Missing required property 'ShardHealthStats.unassignedPrimaryShards' when using client v 8.16 against elasticsearch 7.17.23

[karlney]

Java API client version

8.16.1

Java version

21.0.4

Elasticsearch Version

7.17.23

Problem description

When upgrading from v 8.15.2 of the client to 8.16.1 I start getting the following errors towards my elasticsearch cluster when fetching the cluster health

co.elastic.clients.transport.TransportException: node: http://localhost:38629/, status: 200, [es/cluster.health] Failed to decode response java.util.concurrent.CompletionException: co.elastic.clients.transport.TransportException: node: http://localhost:38629/, status: 200, [es/cluster.health] Failed to decode response at java.base@21.0.3/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) at java.base@21.0.3/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347) at java.base@21.0.3/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636) at java.base@21.0.3/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base@21.0.3/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at app//co.elastic.clients.transport.ElasticsearchTransportBase.lambda$performRequestAsync$0(ElasticsearchTransportBase.java:218) at java.base@21.0.3/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base@21.0.3/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base@21.0.3/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base@21.0.3/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at app//co.elastic.clients.transport.rest_client.RestClientHttpClient$1.onSuccess(RestClientHttpClient.java:116) at app//org.elasticsearch.client.RestClient$FailureTrackingResponseListener.onSuccess(RestClient.java:680) at app//org.elasticsearch.client.RestClient$1.completed(RestClient.java:403) at app//org.elasticsearch.client.RestClient$1.completed(RestClient.java:397) at app//org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) at app//org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:182) at app//org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) at app//org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) at app//org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) at app//org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:87) at app//org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:40) at app//org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at app//org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at app//org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at app//org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at app//org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at app//org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at app//org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583) Caused by: co.elastic.clients.transport.TransportException: node: http://localhost:38629/, status: 200, [es/cluster.health] Failed to decode response at app//co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:409) at app//co.elastic.clients.transport.ElasticsearchTransportBase.getApiResponse(ElasticsearchTransportBase.java:367) at app//co.elastic.clients.transport.ElasticsearchTransportBase.lambda$performRequestAsync$0(ElasticsearchTransportBase.java:212) ... 23 more Caused by: co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch.cluster.health.IndexHealthStats: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ShardHealthStats.unassignedPrimaryShards' (JSON path: indices['document.document-20241025'].shards['0']) (line no=1, column no=739, offset=-1) at app//co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134) at app//co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:353) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:333) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78) at app//co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) at app//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:349) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:333) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:78) at app//co.elastic.clients.json.ObjectDeserializer.deserializeInner(ObjectDeserializer.java:214) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:166) at app//co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:157) at app//co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:77) at app//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) at app//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) at app//co.elastic.clients.transport.ElasticsearchTransportBase.decodeTransportResponse(ElasticsearchTransportBase.java:404) ... 25 more Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ShardHealthStats.unassignedPrimaryShards' at app//co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats.<init>(ShardHealthStats.java:89) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats.<init>(ShardHealthStats.java:64) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats$Builder.build(ShardHealthStats.java:276) at app//co.elastic.clients.elasticsearch.cluster.health.ShardHealthStats$Builder.build(ShardHealthStats.java:191) at app//co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80) at app//co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) at app//co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:349) ... 45 more

Is there some workaround or do we need to wait on a patch?

[l-trotta]

Hello! Unfortunately this kind of things can happen when the client version and the server version start diverging, so this is probably a new field that is mandatory in new server versions, but that old server versions don't have. As the documentation states, the java client is forward compatible, meaning we guarantee its functionalities for versions of the server greater than those of the client, so there's won't be a patch for this.
What you can do as a workaround is disable the check just for that specific call, by performing it inside a try block configured like this:

try (ApiTypeHelper.DisabledChecksHandle h = ApiTypeHelper.DANGEROUS_disableRequiredPropertiesCheck(true)) {
    esClient.healthReport();
}