Failing to start service after upgrading from 2.6.2 to 2.6.5

[rnotrifork]

Description

Registry
Version: 2.6.5
Persistence type: kafkasql

Environment

Kubernetes v1.30.4-gke.1348000

Steps to Reproduce

1. Deploy apicurio-registry to kubernetes cluster.
2. Configure the pod to use readOnlyRootFilesystem: true
3. Upgrade apicurio/apicurio-registry-kafkasql image from 2.6.2 to 2.6.5

Expected vs Actual Behaviour

Expected service to spin up and get ready.
Actual behaviour is that it end in a CrashLoop.

Logs

INFO exec -a "java" java  -cp "." -jar /deployments/apicurio-registry-storage-kafkasql-2.6.5.Final-runner.jar 
INFO running in /deployments
__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2024-10-09 12:56:17 INFO <> [null] (main) ---------------------------------------------------------------------
2024-10-09 12:56:17 INFO <> [null] (main) Legacy REST API date formats enabled (this is currently the default).
2024-10-09 12:56:17 INFO <> [null] (main) 
2024-10-09 12:56:17 INFO <> [null] (main) For maximum compatibility and to ease upgrades from older versions
2024-10-09 12:56:17 INFO <> [null] (main) of Registry, the date format used in the REST API is not compliant
2024-10-09 12:56:17 INFO <> [null] (main) with OpenAPI standards (due to a bug in older versions).  Please
2024-10-09 12:56:17 INFO <> [null] (main) make sure you upgrade all of your client applications to use the
2024-10-09 12:56:17 INFO <> [null] (main) latest client version.  The next release will fix the date format
2024-10-09 12:56:17 INFO <> [null] (main) bug, which will result in older clients no longer being compatible
2024-10-09 12:56:17 INFO <> [null] (main) with the REST API.
2024-10-09 12:56:17 INFO <> [null] (main) 
2024-10-09 12:56:17 INFO <> [null] (main) If you would like to fix the date format bug in THIS version of
2024-10-09 12:56:17 INFO <> [null] (main) Registry (great!) please set the following ENV variable + value:
2024-10-09 12:56:17 INFO <> [null] (main) 
2024-10-09 12:56:17 INFO <> [null] (main) REGISTRY_APIS_V2_DATE_FORMAT=yyyy-MM-dd'T'HH:mm:ss'Z'
2024-10-09 12:56:17 INFO <> [null] (main) 
2024-10-09 12:56:17 INFO <> [null] (main) Doing this will result in a REST API that is OpenAPI compliant, but
2024-10-09 12:56:17 INFO <> [null] (main) please remember to upgrade all your client applications first!
2024-10-09 12:56:17 INFO <> [null] (main) ---------------------------------------------------------------------
2024-10-09 12:56:39 WARN <> [null] (main) The 'quarkus.log.console.color' config property is deprecated and should not be used anymore
2024-10-09 12:57:09 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) SqlRegistryStorage constructed successfully.  JDBC URL: jdbc:h2:mem:registry_db
2024-10-09 12:57:10 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Checking to see if the DB is initialized.
2024-10-09 12:57:12 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Database not initialized.
2024-10-09 12:57:13 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Initializing the Apicurio Registry database.
2024-10-09 12:57:13 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) 	Database type: h2
2024-10-09 12:57:14 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Checking to see if the DB is up-to-date.
2024-10-09 12:57:14 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Build's DB version is 15
2024-10-09 12:57:15 INFO <_> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage] (main) Using Kafka-SQL artifactStore.
2024-10-09 12:57:15 INFO <_> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage] (main) SQL store initialized, starting consumer thread.
2024-10-09 12:57:15 INFO <_> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage] (main) Starting KSQL consumer thread on topic: _kafkasql-journal
2024-10-09 12:57:15 INFO <_> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage] (main) Bootstrap servers: kafka-brokers.kafka.svc:9092
2024-10-09 12:57:15 INFO <_> [io.apicurio.registry.utils.kafka.AsyncProducer] (main) Creating new resilient producer.
2024-10-09 12:57:16 ERROR <> [io.quarkus.runtime.ApplicationLifecycleManager] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: java.lang.RuntimeException: Cannot invoke method: io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage#initialize on io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Subclass@1de08775
	at io.quarkus.arc.impl.Reflections.invokeMethod(Reflections.java:188)
	at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Bean.doCreate(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Bean.create(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c101(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Observer_onStart_GvKzwrQiqcFWu0a7W8wCqLRzan8.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
	at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155)
	at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:106)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
	... 7 more
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.arc.impl.Reflections.invokeMethod(Reflections.java:182)
	... 24 more
Caused by: java.lang.ExceptionInInitializerError: Cannot unpack libzstd-jni-1.5.6-3: Read-only file system
	at java.base/java.io.UnixFileSystem.createFileExclusively(Native Method)
	at java.base/java.io.File.createTempFile(File.java:2170)
	at com.github.luben.zstd.util.Native.load(Native.java:130)
	at com.github.luben.zstd.util.Native.load(Native.java:86)
	at com.github.luben.zstd.Zstd.<clinit>(Zstd.java:12)
	at org.apache.kafka.common.compress.ZstdCompression.<clinit>(ZstdCompression.java:41)
	at org.apache.kafka.clients.producer.ProducerConfig.<clinit>(ProducerConfig.java:384)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:295)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:322)
	at io.apicurio.registry.utils.kafka.AsyncProducer.getProducer(AsyncProducer.java:86)
	at io.apicurio.registry.utils.kafka.AsyncProducer.apply(AsyncProducer.java:61)
	at io.apicurio.registry.utils.kafka.AsyncProducer.apply(AsyncProducer.java:41)
	at io.apicurio.registry.utils.kafka.KafkaSqlFactory_ProducerMethod_createKafkaProducer_dVZIBgkIj7B-jknZHrjTklwaR1Q_ClientProxy.apply(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlSubmitter.send(KafkaSqlSubmitter.java:69)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlSubmitter_Subclass.send$$superforward(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlSubmitter_Subclass$$function$$1.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.apicurio.common.apps.logging.LoggingInterceptor.logMethodEntry(LoggingInterceptor.java:53)
	at io.apicurio.common.apps.logging.LoggingInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlSubmitter_Subclass.send(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlSubmitter_ClientProxy.send(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.upgrade.KafkaSqlUpgraderManager.init(KafkaSqlUpgraderManager.java:176)
	at io.apicurio.registry.storage.impl.kafkasql.upgrade.KafkaSqlUpgraderManager_ClientProxy.init(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage.startConsumerThread(KafkaSqlRegistryStorage.java:268)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage.handleSqlStorageEvent(KafkaSqlRegistryStorage.java:199)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Subclass.handleSqlStorageEvent$$superforward(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Subclass$$function$$82.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.apicurio.common.apps.logging.LoggingInterceptor.logMethodEntry(LoggingInterceptor.java:53)
	at io.apicurio.common.apps.logging.LoggingInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.apicurio.registry.metrics.health.readiness.PersistenceTimeoutReadinessInterceptor.intercept(PersistenceTimeoutReadinessInterceptor.java:32)
	at io.apicurio.registry.metrics.health.readiness.PersistenceTimeoutReadinessInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)
	at io.apicurio.registry.metrics.health.liveness.PersistenceExceptionLivenessInterceptor.intercept(PersistenceExceptionLivenessInterceptor.java:28)
	at io.apicurio.registry.metrics.health.liveness.PersistenceExceptionLivenessInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.apicurio.registry.metrics.StorageMetricsInterceptor.intercept(StorageMetricsInterceptor.java:48)
	at io.apicurio.registry.metrics.StorageMetricsInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Subclass.handleSqlStorageEvent(Unknown Source)
	at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Observer_handleSqlStorageEvent_aD0SpxuOCT_mfMReL5VVVfvdQGw.notify(Unknown Source)
	at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
	at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:324)
	at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
	at io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage.initialize(AbstractSqlRegistryStorage.java:213)
	... 29 more

[apicurio-bot]

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.

[carlesarnal]

I'm afraid this is apparently a known issue with the kafka clients version 3.8 when Kafka runs in a (relatively) restricted environment. Apart from some existing workarounds, there's not much we can do (I don't have any other ideas apart from revert to a previous version) until they release a new version.