[Bug] Podman discovery not working

[mwangggg]

Current Behavior

When I run sh smoketest.sh, the topology view in Cryostat doesn't show the podman realm.

The logs show:

Sep 01, 2023 6:35:26 PM io.cryostat.core.log.Logger info
INFO: HTTPS service running on https://localhost:8181
Sep 01, 2023 6:35:26 PM io.cryostat.core.log.Logger info
INFO: Deployed io.cryostat.net.HttpServer Verticle [0139d32f-0fa8-4605-a66b-c07f70dc582c]
Sep 01, 2023 6:35:28 PM io.cryostat.core.log.Logger info
INFO: Testing DockerPlatformStrategy Availability via /var/run/docker.sock
Sep 01, 2023 6:35:28 PM io.cryostat.core.log.Logger info
INFO: DockerPlatformStrategy available? false
Sep 01, 2023 6:35:28 PM io.cryostat.core.log.Logger info
INFO: Testing PodmanPlatformStrategy Availability via /run/user/0/podman/podman.sock
Sep 01, 2023 6:35:29 PM org.slf4j.impl.JDK14LoggerAdapter fillCallerData
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2502 ms, time limit is 2000 ms
Sep 01, 2023 6:35:30 PM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
	at io.cryostat.platform.internal.PodmanPlatformStrategy.testPodmanApi(PodmanPlatformStrategy.java:122)
	at io.cryostat.platform.internal.PodmanPlatformStrategy.isAvailable(PodmanPlatformStrategy.java:86)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.cryostat.platform.PlatformModule.provideSelectedPlatformStrategies(PlatformModule.java:99)
	at io.cryostat.platform.PlatformModule_ProvideSelectedPlatformStrategiesFactory.provideSelectedPlatformStrategies(PlatformModule_ProvideSelectedPlatformStrategiesFactory.java:59)
	at io.cryostat.platform.PlatformModule_ProvideSelectedPlatformStrategiesFactory.get(PlatformModule_ProvideSelectedPlatformStrategiesFactory.java:46)
	at io.cryostat.platform.PlatformModule_ProvideSelectedPlatformStrategiesFactory.get(PlatformModule_ProvideSelectedPlatformStrategiesFactory.java:15)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.platform.PlatformModule_ProvidePlatformStrategyFactory.get(PlatformModule_ProvidePlatformStrategyFactory.java:40)
	at io.cryostat.platform.PlatformModule_ProvidePlatformStrategyFactory.get(PlatformModule_ProvidePlatformStrategyFactory.java:13)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.platform.PlatformModule_ProvideAuthManagerFactory.get(PlatformModule_ProvideAuthManagerFactory.java:50)
	at io.cryostat.platform.PlatformModule_ProvideAuthManagerFactory.get(PlatformModule_ProvideAuthManagerFactory.java:16)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.DaggerCryostat_Client$ClientImpl.recordingMetadataLabelsPostHandler(DaggerCryostat_Client.java:1002)
	at io.cryostat.DaggerCryostat_Client$ClientImpl.setOfRequestHandler(DaggerCryostat_Client.java:1471)
	at io.cryostat.DaggerCryostat_Client$ClientImpl.webServer(DaggerCryostat_Client.java:1780)
	at io.cryostat.Cryostat.lambda$start$1(Cryostat.java:75)
	at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onSuccess(FutureImpl.java:262)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
	at io.vertx.core.impl.future.Mapping.onSuccess(Mapping.java:40)
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:86)
	at io.vertx.core.impl.ContextBase.execute(ContextBase.java:225)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
	at io.vertx.core.Promise.complete(Promise.java:66)
	at io.vertx.core.impl.DeploymentManager.lambda$null$4(DeploymentManager.java:209)
	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
	at io.vertx.core.Promise.tryComplete(Promise.java:121)
	at io.vertx.core.Promise.complete(Promise.java:77)
	at io.cryostat.net.HttpServer.lambda$start$0(HttpServer.java:106)
	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
	at io.vertx.core.impl.future.FixedMapping.onSuccess(FixedMapping.java:31)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
	at io.vertx.core.impl.future.FixedMapping.onSuccess(FixedMapping.java:31)
	at io.vertx.core.impl.future.FutureImpl$ListenerArray.onSuccess(FutureImpl.java:262)
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)

Sep 01, 2023 6:35:30 PM io.cryostat.core.log.Logger info
INFO: PodmanPlatformStrategy available? false
Sep 01, 2023 6:35:30 PM org.slf4j.impl.JDK14LoggerAdapter fillCallerData
WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3503 ms, time limit is 2000 ms
Sep 01, 2023 6:35:30 PM io.cryostat.core.log.Logger info
INFO: Selecting configured AuthManager "io.cryostat.net.BasicAuthManager"
Sep 01, 2023 6:35:30 PM io.cryostat.core.log.Logger info
INFO: Local save path for flight recordings set as /opt/cryostat.d/recordings.d

[andrewazores]

Affects 2.4.0 only or 2.3.1 as well?

[andrewazores]

Possibly related to #1629

[andrewazores]

This is actually caused by the same root cause as #1629 . When --cpus is set with a value of 2.0 or less, the JVM detects the Runtime#availableProcessors() value of 2 or 1, which sets the parallelism of the ForkJoinPool.commonPool() which we use in several places, including during application startup to test various platforms' availabilities. When there are fewer than 3 threads we get into a deadlock situation where no worker thread from the pool is available to perform the detection check, so it times out and disables the platform strategy.