[Bug] javax.naming.ConfigurationException: Environment variable CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD must be set and non-blank

[Lsq707]

Current Behavior

I try to run the run-docker.sh on macOS with docker 24.0.7 but got following error:

INFO: Local config path set as /opt/cryostat.d/conf.d
Jan 01, 2024 11:10:55 AM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
java.lang.RuntimeException: javax.naming.ConfigurationException: Environment variable CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD must be set and non-blank
	at io.cryostat.storage.StorageModule.provideEntityManagerFactory(StorageModule.java:73)
	at io.cryostat.storage.StorageModule_ProvideEntityManagerFactoryFactory.provideEntityManagerFactory(StorageModule_ProvideEntityManagerFactoryFactory.java:44)
	at io.cryostat.storage.StorageModule_ProvideEntityManagerFactoryFactory.get(StorageModule_ProvideEntityManagerFactoryFactory.java:35)
	at io.cryostat.storage.StorageModule_ProvideEntityManagerFactoryFactory.get(StorageModule_ProvideEntityManagerFactoryFactory.java:13)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.storage.StorageModule_ProvideEntityManagerFactory.get(StorageModule_ProvideEntityManagerFactory.java:35)
	at io.cryostat.storage.StorageModule_ProvideEntityManagerFactory.get(StorageModule_ProvideEntityManagerFactory.java:13)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.discovery.DiscoveryModule_ProvidePluginInfoDaoFactory.get(DiscoveryModule_ProvidePluginInfoDaoFactory.java:43)
	at io.cryostat.discovery.DiscoveryModule_ProvidePluginInfoDaoFactory.get(DiscoveryModule_ProvidePluginInfoDaoFactory.java:14)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at io.cryostat.discovery.DiscoveryModule_ProvideDiscoveryStorageFactory.get(DiscoveryModule_ProvideDiscoveryStorageFactory.java:80)
	at io.cryostat.discovery.DiscoveryModule_ProvideDiscoveryStorageFactory.get(DiscoveryModule_ProvideDiscoveryStorageFactory.java:22)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at dagger.internal.DelegateFactory.get(DelegateFactory.java:36)
	at io.cryostat.configuration.ConfigurationModule_ProvideCredentialsManagerFactory.get(ConfigurationModule_ProvideCredentialsManagerFactory.java:68)
	at io.cryostat.configuration.ConfigurationModule_ProvideCredentialsManagerFactory.get(ConfigurationModule_ProvideCredentialsManagerFactory.java:20)
	at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
	at dagger.internal.DelegateFactory.get(DelegateFactory.java:36)
	at io.cryostat.DaggerCryostat_Client$ClientImpl.credentialsManager(DaggerCryostat_Client.java:1753)
	at io.cryostat.Cryostat.start(Cryostat.java:58)
	at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:195)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
	at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: javax.naming.ConfigurationException: Environment variable CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD must be set and non-blank
	... 33 more

^C+ dockerCleanUp
+ '[' -n cryostat ']'
+ docker rm -f cryostat
Error response from daemon: No such container: cryostat
+ docker network rm -f cryostat-docker
cryostat-docker

Do i need to set CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD and how ?
Or is there any issues supporting macOS ？

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- OS: macOS 13.6.1
- Environment: docker 24.0.7
- Version: main branch

Anything else?

No response

[andrewazores]

Hi @Lsq707 !

1. We don't have any MacOS users on our development team and we primarily target deployment on Linux, but so long as you have a working OCI container environment and a POSIX filesystem, things should work.
2. This environment variable is required, but the run.sh and run-docker.sh scripts are supposed to set it to a random value automatically if you don't supply one:

https://github.com/cryostatio/cryostat/blob/c563279c465e9543e81ffc3d49923b2908019813/run-docker.sh#L64

https://github.com/cryostatio/cryostat/blob/c563279c465e9543e81ffc3d49923b2908019813/run-docker.sh#L168

I wonder if this is some shell difference and that variable isn't getting set. What happens if you try setting it like this?

$ CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD=somesecret sh run-docker.sh

[Lsq707]

Hi @andrewazores,

Thank you for answering.

After executing $ CRYOSTAT_JMX_CREDENTIALS_DB_PASSWORD=somesecret sh run-docker.sh
i got this error

Jan 02, 2024 1:55:44 AM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://2e93ceb98765:9091/jmxrmi closed
Jan 02, 2024 1:55:44 AM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
org.openjdk.jmc.rjmx.ConnectionException caused by java.lang.SecurityException: Authentication failed! Invalid username or password
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:345)
	at io.cryostat.core.net.JFRJMXConnection.attemptConnect(JFRJMXConnection.java:340)
	at io.cryostat.core.net.JFRJMXConnection.connect(JFRJMXConnection.java:296)
	at io.cryostat.core.net.JFRJMXConnection.getJvmIdentifier(JFRJMXConnection.java:149)
	at io.cryostat.core.net.JFRConnection.getJvmId(JFRConnection.java:52)
	at io.cryostat.net.TargetConnectionManager.lambda$executeConnectedTaskAsync$2(TargetConnectionManager.java:135)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.SecurityException: Authentication failed! Invalid username or password
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticationFailure(JMXPluggableAuthenticator.java:231)
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:208)
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:233)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:587)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:705)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:704)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
	at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:304)
	at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:280)
	at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:165)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
	at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:575)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:560)
	at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:338)
	... 10 more
Caused by: javax.security.auth.login.FailedLoginException: Invalid username or password
	at java.management/com.sun.jmx.remote.security.FileLoginModule.attemptAuthentication(FileLoginModule.java:460)
	at java.management/com.sun.jmx.remote.security.FileLoginModule.login(FileLoginModule.java:309)
	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at java.management/com.sun.jmx.remote.security.JMXPluggableAuthenticator.authenticate(JMXPluggableAuthenticator.java:195)
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$AccessFileCheckerAuthenticator.authenticate(ConnectorBootstrap.java:233)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:231)
	at java.management.rmi/javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:198)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:587)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:828)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:705)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:704)
	... 3 more

Hibernate: 
    /* select
        generatedAlias0 
    from
        PluginInfo as generatedAlias0 */ select
            plugininfo0_.id as id1_1_,
            plugininfo0_.callback as callback2_1_,
            plugininfo0_.realm as realm3_1_,
            plugininfo0_.subtree as subtree4_1_ 
        from
            PluginInfo plugininfo0_
Jan 02, 2024 1:55:44 AM io.cryostat.core.log.Logger info
INFO: Creating connection for service:jmx:rmi:///jndi/rmi://2e93ceb98765:9091/jmxrmi
Jan 02, 2024 1:55:44 AM io.cryostat.core.log.Logger info
INFO: Removing cached connection for service:jmx:rmi:///jndi/rmi://2e93ceb98765:9091/jmxrmi: EXPLICIT
Jan 02, 2024 1:55:44 AM io.cryostat.core.log.Logger info
INFO: connection attempt failed.

I was wondering if by running the run-docker.sh script, I should be able to access the application via localhost:8181 ？

Btw, I was referring to this blog: https://developers.redhat.com/blog/2021/01/25/introduction-to-containerjfr-jdk-flight-recorder-for-containers# but it seems outdated. Is there any new user guideline available?

[andrewazores]

That error in the log is actually not a problem - what's happening is Cryostat is successfully running, and sees itself on the network, but has JMX authentication enabled and so it fails to connect to itself.

You should be able to get to the Cryostat UI at https://localhost:8181.

That blog post is indeed out of date. Please check https://cryostat.io for the latest documentation.

[Lsq707]

HI @andrewazores
Is the somketest open for public ?

i want to try the sample applications but have no authentication

[andrewazores]

@Lsq707 sure - the run.sh and smoketest.sh spin up an instance on your local machine :-) there are no "public" Cryostat instances, it is always a self-hosted deployment like this.

The test credentials are defined within smoketest.sh - you may log in with username user and password pass.

[Lsq707]

Thanks @andrewazores !

But while i am trying to view a recording in grafana http://grafana:3000/ in smoketest-docker.sh i got this error, the recoding was uploaded successfully.

• http://grafana:3000/ page:
  
• Recoding uploaded
  
• The console outputs:
  
  {"meta":{"type":"text/plain","status":"Unauthorized"},"data":{"reason":"HTTP Authorization Failure"}}

Is this because the firewall block ?

[andrewazores]

Can you try manually opening http://localhost:3000 instead of http://grafana:3000? This looks to me like the smoketest script makes an assumption about containers' names being resolvable as hostnames, and that doesn't look like it's true on your system.

I don't know if there is any "firewall block" involved - that would depend on what firewall you have configured on your machine.

---

It's probably worth mentioning that the run.sh/run-docker.sh/smoketest.sh scripts you find in this repository are meant for developers' use, really. As an end user the intended deployment method is https://github.com/cryostatio/cryostat-operator or https://github.com/cryostatio/cryostat-helm , both of which presume your applications live in Kubernetes or OpenShift. If you want to drop Cryostat in to a Docker (or Podman) system then it's probably best to use one of these as inspiration and build out your own deployment of Cryostat tailored to your machines and environment, rather than trying to use these development scripts that have implicit assumptions around Linux, and primarily Podman instead of Docker as well.

[Lsq707]

Able to access via http://0.0.0.0:3000

Ok thanks for the advice