[BUG] - Non-interactive login not working at all, interactive login not working on SLURM

[mjacksonhill]

Bug description

I am trying to automate login for some eventual scripting using SLURM. I can only ever get login to work using interactive login (-u -p) from the terminal. I have read some of the other issues on here relating to the refresh/access tokens on compute clusters, but I can't figure out what to change to make this work. Confusingly, it doesn't even work on the login node.

I should also note that all my failed login attempts are preceded by a successful one-- somehow the token is not being imported correctly upon subsequent startups. Is there any way to make this work? Am I missing some key step in storing the token? My sirius configuration directory is the default one.

Steps To Reproduce

Successful Login:

sirius login -u email.address@site.com -p

Unsuccessful Login Case 1 (env variables on login or compute node):

#contents of /home/mat.hill/.sirius.env
SIRIUS_USER=email.address@site.com
SIRIUS_PASS='somepass'

#load env variables
source /home/mat.hill/.sirius.env

#attempt login in interactive shell or on compute node with env variables while still successfully logged in
`sirius login --user-env=SIRIUS_USER --password-env=SIRIUS_PASS`
`srun  -p debug sirius login --user-env=SIRIUS_USER --password-env=SIRIUS_PASS`

results in:

WARNING: No login Found: Not Logged in, No valid refresh token Available
Nov 23, 2024 3:37:58 AM de.unijena.bioinf.ms.frontend.core.ApplicationCore <clinit>
INFO: Web API initialized.
Nov 23, 2024 3:38:08 AM de.unijena.bioinf.ms.frontend.SiriusCLIApplication run
SEVERE: Unexpected Error!
java.lang.RuntimeException: Error when storing refresh token. You may have to re-login.
        at de.unijena.bioinf.ms.frontend.subtools.login.LoginOptions$LoginWorkflow.run(LoginOptions.java:241)
        at de.unijena.bioinf.ms.frontend.Run.compute(Run.java:65)
        at de.unijena.bioinf.ms.frontend.SiriusCLIApplication.run(SiriusCLIApplication.java:148)
        at de.unijena.bioinf.ms.frontend.SiriusCLIApplication.runMain(SiriusCLIApplication.java:77)
        at de.unijena.bioinf.ms.middleware.SiriusMiddlewareApplication.main(SiriusMiddlewareApplication.java:222)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
Caused by: de.unijena.bioinf.auth.LoginException: Not Logged in, No valid refresh token Available
        at de.unijena.bioinf.auth.AuthServices.writeRefreshToken(AuthServices.java:118)
        at de.unijena.bioinf.auth.AuthServices.writeRefreshToken(AuthServices.java:106)
        at de.unijena.bioinf.ms.frontend.subtools.login.LoginOptions$LoginWorkflow.run(LoginOptions.java:239)
        ... 9 more
Caused by: java.lang.IllegalStateException: Cannot save refresh token if user is not logged in.
        ... 12 more

Unsuccessful Login Case 2 (interactive on compute node):

#attempt interactive login on compute node
`srun -p debug --pty sirius login -u email.address@site.com -p`

WARNING: No login Found: Not Logged in, No valid refresh token Available
Nov 23, 2024 3:45:28 AM de.unijena.bioinf.ms.frontend.core.ApplicationCore <clinit>
INFO: Web API initialized.
Enter value for --password (Console password input.): 
Nov 23, 2024 3:45:53 AM de.unijena.bioinf.ms.frontend.subtools.login.LoginOptions$LoginWorkflow run
SEVERE: Could not login to Authentication Server!
java.net.SocketException: Network is unreachable
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Net.java:589)
        at java.base/sun.nio.ch.Net.connect(Net.java:578)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:751)
        at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
        at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
        at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
        at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
        at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
        at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
        at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at com.github.scribejava.httpclient.okhttp.OkHttpHttpClient.doExecute(OkHttpHttpClient.java:128)
        at com.github.scribejava.httpclient.okhttp.OkHttpHttpClient.execute(OkHttpHttpClient.java:101)
        at com.github.scribejava.core.oauth.OAuthService.execute(OAuthService.java:114)
        at com.github.scribejava.core.oauth.OAuth20Service.sendAccessTokenRequestSync(OAuth20Service.java:151)
        at com.github.scribejava.core.oauth.OAuth20Service.getAccessTokenPasswordGrant(OAuth20Service.java:347)
        at de.unijena.bioinf.auth.AuthService.requestAccessTokenPasswordFlow(AuthService.java:163)
        at de.unijena.bioinf.auth.AuthService.login(AuthService.java:257)
        at de.unijena.bioinf.ms.frontend.subtools.login.LoginOptions$LoginWorkflow.run(LoginOptions.java:188)
        at de.unijena.bioinf.ms.frontend.Run.compute(Run.java:65)
        at de.unijena.bioinf.ms.frontend.SiriusCLIApplication.run(SiriusCLIApplication.java:148)
        at de.unijena.bioinf.ms.frontend.SiriusCLIApplication.runMain(SiriusCLIApplication.java:77)
        at de.unijena.bioinf.ms.middleware.SiriusMiddlewareApplication.main(SiriusMiddlewareApplication.java:222)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
        Suppressed: java.net.SocketTimeoutException: Connect timed out
                at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
                at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
                ... 36 more
        Suppressed: java.net.SocketTimeoutException: Connect timed out
                at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546)
                at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
                ... 36 more
        Suppressed: java.net.SocketException: Network is unreachable
                ... 40 more

Nov 23, 2024 3:45:53 AM de.unijena.bioinf.ms.frontend.SiriusCLIApplication run
SEVERE: Unexpected Error!
java.lang.RuntimeException: Error when storing refresh token. You may have to re-login.
        at de.unijena.bioinf.ms.frontend.subtools.login.LoginOptions$LoginWorkflow.run(LoginOptions.java:241)
        at de.unijena.bioinf.ms.frontend.Run.compute(Run.java:65)
        at de.unijena.bioinf.ms.frontend.SiriusCLIApplication.run(SiriusCLIApplication.java:148)
        at de.unijena.bioinf.ms.frontend.SiriusCLIApplication.runMain(SiriusCLIApplication.java:77)
        at de.unijena.bioinf.ms.middleware.SiriusMiddlewareApplication.main(SiriusMiddlewareApplication.java:222)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
Caused by: de.unijena.bioinf.auth.LoginException: Not Logged in, No valid refresh token Available
        at de.unijena.bioinf.auth.AuthServices.writeRefreshToken(AuthServices.java:118)
        at de.unijena.bioinf.auth.AuthServices.writeRefreshToken(AuthServices.java:106)
        at de.unijena.bioinf.ms.frontend.subtools.login.LoginOptions$LoginWorkflow.run(LoginOptions.java:239)
        ... 9 more
Caused by: java.lang.IllegalStateException: Cannot save refresh token if user is not logged in.
        ... 12 more

System Info:

• OS and Version: CentOS Linux 7 (Core)
• SIRIUS Version 6.0.7 (installed via conda)
• Hardware SLURM cluster

[mfleisch]

Hi,
The problem you are experiencing is probably due to a shared user home directory on your compute cluster. SIRIUS persistently stores refresh tokens so that a normal desktop user does not have to re-login each time. These persistent refresh tokens are single-use (for security reasons) and are replaced after each refresh. If this single-use policy is violated, the entire token chain is invalidated and a relogin is required.

Long story short, if a token (default location: ~/.sirius-6.x) is shared between different SIRIUS instances, the token's single use policy is violated and the token chain becomes invalid. Currently this can be resolved by ensuring that each SIRIUS instance uses its own location to store the token. This can be specified using the --workspace parameter.

FYI: The next release will introduce the option to not persist tokens, which will avoid this problem altogether.

Closing this issue. Feel free to reopen if this answer does not solve your problem.

[mjacksonhill]

Hi,

Thanks for the insight into the token mechanism. I had gleaned some of this from reading the threads on other issues, but this definitely helps. I've played with the --workspace parameter but cannot get it to work. Attempting to log in at all from a compute node fails:

sirius login -u email@address -p
sirius login --user-env=SIRIUS_USER --password-env=SIRIUS_PASS

#both give following error

Nov 23, 2024 7:01:10 PM de.unijena.bioinf.ms.frontend.SiriusCLIApplication run
SEVERE: Unexpected Error!
java.lang.RuntimeException: Error when storing refresh token. You may have to re-login.

The remaining option of logging in interactively on a login node and copying the folder to a new location passed as --workspace does not work:

sirius login -u email@address -p
cp -r /home/mat/hill/.sirius-6.0 /some/other/directory/test-workspace

#from compute node
sirius [...] --workspace /some/other/directory/test-workspace

SEVERE: Error when using given refresh_token. Not logged in. Re-login with PW Flow might be necessary