Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use OAuth client id with a https (secure) redirect url #494

Closed
sevalsixarci opened this issue Nov 1, 2024 · 34 comments
Closed

Use OAuth client id with a https (secure) redirect url #494

sevalsixarci opened this issue Nov 1, 2024 · 34 comments
Assignees
Labels
bug Something isn't working

Comments

@sevalsixarci
Copy link

Hi,
I'm exploring the project to replicate my current email setup which use mu, mbsync, msmtp, and pizauth (OAuth token fetcher/daemon). I compiled himalaya from git with OAuth2 and keyring features enabled (the repository's head at the time of compilation was commit 92814d6).

The issue is that I can read mail from account using standard IMAP, but OAuth authentication fails for my Outlook work accounts. Since I am authenticating as Thunderbird (authorized with HTTPS redirects on localhost), attempting to fetch mail gives the following error:

To complete your OAuth 2.0 setup, click on the following link:

https://login.microsoftonline.com/common/oauth2/v2.0/authorize?response_type=code&client_id=XXXXXXXXXX&state=o2cFdGrpKm_qDCcEfM-hMw&code_challenge=WK7Q_eEk1UngQ9Cj4d-y35pC3B-Xwxs-p7HgEydWFtE&code_challenge_method=S256&redirect_uri=https%3A%2F%2Flocalhost%3A49152&scope=https%3A%2F%2Foutlook.office.com%2FIMAP.AccessAsUser.All
Error:
0: cannot wait for oauth2 redirection error
1: stream did not contain valid UTF-8

Config snippet:

backend.type = "imap"
backend.auth.type = "oauth2"
backend.auth.method = "xoauth2"
backend.host = "outlook.office365.com"
backend.port = 993
backend.login = "XXXXXXXXXXXXXX"
backend.auth.client-id = "XXXXXXXXXXXXXX"
backend.auth.auth-url = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize"
backend.auth.token-url = "https://login.microsoftonline.com/common/oauth2/v2.0/token"
backend.auth.pkce = true
backend.auth.scope = "https://outlook.office.com/IMAP.AccessAsUser.All"
backend.auth.redirect-scheme = "https"
backend.auth.redirect-host = "localhost"

I also tried to bypass this by reusing the OAuth tokens from pizauth, with these config lines (but I think I may have misunderstood some of the config keys):

backend.auth.access-token.cmd = "pizauth show accountXXXX"
backend.auth.refresh-token.cmd = "pizauth refresh accountXXXX" 

Some logs:

djus@satellite:~/dev$ himalaya account configure accountXXXX --trace
2024-11-01T18:55:54.971161Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-11-01T18:55:54.972197Z  INFO himalaya::account::command::configure: executing configure account command
2024-11-01T18:55:54.972255Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXX-imap-oauth2-client-secret, and no target    
2024-11-01T18:55:54.972278Z DEBUG keyring: created entry SsCredential { attributes: {"username": "accountXXXX-imap-oauth2-client-secret", "target": "default", "service": "himalaya-cli", "application": "rust-keyring"}, label: "accountXXXX-imap-oauth2-client-secret@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.972294Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXX-imap-oauth2-access-token, and no target    
2024-11-01T18:55:54.972302Z DEBUG keyring: created entry SsCredential { attributes: {"target": "default", "username": "accountXXXX-imap-oauth2-access-token", "service": "himalaya-cli", "application": "rust-keyring"}, label: "accountXXXX-imap-oauth2-access-token@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.972311Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXX-imap-oauth2-refresh-token, and no target    
2024-11-01T18:55:54.972320Z DEBUG keyring: created entry SsCredential { attributes: {"application": "rust-keyring", "target": "default", "service": "himalaya-cli", "username": "accountXXXX-imap-oauth2-refresh-token"}, label: "accountXXXX-imap-oauth2-refresh-token@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.972330Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXX-smtp-oauth2-client-secret, and no target    
2024-11-01T18:55:54.972338Z DEBUG keyring: created entry SsCredential { attributes: {"username": "accountXXXX-smtp-oauth2-client-secret", "target": "default", "application": "rust-keyring", "service": "himalaya-cli"}, label: "accountXXXX-smtp-oauth2-client-secret@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.972351Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXX-smtp-oauth2-access-token, and no target    
2024-11-01T18:55:54.972362Z DEBUG keyring: created entry SsCredential { attributes: {"service": "himalaya-cli", "username": "accountXXXX-smtp-oauth2-access-token", "target": "default", "application": "rust-keyring"}, label: "accountXXXX-smtp-oauth2-access-token@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.972372Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXX-smtp-oauth2-refresh-token, and no target    
2024-11-01T18:55:54.972382Z DEBUG keyring: created entry SsCredential { attributes: {"service": "himalaya-cli", "target": "default", "application": "rust-keyring", "username": "accountXXXX-smtp-oauth2-refresh-token"}, label: "accountXXXX-smtp-oauth2-refresh-token@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.972393Z DEBUG keyring: get keyring secret key="accountXXXX-imap-oauth2-access-token"
2024-11-01T18:55:54.972583Z DEBUG keyring: get password from entry SsCredential { attributes: {"target": "default", "username": "accountXXXX-imap-oauth2-access-token", "service": "himalaya-cli", "application": "rust-keyring"}, label: "accountXXXX-imap-oauth2-access-token@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.973015Z TRACE perform: zbus::connection::handshake::client: Initializing
2024-11-01T18:55:54.973044Z TRACE perform:authenticate: zbus::connection::handshake::client: Trying EXTERNAL mechanism
2024-11-01T18:55:54.973122Z TRACE perform:authenticate:write_command{command=Auth(Some(External), Some([49, 48, 48, 48]))}:write_commands{commands=[Auth(Some(External), Some([49, 48, 48, 48]))] extra_bytes=None}: zbus::connection::handshake::common: Wrote all commands
2024-11-01T18:55:54.973173Z TRACE perform:authenticate:read_command:read_commands{n_commands=1}: zbus::connection::handshake::common: Reading OK 1ed58543edcf355b31594b21aa039eab

2024-11-01T18:55:54.973202Z TRACE perform:authenticate: zbus::connection::handshake::client: Received OK from server
2024-11-01T18:55:54.973328Z TRACE perform:send_secondary_commands{challenge_response=None}:write_commands{commands=[NegotiateUnixFD, Begin] extra_bytes=Some([108, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 109, 0, 0, 0, 1, 1, 111, 0, 21, 0, 0, 0, 47, 111, 114, 103, 47, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 47, 68, 66, 117, 115, 0, 0, 0, 3, 1, 115, 0, 5, 0, 0, 0, 72, 101, 108, 108, 111, 0, 0, 0, 6, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0, 2, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0])}: zbus::connection::handshake::common: Wrote all commands
2024-11-01T18:55:54.973404Z TRACE perform:receive_secondary_responses{expected_n_responses=1}:read_commands{n_commands=1}: zbus::connection::handshake::common: Reading AGREE_UNIX_FD

2024-11-01T18:55:54.973424Z TRACE perform: zbus::connection::handshake::client: Handshake done
2024-11-01T18:55:54.978136Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 2, sender: UniqueName(":1.265"), path: ObjectPath("/org/freedesktop/secrets"), iface: InterfaceName("org.freedesktop.Secret.Service"), member: MemberName("OpenSession"), body: Signature("sv"), fds: [] }
2024-11-01T18:55:54.978174Z TRACE zbus::connection::socket: Sent message with serial: 2
2024-11-01T18:55:54.978192Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.978278Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: Signal, serial: 4294967295, sender: UniqueName("org.freedesktop.DBus"), path: ObjectPath("/org/freedesktop/DBus"), iface: InterfaceName("org.freedesktop.DBus"), member: MemberName("NameAcquired"), body: Signature("s"), fds: [] }
2024-11-01T18:55:54.978298Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.978309Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: Signal, serial: 4294967295, sender: UniqueName("org.freedesktop.DBus"), path: ObjectPath("/org/freedesktop/DBus"), iface: InterfaceName("org.freedesktop.DBus"), member: MemberName("NameAcquired"), body: Signature("s"), fds: [] })
2024-11-01T18:55:54.978320Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.982051Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2081, sender: UniqueName(":1.41"), reply-serial: 2, body: Signature("vo"), fds: [] }
2024-11-01T18:55:54.982078Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.982093Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2081, sender: UniqueName(":1.41"), reply-serial: 2, body: Signature("vo"), fds: [] })
2024-11-01T18:55:54.982104Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.986393Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 3, sender: UniqueName(":1.265"), path: ObjectPath("/org/freedesktop/secrets"), iface: InterfaceName("org.freedesktop.Secret.Service"), member: MemberName("SearchItems"), body: Signature("a{ss}"), fds: [] }
2024-11-01T18:55:54.986422Z TRACE zbus::connection::socket: Sent message with serial: 3
2024-11-01T18:55:54.986794Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2082, sender: UniqueName(":1.41"), reply-serial: 3, body: Signature("aoao"), fds: [] }
2024-11-01T18:55:54.986817Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.986831Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2082, sender: UniqueName(":1.41"), reply-serial: 3, body: Signature("aoao"), fds: [] })
2024-11-01T18:55:54.986842Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.986966Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 4, sender: UniqueName(":1.265"), path: ObjectPath("/org/freedesktop/secrets"), iface: InterfaceName("org.freedesktop.Secret.Service"), member: MemberName("ReadAlias"), body: Signature("s"), fds: [] }
2024-11-01T18:55:54.986988Z TRACE zbus::connection::socket: Sent message with serial: 4
2024-11-01T18:55:54.987509Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2083, sender: UniqueName(":1.41"), reply-serial: 4, body: Signature("o"), fds: [] }
2024-11-01T18:55:54.987529Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.987541Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2083, sender: UniqueName(":1.41"), reply-serial: 4, body: Signature("o"), fds: [] })
2024-11-01T18:55:54.987550Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.987676Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 5, sender: UniqueName(":1.265"), path: ObjectPath("/org/freedesktop/secrets/collection/kdewallet"), iface: InterfaceName("org.freedesktop.Secret.Collection"), member: MemberName("SearchItems"), body: Signature("a{ss}"), fds: [] }
2024-11-01T18:55:54.987698Z TRACE zbus::connection::socket: Sent message with serial: 5
2024-11-01T18:55:54.988061Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2084, sender: UniqueName(":1.41"), reply-serial: 5, body: Signature("ao"), fds: [] }
2024-11-01T18:55:54.988082Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.988096Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2084, sender: UniqueName(":1.41"), reply-serial: 5, body: Signature("ao"), fds: [] })
2024-11-01T18:55:54.988106Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.988574Z DEBUG keyring: find keyring secret key="accountXXXX-imap-oauth2-client-secret"
2024-11-01T18:55:54.988650Z DEBUG keyring: get password from entry SsCredential { attributes: {"username": "accountXXXX-imap-oauth2-client-secret", "target": "default", "service": "himalaya-cli", "application": "rust-keyring"}, label: "accountXXXX-imap-oauth2-client-secret@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-01T18:55:54.988755Z TRACE perform: zbus::connection::handshake::client: Initializing
2024-11-01T18:55:54.988768Z TRACE perform:authenticate: zbus::connection::handshake::client: Trying EXTERNAL mechanism
2024-11-01T18:55:54.988807Z TRACE perform:authenticate:write_command{command=Auth(Some(External), Some([49, 48, 48, 48]))}:write_commands{commands=[Auth(Some(External), Some([49, 48, 48, 48]))] extra_bytes=None}: zbus::connection::handshake::common: Wrote all commands
2024-11-01T18:55:54.988970Z TRACE perform:authenticate:read_command:read_commands{n_commands=1}: zbus::connection::handshake::common: Reading OK 1ed58543edcf355b31594b21aa039eab

2024-11-01T18:55:54.988994Z TRACE perform:authenticate: zbus::connection::handshake::client: Received OK from server
2024-11-01T18:55:54.989091Z TRACE perform:send_secondary_commands{challenge_response=None}:write_commands{commands=[NegotiateUnixFD, Begin] extra_bytes=Some([108, 1, 0, 1, 0, 0, 0, 0, 6, 0, 0, 0, 109, 0, 0, 0, 1, 1, 111, 0, 21, 0, 0, 0, 47, 111, 114, 103, 47, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 47, 68, 66, 117, 115, 0, 0, 0, 3, 1, 115, 0, 5, 0, 0, 0, 72, 101, 108, 108, 111, 0, 0, 0, 6, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0, 2, 1, 115, 0, 20, 0, 0, 0, 111, 114, 103, 46, 102, 114, 101, 101, 100, 101, 115, 107, 116, 111, 112, 46, 68, 66, 117, 115, 0, 0, 0, 0])}: zbus::connection::handshake::common: Wrote all commands
2024-11-01T18:55:54.989154Z TRACE perform:receive_secondary_responses{expected_n_responses=1}:read_commands{n_commands=1}: zbus::connection::handshake::common: Reading AGREE_UNIX_FD

2024-11-01T18:55:54.989172Z TRACE perform: zbus::connection::handshake::client: Handshake done
2024-11-01T18:55:54.993651Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 7, sender: UniqueName(":1.266"), path: ObjectPath("/org/freedesktop/secrets"), iface: InterfaceName("org.freedesktop.Secret.Service"), member: MemberName("OpenSession"), body: Signature("sv"), fds: [] }
2024-11-01T18:55:54.993683Z TRACE zbus::connection::socket: Sent message with serial: 7
2024-11-01T18:55:54.993699Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.993777Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: Signal, serial: 4294967295, sender: UniqueName("org.freedesktop.DBus"), path: ObjectPath("/org/freedesktop/DBus"), iface: InterfaceName("org.freedesktop.DBus"), member: MemberName("NameAcquired"), body: Signature("s"), fds: [] }
2024-11-01T18:55:54.993798Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.993808Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: Signal, serial: 4294967295, sender: UniqueName("org.freedesktop.DBus"), path: ObjectPath("/org/freedesktop/DBus"), iface: InterfaceName("org.freedesktop.DBus"), member: MemberName("NameAcquired"), body: Signature("s"), fds: [] })
2024-11-01T18:55:54.993822Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:54.997104Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2087, sender: UniqueName(":1.41"), reply-serial: 7, body: Signature("vo"), fds: [] }
2024-11-01T18:55:54.997128Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:54.997136Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2087, sender: UniqueName(":1.41"), reply-serial: 7, body: Signature("vo"), fds: [] })
2024-11-01T18:55:54.997147Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:55.001346Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 8, sender: UniqueName(":1.266"), path: ObjectPath("/org/freedesktop/secrets"), iface: InterfaceName("org.freedesktop.Secret.Service"), member: MemberName("SearchItems"), body: Signature("a{ss}"), fds: [] }
2024-11-01T18:55:55.001373Z TRACE zbus::connection::socket: Sent message with serial: 8
2024-11-01T18:55:55.001698Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2088, sender: UniqueName(":1.41"), reply-serial: 8, body: Signature("aoao"), fds: [] }
2024-11-01T18:55:55.001719Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:55.001725Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2088, sender: UniqueName(":1.41"), reply-serial: 8, body: Signature("aoao"), fds: [] })
2024-11-01T18:55:55.001733Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
2024-11-01T18:55:55.001847Z TRACE zbus::connection::socket: Sending message: Msg { type: MethodCall, serial: 9, sender: UniqueName(":1.266"), path: ObjectPath("/org/freedesktop/secrets/collection/kdewallet/2"), iface: InterfaceName("org.freedesktop.Secret.Item"), member: MemberName("GetSecret"), body: Signature("o"), fds: [] }
2024-11-01T18:55:55.001866Z TRACE zbus::connection::socket: Sent message with serial: 9
2024-11-01T18:55:55.002179Z TRACE socket reader: zbus::connection::socket_reader: Message received on the socket: Msg { type: MethodReturn, serial: 2089, sender: UniqueName(":1.41"), reply-serial: 9, body: Signature("(oayays)"), fds: [] }
2024-11-01T18:55:55.002202Z TRACE socket reader: zbus::connection::socket_reader: Error broadcasting message to stream for `None`: SendError(..)
2024-11-01T18:55:55.002209Z TRACE socket reader: zbus::connection::socket_reader: Broadcasted to all streams: Ok(Msg { type: MethodReturn, serial: 2089, sender: UniqueName(":1.41"), reply-serial: 9, body: Signature("(oayays)"), fds: [] })
2024-11-01T18:55:55.002218Z TRACE socket reader: zbus::connection::socket_reader: Waiting for message on the socket..
To complete your OAuth 2.0 setup, click on the following link:

https://login.microsoftonline.com/common/oauth2/v2.0/authorize?response_type=code&client_id=XXXXXXXXXX&state=o2cFdGrpKm_qDCcEfM-hMw&code_challenge=WK7Q_eEk1UngQ9Cj4d-y35pC3B-Xwxs-p7HgEydWFtE&code_challenge_method=S256&redirect_uri=https%3A%2F%2Flocalhost%3A49152&scope=https%3A%2F%2Foutlook.office.com%2FIMAP.AccessAsUser.All
Error: 
   0: cannot wait for oauth2 redirection error
   1: stream did not contain valid UTF-8

Location:
   /home/djus/.cargo/git/checkouts/himalaya-1e70121402046f0a/92814d6/src/account/command/configure.rs:79

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 3 frames hidden ⋮                               
   4: himalaya::account::command::configure::AccountConfigureCommand::execute::{{closure}}::h89b30b4e2fb702d7
      at <unknown source file>:<unknown line>
   5: himalaya::cli::HimalayaCommand::execute::{{closure}}::hbe84badc29564940
      at <unknown source file>:<unknown line>
   6: tokio::runtime::park::CachedParkThread::block_on::hd6b654ade072cf0d
      at <unknown source file>:<unknown line>
   7: tokio::runtime::context::runtime::enter_runtime::hde46c34d4f1a400f
      at <unknown source file>:<unknown line>
   8: tokio::runtime::runtime::Runtime::block_on::habc584feb0f754d1
      at <unknown source file>:<unknown line>
   9: himalaya::main::hb8f6325a19ce1083
      at <unknown source file>:<unknown line>
  10: std::sys::backtrace::__rust_begin_short_backtrace::hc3ace97b884b8485
      at <unknown source file>:<unknown line>
  11: std::rt::lang_start::{{closure}}::h778edc4c447b525b
      at <unknown source file>:<unknown line>
  12: std::rt::lang_start_internal::h5e7c81cecd7f0954
      at <unknown source file>:<unknown line>
  13: main<unknown>
      at <unknown source file>:<unknown line>
  14: __libc_start_call_main<unknown>
      at <unknown source file>:<unknown line>
  15: __libc_start_main@GLIBC_2.2.5<unknown>
      at <unknown source file>:<unknown line>
  16: _start<unknown>
      at <unknown source file>:<unknown line>
@soywod
Copy link
Member

soywod commented Nov 5, 2024

Sorry for the delay. Does the error displays straight, or after a delay? Do you have time to click on the link? I tried recently and the Outlook flow worked as expected for me. I will give another shot in the nearest day and let you know.

@soywod soywod self-assigned this Nov 5, 2024
@soywod soywod added the bug Something isn't working label Nov 5, 2024
@soywod soywod added this to Pimalaya Nov 5, 2024
@soywod soywod moved this to In Progress in Pimalaya Nov 5, 2024
@soywod
Copy link
Member

soywod commented Nov 5, 2024

Oh wait I just noticed the redirect scheme with https, it may be the issue. The spawn server is really basic and may not support https in fact.

@soywod
Copy link
Member

soywod commented Nov 5, 2024

I also tried to bypass this by reusing the OAuth tokens from pizauth, with these config lines (but I think I may have misunderstood some of the config keys):

backend.auth.access-token.cmd = "pizauth show accountXXXX"
backend.auth.refresh-token.cmd = "pizauth refresh accountXXXX" 

Those are shell commands to retrieve the access token or the refresh token. So pizauth show accountXXXX should definitely work (and should not require any himalaya account configure).

@sevalsixarci
Copy link
Author

Oh wait I just noticed the redirect scheme with https, it may be the issue. The spawn server is really basic and may not support https in fact.

Yes. I've dug into this explanation by reading the code that manages the oauth flow.

In oauth/src/v2_0/authorization_code_grant.rs it's also actually hardcoded the http scheme.

@sevalsixarci
Copy link
Author

I also tried to bypass this by reusing the OAuth tokens from pizauth, with these config lines (but I think I may have misunderstood some of the config keys):

backend.auth.access-token.cmd = "pizauth show accountXXXX"
backend.auth.refresh-token.cmd = "pizauth refresh accountXXXX" 

Those are shell commands to retrieve the access token or the refresh token. So pizauth show accountXXXX should definitely work (and should not require any himalaya account configure).

Good to know.

I retryed this approach without success.

Trace of this try:


djus@satellite:~$ himalaya envelope list -a accountXXXXX --trace
2024-11-06T08:22:34.133346Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-11-06T08:22:34.134451Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-11-06T08:22:34.134521Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXXX-imap-oauth2-client-secret, and no target    
2024-11-06T08:22:34.134544Z DEBUG keyring: created entry SsCredential { attributes: {"username": "accountXXXXX-imap-oauth2-client-secret", "application": "rust-keyring", "target": "default", "service": "himalaya-cli"}, label: "accountXXXXX-imap-oauth2-client-secret@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-06T08:22:34.134561Z DEBUG keyring: creating entry with service himalaya-cli, user accountXXXXX-smtp-oauth2-client-secret, and no target    
2024-11-06T08:22:34.134571Z DEBUG keyring: created entry SsCredential { attributes: {"username": "accountXXXXX-smtp-oauth2-client-secret", "service": "himalaya-cli", "target": "default", "application": "rust-keyring"}, label: "accountXXXXX-smtp-oauth2-client-secret@himalaya-cli:default (keyring v3.6.1)", target: Some("default") }    
2024-11-06T08:22:34.134643Z DEBUG email::imap: building 1 IMAP clients
Error: 
   0: cannot build IMAP client
   1: cannot connect to IMAP server outlook.office365.com:993 using TCP
   2: cannot receive greeting from server
   3: Stream was closed

Location:
   /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pimalaya-tui-0.1.0/src/himalaya/backend.rs:712

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 3 frames hidden ⋮                               
   4: pimalaya_tui::himalaya::backend::BackendBuilder::build::{{closure}}::hdca2506f1d3d6d69
      at <unknown source file>:<unknown line>
   5: himalaya::email::envelope::command::list::ListEnvelopesCommand::execute::{{closure}}::hb863fa8425f1097d
      at <unknown source file>:<unknown line>
   6: himalaya::cli::HimalayaCommand::execute::{{closure}}::hbe84badc29564940
      at <unknown source file>:<unknown line>
   7: tokio::runtime::park::CachedParkThread::block_on::hd6b654ade072cf0d
      at <unknown source file>:<unknown line>
   8: tokio::runtime::context::runtime::enter_runtime::hde46c34d4f1a400f
      at <unknown source file>:<unknown line>
   9: tokio::runtime::runtime::Runtime::block_on::habc584feb0f754d1
      at <unknown source file>:<unknown line>
  10: himalaya::main::hb8f6325a19ce1083
      at <unknown source file>:<unknown line>
  11: std::sys::backtrace::__rust_begin_short_backtrace::hc3ace97b884b8485
      at <unknown source file>:<unknown line>
  12: std::rt::lang_start::{{closure}}::h778edc4c447b525b
      at <unknown source file>:<unknown line>
  13: std::rt::lang_start_internal::h5e7c81cecd7f0954
      at <unknown source file>:<unknown line>
  14: main<unknown>
      at <unknown source file>:<unknown line>
  15: __libc_start_call_main<unknown>
      at <unknown source file>:<unknown line>
  16: __libc_start_main@GLIBC_2.2.5<unknown>
      at <unknown source file>:<unknown line>
  17: _start<unknown>
      at <unknown source file>:<unknown line>

@soywod
Copy link
Member

soywod commented Nov 6, 2024

Great, we are going forward now.

I see in the logs that it uses keyring for the client secret. I don't know how you put your client secret with pizauth, but Himalaya needs to know how to retrieve it. The default behaviour uses keyring, you need a account configure to set it up. If your client secret is accessible from a command, you can use the same syntax as for access-token and refresh-token:

backend.auth.client-secret.cmd = "shell-command-that-retrieves-secret"

# or using raw, not safe and not recommended (yet useful for testing purpose)
backend.auth.client-secret.raw = "S€cяэT"

@sevalsixarci
Copy link
Author

sevalsixarci commented Nov 6, 2024

With pizauth I simply ignored the client secret. In the source code of Thunderbird (of which I am pretending to be) there are any of it and, in my setup, authenticating just works without (I am able to receive and send, via smtp).

About Himalaya: I first tried to configure the account without reusing pizauth; the interactive UI asked for client secret and I think that I just sent a blank line or I pasted a old client secret of Thunderbird (which was paired with a different client id).

--
Now after adding backend.auth.client-secret.raw = "" (blank) in my config I just tried theese attempts:

  1. reset account (I think it cleans keyring, right?) with himalaya account configure --reset accountXXXXX;
  2. renamed account (in config);
  3. tried with the last know client secret of Thunderbird.

Every attempts failed with the same trace of my last message.

Also I add that the connection attempt blocks for a minute (time says "real 1m9").

@soywod
Copy link
Member

soywod commented Nov 7, 2024

With pizauth I simply ignored the client secret. In the source code of Thunderbird (of which I am pretending to be) there are any of it and, in my setup, authenticating just works without (I am able to receive and send, via smtp).

Looks like I miss some piece of information there. I thought the client secret was mandatory. The code expects a client secret as well. So either Thunderbird has a special authority that I'm not aware of, or the flow should accept not to have a client secret.

I will investigate and let you.

@soywod
Copy link
Member

soywod commented Nov 7, 2024

Is your client id 9e5f94bc-e8a4-4e73-b8be-63364c29d753 by any chance? So far, it looks like Thunderbird has a special authorization: https://support.mozilla.org/fr/kb/microsoft-oauth-authentification-thunderbird

Can you tell me a bit more how you configured your Thunderbird and pizauth? Is it for a personal account (like personal outlook account) or for a company?

@sevalsixarci
Copy link
Author

Is your client id 9e5f94bc-e8a4-4e73-b8be-63364c29d753 by any chance? […]

Yes, that is the client id of Thunderbird (the desktop version) that I'm using.

So far, it looks like Thunderbird has a special authorization: https://support.mozilla.org/fr/kb/microsoft-oauth-authentification-thunderbird

Yes, with Outlook the client id must be "authorized" by the organization. My organization has authorized Thunderbird.

Can you tell me a bit more how you configured your Thunderbird and pizauth?

Thunderbird is not running on my machine.
I'm using their id with pizauth according to their (pizauth) documentation.
pizauth is a daemon utility which has the only task of fetch (and refresh as necessary) OAuth tokens.

Is it for a personal account (like personal outlook account) or for a company?

For work accounts (two, one is issued by my gov), so company.

@sevalsixarci
Copy link
Author

Looks like I miss some piece of information there. I thought the client secret was mandatory. The code expects a client secret as well. So either Thunderbird has a special authority that I'm not aware of, or the flow should accept not to have a client secret.

Apparently that is the case.

I quote from pizauth readme:

At a minimum you need to find out from your provider:

  • The authorisation URI.
  • The token URI.
  • Your "Client ID" (and in many cases also your "Client secret"), which identify your software.

Emphasis mine.

@soywod
Copy link
Member

soywod commented Nov 11, 2024

Interesting. I can try to make the client secret optional and see if it works for you. I let you know.

soywod added a commit that referenced this issue Nov 21, 2024
@soywod soywod added the question Further information is requested label Nov 21, 2024
@soywod
Copy link
Member

soywod commented Nov 21, 2024

I made the client secret optional, could you try again with a pre-built binary?

@soywod soywod moved this from In Progress to Todo in Pimalaya Nov 21, 2024
@sevalsixarci
Copy link
Author

Hi, the pre-built binary miss the oauth2 feature so I tried to rebuild from git locally (I'm running cargo 1.81.0).

With same account/config (I only changed the name) I think this fails nevertheless with some kind of bootloop (while trying to fetch inbox contents?), trace:

himalaya envelope list -a bug494 --trace
2024-11-21T17:10:58.988879Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-11-21T17:10:59.008649Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-11-21T17:10:59.008947Z DEBUG email::imap: building 1 IMAP clients
2024-11-21T17:10:59.146727Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.419843Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.420135Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.420276Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.420399Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422422Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422509Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422542Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422565Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422588Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422610Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422633Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422656Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422681Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422704Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422729Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422752Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422775Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
2024-11-21T17:12:05.422797Z DEBUG client::build: imap_next::stream: more input needed
2024-11-21T17:12:05.422836Z DEBUG client::build: imap_next::stream: read 0/1024 bytes
[trucated]

Screencast:
Screencast_20241121_181053.webm

By the way the process continued to run until this very moment (so for 7 minutes) before being apparently killed by my OS (I think, the terminal program simply disappeared).

P.S. I'm on a GNU/Linux system (openSUSE Tumbleweed).

@KoviRobi
Copy link
Contributor

KoviRobi commented Nov 21, 2024

I'm getting the same stuck issue with a public application too.

I've realised how to create public (no secret) applications:
https://learn.microsoft.com/en-us/entra/identity-platform/msal-client-applications

Basically when you create the authentication/platform configuration URL in the image below, if you choose (3) then you create a confidential application which needs a secret, and if you choose (4) then you don't need a secret

image

So this might make the issue more easy to reproduce?

And with this knowledge, I've managed to fetch my mail by converting my public app to a confidential app

@soywod
Copy link
Member

soywod commented Nov 22, 2024

The pre-built binary miss the oauth2 feature so I tried to rebuild from git locally (I'm running cargo 1.81.0).

Yes indeed, I forgot this detail.

With same account/config (I only changed the name) I think this fails nevertheless with some kind of bootloop (while trying to fetch inbox contents?), trace:

In fact it cannot connect to the TLS stream, we see on the screencast that it hangs for a while before looping. I justed push a fix that prevents the loop: you can pull and try again, you should get sth similar to #494 (comment):

cannot connect to IMAP server outlook.office365.com:993 using TCP

This error comes from the initial TCP connection to the server:

let tcp_stream = TcpStream::connect((host.as_str(), port))
    .await
    .map_err(ClientError::ConnectTcp)?;

Which means that it fails connecting to the host via the given port.

Do you confirm that a simple telnet outlook.office365.com 993 works?
Did you try with STARTTLS?

backend.port = 143
backend.encryption = "start-tls"

I cannot reproduce the issue with my personal Outlook account, both TLS and STARTTLS work. Not sure how to help you.

@soywod
Copy link
Member

soywod commented Nov 22, 2024

If it was an auth issue, we would get TLS logs, and the error would look like in #501.

@sevalsixarci
Copy link
Author

With same account/config (I only changed the name) I think this fails nevertheless with some kind of bootloop (while trying to fetch inbox contents?), trace:

In fact it cannot connect to the TLS stream, we see on the screencast that it hangs for a while before looping. I justed push a fix that prevents the loop: you can pull and try again, you should get sth similar to #494 (comment):

cannot connect to IMAP server outlook.office365.com:993 using TCP

I can confirm, now it outputs this:

$ ./target/debug/himalaya envelope list -a bug494 --trace
2024-11-23T18:36:21.310851Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-11-23T18:36:21.372656Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-11-23T18:36:21.373012Z DEBUG email::imap: building 1 IMAP clients
2024-11-23T18:36:21.495895Z TRACE client::build: imap_next::stream: more input needed
Error: 
   0: cannot build IMAP client
   1: cannot connect to IMAP server outlook.office365.com:993 using TCP
   2: cannot receive greeting from server
   3: Stream was closed

Location:
   /home/djus/.cargo/git/checkouts/tui-660ce218c336e983/f59e1ac/src/himalaya/backend.rs:712

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 5 frames hidden ⋮                               
   6: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual::h7a98fa1831eafa8a
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/result.rs:1959
   7: pimalaya_tui::himalaya::backend::BackendBuilder::build::{{closure}}::hd7f98a8724d39035
      at /home/djus/.cargo/git/checkouts/tui-660ce218c336e983/f59e1ac/src/himalaya/backend.rs:712
   8: himalaya::email::envelope::command::list::ListEnvelopesCommand::execute::{{closure}}::hfa766210dd32a3ab
      at /home/djus/dev/test/himalaya/src/email/envelope/command/list.rs:168
   9: himalaya::email::envelope::command::EnvelopeSubcommand::execute::{{closure}}::hdfff6aab4591f515
      at /home/djus/dev/test/himalaya/src/email/envelope/command/mod.rs:31
  10: himalaya::cli::HimalayaCommand::execute::{{closure}}::h472ca1fe2bc6ea7e
      at /home/djus/dev/test/himalaya/src/cli.rs:150
  11: himalaya::main::{{closure}}::hbc70bdc818226c72
      at /home/djus/dev/test/himalaya/src/main.rs:37
  12: <core::pin::Pin<P> as core::future::future::Future>::poll::h846c7d3354785c5e
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/future/future.rs:124
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h654d96a9f4af40b8
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/park.rs:281
  14: tokio::runtime::coop::with_budget::h6ffeb143091c8991
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/coop.rs:107
  15: tokio::runtime::coop::budget::hf4d41079a1850afe
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/coop.rs:73
  16: tokio::runtime::park::CachedParkThread::block_on::he2675d2a1e23224f
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/park.rs:281
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on::h96ed57ad7cd9cb89
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context/blocking.rs:66
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::hc082c65abc791a55
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/mod.rs:87
  19: tokio::runtime::context::runtime::enter_runtime::hbc87b08ad48d7c86
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context/runtime.rs:65
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h2fa583ad25779d46
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/mod.rs:86
  21: tokio::runtime::runtime::Runtime::block_on_inner::h88212693eeb71782
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/runtime.rs:370
  22: tokio::runtime::runtime::Runtime::block_on::hb8cb5c0bc3e9e187
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/runtime.rs:340
  23: himalaya::main::h8bfa708fd8548ebf
      at /home/djus/dev/test/himalaya/src/main.rs:46
  24: core::ops::function::FnOnce::call_once::h20edc6c027b3bbd6
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/ops/function.rs:250
                                ⋮ 16 frames hidden ⋮                

Do you confirm that a simple telnet outlook.office365.com 993 works?

Telnet seems to be able to connect:

Trying 52.98.237.146...
Connected to outlook.office365.com.
Escape character is '^]'.
Connection closed by foreign host.

Did you try with STARTTLS?

Just tried, it fails:

$ ./target/debug/himalaya envelope list -a bug494 --trace
2024-11-23T18:41:19.137627Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-11-23T18:41:19.147011Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-11-23T18:41:19.147351Z DEBUG email::imap: building 1 IMAP clients
2024-11-23T18:41:19.369318Z DEBUG client::build: start_tls::imap: read and discarded 160 bytes: "* OK The Microsoft Exchange IMAP4 service is ready. [TQBJADAAUAAyADkAMwBDAEEAMAAwADAANAAuAEkAVABBAFAAMgA5ADMALgBQAFIATwBEAC4ATwBVAFQATABPAE8ASwAuAEMATwBNAA==]\r\n"
2024-11-23T18:41:19.369652Z DEBUG client::build: start_tls::imap: wrote 13 bytes: "A1 STARTTLS\r\n"
2024-11-23T18:41:19.425892Z DEBUG client::build: start_tls::imap: read and discarded 34 bytes: "A1 OK Begin TLS negotiation now.\r\n"
2024-11-23T18:41:19.426330Z DEBUG client::build: rustls::client::hs: No cached session for DnsName("outlook.office365.com")    
2024-11-23T18:41:19.427196Z DEBUG client::build: rustls::client::hs: Not resuming any session    
2024-11-23T18:41:19.691568Z DEBUG client::build: rustls::webpki::anchors: add_parsable_certificates processed 148 valid and 0 invalid certs    
2024-11-23T18:41:19.691687Z DEBUG client::build: rustls_platform_verifier::verification::others: Loaded 148 CA certificates from the system    
2024-11-23T18:41:19.691884Z TRACE client::build: rustls::client::hs: Sending ClientHello Message {
    version: TLSv1_0,
    payload: Handshake {
        parsed: HandshakeMessagePayload {
            typ: ClientHello,
            payload: ClientHello(
                ClientHelloPayload {
                    client_version: TLSv1_2,
                    random: c85756d472d62d57a312da7d53a2210edd03420e4a6bb2ad6d9fa73bebfc522e,
                    session_id: d1c0dfa9a8377a54b67f58bd5d58773669a6d08e0ee54e05e9ea11f1df87e2ba,
                    cipher_suites: [
                        TLS13_AES_256_GCM_SHA384,
                        TLS13_AES_128_GCM_SHA256,
                        TLS13_CHACHA20_POLY1305_SHA256,
                        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
                        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
                        TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
                        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
                        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
                        TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
                        TLS_EMPTY_RENEGOTIATION_INFO_SCSV,
                    ],
                    compression_methods: [
                        Null,
                    ],
                    extensions: [
                        EcPointFormats(
                            [
                                Uncompressed,
                            ],
                        ),
                        PresharedKeyModes(
                            [
                                PSK_DHE_KE,
                            ],
                        ),
                        Protocols(
                            [
                                ProtocolName(
                                    696d6170,
                                ),
                            ],
                        ),
                        ServerName(
                            [
                                ServerName {
                                    typ: HostName,
                                    payload: HostName(
                                        DnsName(
                                            "outlook.office365.com",
                                        ),
                                    ),
                                },
                            ],
                        ),
                        ExtendedMasterSecretRequest,
                        CertificateStatusRequest(
                            Ocsp(
                                OcspCertificateStatusRequest {
                                    responder_ids: [],
                                    extensions: ,
                                },
                            ),
                        ),
                        KeyShare(
                            [
                                KeyShareEntry {
                                    group: X25519,
                                    payload: 00561a501abcdf0ea233cc11bb21cdd09ea6f20f0ba1212f8e5045badc8f7024,
                                },
                            ],
                        ),
                        NamedGroups(
                            [
                                X25519,
                                secp256r1,
                                secp384r1,
                            ],
                        ),
                        SignatureAlgorithms(
                            [
                                ECDSA_NISTP384_SHA384,
                                ECDSA_NISTP256_SHA256,
                                ED25519,
                                RSA_PSS_SHA512,
                                RSA_PSS_SHA384,
                                RSA_PSS_SHA256,
                                RSA_PKCS1_SHA512,
                                RSA_PKCS1_SHA384,
                                RSA_PKCS1_SHA256,
                            ],
                        ),
                        SupportedVersions(
                            [
                                TLSv1_3,
                                TLSv1_2,
                            ],
                        ),
                        SessionTicket(
                            Request,
                        ),
                    ],
                },
            ),
        },
        encoded: 010000fa0303c85756d472d62d57a312da7d53a2210edd03420e4a6bb2ad6d9fa73bebfc522e20d1c0dfa9a8377a54b67f58bd5d58773669a6d08e0ee54e05e9ea11f1df87e2ba0014130213011303c02cc02bcca9c030c02fcca800ff0100009d000b00020100002d0002010100100007000504696d61700000001a00180000156f75746c6f6f6b2e6f66666963653336352e636f6d00170000000500050100000000003300260024001d002000561a501abcdf0ea233cc11bb21cdd09ea6f20f0ba1212f8e5045badc8f7024000a00080006001d00170018000d00140012050304030807080608050804060105010401002b0005040304030300230000,
    },
}    
2024-11-23T18:41:19.769015Z TRACE client::build: rustls::client::hs: We got ServerHello ServerHelloPayload {
    extensions: [
        CertificateStatusAck,
        ExtendedMasterSecretAck,
        RenegotiationInfo(
            ,
        ),
    ],
    legacy_version: TLSv1_2,
    random: 674221cf7f88d351eace92784c58cddd696e13b741f48fd23c7c8e7bdfbc7fe6,
    session_id: 232c000067f39ed2692483d2e68d58552742dac1ac72fdf2b9badffa5461a1a2,
    cipher_suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
    compression_method: Null,
}    
2024-11-23T18:41:19.769207Z DEBUG client::build: rustls::client::hs: ALPN protocol is None    
2024-11-23T18:41:19.769280Z DEBUG client::build: rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384    
2024-11-23T18:41:19.769364Z DEBUG client::build: rustls::client::tls12::server_hello: Server may staple OCSP response    
2024-11-23T18:41:19.769623Z TRACE client::build: rustls::client::tls12: Server stapled OCSP response is [48, 130, 1, 211, 10, 1, 0, 160, 130, TRUNCATED_BECAUSE_IS_TOO-LONG]]    
2024-11-23T18:41:19.769818Z DEBUG client::build: rustls::client::tls12: ECDHE curve is EcParameters { curve_type: NamedCurve, named_group: secp384r1 }    
2024-11-23T18:41:19.769904Z TRACE client::build: rustls::client::tls12: Server cert is CertificateChain([CertificateDer(0x308208a53082 TRUNCATED_BECAUSE_IS_TOO-LONG]), CertificateDer(0x308204e63082 TRUNCATED_BECAUSE_IS_TOO-LONG])])    
2024-11-23T18:41:19.770465Z DEBUG client::build: rustls::client::tls12: Server DNS name is DnsName("outlook.office365.com")    
2024-11-23T18:41:19.771161Z TRACE client::build: rustls::webpki::server_verifier: Unvalidated OCSP response: [48, 130, 1, TRUNCATED_BECAUSE_IS_TOO-LONG]    
2024-11-23T18:41:19.841654Z TRACE client::build: imap_next::stream: more input needed
2024-11-23T18:42:25.880370Z TRACE client::build: imap_next::stream: read 32/1024 bytes
2024-11-23T18:42:25.880819Z DEBUG client::build: imap_client::tasks::resolver: received greeting: Greeting { kind: Bye, code: None, text: Text("Connection is closed. 13") }
2024-11-23T18:42:25.881222Z TRACE client::build: imap_next::stream: wrote 16/16 bytes
Error: 
   0: cannot build IMAP client
   1: cannot connect to IMAP server outlook.office365.com:143 using STARTTLS
   2: stream error
   3: peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/index.html#unexpected-eof

Location:
   /home/djus/.cargo/git/checkouts/tui-660ce218c336e983/f59e1ac/src/himalaya/backend.rs:712

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 5 frames hidden ⋮                               
   6: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual::h7a98fa1831eafa8a
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/result.rs:1959
   7: pimalaya_tui::himalaya::backend::BackendBuilder::build::{{closure}}::hd7f98a8724d39035
      at /home/djus/.cargo/git/checkouts/tui-660ce218c336e983/f59e1ac/src/himalaya/backend.rs:712
   8: himalaya::email::envelope::command::list::ListEnvelopesCommand::execute::{{closure}}::hfa766210dd32a3ab
      at /home/djus/dev/test/himalaya/src/email/envelope/command/list.rs:168
   9: himalaya::email::envelope::command::EnvelopeSubcommand::execute::{{closure}}::hdfff6aab4591f515
      at /home/djus/dev/test/himalaya/src/email/envelope/command/mod.rs:31
  10: himalaya::cli::HimalayaCommand::execute::{{closure}}::h472ca1fe2bc6ea7e
      at /home/djus/dev/test/himalaya/src/cli.rs:150
  11: himalaya::main::{{closure}}::hbc70bdc818226c72
      at /home/djus/dev/test/himalaya/src/main.rs:37
  12: <core::pin::Pin<P> as core::future::future::Future>::poll::h846c7d3354785c5e
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/future/future.rs:124
  13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h654d96a9f4af40b8
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/park.rs:281
  14: tokio::runtime::coop::with_budget::h6ffeb143091c8991
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/coop.rs:107
  15: tokio::runtime::coop::budget::hf4d41079a1850afe
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/coop.rs:73
  16: tokio::runtime::park::CachedParkThread::block_on::he2675d2a1e23224f
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/park.rs:281
  17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on::h96ed57ad7cd9cb89
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context/blocking.rs:66
  18: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::hc082c65abc791a55
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/mod.rs:87
  19: tokio::runtime::context::runtime::enter_runtime::hbc87b08ad48d7c86
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/context/runtime.rs:65
  20: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h2fa583ad25779d46
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/scheduler/multi_thread/mod.rs:86
  21: tokio::runtime::runtime::Runtime::block_on_inner::h88212693eeb71782
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/runtime.rs:370
  22: tokio::runtime::runtime::Runtime::block_on::hb8cb5c0bc3e9e187
      at /home/djus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/runtime.rs:340
  23: himalaya::main::h8bfa708fd8548ebf
      at /home/djus/dev/test/himalaya/src/main.rs:46
  24: core::ops::function::FnOnce::call_once::h20edc6c027b3bbd6
      at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/ops/function.rs:250
                                ⋮ 16 frames hidden ⋮     

I cannot reproduce the issue with my personal Outlook account, both TLS and STARTTLS work. Not sure how to help you.

Could there be some error in my configuration? I only configured himalaya by reading the example in the readme and the sample in the repo.

@soywod
Copy link
Member

soywod commented Nov 23, 2024

This looks way better, we see now TLS negociation. @duesee any idea why the server just sends BYE straight after TLS negociation?

@duesee
Copy link

duesee commented Nov 24, 2024

Two observations: First, after STARTTLS, the server doesn't send a second Greeting. So, the BYE is not a Greeting but just an unsolicited response. Second, the response comes a minute after TLS negotiation. So, I think this is a timeout because we expect a greeting even though we should go ahead and send some commands.

But I'm not sure about our STARTTLS handling state right now. Did you already change it as discussed to use the streamlined approach w/o parsing? A caveat I missed when we talked last time was that after STARTTLS the connection must not expect a second greeting. This is easy to solve with a Client::without_greeting() constructor, though. But I don't know currently what is the state of our code base.

See duesee/imap-next#294.

@soywod
Copy link
Member

soywod commented Nov 24, 2024

But I don't know currently what is the state of our code base.

The Himalaya suite actually uses this version of imap-client, so I get real-world feedbacks about our current changes.

after STARTTLS, the server doesn't send a second

Looks like something is wrong is the STARTTLS algorithm, let me check back.

@soywod
Copy link
Member

soywod commented Nov 29, 2024

@sevalsixarci I was able to reproduce and to identify what is missing (see duesee/imap-next#296). Now I have another error, somehow it cannot LOGIN: authentication failed mechanism=Plain err=ResolveTask(UnexpectedNoResponse(StatusBody { kind: No, code: None, text: Text("AUTHENTICATE failed.") })). It looks like Outlook does not allow to use his own password for authentication, I will investigate.

Once the PR is merged, I will close this issue in favor of #501 (duplicate).

@soywod
Copy link
Member

soywod commented Nov 29, 2024

I just pushed a fix meanwhile, could you try again?

@sevalsixarci
Copy link
Author

sevalsixarci commented Nov 29, 2024

Hi, now himalaya works. I see the content of my mailbox.
So, I think, we can say that the issue is resolved.

Do you need traces or other test?

@soywod
Copy link
Member

soywod commented Nov 29, 2024

No it's fine, in fact the issue comes from Microsoft. It looks like it's not possible anymore to login to personnal account with plain auth (OAuth is now required). I also tried with OAuth and it works as expected. Thank you for your patience 🙏

@soywod soywod closed this as completed Nov 29, 2024
@github-project-automation github-project-automation bot moved this from Pending to Done in Pimalaya Nov 29, 2024
@soywod soywod removed the question Further information is requested label Nov 29, 2024
@sevalsixarci
Copy link
Author

I thank you as well for your patience, clarity and kindness. (:

@duesee
Copy link

duesee commented Nov 29, 2024

To recap quickly: It was OAuth from the beginning and we found the STARTTLS bug accidentally, right?

Then I would recommend to you @sevalsixarci to use "implicit TLS" (imaps on port 993) again to get a more secure and faster connection :-)

@soywod
Copy link
Member

soywod commented Nov 29, 2024

It was OAuth from the begining but the TCP connection on port 993 was kind of timing out? Then STARTTLS helped to go forward till the bug we discovered. BTW, @sevalsixarci could you try again TLS on port 993?

@sevalsixarci
Copy link
Author

sevalsixarci commented Nov 29, 2024

BTW, @sevalsixarci could you try again TLS on port 993?

Umh… it fails:

$ himalaya envelope list -a bug494 --trace
2024-11-29T19:04:28.336087Z DEBUG keyring::service: define global service name name="himalaya-cli"
2024-11-29T19:04:28.337398Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-11-29T19:04:28.337515Z DEBUG email::imap: building 1 IMAP clients
2024-11-29T19:04:28.427865Z TRACE client::build: imap_next::stream: more input needed
Error: 
   0: cannot build IMAP client
   1: cannot connect to IMAP server outlook.office365.com:993 using TCP
   2: cannot receive greeting from server
   3: Stream was closed

Location:
   /home/djus/.cargo/git/checkouts/tui-660ce218c336e983/7acc0db/src/himalaya/backend.rs:712

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 3 frames hidden ⋮                               
   4: pimalaya_tui::himalaya::backend::BackendBuilder::build::{{closure}}::hccce85c257203a15
      at <unknown source file>:<unknown line>
   5: himalaya::email::envelope::command::list::ListEnvelopesCommand::execute::{{closure}}::hca669e03cd1e4dd2
      at <unknown source file>:<unknown line>
   6: himalaya::cli::HimalayaCommand::execute::{{closure}}::h272ea11f9d20bc35
      at <unknown source file>:<unknown line>
   7: tokio::runtime::park::CachedParkThread::block_on::h4b52899c96a15352
      at <unknown source file>:<unknown line>
   8: tokio::runtime::context::runtime::enter_runtime::h4d1bfba26c1d7a6b
      at <unknown source file>:<unknown line>
   9: tokio::runtime::runtime::Runtime::block_on::ha202648d4195ed97
      at <unknown source file>:<unknown line>
  10: himalaya::main::hd2768b1d807eac29
      at <unknown source file>:<unknown line>
  11: std::sys::backtrace::__rust_begin_short_backtrace::h3dbfb340f1481efc
      at <unknown source file>:<unknown line>
  12: std::rt::lang_start::{{closure}}::h1596f64f78c9b506
      at <unknown source file>:<unknown line>
  13: std::rt::lang_start_internal::h4d90db0530245041
      at <unknown source file>:<unknown line>
  14: main<unknown>
      at <unknown source file>:<unknown line>
  15: __libc_start_call_main<unknown>
      at <unknown source file>:<unknown line>
  16: __libc_start_main@GLIBC_2.2.5<unknown>
      at <unknown source file>:<unknown line>
  17: _start<unknown>
      at <unknown source file>:<unknown line>

So STARTTLS on port 143 works but the preferred conf with port 993 don't.

@soywod
Copy link
Member

soywod commented Nov 29, 2024

Really strange. I cannot reproduce the error, my personal account works both using TLS 993 and STARTTLS 143. Your logs really show that it does not even reach the server, it fails before even TLS negociation. Must be a Microsoft mis-configuration issue from your side? At least the STARTTLS variant works for you. What a pain…

@sevalsixarci
Copy link
Author

Must be a Microsoft mis-configuration issue from your side?

Dunno.

This is my config:

[accounts.bug494]
email = "MAIL_REDACTED"
backend.type = "imap"
backend.auth.type = "oauth2"
backend.auth.method = "xoauth2"
backend.host = "outlook.office365.com"
backend.port = 993
#backend.port = 143
#backend.encryption = "start-tls"
backend.login = "MAIL_REDACTED"
backend.auth.client-id = "9e5f94bc-e8a4-4e73-b8be-63364c29d753"
backend.auth.auth-url = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize"
backend.auth.token-url = "https://login.microsoftonline.com/common/oauth2/v2.0/token"
backend.auth.access-token.cmd = "pizauth show bug494"
backend.auth.refresh-token.cmd = "pizauth show bug494"
backend.auth.pkce = true
backend.auth.scope = "https://outlook.office.com/IMAP.AccessAsUser.All"

@sevalsixarci
Copy link
Author

sevalsixarci commented Nov 29, 2024

I mean it can. My account is not personal (I deprecate M$ things, from UX and politics POV), is a work account. Maybe the management has enabled some subtle different behavior bout TLS for the sake of "security"; I mean they refuse to let me send mail outside the Outlook UI because they are not able to read how to authorize Thunderbird (so I read only and don't answer, I refuse) .

Any way, rant aside (I apologize), it seems that TLS negotiation still fails, but --trace don't seems so useful in this case, how can I help you to get more information?
I am able to read Rust code and I am willing to learn how to use a debugger.

--
EDIT: I think that I am out of the topic of this issue, do I have to open another thread/issue?

@duesee
Copy link

duesee commented Nov 30, 2024

Does a simple ....

openssl s_client -crlf -connect outlook.office365.com:993

... show you the IMAP greeting?

@sevalsixarci
Copy link
Author

Does a simple ....

openssl s_client -crlf -connect outlook.office365.com:993

... show you the IMAP greeting?

Yes, after the handshake:

OMISSIS...
* OK The Microsoft Exchange IMAP4 service is ready. [TQBJADIAUAAyADkAMwBDAEEAMAAwADAANQAuAEkAVABBAFAAMgA5ADMALgBQAFIATwBEAC4ATwBVAFQATABPAE8ASwAuAEMATwBNAA==]

But the account that with himalaya wont connect using TLS 993 does connect with different stack (my current setup rests on mbsync).
I would say that the problem is not the account per se.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

4 participants