[Bug]: Curl with WolfSSL fails when connecting to open.spotify.com - CA signer not available for verification

[ading2210]

Contact Details

allen@ading.dev

Version

Latest from git, commit 429e7c7

Description

Curl with WolfSSL is failing when trying to connect to open.spotify.com with the following error:

$ src/curl -4 https://open.spotify.com -v
* Host open.spotify.com:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.131.42, 151.101.195.42, 151.101.67.42, 151.101.3.42
*   Trying 151.101.131.42:443...
* ALPN: curl offers http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
*  CA signer not available for verification
* closing connection #0
curl: (77)  CA signer not available for verification

I'm not sure if there are other websites which cause the same error. open.spotify.com is the only one I can reproduce the issue with.

Curl version info:

$ src/curl -V
curl 8.11.0-DEV (aarch64-unknown-linux-gnu) libcurl/8.11.0-DEV wolfSSL/5.7.4 zlib/1.3 libpsl/0.21.2
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM PSL SSL threadsafe UnixSockets

I am fairly sure the underlying issue is with WolfSSL since the request works fine in curl with OpenSSL.

When using OpenSSL:

$ curl -4 https://open.spotify.com -v
* Host open.spotify.com:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.3.42, 151.101.131.42, 151.101.195.42, 151.101.67.42
*   Trying 151.101.3.42:443...
* Connected to open.spotify.com (151.101.3.42) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=open.spotify.com
*  start date: Oct 29 02:06:43 2024 GMT
*  expire date: Nov 28 02:06:42 2024 GMT
*  subjectAltName: host "open.spotify.com" matched cert's "open.spotify.com"
*  issuer: C=US; O=Certainly; CN=Certainly Intermediate R1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://open.spotify.com/
...

My configure command:

./configure --enable-curl --enable-distro

Target environment:

$ uname -a
Linux LAPTOP-SL2KEVM7 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:16:34 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Reproduction steps

Compile WolfSSL from the latest source:

git clone https://github.com/wolfSSL/wolfssl -b master --depth=1
cd wolfssl
./autogen.sh
./configure --enable-curl --enable-distro
make
sudo make install

Compile Curl from the latest source using WolfSSL:

git clone https://github.com/curl/curl/ -b master --depth=1
cd curl
autoreconf -fi
./configure --with-wolfssl
make

Try to download https://open.spotify.com and observe the error:

src/curl -4 https://open.spotify.com -v

Relevant log output

Build log: https://pastebin.com/b0SYabhd

I couldn't include it in the body of this report since it was too long for Github.

[anhu]

Hi @ading2210 ,

My name is Anthony and I am a member of the wolfSSL team. I have reproduced what you are seeing but I did not add --enable-distro to the wolfssl configure command and I needed to add --without-libpsl to the curl configure command. Either I or another engineer will look into this.

In the meantime, can you let us know a bit about yourself and your project. Here at wolfSSL we love to know how people are using our code. Can I ask where you are located and what your goals are?

Warm regards, Anthony

[anhu]

I also had to add --enable-scp to the wolfssl configuration but even then I was still getting the same error. Still investigating.

[gasbytes]

Hey @ading2210

I just took over this ticket, since Anthony is traveling this week, so I'm going through your logs (by the way, thanks for providing all the info to reproduce it right away).
While I investigate the issue, do you mind sharing a bit more about the project that you are working on and what your goals are? Just to get more context.

Thanks and warm regards,
Reda Chouk

[ading2210]

Hi @gasbytes

The project that I am using WolfSSL for is a port of the libcurl C library to WebAssembly. WolfSSL is used here to provide TLS support while having the smallest file size, and even when it's compiled to WASM (something that I don't think was ever officially supported), it works really well for this purpose. Most people using my project are using it to proxy HTTPS requests in the browser (as sort of an encrypted CORS proxy), and thus I need the compatibility to be as good as possible. I actually found out about this bug because of a report from a user of my library, and I eventually traced the issue to WolfSSL.

Thanks,
Allen

[gasbytes]

Hey @ading2210

Thanks for the info, seems like a cool project.
I'm still investigating, but I might have found the problem and I'm currently working on a fix.

Thanks,
Reda Chouk

[gasbytes]

Hey @ading2210,

Seems like in the file /etc/ssl/cert.pem it was missing Certainly as trusted CA, that explain why wolfssl returns to curl ASN_NO_SIGNER_E, which means that it couldn't verify Certainly as a trusted authority, hence the error.

We don't manage CA's, that's usually up to the customer/user and their system-wide certificate store. The usual solution is to provide the certificate in question at run time.

Curl has a really nice flag that lets you provide the certificates directly from command line: --cacert.

To fix the issue, first you need to get the certificate, you can easily extract it and save it locally using openssl like so:
$ openssl s_client -showcerts -connect open.spotify.com:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > certainly_cert.pem

after this, you can provide the certificate to curl like so, and you will receive the payload correctly:

$ src/curl -4 "https://open.spotify.com/" -v --cacert certainly_cert.pem
* Host open.spotify.com:443 was resolved.
* IPv6: (none)
* IPv4: 151.101.87.42
*   Trying 151.101.87.42:443...
* ALPN: curl offers http/1.1
* successfully set certificate verify locations:
*  CAfile: certainly_cert.pem
*  CApath: none
* ALPN: server accepted http/1.1
* SSL connection using TLSv1.3 / TLS13-AES128-GCM-SHA256
* Connected to open.spotify.com (151.101.87.42) port 443
* using HTTP/1.x
> GET / HTTP/1.1
> Host: open.spotify.com
> User-Agent: curl/8.11.0-DEV
> Accept: */*
...

Let me know if this solution works for you! If you run into any further issues or need more help with curl or certificates, feel free to reach out.

Warm regards,
Reda Chouk

[ading2210]

Thanks, this workaround seems to work for me. Though I'm not sure as to why curl with OpenSSL works fine here without any workaround, because in both cases the system certificate store was used but curl with OpenSSL was able to connect just fine. The same is the case if I use the CA cert bundle provided by curl themselves (which was extracted from Firefox's source code). I'm not sure if I'm misunderstanding something here, but it seems to me that needing to add another CA cert for this to work is unexpected behavior.

[gasbytes]

Hello @ading2210

Sorry for the late answer.
I wanted to ask you if adding, given your current available sys certificates, to the wolfssl config this flag:

--enable-altcertchains

From my understanding OpenSSL takes a relaxed approach when validating the peer certificates, using --enable-altcertchains it should match this behaviour.

Another possibility is that openssl may be loading the default system CA certs, and to match that behaviour it would require --enable-sys-ca-certs and also calling wolfSSL_CTX_load_system_CA_certs.

Let me know if this fixes the issue without needing to add the missing intermediate certificate R1 to the store.

Warm regards,
Reda Chouk

[ading2210]

--enable-altcertchains doesn't seem to help: https://pastebin.com/758wzWLc

--enable-sys-ca-certs doesn't really change anything either: https://pastebin.com/Y8zRTumj

Using both flags together has the same result.

[gasbytes]

Hi @ading2210,

Based on the CURL documentation, using --ca-native should enable the operating system's native CA store for certificate verification (https://curl.se/docs/sslcerts.html), but the "unrecognized option" error I saw from my logs on Linux suggests it might not be supported in your build or version. If that’s the case, providing the --cacert option directly, as I suggested, seems like the most reliable workaround.

The inconsistencies between OpenSSL and wolfSSL you mentioned are also discussed here: curl/curl#11883. These differences in how certificates are handled on Linux might explain the behavior you’re seeing. I’d recommend testing with the native CA store option if your environment supports it, but falling back to explicitly loading certificates if not.

Let us know if this resolves the issue or if further clarification is needed.

Thanks,
Reda Chouk