Elasticsearch CA certificates are rejected by Python 3.13

[pquentin]

Elasticsearch Version

8.16.1

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin arm64

Problem Description

Python 3.13 (released in October 2024) enabled by default the VERIFY_X509_STRICT flag for improved RFC 5280 compliance. This setting maps to the X509_V_FLAG_X509_STRICT OpenSSL flag documented as:

The X509_V_FLAG_X509_STRICT flag disables workarounds for some broken certificates and makes the verification strictly apply X509 rules.

The CA certificates generated by Elasticsearch (either by default on startup) or by elasticsearch-certutil are not compliant, at least because they're missing the key usage extension.

Steps to Reproduce

Run Elasticsearch:

$ docker run --name es01 -p 9200:9200 -it -m 1GB docker.elastic.co/elasticsearch/elasticsearch:8.16.1
... wait for startup
$ docker container cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .

Try connecting to it using the Elasticsearch Python client:

from elasticsearch import Elasticsearch

client = Elasticsearch(
    "https://localhost:9200",
    ca_certs="http_ca.crt",
    basic_auth=("elastic", "...")
)
print(client.info())

This fails with:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA cert does not include key usage extension (_ssl.c:1020)

Alternatively, you can inspect the http_ca.crt file:

$ openssl x509 -in http_ca.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            a4:50:12:ea:89:c9:78:fe:9e:9a:4b:7c:64:18:e0:13:04:d6:fb:58
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Elasticsearch security auto-configuration HTTP CA
        Validity
            Not Before: Nov 29 14:07:47 2024 GMT
            Not After : Nov 29 14:07:47 2027 GMT
        Subject: CN=Elasticsearch security auto-configuration HTTP CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    [...]]
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                F4:BC:A2:F5:ED:8B:FD:93:F2:AE:76:82:A2:58:9E:EE:58:82:B9:BC
            X509v3 Authority Key Identifier: 
                F4:BC:A2:F5:ED:8B:FD:93:F2:AE:76:82:A2:58:9E:EE:58:82:B9:BC
            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        [...]

In X509v3 extensions, you can see that the Key Usage extension is missing. If by contrast, I'm looking at the test CA certificate generated by trustme and used in the Python client:

$ openssl x509 -in .buildkite/certs/ca.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            21:80:a5:61:65:e2:4e:c0:7c:68:ca:c4:10:ca:f3:76:b9:39:ac:eb
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: O=trustme v1.2.0, OU=Testing CA #biw1Wc10lpqCFQL5
        Validity
            Not Before: Jan  1 00:00:00 2000 GMT
            Not After : Jan  1 00:00:00 3000 GMT
        Subject: O=trustme v1.2.0, OU=Testing CA #biw1Wc10lpqCFQL5
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    [...]
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                2A:C6:19:C3:BD:BF:45:00:59:2B:03:F7:73:FF:C7:63:13:36:22:5B
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:9
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        [...]

You can see that Key Usage is included, and indeed connections to Python 3.13 work.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)