Sentinel cannot connect to master with TLS enabled

[Leo791]

Describe your question

I need Sentinel to be able to connect to the master while having TLS enabled. Which is currently not happening.

I can connect to the sentinel using the following command: redis-cli -p ${SENTINEL_PORT} --tls --cert ${REDIS_TLS_CERT} --key ${REDIS_TLS_CERT_KEY} --cacert ${REDIS_TLS_CA_KEY} -a ${REDIS_PASSWORD} ping

The master successfully connects to its replica but then as soon as the sentinel tries connecting to an error loop occurs:

ACL_MODE is not true, skipping ACL file modification
Starting redis service in standalone mode.....
1:C 29 Aug 2024 18:18:46.892 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 29 Aug 2024 18:18:46.892 # Redis version=6.2.14, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 29 Aug 2024 18:18:46.892 # Configuration loaded
1:M 29 Aug 2024 18:18:46.902 * monotonic clock: POSIX clock_gettime
1:M 29 Aug 2024 18:18:46.909 * Running mode=standalone, port=6379.
1:M 29 Aug 2024 18:18:46.909 # Server initialized
1:M 29 Aug 2024 18:18:46.913 * Ready to accept connections
1:M 29 Aug 2024 18:18:55.743 * Replica 10.244.2.73:6379 asks for synchronization
1:M 29 Aug 2024 18:18:55.743 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '83eb294ea9016ddda55a92c1cc0d83a5a8fe43d9', my replication IDs are 'eae74c4d984e3deb8b374d4306435edb781d8e4f' and '0000000000000000000000000000000000000000')
1:M 29 Aug 2024 18:18:55.743 * Replication backlog created, my new replication IDs are '7c21d9ad3096d7bceda3e0b32fcad5844ee2b270' and '0000000000000000000000000000000000000000'
1:M 29 Aug 2024 18:18:55.744 * Starting BGSAVE for SYNC with target: disk
1:M 29 Aug 2024 18:18:55.746 * Background saving started by pid 23
23:C 29 Aug 2024 18:18:55.749 * DB saved on disk
23:C 29 Aug 2024 18:18:55.750 * RDB: 1 MB of memory used by copy-on-write
1:M 29 Aug 2024 18:18:55.830 * Background saving terminated with success
1:M 29 Aug 2024 18:18:55.832 * Synchronization with replica 10.244.2.73:6379 succeeded
1:M 29 Aug 2024 18:19:55.152 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
1:M 29 Aug 2024 18:19:55.152 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

What version of redis-operator are you using?

redis-operator version: v0.18.0

Additional context

I'll provide my redis-sentinel definition:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sentinel-config
  namespace: redis-test2
data:
  redis-sentinel-additional.conf: |
    daemonize no
    pidfile "/var/run/redis-sentinel.pid"
    dir "/tmp"
    port 0
    tls-port 26379
    sentinel resolve-hostnames no
    sentinel announce-hostnames no
    sentinel down-after-milliseconds mymaster 30000
    sentinel failover-timeout mymaster 180000
    sentinel parallel-syncs mymaster 1
---
apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisSentinel
metadata:
  name: redis-test2
  namespace: redis-test2
spec:
  clusterSize: 3
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 5
    periodSeconds: 15
    successThreshold: 1
    failureThreshold: 5
  readinessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 5
    periodSeconds: 15
    successThreshold: 1
    failureThreshold: 5
  redisSentinelConfig:
    redisReplicationName: redis-test2
    masterGroupName: 'mymaster'
    redisPort: '6379'
    quorum: '2'
    parallelSyncs: '1'
    failoverTimeout: '180000'
    downAfterMilliseconds: '30000'
    additionalSentinelConfig: sentinel-config
  kubernetesConfig:
    image: quay.io/opstree/redis-sentinel:v6.2.14
    imagePullPolicy: IfNotPresent
    redisSecret:
      name: redis-test2-secret
      key: password
    resources:
      requests:
        cpu: 101m
        memory: 128Mi
      limits:
        cpu: 101m
        memory: 128Mi
  TLS:
    secret:
      secretName: redis-test2-custom-tls
      optional: false

My redis-replication definition:

apiVersion: redis.redis.opstreelabs.in/v1beta2
kind: RedisReplication
metadata:
  name: redis-test2
spec:
  TLS:
    secret:
      optional: false
      secretName: redis-test2-custom-tls
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - locala
            - localb
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - redis-replication
          topologyKey: topology.kubernetes.io/zone
        weight: 90
  clusterSize: 2
  kubernetesConfig:
    image: quay.io/opstree/redis:v6.2.14
    imagePullPolicy: IfNotPresent
    redisSecret:
      key: password
      name: redis-test2-secret
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
    updateStrategy: {}
  podSecurityContext:
    fsGroup: 1000
    runAsUser: 1000
  storage:
    volumeClaimTemplate:
      metadata: {}
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
      status: {}
    volumeMount:
      mountPath:
      - mountPath: /etc/redis/redis.conf
        name: custom-config
        subPath: redis.conf
      volume:
      - name: custom-config
        secret:
          secretName: redis-test2-conf-secret

And my custom redis conf:

kubectl exec -it pod/redis-test-0 -n redis-test -- sh
/data $ cat /etc/redis/redis.conf
bind 0.0.0.0 ::
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize no
supervised no
pidfile /var/run/redis.pid
masterauth password
requirepass password
protected-mode yes
port 0
tls-port 6379
tls-cert-file /tls/tls.crt
tls-key-file /tls/tls.key
tls-ca-cert-file /tls/ca.crt
tls-auth-clients optional
tls-replication yes
maxmemory-policy noeviction

[woodliu]

@Leo791 I think the problem is Sentinel should set parameter tls-replication too.
https://redis.io/docs/latest/operate/oss_and_stack/management/security/encryption/#sentinel
Please try it.