Failed to backup in Red Hat OCP cluster #914

karthik-murugan-hpe · 2023-12-07T07:25:31Z

karthik-murugan-hpe
Dec 7, 2023

I'm trying to test backup of Postgres DB by deploying k8up in a OCP cluster.

Pre-Backup pod is defined as,

apiVersion: k8up.io/v1
kind: PreBackupPod
metadata:
  name: backup-cp
spec:
  backupCommand: sh -c 'PGDATABASE="$POSTGRES_DB" PGUSER="$POSTGRES_USER" PGPASSWORD="$POSTGRES_PASSWORD" pg_dump -h "$DB_HOST" --clean'
  fileExtension: .sql
  pod:
    spec:
      securityContext:
        runAsUser: 1000780000
        runAsGroup: 1000780000
        fsGroup: 1000780000
        #runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - env:
            - name: POSTGRES_USER
              value: cp
            - name: POSTGRES_PASSWORD
              value: mypassword
            - name: DB_HOST
              value: postgres-pgpool
            - name: POSTGRES_DB
              value: cp
          name: postgres
          image: docker.io/bitnami/postgresql-repmgr:14.5.0-debian-11-r31
          command:
            - 'sleep'
            - 'infinity'
          imagePullPolicy: Always
          securityContext:
            #privileged: false
            #allowPrivilegeEscalation: false

Backup is defined as,

apiVersion: k8up.io/v1
kind: Backup
metadata:
  name: backup-test
spec:
  failedJobsHistoryLimit: 2
  successfulJobsHistoryLimit: 2
  podSecurityContext:
    runAsUser: 1000780000
    runAsGroup: 1000780000
    fsGroup: 1000780000
    #runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  backend:
    repoPasswordSecretRef:
      name: enc-secret
      key: KEY
    s3:
      endpoint: https://s3.us-east-2.amazonaws.com
      bucket: k8up-backup-test-2
      accessKeyIDSecretRef:
        name: s3-secret
        key: ACCESSKEY
      secretAccessKeySecretRef:
        name: s3-secret
        key: SECRETKEY
    envFrom:
      - configMapRef:
          name: backup-env

On trigerring the backup, the pods fail with this error. Please see the error in last two lines. Is the permission error causing the failure to backup or the error in last line?

root@JUG4IP2EVM:~/sandbox/k8up/postgres# oc logs backup-backup-test-1-h9pqx -f
2023-12-07T06:45:29Z    INFO    k8up    Starting k8up…  {"version": "2.7.2", "date": "2023-10-09T10:13:29Z", "commit": "45d99dd90dbb2a080e6832c34e96b371216a3e0b", "go_os": "linux", "go_arch": "amd64", "go_version": "go1.19.13", "uid": 1000780000, "gid": 1000780000}
2023-12-07T06:45:29Z    INFO    k8up.restic     initializing
2023-12-07T06:45:29Z    INFO    k8up.restic     setting up a signal handler
2023-12-07T06:45:29Z    INFO    k8up.restic.restic      using the following restic options      {"options": [""]}
2023-12-07T06:45:29Z    INFO    k8up.restic     sleeping until init     {"duration": "5s"}
2023-12-07T06:45:34Z    INFO    k8up.restic.restic.RepoInit.command     restic command  {"path": "/usr/local/bin/restic", "args": ["init", "--option", ""]}
2023-12-07T06:45:34Z    INFO    k8up.restic.restic.RepoInit.command     Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2023-12-07T06:45:36Z    INFO    k8up.restic.restic.unlock       unlocking repository    {"all": false}
2023-12-07T06:45:36Z    INFO    k8up.restic.restic.unlock.command       restic command  {"path": "/usr/local/bin/restic", "args": ["unlock", "--option", ""]}
2023-12-07T06:45:36Z    INFO    k8up.restic.restic.unlock.command       Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2023-12-07T06:45:39Z    INFO    k8up.restic.restic.unlock.restic.stderr unable to open cache: mkdir /.cache/restic: permission denied
2023-12-07T06:45:40Z    INFO    k8up.restic.restic.snapshots    getting list of snapshots
2023-12-07T06:45:40Z    INFO    k8up.restic.restic.snapshots.command    restic command  {"path": "/usr/local/bin/restic", "args": ["snapshots", "--option", "", "--json"]}
2023-12-07T06:45:40Z    INFO    k8up.restic.restic.snapshots.command    Defining RESTIC_PROGRESS_FPS    {"frequency": 0.016666666666666666}
2023-12-07T06:45:43Z    INFO    k8up.restic.restic.snapshots.restic.stderr      unable to open cache: mkdir /.cache/restic: permission denied
2023/12/07 06:45:47 unable to start k8up: backup of annotated pods failed: Could not create kubernetes client: Get "https://172.30.0.1:443/api?timeout=32s": Service Unavailable

Kidswiss · 2023-12-07T08:09:01Z

Kidswiss
Dec 7, 2023
Maintainer

Hi

If you're using the prebackup command, then you don't have to specify the podSecurityContext on the backup. It will read everything from the stdin. I'm pretty sure that's the cause of the permission denied error.

We're doing redis backups on ocp like this:

redis pod:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8up.io/backupcommand: /scripts/backup.sh
    k8up.io/file-extension: .tar
  name: redis-master-0
  namespace: myredis
spec:
  containers:
...

The backup:

apiVersion: k8up.io/v1
kind: Backup
metadata:
  creationTimestamp: "2023-12-06T23:14:00Z"
spec:
  backend:
    repoPasswordSecretRef:
      key: password
      name: k8up-repository-password
    s3:
      accessKeyIDSecretRef:
        key: AWS_ACCESS_KEY_ID
        name: backup-bucket-credentials
      bucket: redis-app1-prod-wv8ss-backup
      endpoint: https://myobjectstorage
      secretAccessKeySecretRef:
        key: AWS_SECRET_ACCESS_KEY
        name: backup-bucket-credentials
  keepJobs: 0

We've also marked all the PVCs in the namespace so they don't get backed up, otherwise we'd end up with backing up the data twice.

Also, is the postgresql you're trying to backup running as a pod as well? Because then you can actually skip the pre-backup pod and set the backup annotations directly on there. See my example above. Here are the docs: https://k8up.io/k8up/2.7/how-tos/application-aware-backups.html

The main use-case for pre-backup pods is to backup stuff that doesn't have a pod running on the cluster. For example if you want to backup a database that is not running on your cluster.

As for unable to start k8up: backup of annotated pods failed: Could not create kubernetes client: Get "https://172.30.0.1:443/api?timeout=32s": Service Unavailable. This looks like the k8s client of k8up is not able to connect to the k8s API endpoint. Please make sure that the API Server is running correctly. I'm not sure if the permission errors have an impact on this though.

0 replies

karthik-murugan-hpe · 2023-12-07T08:27:10Z

karthik-murugan-hpe
Dec 7, 2023
Author

Hi,

Thanks for that. After removing the podSecurityContext, the permission error is gone. However I still see the error in last line.

I need PreBackup pod because I have to run pg_dump against 3 databases in PostgresSQL. If I use annotation, it backs up to a single snapshot in S3 and restoring them will be even trickier.

K8up seems to be picking up the Kubernetes API server details from the mounted service files. Not clear how it could go wrong.

5 replies

Kidswiss Dec 7, 2023
Maintainer

Is there maybe a bogus kubernetes service in that namespace that points somewhere else?

kubectl get service | grep 172.30.0.1 could help you identify that.

karthik-murugan-hpe Dec 7, 2023
Author

Apparently the IP address is referred from a environment variable passed by K8s service.

KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT=tcp://172.30.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=172.30.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=172.30.0.1
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://172.30.0.1:443

I tried to hit the endpoint by curl and getting same error code of 503.

backup-backup-test-0-hvjf9:/$ K8S=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT
backup-backup-test-0-hvjf9:/$ TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
backup-backup-test-0-hvjf9:/$ CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
backup-backup-test-0-hvjf9:/$ NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
backup-backup-test-0-hvjf9:/$ curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/healthz
curl: (56) CONNECT tunnel failed, response 503

karthik-murugan-hpe Dec 7, 2023
Author

I tried these commands from another namespace in same cluster and the curl command yields a "200 OK" response. I suspect, the issue is beyond k8up

Kidswiss Dec 7, 2023
Maintainer

I agree, this looks like there's something else a bit awry here.

karthik-murugan-hpe Dec 7, 2023
Author

Thanks @Kidswiss for your support!

karthik-murugan-hpe · 2023-12-07T12:59:01Z

karthik-murugan-hpe
Dec 7, 2023
Author

Finally, I figured it was due to proxy settings added in Backup resource:

    envFrom:
      - configMapRef:
          name: backup-env

I added a no_proxy to skip internal address of K8s. I needed the proxy only for communicating with AWS S3

apiVersion: v1
data:
  HTTPS_PROXY: web-proxy.acme.net:8080
  NO_PROXY: .svc,.default,.cluster.local,172.30.0.1
kind: ConfigMap
metadata:
  creationTimestamp: "2023-12-06T17:22:17Z"
  name: backup-env
  namespace: test-backup
  resourceVersion: "29156058"
  uid: ff81605c-6c3c-4d68-b9a5-662039fb6891

1 reply

Kidswiss Dec 8, 2023
Maintainer

Great!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to backup in Red Hat OCP cluster #914

{{title}}

Replies: 3 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Failed to backup in Red Hat OCP cluster #914

karthik-murugan-hpe Dec 7, 2023

Replies: 3 comments · 6 replies

Kidswiss Dec 7, 2023 Maintainer

karthik-murugan-hpe Dec 7, 2023 Author

Kidswiss Dec 7, 2023 Maintainer

karthik-murugan-hpe Dec 7, 2023 Author

karthik-murugan-hpe Dec 7, 2023 Author

Kidswiss Dec 7, 2023 Maintainer

karthik-murugan-hpe Dec 7, 2023 Author

karthik-murugan-hpe Dec 7, 2023 Author

Kidswiss Dec 8, 2023 Maintainer

karthik-murugan-hpe
Dec 7, 2023

Replies: 3 comments 6 replies

Kidswiss
Dec 7, 2023
Maintainer

karthik-murugan-hpe
Dec 7, 2023
Author

Kidswiss Dec 7, 2023
Maintainer

karthik-murugan-hpe Dec 7, 2023
Author

karthik-murugan-hpe Dec 7, 2023
Author

Kidswiss Dec 7, 2023
Maintainer

karthik-murugan-hpe Dec 7, 2023
Author

karthik-murugan-hpe
Dec 7, 2023
Author

Kidswiss Dec 8, 2023
Maintainer