[BUG] http: panic serving: runtime error: invalid memory address or nil pointer dereference

[pyrxm]

Note: Make sure to check out known issues (https://akv2k8s.io/troubleshooting/known-issues/) before submitting

Components and versions
Select which component(s) the bug relates to with [X].

[ ] Controller, version: 1.6.0 (docker image tag)
[x] Env-Injector (webhook), version: 1.6.0 (docker image tag)
[ ] Other

Describe the bug

Replicaset returns event Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: Post "https://akv2k8s-envinjector.akv2k8s.svc:443/pods?timeout=10s": EOF, env injector logs shows invalid memory address or nil pointer dereference error.

To Reproduce
Steps to reproduce the behavior:

1. Install v1.6.0 (with --set "controller.enabled=false" and `--set "env_injector.authService=false")
2. Follow Inject Secret tutorial from documentation.

Expected behavior

Secret injection to work without error.

Logs
If applicable, add logs to help explain your problem.

# v1.5.0 installed
❯ helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
akv2k8s akv2k8s         25              2024-01-05 14:09:01.384911 +0000 UTC    deployed        akv2k8s-2.5.0   1.5.0      

# Example app running
❯ kubectl get pods --namespace xm-test -l app=akvs-secret-app
NAME                               READY   STATUS    RESTARTS   AGE
akvs-secret-app-79b489bc7f-plcrt   1/1     Running   0          72m

# Upgrade to v1.6.0
❯ helm upgrade akv2k8s --install --create-namespace --namespace akv2k8s --repo http://charts.spvapi.no --version 2.6.0 akv2k8s --set "controller.enabled=false" --set "env_injector.authService=false"
Release "akv2k8s" has been upgraded. Happy Helming!

# Verify upgraded
❯ helm list           
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
akv2k8s akv2k8s         26              2024-01-05 15:30:47.644243 +0000 UTC    deployed        akv2k8s-2.6.0   1.6.

# Restart example deployment
❯ kubectl rollout --namespace xm-test restart deployment akvs-secret-app
deployment.apps/akvs-secret-app restarted

# Pod isn't re-created
❯ kubectl get pods --namespace xm-test -l app=akvs-secret-app
NAME                               READY   STATUS    RESTARTS   AGE
akvs-secret-app-79b489bc7f-plcrt   1/1     Running   0          80m

# Events show EOF from POST to https://akv2k8s-envinjector.akv2k8s.svc:443/pods?timeout=10s
❯ kubectl get events --namespace xm-test                            
LAST SEEN   TYPE      REASON              OBJECT                                 MESSAGE
12s         Warning   FailedCreate        replicaset/akvs-secret-app-8597ff46d   Error creating: Internal error occurred: failed calling webhook "pods.env-injector.admission.spv.no": failed to call webhook: Post "https://akv2k8s-envinjector.akv2k8s.svc:443/pods?timeout=10s": EOF
33s         Normal    ScalingReplicaSet   deployment/akvs-secret-app             Scaled up replica set akvs-secret-app-8597ff46d to 1

# Check logs (see below)
❯ kubectl logs akv2k8s-envinjector-8889bcb89-d6n28

# Rollback to previous version
❯ helm rollback akv2k8s 25
Rollback was a success! Happy Helming!

# Confirmed downgraded
❯ helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
akv2k8s akv2k8s         27              2024-01-05 15:35:35.912045 +0000 UTC    deployed        akv2k8s-2.5.0   1.5.0

# Pod is now restarting as expected, secrets injected
❯ kubectl get pods --namespace xm-test -l app=akvs-secret-app                           
NAME                               READY   STATUS        RESTARTS   AGE
akvs-secret-app-79b489bc7f-plcrt   1/1     Terminating   0          85m
akvs-secret-app-8597ff46d-jd5s7    1/1     Running       0          12s

Logs produced by akv2k8s-envinjector:

❯ kubectl logs akv2k8s-envinjector-8889bcb89-d6n28
I0105 15:32:22.840798       1 main.go:139] "found pod to mutate" pod="xm-test/"
I0105 15:32:22.840859       1 pod.go:317] "mutate init-containers" xm-test/="(MISSING)"
I0105 15:32:22.840868       1 pod.go:323] "mutate containers" xm-test/="(MISSING)"
I0105 15:32:22.840875       1 pod.go:138] "found container to mutate" container="xm-test/akv2k8s-env-test"
I0105 15:32:22.840881       1 pod.go:141] "checking for env vars to inject" container="xm-test/akv2k8s-env-test"
I0105 15:32:22.840891       1 pod.go:144] "found env var to inject" env="secret-sync@azurekeyvault" container="xm-test/akv2k8s-env-test"
I0105 15:32:22.840911       1 registry.go:130] "using registry" imageRegistry="index.docker.io"
I0105 15:32:22.840918       1 registry.go:135] "using cloudConfig for registry authentication" config.authType="azureCloudConfig"
2024/01/05 15:32:22 http: panic serving [redacted]:48516: runtime error: invalid memory address or nil pointer dereference
goroutine 216 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0x1b69e20?, 0x2f64150?})
        /usr/local/go/src/runtime/panic.go:920 +0x270
github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.getContainerRegistryRemoteOptions({0x2107ad8, 0x2fbeaa0}, {0x211f660, 0xc000398820}, {{0xc0002272c0, 0x7}, {0x0, 0x0, 0x0}, {0xc000012a80, ...}, ...}, ...)
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:136 +0x339
github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry.(*Registry).GetImageConfig(0xc0003190b0, {0x2107ad8, 0x2fbeaa0}, {0x211f660, 0xc000398820}, {0xc0002272c0, 0x7}, 0xc0001aef00, 0xc00011c588, {0x0})
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/pkg/docker/registry/registry.go:109 +0x3f0
main.getContainerCmd({0x2107ad8, 0x2fbeaa0}, {0x211f660, 0xc000398820}, 0xc0001aef00, 0x0?, {0xc0002272c0, 0x7}, {0x20e8020, 0xc0003190b0})
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:39 +0x3c5
main.podWebHook.mutateContainers({{0x211f660, 0xc000398820}, {0xc0002272c0, 0x7}, {0xc00052a2d0, 0x24}, {0xc000054046, 0x10}, 0x0, 0x0, ...}, ...)
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:165 +0x4db
main.podWebHook.mutatePodSpec({{0x211f660, 0xc000398820}, {0xc0002272c0, 0x7}, {0xc00052a2d0, 0x24}, {0xc000054046, 0x10}, 0x0, 0x0, ...}, ...)
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:324 +0x765
main.vaultSecretsMutator({0x2107b10?, 0xc00064efc0?}, {0x211b800?, 0xc00011c480})
        /go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:160 +0x30e
github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate(0xd67351?, {0x2107b10?, 0xc00064efc0?}, {0x211b800?, 0xc00011c480?})
        /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/mutator.go:25 +0x37
github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.mutatingAdmissionReview({{0x20e9740, 0xc000315160}, {0x20e72a0, 0x1f2fb28}, {{0x1e4050c, 0x1a}, {0x211b800, 0xc000440d80}}, {0x2107a68, 0xc000012d70}}, ...)
        /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:128 +0xab
github.com/slok/kubewebhook/pkg/webhook/mutating.mutationWebhook.Review({{0x20e9740, 0xc000315160}, {0x20e72a0, 0x1f2fb28}, {{0x1e4050c, 0x1a}, {0x211b800, 0xc000440d80}}, {0x2107a68, 0xc000012d70}}, ...)
        /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/mutating/webhook.go:120 +0x28e
github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review(0xc0005e06e0, {0x2107b10, 0xc00064ef90}, 0xc00064ee70)
        /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/webhook/internal/instrumenting/instrumenting.go:42 +0x1fd
github.com/slok/kubewebhook/pkg/http.HandlerFor.func1({0x20fa7a0, 0xc0005620e0}, 0xc000542900)
        /go/pkg/mod/github.com/slok/kubewebhook@v0.11.0/pkg/http/handler.go:64 +0x1e5
net/http.HandlerFunc.ServeHTTP(0xc000542800?, {0x20fa7a0?, 0xc0005620e0?}, 0x1?)
        /usr/local/go/src/net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00017a3c0, {0x20fa7a0, 0xc0005620e0}, 0xc000542700)
        /go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1c5
net/http.serverHandler.ServeHTTP({0x20f03f8?}, {0x20fa7a0?, 0xc0005620e0?}, 0x6?)
        /usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc0001d6240, {0x2107b10, 0xc00043e6f0})
        /usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 83
        /usr/local/go/src/net/http/server.go:3086 +0x5cb

Additional context

Looking to upgrade ASAP to 1.6.0 or higher as 1.5.0 is currently mutating security contexts as per #591 and causing issues. Currently on 1.5.0 to fix #547 for Certificate Injection.

[pyrxm]

Not sure if this relates to changes in #631 given the error trace looks like it is running registry functions when this occurs... or something with the net/http package given GHSA-4374-p667-p6c8 required it was bumped in #621

(Not a strong golang user)

[pyrxm]

Just to follow up, I forked the repo and undid #631 and this was no longer an issue.