Enabling DevOps fails with remote error: tls: bad certificate

[Santhosh-Radhakrishna]

Following the standard installation document in my config i enabled devops and on installation installer fails with the below error

"Error from server (InternalError): Internal error occurred: failed calling webhook \"s2ibuildertemplate.kb.io\": failed to call webhook: Post \"https://webhook-server-service.kubesphere-devops-system.svc:443/validate-devops-kubesphere-io-v1alpha1-s2ibuildertemplate?timeout=10s\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-07-01T11:12:31Z is after 2024-02-14T06:08:48Z"

looking the pod status

❯ kubectl get pod -n kubesphere-devops-system
NAME                                 READY   STATUS      RESTARTS   AGE
devops-28663860-ns4np                0/1     Completed   0          22m
devops-apiserver-78cd769fc6-gzwqc    1/1     Running     0          33m
devops-controller-655645dbd7-cxngj   1/1     Running     0          33m
devops-jenkins-7456d589f7-zm4tx      1/1     Running     0          74m
s2ioperator-0                        1/1     Running     0          74m

and looking at the s2ioperator-0 pod logs it says below

❯ k -n kubesphere-devops-system logs -f s2ioperator-0
I0701 10:59:25.988718       1 main.go:49] entrypoint "msg"="setting up client for manager"
I0701 10:59:25.989009       1 main.go:57] entrypoint "msg"="setting up manager"
I0701 10:59:27.090343       1 request.go:645] Throttling request took 1.002489717s, request: GET:https://10.100.0.1:443/apis/telemetry.istio.io/v1alpha1?timeout=32s
I0701 10:59:27.983426       1 listener.go:44] controller-runtime/metrics "msg"="metrics server is starting to listen"  "addr"=":8080"
I0701 10:59:27.983735       1 main.go:70] entrypoint "msg"="Registering Components."
I0701 10:59:27.983756       1 main.go:73] entrypoint "msg"="setting up scheme"
I0701 10:59:27.983894       1 main.go:80] entrypoint "msg"="Setting up controller"
I0701 10:59:27.984020       1 webhook.go:93] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"devops.kubesphere.io","Version":"v1alpha1","Kind":"S2iBuilderTemplate"}
I0701 10:59:27.984066       1 webhook.go:124] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"devops.kubesphere.io","Version":"v1alpha1","Kind":"S2iBuilderTemplate"} "path"="/validate-devops-kubesphere-io-v1alpha1-s2ibuildertemplate"
I0701 10:59:27.984143       1 server.go:130] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-devops-kubesphere-io-v1alpha1-s2ibuildertemplate"
I0701 10:59:27.984246       1 webhook.go:103] controller-runtime/builder "msg"="Registering a mutating webhook"  "GVK"={"Group":"devops.kubesphere.io","Version":"v1alpha1","Kind":"S2iBuilder"} "path"="/mutate-devops-kubesphere-io-v1alpha1-s2ibuilder"
I0701 10:59:27.984288       1 server.go:130] controller-runtime/webhook "msg"="registering webhook" "path"="/mutate-devops-kubesphere-io-v1alpha1-s2ibuilder"
I0701 10:59:27.984346       1 webhook.go:124] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"devops.kubesphere.io","Version":"v1alpha1","Kind":"S2iBuilder"} "path"="/validate-devops-kubesphere-io-v1alpha1-s2ibuilder"
I0701 10:59:27.984392       1 server.go:130] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-devops-kubesphere-io-v1alpha1-s2ibuilder"
I0701 10:59:27.984450       1 webhook.go:93] controller-runtime/builder "msg"="skip registering a mutating webhook, admission.Defaulter interface is not implemented"  "GVK"={"Group":"devops.kubesphere.io","Version":"v1alpha1","Kind":"S2iRun"}
I0701 10:59:27.984476       1 webhook.go:124] controller-runtime/builder "msg"="Registering a validating webhook"  "GVK"={"Group":"devops.kubesphere.io","Version":"v1alpha1","Kind":"S2iRun"} "path"="/validate-devops-kubesphere-io-v1alpha1-s2irun"
I0701 10:59:27.984517       1 server.go:130] controller-runtime/webhook "msg"="registering webhook" "path"="/validate-devops-kubesphere-io-v1alpha1-s2irun"
I0701 10:59:27.984569       1 main.go:102] entrypoint "msg"="start collect s2i metrics"
I0701 10:59:27.984589       1 main.go:106] entrypoint "msg"="start webhook handler"
I0701 10:59:27.984606       1 main.go:110] entrypoint "msg"="Starting the Cmd."
I0701 10:59:28.382820       1 internal.go:406] controller-runtime/manager "msg"="starting metrics server"  "path"="/metrics"
I0701 10:59:28.383588       1 controller.go:158] controller-runtime/manager/controller/s2ibuilder-controller "msg"="Starting EventSource"  "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"runCount":0}}}
I0701 10:59:28.383832       1 controller.go:158] controller-runtime/manager/controller/s2irun-controller "msg"="Starting EventSource"  "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"builderName":""},"status":{}}}
I0701 10:59:28.383993       1 server.go:179] controller-runtime/webhook/webhooks "msg"="starting webhook server"
I0701 10:59:28.384279       1 certwatcher.go:128] controller-runtime/certwatcher "msg"="Updated current TLS certificate"
I0701 10:59:28.384410       1 server.go:222] controller-runtime/webhook "msg"="serving webhook server"  "host"="" "port"=443
I0701 10:59:28.384847       1 certwatcher.go:84] controller-runtime/certwatcher "msg"="Starting certificate watcher"
I0701 10:59:28.486803       1 controller.go:158] controller-runtime/manager/controller/s2irun-controller "msg"="Starting EventSource"  "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"template":{"metadata":{"creationTimestamp":null},"spec":{"containers":null}}},"status":{}}}
I0701 10:59:28.487041       1 controller.go:158] controller-runtime/manager/controller/s2ibuilder-controller "msg"="Starting EventSource"  "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"builderName":""},"status":{}}}
I0701 10:59:28.487073       1 controller.go:165] controller-runtime/manager/controller/s2ibuilder-controller "msg"="Starting Controller"
I0701 10:59:28.587879       1 controller.go:192] controller-runtime/manager/controller/s2ibuilder-controller "msg"="Starting workers"  "worker count"=1
I0701 10:59:28.588049       1 controller.go:158] controller-runtime/manager/controller/s2irun-controller "msg"="Starting EventSource"  "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0701 10:59:28.688885       1 controller.go:165] controller-runtime/manager/controller/s2irun-controller "msg"="Starting Controller"
I0701 10:59:28.688928       1 controller.go:192] controller-runtime/manager/controller/s2irun-controller "msg"="Starting workers"  "worker count"=1
2024/07/01 11:03:56 http: TLS handshake error from 172.16.44.27:40400: remote error: tls: bad certificate
2024/07/01 11:03:57 http: TLS handshake error from 172.16.44.27:40416: remote error: tls: bad certificate
2024/07/01 11:03:57 http: TLS handshake error from 172.16.44.27:40426: remote error: tls: bad certificate
2024/07/01 11:04:00 http: TLS handshake error from 172.16.43.129:53910: remote error: tls: bad certificate
2024/07/01 11:04:00 http: TLS handshake error from 172.16.43.129:53916: remote error: tls: bad certificate
2024/07/01 11:04:01 http: TLS handshake error from 172.16.44.27:40428: remote error: tls: bad certificate

all i did to enable was below in my cluste-config.yml

  devops: 
    enabled: true        
    jenkinsCpuReq: 0.5
    jenkinsCpuLim: 1
    jenkinsMemoryReq: 4Gi
    jenkinsMemoryLim: 4Gi  # Recommend keep same as requests.memory.
    jenkinsVolumeSize: 16Gi

[Santhosh-Radhakrishna]

though the installer reports failed in the UI i see it all green

Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
Start installing alerting
Start installing auditing
Start installing devops
Start installing events
Start installing servicemesh
**************************************************
Waiting for all tasks to be completed ...
task alerting status is successful  (1/9)
task network status is successful  (2/9)
task multicluster status is successful  (3/9)
task openpitrix status is successful  (4/9)
task auditing status is successful  (5/9)
task events status is successful  (6/9)
task servicemesh status is successful  (7/9)
task monitoring status is successful  (8/9)
task devops status is failed  (9/9)
**************************************************
Collecting installation results ...