kuberbacproxy & rolling restart not working

[treknado]

Hello,
we are running in some troubles with the operator:
we have a dedicated namespace for the operator and our opensearch clusters

operator version: 2.4.0
opensearch version. 2.10.0

1)

the kube-rbac-proxy is running into TLS handshake errors with the kubernetes node the pod is running on.
2023/12/07 10:01:42 http: TLS handshake error from 10.x.x.x:xxxxx: EOF

we already found the following setting to disable the kubeRbacProxy.

kubeRbacProxy:
  enable: false

what is the purpose of this proxy? what are disadvantages to disable this proxy ? which functionalities are not working without the proxy ?

2)

the operator can not do a succesful rolling restart of our os-clusters. we can see the events in the os-cluster namespace every 5minutes: Starting to rolling restart but nothing happens.

operator logs:

{"level":"info","ts":"2023-12-07T10:10:49.958Z","msg":"Reconciling OpenSearchCluster","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"os-cluster-1"},"namespace":"os-cluster-1","name":"opensearch-cluster","reconcileID":"xXxXx","cluster":{"name":"opensearch-cluster","namespace":"os-cluster-1"}}
{"level":"info","ts":"2023-12-07T10:10:49.966Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"os-cluster-1"},"namespace":"os-cluster-1","name":"opensearch-cluster","reconcileID":"xXxXx","interface":"transport"}
{"level":"info","ts":"2023-12-07T10:10:49.971Z","msg":"Generating certificates","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"os-cluster-1"},"namespace":"os-cluster-1","name":"opensearch-cluster","reconcileID":"xXxXx","interface":"http"}
{"level":"info","ts":"2023-12-07T10:10:49.977Z","msg":"The existing disk size 100Gi is same as passed in disk size 100Gi","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"os-cluster-1"},"namespace":"os-cluster-1","name":"opensearch-cluster","reconcileID":"xXxXx"}

best regards

[treknado]

can someone help here, please ?

[swoehrl-mw]

Hi @treknado. Can you please try to replicate your situation with the current operator version (2.6.0)? There were some fixes regarding restart and change detection.

The idea of the kubeRbacProxy is to protect access to the operator metrics endpoint via Kubernetes RBAC.

[treknado]

hey @swoehrl-mw , our current operator version is 2.6.0.
still no rolling restarts.

the operator encounters an OOMKilled - Exit Code: 137 and has hundreds of restarts, regardless of how much memory the operator gets.

[swoehrl-mw]

Hi @treknado. The OOM Kill is a known problem, please watch #700.
The operator is designed to pick up again after a operator pod restart, so that should not be a real problem, as long as the operator can run for a few minutes at least.
Regarding the rolling restart: Does your OS cluster actually need a restart? Please check the operator logs for lines with "Starting rolling restart of the StatefulSet", "Preparing to restart pod" and "Couldn't proceed with rolling restart for Statefulset". These should give a clue what the operator is actually doing. Also check if your OS cluster is in green state, as that will prevent the restart from continuing.