[ISSUE] Zeebe 8.7.0-alpha1 cannot connect to AWS OpenSearch

[Szik]

Describe the issue:

Running a C8 SM cluster for 8.7.0-alpha1 for Monorepo components, only Zeebe is unable to establish a connection to OpensSearch (v2.15) on AWS.

Actual behavior:

Zeebe recieves the error code 401, not Authorized where as all other components that connect to OpenSearch do connect.

Expected behavior:

Zeebe can connect to OpenSearch just as all other components.

How to reproduce:
Setup OpenSearch on AWS with version 2.15 (BasicAuth is used) and configure values.yaml with host, username and password as setted up for opensearch. (elasticsearch disabled)
startup cluster and check the loggs of zeebe.
when deploying and running a porocess via webModeler, no process is visible in operate or optimize. zeebe logs record error of authorization.

Logs:

2024-11-04 12:45:14.080 [Broker-0] [zb-fs-workers-0] [Exporter-1] WARN                                                                                                                                          │
│       io.camunda.search.connect.os.OpensearchConnector - AWS not configured due to: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemP │
│ ropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), Container │
│ CredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment vari │
│ able (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment  │
│ variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityToken │
│ File must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCrede │
│ ntialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProv │
│ ider(): Failed to load credentials from IMDS.]                                                                                                                                                                  │
│ 2024-11-04 12:45:14.080 [Broker-0] [zb-fs-workers-0] [Exporter-1] WARN                                                                                                                                          │
│       io.camunda.search.connect.os.OpensearchConnector - Username and/or password for are empty. Basic authentication for OpenSearch is not used.                                                               │
│ 2024-11-04 12:45:14.160 [Broker-0] [zb-fs-workers-0] [Exporter-1] ERROR                                                                                                                                         │
│       io.camunda.zeebe.broker.exporter - Failed to open exporter 'CamundaExporter'. Retrying...                                                                                                                 │
│ io.camunda.exporter.exceptions.OpensearchExporterException: Failed retrieving mappings from index/index templates with pattern [tasklist-form-8.4.0_*,identity-users-8.7.0_*,operate-process-8.3.0_*,tasklist-m │
│ etric-8.3.0_*,identity-authorizations-8.7.0_*,operate-decision-8.3.0_*,operate-metric-8.3.0_*,operate-decision-requirements-8.3.0_*]                                                                            │
│     at io.camunda.exporter.schema.opensearch.OpensearchEngineClient.getMappings(OpensearchEngineClient.java:146) ~[camunda-exporter-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                          │
│     at io.camunda.exporter.schema.SchemaManager.validateIndices(SchemaManager.java:149) ~[camunda-exporter-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                   │
│     at io.camunda.exporter.schema.SchemaManager.startup(SchemaManager.java:45) ~[camunda-exporter-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                            │
│     at io.camunda.exporter.CamundaExporter.open(CamundaExporter.java:104) ~[camunda-exporter-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                                 │
│     at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.lambda$openExporter$0(ExporterContainer.java:118) ~[zeebe-broker-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                            │
│     at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:59) ~[zeebe-util-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                         │
│     at io.camunda.zeebe.util.jar.ThreadContextUtil.runWithClassLoader(ThreadContextUtil.java:35) ~[zeebe-util-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                │
│     at io.camunda.zeebe.broker.exporter.stream.ExporterContainer.openExporter(ExporterContainer.java:117) ~[zeebe-broker-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                     │
│     at io.camunda.zeebe.broker.exporter.stream.ExporterDirector.lambda$startActiveExportingMode$16(ExporterDirector.java:532) ~[zeebe-broker-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                 │
│     at io.camunda.zeebe.scheduler.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                             │
│     at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:85) [zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                                         │
│     at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:42) [zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                                        │
│     at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:122) [zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                                     │
│     at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:130) [zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                      │
│     at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:108) [zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                                  │
│     at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:227) [zeebe-scheduler-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                                                                     │
│ Caused by: java.io.IOException: Unauthorized access                                                                                                                                                             │
│     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport.extractAndWrapCause(ApacheHttpClient5Transport.java:1150) ~[opensearch-java-2.14.0.jar:?]                                         │
│     at org.opensearch.client.transport.httpclient5.ApacheHttpClient5Transport.performRequest(ApacheHttpClient5Transport.java:158) ~[opensearch-java-2.14.0.jar:?]                                               │
│     at org.opensearch.client.opensearch.indices.OpenSearchIndicesClient.getMapping(OpenSearchIndicesClient.java:919) ~[opensearch-java-2.14.0.jar:?]                                                            │
│     at org.opensearch.client.opensearch.indices.OpenSearchIndicesClient.getMapping(OpenSearchIndicesClient.java:933) ~[opensearch-java-2.14.0.jar:?]                                                            │
│     at io.camunda.exporter.schema.opensearch.OpensearchEngineClient.getCurrentMappings(OpensearchEngineClient.java:269) ~[camunda-exporter-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                   │
│     at io.camunda.exporter.schema.opensearch.OpensearchEngineClient.getMappings(OpensearchEngineClient.java:130) ~[camunda-exporter-8.7.0-SNAPSHOT.jar:8.7.0-SNAPSHOT]                                          │
│     ... 15 more  

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform: GCP, AWS
• Helm CLI version: 3.13.1
• Chart version: snapshot-alpha
• Values file:

global:
  opensearch:
    auth:
      username: admin
      password: ******
    enabled: true
    url:
      scheme: "https"  # Ensure the scheme is set to http
      host: "search-qa-opensearch-testing-o36abofz5ejohm6a3vguqbmsiy.eu-central-1.es.amazonaws.com"  # The service name of your OpenSearch instance
      port: 443
  elasticsearch:
    enabled: false

elasticsearch:
  enabled: false