Replace apm-server OTLP endpoint with Otel native processors and Elasticsearch

[rogercoll]

Changes

OpenTelemetry native pipeline using Elastic APM enrichment and aggregation processors. Aligns with 8.16 K8s Onboarding flow configuration: https://github.com/elastic/opentelemetry/blob/8.16/resources/kubernetes/operator/helm/values.yaml

Note: The K8s daemonset could forward the gathered signals to the deployment instance, and just have a collector sending data to Elasticsearch. This PR uses the daemonset to send directly to ES, thus not requiring the deployment collector to collect K8s metrics and logs.

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[jasonrhodes]

Thanks for adding this! A few questions off the bat

• What is EDOT
• Where does the APM server we are currently using fit into this?
• Receivers, connectors, processors, exporters - these are all generic OTEL concepts that can be read about in the OTEL docs, is that right?
  • Do we provide custom Elastic versions of any of these, and if so do we have docs for those as well, especially configuration?
• How do these components map to configurations and env variables and config overrides? (This might not be appropriate for this diagram but it's something that comes up a lot)

[rogercoll]

What is EDOT

Elastic Distributions of OpenTelemetry

Where does the APM server we are currently using fit into this?

This PR removes the apm-server dependency and uses the new 8.16 OpenTelemetry native workflow (through the Otel Elasticsearch exporter). Some of the apm-server functionalities have moved to OpenTelemetry components that will be shipping with this PR's configuration.

Receivers, connectors, processors, exporters - these are all generic OTEL concepts that can be read about in the OTEL docs, is that right?

Elastic is gathering all OpenTelemetry related documentation is the elastic/opentelemetry repository, you can read more information about the components/collector in https://github.com/elastic/opentelemetry/blob/main/docs/configure-upstream-collector.md

How do these components map to configurations and env variables and config overrides? (This might not be appropriate for this diagram but it's something that comes up a lot)

That's an interesting topic, and you are right that you need to open the configuration to see their usage. At the moment, those env variables are being used to authenticate with the Elasticsearch endpoint (with the Elasticsearch exporter. I will add some reference regarding this.

Thanks!