Redo export of metrics, traces and logs

[arealmaas]

Introduction

A short and concise description of the functionality this issue addresses.

Description

After a meeting with platform about the steps we need to take to test out Grafana, we discussed some other points related to that.

• We should discuss whether we should fully transition to OpenTelemetry with the collector pattern and remove App Insights-specific components from our code. With this approach, we can use the Container Apps OpenTelemetry agent to send metrics and logs to App Insights and potentially other sources.
• They’re interested in metrics from us, so the on-call team, for instance, can get a holistic view of the status across all systems in Altinn.
• We likely don’t need to do much with traces and logs, as App Insights is already handling these effectively. However, for metrics, we should consider testing the Grafana instance they have. A prerequisite for this is setting up an Azure Monitor Workspace to send metrics to, which they would then use as a datasource.

Implementation

• Add an Azure Monitor Workspace via IaC.
• Use the OT agent in Container Apps (see documentation) to send metrics to Azure Monitor and logs to App Insights.
• Gain access to Grafana and add Azure Monitor as a datasource.
• Increase the coverage of metrics and alerting. Ensure that we get notifications about crashing applications etc.

Tasks

Beta Give feedback

1. Add an Azure Monitor Workspace in our IAC #1462
   0 of 2
   monitoring +1
   
2. Add the OT agent to our container app environment #1463
   monitoring +1
   
3. Add OTLP exporters in our apps that send to the new OT agent #1465
   monitoring +1
   
4. Get access to Grafana, and add AMW as a datasource #1466
   0 of 2
   monitoring +1
   
5. Investigate the possibility to export the logs/traces from container apps(system logs and stdout) into application insights

Threat modelling

Beta Give feedback

1. Does this change introduce any potential security issues?

Acceptance criteria

GIVEN ...
WHEN ....
THEN ...

GIVEN ...
WHEN ....
THEN ...