[ci] intermittent failures on Observability Integration tests

[DnPlas]

Bug Description

The test_prometheus_grafana_integration_istio_pilot test case that is part of the Observability Integration tests is failing intermittently in the CI, potentially because the test case is querying for metrics that may not be ready when the assertion is called.

This issue is similar to canonical/seldon-core-operator#244 and could be worked as part of canonical/bundle-kubeflow#823.

To Reproduce

So far I have only been able to observe this error in the CI.

Environment

On push or on pull CI.

Relevant Log Output

File "/home/runner/work/istio-operators/istio-operators/tests/test_cos_integration.py", line 84, in test_prometheus_grafana_integration_istio_pilot
    response_metric = response["data"]["result"][0]["metric"]
IndexError: list index out of range

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5403.

This message was autogenerated

[DnPlas]

Another ci failure