feat: Include logs of all containers in error message

[Raffy23]

This PR

• improves the debugging experience by including all logs of all containers in a pod when reporting back the logs (related to Add troubleshooting step: logs --all-containers #123)
• adds the failure reason and message / exit code of a container to the error log

Solves

• Fail with a .finished event with proper message and status if initcontainer fails #124

[github-actions]

Unit Test Results

    1 files  13 suites   1s ⏱️
101 tests 95 ✔️ 6 💤 0 ❌

Results for commit 5b376da.

♻️ This comment has been updated with latest results.

[pchila]

Just by quickly looking at the code: I am not sure we want to include always the full logs of all containers all the time (think about the successful runs the would include logs from the init container every time) because the output could become polluted.
I would prefer to have the logs of the container that failed (init or main container) in case of failure and only the logs of the main container in case of success... What do you think @christian-kreuzberger-dtx ?

[christian-kreuzberger-dtx]

@pchila, I believe this is exactly how @Raffy23 implemented it! (Or did he force push some changes :D?) Relevant Code:

if container.status.Reason != "Completed" {

[christian-kreuzberger-dtx]

After some deeper review, here are my change requests:

• Do not output initcontainer logs UNLESS initcontainer or any other container fails (if initcontainer succeeds but the 2nd container fails, we need both log outputs)
• Always output container logs of non-initcontainers
• Ensure minimum interference with existing output, e.g.:
  previous output

Job job-executor-service-job-613f1259-f94f-4948-a3c0-7c97-1 finished successfully!

Logs:
...

• Please keep the container name though, that's super helpful

[github-actions]

The following Docker Images have been built:

• keptncontrib/job-executor-service:0.2.0-dev-pr-214,keptncontrib/job-executor-service:0.2.0-dev-pr-214.202205030805
• keptncontrib/job-executor-service-initcontainer:0.2.0-dev-pr-214,keptncontrib/job-executor-service-initcontainer:0.2.0-dev-pr-214.202205030805

[pchila]

LGTM

[christian-kreuzberger-dtx]

It's not solving #125 (still gives me Task 'Run helm' failed: max poll count reached for job job-executor-service-job-18a4f937-0918-412f-a38a-bdeb-1. Timing out after 5m1.048032241s), The reason behind is that the pod is not spawning at all :D
Other then that, it nicely solves #124 and it provides the logs that we needed in a readable and structured format.
LGTM

[christian-kreuzberger-dtx]

Log Example:

Task 'Run helm' failed: job job-executor-service-job-cba897ca-b3d1-4840-8407-aa77-1 failed. Reason: BackoffLimitExceeded, Message: Job has reached the specified backoff limit

Logs: 
Container init-job-executor-service-job-cba897ca-b3d1-4840-8407-aa77-1 of pod job-executor-service-job-cba897ca-b3d1-4840-8407-aa77-1-fthxr:
2022/05/05 11:53:25 successfully moved file /charts/helloservice.tgz to /keptn/charts/helloservice.tgz


Container job-executor-service-job-cba897ca-b3d1-4840-8407-aa77-1 of pod job-executor-service-job-cba897ca-b3d1-4840-8407-aa77-1-fthxr terminated with an error (Reason: Error, ExitCode: 1):
Release "helloservice" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: Service "helloservice" in namespace "podtato-head-production" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "helloservice": current value is "podtato-head-production-helloservice"