Error during git put tasks leads to data loss (eg tfstate changes)

[gberche-orange]

Expected behavior

Upon mutations (such as tfstate resulting of a terraform apply), resulting git commit should not be lost upon transient git unavailability.

• A non-zero retry should be configured through the attempts support

• Possibly retry the job forever, as to enable operators to

  • fix root cause (and the job would then resume)
  • fly in to retrieve the git commits as a workaround
  • explicitly cancel the hang jobs (e.g. in case of pipeline deadlocks)

• Consider displaying a git patch output in the concourse console, as to enable operators to manually reapply it later on as a manual workaround. This may leak credentials to operators though.

Observed behavior

Put task fails immediately:

remote: error: cannot lock ref 'refs/heads/master': is at 741eed27503195c717bd8925140684050f5202d2 but expected a9157d66de44c3ae0d4fa0dbc91abc18cfebd8d8        
    
To https://elpaaso-gitlab.my.domain.com/fe-group/secrets.git
    
 ! [remote rejected] HEAD -> master (failed to update ref)
    
error: failed to push some refs to 'https://redacted_user:redacted_password@elpaaso-gitlab.my.domain.com/fe-group/secrets.git'
    
failed with non-rebase error

See related #231

Human workaround through fly hijack in the build was not possible (around 30 mins later).

Affected release

Reproduced on version 3.2.2

[gberche-orange]

Notes during backlog review:

Evaluated impact:

• Only terraform tfstate is source of truth in git for data that can't be regenerated using retries

Additional considered alternatives:

• accept the limitation, and try to recover manually lost data (e.g. using terraform import)
• move tf data outside of git: S3, credhub
• on git put failure store to alternate storage: S3, Credhub

=> displaying Git patch seems best approach for now