☂️ Enhance and Stabilise Druid E2E tests

[unmarshall]

What you would like to be added:

e2e tests for etcd-druid should only test etcd-druid and not etcd-backup-restore. Eventually remove all backup-restore or etcd specific tests. These should be done in etcd-backup-restore and etcd-wrapper repositories respectively.

• Use KO to build images so that these are faster.

• Generate all PKI artifacts to be used for e2e tests. This utility should be re-used for any tests (other than e2e tests) that require PKI artifacts.

• Use local provider instead of emulators for etcd-druid e2e tests. hostPath can be configured when creating the backup-secret. Kind already comes bundled with rancher's local-path-provisioner making it possible to use local store as backup-bucket. As a consequence we remove hack/e2e-test/infrastructure. This also considerably simplifies our e2e setup and the time it takes to do the setup also reduces. (Do we need to test with(out) a backup-bucket?) - This needs to be decided.

• Use separate namespace for running e2e tests concurrently (using go-native tests)

• In skaffold.yaml change the image to just etcd-druid. When you invoke kind-up then it will setup a KIND cluster and a local registry. It will also automatically integrate with this registry. This avoids explicit loading of images via kind load and completely removes the need to push dev images to gcr container registry.

• Remove ginkgo with native golang tests. We are removing ginkgo usage from druid. A lot of unit and some IT tests have already been migrated and will be merged with Druid Refactor to Address Multiple Controller Conflicts #777 .

• For tests that have failed, preserve their namespaces so that developers can debug. For tests that have passed cleanup the respective namespaces.

• Do not stop the kind cluster at the end of the test run. Or at least have an option to not do that. For concourse pipeline mandatory cleanup is required but for local runs where an ability to analyse failure is required there we can switch it off.

• Need to add proper compaction and copy-backups-task testing to the e2e test suite. (Add copy-backups-task tests to e2e test suite #705)

• Add secret controller tests to e2e test suite #706

• Introduce etcd-druid upgrade tests #807

  • Add to CI pipeline with PR branch vs master branch (or previous release)

• Test backward compatibility to previous druid version (support for downgrade)

• Test reconciliation after error in previous reconciliation which had caused the etcd cluster to be unready. This test would catch cases such as the one described in Deleting the etcd-bootstrap configmap leads to etcd reconciliation to never succeed #818

• Enable etcd-components webhook and run the e2e tests to check the functionality offered by this webhoook.

• (Performance/Benchmark): Record and publish (as logs) startup times for etcd clusters. Ideally these should be recorded as metrics for all clusters managed by druid on dev/staging/canary/live landscapes. This will help understand any deterioration in the startup times across releases.

• Have capability to put breakpoint to any test and enable debugging from the IDE.

• Should be able to do fast iterations with quasi hot deploy (golang unfortunately does not support real hot-deploy).

Motivation (Why is this needed?):
E2E tests should be simple, comprehensive, fast and stable.

[shreyas-s-rao]

Manual tests that I generally run before merging large PRs, cover different combinations of druid auto-reconcile enabled, single/multi node etcds, backups disabled/enabled (with different providers), TLS disabled/enabled, etc, for various scenarios like:

• etcd creation
• reconciliation
• spec changes
• scale-up of replicas (with different combinations of TLS disabled/enabled)
• hibernation/unhibernation (scale down to 0 and back up to original replicas)
• upgrade of druid from old to new version (with checks for etcd status reconciliation, and later spec reconciliation)
• compaction jobs
• copy-backups tasks

Ex: list of manual tests executed before merging #777

TEST NAME	Druid Auto-Reconcile	Single/Multi Node	Backups (provider)	Etcd Client TLS	Etcd Peer TLS	EtcdBR TLS	TEST RESULT
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Single	NA	FALSE	FALSE	FALSE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Single	NA	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Single	AWS	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	NA	FALSE	FALSE	FALSE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	NA	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	AWS	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	GCP	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	Azure	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	Openstack	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	FALSE	Multi	Local	TRUE	TRUE	TRUE	TRUE
Perform etcd spec changes, check if reconciliation triggered	FALSE	Multi	GCP	TRUE	TRUE	TRUE	TRUE
Scale-up etcd from single-node non-TLS to multi-node non-TLS, hibernate, unhibernate	FALSE	Single	GCP	FALSE	FALSE	FALSE	TRUE
Scale-up etcd from single-node non-TLS to multi-node TLS, hibernate, unhibernate	FALSE	Single	GCP	FALSE	FALSE	FALSE	TRUE
Scale-up etcd from single-node TLS to multi-node TLS, hibernate, unhibernate	FALSE	Single	NA	TRUE	TRUE	TRUE	TRUE
Upgrade druid from master to #777, check status updates, add reconcile annotation, check reconciliation	FALSE	Multi	GCP	TRUE	TRUE	TRUE	TRUE
Deploy etcdcopybackupstask, check success	FALSE	Multi	Local	TRUE	TRUE	TRUE	TRUE
Configure compaction with low threshold, populate etcd, check if compaction jobs are triggered and run	FALSE	Single	AWS	TRUE	TRUE	TRUE	TRUE
Deploy etcd, check reconciliation, hibernate, unhibernate, delete etcd	TRUE	Multi	GCP	TRUE	TRUE	TRUE	TRUE
Perform etcd spec changes, check if reconciliation triggered	TRUE	Multi	GCP	TRUE	TRUE	TRUE	TRUE

[unmarshall]

#833 introduced namespace separation but this will be completely re-written.