Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to read JSON credential files present in a directory passed through the <PROVIDER>_APPLICATION_CREDENTIALS environment variable #730

Merged
merged 3 commits into from
May 17, 2024

Conversation

renormalize
Copy link
Member

@renormalize renormalize commented Apr 28, 2024

What this PR does / why we need it:

The dual way of passing credentials to etcd-backup-restore, but the file-type agnostic way of passing credentials in gardener/etcd-druid caused confusion in the community consuming both etcd-druid and etcd-backup-restore.

Passing credentials through a JSON file in etcd-druid does not cause it to error, but is not supported by etcd-druid - and thereby causes etcd-backup-restore to error as a result, even though etcd-backup-restore supports credentials in JSON files.
If consumers familiar with etcd-backup-restore who pass credentials as JSON typically, use the same method to pass credentials in etcd-druid, etcd-backup-restore errors.

To temporarily solve this, if credentials are passed through a JSON file in etcd-druid (i.e. as a file in a directory whose path is exported in <PROVIDER>_APPLICATION_CREDENTIALS), etcd-backup-restore handles this and fetches the credentials from the JSON file. If a JSON file is present in the directory, all other files are ignored.

However, as mentioned in #729, credentials will only be passed to etcd-backup-restore as individual files in a directory in 3 releases from the next (as of writing, v0.31.0), and credentials in the form of JSON files will be deprecated.

This PR makes the following changes:

  • Add support to read JSON credential files present in a directory which is passed through the <PROVIDER>_APPLICATION_CREDENTIALS environment variable for all (applicable) providers.

  • Remove examples which suggest that cloud provider credentials can be passed through a JSON file since this will be deprecated in future releases.

  • Improve error handling in various SnapStore related files for each cloud provider.

  • Updated docs to discourage usage of JSON files to pass credentials.

  • Housekeeping in pkg/server/backuprestoreserver.go. Changed an unused named function parameter to _, and removed a tautological if condition.

Tested with the following cloud providers to test the feature and check for regressions:

  • AWS S3
  • Azure Blob Storage
  • GCS (currently only a service_account.json is accepted so this issue will not be faced.)
  • OpenStack Swift

Which issue(s) this PR fixes:
Fixes #729

Special notes for your reviewer:

Please update the block header and the release note to something that you feel is more appropriate.

Release note:

Passing object store credentials to etcd-backup-restore in the form of a JSON file is now deprecated. Support to pass JSON credentials via file mount (File Path Set through Env: <ProviderName>_APPLICATION_CREDENTIALS_JSON) will be dropped by v0.31.0.
Please switch to passing credentials through non-JSON file as mentioned in https://github.com/gardener/etcd-backup-restore/tree/master/example/storage-provider-secrets and Set File Path through Env: `<ProviderName>_APPLICATION_CREDENTIALS`

…ed through the `${PROVIDER}_APPLICATION_CREDENTIALS` environment variable.

* Add support to read JSON credential files present in a directory which
  is passed through the `${PROVIDER}_APPLICATION_CREDENTIALS` environment
  variable for all (applicable) providers.

* Remove examples which suggest that cloud provider credentials can be
  passed through a JSON file since this will be deprecated in future releases.

* Improve error handling in various `SnapStore` related files for each
  cloud provider.

* Updated docs to discourage usage of JSON files to pass credentials.

* Housekeeping in `pkg/server/backuprestoreserver.go`.
  Changed an unused named function parameter to `_`, and removed a
  tautological if condition.
@renormalize renormalize added kind/bug Bug platform/aws Amazon web services platform/infrastructure platform/openstack OpenStack platform/infrastructure component/etcd-druid ETCD Druid platform/alicloud Alicloud platform/infrastructure platform/azure Microsoft Azure platform/infrastructure labels Apr 28, 2024
@renormalize renormalize added this to the v0.29.0 milestone Apr 28, 2024
@renormalize renormalize requested a review from a team as a code owner April 28, 2024 11:18
@gardener-robot gardener-robot added needs/review Needs review size/l Size of pull request is large (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else labels Apr 28, 2024
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Apr 28, 2024
Copy link
Contributor

@anveshreddy18 anveshreddy18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @renormalize. I left a couple of comments. PTAL

pkg/snapstore/abs_snapstore.go Show resolved Hide resolved
pkg/snapstore/abs_snapstore.go Outdated Show resolved Hide resolved
@gardener-robot gardener-robot added the needs/changes Needs (more) changes label Apr 29, 2024
@renormalize
Copy link
Member Author

Would 3 releases be necessary for deprecation @gardener/etcd-druid-maintainers? We're still in a 0 major version and given the release cadence of etcd-backup-restore, this will take a long time as @anveshreddy18 has mentioned in a review comment.

Being in "alpha" gives us the freedom to deprecate features without any repercussions. However, I do understand that etcd-backup-restore has been used by a lot of members who have come to rely on their particular workflows with regards to passing credentials, and 1 release is not enough time. Would 2 be good enough?

docs/deployment/getting_started.md Outdated Show resolved Hide resolved
pkg/snapstore/abs_snapstore.go Show resolved Hide resolved
pkg/snapstore/s3_snapstore.go Outdated Show resolved Hide resolved
* Remove JSON method from the various ways to pass credentials section.

* Remove a confusing line about passing credentials as individual files.
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 5, 2024
pkg/snapstore/utils.go Outdated Show resolved Hide resolved
docs/deployment/getting_started.md Outdated Show resolved Hide resolved
pkg/snapstore/utils.go Outdated Show resolved Hide resolved
pkg/snapstore/utils.go Outdated Show resolved Hide resolved
pkg/snapstore/utils.go Outdated Show resolved Hide resolved
* Fix comments documenting functions

* Enhance `docs/deployment/getting_started.md`

* Unnecessary argument for error string creation in `pkg/snapstore/utils.go` removed
@gardener-robot-ci-3 gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 13, 2024
@renormalize
Copy link
Member Author

I've addressed your comments @ishan16696, feel free to let me know if you have any more suggestions!

@renormalize renormalize changed the title Add support to read JSON credential files present in a directory passed through the ${PROVIDER}_APPLICATION_CREDENTIALS environment variable Add support to read JSON credential files present in a directory passed through the <PROVIDER>_APPLICATION_CREDENTIALS environment variable May 13, 2024
Copy link
Member

@ishan16696 ishan16696 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. just one nit

pkg/snapstore/abs_snapstore.go Show resolved Hide resolved
Copy link
Member

@ishan16696 ishan16696 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!!

Copy link
Contributor

@anveshreddy18 anveshreddy18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!! Thanks for the PR.

Before merging, pls make sure to test with OpenStack Swift as well. Thanks

@renormalize
Copy link
Member Author

Thanks for the approval @ishan16696 @anveshreddy18!
I've tested it with OpenStack Swift as well, and I've not found any issues.

Thanks for the feedback that made this PR better.

Will merge soon.

@renormalize renormalize merged commit f47e340 into gardener:master May 17, 2024
9 checks passed
@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label May 17, 2024
@renormalize renormalize deleted the json-credential-handling branch July 29, 2024 14:42
@renormalize renormalize self-assigned this Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/etcd-druid ETCD Druid kind/bug Bug needs/changes Needs (more) changes needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) needs/review Needs review needs/second-opinion Needs second review by someone else platform/alicloud Alicloud platform/infrastructure platform/aws Amazon web services platform/infrastructure platform/azure Microsoft Azure platform/infrastructure platform/openstack OpenStack platform/infrastructure size/l Size of pull request is large (see gardener-robot robot/bots/size.py) status/closed Issue is closed (either delivered or triaged)
Projects
None yet
5 participants