diff --git a/designs/helm-chart-autogenerate-plugin.md b/designs/helm-chart-autogenerate-plugin.md new file mode 100644 index 00000000000..329daa55486 --- /dev/null +++ b/designs/helm-chart-autogenerate-plugin.md @@ -0,0 +1,567 @@ +| Authors | Creation Date | Status | Extra | +|----------------------------------------|---------------|--------------|-------| +| @dashanji,@camilamacedo86,@LCaparelli | Sep, 2023 | Implemented | - | + +# New Plugin to allow project distribution via helm charts + +This proposal aims to introduce an optional mechanism that allows users +to generate a Helm Chart from their Kubebuilder-scaffolded project. This will +enable them to effectively package and distribute their solutions. + +To achieve this goal, we are proposing a new native Kubebuilder +[plugin](https://book.kubebuilder.io/plugins/plugins) (i.e., `helm-chart/v1-alpha`) +which will provide the necessary scaffolds. The plugin will function similarly +to the existing [Grafana Plugin](https://book.kubebuilder.io/plugins/grafana-v1-alpha), generating or regenerating HelmChart files using the init and edit +sub-commands (i.e., `kubebuilder init|edit --plugins helm-chart/v1-alpha`). + +An alternative solution could be to implement an alpha command, +similar to the [helper provided to upgrade projects](https://book.kubebuilder.io/reference/rescaffold) that would +provide the HelmChart under the `dist`directory, similar to what +is done by [helmify](https://github.com/arttor/helmify). + +## Example of Usage + +**To enable the helm-chart generation when a project is initialized** + +> kubebuilder init --plugins=`go/v4,helm/v1-alpha` + +**To enable the helm-chart generation after the project be scaffolded** + +> kubebuilder edit --plugins=`helm/v1-alpha` + +> Note that the HelmChart should be scaffold under the `dist/` directory in both scenarios: +> ```shell +> example-project/ +> dist/ +> chart/ +> ``` + + +**To sync the HelmChart with the latest changes and add the manifests generated** + +> kubebuilder edit --plugins=`helm/v1-alpha` + +The above command will be responsible for ensuring that the Helm Chart is properly updated +with the latest changes in the project, including the files generated by +controller-gen when users run make manifests. + +## Open Questions + +### 1) How to manage and scaffold the CRDs for the HelmChart? + +According to [Helm Best Practices for Custom Resource Definitions](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you), +there are two main methods for handling CRDs: + +- **Method 1:Let Helm Do It For You:** Place CRDs in the `crds/` directory. Helm installs these CRDs during the initial + install but does not manage upgrades or deletions. +- **Method 2:Separate Charts:** Place the CRD definition in one chart + and the resources using the CRD in another chart. + This method requires separate installations for each chart. + +**Raised Considerations and Concerns** +- **Use Helm crd directory** The upgraded chart versions will silently ignore CRDs even if they differ from the installed versions. This could lead to surprising and unexpected behavior. Therefore, Kubebuilder should not encourage or promote this approach. +- **Templates Folder**: Moving CRDs to the `templates` folder facilitates upgrades but uninstalls CRDs when the operator is uninstalled. However, it allows users easier manage the CRDs and install them on upgrades. It is a common approach adopted by maintainers but is not considered a good practice by Helm itself. +- **Separate Helm Chart for CRDs:** This approach allows control over both CRD and operator versions without deleting CRDs when the operator chart is deleted. Also, follows the HelmChart best practices. Another problem with this approach is to ensure that the CRDs will be applied before the CRs since both will be under the template directory. +- **When Webhooks are used:** If a CRD specifies, for example, a conversion webhook, the "API chart" needs to contain the CRDs and the webhook `service/workload`. + It would also make sense to include `validating/mutating` webhooks, requiring the scaffolding of separate main modules and image builds for + webhooks and controllers which does not shows to be compatible with Kubebuilder Golang scaffold. + +**Proposed Solution** + +Follow the same approach adopted by [Cert-Manager](https://cert-manager.io/docs/installation/helm/). +Add the CRDs under the `template` directory and have a spec in the `values.yaml` +which will define if the CRDs should or not be applied + +```shell +helm install|upgrade \ + myrelease \ + --namespace my-namespace \ + --set `crds.enabled=true` +``` + +Also, add another spec to the `values.yaml` to not allow the CRDs +be deleted when the helm is uninstalled: + +```yaml + # START annotations {{- if .Values.crds.keep }} + annotations: + helm.sh/resource-policy: keep + # END annotations {{- end }} +``` + +Additionally, we might want to +scaffold separate charts for the APIs and support both. +An example of this approach provided as feedback +was [karpenter-provider-aws](https://github.com/aws/karpenter-provider-aws/tree/main/charts). + +We should either make clear the usage of both supported +ways and clarify their limitations. However, the proposed +solution would result in the following layout: + +``` +example-project/ + dist/ + chart/ + example-project-crd/ + ├── Chart.yaml + ├── templates/ + │ ├── _helpers.tpl + │ ├── crds/ + │ │ └── + └── values.yaml + example-project/ + ├── Chart.yaml + ├── templates/ + │ ├── _helpers.tpl + │ ├── crds/ + │ └── + │ ├── ... +``` + +### 2) How to manage dependencies such as Cert-Manager and Prometheus? + +Helm charts allow maintainers to define dependencies via the `Chart.yaml` file. +However, in the initial version of this plugin at least, we do not need to consider management of dependencies. + +Adding dependencies such as **Cert-Manager** and **Prometheus** directly in the `Chart.yaml` +could introduce issues since these components are intended to be installed only once per cluster. +Attempting to manage multiple installations could lead to conflicts and cause unintended behaviors, +especially in shared cluster environments. + +To avoid these issues, the plugin for now will not scaffold this file and will not try to +manage it. Instead, users will be responsible for managing these dependencies outside of +the generated Helm chart, ensuring they are correctly installed and only installed once in the cluster. + +## Motivation + +Currently, projects scaffolded with Kubebuilder can be distributed via YAML. Users can run +`make build-installer IMG=/:tag`, which will generate `dist/install.yaml`. +Therefore, its consumers can install the solution by applying this YAML file, such as: +`kubectl apply -f https://raw.githubusercontent.com////dist/install.yaml`. + +However, many adopt solutions require the Helm Chart format, such as FluxCD. Therefore, +maintainers are looking to also provide their solutions via Helm Chart. Users currently face the challenges of lacking +an officially supported distribution mechanism for Helm Charts. They seek to: + +- Harness the power of Helm Chart as a package manager for the project, enabling seamless adaptation to diverse deployment environments. +- Take advantage of Helm's dependency management capabilities to simplify the installation process of project dependencies, such as cert-manager. +- Seamlessly integrate with Helm's ecosystem, including FluxCD, to efficiently manage the project. + +Consequently, this proposal aims to introduce a method that allows Kubebuilder users to easily distribute their projects through Helm Charts, a strategy that many well-known projects have adopted: + +- [mongodb](https://artifacthub.io/packages/helm/mongodb-helm-charts/community-operator) +- [cert-manager](https://cert-manager.io/v1.6-docs/installation/helm/#1-add-the-jetstack-helm-repository) +- [prometheus](https://bitnami.com/stack/prometheus-operator/helm) +- [aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller/tree/main/helm/aws-load-balancer-controller) + +**NOTE:** For further context see the [discussion topic](https://github.com/kubernetes-sigs/kubebuilder/discussions/3074) + +## Goals + +- Allow Kubebuilder users distribute their projects using Helm easily. +- Make the best effort to preserve any customizations made to the Helm Charts by the users, which means we will skip syncs in the `values.ymal`. +- Stick with Helm layout definitions and externalize into the relevant values-only options to distribute the default scaffold done by Kubebuilder. We should follow https://helm.sh/docs/chart_best_practices. + +## Non-Goals + +- Converting any Kustomize configuration to Helm Charts like [helmify](https://github.com/arttor/helmify) does. +- Support the deprecated plugins. This option should be supported from `go/v4` and `kustomize/v2` +- Introduce support for Helm in addition to Kustomize, or replace Kustomize with Helm entirely, similar to the approach +taken by Operator-SDK, thereby allowing users to utilize Helm Charts to build their Project. +- Attend standard practices that deviate from Helm Chart layout, definition, or conventions to workaround its limitations. + +## User Stories + +- As a developer, I want to be able to generate a helm chart from a kustomize directory so that I can distribute the helm chart to my users. Also, I want the + generation to be as simple as possible without the need to write any additional duplicate files. +- As a user, I want the helm chart can cover all potential configurations when I deploy it on the Kubernetes cluster. +- As a platform engineer, I want to be able to manage different versions and configurations of a project across multiple clusters and environments based on the same distribution artifact (Helm Chart), with versioning and dependency locking for supply chain security. + +## Implementation Details/Notes/Constraints + +### Plugin Layout + +- **Location and Versioning**: The new plugin should follow Kubebuilder standards and +be implemented under `pkg/plugins/optional`. It should be introduced as an alpha version +(`v1alpha`), similar to the [Grafana plugin](https://github.com/kubernetes-sigs/kubebuilder/tree/master/pkg/plugins/optional/grafana/v1alpha). + +- **The data should be tracked in PROJECT File**: Usage of the plugin should be tracked in the `PROJECT` +file with the input via flags and options if required. Example entry in the `PROJECT` file: + +```yaml +... +plugins: + helm.go.kubebuilder.io/v1-alpha: + options: ## (If ANY) + : +``` + +Ensure that user-provided input is properly tracked, similar to how it's done in +other plugins [(see the code in the plugin.go)](https://github.com/kubernetes-sigs/kubebuilder/blob/c058fb95fe0ccd8d2a3147990251ca501df5eb26/pkg/plugins/golang/deploy-image/v1alpha1/plugin.go#L58-L75) +and the [(code source to track the data)](https://github.com/kubernetes-sigs/kubebuilder/blob/c058fb95fe0ccd8d2a3147990251ca501df5eb26/pkg/plugins/golang/deploy-image/v1alpha1/api.go#L191-L217) +of the deploy-image plugin for reference. + +**NOTE** We might not need options/flags in the first implementation. However, we should still track the plugin as +we do for the Grafana plugin. + +### Plugin Implementation Structure + +Following the structure implementation for the source code of this plugin: + +```shell +. +├── helm-chart +│ └── v1alpha1 +│ ├── init.go +│ ├── edit.go +│ ├── plugin.go +│ └── scaffolds +│ ├── init.go +│ ├── edit.go +│ └── internal +│ └── templates +``` + +### SubCommand implementation + +For each subCommand we will need to check the resources which +are scaffold for each subCommand via the [kustomize](https://kubebuilder.io/plugins/kustomize-v2) plugin +and ensure that we will implement the subCommand of the HelmChart plugin +to the respective scaffolds as well. + +#### To Sync the Manifests Created with `controller-gen` + +Users will need to call the subcommand `edit` passing the plugin to +ensure that the Helm chart is properly synced. + +Therefore, the `PostScaffold` of this command could perform steps such as: + +- **Run `make manifests`**: Generate the latest CRDs and other manifests. +- **Copy the files to Helm chart templates**: + - Copy CRDs: `cp config/crd/bases/*.yaml chart/example-project-crd/templates/crds/` + - Copy RBAC manifests: `cp config/rbac/*.yaml chart/example-project/templates/rbac/` + - Copy webhook configurations: `cp config/webhook/*.yaml chart/example-project/templates/webhook/` + - Copy the manager manifest: `cp config/default/manager.yaml chart/example-project/templates/manager/manager.ymal0` +- **Replace placeholders with Helm values**: Ensure that customized fields, such as the namespace, are properly replaced accordingly. + Example: Replace `name: system` with `{{ .Values.Release.name }}`. + +This ensures the Helm chart is always up-to-date with the latest +manifests generated by Kubebuilder, maintaining consistency with the +configured namespace and other customizable fields. + +We will need to use the utils helpers such as [ReplaceInFile](https://github.com/kubernetes-sigs/kubebuilder/blob/c058fb95fe0ccd8d2a3147990251ca501df5eb26/pkg/plugin/util/util.go#L303-L323) +or [EnsureExistAndReplace](https://github.com/kubernetes-sigs/kubebuilder/blob/c058fb95fe0ccd8d2a3147990251ca501df5eb26/pkg/plugin/util/util.go#L276) +to achieve this goal. + +### HelmChart Values Scaffolded by the Plugin + +- **Allow values.yaml to be fully re-generated with the flag --force**: + +By default, the `values.yaml` file should not +be overwritten. However, users should have the option to overwrite it using +a flag (`--force=true`). + +This can be implemented in the specific template as done for other plugins: + +```go +if f.Force { + f.IfExistsAction = machinery.OverwriteFile +} else { + f.IfExistsAction = machinery.Error +} +``` + +**NOTE:** We will evaluate the cases when we implement `webhook.go` and `api.go` +for the HelmChart plugin. However, we might use the force flag to replicate +the same behavior implemented in the subCommands of the kustomize plugin. +For instance, if the flag is used when creating an API, it forces +the overwriten of the generated samples. Similarly, if the api subCommand +of the HelmChart plugin is called with `--force`, we should replace +all samples with the latest versions instead of only adding the new one. + +- **Helm Chart Templates should have conditions**: + +Ensure templates install resources based on +conditions defined in the `values.yaml`. Example for CRDs: + +``` +# To install CRDs +{{- if .Values.crd.enable }} +... +{{- end }} +``` + +- **Customizable Values**: Set customizable values in the `values.yaml`, + such as defining ServiceAccount names, and whether they should be created or not. + Furthermore, we should include comments to help end-users understand the source + of configurations. Example: + +```yaml +{{- if .Values.rbac.enable }} +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + app.kubernetes.io/name: project-v4 + app.kubernetes.io/managed-by: kustomize + name: {{ .Values.rbac.serviceAccountName }} + namespace: {{ .Release.Namespace }} +{{- end }} +``` + +- **Example of values.yaml**: + +Following an example to illustrate the expected result of this plugin: + +```yaml +# Install CRDs under the template +crd: + enable: false + keep: true + +# Webhook configuration sourced from the `config/webhook` +webhook: + enabled: true + conversion: + enabled: true + +## RBAC configuration under the `config/rbac` directory +rbac: + create: true + serviceAccountName: "controller-manager" + +# Cert-manager configuration +certmanager: + enabled: false + issuerName: "letsencrypt-prod" + commonName: "example.com" + dnsName: "example.com" + +# Network policy configuration sourced from the `config/network_policy` +networkPolicy: + enabled: false + +# Prometheus configuration +prometheus: + enabled: false + +# Manager configuration sourced from the `config/manager` +manager: + replicas: 1 + image: + repository: "controller" + tag: "latest" + resources: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 100m + memory: 64Mi + +# Metrics configuration sourced from the `config/metrics` +metrics: + enabled: true + +# Leader election configuration sourced from the `config/leader_election` +leaderElection: + enabled: true + role: "leader-election-role" + rolebinding: "leader-election-rolebinding" + + +# Controller Manager configuration sourced from the `config/manager` +controllerManager: + manager: + args: + - --metrics-bind-address=:8443 + - --leader-elect + - --health-probe-bind-address=:8081 + containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + image: + repository: controller + tag: latest + resources: + limits: + cpu: 500m + memory: 128Mi + requests: + cpu: 10m + memory: 64Mi + replicas: 1 + serviceAccount: + annotations: {} + +# Kubernetes cluster domain configuration +kubernetesClusterDomain: cluster.local + +# Metrics service configuration sourced from the `config/metrics` +metricsService: + ports: + - name: https + port: 8443 + protocol: TCP + targetPort: 8443 + type: ClusterIP + +# Webhook service configuration sourced from the `config/webhook` +webhookService: + ports: + - port: 443 + protocol: TCP + targetPort: 9443 + type: ClusterIP +``` + +### Optional configurations should be disabled by default + +The HelmChart plugin should not scaffold optional options enabled +when those are scaffolded as disabled by the default implementation +of `kustomize/v2` and consequently the `go/v4` plugin used by default. Example: + +The dependency on Cert-Manager is disabled by default. + +```yaml +From config/default/kusyomization.yaml +# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required. +#- ../certmanager +``` + +Therefore, by default the `values.yaml` should be scaffolded with: + +``` +# Cert-manager configuration +certmanager: + enabled: false +``` + +### Layout of the Helm-Chart + +Following an example of the expected result of this plugin: + +```shell +example-project/ + dist/ + chart/ + example-project-crd/ + ├── Chart.yaml + ├── templates/ + │ ├── _helpers.tpl + │ ├── crds/ + │ │ └── + └── values.yaml + example-project/ + ├── Chart.yaml + ├── templates/ + │ ├── _helpers.tpl + │ ├── crds/ + │ └── + │ ├── certmanager/ + │ │ └── certificate.yaml + │ ├── manager/ + │ │ └── manager.yaml + │ ├── network-policy/ + │ │ ├── allow-metrics-traffic.yaml + │ │ └── allow-webhook-traffic.yaml // Should be added by the plugin subCommand webhook.go + │ ├── prometheus/ + │ │ └── monitor.yaml + │ ├── rbac/ + │ │ ├── kind_editor_role.yaml + │ │ ├── kind_viewer_role.yaml + │ │ ├── leader_election_role.yaml + │ │ ├── leader_election_role_binding.yaml + │ │ ├── metrics_auth_role.yaml + │ │ ├── metrics_auth_role_binding.yaml + │ │ ├── metrics_reader_role.yaml + │ │ ├── role.yaml + │ │ ├── role_binding.yaml + │ │ └── service_account.yaml + │ ├── samples/ + │ │ └── kind_version_admiral.yaml + │ ├── webhook/ + │ │ ├── manifests.yaml + │ │ └── service.yaml + └── values.yaml +``` + +### Update the README + +A README.md is scaffold for the projects. (see its implementation [here](https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/plugins/golang/v4/scaffolds/internal/templates/readme.go)). +Therefore, if the project is scaffold with the HelmChart plugin then, +we should update the [Distribution](https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/README.md#project-distribution) +section to add the info and steps over how to keep the HelmChart synced. + +### Tests and samples + +To ensure that the new plugin will work well we will need to: +- Implement e2e tests for the plugin. (for reference see the e2e tests for the [DeployImage](https://github.com/kubernetes-sigs/kubebuilder/tree/master/test/e2e/deployimage)) +- Ensure that the plugin is scaffold with all samples under the [testdata](https://github.com/kubernetes-sigs/kubebuilder/tree/master/testdata) directory (we will need call the plugin in [test/testdata/generate.sh](https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/testdata/generate.sh#L115-L119)) + +### Documentation + +The new plugin should either be properly documented such as the others. +For reference see: +- [Available Plugins](https://book.kubebuilder.io/plugins/available-plugins) +- [DeployImage Plugin](https://book.kubebuilder.io/plugins/deploy-image-plugin-v1-alpha) + +## Risks and Mitigations + +**Difficulty in Maintaining the Solution** + +Maintaining the solution may prove challenging in the long term, +particularly if it does not gain community adoption and, consequently, collaboration. +To mitigate this risk, the proposal aims to introduce an optional alpha plugin or to +implement it through an alpha command. This approach provides us with greater flexibility +to make adjustments or, if necessary, to deprecate the feature without definitively +compromising support. + +## Proof of Concept + +In order to prove that would be possible we could +refer to the open source tool +[helmify](https://github.com/arttor/helmify). + +## Drawbacks + +**Inability to Handle Complex Kubebuilder Scenarios** + +The proposed plugin may struggle to appropriately handle complex scenarios commonly encountered +in Kubebuilder projects, such as intricate webhook configurations. Kubebuilder’s scaffolded +projects can have sophisticated webhook setups, and translating these accurately into Helm +Charts may prove challenging. This could result in Helm Charts that are not fully reflective +of the original project’s functionality or configurations. + +**Incomplete Generation of Valid and Deployable Helm Charts** + +The proposed solution may not be capable of generating a fully valid and deployable Helm Chart +for all use cases supported by Kubebuilder. Given the diversity and complexity of potential +configurations within Kubebuilder projects, there is a risk that the generated Helm Charts +may require significant manual intervention to be functional. This drawback undermines the +goal of simplifying distribution via Helm Charts and could lead to frustration for users who +expect a seamless and automated process. + +## Alternatives + +**Via a new command (Alternative Option)** + +By running the following command, the plugin will generate a helm chart from the specific kustomize directory and output it to the directory specified by the `--output` flag. + +```shell +kubebuilder alpha generate-helm-chart --from= --output= +``` + +The main drawback of this option is that it does not adhere to the Kubebuilder ecosystem. +Additionally, we would not take advantage of Kubebuilder library features, such as avoiding +overwriting the `values.yaml`. It might also be harder to support and maintain since we would +not have the templates as we usually do. + +Lastly, another con is that it would not allow us to scaffold projects with the plugin +enabled and in the future provide further configurations and customizations for this plugin. +These configurations would be tracked in the `PROJECT` file, allowing integration with other +projects, extensions, and the re-scaffolding of the HelmChart while preserving the inputs +provided by the user via plugins flags as it is done for example for +the [Deploy Image](https://book.kubebuilder.io/plugins/deploy-image-plugin-v1-alpha) plugin.