Store modules settings in separate CRDs

[nabokihms]

Problem overview

ConfigMap has its limits.

• It is not convenient to write settings for many modules:
  • Users need to always remember about the |
  • Suffix Enabled is special
  • Differences between boolean declarations
  • No syntax highlighting

apiVersion: v1
kind: ConfigMap
metadata:
  name: deckhouse
  namespace: d8-system
spec:
  userAuthzEnabled: "true"
  userAuthz: |
    enabledMultitenancy: true

• It is hard to orient in a big configmap.
• There is no feedback for the user whether their changes are accepted or not.
• We need to validate, edit and migrate the whole configmap, which is a single point of failure.

Describe the solution you'd like to see

1. Create CRD manually (addon-operator will not be managing CRDs).
2. The structure of CRD fields must be like this:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleSettings
metadata:
  name: prometheus # the name of the module
  namespace: d8-system
enabled: true / false
spec:
  {module settings here ... }
status:
  lastCheckTime: 2019-08-08T12:00:00
  condition: accepted

3. User tells add-operator which apiVersion and kind of the CRD to listen instead of ConfigMap.
4. On every change, addon-operator validates spec parameters with an OpenAPI config schema. If validation pass, it sets custom resource status to accepted.
5. If validation fails, it sets custom resource status to validation error with an error message:

status:
  lastCheckTime: 2019-08-08T12:00:00
  condition: error
  message: |
    ...

5. If the module is not supported by the addon-operator, it will write unknownModule to the status:

status:
  lastCheckTime: 2019-08-08T12:00:00
  condition: unknownModule

Additional context

We also need an implementation of MutatingWebhook to write migrations for modules settings to store them in Git.

[nabokihms]

1. Instead of validating CRs in add-operator, consider preventing saving incorrect values to the cluster utilizing Kubernetes native feature - validating webhook. We can implement it using the same code (open API specs store) to perform validation. The only thing left is to connect our code to Kubernetes API.

2. By storing modules settings with a single custom resource definition we benefit from it things like seen all modules configuration with one kubectl command. However, it would be hard to write migrations. We need to add a field to the resource spec with a config version and then develop our own migration system based on mutating or conversion webhook.

[diafour]

Done in the Deckhouse using additional module and small changes in the Addon-operator. We may integrate ModuleConfig into addon-operator and drop configuring via ConfigMap in the future.