Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes experimentation #1054

Draft
wants to merge 39 commits into
base: main
Choose a base branch
from

Commits on Sep 1, 2024

  1. Unstructured configs that configure and deploy things to k8s

    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    19007a5 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. more k8s & helm experimentation.

    There's a testObservability chart that will deploy jaeger, prometheus, and grafana - testing values are within the helmValues/localTesting/testObservability.yaml file.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    3e7d858 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Another checkpoint in slowly (& inconsistently) getting more services…

    … to be deployed hierarchically via helm packages
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    785312e View commit details
    Browse the repository at this point in the history
  2. K8s Checkpoint. With minimal testing, the proxy, ES/OS, grafana seem …

    …to work as expected.
    
    Jaeger & Prometheus have been deployed, but I don't have a collector and haven't confirmed that they're behaving as expected.
    I still need to 1) setup kafka, flip the proxy to write to kafka; 2) setup the replayer 3) setup otel-collectors; 4) setup RFS.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    10fac7d View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. K8s checkpoint. Working on getting kafka resources deployed as a helm…

    … package.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    f79142e View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Configuration menu
    Copy the full SHA
    0bffeca View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Minor cleanup

    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    0e201c2 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. another checkpoint

    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    9c6f6f4 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Configuration menu
    Copy the full SHA
    e33f89a View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Experiment with k8 structure pattern

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    1cd8782 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. Partial working state for test environment

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    4abd50e View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Checkpoint - Basic replayer functionality

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    cdbe2b5 View commit details
    Browse the repository at this point in the history
  2. Update services.yaml endpoint

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    03e3431 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Basic working Replayer and RFS in local k8

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    032227b View commit details
    Browse the repository at this point in the history
  2. Swap out to use Kafka operator

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    8054cdc View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Add opensearch operator (not working)

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    53057ae View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2024

  1. Checkpoint after moving to shared volume charts, otel collector in sh…

    …ared library WIP
    
    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    8614014 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Add documentation for getting started with K8 and minor changes

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    e174444 View commit details
    Browse the repository at this point in the history
  2. another checkpoint

    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    29ff913 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Start structure change

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    cc31d0d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    9b6115e View commit details
    Browse the repository at this point in the history
  3. Merge branch 'KubernetesExperimentation' of github.com:gregschohn/ope…

    …nsearch-migrations into KubernetesExperimentation
    gregschohn committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    5981d40 View commit details
    Browse the repository at this point in the history
  4. Checkpoint 10-24 fill in structure more, a couple more services need …

    …refactoring
    
    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    233ac38 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Modify volume structure and checkpoint

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    b1567d4 View commit details
    Browse the repository at this point in the history
  2. Checkpoint probably working state after refactoring

    Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
    lewijacn committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    76cab7f View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2024

  1. Reorder MA helm chart dependencies

    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 27, 2024
    Configuration menu
    Copy the full SHA
    00001e4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c70cf6d View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Move original captureProxy chart from k8s and fold the k8 version int…

    …o this too. This commit also creates general purpose helper scripts.
    
    Helper scripts for the chart to handle threading command line params through ConfigMaps and into the command line.  This takes parameters from the values file and put them into ConfigMaps, always creating an immutable one for a default and possibly another for the user to override the setting at runtime.  That runtime configmap is created with no value.
    
    The pod itself pulls the configmap values in as env variables, preferring the overridden one if it was present.  It then strings all of those into the command line invocation.
    Notice that there are 3 types of values for parameters - ones with arity 0, arity 1, and arity > 2.  For arity 0, the keyname will be 'present', for arity 1, the keyname=value, and for greater arities, the data should be passed as a yaml list via the keyname 'list'.  For lists, we'll use yq and xargs to parse the contents of the yaml list to turn that into the command line - so those two utilities need to be present in the image that we're running in to take advantage of that feature.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    f9751f9 View commit details
    Browse the repository at this point in the history
  2. Cleanup to refactor the common helm helpers and to try to support ins…

    …talling the capture proxy by itself.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    38799c3 View commit details
    Browse the repository at this point in the history
  3. Simple but useful template that can be used JUST to make values templ…

    …ates.
    
    Example syntax: `helm template charts/valueGenerator --set-file "proxy=helmValues/localTesting/captureProxy.yaml"  --set-file "source=helmValues/localTesting/sourceElasticsearchCluster.yaml" --set-file "target=helmValues/localTesting/targetOpenSearchCluster.yaml"   | grep -v "^---$" | helm install mcc charts/aggregates/mockCustomerClusters -f - --dry-run`
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    e6aa8eb View commit details
    Browse the repository at this point in the history
  4. Canonicalize/merge the charts for replayer, PVC components, and MA (o…

    …verall).
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    59b0c62 View commit details
    Browse the repository at this point in the history
  5. Checking bugfixes so that the proxy can startup with helm install com…

    …ponent/captureProxy or aggregates/mockCustomerClusters.
    
    To install the aggregate package, run the values through a pre-template first, e.g.:
    ```
    helm template charts/valueGenerator --set-file "proxy=helmValues/localTesting/captureProxy.yaml"  --set-file "source=helmValues/localTesting/sourceElasticsearchCluster.yaml" --set-file "target=helmValues/localTesting/targetOpenSearchCluster.yaml"   | grep -v "^---$" | helm install ma charts/aggregates/mockCustomerClusters -f -
    ```
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    4ccc3f1 View commit details
    Browse the repository at this point in the history
  6. excise the roots for the jaeger and prometheus localTesting values so…

    … that they'll follow the same conventions as the other files in this directory.
    
    The expectation is that these support independent installs for those charts OR will be used in an aggregate install but only after applying the valueGenerator template.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    296886d View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Checkpoint - Further changes that destabilize the helm charts but are…

    … working toward bringing the console and proxy into the main MA chart.
    
    Notice that the proxy is still available as a standalone chart or through the mockCustomerClusters
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ff16308 View commit details
    Browse the repository at this point in the history
  2. Checkpoint that's working on introducing global/shared configuration …

    …variables across pods.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ef3018a View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. Move helmCommon into sharedResources directory

    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    901cbfa View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2024

  1. Add camel case parameter names in addition to snake case ones.

    This helps canonicalize parameter passing from configs.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    bc5ff89 View commit details
    Browse the repository at this point in the history
  2. Checkpoint - BulkLoad, Console, and Replayer are coming along as K8s …

    …pods.
    
    All containers will use an initContainer to setup their arguments from configMaps.  That will require restarting the pod to pickup new configs, but I want require full-recycling anyway (as the applications re easier to write and manage).
    BulkLoad won't come up because it requires some runtime parameters.
    Console uses a sidecar to update /etc/migration_services.yaml w/ the help of a new container/image that runs some python code to monitor the configs and write to a shared file.
    Replayer requires more work to support it - positional argument parsing support needs to be added; shared configs; and aliases for parameter names (--otelCollectorEndpoint may be the same or might be different for different services...
    We should namespace the configMaps w/ the service name - and global ones will be just that... that might complicate the sidecar logic, but we'll see.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Nov 9, 2024
    Configuration menu
    Copy the full SHA
    156142c View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2024

  1. Checkpoint to start cleaning up namespaces and unifying namespace man…

    …agement for resources.
    
    Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
    gregschohn committed Nov 11, 2024
    Configuration menu
    Copy the full SHA
    2ab2578 View commit details
    Browse the repository at this point in the history