-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kubernetes experimentation #1054
base: main
Are you sure you want to change the base?
Kubernetes experimentation #1054
Commits on Sep 1, 2024
-
Unstructured configs that configure and deploy things to k8s
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 19007a5 - Browse repository at this point
Copy the full SHA 19007a5View commit details
Commits on Sep 2, 2024
-
more k8s & helm experimentation.
There's a testObservability chart that will deploy jaeger, prometheus, and grafana - testing values are within the helmValues/localTesting/testObservability.yaml file. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3e7d858 - Browse repository at this point
Copy the full SHA 3e7d858View commit details
Commits on Sep 5, 2024
-
Another checkpoint in slowly (& inconsistently) getting more services…
… to be deployed hierarchically via helm packages Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 785312e - Browse repository at this point
Copy the full SHA 785312eView commit details -
K8s Checkpoint. With minimal testing, the proxy, ES/OS, grafana seem …
…to work as expected. Jaeger & Prometheus have been deployed, but I don't have a collector and haven't confirmed that they're behaving as expected. I still need to 1) setup kafka, flip the proxy to write to kafka; 2) setup the replayer 3) setup otel-collectors; 4) setup RFS. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 10fac7d - Browse repository at this point
Copy the full SHA 10fac7dView commit details
Commits on Sep 9, 2024
-
K8s checkpoint. Working on getting kafka resources deployed as a helm…
… package. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f79142e - Browse repository at this point
Copy the full SHA f79142eView commit details
Commits on Sep 30, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0bffeca - Browse repository at this point
Copy the full SHA 0bffecaView commit details
Commits on Oct 1, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0e201c2 - Browse repository at this point
Copy the full SHA 0e201c2View commit details
Commits on Oct 4, 2024
-
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 9c6f6f4 - Browse repository at this point
Copy the full SHA 9c6f6f4View commit details
Commits on Oct 10, 2024
-
Configuration menu - View commit details
-
Copy full SHA for e33f89a - Browse repository at this point
Copy the full SHA e33f89aView commit details
Commits on Oct 11, 2024
-
Experiment with k8 structure pattern
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 1cd8782 - Browse repository at this point
Copy the full SHA 1cd8782View commit details
Commits on Oct 12, 2024
-
Partial working state for test environment
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 4abd50e - Browse repository at this point
Copy the full SHA 4abd50eView commit details
Commits on Oct 14, 2024
-
Checkpoint - Basic replayer functionality
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for cdbe2b5 - Browse repository at this point
Copy the full SHA cdbe2b5View commit details -
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 03e3431 - Browse repository at this point
Copy the full SHA 03e3431View commit details
Commits on Oct 16, 2024
-
Basic working Replayer and RFS in local k8
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 032227b - Browse repository at this point
Copy the full SHA 032227bView commit details -
Swap out to use Kafka operator
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 8054cdc - Browse repository at this point
Copy the full SHA 8054cdcView commit details
Commits on Oct 17, 2024
-
Add opensearch operator (not working)
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 53057ae - Browse repository at this point
Copy the full SHA 53057aeView commit details
Commits on Oct 20, 2024
-
Checkpoint after moving to shared volume charts, otel collector in sh…
…ared library WIP Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 8614014 - Browse repository at this point
Copy the full SHA 8614014View commit details
Commits on Oct 22, 2024
-
Add documentation for getting started with K8 and minor changes
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for e174444 - Browse repository at this point
Copy the full SHA e174444View commit details -
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 29ff913 - Browse repository at this point
Copy the full SHA 29ff913View commit details
Commits on Oct 24, 2024
-
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for cc31d0d - Browse repository at this point
Copy the full SHA cc31d0dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9b6115e - Browse repository at this point
Copy the full SHA 9b6115eView commit details -
Merge branch 'KubernetesExperimentation' of github.com:gregschohn/ope…
…nsearch-migrations into KubernetesExperimentation
Configuration menu - View commit details
-
Copy full SHA for 5981d40 - Browse repository at this point
Copy the full SHA 5981d40View commit details -
Checkpoint 10-24 fill in structure more, a couple more services need …
…refactoring Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 233ac38 - Browse repository at this point
Copy the full SHA 233ac38View commit details
Commits on Oct 25, 2024
-
Modify volume structure and checkpoint
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for b1567d4 - Browse repository at this point
Copy the full SHA b1567d4View commit details -
Checkpoint probably working state after refactoring
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Configuration menu - View commit details
-
Copy full SHA for 76cab7f - Browse repository at this point
Copy the full SHA 76cab7fView commit details
Commits on Oct 27, 2024
-
Reorder MA helm chart dependencies
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 00001e4 - Browse repository at this point
Copy the full SHA 00001e4View commit details -
Configuration menu - View commit details
-
Copy full SHA for c70cf6d - Browse repository at this point
Copy the full SHA c70cf6dView commit details
Commits on Oct 31, 2024
-
Move original captureProxy chart from k8s and fold the k8 version int…
…o this too. This commit also creates general purpose helper scripts. Helper scripts for the chart to handle threading command line params through ConfigMaps and into the command line. This takes parameters from the values file and put them into ConfigMaps, always creating an immutable one for a default and possibly another for the user to override the setting at runtime. That runtime configmap is created with no value. The pod itself pulls the configmap values in as env variables, preferring the overridden one if it was present. It then strings all of those into the command line invocation. Notice that there are 3 types of values for parameters - ones with arity 0, arity 1, and arity > 2. For arity 0, the keyname will be 'present', for arity 1, the keyname=value, and for greater arities, the data should be passed as a yaml list via the keyname 'list'. For lists, we'll use yq and xargs to parse the contents of the yaml list to turn that into the command line - so those two utilities need to be present in the image that we're running in to take advantage of that feature. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f9751f9 - Browse repository at this point
Copy the full SHA f9751f9View commit details -
Cleanup to refactor the common helm helpers and to try to support ins…
…talling the capture proxy by itself. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 38799c3 - Browse repository at this point
Copy the full SHA 38799c3View commit details -
Simple but useful template that can be used JUST to make values templ…
…ates. Example syntax: `helm template charts/valueGenerator --set-file "proxy=helmValues/localTesting/captureProxy.yaml" --set-file "source=helmValues/localTesting/sourceElasticsearchCluster.yaml" --set-file "target=helmValues/localTesting/targetOpenSearchCluster.yaml" | grep -v "^---$" | helm install mcc charts/aggregates/mockCustomerClusters -f - --dry-run` Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for e6aa8eb - Browse repository at this point
Copy the full SHA e6aa8ebView commit details -
Canonicalize/merge the charts for replayer, PVC components, and MA (o…
…verall). Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 59b0c62 - Browse repository at this point
Copy the full SHA 59b0c62View commit details -
Checking bugfixes so that the proxy can startup with helm install com…
…ponent/captureProxy or aggregates/mockCustomerClusters. To install the aggregate package, run the values through a pre-template first, e.g.: ``` helm template charts/valueGenerator --set-file "proxy=helmValues/localTesting/captureProxy.yaml" --set-file "source=helmValues/localTesting/sourceElasticsearchCluster.yaml" --set-file "target=helmValues/localTesting/targetOpenSearchCluster.yaml" | grep -v "^---$" | helm install ma charts/aggregates/mockCustomerClusters -f - ``` Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 4ccc3f1 - Browse repository at this point
Copy the full SHA 4ccc3f1View commit details -
excise the roots for the jaeger and prometheus localTesting values so…
… that they'll follow the same conventions as the other files in this directory. The expectation is that these support independent installs for those charts OR will be used in an aggregate install but only after applying the valueGenerator template. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 296886d - Browse repository at this point
Copy the full SHA 296886dView commit details
Commits on Nov 5, 2024
-
Checkpoint - Further changes that destabilize the helm charts but are…
… working toward bringing the console and proxy into the main MA chart. Notice that the proxy is still available as a standalone chart or through the mockCustomerClusters Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for ff16308 - Browse repository at this point
Copy the full SHA ff16308View commit details -
Checkpoint that's working on introducing global/shared configuration …
…variables across pods. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for ef3018a - Browse repository at this point
Copy the full SHA ef3018aView commit details
Commits on Nov 6, 2024
-
Move helmCommon into sharedResources directory
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 901cbfa - Browse repository at this point
Copy the full SHA 901cbfaView commit details
Commits on Nov 9, 2024
-
Add camel case parameter names in addition to snake case ones.
This helps canonicalize parameter passing from configs. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for bc5ff89 - Browse repository at this point
Copy the full SHA bc5ff89View commit details -
Checkpoint - BulkLoad, Console, and Replayer are coming along as K8s …
…pods. All containers will use an initContainer to setup their arguments from configMaps. That will require restarting the pod to pickup new configs, but I want require full-recycling anyway (as the applications re easier to write and manage). BulkLoad won't come up because it requires some runtime parameters. Console uses a sidecar to update /etc/migration_services.yaml w/ the help of a new container/image that runs some python code to monitor the configs and write to a shared file. Replayer requires more work to support it - positional argument parsing support needs to be added; shared configs; and aliases for parameter names (--otelCollectorEndpoint may be the same or might be different for different services... We should namespace the configMaps w/ the service name - and global ones will be just that... that might complicate the sidecar logic, but we'll see. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 156142c - Browse repository at this point
Copy the full SHA 156142cView commit details
Commits on Nov 11, 2024
-
Checkpoint to start cleaning up namespaces and unifying namespace man…
…agement for resources. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 2ab2578 - Browse repository at this point
Copy the full SHA 2ab2578View commit details