Kubernetes experimentation #1054
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
There's a testObservability chart that will deploy jaeger, prometheus, and grafana - testing values are within the helmValues/localTesting/testObservability.yaml file. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
… to be deployed hierarchically via helm packages Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…to work as expected. Jaeger & Prometheus have been deployed, but I don't have a collector and haven't confirmed that they're behaving as expected. I still need to 1) setup kafka, flip the proxy to write to kafka; 2) setup the replayer 3) setup otel-collectors; 4) setup RFS. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
… package. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1054 +/- ##
============================================
+ Coverage 80.66% 80.74% +0.08%
- Complexity 2892 2918 +26
============================================
Files 383 399 +16
Lines 14359 14829 +470
Branches 989 1007 +18
============================================
+ Hits 11582 11973 +391
- Misses 2183 2250 +67
- Partials 594 606 +12
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
…ared library WIP Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
…nsearch-migrations into KubernetesExperimentation
…refactoring Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Tanner Lewis <lewijacn@amazon.com>
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
gregschohn
force-pushed
the
KubernetesExperimentation
branch
from
763de17
to
ad8be18
Compare
…o this too. This commit also creates general purpose helper scripts. Helper scripts for the chart to handle threading command line params through ConfigMaps and into the command line. This takes parameters from the values file and put them into ConfigMaps, always creating an immutable one for a default and possibly another for the user to override the setting at runtime. That runtime configmap is created with no value. The pod itself pulls the configmap values in as env variables, preferring the overridden one if it was present. It then strings all of those into the command line invocation. Notice that there are 3 types of values for parameters - ones with arity 0, arity 1, and arity > 2. For arity 0, the keyname will be 'present', for arity 1, the keyname=value, and for greater arities, the data should be passed as a yaml list via the keyname 'list'. For lists, we'll use yq and xargs to parse the contents of the yaml list to turn that into the command line - so those two utilities need to be present in the image that we're running in to take advantage of that feature. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…talling the capture proxy by itself. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…ates. Example syntax: `helm template charts/valueGenerator --set-file "proxy=helmValues/localTesting/captureProxy.yaml" --set-file "source=helmValues/localTesting/sourceElasticsearchCluster.yaml" --set-file "target=helmValues/localTesting/targetOpenSearchCluster.yaml" | grep -v "^---$" | helm install mcc charts/aggregates/mockCustomerClusters -f - --dry-run` Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…verall). Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…ponent/captureProxy or aggregates/mockCustomerClusters. To install the aggregate package, run the values through a pre-template first, e.g.: ``` helm template charts/valueGenerator --set-file "proxy=helmValues/localTesting/captureProxy.yaml" --set-file "source=helmValues/localTesting/sourceElasticsearchCluster.yaml" --set-file "target=helmValues/localTesting/targetOpenSearchCluster.yaml" | grep -v "^---$" | helm install ma charts/aggregates/mockCustomerClusters -f - ``` Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
… that they'll follow the same conventions as the other files in this directory. The expectation is that these support independent installs for those charts OR will be used in an aggregate install but only after applying the valueGenerator template. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
… working toward bringing the console and proxy into the main MA chart. Notice that the proxy is still available as a standalone chart or through the mockCustomerClusters Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…variables across pods. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
gregschohn
force-pushed
the
KubernetesExperimentation
branch
from
ad8be18
to
ef3018a
Compare
Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
This helps canonicalize parameter passing from configs. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…pods. All containers will use an initContainer to setup their arguments from configMaps. That will require restarting the pod to pickup new configs, but I want require full-recycling anyway (as the applications re easier to write and manage). BulkLoad won't come up because it requires some runtime parameters. Console uses a sidecar to update /etc/migration_services.yaml w/ the help of a new container/image that runs some python code to monitor the configs and write to a shared file. Replayer requires more work to support it - positional argument parsing support needs to be added; shared configs; and aliases for parameter names (--otelCollectorEndpoint may be the same or might be different for different services... We should namespace the configMaps w/ the service name - and global ones will be just that... that might complicate the sidecar logic, but we'll see. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
…agement for resources. Signed-off-by: Greg Schohn <greg.schohn@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Issues Resolved
https://opensearch.atlassian.net/browse/MIGRATIONS-1622
Testing
Manual testing only w/ minikube on my mac. See
localTesting.shfor some rough examples.Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.