Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stable experiments #25

Merged
merged 17 commits into from
Sep 15, 2023
Merged

Stable experiments #25

merged 17 commits into from
Sep 15, 2023

Conversation

timebertt
Copy link
Owner

This PR makes the experiment environments more stable to get (more) reproducible test results for the Master's thesis:

  • the shoot control plane is now running on dedicated seed nodes (non-overcommit flavors), etcd and kube-apiserver are running on isolated high-CPU nodes: f50df33, f50df33
  • the shoot nodes are using non-overcommit flavors: f50df33
  • static requests and limits for etcd, kube-apiserver (no autoscaling, Guaranteed QoS class without CPU throttling, static performance): 7899b35
  • only use a single replica of kube-apiserver (otherwise, API request distribution across instances varies between experiment runs): 7899b35
  • prevent experiment disruptions due to regular shoot reconciliations (might restart control plane components): shoot is only reconciled in its maintenance window
  • continuous profiling is always enabled (static performance impact on all experiment runs): f05f380

The PR also drops a few scalability limitations in the operator and the experiment setup itself, which were discovered while working on more reproducible test results:

  • increase concurrency of controllers to reasonable values: b46c39d
  • reduce the log volume of the operator (causes high i/o wait time in operator on low-IOPS root disks): e61ffad, 772f388, e4e66e4
  • drop kyverno webhooks for patching load test objects (add high latency to API calls): b92c8ac

@timebertt
Add helper for keeping experiment worker scaled up
e79aa95
@timebertt
Ensure same control plane scale between load test runs
7899b35
@timebertt
Add tests for kyverno policies
09a861d
@timebertt
Don't use webhooks for patching websites in load tests
b92c8ac 
Webhooks add a tremendous amount of latency to API calls, especially during load tests when the kyverno admission controller is overloaded.
High latency of API calls extremely decreases the operator's reconciliation rate which makes it difficult to achieve the desired load.
We are not load testing kyverno or the API server's webhook machinery, so get that off the critical path.
@timebertt
Reduce log volume on cluster
e61ffad
@timebertt
Reduce log volume of operator
772f388
@timebertt
Reduce log volume of sharding reconcilers
e4e66e4
@timebertt
Increase concurrency of controllers
b46c39d
@timebertt
Use non-overcommit machine flavors for more predictable performance
f50df33
@timebertt
Move shoot to project timebertt
438d25c
@timebertt
Disable hibernation
6602c28
@timebertt
Drop no longer needed metric filters
c46d76f
@timebertt
Drop unneeded kustomize tool
0d9ab43
@timebertt
Ensure correct owners of DNS records
8dc66a8
@timebertt
Extract grafana password handling into script
2463451
@timebertt
Add Ingress for parca
f05f380
@timebertt
Schedule etcd-main and kube-apiserver to high-cpu worker pool
2c71755
@timebertt timebertt merged commit f07f223 into master Sep 15, 2023
5 checks passed
@timebertt timebertt deleted the stable-experiments branch September 15, 2023 09:31
@timebertt timebertt mentioned this pull request Oct 24, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@timebertt