K8SPXC-1152: restore stucks on operator restart #1610
+766
−511
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pooknull
requested review from
tplavcic,
nmarukovich,
ptankov,
hors,
egegunes and
inelpandzic
as code owners
inelpandzic
previously approved these changes
Feb 6, 2024
| if err != nil { | ||
| return rr, errors.Wrap(err, "run pitr") | ||
| switch err { | ||
| case errWaitingPods, errWaitingPVC: |
There was a problem hiding this comment.
Could we check these errors on the if line so we can avoid this inner switch?
egegunes
reviewed
Feb 6, 2024
| } | ||
| return rr, nil | ||
| } else { | ||
| if cluster.Status.ObservedGeneration == cluster.Generation && cluster.Status.PXC.Status == api.AppStateReady { |
There was a problem hiding this comment.
not sure about this condition. why do we say waiting for cluster to start only if cluster.Status.PXC.Status is ready?
Comment on lines
92
to
94
| rr := reconcile.Result{ | ||
| RequeueAfter: time.Second * 5, | ||
| } |
There was a problem hiding this comment.
honestly I'm not happy to depend on RequeueAfter but I guess there's no other way
There was a problem hiding this comment.
There is a way to not depend on RequeueAfter, but it will take more time to implement. I would like to do it in a separate PR.
Comment on lines
144
to
148
| if err := s.k8sClient.Get(ctx, types.NamespacedName{Name: svc.Name, Namespace: svc.Namespace}, svc); err != nil { | ||
| if k8serrors.IsNotFound(err) { | ||
| initInProcess = false | ||
| } | ||
| } |
There was a problem hiding this comment.
Suggested change
| if err := s.k8sClient.Get(ctx, types.NamespacedName{Name: svc.Name, Namespace: svc.Namespace}, svc); err != nil { | |
| if k8serrors.IsNotFound(err) { | |
| initInProcess = false | |
| } | |
| } | |
| if err := s.k8sClient.Get(ctx, types.NamespacedName{Name: svc.Name, Namespace: svc.Namespace}, svc); k8serrors.IsNotFound(err) { | |
| initInProcess = false | |
| } |
commit: f86ff81 |
inelpandzic
approved these changes
Feb 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://perconadev.atlassian.net/browse/K8SPXC-1152
CHANGE DESCRIPTION
Problem:
If operator pod is restarted during a restore, it can't continue to the restore process.
Cause:
The current design of the restore process is not designed to continue on operator restart.
Solution:
We should refactor the restore code so that the operator can catch up with the current state of the restore and continue.
CHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
compare/*-oc.yml)?Config/Logging/Testability