Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement local stream offset management instead of using consumer groups. #66

Open
theferrit32 opened this issue Sep 12, 2022 · 1 comment

Comments

@theferrit32
Copy link
Contributor

Since the clinvar-raw producer application has a kubernetes Application-scoped persistent volume now, we can store additional info there like the latest processed offset (similar to how genegraph does it). When pod is redeployed or restarted or updated within the scope of the same Application, the persistent volume of the StatefulSet is not destroyed and the code can restart from the same offset stored there. If the StatefulSet is fully deleted and recreated, it will lose track of where it had processed to previously, but this is probably both okay and desirable.

@theferrit32
Copy link
Contributor Author

This will avoid any of the timeout issues related to this:
#62

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant