Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initializing workers with data from parent process #11

Open
Rakdar opened this issue Apr 26, 2017 · 0 comments
Open

Initializing workers with data from parent process #11

Rakdar opened this issue Apr 26, 2017 · 0 comments

Comments

@Rakdar
Copy link

Rakdar commented Apr 26, 2017

Is there a way to send data once to a process after it has been started, but before it gets work from the queue? This way i could reduce the overhead i'll have to send to the worker every time i've got stuff to do.

My use case is to calculate distances from ~20 million coordinates against a set of ~1000. My current solution is the send batches (e.g. 5000) combined with the other 1k in a rpc-call. But this, of course, results in me managing a buffer myself, until i have those 5k coordinates ready, and 20% communcation/parsing overhead due to the repeating set of 1k coordinates.

Another solution would be for the workers to fetch the 1k coordinates on startup themselves from an independend resource, but that doesnt sounds correct either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant