You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is there a way to send data once to a process after it has been started, but before it gets work from the queue? This way i could reduce the overhead i'll have to send to the worker every time i've got stuff to do.
My use case is to calculate distances from ~20 million coordinates against a set of ~1000. My current solution is the send batches (e.g. 5000) combined with the other 1k in a rpc-call. But this, of course, results in me managing a buffer myself, until i have those 5k coordinates ready, and 20% communcation/parsing overhead due to the repeating set of 1k coordinates.
Another solution would be for the workers to fetch the 1k coordinates on startup themselves from an independend resource, but that doesnt sounds correct either.
The text was updated successfully, but these errors were encountered:
Is there a way to send data once to a process after it has been started, but before it gets work from the queue? This way i could reduce the overhead i'll have to send to the worker every time i've got stuff to do.
My use case is to calculate distances from ~20 million coordinates against a set of ~1000. My current solution is the send batches (e.g. 5000) combined with the other 1k in a rpc-call. But this, of course, results in me managing a buffer myself, until i have those 5k coordinates ready, and 20% communcation/parsing overhead due to the repeating set of 1k coordinates.
Another solution would be for the workers to fetch the 1k coordinates on startup themselves from an independend resource, but that doesnt sounds correct either.
The text was updated successfully, but these errors were encountered: