-
Notifications
You must be signed in to change notification settings - Fork 182
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How many queues can I create? #487
Comments
Exq polls redis for each and every queue independently using RPOPLPUSH. Let's say your I think you are trying to design a queue per user? If the count could go up to 100k, I would suggest not to go this route. Also, Exq implementation itself might not be optimized. Most deployments I have seen are < 100 queues. |
Agreed with @ananthakumaran. There's also some performance tests you can try here and adapt (we did some optimization a while back but not related to this case): https://github.com/akira/exq/blob/master/test/performance_test.exs |
@akira @ananthakumaran Thank you for your insights on this. I appologize that I didn't respond sooner, but I have been thinking about this problem, especially as the application continues to grow and place more load on Exq. I am noticing more "gremlins" with regards to jobs being processed predictably. Sometimes, I notice that a queue which is subscribed to, simply doesn't execute the jobs within. I have the following thought for a potential workaround to buy me some more time. Basically, out of the thousands of queues which exist in the application, only around 10 - 20 ever have any jobs queued up at a given time. I'm thinking that I can "prune" the queues which don't have any jobs every 5 minutes. Then, when I need to queue up a job for that client later on, the application will create the new queue and subscribe to it dynamically. I expected I could make the "prune" function as follows: def prune() do
{:ok, queues} = Exq.Api.queue_size(Exq.Api)
Enum.each(queues, fn {queue, jobs} ->
if jobs == 0 do
Exq.Api.remove_queue(Exq.Api, queue)
end
end)
end The problem I am facing here though, is that It seems like the queues remain in the Exq cache even after they are removed, causing the application to continue executing the |
Okay, quick update. After switching the above code to use |
This is not necessary and might delete actual jobs. unsubscribe is all you need, though it needs to be run on all worker nodes
From what I understand, you are delaying the job execution (after enqueue) by a few minutes for queues with infrequent jobs. If this is ok, then unsubscribe might work. |
Hey @ananthakumaran, thank you for your thoughtful response. I enqueue jobs for users immediately based on incoming webhooks, with the following logic: def enqueue_job(user) do
# Check for an existing subscription for this user.
{:ok, existing_subscriptions} = Exq.subscriptions(Exq)
subscription_already_exists? = Enum.find(existing_subscriptions, &(&1 == user.name))
unless subscription_already_exists? do
# Subscribe to the queue if a subscription is not already in place.
Exq.subscribe(Exq, user.name)
end
# Enqueue the job immediately, with the subscription is in place.
Exq.enqueue(Exq, user.name, Foo.Worker, [])
end
def prune() do
# Get list of queues along with their queue size
{:ok, queues} = Exq.Api.queue_size(Exq.Api)
{:ok, subscriptions} = Exq.subscriptions(Exq)
Enum.each(queues, fn {queue, jobs} ->
if queue in subscriptions and jobs == 0 do
# If there are no jobs in the subscribed queue, unsubscribe from that queue
Exq.unsubscribe(Exq, queue)
end
end)
end The I ran some tests and this really improved things a lot, since 99% of the queues which were created dynamically have 0 pending jobs at any given moment. This pruning approach keeps the ~10,000 subscriptions which were in place previously (making 20 With this in mind, do you see anything which I might have failed to consider? Once again, I really appreciate your thoughts. Eric |
If you know when jobs are getting enqueued, then this approach would work, though I haven't given much thought about how it would play with multiple worker nodes. Exq also has a Dequeue behaviour which can be overridden. This was added to support rate limiter (see https://github.com/ananthakumaran/exq_limit), you might be able to re-purpose it for your use case. |
Please forgive my ignorance, but what is the reason Exq uses polling as opposed to Pub/Sub? |
Pub/Sub is not persistent. There is a blocking variant of rpoplpush called brpoplpush, but it doesn't support multiple lists redis/redis#1785. Exq also needs to be compatible with Sidekiq, which restricts what data structures are allowed If I were to write a Job processing library today, I would use redis streams, but that's not possible with Exq due to Sidekiq compatibility. |
That makes sense, thanks! |
Sorry, this is not an issue. I'm just wondering if someone from the Exq core team would be able to advise on how many queues can be created. I imagine this might come down to a Redis limitation? What is the upper limit for the number of queues I can create?
So far, my application is managing ~5,600 queues gracefully. How long can this scale for?
The text was updated successfully, but these errors were encountered: