Milvus supporting 1 million partitions, how to scale the cluster to support that?

[xiaobingxia-at]

Hello Milvus experts, for some reason, I want to have my Milvus cluster to support a large number of physical partitions, like 1 million?
I understand we don't recommend to do that. But still out of curiosity, if we want to do that:

1. which part of cluster we should scale up/out to support that?
2. which configuration I should tune to help support that?
3. What's the concern and bottlenecks of large number of physical partitions? What's the consequence?
   Thanks!

[xiaofan-luan]

I don't think it's possible to do so.
10000 partitions might be doable, 1m definitely not.
The bottle neck will be on coordinator -> too manage that much partitions means load balance will be difficult to do.
Also the growing data ingestion will be challenging (if partitions are written evenly).

With proper optimization 100K might be doable, but this needs special optimization for sure.

[xiaofan-luan]

you can try to relax the limit to 10000 and create 10 collections so you could create 100k tenants. I'm not sure how things will going on with more than 100k partitions because that is way beyond the limit. If not all the tenants write at the same time that might work.

[xiaobingxia-at]

Thank you so much for your detailed reply! really appreciate it!
Can you please elaborate more about the optimization?

• Right now I know I need to give a larger coordinator node. (The number of Coordinator node can only be 1, right? We can only scale up)
• Why growing data ingestion will be challenging? which node is struggling? Which configuration can be tuned to optimize that?
• Another other optimizations we can do?

Another question is: I'm thinking about using the partition key. But with that, I can only load the whole collection into memory cache, instead of loading individual partitions, is that correct? that means my loading time will be much longer?
Thanks!

[xiaofan-luan]

Usually you didn't need to load realease. The easiest way to handle it is to enable mmap feature so could data will be cached in memory page cache and cold data will be cached on disk.

Only when failure recovery happens Milvus will load all data from S3.

With the larger cluster, load cost should be less than couple of 10 minutes and if only one node crash it takes less than 1 minutes to recover

[xiaofan-luan]

I notice you have many questions and I'd glad to take a half hour call to answer all these questions.
ping me at james.luan@zilliz.com if necessary

[xiaobingxia-at]

@xiaofan-luan In my scenario, I need to create 10 million partitions in total, distributed to many different collection.
But for 99.9% of them, after ingestion, I never touched them. Maybe only for 10k partitions, I will actively do some searches.
Do you think that is do-able?

[xiaobingxia-at]

ok. To be specifically, I will release 99.9% of partitions by calling "client.release(collection_name)". I assume they are completely wiped out from the system?
And I just call "client.load_partition()" for only 10k partitions.
So you still think that's won't go well?

[xiaobingxia-at]

There are only 10k partitions being ingested at the same time.

[xiaofan-luan]

I don't think this is a doable in milvus for now.

And support 1m parition is not our plan. Parititonkey is suggested for tenants more than 10k

[xiaobingxia-at]

I sent you an email~~Thank you again!

[xiaobingxia-at]

One more question, if using partition key, we can only load the whole collection, correct?
Since our whole data volume is too huge, only minor percentage of data is alive. If loading the whole collection, that is too expensive. We want to load data into cache at the partition level.

So if using partition key, given a partition value, can we load specific partition to cache? instead of the whole collection?