Dimension's priority value to support High number of Dimensions #277
Replies: 5 comments 7 replies
-
|
An idea I had: You can see that 1, 2, 4, 8 keep repeating with a different number of 0s. We can store this in the DB as:
We can represent a very large number of dimensions this way. When we want to create a context and compute it's weight, we can add the value column and the
This is easy to compute and can even be done on the client, we don't lose any performance all our previously held rules still apply |
Beta Was this translation helpful? Give feedback.
-
|
We can sort first on number of 0s then on the value. The SQL would look like this. If number_of_zero is equal, then we have to compare value |
Beta Was this translation helpful? Give feedback.
-
|
@Datron, your suggestion of storing the weightage using hex values is neat, but with the weightage computation logic that you proposed for contexts, it could lead to collisions |
Beta Was this translation helpful? Give feedback.
-
|
Yeah I applied a very naive way of adding hexs. We need to come up with a way to add them better, that would work |
Beta Was this translation helpful? Give feedback.
-
|
https://chatgpt.com/share/672788a6-787c-8009-b4e7-92f65cebdf8c The above answer has some useful clues - particularly storing large numbers in fixed length binary form (which is better than storing as strings - from a space and sort time perspective). We can actually take the max number of dimensions as a parameter before we onboard a tenant and set it up. |
Beta Was this translation helpful? Give feedback.
-
|
@knutties isn't the max number of dimensions too dynamic ? |
Beta Was this translation helpful? Give feedback.
-
|
We can always give a way to increase the max number of dimensions - one thing to remember is that if dimension count increases beyond the padding length - some maintenance activity is needed to re-compute the priorities in anycase. Basically it cannot be a completely free activity forever. |
Beta Was this translation helpful? Give feedback.
-
|
Postgresql's Numeric Type Approach For dimension we can have another column named order which will store order of dimension's specificity , since context's weightage value can be too large , so we will use postgresql Numeric type Rust/ Diesel supports BigDecimal type which is compatible with Numeric type |
Beta Was this translation helpful? Give feedback.
-
|
Some performance metrics Table schema When big_value column type is string When big_value column type is BigInt When big_value column type is Numeric There isn't seems major difference between Numeric and BigInt type , but String type has worst performance |
Beta Was this translation helpful? Give feedback.
-
|
@pratikmishra356 Can you share the query you ran, also index the big_value column |
Beta Was this translation helpful? Give feedback.
-
|
I have indexed the big_value column not much significant difference query : |
Beta Was this translation helpful? Give feedback.
-
|
Migration Process post above changes We will add
Problem During migration |
Beta Was this translation helpful? Give feedback.
-
Current scenario
We are using 2's exponential as priority's value. Priority of Dimension defines which dimension's specificity in config.
Problem : User has to declare this exponential value, as value is exponential it has limitation on the total number of dimensions that can be defined
New Approach
We will rename
**priority**to**order**, where users can give the order of dimension, Less order means less specificity.During creation of context we will store dimension's order in form of binary string
for example lets say we have 4 dimensions - state , city , vehicle_type, time in same order
so for this context [context."$vehicle_type == 'cab'"]-> value will be "0100" most significant digit will represent the most specific dimension , in this case first 0 will represent time dimension, next 1 will represent
vehicle_typedimension.Context Fetch from DB: we will fetch based on sorting on this context's order value, and in backend we will assign index of that context as the context's order for the user
Beta Was this translation helpful? Give feedback.
All reactions