Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support hashing of search document ID #35

Open
eminano opened this issue Jun 6, 2024 · 0 comments
Open

Support hashing of search document ID #35

eminano opened this issue Jun 6, 2024 · 0 comments
Assignees
Labels

Comments

@eminano
Copy link
Collaborator

eminano commented Jun 6, 2024

Currently the search adapter supports using the primary key as the document ID. It does so by prepending the value of the primary column (or combination of values in the case of a composite primary key) by the table pgstream id.
However, depending on the value of the primary key column, this can get large quickly. Opensearch currently has a size limit of the document ID field of 512 bytes. In order to ensure this limit is not reached, we could hash the primary key value(s) to ensure the limit is not reached.

The ideal solution would be to add an option to the library where the user can provide the hashing mechanism of their choosing, so that the same can be used when querying the data. By default, we'll expose a hashing mechanism that will be used when using the CLI as a default when hashing is selected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant