Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hasher throws out of memory exception on very large Uint8array instances crashing node #63

Open
wayfarer3130 opened this issue Sep 6, 2023 · 2 comments

Comments

@wayfarer3130
Copy link

Trying to hash a 256 mb Uint8array crashes the node instance with an out of memory. It appears the base object is just too large to be handled this way. It would be nice to not convert array buffers into strings, but hash them raw.

@SkeLLLa
Copy link
Owner

SkeLLLa commented Sep 8, 2023

Hi. Good catch. Would you like to investigate it and make a PR?

@wayfarer3130
Copy link
Author

It is a significant change to do this correctly - the right way of doing this is to provide hash generator to the child nodes so that you never create a full string, but do it piecemeal. The issue with that is it is quite different in design from the current behaviour. That would make it quite a bit faster, because you wouldn't be combining/moving strings in large numbers the way it does now, but trying to do that the way it is now would be hard.

There is a pair of simple changes that would "fix" things - that is, would make them reasonably fast, without changing the structure much:

  1. If the incoming object is an ArrayBuffer type, then return the hash code directly
  2. For the "toString" method on values, return the hash code of ArrayBuffer instances rather than a string representation

Those two changes would allow either simple objects or things contains arrays to be hashed at a reasonable speed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants