Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn ArangoStore into a crawlee storage provider #19

Open
eaton opened this issue Nov 4, 2022 · 1 comment
Open

Turn ArangoStore into a crawlee storage provider #19

eaton opened this issue Nov 4, 2022 · 1 comment

Comments

@eaton
Copy link
Contributor

eaton commented Nov 4, 2022

Does what it says on the tin. It's not ultra-high priority, but it would be an easy way to get around some of the Request vs UniqueUrl issues: both would be easily queryable.

This may also include revisiting how the DataSet and Fragment Entity types are handled; they may be better matches for Crawlee-style Datasets and KVS data; if they're stored in Arango anyways, using the different storage approach wouldn't prevent us from doing complex queries with DS/KVS data.

@eaton eaton added this to Spidergram Nov 19, 2022
@eaton
Copy link
Contributor Author

eaton commented Jan 3, 2023

This can be broken into a couple of steps:

  • Custom request queue provider
  • Custom Dataset provider
  • Custom KVS provider
  • Potentially use TypeFS for another KVS provider?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant