⚡ Optimize the performance of the search engine

[neon-mmd]

What would you like to share?

Work Expected From The Issue

Provide several optimizations to improve the performance of the search engine by making the following changes:

• Replace FuturesUnordered with tokio::JoinSet to improve peformance.
• Migrate SharedCache intialization code from run function to search function as the SharedCache struct is initialized and used once in the entire app.
• Asyncify all code in the backend except places where code can be parallelized using rayon crate.
• Redesign the code fetching the cached results to fetch cached results in one single round trip.
• Place all heavy computation tasks under tokio::spawn_blocking.
• Hash urls parallely using rayon once every call to the search route.
• Compute a random user agent once every call to the search route.
• Optionally: explore the possibility of redesiging a lock-free implementation for fetching and caching search results in the cache server.

The issue expects the following files to be changed/modified:

• Cargo.lock
• Cargo.toml
• src/bin/websurfx.rs
• src/cache/cacher.rs
• src/cache/encryption.rs
• src/cache/redis_cacher.rs
• src/config/parser.rs
• src/handler/mod.rs
• src/lib.rs
• src/models/aggregation_models.rs
• src/results/aggregator.rs
• src/results/mod.rs
• src/results/user_agent.rs
• src/server/router.rs
• src/server/routes/search.rs
• src/templates/partials/bar.rs
• src/templates/partials/footer.rs
• src/templates/partials/header.rs
• src/templates/partials/navbar.rs
• src/templates/partials/search_bar.rs
• src/templates/partials/settings_tabs/cookies.rs
• src/templates/partials/settings_tabs/engines.rs
• src/templates/partials/settings_tabs/general.rs
• src/templates/partials/settings_tabs/user_interface.rs
• src/templates/views/about.rs
• src/templates/views/index.rs
• src/templates/views/not_found.rs
• src/templates/views/search.rs
• src/templates/views/settings.rs
• tests/index.rs

Note

All the files that are expected to be changed are located under the codebase (websurfx directory).

Reason Behind These Changes

The reason behind having these changes is to improve the performance of the engine which can reduce the time it takes to display the search results, which can drastically improve the user experience.

Do you want to work on this issue?

Yes

Additional information

No response

[github-actions]

The issue has been unlocked and is now ready for dev. If you would like to work on this issue, you can comment to have it assigned to you. You can learn more in our contributing guide https://github.com/neon-mmd/websurfx/blob/rolling/CONTRIBUTING.md