* 🔧 Config: Updated `number_of_https_connections` for the HTTPS connection pool.
* (feat) : added the max idle pool per host option
* 📝 docs(config): add documentation back to the `http_adaptive_window_size` option (#532)
---------
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Added new HTTP connection setting to the reqwest::ClientBuilder to timeout requests for fetching the search results from the upstream search engines.
* Provided a config option under the server section of the config to allow users to keep tcp connections alive for each request for a certain period of time
* Update src/results/aggregator.rs
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
* Update src/results/aggregator.rs
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
* Fixed import issue in `tcp_connection_keepalive`
* updated size to u8
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
* Fixed sizer eror in `parser.rs`
---------
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: alamin655 <129589283+alamin655@users.noreply.github.com>
* ♻️ refactor: initialize & store the config & cache structs as a constant (#486)
- initializes & stores the config & cache structs as a static constant.
- Pass the config & cache structs as a static reference to all the
functions handling their respective route.
* ⚡ perf: replace hashmaps with vectors for fetching & aggregating results (#486)
- replace hashmaps with vectors for fetching, collecting & aggregating results as it tends to be contigous & cache efficient data structure.
- refactor & redesign algorithms for fetching & aggregating results
centered around vectors in aggregate function.
* ➕ build: add the future crate (#486)
* ⚡ perf: use `futureunordered` for collecting results fetched from the tokio spawn tasks (#486)
- using the `futureunordered` instead of vector for collecting results
reduces the time it takes to fetch the results as the results do not
need to come in specific order so any result that gets fetched first
gets collected in the `futureunordered` type.
Co-authored-by: Spencerjibz <spencernajib2@gmail.com>
* ⚡ perf: initialize new async connections parallely using tokio spawn tasks (#486)
* ⚡ perf: initialize redis pipeline struct once with the default size of 3 (#486)
* ⚡ perf: reduce branch predictions by reducing conditional code branches (#486)
* ✅ test(unit): provide unit test for the `get_safesearch_level` function (#486)
* ⚡ perf: reduce clones & use index based loop to improve search results filtering performance (#486)
* 🚨 fix(clippy): make clippy/format checks happy (#486)
* 🚨 fix(build): make the cargo build check happy (#486)
* ⚡ perf: reduce the amount of clones, to_owneds & to_strings (#486)
* ⚡ perf: use async crates & methods & make functions async (#486)
* 🔖 chore(release): bump the app version (#486)
---------
Co-authored-by: Spencerjibz <spencernajib2@gmail.com>
Adds the EngineHandler struct
Removes vulnerability where an attacker could send requests cookies with fake engine names and crash the server.
Merged RawSearchResult and SearchResult, as they were functionally identical.
Refactor aggregation function
Rename visiting_url to url, as they are always the same (see upstream engine scalping).
Refactor parsing function to be more readable.
