0
0
mirror of https://github.com/neon-mmd/websurfx.git synced 2024-12-22 12:28:21 -05:00
Commit Graph

293 Commits

Author SHA1 Message Date
Evan Yang
2e64fd5cbc
feat(config): config option to use operating system certificates alongside rustls certificates (#620)
* TLS certificates

* 🚨 fix: make cargo checks happy (#523)

* 🚨 fix: make cargo format checks happy (#557)

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2024-12-05 23:09:01 +03:00
Jorge Botto
ef0ae2f0aa
Fixing Librex SearchResultParser results selector (#619) 2024-11-02 15:36:14 +00:00
Laptop
56bfcbba0e
less javascript (#621) 2024-10-18 10:16:32 +00:00
neon_arch
d52da3aeb4
Clear button for the search bar (#608)
* feat(icons): add a new icon for clear button on the search bar (#394)

*  feat: add clear button near the search bar input field (#394)

*  feat: add a function to clear search input text on click event (#394)

*  feat: disable unstable webkit search and clear button (#394)

* 🔖 chore(release): bump the app version (#394)

* 🩹 fix: rename the pseudo css selector according to the `mdn` docs (#394)

Co-authored-by: Jann Marc Villablanca <31008330+jfvillablanca@users.noreply.github.com>

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jann Marc Villablanca <31008330+jfvillablanca@users.noreply.github.com>
2024-10-08 12:09:41 +00:00
Zacherina
42c30aaaba
🐛 fix(search): Resolve random delay condition during search for individual/non-production self-hosted server (#617) 2024-10-07 02:58:06 +03:00
Jurijs Bibicevs
959d0c52b1
♻️ refactor: replace deprecated per_second with seconds_per_request in rate-limiting middleware code (#616) 2024-10-06 19:30:11 +03:00
Łukasz Mariański
c796ae8bb7
Option to use a proxy for outgoing upstream search engine requests (#573)
* Add option to use a proxy for outgoing (search engine) requests.

* Enable socks feature in reqwest

* Fix formatting

* add proxy feature

* Update src/config/parser.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update websurfx/config.lua

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update Cargo.toml

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* fix

* Update Cargo.toml

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2024-10-05 03:47:36 +00:00
Kekma
2a4dd07752
feat: config option to configure the number of connections to have in the reqwest pool (#552)
* 🔧 Config: Updated `number_of_https_connections` for the HTTPS connection pool.

* (feat) : added the max idle pool per host option

* 📝 docs(config): add documentation back to the `http_adaptive_window_size` option (#532)

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2024-09-30 15:22:25 +03:00
neon_arch
1883093dc7 🚨 chore: make rust format checks happy (#603) 2024-09-06 18:16:02 +05:30
neon_arch
28231d99b8 perf: move the random delay code in the search function (#603) 2024-09-05 22:14:15 +05:30
neon_arch
4afc07f1d5 perf: reduce the usage of clones (#603) 2024-09-05 22:12:34 +05:30
neon_arch
f2d2068bcf perf: replace String & vectors with Cow data types (#603) 2024-09-05 22:10:02 +05:30
neon_arch
b1bcf41061 perf: replace Vec<T> with Box<[T]> & refactor calculate_tfidf function (#603) 2024-09-05 22:07:10 +05:30
neon_arch
39af9096ef perf: replace Vec<T> with Box<T> & initialize vectors with capacity by default (#603) 2024-09-05 22:03:23 +05:30
neon_arch
4bb6c5e90b perf: initialize vectors with capacity by default & use Arc<T> to partially clone memory cache struct (#603) 2024-09-05 21:57:18 +05:30
neon_arch
acee5d892d 🐛 fix: replace deprecated set_ex command with set_options in cache_json function (#592) 2024-09-02 21:17:31 +05:30
neon_arch
9a5f1c5f44 🐛 fix: reimplement caching code within the search function (#592)
- reduce resource usage &
- only cache search results which has not been cached before.
2024-09-02 21:10:54 +05:30
ddotthomas
5d06cce220
feat(config): option to keep the websurfx server connection alive for a certain period for subsequent requests (#568) 2024-05-09 17:24:12 +00:00
Todor Todorov
93fd8f8565
🎨 style: change http/https header word/sentence casing (#555)
Currently all HTTP request headers are in snake case and upper case.
The adopted standard is kebab case and title case.

Here you will find more details:
https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Standard_request_fields
2024-04-25 20:31:47 +03:00
Kekma
b1df4f1154
feat(config): config option to timeout idle connections within the reqwest pool (#551) 2024-03-30 16:05:44 +03:00
Spencer
bb50e8bb25
Ranking of aggregated search results based on relevancy of the search result to the user's search query (#549)
* add sorting by relevance and merge new changes

* fix conflicts

* Update src/models/aggregation_models.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update src/models/aggregation_models.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update Cargo.toml

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update Cargo.toml

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update Cargo.toml

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* enable non-static-synonyms features

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2024-03-25 09:16:49 +00:00
Kekma
c584a7d601
feat(config): config option to keep tcp connection alive for a certain period for subsequent requests (#548)
* Added new HTTP connection setting to the reqwest::ClientBuilder to timeout requests for fetching the search results from the upstream search engines.

* Provided a config option under the server section of the config to allow users to keep tcp connections alive for each request for a certain period of time

* Update src/results/aggregator.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Update src/results/aggregator.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Fixed import issue in `tcp_connection_keepalive`

* updated size to u8

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Fixed sizer eror in `parser.rs`

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: alamin655 <129589283+alamin655@users.noreply.github.com>
2024-03-25 12:11:56 +03:00
Kekma
2149e32c9e
feat: config option to timeout requests for fetching search results from the upstream search engines (#545) 2024-03-18 22:33:52 +03:00
neon_arch
991f3f59de
perf: several optimizations for improving the performance of the engine (#540)
* ♻️ refactor: initialize & store the config & cache structs as a constant (#486)
- initializes & stores the config & cache structs as a static constant.
- Pass the config & cache structs as a static reference to all the
  functions handling their respective route.

*  perf: replace hashmaps with vectors for fetching & aggregating results (#486)
- replace hashmaps with vectors for fetching, collecting & aggregating results as it tends to be contigous & cache efficient data structure.
- refactor & redesign algorithms for fetching & aggregating results
  centered around vectors in aggregate function.

*  build: add the future crate (#486)

*  perf: use `futureunordered` for collecting results fetched from the tokio spawn tasks (#486)
- using the `futureunordered` instead of vector for collecting results
  reduces the time it takes to fetch the results as the results do not
  need to come in specific order so any result that gets fetched first
  gets collected in the `futureunordered` type.

Co-authored-by: Spencerjibz <spencernajib2@gmail.com>

*  perf: initialize new async connections parallely using tokio spawn tasks (#486)

*  perf: initialize redis pipeline struct once with the default size of 3 (#486)

*  perf: reduce branch predictions by reducing conditional code branches (#486)

*  test(unit): provide unit test for the `get_safesearch_level` function (#486)

*  perf: reduce clones & use index based loop to improve search results filtering performance (#486)

* 🚨 fix(clippy): make clippy/format checks happy (#486)

* 🚨 fix(build): make the cargo build check happy (#486)

*  perf: reduce the amount of clones, to_owneds & to_strings (#486)

*  perf: use async crates & methods & make functions async (#486)

* 🔖 chore(release): bump the app version (#486)

---------

Co-authored-by: Spencerjibz <spencernajib2@gmail.com>
2024-03-11 12:01:30 +03:00
aocosmic
11d23fd1b2
💄 chore(simple): modernize the appearance of about page & make it responsive. (#530)
* lazyload search result text with css content-visibility

* typo, lightning-fast is grammatically correct

* revamped logo to look more modern

* added class to control how new logo looks and behaves

* added class to control how new logo looks and behaves

* Replace old websurfx logo in readme file

* major visual revamp

* removed logo files and switched to inline svg

* added `--logo-color` to make svg logo colorscheme adaptive

* Update public/static/themes/simple.css

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* [CodeFactor] Apply fixes

* chores: fix duplicate footer selector and remove unknown pseudo-class

* chores: remove missed  duplicate footer selector

* chores: pass formatter check by deleting empty line at 18

* fix search result title disappearing after being visited

* 💄 style: revamped the about page to look more modern and responsive

* [CodeFactor] Apply fixes

* chore: remove invalid css property

* chore: remove lines between or after svg string literals to conform with proper rust formatting

* chore: remove blank lines between svg literals to conform with proper Rust formatting

* refactor: changed decimal values to percentage, added quotation in urls, add blank lines to before rule declarations and between font-face rules

* refactor: correctly remove blank lines between and after svg string literals to conform with proper rust formatting

* chore: revert percentage values into decimals and trimmed blank lines before first rule inside media queries

* combine font face queries into one

* fix: added missing quotation marks on font face url

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: alamin655 <mdalamin655@outlook.com>
Co-authored-by: codefactor-io <support@codefactor.io>
Co-authored-by: alamin655 <129589283+alamin655@users.noreply.github.com>
2024-03-02 09:50:35 +03:00
ddotthomas
41ab8a2a76
config option to configure the https adaptive window size for requests (#529)
* Added config option to enable the reqwest client adaptive window

* Change adaptive window config name

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* Modified documentation

* Trimmed down aggregate parameters

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2024-02-28 15:08:29 +03:00
Jann Marc Villablanca
669e365913
feat: add new helper function to fetch upstream search engine JSON response (#504)
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2024-01-30 13:37:50 +00:00
abdulahad5112
851ea314a7
🐛 parsed_cet not found in scope error when building the app with the no-cache feature #498 (#502)
Co-authored-by: alamin655 <129589283+alamin655@users.noreply.github.com>
2024-01-30 07:14:31 +00:00
abdulahad5112
fbf73634ee
🐛 Undeclared mini-mocha crate error when building the app with features other than memory-cache #493 (#501) 2024-01-30 07:09:33 +00:00
Spencerjibz
51214dc23a add support for caching N number of key-values pairs instead of only 3 2024-01-24 22:56:11 +00:00
Spencerjibz
6b9469e4b3 cache the next, current and previous results in a separate task 2024-01-22 19:29:08 +00:00
Spencerjibz
f5cf5f9151 changed cache_results method to support multiple values 2024-01-22 19:26:35 +00:00
Spencerjibz
c762f9cf8e use redis pipeline to set multiple values at once 2024-01-22 19:20:16 +00:00
ddotthomas
31c9c676ee Change get_safe_search_level logic back 2024-01-12 09:51:31 -07:00
ddotthomas
ca96a76958 change: renamed get_safesearch_level variables 2024-01-11 11:58:09 -07:00
alamin655
326131aac4
Merge branch 'rolling' into FIX/463_results_from_different_search_engines_get_cached_as_the_same_key 2024-01-11 16:47:15 +05:30
Evan Yang
efa8efc6c7
Compression and encryption for the cached search results (#443)
* attempt1

* rough draft

* add features and their optional dependancies

* add encryption and compression error variants

* add  a sample implementation to cache trait

* Update src/cache/cacher.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* adjust comment so feature flag would apply?

* adjust feature flag so it applies?

* formatting

* Update src/cache/cacher.rs

update documentation

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* [features]Add base64 and chacha20 dependencies for compress-cache-results and encrypt-cache-results

* move encryption key and cipher logic to separate sub module

* added cacha20 and cec-results feature

* added cacha20 and cec-results feature

* added compression and encryption helper functions to trait implementations

* added compression and encryption  implementation for inMemoryCache

* base64 is only requried when redis-cache feature is enabled

* add error case for base64 and encryption/compression implementation to redisCache

* Refactor cacher to remove regex dependency

* fmt cache error and cacher

* Update Cargo.toml

disabling the unneeded default-features

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* fix unused import warning for mimalloc

* remove deprecated method

* add doc comments for encryption module

* fix known bugs and use cfg-if module

* make cfg-if an optional dependency

* use feature-flag instead of maco lint

* add comment to explain type complexity

* bump app version

* Update src/cache/encryption.rs

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>

* fixed type complexity and add docs for types

---------

Co-authored-by: Spencer Najib <spencernajib2@gmail.com>
Co-authored-by: alamin655 <mdalamin655@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
Co-authored-by: Spencerjibz <=spencernajib2@gmail.com>
Co-authored-by: spencer <spencer@DESKTOP-SIF13AR>
2024-01-11 11:10:35 +00:00
ddotthomas
80e950de3b refactor: changed Cookie to use Cow to facilitate using references when building 2024-01-10 14:14:14 -07:00
neon_arch
a47e28587c
Merge branch 'rolling' into FIX/463_results_from_different_search_engines_get_cached_as_the_same_key 2024-01-10 07:25:30 +03:00
ddotthomas
d912bff94e change: revert Cookie Strings back to &str 2024-01-08 11:46:21 -07:00
ddotthomas
6e9250c03a fix: safe search url parameter ignored 2024-01-08 09:19:36 -07:00
ddotthomas
7d762b3726 refactor: moved settings parsing out of results 2024-01-07 16:29:39 -07:00
Леонтий Вартанян
33846cee34
💄 style: replace the search button with a magnifying glass (#483)
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2024-01-06 21:17:11 +03:00
ddotthomas
dde117e7e6 modify: changed cache_key format data 2024-01-04 11:05:09 -07:00
ddotthomas
86b0d3d6c9 Moved parsing of cookie_value, config to cache_key 2024-01-02 16:26:04 -07:00
neon_arch
36e2ac93be
Merge branch 'rolling' into FIX/463_results_from_different_search_engines_get_cached_as_the_same_key 2024-01-02 12:29:54 +03:00
ddotthomas
db1115f19e fix: add comma separator and format macro 2024-01-01 19:36:18 -07:00
Jann Marc Villablanca
50aa52c485
Bing for the search engine (#473) (#473)
fix: add closing curly brace

- accidentally removed from previous PR. i hope i did not ruin some
  build

* feat: implement mojeek engine

* Merge branch 'rolling' into FEAT_316_bing_search_engine

*  feat: implement bing engine

* Fix: include cookie string to header_map

* Fix: include <strong> tags from upstream search

* Merge branch 'rolling' into FEAT_316_bing_search_engine

Co-authored-by: neon-mmd <132049916+neon-mmd@users.noreply.github.com>
2024-01-01 14:57:31 +03:00
Леонтий Вартанян
5020f36c90
♻️ refactor: standardize the content-type header by using an enum value over typing it manually (#474)
* ♻️  refactor: change content-type

* 🐛  fix: change parameters that were passed to the settings function

---------

Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2023-12-31 13:31:07 +03:00
Siddharth Tiwari
5b4864424a
perf: Compression for the page responses of the search engine (#472) 2023-12-31 12:57:50 +03:00