neon_arch
03384d4e04
🛠️ fix: implement hybrid caching and improve documentation ( #244 )
2023-09-17 12:47:02 +03:00
Zsombor Gegesy
e69126c5ea
Fix git rebase problems, and the failing doctest
2023-09-15 23:25:17 +02:00
Zsombor Gegesy
1e7805cf42
Rename features, make the memory-cache the default
2023-09-15 23:21:19 +02:00
Zsombor Gegesy
76795c43cc
Make the cache types compile time feature flags too, and make it more configurable!
2023-09-15 23:21:19 +02:00
Zsombor Gegesy
320f5f4720
Remove unnecessary clones
2023-09-15 23:21:19 +02:00
Zsombor Gegesy
519ebe0fd8
Instead of caching jsons, we can cache the original structure
2023-09-15 23:21:19 +02:00
Zsombor Gegesy
996ff84c5b
Cache refactor - add an in-memory cache, so redis is not needed
2023-09-15 23:21:19 +02:00
neon_arch
ca4447fcdf
Merge branch 'rolling' into reorganize-code-and-restructure-the-codebase
2023-09-13 18:31:37 +03:00
neon_arch
c60fdb8366
🧹 chore: make github actions happy ( #205 )
2023-09-12 18:53:32 +03:00
neon_arch
1a222217c4
🧹 chore: make github actions happy ( #205 )
2023-09-12 17:59:33 +03:00
neon_arch
fb231de416
Merge branch 'rolling' into change-document-style-with-linter-warnings
2023-09-12 17:49:46 +03:00
neon_arch
b9d651c378
🧹 chore: make github actions happy ( #203 )
2023-09-11 20:16:42 +03:00
alamin655
8e56ac8ba2
Update parser.rs
2023-09-11 12:05:13 +05:30
alamin655
0132a63e6f
Apply suggestions from code review
2023-09-11 11:58:31 +05:30
alamin655
3c1aaf63f8
Update src/config/parser.rs
2023-09-11 11:56:12 +05:30
alamin655
531611f077
Update src/config/parser.rs
2023-09-11 11:54:21 +05:30
alamin655
70a5a2463b
Update src/config/parser.rs
2023-09-11 11:46:56 +05:30
alamin655
2790eefba8
Merge branch 'rolling' into feat-rate-limiter-for-websurfx
2023-09-11 11:40:10 +05:30
neon_arch
12bfc5276a
🧹 chore: make clippy happy ( #201 )
2023-09-10 18:56:54 +03:00
neon_arch
ac4e29a93f
🧹 chore: make github actions happy ( #201 )
2023-09-10 18:41:24 +03:00
alamin655
86991a2f9a
Merge branch 'rolling' into feat-rate-limiter-for-websurfx
2023-09-10 20:56:21 +05:30
alamin655
60b2fcc27a
Merge branch 'rolling' into feat-disallow-user-to-search-via-lists
2023-09-10 20:51:10 +05:30
neon_arch
82a78e3b42
Merge branch 'rolling' into optimize-and-make-code-idiomatic
2023-09-10 18:11:12 +03:00
neon_arch
8dfe2e1b2d
Merge branch 'rolling' into optimize-and-make-code-idiomatic-3
2023-09-10 17:58:28 +03:00
neon_arch
db009454c8
⚙️ refactor: change import path for search route ( #207 )
2023-09-03 21:03:58 +03:00
neon_arch
493c56bd02
⚙️ refactor: reorganize code & restructure codebase for better maintainability ( #207 )
2023-09-03 20:50:50 +03:00
neon_arch
453dbdc47d
🧹 chore: make rustfmt happy ( #205 )
2023-09-03 19:34:22 +03:00
neon_arch
049b1c1ddd
⚙️ refactor: change & add documentation to the code based on the lints ( #205 )
2023-09-03 19:23:34 +03:00
neon_arch
0d2d449889
⚙️ refactor: add lints to the codebase to ensure proper code style ( #205 )
2023-09-03 19:21:23 +03:00
neon_arch
51937a0d49
✨ feat: pass the new config option into the middleware config ( #203 )
2023-09-02 20:22:24 +03:00
neon_arch
8cba040d80
✨ feat: add code to parse the new config option ( #203 )
2023-09-02 20:21:48 +03:00
neon_arch
4eb75a8cb1
✨ feat: add code to parse the new config option ( #203 )
2023-09-02 20:19:43 +03:00
neon_arch
3f73e85455
🧹 chore: make rustfmt happy ( #201 )
2023-09-02 19:07:23 +03:00
neon_arch
05c3e6cc12
✨ feat: implement new search parameter safe_search & code to disallow
...
users when safe_search is set to 4 (#201 )
2023-09-02 17:50:06 +03:00
neon_arch
410257c276
✨ feat: implement new fields, traits and functions ( #201 )
2023-09-02 17:48:27 +03:00
neon_arch
f4c496c02c
✨ feat: implement default trait for style struct ( #201 )
2023-09-02 17:46:32 +03:00
neon_arch
cbad560139
✨ feat: pass new safe search option in the results function in the
...
trait (#201 )
2023-09-02 17:45:17 +03:00
neon_arch
6fa45ecb12
✨ feat: pass the config option into the results function ( #201 )
2023-09-02 17:44:05 +03:00
neon_arch
37e650eb8a
✨ feat: add code to parse the new config option ( #201 )
2023-09-02 17:42:55 +03:00
neon_arch
d6463f0872
✨ feat: add condition to filter results only when safe_search level is
...
set to 3 or above (#201 )
2023-09-02 17:38:46 +03:00
neon_arch
a799af3125
🧹 chore: add fixes to code & make rustfmt & clippy happy ( #180 )( #178 )
2023-08-30 18:09:58 +03:00
neon_arch
4c298ce18c
⚙️ refactor: add code to initialize redis cache struct only once ( #180 )( #178 )
2023-08-29 20:15:06 +03:00
neon_arch
7b1f93b232
⚙️ refactor: add compress middleware to reduce memory usage ( #180 )( #178 )
2023-08-29 20:12:15 +03:00
neon_arch
1de52decd3
⚙️ refactor: add clone trait to RedisCache struct ( #180 )( #178 )
2023-08-29 20:10:32 +03:00
neon_arch
028463bb36
⚙️ refactor: replace jemalloc with mimalloc ( #178 )( #180 )
2023-08-29 20:07:28 +03:00
neon_arch
fceacdb916
🧹 chore: make cargo format happy ( #180 )( #178 )
2023-08-27 21:17:21 +03:00
neon_arch
fd6cb46f87
⚙️ refactor: replace string type to &str type ( #180 )( #178 )
2023-08-27 21:07:57 +03:00
neon_arch
b2c72bdfc3
⚙️ refactor: replace pass by value with pass by reference ( #180 )( #178 )
2023-08-27 21:06:25 +03:00
neon_arch
13ce420642
⚙️ refactor: add several minor optimizations ( #180 )( #178 )
2023-08-27 21:04:41 +03:00
neon_arch
2885f23ec9
⚙️ refactor: replace vecs with smallvecs for smaller data sizes & replace to_strings with to_owned ( #180 )( #178 )
2023-08-27 21:02:23 +03:00
neon_arch
2a68081ae2
⚙️ refactor: remove deprecated intoreport functions & add minor optimizations ( #180 )( #178 )
2023-08-27 21:00:22 +03:00
neon_arch
f5f0488954
⚙️ refactor: add several optimizations to the engine code ( #180 )( #178 )
2023-08-27 20:59:08 +03:00
neon_arch
4ccd0486e7
⚙️ refactor: replace oncecell with oncelock from std library ( #180 )( #178 )
2023-08-27 20:57:33 +03:00
neon_arch
7b33744c9d
⚙️ refactor: implement default trait for Style struct ( #180 )( #178 )
2023-08-27 20:56:29 +03:00
neon_arch
e4476aae28
⚙️ refactor: replace rlua with mlua code implementation ( #180 )( #178 )
2023-08-27 20:55:34 +03:00
neon_arch
5f1a43976f
⚙️ refactor: add error.rs module ( #180 )( #178 )
2023-08-27 20:53:55 +03:00
neon_arch
01d8c7ae4c
⚙️ refactor: add new pooling error type for pooling code ( #180 )( #178 )
2023-08-27 20:52:16 +03:00
neon_arch
db93c31603
⚙️ refactor: implement async pooling for redis connections ( #180 )( #178 )
2023-08-27 20:50:42 +03:00
neon_arch
4afc0d16f0
⚙️ refactor: add dhat profiler configuration with feat flag ( #180 )( #178 )
2023-08-27 20:46:19 +03:00
xffxff
c3a7c917f6
make format happy
2023-08-24 09:50:19 +08:00
xffxff
23ff24bdf3
add a test to check if the regex wildcard .* matches any character
2023-08-24 09:46:01 +08:00
xffxff
a2fc10ca39
add a test for invalid regex
2023-08-24 09:36:08 +08:00
xffxff
4280545e8c
add a test for non-existent file
2023-08-24 09:32:22 +08:00
xffxff
5c0397c456
add some comments to filter_with_lists and add a basic test
2023-08-24 09:29:08 +08:00
zhou fan
2f1fa00f87
Merge branch 'rolling' into feat-error-box-for-engine-errors
2023-08-24 08:16:32 +08:00
zhou fan
575a7f95ba
Merge branch 'rolling' into feat-error-box-for-engine-errors
2023-08-24 08:02:03 +08:00
neon_arch
44216e4d4c
✨ feat: optimise search results filtering code ( #163 )
2023-08-23 13:11:09 +03:00
neon_arch
9c71c9f617
🧹 chore: make rustfmt happy
2023-08-22 19:29:13 +03:00
neon_arch
b17057471b
✨ feat: rename functions ( #163 )
2023-08-22 19:22:37 +03:00
neon_arch
09227d8c93
✨ feat: move implementation of config file path to reduce duplication ( #163 )
2023-08-22 19:20:29 +03:00
neon_arch
fda6c3a9be
✨ feat: rename public_paths to paths ( #163 )
2023-08-22 19:18:43 +03:00
neon_arch
4f28711218
✨ feat: add code to filter aggregated search results using lists ( #163 )
2023-08-22 19:16:37 +03:00
Milim
87e230d731
fix debug level not being set properly
2023-08-19 09:09:09 +02:00
Milim
21403d0a48
fix dumb rustfmt version and bump dependency versions
2023-08-18 22:04:34 +02:00
Milim
5aca5c0d0d
Improve aggregation
...
Adds the EngineHandler struct
Removes vulnerability where an attacker could send requests cookies with fake engine names and crash the server.
Merged RawSearchResult and SearchResult, as they were functionally identical.
2023-08-18 10:43:53 +02:00
Milim
15dfda6ea9
Improve Aggregation function & config parser
...
Refactor aggregation function
Rename visiting_url to url, as they are always the same (see upstream engine scalping).
Refactor parsing function to be more readable.
2023-08-17 22:48:20 +02:00
Milim
af3b1cb308
Use non-blocking sleep function
...
std:🧵 :sleep blocks the entire thread and thus also forces all other current requests to be halted.
2023-08-17 21:45:48 +02:00
neon_arch
c4935f202a
✨ feat: add images, error_box & new message when no results are provided ( #185 )
2023-08-10 04:32:47 +03:00
neon_arch
272fdefff5
Merge branch 'rolling' into improve-results-caching
2023-08-06 20:53:14 +03:00
neon_arch
a596c07547
🔧 chore: add code to cache the previous & next pages ( #160 )
2023-08-06 20:31:30 +03:00
neon_arch
ca13fb7ad4
Merge branch 'improve-logging-based-on-levels-and-opts' of github.com:neon-mmd/websurfx into improve-logging-based-on-levels-and-opts
2023-08-05 19:20:58 +03:00
neon_arch
653d08c801
🔧 chore: fix link in the logs & add logs for engine error ( #170 )
2023-08-05 19:16:27 +03:00
neon_arch
0d8c12e26b
Merge branch 'rolling' into improve-logging-based-on-levels-and-opts
2023-08-04 12:55:05 +03:00
neon_arch
9d3a8e065a
Merge branch 'rolling' into patch-csrf-security-with-cors
2023-08-04 12:35:26 +03:00
neon_arch
8b0f2f50f0
🔧 chore: improve logging based on levels & config options ( #173 )
2023-08-03 23:09:08 +03:00
neon_arch
bef8956da6
🛠️ fix: add code to prevent csrf attacks using cors ( #172 )
2023-08-03 17:44:13 +03:00
neon_arch
e02fafd69c
✨ feat: fix doc tests
2023-08-03 01:03:56 +03:00
neon_arch
4b4dc28cd2
✨ feat: add control check to avoid reinitializing logging & fix tests
2023-08-03 00:12:09 +03:00
neon_arch
11bcf9c98a
✨ feat: remove space from the end of the line
2023-08-02 20:07:29 +03:00
neon_arch
d8bd2feb6e
✨ feat: add new config option to manage threads and improve logging
2023-08-02 20:05:39 +03:00
neon_arch
a8b6a9bce2
Merge branch 'rolling' into patch-searx-results-on-first-page
2023-07-31 13:57:34 +03:00
neon_arch
a8791deb13
🛠️ fix: add & improve code to handle page handling in searx & duckduckgo
2023-07-30 20:14:40 +03:00
neon_arch
9cb582a10d
✨ feat: improve code documentation
2023-07-30 17:08:47 +03:00
neon_arch
25d6af0251
🧹 chore: reformat code to make rustfmt happy
2023-07-30 16:58:19 +03:00
neon_arch
2d47e8d730
✨ feat: provide the functionality to use the new config option
2023-07-30 10:53:48 +03:00
neon_arch
6d3396b6f9
✨ feat: provide functionality to parse the new config option
2023-07-30 10:52:03 +03:00
neon_arch
a28d559426
✨ feat: add documentation for the outputs.iter() code block
2023-07-17 17:03:50 +03:00
neon_arch
987e667c3d
✨ feat: rename results_json to results
2023-07-17 13:17:24 +03:00
neon_arch
7e3b59e8c0
✨ feat: remove the empty_result_set option and helper functions
2023-07-17 13:03:20 +03:00
neon_arch
c5fca32553
✨ feat: fix spelling mistake in the comment
2023-07-17 10:50:15 +03:00
neon_arch
cff7de9d0f
✨ feat: rephrase the comment to break it into smaller sentences.
2023-07-17 10:47:29 +03:00
neon_arch
74e4fc6169
Merge branch 'rolling' into improve-async-multithreading
2023-07-15 19:50:31 +03:00
neon_arch
94ef62eec9
✨ feat: add documentation to code
2023-07-15 13:36:46 +03:00
neon_arch
8133de1758
✨ feat: add check to conditionaly pass empty_result_set to true to UI
2023-07-14 21:29:01 +03:00
neon_arch
a3edf70d1e
✨ feat: add code to pass engine errors to the UI
2023-07-14 21:27:23 +03:00
neon_arch
3aeb3b3e73
✨ feat: add setter functions to SearchResults struct
2023-07-14 21:26:29 +03:00
neon_arch
9ee516e9b1
✨ feat: fix code for clippy suggestions
2023-07-14 21:18:26 +03:00
neon_arch
3e6f7867eb
✨ feat: add code to parse cookies and handle engine selection from UI
2023-07-14 17:20:46 +03:00
neon_arch
27247baf0c
✨ feat: define common behaviour for all engines
2023-07-14 17:17:05 +03:00
neon_arch
c223fed394
✨ feat: remove the time crate import
2023-07-14 17:16:13 +03:00
neon_arch
2f01651be0
✨ feat: fix bugs and add code to handle engine selections
...
Co-authored-by: Sabrina Jewson <58880148+SabrinaJewson@users.noreply.github.com>
2023-07-14 12:56:06 +03:00
neon_arch
0781385393
✨ feat: implement async multithreading and engine selection code
2023-07-11 19:44:38 +03:00
neon_arch
897ab0807f
✨ feat: fix the argument type used
2023-07-11 19:43:32 +03:00
neon_arch
f9b9e87a0e
✨ feat: rewrite code by implementing common engine trait SearchEngine
2023-07-11 19:42:17 +03:00
neon_arch
b72af01e0e
✨ feat: implement common engine trait
2023-07-11 19:41:34 +03:00
neon_arch
ff79c1fcfb
✨ feat: code to parse and handle the new config option
2023-07-11 19:38:59 +03:00
Trickshotblaster
99936bcb1a
Rename getter functions
2023-07-04 15:11:30 -07:00
Milim
4cb38b1b54
Fix docs
2023-07-03 19:38:17 +02:00
Milim
440216871d
Rename Things, refactor some code
...
BREAKING: renames `binding_ip_addr` to `binding_ip` and `redis_connection_url` to `redis_url`.
Renames a lot of internals as well, but they are to many to mention.
2023-07-03 19:30:25 +02:00
Milim
dbe5b533d6
fix spelling
2023-06-29 19:40:49 +02:00
Milim
e2907edf6e
refactor config helper function
2023-06-29 19:10:09 +02:00
Milim
1f90b4e15c
refactor search route and change default page to 0
...
that thing was utterly insane, and i am not sorry for saying this
2023-06-29 18:49:38 +02:00
xffxff
ebb9e9ee2b
make format happy
2023-06-15 06:48:37 +08:00
zhou fan
b6faa09750
Update src/engines/engine_models.rs
...
Co-authored-by: neon_arch <mustafadhuleb53@gmail.com>
2023-06-15 06:28:21 +08:00
xffxff
1ebf888085
log the error from engines
2023-06-15 06:27:45 +08:00
xffxff
9a4cf949e4
improve error handling by using error-stack crate
2023-06-14 21:23:00 +08:00
neon_arch
c2280b7349
chore: add source and an appropriate message to different error variants
2023-06-04 11:56:07 +03:00
neon_arch
4460730ca6
chore: make clippy github action happy
2023-06-01 12:21:45 +03:00
neon_arch
975e8a480b
Merge branch 'rolling' into handle-reqwest-errors
2023-05-31 17:20:08 +00:00
neon_arch
5962cca294
chore: provide a better and more standardized way to handle engine errors
2023-05-31 19:54:51 +03:00
neon_arch
13632f1f99
feat: remove random delays when debug is set to true
2023-05-29 21:28:09 +03:00
neon_arch
60317a3b75
chore: make format ci happy
2023-05-29 21:13:07 +03:00
neon_arch
d4df90160d
feat: add an option to enable/disable debug mode
2023-05-29 21:09:07 +03:00
neon_arch
8e7dc68d2d
feat: add an option to enable or disable logs
2023-05-27 19:50:20 +03:00
neon_arch
dc3308cb5f
chore: rename from theme to public
2023-05-25 11:50:37 +03:00
neon_arch
29456650f1
feat: add ability to put the themes folder on different paths
2023-05-24 17:37:41 +03:00
zhou fan
e6bb981133
Merge branch 'rolling' into ability-to-have-config-on-diff-paths
2023-05-24 17:10:07 +08:00
neon_arch
a7a28ed8c6
chore: make websurfx directory and config file names as constants
2023-05-24 12:01:36 +03:00
neon_arch
89796a054f
feat: add ability to put config file on different paths
2023-05-23 23:47:36 +03:00
XFFXFF
ea013e718e
make format happy
2023-05-23 10:00:35 +00:00
XFFXFF
9773cee38d
change the 'aggregator' option to 'production_use'
2023-05-23 10:00:05 +00:00
XFFXFF
05272884ba
supports the option to add a random delay
2023-05-23 10:00:03 +00:00
XFFXFF
cecffe4155
make format happy
2023-05-23 09:34:46 +00:00
neon_arch
6e19246cb2
fix: reduce the time to build user-agents
2023-05-22 13:07:37 +03:00
MD AL AMIN TALUKDAR
ae2fdb557c
Removed unnecessary comment
2023-05-20 08:01:24 +05:30
MD AL AMIN TALUKDAR
9a204b2f98
Fix page_url assignment in search route
...
The page_url variable in the search route was not being properly assigned in certain cases. This commit fixes the issue by ensuring that page_url is assigned the correct value based on the search parameters.
In the match expression, the conditions have been adjusted to correctly handle the page number and construct the appropriate page_url. This ensures that the generated URL for the search page is accurate and reflects the search query and page number.
This change improves the functionality and reliability of the search route by correctly setting the page_url variable based on the provided search parameters.
2023-05-19 17:13:11 +05:30
neon_arch
019b3322e7
chore: add enum to handle different reqwest errors & add timeout to requests
2023-05-16 12:22:00 +03:00
XFFXFF
3c7edb80d0
refactor: reduce connections created with RedisCache
2023-05-15 00:20:43 +00:00
neon_arch
4364310b24
add support for dockerized container hosting
2023-05-09 18:53:12 +03:00
neon_arch
0502a8f551
improve code to evade bot detection and closes #8
2023-05-08 11:47:50 +03:00
