summaryrefslogtreecommitdiff
path: root/searx/network
AgeCommit message (Collapse)Author
2024-11-23[log] warning with URL in case of 'raise_for_httperror'Markus Heiser
In order to be able to implement error handling, it is necessary to know which URL triggered the exception / the URL has not yet been logged. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-07-13[mod] remove py 3.6 leftoversAllen
2024-03-11[mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTIONMarkus Heiser
In the past, some files were tested with the standard profile, others with a profile in which most of the messages were switched off ... some files were not checked at all. - ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished - the distinction ``# lint: pylint`` is no longer necessary - the pylint tasks have been reduced from three to two 1. ./searx/engines -> lint engines with additional builtins 2. ./searx ./searxng_extra ./tests -> lint all other python files Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-09[fix] issues reported by pylint 3.1.0Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18[fix] spellingjazzzooo
2023-08-27[mod] searx.network: memory optimizationAlexandre Flament
Avoid to create a SSLContext in AsyncHTTPTransportNoHttp See: * https://github.com/encode/httpx/blob/0f61aa58d66680c239ce43c8cdd453e7dc532bfc/httpx/_transports/default.py#L271 * https://github.com/encode/httpx/issues/2298
2023-08-21Bump httpx 0.21.2 from to 0.24.1Alexandre Flament
2023-03-19[mod] Shuffle httpx's default ciphers of a SSL context randomly.Markus Heiser
From the analyse of @9Ninety [1] we know that DDG (and may be other engines / I have startpage in mind) does some kind of TLS fingerprint to block bots. This patch shuffles the default ciphers from httpx to avoid a cipher profile that is known to httpx (and blocked by DDG). [1] https://github.com/searxng/searxng/issues/2246#issuecomment-1467895556 ---- From `What Is TLS Fingerprint and How to Bypass It`_ > When implementing TLS fingerprinting, servers can't operate based on a > locked-in whitelist database of fingerprints. New fingerprints appear > when web clients or TLS libraries release new versions. So, they have to > live off a blocklist database instead. > ... > It's safe to leave the first three as is but shuffle the remaining ciphers > and you can bypass the TLS fingerprint check. .. _What Is TLS Fingerprint and How to Bypass It: https://www.zenrows.com/blog/what-is-tls-fingerprint#how-to-bypass-tls-fingerprinting Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> Closes: https://github.com/searxng/searxng/issues/2246
2023-01-28search.suspended_time settings: bug fixesAlexandre Flament
* fix type in settings.yml: replace suspend_times by suspended_times * always use delay defined in settings.yml: * HTTP status 402 and 403: read the value from settings.yml instead of using the hardcoded value of 1 day. * startpage engine: CAPTCHA suspend the engine for one day instead of one week
2023-01-15Add search.suspended_times settingsLéon Tiekötter
Make suspended_time changeable in settings.yml Allow different values to be set for different exceptions. Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-11-06Update network.pyEvhorizon
2022-10-14searx.network: add "verify" option to the networksAlexandre Flament
Each network can define a verify option: * false to disable certificate verification * a path to existing certificate. SearXNG uses SSL_CERT_FILE and SSL_CERT_DIR when they are defined see https://www.python-httpx.org/environment_variables/#ssl_cert_file
2022-07-08bing.py: resolve bing.com/ck/a redirectionsAlexandre Flament
add a new function searx.network.multi_requests to send multiple HTTP requests at once
2022-06-03[fix] prepare for pylint 2.14.0Markus Heiser
Remove issue reported by Pylint 2.14.0: - no-self-use: has been moved to optional extension [1] - The refactoring checker now also raises 'consider-using-generator' messages for max(), min() and sum(). [2] .pylintrc: - <option name>-hint has been removed since long, Pylint 2.14.0 raises an error on invalid options - bad-continuation and bad-whitespace have been removed [3] [1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0 [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-29[mod] remove deprecate codeAlexandre Flament
remove code to support Python 3.5 and Python 3.6
2022-01-27Merge pull request #817 from not-my-profile/pyright-01Alexandre Flament
Pyright 01
2022-01-27Allow 'using_tor_proxy' to be set for each engine individuallyLéon Tiekötter
Check 'using_tor_proxy' for each engine individually instead of checking globally [fix] searx.network: update _rdns test to the last httpx version Co-authored-by: Alexandre Flament <alex@al-f.net>
2022-01-27[pyright:basic] searx.webappMartin Fischer
2022-01-17[typing] add type hints for dictionariesMartin Fischer
2022-01-05[mod] upgrade httpx 0.21.2Alexandre Flament
httpx 0.21.2 and httpcore 0.14.4 fix multiple issues: * https://github.com/encode/httpx/releases/tag/0.21.2 * https://github.com/encode/httpcore/releases/tag/0.14.4 so most of the workarounds in searx.network have been removed.
2021-12-27[format.python] initial formatting of the python codeMarkus Heiser
This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-12[enh] verify that Tor proxy works every time searx startsAlexandre Flament
based on @MarcAbonce commit on searx
2021-09-28[mod] searx.network.stream returns a tuple (response, stream)Alexandre Flament
2021-09-28[fix] searx.network.stream: fix memory leakAlexandre Flament
2021-09-17Merge pull request #261 from dalf/upgrade_httpxAlexandre Flament
[upd] upgrade httpx 0.19.0
2021-09-17[pylint] fix global-variable-not-assigned issuesMarkus Heiser
If there is no write access, there is no need for global. Remove global statement if there is no assignment. global-variable-not-assigned: Using global for names but no assignment is done Used when a variable is defined through the "global" statement but no assignment to this variable is done. In Pylint 2.11 the global-variable-not-assigned checker now catches global variables that are never reassigned in a local scope and catches (reassigned) functions [1][2] [1] https://pylint.pycqa.org/en/latest/whatsnew/2.11.html [2] https://github.com/PyCQA/pylint/issues/1375 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17[mod] searx.network: remove redundant codeAlexandre Flament
searx.client.new_client: the proxies parameter is a dictonnary, and the protocol (key of the dictionnary) is already normalized (see usage of searx.network.network.PROXY_PATTERN_MAPPING)
2021-09-17[upd] upgrade httpx 0.19.0Alexandre Flament
adjust searx.network module to the new internal API see https://github.com/encode/httpx/pull/1522
2021-09-14Merge pull request #297 from dalf/engine-logger-enhAlexandre Flament
debug mode: more readable logging
2021-09-11[fix] PR #257: use the image_proxy network instead of the default networkAlexandre Flament
2021-09-11[mod] debug mode: log HTTP requests with network nameAlexandre Flament
For example wikipedia requests use the logger name "searx.network.wikipedia" Log is disable when searx_debug is False
2021-09-07[pylint] searx: drop no longer needed 'missing-function-docstring'Markus Heiser
Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914168470 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-31[pylint] Pylint 2.10 - unused-variableMarkus Heiser
Pylint 2.10 fixed [1]: Fixed bug with cell-var-from-loop checker: it no longer has false negatives when both unused-variable and used-before-assignment are disabled. [1] https://pylint.pycqa.org/en/latest/whatsnew/2.10.html Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-24[mod] /image_proxy: use HTTP/1 instead of HTTP/2Alexandre Flament
httpx: HTTP/2 is slow when a lot data is downloaded. https://github.com/dalf/pyhttp-benchmark also, the usage of HTTP/1 decreases the load average
2021-08-24[fix] image_proxy: always close the httpx responeAlexandre Flament
previously, when the content type was not an image and some other error, the httpx response was not closed
2021-08-24[mod] /image_proxy: don't decompress imagesAlexandre Flament
2021-06-01[mod] move all default settings into searx.settings_defaultsAlexandre Flament
2021-05-24[coding-style] searx/network/network.py - normalized indentationsMarkus Heiser
No functional change! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[coding-style] searx/network/client.py - normalized indentationsMarkus Heiser
No functional change! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[pylint] searx/network/raise_for_httperror.pyMarkus Heiser
No functional change! - fix messages from pylint - add ``global NETWORKS`` - normalized indentations Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[pylint] searx/network/network.py & add global (NETWORKS)Markus Heiser
No functional change! - fix messages from pylint - add ``global NETWORKS`` Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[pylint] searx/network/client.py & add global (TRANSPORT_KWARGS)Markus Heiser
No functional change! - fix messages from pylint - add ``global TRANSPORT_KWARGS`` - normalized python_socks imports Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-24[pylint] searx/network/__init__.py & add global (THREADLOCAL)Markus Heiser
No functional change! - fix messages from pylint - add ``global THREADLOCAL`` - normalized various indentation Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-22[fix] offline engine: don't crash on time recordingAlexandre Flament
2021-05-05[mod] searx.network.client: the same configuration reuses the same ↵Alexandre Flament
ssl.SSLContext before there was one ssl.SSLContext per client. see https://github.com/encode/httpx/issues/978
2021-04-27[fix] searx.network: fix rare cases where LOOP is NoneAlexandre Flament
* searx.network.client.LOOP is initialized in a thread * searx.network.__init__ imports LOOP which may happen before the thread has initialized LOOP This commit adds a new function "searx.network.client.get_loop()" to fix this issue
2021-04-21[enh] rewrite and enhance metricsAlexandre Flament
2021-04-21[mod] refactoring: processorsAlexandre Flament
Report to the user suspended engines. searx.search.processor.abstract: * manages suspend time (per network). * reports suspended time to the ResultContainer (method extend_container_if_suspended) * adds the results to the ResultContainer (method extend_container) * handles exceptions (method handle_exception)
2021-04-12[httpx] replace searx.poolrequests by searx.networkAlexandre Flament
settings.yml: * outgoing.networks: * can contains network definition * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections, keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time) * local_addresses can be "192.168.0.1/24" (it supports IPv6) * support_ipv4 & support_ipv6: both True by default see https://github.com/searx/searx/pull/1034 * each engine can define a "network" section: * either a full network description * either reference an existing network * all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion