summaryrefslogtreecommitdiff
path: root/searx/utils.py
AgeCommit message (Collapse)Author
2022-07-30[fix] pyright repported errorsAlexandre Flament
The errors make pyright usage useless since a new error won't be seen [1]. [1] https://github.com/searxng/searxng/pull/1569 ``` searx/compat.py:11:27 - error: Expression of type "Type[cached_property[_T@cached_property]]" cannot be assigned to declared type "Type[cached_property]" "Type[cached_property[_T@cached_property]]" is incompatible with "Type[cached_property]" Type "Type[cached_property[_T@cached_property]]" cannot be assigned to type "Type[cached_property]" (reportGeneralTypeIssues) searx/utils.py:69:36 - error: Expression of type "None" cannot be assigned to parameter of type "str" Type "None" cannot be assigned to type "str" (reportGeneralTypeIssues) searx/utils.py:573:85 - error: Expression of type "None" cannot be assigned to parameter of type "int" Type "None" cannot be assigned to type "int" (reportGeneralTypeIssues) searx/webapp.py:1306:22 - error: Argument of type "str" cannot be assigned to parameter "__a" of type "BytesPath" in function "join" Type "str" cannot be assigned to type "BytesPath" "str" is incompatible with "bytes" "str" is incompatible with protocol "PathLike[bytes]" "__fspath__" is not present (reportGeneralTypeIssues) searx/webapp.py:1306:68 - error: Argument of type "Literal['themes']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join" Type "Literal['themes']" cannot be assigned to type "BytesPath" "Literal['themes']" is incompatible with "bytes" "Literal['themes']" is incompatible with protocol "PathLike[bytes]" "__fspath__" is not present (reportGeneralTypeIssues) searx/webapp.py:1306:78 - error: Argument of type "str | Any | None" cannot be assigned to parameter "paths" of type "BytesPath" in function "join" Type "str | Any | None" cannot be assigned to type "BytesPath" Type "str" cannot be assigned to type "BytesPath" "str" is incompatible with "bytes" "str" is incompatible with protocol "PathLike[bytes]" "__fspath__" is not present (reportGeneralTypeIssues) searx/webapp.py:1306:85 - error: Argument of type "Literal['img']" cannot be assigned to parameter "paths" of type "BytesPath" in function "join" Type "Literal['img']" cannot be assigned to type "BytesPath" "Literal['img']" is incompatible with "bytes" "Literal['img']" is incompatible with protocol "PathLike[bytes]" "__fspath__" is not present (reportGeneralTypeIssues) searx/engines/mongodb.py:8:6 - warning: Import "pymongo" could not be resolved (reportMissingImports) searx/engines/mysql_server.py:9:8 - warning: Import "mysql.connector" could not be resolved (reportMissingImports) searx/engines/postgresql.py:9:8 - warning: Import "psycopg2" could not be resolved from source (reportMissingModuleSource) searx/engines/xpath.py:187:28 - warning: "categories" is not defined (reportUndefinedVariable) searx/search/__init__.py:184:82 - warning: "flask" is not defined (reportUndefinedVariable) searx/search/checker/background.py:19:26 - error: Type of "schedule" is partially unknown Type of "schedule" is "(delay: Any, func: Any, *args: Any) -> Literal[True]" (reportUnknownVariableType) searx/shared/__init__.py:8:12 - warning: Import "uwsgi" could not be resolved (reportMissingImports) searx/shared/shared_uwsgi.py:5:8 - warning: Import "uwsgi" could not be resolved (reportMissingImports) ```
2022-06-03[fix] prepare for pylint 2.14.0Markus Heiser
Remove issue reported by Pylint 2.14.0: - no-self-use: has been moved to optional extension [1] - The refactoring checker now also raises 'consider-using-generator' messages for max(), min() and sum(). [2] .pylintrc: - <option name>-hint has been removed since long, Pylint 2.14.0 raises an error on invalid options - bad-continuation and bad-whitespace have been removed [3] [1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0 [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-22[test.pyright] suppress unneeded error & warning messagesMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-04-16searx.utils.html_to_text: replace <br/> by a spaceAlexandre Flament
2022-01-30[mod] searx.utils: more typingAlexandre Flament
2022-01-29[mod] add documentation about searx.utilsAlexandre Flament
This module is a toolbox for the engines. Is should be documented. In addition, searx/utils.py is checked by pylint.
2021-12-27[format.python] initial formatting of the python codeMarkus Heiser
This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-12[fix] fix match_language issue to make zh-TW match to zh-Hant-TWMarc Abonce Seguin
pybabel separates locales with underscores but we use hyphens everywhere babel doesn't directly touch
2021-10-06[fix] don't mix loaded modules with imported modules (sys.modules)Markus Heiser
The utils.load_module() function is used to load a python file (aka module) and return the module's namespace. SearXNG uses this function to load *engines and answerers* from arbitrary locations with arbitrary modifications. These are not real python modules and it is not intended to mix this *engines and answerers* with the python modules registered in sys.modules. Closes: https://github.com/searxng/searxng/issues/312 Suggested-by: @dalf in https://github.com/searxng/searxng/issues/312 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-24[mod] searx.utils.dict_subset: rewrite with comprehensionAlexandre Flament
2021-07-30version based on the git repositoryAlexandre Flament
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH: there are read from the git repository. It is possible to call python -m searx.version freeze to freeze the current version. Useful when the code is installed outside git (distro package, docker, etc...)
2021-06-09[fix] strip spaces from searx user agentAlexandre Flament
h11 (used by httpx) rejects HTTP request with a trailing space in HTTP headers
2021-06-01[mod] move all default settings into searx.settings_defaultsAlexandre Flament
2021-05-28[mod] utils.get_value() - avoidance of a recursionMarkus Heiser
In a comment [1] dalf suggested to avoid a recursion of get_value() [1] https://github.com/searxng/searxng/pull/99#discussion_r640833716 Suggested-by: Alexandre Flament <alex@al-f.net> Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28[enh] add settings option to enable/disable search formatsMarkus Heiser
Access to formats can be denied by settings configuration:: search: formats: [html, csv, json, rss] Closes: https://github.com/searxng/searxng/issues/95 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-10[enh] replace requests by httpxAlexandre Flament
2020-12-20[fix] pylint: use "raise ... from ..."Alexandre Flament
2020-12-03[mod] bing_news: use eval_xpath_getindexAlexandre Flament
remove unused function searx.utils.list_get
2020-12-03[enh] record details exception per engineAlexandre Flament
add an new API /stats/errors
2020-12-01[mod] pylint: numerous minor code fixesAlexandre Flament
2020-11-14[mod] remove unused importAlexandre Flament
use from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA so it is possible to easily remove all unused import using autoflake: autoflake --in-place --recursive --remove-all-unused-imports searx tests
2020-10-28[mod] duckduckgo_definitions: display only user friendly attributes / URLAlexandre Flament
various bug fixes
2020-10-07[mod] Add searx.data moduleAlexandre Flament
Instead of loading the data/*.json in different location, load these files in the new searx.data module.
2020-10-06[fix] drop Python 2: use importlib instead of imp.load_sourceAlexandre Flament
imp.load_source is not documented in Python 3 see documentation : https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly partial fix of https://github.com/searx/searx/issues/1674
2020-10-03[mod] searx.utils.normalize_url: remove Yahoo hackAlexandre Flament
* The hack for Yahoo URLs is not necessary anymore. (see searx.engines.yahoo.parse_url) * move the URL normalization in extract_url to normalize_url
2020-10-02[mod] searx/utils.py: add docstringAlexandre Flament
2020-10-02[mod] move extract_text, extract_url to searx.utilsAlexandre Flament
2020-09-22[mod] add searx/webutils.pyAlexandre Flament
contains utility functions and classes used only by webapp.py
2020-09-13[fix] searx.utils.HTMLTextExtractor: invalid HTML don't raise an ExceptionAlexandre Flament
Close #2188
2020-09-10Drop Python 2 (5/n): searx.utils.is_valid_lang, input parameter is a str ↵Alexandre Flament
instead of bytes Fix bug in translated.py and dictzone.py
2020-09-10Drop Python 2 (4/n): SearchQuery.query is a str instead of bytesDalf
2020-09-10Drop Python 2 (1/n): remove unicode string and url_utilsDalf
2019-11-15[mod] speed optimizationDalf
compile XPath only once avoid redundant call to urlparse get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
2019-10-16fix pep 8 checkNoémi Ványi
2019-10-16add initial support for offline engines && command engineNoémi Ványi
2019-10-16[fix] convert bytes type to string in language detection (fixes dictzone)Adam Tauber
2019-08-02[fix] fix flickr_noapi decoding (#1655)Alexandre Flament
Characters that were not ASCII were incorrectly decoded. Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
2019-07-19[mod] use cache in _match_language function to speed up searx start time ↵Dalf
significantly
2019-07-17[fix] secret_key can be bytes instead of a string (#1602)rachmadani haryono
Fix #1600 In settings.yml, the secret_key can be written as string or as base64 encoded data using !!binary notation.
2018-08-05fetch_firefox_version.py : compatible with Python 3 and minor fixes.Alex
2018-08-05[mod] fetch firefox versions in a standalone scriptAlexandre Flament
2018-07-09Merge pull request #1303 from MarcAbonce/bingAdam Tauber
Fix bing "garbage" results
2018-06-23[fix] update user agent versionsMarc Abonce Seguin
this fixes duckduckgo error response
2018-06-14[fix] add basestring for py3Adam Tauber
2018-05-20fix bing "garbage" results (issue #1275)Marc Abonce Seguin
2018-03-27refactor engine's search language handlingMarc Abonce Seguin
Add match_language function in utils to match any user given language code with a list of engine's supported languages. Also add language_aliases dict on each engine to translate standard language codes into the custom codes used by the engine.
2017-12-01[fix] convert json engine result attributes to string - closes #1006Adam Tauber
2017-11-21[fix] remove trailing 0x00 from csv outputAdam Tauber
2017-11-21[fix] resurrect csv output in py2Adam Tauber
2017-09-08fix hmac python3 compatibilityNoémi Ványi
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion