summaryrefslogtreecommitdiff
path: root/searx/engines/wikidata.py
AgeCommit message (Collapse)Author
2024-05-10[fix] engine wikidata - WIKIDATA_UNITS has been changed in #3378Markus Heiser
This patch is a leftover from [1] in which the WIKIDATA_UNITS values has become a dictionary. [1] https://github.com/searxng/searxng/pull/3378 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2024-03-11[mod] pylint all engines without PYLINT_SEARXNG_DISABLE_OPTIONMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19wikipedia wikidata infobox + disable wikisource (#2806)Émilien (perso)
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18[fix] spellingjazzzooo
2023-04-17[fix] doc of locales.get_engine_locale() / zh-classical is missleadingMarkus Heiser
Wikipedia's zh-classical is not zh_Hant (see doc-string of engines.wikipedia). Fixed the example in the doc-string of locales.get_engine_locale() to 'zh_TW'. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15[fix] searxng_extra/update/update_engine_descriptions.py (part 1)Markus Heiser
Follow up of #2269 The script to update the descriptions of the engines does no longer work since PR #2269 has been merged. searx/engines/wikipedia.py ========================== 1. There was a misusage of zh-classical.wikipedia.org: - `zh-classical` is dedicate to classical Chinese [1] which is not traditional Chinese [2]. - zh.wikipedia.org has LanguageConverter enabled [3] and is going to dynamically show simplified or traditional Chinese according to the HTTP Accept-Language header. 2. The update_engine_descriptions.py needs a list of all wikipedias. The implementation from #2269 included only a reduced list: - https://meta.wikimedia.org/wiki/Wikipedia_article_depth - https://meta.wikimedia.org/wiki/List_of_Wikipedias searxng_extra/update/update_engine_descriptions.py ================================================== Before PR #2269 there was a match_language() function that did an approximation using various methods. With PR #2269 there are only the types in the data model of the languages, which can be recognized by babel. The approximation methods, which are needed (only here) in the determination of the descriptions, must be replaced by other methods. [1] https://en.wikipedia.org/wiki/Classical_Chinese [2] https://en.wikipedia.org/wiki/Traditional_Chinese_characters [3] https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter Closes: https://github.com/searxng/searxng/issues/2330 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] wikipedia & wikidata: upgrade to data_type: traits_v1Markus Heiser
BTW this fix an issue in wikipedia: SearXNG's locales zh-TW and zh-HK are now using language `zh-classical` from wikipedia (and not `zh`). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-24[mod] Wikipedia: fetch engine traits (data_type: supported_languages)Markus Heiser
Implements a fetch_traits function for the Wikipedia engines. .. note:: Does not include migration of the request methode from 'supported_languages' to 'traits' (EngineTraits) object! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-03-21Pass black formatting testSolirs
2023-03-21WIKIDATA: Add description for resultsSolirs
2022-09-27[fix] typos / reported by @kianmeng in searx PR-3366Markus Heiser
[PR-3366] https://github.com/searx/searx/pull/3366 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-06-11Wikidata engine: ignore dummy entitiesAlexandre Flament
Close #641
2022-06-11Wikidata engine: minor change of the SPARQL requestAlexandre Flament
The engine can be slow especially when the query won't return any answer. See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to Related to #1290
2022-06-03[fix] prepare for pylint 2.14.0Markus Heiser
Remove issue reported by Pylint 2.14.0: - no-self-use: has been moved to optional extension [1] - The refactoring checker now also raises 'consider-using-generator' messages for max(), min() and sum(). [2] .pylintrc: - <option name>-hint has been removed since long, Pylint 2.14.0 raises an error on invalid options - bad-continuation and bad-whitespace have been removed [3] [1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0 [2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-07[pylint] searx/engines/wikidata.py (no functional change)Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-07[fix] wikidata: URL decoding and file extension handlingLéon Tiekötter
Add '.png' to the second img_src_name if it has the extension '.svg'. Use urllib.parse.unquote for URL decoding.
2022-02-06[fix] wikidata engine: select image with higher (not lower) priorityMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-02-06wikidata: load thumbnail instead of full imageLéon Tiekötter
2022-02-06[fix] wikidata info box imagesLéon Tiekötter
Wikidata info box images are now loaded from uploads.wikimedia.org instead of commons.wikimedia.org to prevent redirects Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27[fix] issues reported by pylintMarkus Heiser
Fix pylint issues from commit (3d96a983) [format.python] initial formatting of the python code Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27[format.python] initial formatting of the python codeMarkus Heiser
This patch was generated by black [1]:: make format.python [1] https://github.com/psf/black Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27[format.python] disable py code formatting for some hunks of codeMarkus Heiser
Disable the python code formatting from python-black, where the readability of code suffers by formatting. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-06[mod] one logger per engine - drop obsolete logger.getChildMarkus Heiser
Remove the no longer needed `logger = logger.getChild(...)` from engines. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-12[httpx] replace searx.poolrequests by searx.networkAlexandre Flament
settings.yml: * outgoing.networks: * can contains network definition * propertiers: enable_http, verify, http2, max_connections, max_keepalive_connections, keepalive_expiry, local_addresses, support_ipv4, support_ipv6, proxies, max_redirects, retries * retries: 0 by default, number of times searx retries to send the HTTP request (using different IP & proxy each time) * local_addresses can be "192.168.0.1/24" (it supports IPv6) * support_ipv4 & support_ipv6: both True by default see https://github.com/searx/searx/pull/1034 * each engine can define a "network" section: * either a full network description * either reference an existing network * all HTTP requests of engine use the same HTTP configuration (it was not the case before, see proxy configuration in master)
2021-01-14[enh] engines: add about variableAlexandre Flament
move meta information from comment to the about variable so the preferences, the documentation can show these information
2020-12-11[enh] add raise_for_httperrorAlexandre Flament
check HTTP response: * detect some comme CAPTCHA challenge (no solving). In this case the engine is suspended for long a time. * otherwise raise HTTPError as before the check is done in poolrequests.py (was before in search.py). update qwant, wikipedia, wikidata to use raise_for_httperror instead of raise_for_status
2020-12-04[fix] wikidata: fix crash when the item has no description at all and at ↵Alexandre Flament
least one URL.
2020-12-01[mod] wikidata: WDGeoAttribute class: doesn't change the method signature of ↵Alexandre Flament
get_str
2020-11-14[mod] remove unused importAlexandre Flament
use from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA so it is possible to easily remove all unused import using autoflake: autoflake --in-place --recursive --remove-all-unused-imports searx tests
2020-10-28[mod] wikidata engine: use one SPARQL request instead of 2 HTTP requests.Alexandre Flament
2020-10-02[mod] move extract_text, extract_url to searx.utilsAlexandre Flament
2020-09-10Drop Python 2 (1/n): remove unicode string and url_utilsDalf
2020-08-27only return one url per "type" in Wikidata (#2151)Marc Abonce Seguin
i.e. only one official website, one Twitter, etc.
2020-06-09[enh] add official site link to the top of the infobox - closes #1644Adam Tauber
2019-11-15[mod] speed optimizationDalf
compile XPath only once avoid redundant call to urlparse get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
2019-07-29[fix] wikidata engine: faster processing, remove one HTTP redirection.Dalf
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection) * url_detail: remove the disabletidy=1 deprecated parameter * Add eval_xpath function: compile once for all xpath. * Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}') * Create an etree.HTMLParser() instead of using the global one (see #1575)
2019-01-07Revert "remove 'all' option from search languages"Noémi Ványi
This reverts commit 4d1770398a6af8902e75c0bd885781584d39e796.
2019-01-06[fix] check language aliases when setting search languageMarc Abonce Seguin
2018-07-06Fix formattingLéo Bourrel
2018-07-05Update path to wikidata imageLéo Bourrel
2018-04-08[fix] make search requests on wikidata more accurateMarc Abonce Seguin
2018-03-27refactor engine's search language handlingMarc Abonce Seguin
Add match_language function in utils to match any user given language code with a list of engine's supported languages. Also add language_aliases dict on each engine to translate standard language codes into the custom codes used by the engine.
2017-12-06remove 'all' option from search languagesmarc
2017-05-15[enh] py3 compatibilityAdam Tauber
2016-12-15tests for _fetch_supported_languages in enginesmarc
and refactor method to make it testable without making requests
2016-12-13[mod] fetch supported languages for several enginesmarc
utils/fetch_languages.py gets languages supported by each engine and generates engines_languages.json with each engine's supported language.
2016-12-13[enh] add supported_languages on engines and auto-generate languages.pymarc
2016-08-05[fix] merge infoboxes based on weightmarc
also minor changes in attributes and images from wikidata
2016-08-05wikidata refactor and more attributes (see issue #560)marc
2016-08-05[enh] multilingual wikidataa01200356
disambiguation and tags are in local language TOFIX: needs to query the api every time to know each label's name
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion