summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2021-07-21[brand] searx.sh & morty.sh - renamed 'searx' into 'SearXNG'Markus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] utils/.searx.sh: add commands: install [dot-config|init-src]Markus Heiser
This patch implements two new commands: install dot-config : - copy ./config.sh to ${SEARX_SRC} - implemented in new function install_DOT_CONFIG install init-src: - sync files (SEARX_SRC_INIT_FILES) with ${SEARX_SRC} - implemented in new function init_SEARX_SRC() new functions: verify_continue_install() and prompt_installation_status() These functions are used in installation procedures to get the status of the installation procedure. The status is based on the existing function: usage: install_searx_get_state Prompts a string indicating the status of the installation procedure missing-searx-clone: There is no clone at ${SEARX_SRC} missing-searx-pyenv: There is no pyenv in ${SEARX_PYENV} installer-modified: There are files modified locally in the installer (clone), see ${SEARX_SRC_INIT_FILES} description. python-installed: Scripts can be executed in instance's environment - user: ${SERVICE_USER} - pyenv: ${SEARX_PYENV} Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] normalize .config.sh with settings.ymlMarkus Heiser
In commit 94851790 we have centralized all SearXNG setups in the settings.yml file: 94851790 [mod] move brand options from Makefile to settings.yml This step has not yet been completed for the installation procedures! Since all SearXNG setups are done in the settings.yml these environment variables needs to be removed from the ./conf.sh file. Scripts and other tasks running outside of an instance got the needed values from the ./utils/brand.env file. By example: ATM the environment variables of the ./config.sh file are in conflict with them from settings.yml: - PUBLIC_URL --> {server:base_url} - SEARX_INTERNAL_HTTP --> {server:bind_address}.{server:port} - GIT_BRANCH --> {brand:GIT_URL} These environment variable of a SearXNG instance and additional - SEARX_SETTINGS_TEMPLATE has been remove from the '.config.sh' file. With this patch, the main focus of ./conf.sh resists on environment variables needed for the installation of morty, filtron software. modified .config.sh: - removed no longer supported variables (see above) - add comment about: SearXNG setup in settings.yml modified utils/searx.sh: - SEARX_INTERNAL_HTTP no longer take from .config.sh - SEARX_SETTINGS_PATH /etc/searx/settings.yml - SEARX_SETTINGS_TEMPLATE obsolete modified utils/lib_install.sh: Initialize environment variables SEARX_PYENV, SEARX_SETTINGS_PATH and PUBLIC_URL. modified: utils/morty.sh Add missing hint about SEARX_SETTINGS_PATH and move PUBLIC_URL to utils/lib_install.sh modified: utils/morty.sh Move PUBLIC_URL to utils/lib_install.sh Renamed utils/templates/etc/searx/use_default_settings.yml -> settings.yml - removed option which can't be modified after installation - add some comments with examples Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[fix] utils/build_env.py and add documentationMarkus Heiser
modified docs/admin/engines/settings.rst - Fix documentation and add section 'brand'. - Add remarks about **buildenv** variables. - Add remarks about settings from environment variables $SEARX_DEBUG, $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET modified docs/admin/installation-searx.rst & docs/build-templates/searx.rst Fix template location /templates/etc/searx/settings.yml modified docs/dev/makefile.rst Add description of the 'make buildenv' target and describe - we have all SearXNG setups are centralized in the settings.yml file - why some tasks need a utils/brand.env (aka instance's buildenv) modified manage Settings file from repository's working tree are used by default and ask user if a /etc/searx/settings.yml file exists. modified searx/settings.yml Add comments about when it is needed to run 'make buildenv' modified searx/settings_defaults.py Default for server:port is taken from enviroment variable SEARX_PORT. modified utils/build_env.py - Some defaults in the settings.yml are taken from the environment, e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA). When the 'brand.env' file is created these enviroment variables should be unset first. - The CONTACT_URL enviroment is not needed in the utils/brand.env Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] load .config.sh from installation (utils/lib_install.sh)Markus Heiser
**new** utils/lib_install.sh: Used to initialize installation procedures - Modified source_dot_config function that - loads .config.sh from an existing installation (at SEARX_SRC). - initialize **SEARX_SRC_INIT_FILES** - functions like: - install_log_searx_instance() - install_searx_get_state() **modified** utils/searx.sh - obsolete environment SEARX_INSTANCE_NAME has been replaced by string 'SearXNG'. **modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh - source utils/lib_install.sh - normalize logging of environment variables using new function install_log_searx_instance() **modified** utils/lib.sh - fix marginal typos **Installation scripts** The utils/lib_install.sh is sourced by the installations scripts: - utils/searx.sh - utils/morty.sh - utils/filtron.sh If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function loads this configuration (instead of './.config.sh'). **SEARX_SRC_INIT_FILES** Array of file names to sync into a installation at $SEARX_SRC. The file names are relative to the $REPO_ROOT. Set by function init_SEARX_SRC_INIT_FILES(). Most often theses are files like: - .config.sh - searx/settings.yml - utils/brand.env - ... Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21Merge pull request #213 from return42/drop-brandAlexandre Flament
[mod] drop searx.brand namespace
2021-07-21[mod] make pyenv.OK() less verbose when pyenv is OKMarkus Heiser
Prompt the verbose messages ... PYENV [check] import yaml --> OK PYENV OK only when environment $VERBOSE is active. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] ./manage docker.buildx make variables localMarkus Heiser
The environment variables .. - SEARX_GIT_VERSION - VERSION_GITCOMMIT - SEARX_PYTHON_VERSION - GITHUB_USER - SEARX_IMAGE_NAME - BUILD are local to the docker.buildx() function, other variables like GIT_URL are comming from the utils/brand.env file (rebuild by: make buildenv). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] drop obsolete searx.brand namespaceMarkus Heiser
The usages of the searx.brand namespace has been removed, the searx.brand namespace is now longer needed. The searx.brand namespace was an interim solution which has been added in commit 9e53470b4, see commit message there ... This patch is a first 'proof of concept'. Later we can decide to remove the brand namespace entirely or not. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] drop usage of the searx.brand namespace (templates & /config)Markus Heiser
In the templates and the /config (JSON) the usage of the 'brand.*' name space is replaced by 'searx.get_setting' function. - new_issue_url --> get_setting('brand.new_issue_url') - brand.GIT_URL --> get_setting('brand.git_url') - brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances') - brand.DOCS_URL --> get_setting('brand.docs_url') - brand.ISSUE_URL --> get_setting('brand.issue_url') - brand.CONTACT_URL --> get_setting('general.contact_url', '') The macro 'new_issue' from searx/templates/*/messages/no_results.html is now imported with context:: {% from '__common__/new_issue.html' import new_issue with context %} To get *public instances URL* from context's 'get_setting()' function:: get_setting('brand.public_instances','') Macro's prototype does no longer need the 'new_issue_url' argument and has been changed to:: macro new_issue(engine_name, engine_reliability) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] drop unused setting option brand:twitter_urlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[fix] add missing brand options to the settings SCHEMAMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-19Merge pull request #106 from searxng/dependabot/pip/master/flask-2.0.1Alexandre Flament
Bump flask from 1.1.2 to 2.0.1
2021-07-18[mod] drop usage of the searx.brand namespace (python procs)Markus Heiser
Added function searx.get_setting(name, default=_unset): Returns the value to which ``name`` point. If there is no such name in the settings and the ``default`` is unset, a KeyError exception is raised. In all the python processes .. - make docs - make buildenv - make install (setup.py) the usage of the 'brand.*' name space is replaced by 'searx.get_setting' function. - brand.SEARX_URL --> get_setting('server.base_url') - brand.GIT_URL --> get_setting('brand.git_url') - brand.GIT_BRANCH' --> get_setting('server.base_url') - brand.ISSUE_URL --> get_setting('brand.issue_url') - brand.DOCS_URL --> get_setting('brand.docs_url') - brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances') - brand.CONTACT_URL --> get_setting('general.contact_url', '') - brand.WIKI_URL --> get_setting('brand.wiki_url') - brand.TWITTER_URL --> get_setting('brand.twitter_url', '') Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-17Merge pull request #212 from return42/fix-manageMarkus Heiser
[fix] ./manage script, prompt the unknown command
2021-07-17[fix] ./manage script, prompt the unknown commandMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-17Bump flask from 1.1.2 to 2.0.1dependabot[bot]
Bumps [flask](https://github.com/pallets/flask) from 1.1.2 to 2.0.1. - [Release notes](https://github.com/pallets/flask/releases) - [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/flask/compare/1.1.2...2.0.1) Signed-off-by: dependabot[bot] <support@github.com>
2021-07-17Merge pull request #89 from searxng/dependabot/pip/master/jinja2-3.0.1Alexandre Flament
Bump jinja2 from 2.11.3 to 3.0.1
2021-07-16Merge pull request #211 from MarcAbonce/onions_v3_fix_searxngAlexandre Flament
Update onion engines to v3
2021-07-16Merge pull request #210 from searxng/dependabot/pip/master/sphinx-4.1.1Alexandre Flament
Bump sphinx from 4.0.2 to 4.1.1
2021-07-16Merge pull request #207 from return42/mongodbAlexandre Flament
[enh] add mongodb offline engine
2021-07-16Bump jinja2 from 2.11.3 to 3.0.1dependabot[bot]
Bumps [jinja2](https://github.com/pallets/jinja) from 2.11.3 to 3.0.1. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](https://github.com/pallets/jinja/compare/2.11.3...3.0.1) Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16Merge pull request #209 from searxng/dependabot/pip/master/python-dateutil-2.8.2Alexandre Flament
Bump python-dateutil from 2.8.1 to 2.8.2
2021-07-16Merge pull request #206 from return42/improve-qwantAlexandre Flament
[mod] improve video results of the qwant engine
2021-07-16[mod] qwant engine: fix typos / minor changeAlexandre Flament
minor modification of commit 628b5703f3aeeed117772696f83efb344d6f337e (no functionnal change)
2021-07-16update onion engines to v3Marc Abonce Seguin
remove not_evil which has been down for a while now: https://old.reddit.com/r/onions/search/?q=not+evil&restrict_sr=on&t=year
2021-07-16Bump python-dateutil from 2.8.1 to 2.8.2dependabot[bot]
Bumps [python-dateutil](https://github.com/dateutil/dateutil) from 2.8.1 to 2.8.2. - [Release notes](https://github.com/dateutil/dateutil/releases) - [Changelog](https://github.com/dateutil/dateutil/blob/master/NEWS) - [Commits](https://github.com/dateutil/dateutil/compare/2.8.1...2.8.2) --- updated-dependencies: - dependency-name: python-dateutil dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16Merge pull request #208 from searxng/dependabot/pip/master/uvloop-0.15.3Alexandre Flament
Bump uvloop from 0.15.2 to 0.15.3
2021-07-16Bump sphinx from 4.0.2 to 4.1.1dependabot[bot]
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.0.2 to 4.1.1. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.0.2...v4.1.1) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
2021-07-16Bump uvloop from 0.15.2 to 0.15.3dependabot[bot]
Bumps [uvloop](https://github.com/MagicStack/uvloop) from 0.15.2 to 0.15.3. - [Release notes](https://github.com/MagicStack/uvloop/releases) - [Commits](https://github.com/MagicStack/uvloop/compare/v0.15.2...v0.15.3) --- updated-dependencies: - dependency-name: uvloop dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
2021-07-15[enh] add mongodb offline engineMarkus Heiser
Cherry-Pick: https://github.com/searx/searx/commit/198aad43 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15[mod] improve video results of the qwant engineMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-15Merge pull request #205 from unixfox/patch-2Alexandre Flament
Add missing parameter for mobile UI search
2021-07-15Merge pull request #204 from return42/fix-qwantAlexandre Flament
[fix] Qwant engine - implement API v3 and add engine "qwant videos"
2021-07-15Add missing parameter for mobile UI searchÉmilien Devos
2021-07-14[mod] enable Qwant engines (web, news, images, video) by defaultMarkus Heiser
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA. Let us enable Qwant engines by default. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-14[fix] Qwant engines - implement API v3 and add 'quant videos'Markus Heiser
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is undocumented but can be reverse engineered by reading the network log of https://www.qwant.com/ queries. This implementation is used by different qwant engines in the settings.yml:: - name: qwant categories: general ... - name: qwant news categories: news ... - name: qwant images categories: images ... - name: qwant videos categories: videos ... Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-13Merge pull request #203 from return42/drop-dogpileAlexandre Flament
[fix] drop engine dogpile
2021-07-13[fix] drop engine dogpileMarkus Heiser
The engine was added in commit a4b07460 but now it shows new issues [1]. In the 90'th of the last century, dogpile had its own WEB index, but nowadays it is a meta-search engine [2] Powered by technology, Dogpile returns all the best results from leading search engines including Google and Yahoo! Using dogpile as an engine in SearXNG needs more investigation, a XPath solution like we have is not enough. It is questionable whether it still makes sense to investigate more into a meta-search engine with a ReCAPTCHA in front. With this patch the dogpile engine is removed [1] https://github.com/searxng/searxng/issues/202 [2] https://www.dogpile.com/support/aboutus Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-12Merge pull request #201 from return42/drop-torrentzAlexandre Flament
[drop] engine torrentz: torrentz2.eu and torrentz2.is are offline
2021-07-11[drop] engine torrentz: torrentz2.eu and torrentz2.is are offlineMarkus Heiser
[1] https://torrentfreak.com/torrentz2-eu-domain-suspended-by-registry-on-public-prosecutors-order-200628/ Suggested-by: @rasos https://github.com/searx/searx/issues/1875#issuecomment-877755872 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-10Merge pull request #200 from unixfox/patch-1Markus Heiser
Fix google images
2021-07-10Fix google imagesÉmilien Devos
Proposed fix in https://github.com/searx/searx/pull/2115#issuecomment-876716010
2021-07-03Merge pull request #198 from return42/gpodder.netMarkus Heiser
[enh] add gpodder.net (JSON)
2021-07-03Merge pull request #197 from return42/woxikonMarkus Heiser
Add woxikon.de synonyme (Xpath)
2021-07-03[enh] add gpodder.net (JSON)Allen
Engine just for Podcasts. An API which returns Podcasts and their Info like: website, author etc. Upstream query example: https://gpodder.net/search.json?q=linux
2021-07-03[yamllint] fix some warning issues reported by yamllintMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03[enh] add woxikon.de synonyme (Xpath)Allen
Added synonyme.woxikon.de using the xpath engine. Adds a site which returns word synonyms although just in German. Depending on the query not all synonyms are shown because of not the best xpath selection. But should do the job just fine. Upstream example query: https://synonyme.woxikon.de/synonyme/test.php
2021-07-03Merge pull request #196 from return42/fix-braveMarkus Heiser
[fix] brave engine: shows descriptions with their correct URLs
2021-07-03[fix] brave engine: shows descriptions with their correct URLsMarkus Heiser
BTW add about section to the YAML configuration It now shows descriptions with their correct URLs when there are videos in the search results, pulling content_xpath from snippet-description instead of snippet-content. Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion