summaryrefslogtreecommitdiff
path: root/utils
AgeCommit message (Collapse)Author
2021-07-21[fix] utils/build_env.py and add documentationMarkus Heiser
modified docs/admin/engines/settings.rst - Fix documentation and add section 'brand'. - Add remarks about **buildenv** variables. - Add remarks about settings from environment variables $SEARX_DEBUG, $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET modified docs/admin/installation-searx.rst & docs/build-templates/searx.rst Fix template location /templates/etc/searx/settings.yml modified docs/dev/makefile.rst Add description of the 'make buildenv' target and describe - we have all SearXNG setups are centralized in the settings.yml file - why some tasks need a utils/brand.env (aka instance's buildenv) modified manage Settings file from repository's working tree are used by default and ask user if a /etc/searx/settings.yml file exists. modified searx/settings.yml Add comments about when it is needed to run 'make buildenv' modified searx/settings_defaults.py Default for server:port is taken from enviroment variable SEARX_PORT. modified utils/build_env.py - Some defaults in the settings.yml are taken from the environment, e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA). When the 'brand.env' file is created these enviroment variables should be unset first. - The CONTACT_URL enviroment is not needed in the utils/brand.env Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] load .config.sh from installation (utils/lib_install.sh)Markus Heiser
**new** utils/lib_install.sh: Used to initialize installation procedures - Modified source_dot_config function that - loads .config.sh from an existing installation (at SEARX_SRC). - initialize **SEARX_SRC_INIT_FILES** - functions like: - install_log_searx_instance() - install_searx_get_state() **modified** utils/searx.sh - obsolete environment SEARX_INSTANCE_NAME has been replaced by string 'SearXNG'. **modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh - source utils/lib_install.sh - normalize logging of environment variables using new function install_log_searx_instance() **modified** utils/lib.sh - fix marginal typos **Installation scripts** The utils/lib_install.sh is sourced by the installations scripts: - utils/searx.sh - utils/morty.sh - utils/filtron.sh If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function loads this configuration (instead of './.config.sh'). **SEARX_SRC_INIT_FILES** Array of file names to sync into a installation at $SEARX_SRC. The file names are relative to the $REPO_ROOT. Set by function init_SEARX_SRC_INIT_FILES(). Most often theses are files like: - .config.sh - searx/settings.yml - utils/brand.env - ... Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] make pyenv.OK() less verbose when pyenv is OKMarkus Heiser
Prompt the verbose messages ... PYENV [check] import yaml --> OK PYENV OK only when environment $VERBOSE is active. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21[mod] drop unused setting option brand:twitter_urlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-18[mod] drop usage of the searx.brand namespace (python procs)Markus Heiser
Added function searx.get_setting(name, default=_unset): Returns the value to which ``name`` point. If there is no such name in the settings and the ``default`` is unset, a KeyError exception is raised. In all the python processes .. - make docs - make buildenv - make install (setup.py) the usage of the 'brand.*' name space is replaced by 'searx.get_setting' function. - brand.SEARX_URL --> get_setting('server.base_url') - brand.GIT_URL --> get_setting('brand.git_url') - brand.GIT_BRANCH' --> get_setting('server.base_url') - brand.ISSUE_URL --> get_setting('brand.issue_url') - brand.DOCS_URL --> get_setting('brand.docs_url') - brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances') - brand.CONTACT_URL --> get_setting('general.contact_url', '') - brand.WIKI_URL --> get_setting('brand.wiki_url') - brand.TWITTER_URL --> get_setting('brand.twitter_url', '') Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03[fix] pacman --noprogressbar in shell scriptsMarkus Heiser
From pacman's man page: --noprogressbar Do not show a progress bar when downloading files. This can be useful for scripts that call pacman and capture the output. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-03[fix] utils/lib.sh: in a pipe, do not print ASCII escape codeMarkus Heiser
In 2a12637 an ASCII escape code was added, such escape codes should not be written to pipes (when the output is not a terminal). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-28Merge pull request #174 from searxng/static_scriptAlexandre Flament
[mod] add make targets to manage the build files in the /searx/static directory
2021-06-27[fix] utils/lib.sh: prefix_stdout show cursor (ANSI escape \e[?25h)Markus Heiser
Some piped commands hide the cursor, show cursory when the stream ends. Most often this is a bug of the command which piped. The command should not hide the cursor when it writes to a pipe. I have seen this bug with the package manager (pacman) from ArchLinux. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26[mod] move functions from utils/manage_static.sh to ./manage scriptMarkus Heiser
The functions: - static.build.commit - static.build.commit.drop - static.build.restore are imported into the ./manage script. To avoid name collisions some variables and fucntions has been renamed by adding the prefix *static_*. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26[mod] utils/manage_static.sh: remove git_log_current_branchMarkus Heiser
Marginal simplification of the procedure to get only the last local commit which is not in remotes. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26[mod] utils/manage_static.sh: simplify git restore actionsMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26[mod] utils/manage_static.sh: add script to shellcheckMarkus Heiser
- add script to shellcheck, - fix error messages from shellcheck and - moved global variables to local variables (lower case) No functional change! Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26[mod] add make targets to manage the build files in the /searx/static directoryAlexandre Flament
The idea is to avoid conflict when there are differents branches with changes are made on the static files. A solution is to ask the administrators to build the files from the sources, but it requires to install the npm dependencies. So the solution in this commit keep the sources and the build files in the same git repository. In one branch, the modification of the source (*.less, *.js) are commited without the built files. The built files are commited in a uniq commit, with a commit message "Static build" In case of merge or rebase, this commit can be dropped. New make targets: * static.build.commit.drop: drop the last "Static build". The command checks that there are only build files in the commit. * static.build.commit : call "make static.build.commit.drop" call "make themes.all" commit the files * static.git.restore.staged: git restore --staged <build files> * static.git.restore: git restore <build files> Related to https://github.com/searxng/searxng/issues/137
2021-06-25Merge pull request #173 from return42/fix-lxc-iptablesMarkus Heiser
[mod] utils/lxc.sh: detect conflict of docker & LXC in the iptables
2021-06-23[mod] utils/lxc.sh: detect conflict of docker & LXC in the iptablesMarkus Heiser
Docker is blocking network of existing LXC containers / there is a conflict in the iptables setup of Docker & LXC. With this patch: - utils/lxc.sh checks internet connectivity (instead of silently hang) - Chapter "Internet Connectivity & Docker" describes the problem and made a suggestion for a solution a solution Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-23[fix] utils/searx.sh: signature of YAML settings has been changedMarkus Heiser
In 5a7b12ee we changed the signature of the YAML settings, this patch takes this into account. Related-to: 5a7b12ee [yamllint] searx/settings.yml Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-22[mod] simple theme: include fontsAlexandre Flament
"npm run webfont" to build the fonts directory. It requires fontforge and ttfautohint distro packages. partial revert of commit 7137d2893f33e9158e8563a21d9586119d175269
2021-06-17[enh] ./manage node.env - check build tools firstMarkus Heiser
The node.env build environment require npm, ttfautohint and fontforge installed in the OS. These tools can be installed by:: sudo -H ./utils/searx.sh install buildhost If one of the tools is not installed, the script node.env stops with a appropriate message. BTW: We ignore CentOS-7 as developer & build platform Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-16[mod] simple theme: remove src/less/ion.lessAlexandre Flament
This file is generated by webfont. * It is now generated as searx/static/themes/simple/ion.less * It is generated before the .less compilation. * .gitignore includes this file Add two new package depedencies: fontforge ttfautohint See utils/searx.sh
2021-06-13[fix] utils/morty.sh - PUBLIC_URL_MORTY is based on PUBLIC_URLMarkus Heiser
The public URL of the searx instance: PUBLIC_URL="https://mydomain.xy/searx" The default is SEARX_URL which is taken from ./utils/brand.env. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-12[yamllint] ./utils/templates/etc/searx/*.ymlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28[enh] add settings option to enable/disable search formatsMarkus Heiser
Access to formats can be denied by settings configuration:: search: formats: [html, csv, json, rss] Closes: https://github.com/searxng/searxng/issues/95 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28[fix] typo: 'interactiv' --> 'interactive'Markus Heiser
Reported-by: https://github.com/searxng/searxng/pull/38#pullrequestreview-646940531 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28[filtron.sh] make filtron rule file configurableMarkus Heiser
To select a different file with filtron rules, set environment FILTRON_RULES_TEMPLATE the default is utils/templates/etc/filtron/rules.json The installation is done by the new function install_rules() which offers a multiple choice in case of collisions (known from searx.sh install setup). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25[brand] searxng -- fix links to issue tracker & WEB-GUIMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23[mod] replace makefile boilerplate by 'manage' scriptMarkus Heiser
Replaces the make targets with the bash scripts Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23[fix] do not spit out error messages for utils/lxc.sh --helpMarkus Heiser
Error:: $ utils/lxc.sh --help ERROR: missing command lxc ... This breaks also docs build:: $ make docs ... WARNING: Unexpected return code 42 from command '../utils/lxc.sh --help' ... Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23[enh] utils/lib.sh - commands to build Sphinx-doc & deploy gh-pagesMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23[enh] utils/lib.sh - commands pyenv, pyenv.drop pyenv.(un)installMarkus Heiser
Implement a boilerplate to manage performance optimized virtualenv builds. Shell scripts can use (e.g.) 'pyenv.cmd' to execute command in the virtualenv without having to worry about whether and how the environment is provided. :: pyenv.cmd which python ..../local/py3/bin/python pyenv.cmd which pip ..../local/py3/bin/pip If pyenv.cmd released multiple times the installation will only rebuild if the function 'pyenv.OK' fails. Function 'pyenv.OK' make some test to validate that the virtualenv exists and works as expected. The check also fails if requirements listed requirements-dev.txt and requirements.txt has been edited. Among these tests 'pyenv.OK' calls 'pyenv.check' which implements a python script that validate the python installation. Here is an example how a 'pyenv.check' implementation could look like:: pyenv.check() { cat <<EOF import yaml print('import yaml --> OK') EOF } Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23[fix] utils/searx.sh - Double quote to prevent globbingMarkus Heiser
Fix issues reported by `make test.sh` Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22More brandingAlexandre Flament
2021-04-22searxng brandAlexandre Flament
2021-04-08Allow overriding env vars SEARX_SETTINGS_PATH,UWSGI_SETTINGS_PATH3nprob
2021-03-05[mod] LXC switch to Fedora 33 / Fedora 31 reached its EOLMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05Merge pull request #2600 from dalf/searx-extraAlexandre Flament
Add searx_extra package
2021-03-04Add searx_extra packageAlexandre Flament
Split the utils directory into: * searx_extra contains update scripts, standalone_searx.py * utils contains the files to build and setup searx.
2021-03-04Merge pull request #2618 from thezeroalpha/masterAlexandre Flament
Fix security vulnerabilities in provided nginx configuration
2021-03-04Merge pull request #2623 from return42/fix-centos7Alexandre Flament
[fix] add package `which` to CentOS-7 boilerplate
2021-03-04Merge pull request #2619 from return42/drop-ubu1604Alexandre Flament
[mod] Drop Ubuntu 16.04 (Xenial Xerus) support
2021-03-04Merge pull request #2620 from return42/fix-gitAlexandre Flament
[fix] support git versions <v2.22
2021-03-04Use $host in nginx morty.conf templateAlex Balgavy
2021-03-03[fix] add package 'which' to CentOS-7 boilerplateMarkus Heiser
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer include the which command. Issue: $ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all INFO: [searx-centos7] ./utils/filtron.sh install all ... Install Go in user's HOME ------------------------- download and install go binary .. ... -bash: line 1: which: command not found -->|ERROR - Go Installation not found in PATH!?! -bash: line 2: which: command not found Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03[fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'Markus Heiser
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is responsible for the error message: ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx The correct pip-install comes right after changing to `cd ${SEARX_SRC}`. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03[fix] support git versions <v2.22Markus Heiser
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.:: $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version ... git version 2.17.1 The option `--show-current` was added in git v2.22, the alternative to this option is:: git rev-parse --abbrev-ref HEAD Issue when using option `--show-current`:: [searx-ubu1804] Clone searx sources [searx-ubu1804] ------------------- [searx-ubu1804] [searx-ubu1804] error: unknown option `show-current' [searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged] .... Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03[mod] Drop Ubuntu 16.04 (Xenial Xerus) supportMarkus Heiser
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04 since a longer time. Issues are comming from dependencies (e.g. pip):: [searx-ubu1604] |searx| SyntaxError: invalid syntax [searx-ubu1604] |searx| Traceback (most recent call last): [searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module> [searx-ubu1604] |searx| from pip._internal.cli.main import main [searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60 [searx-ubu1604] |searx| sys.stderr.write(f"ERROR: {exc}") [searx-ubu1604] |searx| ^ [searx-ubu1604] |searx| SyntaxError: invalid syntax Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03Fix security vulnerabilities in suggested nginx configurationAlex Balgavy
The suggested configurations for nginx found in the documentation and templates lead to vulnerabilities allowing host spoofing [1] and path traversal [2], as reported by Gixy [3]. This commit fixes those issues. [1] https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md [2] https://github.com/yandex/gixy/blob/master/docs/en/plugins/aliastraversal.md [3] https://github.com/yandex/gixy
2021-02-25remove articles number from engines_languages.jsonMarc Abonce Seguin
2021-02-24[mod] add utils/fetch_external_bangs.pyAlexandre Flament
Based on duckduckgo bangs Store bangs on a trie to allow autocomplete (not in this commit)
2021-02-23Merge pull request #2595 from dalf/update-wikidata-unitsAlexandre Flament
[mod] update wikidata_units.json and fetch_wikidata_units.py
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion