summaryrefslogtreecommitdiff
path: root/utils
AgeCommit message (Collapse)Author
2021-06-22[mod] simple theme: include fontsAlexandre Flament
"npm run webfont" to build the fonts directory. It requires fontforge and ttfautohint distro packages. partial revert of commit 7137d2893f33e9158e8563a21d9586119d175269
2021-06-17[enh] ./manage node.env - check build tools firstMarkus Heiser
The node.env build environment require npm, ttfautohint and fontforge installed in the OS. These tools can be installed by:: sudo -H ./utils/searx.sh install buildhost If one of the tools is not installed, the script node.env stops with a appropriate message. BTW: We ignore CentOS-7 as developer & build platform Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-16[mod] simple theme: remove src/less/ion.lessAlexandre Flament
This file is generated by webfont. * It is now generated as searx/static/themes/simple/ion.less * It is generated before the .less compilation. * .gitignore includes this file Add two new package depedencies: fontforge ttfautohint See utils/searx.sh
2021-06-13[fix] utils/morty.sh - PUBLIC_URL_MORTY is based on PUBLIC_URLMarkus Heiser
The public URL of the searx instance: PUBLIC_URL="https://mydomain.xy/searx" The default is SEARX_URL which is taken from ./utils/brand.env. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-12[yamllint] ./utils/templates/etc/searx/*.ymlMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-05-28[enh] add settings option to enable/disable search formatsMarkus Heiser
Access to formats can be denied by settings configuration:: search: formats: [html, csv, json, rss] Closes: https://github.com/searxng/searxng/issues/95 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28[fix] typo: 'interactiv' --> 'interactive'Markus Heiser
Reported-by: https://github.com/searxng/searxng/pull/38#pullrequestreview-646940531 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-28[filtron.sh] make filtron rule file configurableMarkus Heiser
To select a different file with filtron rules, set environment FILTRON_RULES_TEMPLATE the default is utils/templates/etc/filtron/rules.json The installation is done by the new function install_rules() which offers a multiple choice in case of collisions (known from searx.sh install setup). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-25[brand] searxng -- fix links to issue tracker & WEB-GUIMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23[mod] replace makefile boilerplate by 'manage' scriptMarkus Heiser
Replaces the make targets with the bash scripts Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23[fix] do not spit out error messages for utils/lxc.sh --helpMarkus Heiser
Error:: $ utils/lxc.sh --help ERROR: missing command lxc ... This breaks also docs build:: $ make docs ... WARNING: Unexpected return code 42 from command '../utils/lxc.sh --help' ... Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23[enh] utils/lib.sh - commands to build Sphinx-doc & deploy gh-pagesMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-23[enh] utils/lib.sh - commands pyenv, pyenv.drop pyenv.(un)installMarkus Heiser
Implement a boilerplate to manage performance optimized virtualenv builds. Shell scripts can use (e.g.) 'pyenv.cmd' to execute command in the virtualenv without having to worry about whether and how the environment is provided. :: pyenv.cmd which python ..../local/py3/bin/python pyenv.cmd which pip ..../local/py3/bin/pip If pyenv.cmd released multiple times the installation will only rebuild if the function 'pyenv.OK' fails. Function 'pyenv.OK' make some test to validate that the virtualenv exists and works as expected. The check also fails if requirements listed requirements-dev.txt and requirements.txt has been edited. Among these tests 'pyenv.OK' calls 'pyenv.check' which implements a python script that validate the python installation. Here is an example how a 'pyenv.check' implementation could look like:: pyenv.check() { cat <<EOF import yaml print('import yaml --> OK') EOF } Signed-off-by: Markus Heiser <markus@darmarit.de>
2021-04-23[fix] utils/searx.sh - Double quote to prevent globbingMarkus Heiser
Fix issues reported by `make test.sh` Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-04-22More brandingAlexandre Flament
2021-04-22searxng brandAlexandre Flament
2021-04-08Allow overriding env vars SEARX_SETTINGS_PATH,UWSGI_SETTINGS_PATH3nprob
2021-03-05[mod] LXC switch to Fedora 33 / Fedora 31 reached its EOLMarkus Heiser
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-05Merge pull request #2600 from dalf/searx-extraAlexandre Flament
Add searx_extra package
2021-03-04Add searx_extra packageAlexandre Flament
Split the utils directory into: * searx_extra contains update scripts, standalone_searx.py * utils contains the files to build and setup searx.
2021-03-04Merge pull request #2618 from thezeroalpha/masterAlexandre Flament
Fix security vulnerabilities in provided nginx configuration
2021-03-04Merge pull request #2623 from return42/fix-centos7Alexandre Flament
[fix] add package `which` to CentOS-7 boilerplate
2021-03-04Merge pull request #2619 from return42/drop-ubu1604Alexandre Flament
[mod] Drop Ubuntu 16.04 (Xenial Xerus) support
2021-03-04Merge pull request #2620 from return42/fix-gitAlexandre Flament
[fix] support git versions <v2.22
2021-03-04Use $host in nginx morty.conf templateAlex Balgavy
2021-03-03[fix] add package 'which' to CentOS-7 boilerplateMarkus Heiser
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer include the which command. Issue: $ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all INFO: [searx-centos7] ./utils/filtron.sh install all ... Install Go in user's HOME ------------------------- download and install go binary .. ... -bash: line 1: which: command not found -->|ERROR - Go Installation not found in PATH!?! -bash: line 2: which: command not found Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03[fix] utils/serax.sh create_pyenv() - drop duplicate 'pip install .'Markus Heiser
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is responsible for the error message: ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx The correct pip-install comes right after changing to `cd ${SEARX_SRC}`. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03[fix] support git versions <v2.22Markus Heiser
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.:: $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version ... git version 2.17.1 The option `--show-current` was added in git v2.22, the alternative to this option is:: git rev-parse --abbrev-ref HEAD Issue when using option `--show-current`:: [searx-ubu1804] Clone searx sources [searx-ubu1804] ------------------- [searx-ubu1804] [searx-ubu1804] error: unknown option `show-current' [searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged] .... Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03[mod] Drop Ubuntu 16.04 (Xenial Xerus) supportMarkus Heiser
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04 since a longer time. Issues are comming from dependencies (e.g. pip):: [searx-ubu1604] |searx| SyntaxError: invalid syntax [searx-ubu1604] |searx| Traceback (most recent call last): [searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module> [searx-ubu1604] |searx| from pip._internal.cli.main import main [searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60 [searx-ubu1604] |searx| sys.stderr.write(f"ERROR: {exc}") [searx-ubu1604] |searx| ^ [searx-ubu1604] |searx| SyntaxError: invalid syntax Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-03-03Fix security vulnerabilities in suggested nginx configurationAlex Balgavy
The suggested configurations for nginx found in the documentation and templates lead to vulnerabilities allowing host spoofing [1] and path traversal [2], as reported by Gixy [3]. This commit fixes those issues. [1] https://github.com/yandex/gixy/blob/master/docs/en/plugins/hostspoofing.md [2] https://github.com/yandex/gixy/blob/master/docs/en/plugins/aliastraversal.md [3] https://github.com/yandex/gixy
2021-02-25remove articles number from engines_languages.jsonMarc Abonce Seguin
2021-02-24[mod] add utils/fetch_external_bangs.pyAlexandre Flament
Based on duckduckgo bangs Store bangs on a trie to allow autocomplete (not in this commit)
2021-02-23Merge pull request #2595 from dalf/update-wikidata-unitsAlexandre Flament
[mod] update wikidata_units.json and fetch_wikidata_units.py
2021-02-23[mod] update currencies.json and fetch_currencies.pyAlexandre Flament
use a sparql request on wikidata to get the list of currencies. currencies.json contains the translation for all supported searx languages. Supersede #993
2021-02-23[mod] update wikidata_units.json and fetch_wikidata_units.pyAlexandre Flament
The fetch_wikidata_units.py result won't change randomly. See comments in the script.
2021-02-02[mod] CI: minor changesAlexandre Flament
* utils/makefile.python: travis-gh-pages renamed ci-gh-pages
2021-02-01Merge pull request #2500 from dalf/github-action-dataAlexandre Flament
[enh] every Sunday, call utils/fetch_*.py scripts and create a PR automatically
2021-02-01Merge pull request #2514 from return42/fix-gh-pagesAlexandre Flament
[fix] Makefile target gh-pages & flatten history of branch gh.pages
2021-02-01[fix] re-add 'pip-exe' target - partial revert 9b48ae47Markus Heiser
Target pip-exe is a prerequisite of the targets: - pyinstall - pyuninstall and was accidentally deleted in commit 9b48ae47. HINT: do not confuse pyinstall with penvinstall pyinstall & pyuninstall Installing into user's HOME using pip from OS, therefore the message is needed. pyenvinstall & pyenvuninstall Installing into virtualenv (./local) using pip which is provided by prerequisite 'pyenv' in the virtualenv. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-02-01[mod] more robust make pyenv / make pyenvinstallAlexandre Flament
"make pyenv" ensures that ./local/py3/bin/python is an executable
2021-02-01Merge pull request #2494 from return42/rm-fabfileAlexandre Flament
[fix] remove Fabric file
2021-01-29[fix] Makefile target gh-pages & flatten history of branch gh.pagesMarkus Heiser
1. This patch fixes error: rm -rf gh-pages/ make V=1 gh-pages make[1]: Leaving directory '/800GBPCIex4/share/searx' [ -d "gh-pages/.git" ] || git clone gh-pages fatal: repository 'gh-pages' does not exist 2. The gh-page build has been moved to ./build/gh-pages this also affects 'travis-gh-pages' 3. The gh-pages commit messages now includes a ref to the repository and commit 4. Since a gh-pages history has only the drawback that the reposetory grows fast, this patch also flattens the history: cd build/gh-pages/; git log --oneline bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden 026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-24[mod] utils/fetch_languages.py: write files at the right locationAlexandre Flament
2021-01-24[fix] lxc.sh - SC2034: ubu2010_boilerplate appears unused.Markus Heiser
$ make test.sh In utils/lxc.sh line 42: ubu2010_boilerplate="$ubu1904_boilerplate" ^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-22[fix] remove Fabric fileMarkus Heiser
The fabfile.py has not been updated since 5 years. I also asked [1] if someone still use Fabric wtihout any response. Lets drop outdated Fabric file. [1] https://github.com/searx/searx/discussions/2400 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-19[mod] checker : replace pycld3 by langdetectAlexandre Flament
pycld3 requires the native library cld3 langdetect is a pure python package
2021-01-16Merge pull request #2473 from return42/fix-setup.pyAlexandre Flament
[fix] setup.py requires pyyaml installed
2021-01-16[fix] setup.py requires pyyaml installedMarkus Heiser
pip install -e . ... Obtaining file:///usr/local/searx/searx-src ERROR: Command errored out with exit status 1: command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2 cwd: /usr/local/searx/searx-src/ Complete output (9 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/searx/searx-src/setup.py", line 10, in <module> from searx.version import VERSION_STRING File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module> import searx.settings_loader File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module> import yaml ModuleNotFoundError: No module named 'yaml' ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-01-14[enh] engines: add about variableAlexandre Flament
move meta information from comment to the about variable so the preferences, the documentation can show these information
2021-01-12[enh] add searx.sharedAlexandre Flament
shared dictionary between the workers (UWSGI or werkzeug) scheduler: run a task once every x seconds (UWSGI or werkzeug)
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion