diff options
Diffstat (limited to '_sources/admin')
27 files changed, 3031 insertions, 0 deletions
diff --git a/_sources/admin/answer-captcha.rst.txt b/_sources/admin/answer-captcha.rst.txt new file mode 100644 index 000000000..7ae29be0c --- /dev/null +++ b/_sources/admin/answer-captcha.rst.txt @@ -0,0 +1,69 @@ +=============================== +Answer CAPTCHA from server's IP +=============================== + +With a SSH tunnel we can send requests from server's IP and solve a CAPTCHA that +blocks requests from this IP. If your SearXNG instance is hosted at +``example.org`` and your login is ``user`` you can setup a proxy simply by +:man:`ssh`: + +.. code:: bash + + # SOCKS server: socks://127.0.0.1:8080 + + $ ssh -q -N -D 8080 user@example.org + +The ``socks://localhost:8080`` from above can be tested by: + +.. tabs:: + + .. group-tab:: server's IP + + .. code:: bash + + $ curl -x socks://127.0.0.1:8080 http://ipecho.net/plain + n.n.n.n + + .. group-tab:: desktop's IP + + .. code:: bash + + $ curl http://ipecho.net/plain + x.x.x.x + +In the settings of the WEB browser open the *"Network Settings"* and setup a +proxy on ``SOCKS5 127.0.0.1:8080`` (see screenshot below). In the WEB browser +check the IP from the server is used: + +- http://ipecho.net/plain + +Now open the search engine that blocks requests from your server's IP. If you +have `issues with the qwant engine +<https://github.com/searxng/searxng/issues/2011#issuecomment-1553317619>`__, +solve the CAPTCHA from `qwant.com <https://www.qwant.com/>`__. + +----- + +.. tabs:: + + .. group-tab:: Firefox + + .. kernel-figure:: answer-captcha/ffox-setting-proxy-socks.png + :alt: FFox proxy on SOCKS5, 127.0.0.1:8080 + + Firefox's network settings + + +.. admonition:: :man:`ssh` manual: + + -D [bind_address:]port + Specifies a local “dynamic” application-level port forwarding. This works + by allocating a socket to listen to port on the local side .. Whenever a + connection is made to this port, the connection is forwarded over the + secure channel, and the application protocol is then used to determine + where to connect to from the remote machine .. ssh will act as a SOCKS + server .. + + -N + Do not execute a remote command. This is useful for just forwarding ports. + diff --git a/_sources/admin/api.rst.txt b/_sources/admin/api.rst.txt new file mode 100644 index 000000000..8bd563ba7 --- /dev/null +++ b/_sources/admin/api.rst.txt @@ -0,0 +1,92 @@ +.. _adminapi: + +================== +Administration API +================== + +Get configuration data +====================== + +.. code:: http + + GET /config HTTP/1.1 + +Sample response +--------------- + +.. code:: json + + { + "autocomplete": "", + "categories": [ + "map", + "it", + "images", + ], + "default_locale": "", + "default_theme": "simple", + "engines": [ + { + "categories": [ + "map" + ], + "enabled": true, + "name": "openstreetmap", + "shortcut": "osm" + }, + { + "categories": [ + "it" + ], + "enabled": true, + "name": "arch linux wiki", + "shortcut": "al" + }, + { + "categories": [ + "images" + ], + "enabled": true, + "name": "google images", + "shortcut": "goi" + }, + { + "categories": [ + "it" + ], + "enabled": false, + "name": "bitbucket", + "shortcut": "bb" + }, + ], + "instance_name": "searx", + "locales": { + "de": "Deutsch (German)", + "en": "English", + "eo": "Esperanto (Esperanto)", + }, + "plugins": [ + { + "enabled": true, + "name": "HTTPS rewrite" + } + ], + "safe_search": 0 + } + + +Embed search bar +================ + +The search bar can be embedded into websites. Just paste the example into the +HTML of the site. URL of the SearXNG instance and values are customizable. + +.. code:: html + + <form method="post" action="https://example.org/"> + <!-- search --> <input type="text" name="q"> + <!-- categories --> <input type="hidden" name="categories" value="general,social media"> + <!-- language --> <input type="hidden" name="lang" value="all"> + <!-- locale --> <input type="hidden" name="locale" value="en"> + <!-- date filter --> <input type="hidden" name="time_range" value="month"> + </form> diff --git a/_sources/admin/architecture.rst.txt b/_sources/admin/architecture.rst.txt new file mode 100644 index 000000000..d0d40715d --- /dev/null +++ b/_sources/admin/architecture.rst.txt @@ -0,0 +1,38 @@ +.. _architecture: + +============ +Architecture +============ + +.. sidebar:: Further reading + + - Reverse Proxy: :ref:`Apache <apache searxng site>` & :ref:`nginx <nginx + searxng site>` + - uWSGI: :ref:`searxng uwsgi` + - SearXNG: :ref:`installation basic` + +Herein you will find some hints and suggestions about typical architectures of +SearXNG infrastructures. + +.. _architecture uWSGI: + +uWSGI Setup +=========== + +We start with a *reference* setup for public SearXNG instances which can be build +up and maintained by the scripts from our :ref:`toolboxing`. + +.. _arch public: + +.. kernel-figure:: arch_public.dot + :alt: arch_public.dot + + Reference architecture of a public SearXNG setup. + +The reference installation activates ``server.limiter``, ``server.image_proxy`` +and ``ui.static_use_hash`` (:origin:`/etc/searxng/settings.yml +<utils/templates/etc/searxng/settings.yml>`) + +.. literalinclude:: ../../utils/templates/etc/searxng/settings.yml + :language: yaml + :end-before: # preferences: diff --git a/_sources/admin/buildhosts.rst.txt b/_sources/admin/buildhosts.rst.txt new file mode 100644 index 000000000..bdb9ae1d4 --- /dev/null +++ b/_sources/admin/buildhosts.rst.txt @@ -0,0 +1,163 @@ +.. _buildhosts: + +========== +Buildhosts +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +To get best results from build, it's recommend to install additional packages on +build hosts (see :ref:`searxng.sh`). + +.. _searxng.sh install buildhost: + +Build and Development tools +=========================== + +To Install tools used by build and development tasks in once: + +.. tabs:: + + .. group-tab:: SearXNG's development tools + + .. code:: sh + + $ sudo -H ./utils/searxng.sh install buildhost + +This will install packages needed by SearXNG: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START distro-packages + :end-before: END distro-packages + +and packages needed to build documentation and run tests: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START build-packages + :end-before: END build-packages + +.. _docs build: + +Build docs +========== + +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org +.. _XeTeX: https://tug.org/xetex/ +.. _dvisvgm: https://dvisvgm.de/ + +.. sidebar:: Sphinx build needs + + - ImageMagick_ + - Graphviz_ + - XeTeX_ + - dvisvgm_ + +Most of the sphinx requirements are installed from :origin:`setup.py` and the +docs can be build from scratch with ``make docs.html``. For better math and +image processing additional packages are needed. The XeTeX_ needed not only for +PDF creation, it's also needed for :ref:`math` when HTML output is build. + +To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered +as images (``sphinx.ext.imgmath`` extension). + +Here is the extract from the :origin:`docs/conf.py` file, setting math renderer +to ``imgmath``: + +.. literalinclude:: ../conf.py + :language: python + :start-after: # sphinx.ext.imgmath setup + :end-before: # sphinx.ext.imgmath setup END + +If your docs build (``make docs.html``) shows warnings like this:: + + WARNING: dot(1) not found, for better output quality install \ + graphviz from https://www.graphviz.org + .. + WARNING: LaTeX command 'latex' cannot be run (needed for math \ + display), check the imgmath_latex setting + +you need to install additional packages on your build host, to get better HTML +output (:ref:`install buildhost <searxng.sh install buildhost>`). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install graphviz imagemagick texlive-xetex librsvg2-bin + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S graphviz imagemagick texlive-bin extra/librsvg + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install graphviz graphviz-gd ImageMagick texlive-xetex-bin librsvg2-tools + + +For PDF output you also need: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + $ sudo apt texlive-latex-recommended texlive-extra-utils ttf-dejavu + + .. group-tab:: Arch Linux + + .. code:: sh + + $ sudo pacman -S texlive-core texlive-latexextra ttf-dejavu + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + $ sudo dnf install \ + texlive-collection-fontsrecommended texlive-collection-latex \ + dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts + +.. _sh lint: + +Lint shell scripts +================== + +.. _ShellCheck: https://github.com/koalaman/shellcheck + +To lint shell scripts we use ShellCheck_ - a shell script static analysis tool +(:ref:`install buildhost <searxng.sh install buildhost>`). + +.. SNIP sh lint requirements + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install shellcheck + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S shellcheck + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install ShellCheck + +.. SNAP sh lint requirements diff --git a/_sources/admin/index.rst.txt b/_sources/admin/index.rst.txt new file mode 100644 index 000000000..b47074a8f --- /dev/null +++ b/_sources/admin/index.rst.txt @@ -0,0 +1,23 @@ +=========================== +Administrator documentation +=========================== + +.. toctree:: + :maxdepth: 2 + + settings/index + installation + installation-docker + installation-scripts + installation-searxng + installation-uwsgi + installation-nginx + installation-apache + update-searxng + answer-captcha + searx.favicons + searx.limiter + api + architecture + plugins + buildhosts diff --git a/_sources/admin/installation-apache.rst.txt b/_sources/admin/installation-apache.rst.txt new file mode 100644 index 000000000..b0b580607 --- /dev/null +++ b/_sources/admin/installation-apache.rst.txt @@ -0,0 +1,388 @@ +.. _installation apache: + +====== +Apache +====== + +.. _Apache: https://httpd.apache.org/ +.. _Apache Debian: + https://cwiki.apache.org/confluence/display/HTTPD/DistrosDefaultLayout#DistrosDefaultLayout-Debian,Ubuntu(Apachehttpd2.x): +.. _apache2.README.Debian: + https://salsa.debian.org/apache-team/apache2/raw/master/debian/apache2.README.Debian +.. _Apache Arch Linux: + https://wiki.archlinux.org/index.php/Apache_HTTP_Server +.. _Apache Fedora: + https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-apache-http-server/index.html +.. _Apache directives: + https://httpd.apache.org/docs/trunk/mod/directives.html +.. _Getting Started: + https://httpd.apache.org/docs/current/en/getting-started.html +.. _Terms Used to Describe Directives: + https://httpd.apache.org/docs/current/en/mod/directive-dict.html +.. _Configuration Files: + https://httpd.apache.org/docs/current/en/configuring.html +.. _ProxyPreserveHost: https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypreservehost +.. _LoadModule: + https://httpd.apache.org/docs/mod/mod_so.html#loadmodule +.. _IncludeOptional: + https://httpd.apache.org/docs/mod/core.html#includeoptional +.. _DocumentRoot: + https://httpd.apache.org/docs/trunk/mod/core.html#documentroot +.. _Location: + https://httpd.apache.org/docs/trunk/mod/core.html#location +.. _uWSGI Apache support: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html +.. _mod_proxy_uwsgi: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-proxy-uwsgi +.. _mod_proxy_http: + https://httpd.apache.org/docs/current/mod/mod_proxy_http.html +.. _mod_proxy: + https://httpd.apache.org/docs/current/mod/mod_proxy.html + + +This section explains how to set up a SearXNG instance using the HTTP server Apache_. +If you did use the :ref:`installation scripts` and do not have any special preferences +you can install the :ref:`SearXNG site <apache searxng site>` using +:ref:`searxng.sh <searxng.sh overview>`: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install apache + +If you have special interests or problems with setting up Apache, the following +section might give you some guidance. + + +.. sidebar:: further read + + - `Apache Arch Linux`_ + - `Apache Debian`_ + - `apache2.README.Debian`_ + - `Apache Fedora`_ + - `Apache directives`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +The Apache HTTP server +====================== + +If Apache_ is not installed, install it now. If apache_ is new to you, the +`Getting Started`_, `Configuration Files`_ and `Terms Used to Describe +Directives`_ documentation gives first orientation. There is also a list of +`Apache directives`_ *to keep in the pocket*. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H apt-get install apache2 + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H pacman -S apache + sudo -H systemctl enable httpd + sudo -H systemctl start http + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H dnf install httpd + sudo -H systemctl enable httpd + sudo -H systemctl start httpd + +Now at http://localhost you should see some kind of *Welcome* or *Test* page. +How this default site is configured, depends on the linux distribution +(compare `Apache directives`_). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + less /etc/apache2/sites-enabled/000-default.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot /var/www/html + + And the *welcome* page is the HTML file at ``/var/www/html/index.html``. + + .. group-tab:: Arch Linux + + .. code:: bash + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot "/srv/http" + <Directory "/srv/http"> + Options Indexes FollowSymLinks + AllowOverride None + Require all granted + </Directory> + + The *welcome* page of Arch Linux is a page showing the directory located + at ``DocumentRoot``. This *directory* page is generated by the Module + `mod_autoindex <https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html>`_: + + .. code:: apache + + LoadModule autoindex_module modules/mod_autoindex.so + ... + Include conf/extra/httpd-autoindex.conf + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the ``DocumentRoot`` directive: + + .. code:: apache + + DocumentRoot "/var/www/html" + ... + <Directory "/var/www"> + AllowOverride None + # Allow open access: + Require all granted + </Directory> + + On fresh installations, the ``/var/www`` is empty and the *default + welcome page* is shown, the configuration is located at:: + + less /etc/httpd/conf.d/welcome.conf + + +.. _Debian's Apache layout: + +Debian's Apache layout +---------------------- + +Be aware, Debian's Apache layout is quite different from the standard Apache +configuration. For details look at the apache2.README.Debian_ +(``/usr/share/doc/apache2/README.Debian.gz``). Some commands you should know on +Debian: + +* :man:`apache2ctl`: Apache HTTP server control interface +* :man:`a2enmod`, :man:`a2dismod`: switch on/off modules +* :man:`a2enconf`, :man:`a2disconf`: switch on/off configurations +* :man:`a2ensite`, :man:`a2dissite`: switch on/off sites + +.. _apache modules: + +Apache modules +-------------- + +To load additional modules, in most distributions you have to uncomment the +lines with the corresponding LoadModule_ directive, except in :ref:`Debian's +Apache layout`. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + :ref:`Debian's Apache layout` uses :man:`a2enmod` and :man:`a2dismod` to + activate or disable modules: + + .. code:: bash + + sudo -H a2enmod ssl + sudo -H a2enmod headers + sudo -H a2enmod proxy + sudo -H a2enmod proxy_http + sudo -H a2enmod proxy_uwsgi + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_ + directives: + + .. code:: apache + + LoadModule ssl_module modules/mod_ssl.so + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_ + directives: + + .. code:: apache + + LoadModule ssl_module modules/mod_ssl.so + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + +.. _apache sites: + +Apache sites +------------ + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + In :ref:`Debian's Apache layout` you create a ``searxng.conf`` with the + ``<Location /searxng >`` directive and save this file in the *sites + available* folder at ``/etc/apache2/sites-available``. To enable the + ``searxng.conf`` use :man:`a2ensite`: + + .. code:: bash + + sudo -H a2ensite searxng.conf + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_ + directive: + + .. code:: apache + + IncludeOptional sites-enabled/*.conf + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/httpd/sites-available + mkdir -p /etc/httpd/sites-enabled + + Create configuration at ``/etc/httpd/sites-available`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/httpd/sites-available/searxng.conf \ + /etc/httpd/sites-enabled/searxng.conf + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_ + directive: + + .. code:: apache + + IncludeOptional sites-enabled/*.conf + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/httpd/sites-available + mkdir -p /etc/httpd/sites-enabled + + Create configuration at ``/etc/httpd/sites-available`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/httpd/sites-available/searxng.conf \ + /etc/httpd/sites-enabled/searxng.conf + + +.. _apache searxng site: + +Apache's SearXNG site +===================== + +.. _mod_uwsgi: https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-uwsgi + +.. sidebar:: uWSGI + + Use mod_proxy_uwsgi_ / don't use the old mod_uwsgi_ anymore. + +To proxy the incoming requests to the SearXNG instance Apache needs the +mod_proxy_ module (:ref:`apache modules`). + +.. sidebar:: HTTP headers + + With ProxyPreserveHost_ the incoming ``Host`` header is passed to the proxied + host. + +Depending on what your SearXNG installation is listening on, you need a http +mod_proxy_http_) or socket (mod_proxy_uwsgi_) communication to upstream. + +The :ref:`installation scripts` installs the :ref:`reference setup +<use_default_settings.yml>` and a :ref:`uwsgi setup` that listens on a socket by default. +You can install and activate your own ``searxng.conf`` like shown in +:ref:`apache sites`. + +.. tabs:: + + .. group-tab:: socket + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START apache socket + :end-before: END apache socket + + .. group-tab:: http + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START apache http + :end-before: END apache http + +.. _restart apache: + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H systemctl restart apache2 + sudo -H service uwsgi restart searxng + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H systemctl restart httpd + sudo -H systemctl restart uwsgi@searxng + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H systemctl restart httpd + sudo -H touch /etc/uwsgi.d/searxng.ini + + +disable logs +============ + +For better privacy you can disable Apache logs. In the examples above activate +one of the lines and `restart apache`_: + +.. code:: apache + + SetEnvIf Request_URI "/searxng" dontlog + # CustomLog /dev/null combined env=dontlog + +The ``CustomLog`` directive disables logs for the entire (virtual) server, use it +when the URL of the service does not have a path component (``/searxng``), so when +SearXNG is located at root (``/``). diff --git a/_sources/admin/installation-docker.rst.txt b/_sources/admin/installation-docker.rst.txt new file mode 100644 index 000000000..09471891b --- /dev/null +++ b/_sources/admin/installation-docker.rst.txt @@ -0,0 +1,197 @@ +.. _installation docker: + +================ +Docker Container +================ + +.. _ENTRYPOINT: https://docs.docker.com/engine/reference/builder/#entrypoint +.. _searxng/searxng @dockerhub: https://hub.docker.com/r/searxng/searxng +.. _searxng-docker: https://github.com/searxng/searxng-docker +.. _[caddy]: https://hub.docker.com/_/caddy +.. _Redis: https://redis.io/ + +---- + +.. sidebar:: info + + - `searxng/searxng @dockerhub`_ + - :origin:`Dockerfile` + - `Docker overview <https://docs.docker.com/get-started/overview>`_ + - `Docker Cheat Sheet <https://docs.docker.com/get-started/docker_cheatsheet.pdf>`_ + - `Alpine Linux <https://alpinelinux.org>`_ + `(wiki) <https://en.wikipedia.org/wiki/Alpine_Linux>`__ + `apt packages <https://pkgs.alpinelinux.org/packages>`_ + - Alpine's ``/bin/sh`` is :man:`dash` + +**If you intend to create a public instance using Docker, use our well maintained +docker container** + +- `searxng/searxng @dockerhub`_. + +.. sidebar:: hint + + The rest of this article is of interest only to those who want to create and + maintain their own Docker images. + +The sources are hosted at searxng-docker_ and the container includes: + +- a HTTPS reverse proxy `[caddy]`_ and +- a Redis_ DB + +The `default SearXNG setup <https://github.com/searxng/searxng-docker/blob/master/searxng/settings.yml>`_ +of this container: + +- enables :ref:`limiter <limiter>` to protect against bots +- enables :ref:`image proxy <image_proxy>` for better privacy +- enables :ref:`cache busting <static_use_hash>` to save bandwidth + +---- + + +Get Docker +========== + +If you plan to build and maintain a docker image by yourself, make sure you have +`Docker installed <https://docs.docker.com/get-docker/>`_. On Linux don't +forget to add your user to the docker group (log out and log back in so that +your group membership is re-evaluated): + +.. code:: sh + + $ sudo usermod -a -G docker $USER + + +searxng/searxng +=============== + +.. sidebar:: ``docker run`` + + - `-\-rm <https://docs.docker.com/engine/reference/run/#clean-up---rm>`__ + automatically clean up when container exits + - `-d <https://docs.docker.com/engine/reference/run/#detached--d>`__ start + detached container + - `-v <https://docs.docker.com/engine/reference/run/#volume-shared-filesystems>`__ + mount volume ``HOST:CONTAINER`` + +The docker image is based on :origin:`Dockerfile` and available from +`searxng/searxng @dockerhub`_. Using the docker image is quite easy, for +instance you can pull the `searxng/searxng @dockerhub`_ image and deploy a local +instance using `docker run <https://docs.docker.com/engine/reference/run/>`_: + +.. code:: sh + + $ mkdir my-instance + $ cd my-instance + $ export PORT=8080 + $ docker pull searxng/searxng + $ docker run --rm \ + -d -p ${PORT}:8080 \ + -v "${PWD}/searxng:/etc/searxng" \ + -e "BASE_URL=http://localhost:$PORT/" \ + -e "INSTANCE_NAME=my-instance" \ + searxng/searxng + 2f998.... # container's ID + +The environment variables UWSGI_WORKERS and UWSGI_THREADS overwrite the default +number of UWSGI processes and UWSGI threads specified in `/etc/searxng/uwsgi.ini`. + +Open your WEB browser and visit the URL: + +.. code:: sh + + $ xdg-open "http://localhost:$PORT" + +Inside ``${PWD}/searxng``, you will find ``settings.yml`` and ``uwsgi.ini``. You +can modify these files according to your needs and restart the Docker image. + +.. code:: sh + + $ docker container restart 2f998 + +Use command ``container ls`` to list running containers, add flag `-a +<https://docs.docker.com/engine/reference/commandline/container_ls>`__ to list +exited containers also. With ``container stop`` a running container can be +stopped. To get rid of a container use ``container rm``: + +.. code:: sh + + $ docker container ls + CONTAINER ID IMAGE COMMAND CREATED ... + 2f998d725993 searxng/searxng "/sbin/tini -- /usr/…" 7 minutes ago ... + + $ docker container stop 2f998 + $ docker container rm 2f998 + +.. sidebar:: Warning + + This might remove all docker items, not only those from SearXNG. + +If you won't use docker anymore and want to get rid of all containers & images +use the following *prune* command: + +.. code:: sh + + $ docker stop $(docker ps -aq) # stop all containers + $ docker system prune # make some housekeeping + $ docker rmi -f $(docker images -q) # drop all images + + +shell inside container +---------------------- + +.. sidebar:: Bashism + + - `A tale of two shells: bash or dash <https://lwn.net/Articles/343924/>`_ + - `How to make bash scripts work in dash <http://mywiki.wooledge.org/Bashism>`_ + - `Checking for Bashisms <https://dev.to/bowmanjd/writing-bash-scripts-that-are-not-only-bash-checking-for-bashisms-and-testing-with-dash-1bli>`_ + +Like in many other distributions, Alpine's `/bin/sh +<https://wiki.ubuntu.com/DashAsBinSh>`__ is :man:`dash`. Dash is meant to be +`POSIX-compliant <https://pubs.opengroup.org/onlinepubs/9699919799>`__. +Compared to debian, in the Alpine image :man:`bash` is not installed. The +:origin:`dockerfiles/docker-entrypoint.sh` script is checked *against dash* +(``make tests.shell``). + +To open a shell inside the container: + +.. code:: sh + + $ docker exec -it 2f998 sh + + +Build the image +=============== + +It's also possible to build SearXNG from the embedded :origin:`Dockerfile`:: + + $ git clone https://github.com/searxng/searxng.git + $ cd searxng + $ make docker.build + ... + Successfully built 49586c016434 + Successfully tagged searxng/searxng:latest + Successfully tagged searxng/searxng:1.0.0-209-9c823800-dirty + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + searxng/searxng 1.0.0-209-9c823800-dirty 49586c016434 13 minutes ago 308MB + searxng/searxng latest 49586c016434 13 minutes ago 308MB + alpine 3.13 6dbb9cc54074 3 weeks ago 5.61MB + + +Command line +============ + +.. sidebar:: docker run + + Use flags ``-it`` for `interactive processes + <https://docs.docker.com/engine/reference/run/#foreground>`__. + +In the :origin:`Dockerfile` the ENTRYPOINT_ is defined as +:origin:`dockerfiles/docker-entrypoint.sh` + +.. code:: sh + + docker run --rm -it searxng/searxng -h + +.. program-output:: ../dockerfiles/docker-entrypoint.sh -h diff --git a/_sources/admin/installation-nginx.rst.txt b/_sources/admin/installation-nginx.rst.txt new file mode 100644 index 000000000..f95354b53 --- /dev/null +++ b/_sources/admin/installation-nginx.rst.txt @@ -0,0 +1,252 @@ +.. _installation nginx: + +===== +NGINX +===== + +.. _nginx: + https://docs.nginx.com/nginx/admin-guide/ +.. _nginx server configuration: + https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers +.. _nginx beginners guide: + https://nginx.org/en/docs/beginners_guide.html +.. _Getting Started wiki: + https://www.nginx.com/resources/wiki/start/ +.. _uWSGI support from nginx: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html +.. _uwsgi_params: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#configuring-nginx +.. _SCRIPT_NAME: + https://werkzeug.palletsprojects.com/en/1.0.x/wsgi/#werkzeug.wsgi.get_script_name + +This section explains how to set up a SearXNG instance using the HTTP server nginx_. +If you have used the :ref:`installation scripts` and do not have any special preferences +you can install the :ref:`SearXNG site <nginx searxng site>` using +:ref:`searxng.sh <searxng.sh overview>`: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install nginx + +If you have special interests or problems with setting up nginx, the following +section might give you some guidance. + + +.. sidebar:: further reading + + - nginx_ + - `nginx beginners guide`_ + - `nginx server configuration`_ + - `Getting Started wiki`_ + - `uWSGI support from nginx`_ + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +The nginx HTTP server +===================== + +If nginx_ is not installed, install it now. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H apt-get install nginx + + .. group-tab:: Arch Linux + + .. code-block:: sh + + sudo -H pacman -S nginx-mainline + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + sudo -H dnf install nginx + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + +Now at http://localhost you should see a *Welcome to nginx!* page, on Fedora you +see a *Fedora Webserver - Test Page*. The test page comes from the default +`nginx server configuration`_. How this default site is configured, +depends on the linux distribution: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + less /etc/nginx/nginx.conf + + There is one line that includes site configurations from: + + .. code:: nginx + + include /etc/nginx/sites-enabled/*; + + .. group-tab:: Arch Linux + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + There is a configuration section named ``server``: + + .. code-block:: nginx + + server { + listen 80; + server_name localhost; + # ... + } + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + There is one line that includes site configurations from: + + .. code:: nginx + + include /etc/nginx/conf.d/*.conf; + + +.. _nginx searxng site: + +NGINX's SearXNG site +==================== + +Now you have to create a configuration file (``searxng.conf``) for the SearXNG +site. If nginx_ is new to you, the `nginx beginners guide`_ is a good starting +point and the `Getting Started wiki`_ is always a good resource *to keep in the +pocket*. + +Depending on what your SearXNG installation is listening on, you need a http or socket +communication to upstream. + +.. tabs:: + + .. group-tab:: socket + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START nginx socket + :end-before: END nginx socket + + .. group-tab:: http + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START nginx http + :end-before: END nginx http + +The :ref:`installation scripts` installs the :ref:`reference setup +<use_default_settings.yml>` and a :ref:`uwsgi setup` that listens on a socket by default. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + Create configuration at ``/etc/nginx/sites-available/`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/sites-available/searxng.conf \ + /etc/nginx/sites-enabled/searxng.conf + + .. group-tab:: Arch Linux + + In the ``/etc/nginx/nginx.conf`` file, in the ``server`` section add a + `include <https://nginx.org/en/docs/ngx_core_module.html#include>`_ + directive: + + .. code:: nginx + + server { + # ... + include /etc/nginx/default.d/*.conf; + # ... + } + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/nginx/default.d + mkdir -p /etc/nginx/default.apps-available + + Create configuration at ``/etc/nginx/default.apps-available`` and place a + symlink to ``default.d``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \ + /etc/nginx/default.d/searxng.conf + + .. group-tab:: Fedora / RHEL + + Create a folder for the *available sites*: + + .. code:: bash + + mkdir -p /etc/nginx/default.apps-available + + Create configuration at ``/etc/nginx/default.apps-available`` and place a + symlink to ``conf.d``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \ + /etc/nginx/conf.d/searxng.conf + +Restart services: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H service uwsgi restart searxng + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H systemctl restart uwsgi@searxng + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H touch /etc/uwsgi.d/searxng.ini + + +Disable logs +============ + +For better privacy you can disable nginx logs in ``/etc/nginx/nginx.conf``. + +.. code:: nginx + + http { + # ... + access_log /dev/null; + error_log /dev/null; + # ... + } diff --git a/_sources/admin/installation-scripts.rst.txt b/_sources/admin/installation-scripts.rst.txt new file mode 100644 index 000000000..2d43f5e37 --- /dev/null +++ b/_sources/admin/installation-scripts.rst.txt @@ -0,0 +1,62 @@ +.. _installation scripts: + +=================== +Installation Script +=================== + +.. sidebar:: Update the OS first! + + To avoid unwanted side effects, update your OS before installing SearXNG. + +The following will install a setup as shown in :ref:`the reference architecture +<arch public>`. First you need to get a clone of the repository. The clone is only needed for +the installation procedure and some maintenance tasks. + +.. sidebar:: further read + + - :ref:`toolboxing` + +Jump to a folder that is readable by *others* and start to clone SearXNG, +alternatively you can create your own fork and clone from there. + +.. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: further read + + - :ref:`inspect searxng` + +To install a SearXNG :ref:`reference setup <use_default_settings.yml>` +including a :ref:`uWSGI setup <architecture uWSGI>` as described in the +:ref:`installation basic` and in the :ref:`searxng uwsgi` section type: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install all + +.. attention:: + + For the installation procedure, use a *sudoer* login to run the scripts. If + you install from ``root``, take into account that the scripts are creating a + ``searxng`` user. In the installation procedure this new created user does + need to have read access to the cloned SearXNG repository, which is not the case if you clone + it into a folder below ``/root``! + +.. sidebar:: further read + + - :ref:`update searxng` + +.. _caddy: https://hub.docker.com/_/caddy + +When all services are installed and running fine, you can add SearXNG to your +HTTP server. We do not have any preferences regarding the HTTP server, you can use +whatever you prefer. + +We use caddy in our :ref:`docker image <installation docker>` and we have +implemented installation procedures for: + +- :ref:`installation nginx` +- :ref:`installation apache` diff --git a/_sources/admin/installation-searxng.rst.txt b/_sources/admin/installation-searxng.rst.txt new file mode 100644 index 000000000..446f45cd2 --- /dev/null +++ b/_sources/admin/installation-searxng.rst.txt @@ -0,0 +1,132 @@ +.. _installation basic: + +========================= +Step by step installation +========================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +In this section we show the setup of a SearXNG instance that will be installed +by the :ref:`installation scripts`. + +.. _install packages: + +Install packages +================ + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START distro-packages + :end-before: END distro-packages + +.. hint:: + + This installs also the packages needed by :ref:`searxng uwsgi` + +.. _create searxng user: + +Create user +=========== + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START create user + :end-before: END create user + +.. _searxng-src: + +Install SearXNG & dependencies +============================== + +Start a interactive shell from new created user and clone SearXNG: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START clone searxng + :end-before: END clone searxng + +In the same shell create *virtualenv*: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START create virtualenv + :end-before: END create virtualenv + +To install SearXNG's dependencies, exit the SearXNG *bash* session you opened above +and start a new one. Before installing, check if your *virtualenv* was sourced +from the login (*~/.profile*): + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START manage.sh update_packages + :end-before: END manage.sh update_packages + +.. tip:: + + Open a second terminal for the configuration tasks and leave the ``(searx)$`` + terminal open for the tasks below. + + +.. _use_default_settings.yml: + +Configuration +============= + +.. sidebar:: ``use_default_settings: True`` + + - :ref:`settings.yml` + - :ref:`settings location` + - :ref:`settings use_default_settings` + - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` + +To create a initial ``/etc/searxng/settings.yml`` we recommend to start with a +copy of the file :origin:`utils/templates/etc/searxng/settings.yml`. This setup +:ref:`use default settings <settings use_default_settings>` from +:origin:`searx/settings.yml` and is shown in the tab *"Use default settings"* +below. This setup: + +- enables :ref:`limiter <limiter>` to protect against bots +- enables :ref:`image proxy <image_proxy>` for better privacy +- enables :ref:`cache busting <static_use_hash>` to save bandwidth + +Modify the ``/etc/searxng/settings.yml`` to your needs: + +.. tabs:: + + .. group-tab:: Use default settings + + .. literalinclude:: ../../utils/templates/etc/searxng/settings.yml + :language: yaml + :end-before: # hostnames: + + To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml` + + .. group-tab:: searx/settings.yml + + .. literalinclude:: ../../searx/settings.yml + :language: yaml + :end-before: # hostnames: + + To see the entire file jump to :origin:`searx/settings.yml` + +For a *minimal setup* you need to set ``server:secret_key``. + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng config + :end-before: END searxng config + + +Check +===== + +To check your SearXNG setup, optional enable debugging and start the *webapp*. +SearXNG looks at the exported environment ``$SEARXNG_SETTINGS_PATH`` for a +configuration file. + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START check searxng installation + :end-before: END check searxng installation + +If everything works fine, hit ``[CTRL-C]`` to stop the *webapp* and disable the +debug option in ``settings.yml``. You can now exit SearXNG user bash session (enter exit +command twice). At this point SearXNG is not demonized; uwsgi allows this. + diff --git a/_sources/admin/installation-uwsgi.rst.txt b/_sources/admin/installation-uwsgi.rst.txt new file mode 100644 index 000000000..78da22f45 --- /dev/null +++ b/_sources/admin/installation-uwsgi.rst.txt @@ -0,0 +1,268 @@ +.. _searxng uwsgi: + +===== +uWSGI +===== + +.. sidebar:: further reading + + - `systemd.unit`_ + - `uWSGI Emperor`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _systemd.unit: https://www.freedesktop.org/software/systemd/man/systemd.unit.html +.. _One service per app in systemd: + https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd +.. _uWSGI Emperor: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html +.. _uwsgi ini file: + https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files +.. _systemd unit template: + http://0pointer.de/blog/projects/instances.html + + +Origin uWSGI +============ + +.. _Tyrant mode: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html#tyrant-mode-secure-multi-user-hosting + +How uWSGI is implemented by distributors varies. The uWSGI project itself +recommends two methods: + +1. `systemd.unit`_ template file as described here `One service per app in systemd`_: + + There is one `systemd unit template`_ on the system installed and one `uwsgi + ini file`_ per uWSGI-app placed at dedicated locations. Take archlinux and a + ``searxng.ini`` as example:: + + systemd template unit: /usr/lib/systemd/system/uwsgi@.service + contains: [Service] + ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/%I.ini + + SearXNG application: /etc/uwsgi/searxng.ini + links to: /etc/uwsgi/apps-available/searxng.ini + + The SearXNG app (template ``/etc/uwsgi/%I.ini``) can be maintained as known + from common systemd units: + + .. code:: sh + + $ systemctl enable uwsgi@searxng + $ systemctl start uwsgi@searxng + $ systemctl restart uwsgi@searxng + $ systemctl stop uwsgi@searxng + +2. The `uWSGI Emperor`_ which fits for maintaining a large range of uwsgi + apps and there is a `Tyrant mode`_ to secure multi-user hosting. + + The Emperor mode is a special uWSGI instance that will monitor specific + events. The Emperor mode (the service) is started by a (common, not template) + systemd unit. + + The Emperor service will scan specific directories for `uwsgi ini file`_\s + (also know as *vassals*). If a *vassal* is added, removed or the timestamp is + modified, a corresponding action takes place: a new uWSGI instance is started, + reload or stopped. Take Fedora and a ``searxng.ini`` as example:: + + to install & start SearXNG instance create --> /etc/uwsgi.d/searxng.ini + to reload the instance edit timestamp --> touch /etc/uwsgi.d/searxng.ini + to stop instance remove ini --> rm /etc/uwsgi.d/searxng.ini + + +Distributors +============ + +The `uWSGI Emperor`_ mode and `systemd unit template`_ is what the distributors +mostly offer their users, even if they differ in the way they implement both +modes and their defaults. Another point they might differ in is the packaging of +plugins (if so, compare :ref:`install packages`) and what the default python +interpreter is (python2 vs. python3). + +While archlinux does not start a uWSGI service by default, Fedora (RHEL) starts +a Emperor in `Tyrant mode`_ by default (you should have read :ref:`uWSGI Tyrant +mode pitfalls`). Worth to know; debian (ubuntu) follow a complete different +approach, read see :ref:`Debian's uWSGI layout`. + +.. _Debian's uWSGI layout: + +Debian's uWSGI layout +--------------------- + +.. _uwsgi.README.Debian: + https://salsa.debian.org/uwsgi-team/uwsgi/-/raw/debian/latest/debian/uwsgi.README.Debian + +Be aware, Debian's uWSGI layout is quite different from the standard uWSGI +configuration. Your are familiar with :ref:`Debian's Apache layout`? .. they do a +similar thing for the uWSGI infrastructure. The folders are:: + + /etc/uwsgi/apps-available/ + /etc/uwsgi/apps-enabled/ + +The `uwsgi ini file`_ is enabled by a symbolic link:: + + ln -s /etc/uwsgi/apps-available/searxng.ini /etc/uwsgi/apps-enabled/ + +More details can be found in the uwsgi.README.Debian_ +(``/usr/share/doc/uwsgi/README.Debian.gz``). Some commands you should know on +Debian: + +.. code:: none + + Commands recognized by init.d script + ==================================== + + You can issue to init.d script following commands: + * start | starts daemon + * stop | stops daemon + * reload | sends to daemon SIGHUP signal + * force-reload | sends to daemon SIGTERM signal + * restart | issues 'stop', then 'start' commands + * status | shows status of daemon instance (running/not running) + + 'status' command must be issued with exactly one argument: '<confname>'. + + Controlling specific instances of uWSGI + ======================================= + + You could control specific instance(s) by issuing: + + SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi <command> <confname> <confname>... + + where: + * <command> is one of 'start', 'stop' etc. + * <confname> is the name of configuration file (without extension) + + For example, this is how instance for /etc/uwsgi/apps-enabled/hello.xml is + started: + + SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi start hello + + +.. _uWSGI maintenance: + +uWSGI maintenance +================= + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description ubuntu-20.04 + :end-before: END searxng uwsgi-description ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description arch + :end-before: END searxng uwsgi-description arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description fedora + :end-before: END searxng uwsgi-description fedora + + +.. _uwsgi setup: + +uWSGI setup +=========== + +Create the configuration ini-file according to your distribution and restart the +uwsgi application. As shown below, the :ref:`installation scripts` installs by +default: + +- a uWSGI setup that listens on a socket and +- enables :ref:`cache busting <static_use_hash>`. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini ubuntu-20.04 + :end-before: END searxng uwsgi-appini ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini arch + :end-before: END searxng uwsgi-appini arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini fedora + :end-before: END searxng uwsgi-appini fedora + + +.. _uWSGI Tyrant mode pitfalls: + +Pitfalls of the Tyrant mode +=========================== + +The implementation of the process owners and groups in the `Tyrant mode`_ is +somewhat unusual and requires special consideration. In `Tyrant mode`_ mode the +Emperor will run the vassal using the UID/GID of the vassal configuration file +(user and group of the app ``.ini`` file). + +.. _#2099@uWSGI: https://github.com/unbit/uwsgi/issues/2099 +.. _#752@uWSGI: https://github.com/unbit/uwsgi/pull/752 +.. _#2425uWSGI: https://github.com/unbit/uwsgi/issues/2425 + +Without option ``emperor-tyrant-initgroups=true`` in ``/etc/uwsgi.ini`` the +process won't get the additional groups, but this option is not available in +2.0.x branch (see `#2099@uWSGI`_) the feature `#752@uWSGI`_ has been merged (on +Oct. 2014) to the master branch of uWSGI but had never been released; the last +major release is from Dec. 2013, since the there had been only bugfix releases +(see `#2425uWSGI`_). To shorten up: + + **In Tyrant mode, there is no way to get additional groups, and the uWSGI + process misses additional permissions that may be needed.** + +For example on Fedora (RHEL): If you try to install a redis DB with socket +communication and you want to connect to it from the SearXNG uWSGI, you will see a +*Permission denied* in the log of your instance:: + + ERROR:searx.redisdb: [searxng (993)] can't connect redis DB ... + ERROR:searx.redisdb: Error 13 connecting to unix socket: /usr/local/searxng-redis/run/redis.sock. Permission denied. + ERROR:searx.plugins.limiter: init limiter DB failed!!! + +Even if your *searxng* user of the uWSGI process is added to additional groups +to give access to the socket from the redis DB:: + + $ groups searxng + searxng : searxng searxng-redis + +To see the effective groups of the uwsgi process, you have to look at the status +of the process, by example:: + + $ ps -aef | grep '/usr/sbin/uwsgi --ini searxng.ini' + searxng 93 92 0 12:43 ? 00:00:00 /usr/sbin/uwsgi --ini searxng.ini + searxng 186 93 0 12:44 ? 00:00:01 /usr/sbin/uwsgi --ini searxng.ini + +Here you can see that the additional "Groups" of PID 186 are unset (missing gid +of ``searxng-redis``):: + + $ cat /proc/186/task/186/status + ... + Uid: 993 993 993 993 + Gid: 993 993 993 993 + FDSize: 128 + Groups: + ... diff --git a/_sources/admin/installation.rst.txt b/_sources/admin/installation.rst.txt new file mode 100644 index 000000000..54d901f76 --- /dev/null +++ b/_sources/admin/installation.rst.txt @@ -0,0 +1,22 @@ +.. _installation: + +============ +Installation +============ + +*You're spoilt for choice*, choose your preferred method of installation. + +- :ref:`installation docker` +- :ref:`installation scripts` +- :ref:`installation basic` + +The :ref:`installation basic` is an excellent illustration of *how a SearXNG +instance is build up* (see :ref:`architecture uWSGI`). If you do not have any +special preferences, it's recommended to use the :ref:`installation docker` or the +:ref:`installation scripts`. + +.. attention:: + + SearXNG is growing rapidly, you should regularly read our :ref:`migrate and + stay tuned` section. If you want to upgrade an existing instance or migrate + from searx to SearXNG, you should read this section first! diff --git a/_sources/admin/plugins.rst.txt b/_sources/admin/plugins.rst.txt new file mode 100644 index 000000000..d97b3dada --- /dev/null +++ b/_sources/admin/plugins.rst.txt @@ -0,0 +1,39 @@ +.. _plugins generic: + +=============== +Plugins builtin +=============== + +.. sidebar:: Further reading .. + + - :ref:`dev plugin` + +Configuration defaults (at built time): + +:DO: Default on + +.. _configured plugins: + +.. jinja:: searx + + .. flat-table:: Plugins configured at built time (defaults) + :header-rows: 1 + :stub-columns: 1 + :widths: 3 1 9 + + * - Name + - DO + - Description + + JS & CSS dependencies + + {% for plgin in plugins %} + + * - {{plgin.name}} + - {{(plgin.default_on and "y") or ""}} + - {{plgin.description}} + + {% for dep in (plgin.js_dependencies + plgin.css_dependencies) %} + | ``{{dep}}`` {% endfor %} + + {% endfor %} diff --git a/_sources/admin/searx.favicons.rst.txt b/_sources/admin/searx.favicons.rst.txt new file mode 100644 index 000000000..b05b3458b --- /dev/null +++ b/_sources/admin/searx.favicons.rst.txt @@ -0,0 +1,251 @@ +.. _favicons: + +======== +Favicons +======== + +.. sidebar:: warning + + Don't activate the favicons before reading the documentation. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Activating the favicons in SearXNG is very easy, but this **generates a +significantly higher load** in the client/server communication and increases +resources needed on the server. + +To mitigate these disadvantages, various methods have been implemented, +including a *cache*. The cache must be parameterized according to your own +requirements and maintained regularly. + +To activate favicons in SearXNG's result list, set a default +``favicon_resolver`` in the :ref:`search <settings search>` settings: + +.. code:: yaml + + search: + favicon_resolver: "duckduckgo" + +By default and without any extensions, SearXNG serves these resolvers: + +- ``duckduckgo`` +- ``allesedv`` +- ``google`` +- ``yandex`` + +With the above setting favicons are displayed, the user has the option to +deactivate this feature in his settings. If the user is to have the option of +selecting from several *resolvers*, a further setting is required / but this +setting will be discussed :ref:`later <register resolvers>` in this article, +first we have to setup the favicons cache. + +Infrastructure +============== + +The infrastructure for providing the favicons essentially consists of three +parts: + +- :py:obj:`Favicons-Proxy <.favicons.proxy>` (aka *proxy*) +- :py:obj:`Favicons-Resolvers <.favicons.resolvers>` (aka *resolver*) +- :py:obj:`Favicons-Cache <.favicons.cache>` (aka *cache*) + +To protect the privacy of users, the favicons are provided via a *proxy*. This +*proxy* is automatically activated with the above activation of a *resolver*. +Additional requests are required to provide the favicons: firstly, the *proxy* +must process the incoming requests and secondly, the *resolver* must make +outgoing requests to obtain the favicons from external sources. + +A *cache* has been developed to massively reduce both, incoming and outgoing +requests. This *cache* is also activated automatically with the above +activation of a *resolver*. In its defaults, however, the *cache* is minimal +and not well suitable for a production environment! + +.. _favicon cache setup: + +Setting up the cache +==================== + +To parameterize the *cache* and more settings of the favicons infrastructure, a +TOML_ configuration is created in the file ``/etc/searxng/favicons.toml``. + +.. code:: toml + + [favicons] + + cfg_schema = 1 # config's schema version no. + + [favicons.cache] + + db_url = "/var/cache/searxng/faviconcache.db" # default: "/tmp/faviconcache.db" + LIMIT_TOTAL_BYTES = 2147483648 # 2 GB / default: 50 MB + # HOLD_TIME = 5184000 # 60 days / default: 30 days + # BLOB_MAX_BYTES = 40960 # 40 KB / default 20 KB + # MAINTENANCE_MODE = "off" # default: "auto" + # MAINTENANCE_PERIOD = 600 # 10min / default: 1h + +:py:obj:`cfg_schema <.FaviconConfig.cfg_schema>`: + Is required to trigger any processes required for future upgrades / don't + change it. + +:py:obj:`cache.db_url <.FaviconCacheConfig.db_url>`: + The path to the (SQLite_) database file. The default path is in the `/tmp`_ + folder, which is deleted on every reboot and is therefore unsuitable for a + production environment. The FHS_ provides the folder for the + application cache + + The FHS_ provides the folder `/var/cache`_ for the cache of applications, so a + suitable storage location of SearXNG's caches is folder ``/var/cache/searxng``. + In container systems, a volume should be mounted for this folder and in a + standard installation (compare :ref:`create searxng user`), the folder must be + created and the user under which the SearXNG process is running must be given + write permission to this folder. + + .. code:: bash + + $ sudo mkdir /var/cache/searxng + $ sudo chown root:searxng /var/cache/searxng/ + $ sudo chmod g+w /var/cache/searxng/ + +:py:obj:`cache.LIMIT_TOTAL_BYTES <.FaviconCacheConfig.LIMIT_TOTAL_BYTES>`: + Maximum of bytes stored in the cache of all blobs. The limit is only reached + at each maintenance interval after which the oldest BLOBs are deleted; the + limit is exceeded during the maintenance period. + + .. attention:: + + If the maintenance period is too long or maintenance is switched + off completely, the cache grows uncontrollably. + +SearXNG hosters can change other parameters of the cache as required: + +- :py:obj:`cache.HOLD_TIME <.FaviconCacheConfig.HOLD_TIME>` +- :py:obj:`cache.BLOB_MAX_BYTES <.FaviconCacheConfig.BLOB_MAX_BYTES>` + + +Maintenance of the cache +------------------------ + +Regular maintenance of the cache is required! By default, regular maintenance +is triggered automatically as part of the client requests: + +- :py:obj:`cache.MAINTENANCE_MODE <.FaviconCacheConfig.MAINTENANCE_MODE>` (default ``auto``) +- :py:obj:`cache.MAINTENANCE_PERIOD <.FaviconCacheConfig.MAINTENANCE_PERIOD>` (default ``6000`` / 1h) + +As an alternative to maintenance as part of the client request process, it is +also possible to carry out maintenance using an external process. For example, +by creating a :man:`crontab` entry for maintenance: + +.. code:: bash + + $ python -m searx.favicons cache maintenance + +The following command can be used to display the state of the cache: + +.. code:: bash + + $ python -m searx.favicons cache state + + +.. _favicon proxy setup: + +Proxy configuration +=================== + +Most of the options of the :py:obj:`Favicons-Proxy <.favicons.proxy>` are +already set sensibly with settings from the :ref:`settings.yml <searxng +settings.yml>` and should not normally be adjusted. + +.. code:: toml + + [favicons.proxy] + + max_age = 5184000 # 60 days / default: 7 days (604800 sec) + + +:py:obj:`max_age <.FaviconProxyConfig.max_age>`: + The `HTTP Cache-Control max-age`_ response directive indicates that the + response remains fresh until N seconds after the response is generated. This + setting therefore determines how long a favicon remains in the client's cache. + As a rule, in the favicons infrastructure of SearXNG's this setting only + affects favicons whose byte size exceeds :ref:`BLOB_MAX_BYTES <favicon cache + setup>` (the other favicons that are already in the cache are embedded as + `data URL`_ in the :py:obj:`generated HTML <.favicons.proxy.favicon_url>`, + which can greatly reduce the number of additional requests). + +.. _register resolvers: + +Register resolvers +------------------ + +A :py:obj:`resolver <.favicon.resolvers>` is a function that obtains the favicon +from an external source. The resolver functions available to the user are +registered with their fully qualified name (FQN_) in a ``resolver_map``. + +If no ``resolver_map`` is defined in the ``favicon.toml``, the favicon +infrastructure of SearXNG generates this ``resolver_map`` automatically +depending on the ``settings.yml``. SearXNG would automatically generate the +following TOML configuration from the following YAML configuration: + +.. code:: yaml + + search: + favicon_resolver: "duckduckgo" + +.. code:: toml + + [favicons.proxy.resolver_map] + + "duckduckgo" = "searx.favicons.resolvers.duckduckgo" + +If this automatism is not desired, then (and only then) a separate +``resolver_map`` must be created. For example, to give the user two resolvers to +choose from, the following configuration could be used: + +.. code:: toml + + [favicons.proxy.resolver_map] + + "duckduckgo" = "searx.favicons.resolvers.duckduckgo" + "allesedv" = "searx.favicons.resolvers.allesedv" + # "google" = "searx.favicons.resolvers.google" + # "yandex" = "searx.favicons.resolvers.yandex" + +.. note:: + + With each resolver, the resource requirement increases significantly. + +The number of resolvers increases: + +- the number of incoming/outgoing requests and +- the number of favicons to be stored in the cache. + +In the following we list the resolvers available in the core of SearXNG, but via +the FQN_ it is also possible to implement your own resolvers and integrate them +into the *proxy*: + +- :py:obj:`searx.favicons.resolvers.duckduckgo` +- :py:obj:`searx.favicons.resolvers.allesedv` +- :py:obj:`searx.favicons.resolvers.google` +- :py:obj:`searx.favicons.resolvers.yandex` + + + +.. _SQLite: + https://www.sqlite.org/ +.. _FHS: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html +.. _`/var/cache`: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html +.. _`/tmp`: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html +.. _TOML: + https://toml.io/en/ +.. _HTTP Cache-Control max-age: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#response_directives +.. _data URL: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs +.. _FQN: https://en.wikipedia.org/wiki/Fully_qualified_name + diff --git a/_sources/admin/searx.limiter.rst.txt b/_sources/admin/searx.limiter.rst.txt new file mode 100644 index 000000000..c23635571 --- /dev/null +++ b/_sources/admin/searx.limiter.rst.txt @@ -0,0 +1,17 @@ +.. _limiter: + +======= +Limiter +======= + +.. sidebar:: info + + The limiter requires a :ref:`Redis <settings redis>` database. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.limiter + :members: diff --git a/_sources/admin/settings/index.rst.txt b/_sources/admin/settings/index.rst.txt new file mode 100644 index 000000000..acc91dbdd --- /dev/null +++ b/_sources/admin/settings/index.rst.txt @@ -0,0 +1,27 @@ +.. _searxng settings.yml: + +======== +Settings +======== + +.. sidebar:: Further reading .. + + - :ref:`engine settings` + - :ref:`engine file` + +.. toctree:: + :maxdepth: 2 + + settings + settings_engine + settings_brand + settings_general + settings_search + settings_server + settings_ui + settings_redis + settings_outgoing + settings_categories_as_tabs + + + diff --git a/_sources/admin/settings/settings.rst.txt b/_sources/admin/settings/settings.rst.txt new file mode 100644 index 000000000..9c6fb01be --- /dev/null +++ b/_sources/admin/settings/settings.rst.txt @@ -0,0 +1,117 @@ +.. _settings.yml: + +================ +``settings.yml`` +================ + +This page describe the options possibilities of the :origin:`searx/settings.yml` +file. + +.. sidebar:: Further reading .. + + - :ref:`use_default_settings.yml` + - :ref:`search API` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. _settings location: + +settings.yml location +===================== + +The initial ``settings.yml`` we be load from these locations: + +1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable. +2. ``/etc/searxng/settings.yml`` + +If these files don't exist (or are empty or can't be read), SearXNG uses the +:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to +see how you can simplify your *user defined* ``settings.yml``. + + + +.. _settings use_default_settings: + +use_default_settings +==================== + +.. sidebar:: ``use_default_settings: true`` + + - :ref:`settings location` + - :ref:`use_default_settings.yml` + - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` + +The user defined ``settings.yml`` is loaded from the :ref:`settings location` +and can relied on the default configuration :origin:`searx/settings.yml` using: + + ``use_default_settings: true`` + +``server:`` + In the following example, the actual settings are the default settings defined + in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and + the ``bind_address``: + + .. code:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + bind_address: "0.0.0.0" + +``engines:`` + With ``use_default_settings: true``, each settings can be override in a + similar way, the ``engines`` section is merged according to the engine + ``name``. In this example, SearXNG will load all the default engines, will + enable the ``bing`` engine and define a :ref:`token <private engines>` for + the arch linux engine: + + .. code:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + - name: bing + disabled: false + + +``engines:`` / ``remove:`` + It is possible to remove some engines from the default settings. The following + example is similar to the above one, but SearXNG doesn't load the the google + engine: + + .. code:: yaml + + use_default_settings: + engines: + remove: + - google + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``keep_only:`` + As an alternative, it is possible to specify the engines to keep. In the + following example, SearXNG has only two engines: + + .. code:: yaml + + use_default_settings: + engines: + keep_only: + - google + - duckduckgo + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: google + tokens: ['$ecretValue'] + - name: duckduckgo + tokens: ['$ecretValue'] diff --git a/_sources/admin/settings/settings_brand.rst.txt b/_sources/admin/settings/settings_brand.rst.txt new file mode 100644 index 000000000..0f1a0d9a9 --- /dev/null +++ b/_sources/admin/settings/settings_brand.rst.txt @@ -0,0 +1,25 @@ +.. _settings brand: + +========== +``brand:`` +========== + +.. code:: yaml + + brand: + issue_url: https://github.com/searxng/searxng/issues + docs_url: https://docs.searxng.org + public_instances: https://searx.space + wiki_url: https://github.com/searxng/searxng/wiki + +``issue_url`` : + If you host your own issue tracker change this URL. + +``docs_url`` : + If you host your own documentation change this URL. + +``public_instances`` : + If you host your own https://searx.space change this URL. + +``wiki_url`` : + Link to your wiki (or ``false``) diff --git a/_sources/admin/settings/settings_categories_as_tabs.rst.txt b/_sources/admin/settings/settings_categories_as_tabs.rst.txt new file mode 100644 index 000000000..732d04678 --- /dev/null +++ b/_sources/admin/settings/settings_categories_as_tabs.rst.txt @@ -0,0 +1,31 @@ +.. _settings categories_as_tabs: + +======================= +``categories_as_tabs:`` +======================= + +A list of the categories that are displayed as tabs in the user interface. +Categories not listed here can still be searched with the :ref:`search-syntax`. + +.. code:: yaml + + categories_as_tabs: + general: + images: + videos: + news: + map: + music: + it: + science: + files: + social media: + +Engines are added to ``categories:`` (compare :ref:`engine categories`), the +categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If +there are no active engines in a category, the tab is not displayed (e.g. if a +user disables all engines in a category). + +On the preferences page (``/preferences``) -- under *engines* -- there is an +additional tab, called *other*. In this tab are all engines listed that are not +in one of the UI tabs (not included in ``categories_as_tabs``). diff --git a/_sources/admin/settings/settings_engine.rst.txt b/_sources/admin/settings/settings_engine.rst.txt new file mode 100644 index 000000000..454fcd241 --- /dev/null +++ b/_sources/admin/settings/settings_engine.rst.txt @@ -0,0 +1,244 @@ +.. _settings engine: + +=========== +``engine:`` +=========== + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`engines-dev` + +In the code example below a *full fledged* example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional. + +.. hint:: + + A few more options are possible, but they are pretty specific to some + engines (:ref:`engine implementations`). + +.. code:: yaml + + - name: example engine + engine: example + shortcut: demo + base_url: 'https://{language}.example.com/' + send_accept_language_header: false + categories: general + timeout: 3.0 + api_key: 'apikey' + disabled: false + language: en_US + tokens: [ 'my-secret-token' ] + weight: 1 + display_error_messages: true + about: + website: https://example.com + wikidata_id: Q306656 + official_api_documentation: https://example.com/api-doc + use_official_api: true + require_api_key: true + results: HTML + + # overwrite values from section 'outgoing:' + enable_http2: false + retries: 1 + max_connections: 100 + max_keepalive_connections: 10 + keepalive_expiry: 5.0 + using_tor_proxy: false + proxies: + http: + - http://proxy1:8080 + - http://proxy2:8080 + https: + - http://proxy1:8080 + - http://proxy2:8080 + - socks5://user:password@proxy3:1080 + - socks5h://user:password@proxy4:1080 + + # other network settings + enable_http: false + retry_on_http_error: true # or 403 or [404, 429] + + +``name`` : + Name that will be used across SearXNG to define this engine. In settings, on + the result page... + +``engine`` : + Name of the python file used to handle requests and responses to and from this + search engine. + +``shortcut`` : + Code used to execute bang requests (in this case using ``!bi``) + +``base_url`` : optional + Part of the URL that should be stable across every request. Can be useful to + use multiple sites using only one engine, or updating the site URL without + touching at the code. + +``send_accept_language_header`` : + Several engines that support languages (or regions) deal with the HTTP header + ``Accept-Language`` to build a response that fits to the locale. When this + option is activated, the language (locale) that is selected by the user is used + to build and send a ``Accept-Language`` header in the request to the origin + search engine. + +.. _engine categories: + +``categories`` : optional + Specifies to which categories the engine should be added. Engines can be + assigned to multiple categories. + + Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the + UI. A search in a tab (in the UI) will query all engines that are active in + this tab. In the preferences page (``/preferences``) -- under *engines* -- + users can select what engine should be active when querying in this tab. + + Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines + in a category, regardless of whether they are active or not, or whether they + are in a tab of the UI or not. For example, ``!dictionaries`` can be used to + query all search engines in that category (group). + +``timeout`` : optional + Timeout of the search with the current search engine. Overwrites + ``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will + modify the global timeout of SearXNG.** + +``api_key`` : optional + In a few cases, using an API needs the use of a secret key. How to obtain them + is described in the file. + +``disabled`` : optional + To disable by default the engine, but not deleting it. It will allow the user + to manually activate it in the settings. + +``inactive``: optional + Remove the engine from the settings (*disabled & removed*). + +``language`` : optional + If you want to use another language for a specific engine, you can define it + by using the ISO code of language (and region), like ``fr``, ``en-US``, + ``de-DE``. + +``tokens`` : optional + A list of secret tokens to make this engine *private*, more details see + :ref:`private engines`. + +``weight`` : default ``1`` + Weighting of the results of this engine. + +``display_error_messages`` : default ``true`` + When an engine returns an error, the message is displayed on the user interface. + +``network`` : optional + Use the network configuration from another engine. + In addition, there are two default networks: + + - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses) + - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses) + +``enable_http`` : optional + Enable HTTP for this engine (by default only HTTPS is enabled). + +``retry_on_http_error`` : optional + Retry request on some HTTP status code. + + Example: + + * ``true`` : on HTTP status code between 400 and 599. + * ``403`` : on HTTP status code 403. + * ``[403, 429]``: on HTTP status code 403 and 429. + +``proxies`` : + Overwrites proxy settings from :ref:`settings outgoing`. + +``using_tor_proxy`` : + Using tor proxy (``true``) or not (``false``) for this engine. The default is + taken from ``using_tor_proxy`` of the :ref:`settings outgoing`. + +.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration + +``max_keepalive_connection#s`` : + `Pool limit configuration`_, overwrites value ``pool_maxsize`` from + :ref:`settings outgoing` for this engine. + +``max_connections`` : + `Pool limit configuration`_, overwrites value ``pool_connections`` from + :ref:`settings outgoing` for this engine. + +``keepalive_expiry`` : + `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from + :ref:`settings outgoing` for this engine. + + +.. _private engines: + +Private Engines (``tokens``) +============================ + +Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through :ref:`offline engines`. Or they would +rather share engines only with their trusted friends or colleagues. + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +To solve this issue the concept of *private engines* exists. + +A new option was added to engines named `tokens`. It expects a list of strings. +If the user making a request presents one of the tokens of an engine, they can +access information about the engine and make search requests. + +Example configuration to restrict access to the Arch Linux Wiki engine: + +.. code:: yaml + + - name: arch linux wiki + engine: archlinux + shortcut: al + tokens: [ 'my-secret-token' ] + +Unless a user has configured the right token, the engine is going to be hidden +from them. It is not going to be included in the list of engines on the +Preferences page and in the output of `/config` REST API call. + +Tokens can be added to one's configuration on the Preferences page under "Engine +tokens". The input expects a comma separated list of strings. + +The distribution of the tokens from the administrator to the users is not carved +in stone. As providing access to such engines implies that the admin knows and +trusts the user, we do not see necessary to come up with a strict process. +Instead, we would like to add guidelines to the documentation of the feature. + + +Example: Multilingual Search +============================ + +SearXNG does not support true multilingual search. You have to use the language +prefix in your search query when searching in a different language. + +But there is a workaround: By adding a new search engine with a different +language, SearXNG will search in your default and other language. + +Example configuration in settings.yml for a German and English speaker: + +.. code-block:: yaml + + search: + default_lang : "de" + ... + + engines: + - name : google english + engine : google + language : en + ... + +When searching, the default google engine will return German results and +"google english" will return English results. + diff --git a/_sources/admin/settings/settings_general.rst.txt b/_sources/admin/settings/settings_general.rst.txt new file mode 100644 index 000000000..75acb4f6d --- /dev/null +++ b/_sources/admin/settings/settings_general.rst.txt @@ -0,0 +1,42 @@ +.. _settings general: + +============ +``general:`` +============ + +.. code:: yaml + + general: + debug: false + instance_name: "SearXNG" + privacypolicy_url: false + donation_url: false + contact_url: false + enable_metrics: true + open_metrics: '' + +``debug`` : ``$SEARXNG_DEBUG`` + Allow a more detailed log if you run SearXNG directly. Display *detailed* error + messages in the browser too, so this must be deactivated in production. + +``donation_url`` : + Set value to ``true`` to use your own donation page written in the + :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable + the donation link altogether. + +``privacypolicy_url``: + Link to privacy policy. + +``contact_url``: + Contact ``mailto:`` address or WEB form. + +``enable_metrics``: + Enabled by default. Record various anonymous metrics available at ``/stats``, + ``/stats/errors`` and ``/preferences``. + +``open_metrics``: + Disabled by default. Set to a secret password to expose an + `OpenMetrics API <https://github.com/prometheus/OpenMetrics>`_ at ``/metrics``, + e.g. for usage with Prometheus. The ``/metrics`` endpoint is using HTTP Basic Auth, + where the password is the value of ``open_metrics`` set above. The username used for + Basic Auth can be randomly chosen as only the password is being validated. diff --git a/_sources/admin/settings/settings_outgoing.rst.txt b/_sources/admin/settings/settings_outgoing.rst.txt new file mode 100644 index 000000000..7d49ab789 --- /dev/null +++ b/_sources/admin/settings/settings_outgoing.rst.txt @@ -0,0 +1,110 @@ +.. _settings outgoing: + +============= +``outgoing:`` +============= + +Communication with search engines. + +.. code:: yaml + + outgoing: + request_timeout: 2.0 # default timeout in seconds, can be override by engine + max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix: "" # information like an email address to the administrator + pool_connections: 100 # Maximum number of allowable connections, or null + # for no limits. The default is 100. + pool_maxsize: 10 # Number of allowable keep-alive connections, or null + # to always allow. The default is 10. + enable_http2: true # See https://www.python-httpx.org/http2/ + # uncomment below section if you want to use a custom server certificate + # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults + # and https://www.python-httpx.org/compatibility/#ssl-configuration + # verify: ~/.mitmproxy/mitmproxy-ca-cert.cer + # + # uncomment below section if you want to use a proxyq see: SOCKS proxies + # https://2.python-requests.org/en/latest/user/advanced/#proxies + # are also supported: see + # https://2.python-requests.org/en/latest/user/advanced/#socks + # + # proxies: + # all://: + # - http://proxy1:8080 + # - http://proxy2:8080 + # + # using_tor_proxy: true + # + # Extra seconds to add in order to account for the time taken by the proxy + # + # extra_proxy_timeout: 10.0 + # + +``request_timeout`` : + Global timeout of the requests made to others engines in seconds. A bigger + timeout will allow to wait for answers from slow engines, but in consequence + will slow SearXNG reactivity (the result page may take the time specified in the + timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`. + +``useragent_suffix`` : + Suffix to the user-agent SearXNG uses to send requests to others engines. If an + engine wish to block you, a contact info here may be useful to avoid that. + +.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration + +``pool_maxsize``: + Number of allowable keep-alive connections, or ``null`` to always allow. The + default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_. + +``pool_connections`` : + Maximum number of allowable connections, or ``null`` # for no limits. The + default is 100. See ``max_connections`` `Pool limit configuration`_. + +``keepalive_expiry`` : + Number of seconds to keep a connection in the pool. By default 5.0 seconds. + See ``keepalive_expiry`` `Pool limit configuration`_. + +.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying + +``proxies`` : + Define one or more proxies you wish to use, see `httpx proxies`_. + If there are more than one proxy for one protocol (http, https), + requests to the engines are distributed in a round-robin fashion. + +``source_ips`` : + If you use multiple network interfaces, define from which IP the requests must + be made. Example: + + * ``0.0.0.0`` any local IPv4 address. + * ``::`` any local IPv6 address. + * ``192.168.0.1`` + * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses + * ``fe80::60a2:1691:e5a2:ee1f`` + * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network. + * ``[ 192.168.0.1, fe80::/126 ]`` + +``retries`` : + Number of retry in case of an HTTP error. On each retry, SearXNG uses an + different proxy and source ip. + +``enable_http2`` : + Enable by default. Set to ``false`` to disable HTTP/2. + +.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults +.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration + +``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR`` + Allow to specify a path to certificate. + see `httpx verification defaults`_. + + In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and + ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables. + see `httpx ssl configuration`_. + +``max_redirects`` : + 30 by default. Maximum redirect before it is an error. + +``using_tor_proxy`` : + Using tor proxy (``true``) or not (``false``) for all engines. The default is + ``false`` and can be overwritten in the :ref:`settings engine` + + diff --git a/_sources/admin/settings/settings_redis.rst.txt b/_sources/admin/settings/settings_redis.rst.txt new file mode 100644 index 000000000..9fb067553 --- /dev/null +++ b/_sources/admin/settings/settings_redis.rst.txt @@ -0,0 +1,49 @@ +.. _settings redis: + +========== +``redis:`` +========== + +.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url + +A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you +will find a description to test your redis connection in SearXNG. When using +sockets, don't forget to check the access rights on the socket:: + + ls -la /usr/local/searxng-redis/run/redis.sock + srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock + +In this example read/write access is given to the *searxng-redis* group. To get +access rights to redis instance (the socket), your SearXNG (or even your +developer) account needs to be added to the *searxng-redis* group. + +``url`` : ``$SEARXNG_REDIS_URL`` + URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`:: + + redis://[[username]:[password]]@localhost:6379/0 + rediss://[[username]:[password]]@localhost:6379/0 + unix://[[username]:[password]]@/path/to/socket.sock?db=0 + +.. _Redis Developer Notes: + +Redis Developer Notes +===================== + +To set up a local redis instance, first set the socket path of the Redis DB +in your YAML setting: + +.. code:: yaml + + redis: + url: unix:///usr/local/searxng-redis/run/redis.sock?db=0 + +Then use the following commands to install the redis instance (:ref:`manage +redis.help`): + +.. code:: sh + + $ ./manage redis.build + $ sudo -H ./manage redis.install + $ sudo -H ./manage redis.addgrp "${USER}" + # don't forget to logout & login to get member of group + diff --git a/_sources/admin/settings/settings_search.rst.txt b/_sources/admin/settings/settings_search.rst.txt new file mode 100644 index 000000000..b8f37b423 --- /dev/null +++ b/_sources/admin/settings/settings_search.rst.txt @@ -0,0 +1,104 @@ +.. _settings search: + +=========== +``search:`` +=========== + +.. code:: yaml + + search: + safe_search: 0 + autocomplete: "" + favicon_resolver: "" + default_lang: "" + ban_time_on_fail: 5 + max_ban_time_on_fail: 120 + suspended_times: + SearxEngineAccessDenied: 86400 + SearxEngineCaptcha: 86400 + SearxEngineTooManyRequests: 3600 + cf_SearxEngineCaptcha: 1296000 + cf_SearxEngineAccessDenied: 86400 + recaptcha_SearxEngineCaptcha: 604800 + formats: + - html + +``safe_search``: + Filter results. + + - ``0``: None + - ``1``: Moderate + - ``2``: Strict + +``autocomplete``: + Existing autocomplete backends, leave blank to turn it off. + + - ``dbpedia`` + - ``duckduckgo`` + - ``google`` + - ``mwmbl`` + - ``startpage`` + - ``swisscows`` + - ``qwant`` + - ``wikipedia`` + +``favicon_resolver``: + To activate favicons in SearXNG's result list select a default + favicon-resolver, leave blank to turn off the feature. Don't activate the + favicons before reading the :ref:`Favicons documentation <favicons>`. + +``default_lang``: + Default search language - leave blank to detect from browser information or + use codes from :origin:`searx/languages.py`. + +``languages``: + List of available languages - leave unset to use all codes from + :origin:`searx/languages.py`. Otherwise list codes of available languages. + The ``all`` value is shown as the ``Default language`` in the user interface + (in most cases, it is meant to send the query without a language parameter ; + in some cases, it means the English language) Example: + + .. code:: yaml + + languages: + - all + - en + - en-US + - de + - it-IT + - fr + - fr-BE + +``ban_time_on_fail``: + Ban time in seconds after engine errors. + +``max_ban_time_on_fail``: + Max ban time in seconds after engine errors. + +``suspended_times``: + Engine suspension time after error (in seconds; set to 0 to disable) + + ``SearxEngineAccessDenied``: 86400 + For error "Access denied" and "HTTP error [402, 403]" + + ``SearxEngineCaptcha``: 86400 + For error "CAPTCHA" + + ``SearxEngineTooManyRequests``: 3600 + For error "Too many request" and "HTTP error 429" + + Cloudflare CAPTCHA: + - ``cf_SearxEngineCaptcha``: 1296000 + - ``cf_SearxEngineAccessDenied``: 86400 + + Google CAPTCHA: + - ``recaptcha_SearxEngineCaptcha``: 604800 + +``formats``: + Result formats available from web, remove format to deny access (use lower + case). + + - ``html`` + - ``csv`` + - ``json`` + - ``rss`` diff --git a/_sources/admin/settings/settings_server.rst.txt b/_sources/admin/settings/settings_server.rst.txt new file mode 100644 index 000000000..cf480683e --- /dev/null +++ b/_sources/admin/settings/settings_server.rst.txt @@ -0,0 +1,57 @@ +.. _settings server: + +=========== +``server:`` +=========== + +.. code:: yaml + + server: + base_url: http://example.org/location # change this! + port: 8888 + bind_address: "127.0.0.1" + secret_key: "ultrasecretkey" # change this! + limiter: false + public_instance: false + image_proxy: false + default_http_headers: + X-Content-Type-Options : nosniff + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer + +``base_url`` : ``$SEARXNG_URL`` + The base URL where SearXNG is deployed. Used to create correct inbound links. + +``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` + Port number and *bind address* of the SearXNG web application if you run it + directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG + services running behind a proxy and using socket communications. + +``secret_key`` : ``$SEARXNG_SECRET`` + Used for cryptography purpose. + +``limiter`` : ``$SEARXNG_LIMITER`` + Rate limit the number of request on the instance, block some bots. The + :ref:`limiter` requires a :ref:`settings redis` database. + +.. _public_instance: + +``public_instance`` : ``$SEARXNG_PUBLIC_INSTANCE`` + + Setting that allows to enable features specifically for public instances (not + needed for local usage). By set to ``true`` the following features are + activated: + + - :py:obj:`searx.botdetection.link_token` in the :ref:`limiter` + +.. _image_proxy: + +``image_proxy`` : ``$SEARXNG_IMAGE_PROXY`` + Allow your instance of SearXNG of being able to proxy images. Uses memory space. + +.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers + +``default_http_headers`` : + Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__ + diff --git a/_sources/admin/settings/settings_ui.rst.txt b/_sources/admin/settings/settings_ui.rst.txt new file mode 100644 index 000000000..75f617683 --- /dev/null +++ b/_sources/admin/settings/settings_ui.rst.txt @@ -0,0 +1,74 @@ +.. _settings ui: + +======= +``ui:`` +======= + +.. _cache busting: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting + +.. code:: yaml + + ui: + static_use_hash: false + default_locale: "" + query_in_title: false + infinite_scroll: false + center_alignment: false + cache_url: https://web.archive.org/web/ + default_theme: simple + theme_args: + simple_style: auto + search_on_category_select: true + hotkeys: default + url_formatting: pretty + +.. _static_use_hash: + +``static_use_hash`` : ``$SEARXNG_STATIC_USE_HASH`` + Enables `cache busting`_ of static files. + +``default_locale`` : + SearXNG interface language. If blank, the locale is detected by using the + browser language. If it doesn't work, or you are deploying a language + specific instance of searx, a locale can be defined using an ISO language + code, like ``fr``, ``en``, ``de``. + +``query_in_title`` : + When true, the result page's titles contains the query it decreases the + privacy, since the browser can records the page titles. + +``infinite_scroll``: + When true, automatically loads the next page when scrolling to bottom of the current page. + +``center_alignment`` : default ``false`` + When enabled, the results are centered instead of being in the left (or RTL) + side of the screen. This setting only affects the *desktop layout* + (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`) + +.. cache_url: + +``cache_url`` : ``https://web.archive.org/web/`` + URL prefix of the internet archive or cache, don't forget trailing slash (if + needed). The default is https://web.archive.org/web/ alternatives are: + + - https://webcache.googleusercontent.com/search?q=cache: + - https://archive.today/ + +``default_theme`` : + Name of the theme you want to use by default on your SearXNG instance. + +``theme_args.simple_style``: + Style of simple theme: ``auto``, ``light``, ``dark``, ``black`` + +``results_on_new_tab``: + Open result links in a new tab by default. + +``search_on_category_select``: + Perform search immediately if a category selected. Disable to select multiple categories. + +``hotkeys``: + Hotkeys to use in the search interface: ``default``, ``vim`` (Vim-like). + +``url_formatting``: + Formatting type to use for result URLs: ``pretty``, ``full`` or ``host``. diff --git a/_sources/admin/update-searxng.rst.txt b/_sources/admin/update-searxng.rst.txt new file mode 100644 index 000000000..b9d15c3f7 --- /dev/null +++ b/_sources/admin/update-searxng.rst.txt @@ -0,0 +1,138 @@ +.. _searxng maintenance: + +=================== +SearXNG maintenance +=================== + +.. sidebar:: further read + + - :ref:`toolboxing` + - :ref:`uWSGI maintenance` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. _update searxng: + +How to update +============= + +How to update depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use the ``update`` command from the :ref:`searxng.sh` +script. + +.. code:: sh + + sudo -H ./utils/searxng.sh instance update + +.. _inspect searxng: + +How to inspect & debug +====================== + +How to debug depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use the ``inspect`` command from the :ref:`searxng.sh` +script. + +.. code:: sh + + sudo -H ./utils/searxng.sh instance inspect + +.. _migrate and stay tuned: + +Migrate and stay tuned! +======================= + +.. sidebar:: info + + - :pull:`1332` + - :pull:`456` + - :pull:`A comment about rolling release <446#issuecomment-954730358>` + +SearXNG is a *rolling release*; each commit to the master branch is a release. +SearXNG is growing rapidly, the services and opportunities are change every now +and then, to name just a few: + +- Bot protection has been switched from filtron to SearXNG's :ref:`limiter + <limiter>`, this requires a :ref:`Redis <settings redis>` database. + +- The image proxy morty is no longer needed, it has been replaced by the + :ref:`image proxy <image_proxy>` from SearXNG. + +- To save bandwidth :ref:`cache busting <static_use_hash>` has been implemented. + To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi + setup`. + +To stay tuned and get in use of the new features, instance maintainers have to +update the SearXNG code regularly (see :ref:`update searxng`). As the above +examples show, this is not always enough, sometimes services have to be set up +or reconfigured and sometimes services that are no longer needed should be +uninstalled. + +.. hint:: + + First of all: SearXNG is installed by the script :ref:`searxng.sh`. If you + have old filtron, morty or searx setup you should consider complete + uninstall/reinstall. + +Here you will find a list of changes that affect the infrastructure. Please +check to what extent it is necessary to update your installations: + +:pull:`1595`: ``[fix] uWSGI: increase buffer-size`` + Re-install uWSGI (:ref:`searxng.sh`) or fix your uWSGI ``searxng.ini`` + file manually. + + +remove obsolete services +------------------------ + +If your searx instance was installed *"Step by step"* or by the *"Installation +scripts"*, you need to undo the installation procedure completely. If you have +morty & filtron installed, it is recommended to uninstall these services also. +In case of scripts, to uninstall use the scripts from the origin you installed +searx from or try:: + + $ sudo -H ./utils/filtron.sh remove all + $ sudo -H ./utils/morty.sh remove all + $ sudo -H ./utils/searx.sh remove all + +.. hint:: + + If you are migrate from searx take into account that the ``.config.sh`` is no + longer used. + +If you upgrade from searx or from before :pull:`1332` has been merged and you +have filtron and/or morty installed, don't forget to remove HTTP sites. + +Apache:: + + $ sudo -H ./utils/filtron.sh apache remove + $ sudo -H ./utils/morty.sh apache remove + +nginx:: + + $ sudo -H ./utils/filtron.sh nginx remove + $ sudo -H ./utils/morty.sh nginx remove + + + +Check after Installation +------------------------ + +Once you have done your installation, you can run a SearXNG *check* procedure, +to see if there are some left overs. In this example there exists a *old* +``/etc/searx/settings.yml``:: + + $ sudo -H ./utils/searxng.sh instance check + + SearXNG checks + -------------- + ERROR: settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/ + INFO: [OK] (old) account 'searx' does not exists + INFO: [OK] (old) account 'filtron' does not exists + INFO: [OK] (old) account 'morty' does not exists + ... + INFO searx.redisdb : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock' + INFO searx.redisdb : connected to Redis |