diff options
| author | return42 <markus.heiser@darmarIT.de> | 2025-01-10 07:15:40 +0000 |
|---|---|---|
| committer | return42 <markus.heiser@darmarIT.de> | 2025-01-10 07:15:40 +0000 |
| commit | bf98d6b3f17d56ff760d9da5e5b4858ea104ffa9 (patch) | |
| tree | 67261a258d820bf1272e33547b975e20acdfe61e /_sources | |
| download | searxng-gh-pages.tar.gz searxng-gh-pages.zip | |
[doc] build from commit 94a0b415ef587e013df9e7350667b752a3822e90gh-pages
Diffstat (limited to '_sources')
119 files changed, 8868 insertions, 0 deletions
diff --git a/_sources/admin/answer-captcha.rst.txt b/_sources/admin/answer-captcha.rst.txt new file mode 100644 index 000000000..7ae29be0c --- /dev/null +++ b/_sources/admin/answer-captcha.rst.txt @@ -0,0 +1,69 @@ +=============================== +Answer CAPTCHA from server's IP +=============================== + +With a SSH tunnel we can send requests from server's IP and solve a CAPTCHA that +blocks requests from this IP. If your SearXNG instance is hosted at +``example.org`` and your login is ``user`` you can setup a proxy simply by +:man:`ssh`: + +.. code:: bash + + # SOCKS server: socks://127.0.0.1:8080 + + $ ssh -q -N -D 8080 user@example.org + +The ``socks://localhost:8080`` from above can be tested by: + +.. tabs:: + + .. group-tab:: server's IP + + .. code:: bash + + $ curl -x socks://127.0.0.1:8080 http://ipecho.net/plain + n.n.n.n + + .. group-tab:: desktop's IP + + .. code:: bash + + $ curl http://ipecho.net/plain + x.x.x.x + +In the settings of the WEB browser open the *"Network Settings"* and setup a +proxy on ``SOCKS5 127.0.0.1:8080`` (see screenshot below). In the WEB browser +check the IP from the server is used: + +- http://ipecho.net/plain + +Now open the search engine that blocks requests from your server's IP. If you +have `issues with the qwant engine +<https://github.com/searxng/searxng/issues/2011#issuecomment-1553317619>`__, +solve the CAPTCHA from `qwant.com <https://www.qwant.com/>`__. + +----- + +.. tabs:: + + .. group-tab:: Firefox + + .. kernel-figure:: answer-captcha/ffox-setting-proxy-socks.png + :alt: FFox proxy on SOCKS5, 127.0.0.1:8080 + + Firefox's network settings + + +.. admonition:: :man:`ssh` manual: + + -D [bind_address:]port + Specifies a local “dynamic” application-level port forwarding. This works + by allocating a socket to listen to port on the local side .. Whenever a + connection is made to this port, the connection is forwarded over the + secure channel, and the application protocol is then used to determine + where to connect to from the remote machine .. ssh will act as a SOCKS + server .. + + -N + Do not execute a remote command. This is useful for just forwarding ports. + diff --git a/_sources/admin/api.rst.txt b/_sources/admin/api.rst.txt new file mode 100644 index 000000000..8bd563ba7 --- /dev/null +++ b/_sources/admin/api.rst.txt @@ -0,0 +1,92 @@ +.. _adminapi: + +================== +Administration API +================== + +Get configuration data +====================== + +.. code:: http + + GET /config HTTP/1.1 + +Sample response +--------------- + +.. code:: json + + { + "autocomplete": "", + "categories": [ + "map", + "it", + "images", + ], + "default_locale": "", + "default_theme": "simple", + "engines": [ + { + "categories": [ + "map" + ], + "enabled": true, + "name": "openstreetmap", + "shortcut": "osm" + }, + { + "categories": [ + "it" + ], + "enabled": true, + "name": "arch linux wiki", + "shortcut": "al" + }, + { + "categories": [ + "images" + ], + "enabled": true, + "name": "google images", + "shortcut": "goi" + }, + { + "categories": [ + "it" + ], + "enabled": false, + "name": "bitbucket", + "shortcut": "bb" + }, + ], + "instance_name": "searx", + "locales": { + "de": "Deutsch (German)", + "en": "English", + "eo": "Esperanto (Esperanto)", + }, + "plugins": [ + { + "enabled": true, + "name": "HTTPS rewrite" + } + ], + "safe_search": 0 + } + + +Embed search bar +================ + +The search bar can be embedded into websites. Just paste the example into the +HTML of the site. URL of the SearXNG instance and values are customizable. + +.. code:: html + + <form method="post" action="https://example.org/"> + <!-- search --> <input type="text" name="q"> + <!-- categories --> <input type="hidden" name="categories" value="general,social media"> + <!-- language --> <input type="hidden" name="lang" value="all"> + <!-- locale --> <input type="hidden" name="locale" value="en"> + <!-- date filter --> <input type="hidden" name="time_range" value="month"> + </form> diff --git a/_sources/admin/architecture.rst.txt b/_sources/admin/architecture.rst.txt new file mode 100644 index 000000000..d0d40715d --- /dev/null +++ b/_sources/admin/architecture.rst.txt @@ -0,0 +1,38 @@ +.. _architecture: + +============ +Architecture +============ + +.. sidebar:: Further reading + + - Reverse Proxy: :ref:`Apache <apache searxng site>` & :ref:`nginx <nginx + searxng site>` + - uWSGI: :ref:`searxng uwsgi` + - SearXNG: :ref:`installation basic` + +Herein you will find some hints and suggestions about typical architectures of +SearXNG infrastructures. + +.. _architecture uWSGI: + +uWSGI Setup +=========== + +We start with a *reference* setup for public SearXNG instances which can be build +up and maintained by the scripts from our :ref:`toolboxing`. + +.. _arch public: + +.. kernel-figure:: arch_public.dot + :alt: arch_public.dot + + Reference architecture of a public SearXNG setup. + +The reference installation activates ``server.limiter``, ``server.image_proxy`` +and ``ui.static_use_hash`` (:origin:`/etc/searxng/settings.yml +<utils/templates/etc/searxng/settings.yml>`) + +.. literalinclude:: ../../utils/templates/etc/searxng/settings.yml + :language: yaml + :end-before: # preferences: diff --git a/_sources/admin/buildhosts.rst.txt b/_sources/admin/buildhosts.rst.txt new file mode 100644 index 000000000..bdb9ae1d4 --- /dev/null +++ b/_sources/admin/buildhosts.rst.txt @@ -0,0 +1,163 @@ +.. _buildhosts: + +========== +Buildhosts +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +To get best results from build, it's recommend to install additional packages on +build hosts (see :ref:`searxng.sh`). + +.. _searxng.sh install buildhost: + +Build and Development tools +=========================== + +To Install tools used by build and development tasks in once: + +.. tabs:: + + .. group-tab:: SearXNG's development tools + + .. code:: sh + + $ sudo -H ./utils/searxng.sh install buildhost + +This will install packages needed by SearXNG: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START distro-packages + :end-before: END distro-packages + +and packages needed to build documentation and run tests: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START build-packages + :end-before: END build-packages + +.. _docs build: + +Build docs +========== + +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org +.. _XeTeX: https://tug.org/xetex/ +.. _dvisvgm: https://dvisvgm.de/ + +.. sidebar:: Sphinx build needs + + - ImageMagick_ + - Graphviz_ + - XeTeX_ + - dvisvgm_ + +Most of the sphinx requirements are installed from :origin:`setup.py` and the +docs can be build from scratch with ``make docs.html``. For better math and +image processing additional packages are needed. The XeTeX_ needed not only for +PDF creation, it's also needed for :ref:`math` when HTML output is build. + +To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered +as images (``sphinx.ext.imgmath`` extension). + +Here is the extract from the :origin:`docs/conf.py` file, setting math renderer +to ``imgmath``: + +.. literalinclude:: ../conf.py + :language: python + :start-after: # sphinx.ext.imgmath setup + :end-before: # sphinx.ext.imgmath setup END + +If your docs build (``make docs.html``) shows warnings like this:: + + WARNING: dot(1) not found, for better output quality install \ + graphviz from https://www.graphviz.org + .. + WARNING: LaTeX command 'latex' cannot be run (needed for math \ + display), check the imgmath_latex setting + +you need to install additional packages on your build host, to get better HTML +output (:ref:`install buildhost <searxng.sh install buildhost>`). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install graphviz imagemagick texlive-xetex librsvg2-bin + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S graphviz imagemagick texlive-bin extra/librsvg + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install graphviz graphviz-gd ImageMagick texlive-xetex-bin librsvg2-tools + + +For PDF output you also need: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + $ sudo apt texlive-latex-recommended texlive-extra-utils ttf-dejavu + + .. group-tab:: Arch Linux + + .. code:: sh + + $ sudo pacman -S texlive-core texlive-latexextra ttf-dejavu + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + $ sudo dnf install \ + texlive-collection-fontsrecommended texlive-collection-latex \ + dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts + +.. _sh lint: + +Lint shell scripts +================== + +.. _ShellCheck: https://github.com/koalaman/shellcheck + +To lint shell scripts we use ShellCheck_ - a shell script static analysis tool +(:ref:`install buildhost <searxng.sh install buildhost>`). + +.. SNIP sh lint requirements + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install shellcheck + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S shellcheck + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install ShellCheck + +.. SNAP sh lint requirements diff --git a/_sources/admin/index.rst.txt b/_sources/admin/index.rst.txt new file mode 100644 index 000000000..b47074a8f --- /dev/null +++ b/_sources/admin/index.rst.txt @@ -0,0 +1,23 @@ +=========================== +Administrator documentation +=========================== + +.. toctree:: + :maxdepth: 2 + + settings/index + installation + installation-docker + installation-scripts + installation-searxng + installation-uwsgi + installation-nginx + installation-apache + update-searxng + answer-captcha + searx.favicons + searx.limiter + api + architecture + plugins + buildhosts diff --git a/_sources/admin/installation-apache.rst.txt b/_sources/admin/installation-apache.rst.txt new file mode 100644 index 000000000..b0b580607 --- /dev/null +++ b/_sources/admin/installation-apache.rst.txt @@ -0,0 +1,388 @@ +.. _installation apache: + +====== +Apache +====== + +.. _Apache: https://httpd.apache.org/ +.. _Apache Debian: + https://cwiki.apache.org/confluence/display/HTTPD/DistrosDefaultLayout#DistrosDefaultLayout-Debian,Ubuntu(Apachehttpd2.x): +.. _apache2.README.Debian: + https://salsa.debian.org/apache-team/apache2/raw/master/debian/apache2.README.Debian +.. _Apache Arch Linux: + https://wiki.archlinux.org/index.php/Apache_HTTP_Server +.. _Apache Fedora: + https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-apache-http-server/index.html +.. _Apache directives: + https://httpd.apache.org/docs/trunk/mod/directives.html +.. _Getting Started: + https://httpd.apache.org/docs/current/en/getting-started.html +.. _Terms Used to Describe Directives: + https://httpd.apache.org/docs/current/en/mod/directive-dict.html +.. _Configuration Files: + https://httpd.apache.org/docs/current/en/configuring.html +.. _ProxyPreserveHost: https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypreservehost +.. _LoadModule: + https://httpd.apache.org/docs/mod/mod_so.html#loadmodule +.. _IncludeOptional: + https://httpd.apache.org/docs/mod/core.html#includeoptional +.. _DocumentRoot: + https://httpd.apache.org/docs/trunk/mod/core.html#documentroot +.. _Location: + https://httpd.apache.org/docs/trunk/mod/core.html#location +.. _uWSGI Apache support: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html +.. _mod_proxy_uwsgi: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-proxy-uwsgi +.. _mod_proxy_http: + https://httpd.apache.org/docs/current/mod/mod_proxy_http.html +.. _mod_proxy: + https://httpd.apache.org/docs/current/mod/mod_proxy.html + + +This section explains how to set up a SearXNG instance using the HTTP server Apache_. +If you did use the :ref:`installation scripts` and do not have any special preferences +you can install the :ref:`SearXNG site <apache searxng site>` using +:ref:`searxng.sh <searxng.sh overview>`: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install apache + +If you have special interests or problems with setting up Apache, the following +section might give you some guidance. + + +.. sidebar:: further read + + - `Apache Arch Linux`_ + - `Apache Debian`_ + - `apache2.README.Debian`_ + - `Apache Fedora`_ + - `Apache directives`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +The Apache HTTP server +====================== + +If Apache_ is not installed, install it now. If apache_ is new to you, the +`Getting Started`_, `Configuration Files`_ and `Terms Used to Describe +Directives`_ documentation gives first orientation. There is also a list of +`Apache directives`_ *to keep in the pocket*. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H apt-get install apache2 + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H pacman -S apache + sudo -H systemctl enable httpd + sudo -H systemctl start http + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H dnf install httpd + sudo -H systemctl enable httpd + sudo -H systemctl start httpd + +Now at http://localhost you should see some kind of *Welcome* or *Test* page. +How this default site is configured, depends on the linux distribution +(compare `Apache directives`_). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + less /etc/apache2/sites-enabled/000-default.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot /var/www/html + + And the *welcome* page is the HTML file at ``/var/www/html/index.html``. + + .. group-tab:: Arch Linux + + .. code:: bash + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot "/srv/http" + <Directory "/srv/http"> + Options Indexes FollowSymLinks + AllowOverride None + Require all granted + </Directory> + + The *welcome* page of Arch Linux is a page showing the directory located + at ``DocumentRoot``. This *directory* page is generated by the Module + `mod_autoindex <https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html>`_: + + .. code:: apache + + LoadModule autoindex_module modules/mod_autoindex.so + ... + Include conf/extra/httpd-autoindex.conf + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the ``DocumentRoot`` directive: + + .. code:: apache + + DocumentRoot "/var/www/html" + ... + <Directory "/var/www"> + AllowOverride None + # Allow open access: + Require all granted + </Directory> + + On fresh installations, the ``/var/www`` is empty and the *default + welcome page* is shown, the configuration is located at:: + + less /etc/httpd/conf.d/welcome.conf + + +.. _Debian's Apache layout: + +Debian's Apache layout +---------------------- + +Be aware, Debian's Apache layout is quite different from the standard Apache +configuration. For details look at the apache2.README.Debian_ +(``/usr/share/doc/apache2/README.Debian.gz``). Some commands you should know on +Debian: + +* :man:`apache2ctl`: Apache HTTP server control interface +* :man:`a2enmod`, :man:`a2dismod`: switch on/off modules +* :man:`a2enconf`, :man:`a2disconf`: switch on/off configurations +* :man:`a2ensite`, :man:`a2dissite`: switch on/off sites + +.. _apache modules: + +Apache modules +-------------- + +To load additional modules, in most distributions you have to uncomment the +lines with the corresponding LoadModule_ directive, except in :ref:`Debian's +Apache layout`. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + :ref:`Debian's Apache layout` uses :man:`a2enmod` and :man:`a2dismod` to + activate or disable modules: + + .. code:: bash + + sudo -H a2enmod ssl + sudo -H a2enmod headers + sudo -H a2enmod proxy + sudo -H a2enmod proxy_http + sudo -H a2enmod proxy_uwsgi + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_ + directives: + + .. code:: apache + + LoadModule ssl_module modules/mod_ssl.so + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file, activate LoadModule_ + directives: + + .. code:: apache + + LoadModule ssl_module modules/mod_ssl.so + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + +.. _apache sites: + +Apache sites +------------ + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + In :ref:`Debian's Apache layout` you create a ``searxng.conf`` with the + ``<Location /searxng >`` directive and save this file in the *sites + available* folder at ``/etc/apache2/sites-available``. To enable the + ``searxng.conf`` use :man:`a2ensite`: + + .. code:: bash + + sudo -H a2ensite searxng.conf + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_ + directive: + + .. code:: apache + + IncludeOptional sites-enabled/*.conf + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/httpd/sites-available + mkdir -p /etc/httpd/sites-enabled + + Create configuration at ``/etc/httpd/sites-available`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/httpd/sites-available/searxng.conf \ + /etc/httpd/sites-enabled/searxng.conf + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file add a IncludeOptional_ + directive: + + .. code:: apache + + IncludeOptional sites-enabled/*.conf + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/httpd/sites-available + mkdir -p /etc/httpd/sites-enabled + + Create configuration at ``/etc/httpd/sites-available`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/httpd/sites-available/searxng.conf \ + /etc/httpd/sites-enabled/searxng.conf + + +.. _apache searxng site: + +Apache's SearXNG site +===================== + +.. _mod_uwsgi: https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-uwsgi + +.. sidebar:: uWSGI + + Use mod_proxy_uwsgi_ / don't use the old mod_uwsgi_ anymore. + +To proxy the incoming requests to the SearXNG instance Apache needs the +mod_proxy_ module (:ref:`apache modules`). + +.. sidebar:: HTTP headers + + With ProxyPreserveHost_ the incoming ``Host`` header is passed to the proxied + host. + +Depending on what your SearXNG installation is listening on, you need a http +mod_proxy_http_) or socket (mod_proxy_uwsgi_) communication to upstream. + +The :ref:`installation scripts` installs the :ref:`reference setup +<use_default_settings.yml>` and a :ref:`uwsgi setup` that listens on a socket by default. +You can install and activate your own ``searxng.conf`` like shown in +:ref:`apache sites`. + +.. tabs:: + + .. group-tab:: socket + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START apache socket + :end-before: END apache socket + + .. group-tab:: http + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START apache http + :end-before: END apache http + +.. _restart apache: + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H systemctl restart apache2 + sudo -H service uwsgi restart searxng + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H systemctl restart httpd + sudo -H systemctl restart uwsgi@searxng + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H systemctl restart httpd + sudo -H touch /etc/uwsgi.d/searxng.ini + + +disable logs +============ + +For better privacy you can disable Apache logs. In the examples above activate +one of the lines and `restart apache`_: + +.. code:: apache + + SetEnvIf Request_URI "/searxng" dontlog + # CustomLog /dev/null combined env=dontlog + +The ``CustomLog`` directive disables logs for the entire (virtual) server, use it +when the URL of the service does not have a path component (``/searxng``), so when +SearXNG is located at root (``/``). diff --git a/_sources/admin/installation-docker.rst.txt b/_sources/admin/installation-docker.rst.txt new file mode 100644 index 000000000..09471891b --- /dev/null +++ b/_sources/admin/installation-docker.rst.txt @@ -0,0 +1,197 @@ +.. _installation docker: + +================ +Docker Container +================ + +.. _ENTRYPOINT: https://docs.docker.com/engine/reference/builder/#entrypoint +.. _searxng/searxng @dockerhub: https://hub.docker.com/r/searxng/searxng +.. _searxng-docker: https://github.com/searxng/searxng-docker +.. _[caddy]: https://hub.docker.com/_/caddy +.. _Redis: https://redis.io/ + +---- + +.. sidebar:: info + + - `searxng/searxng @dockerhub`_ + - :origin:`Dockerfile` + - `Docker overview <https://docs.docker.com/get-started/overview>`_ + - `Docker Cheat Sheet <https://docs.docker.com/get-started/docker_cheatsheet.pdf>`_ + - `Alpine Linux <https://alpinelinux.org>`_ + `(wiki) <https://en.wikipedia.org/wiki/Alpine_Linux>`__ + `apt packages <https://pkgs.alpinelinux.org/packages>`_ + - Alpine's ``/bin/sh`` is :man:`dash` + +**If you intend to create a public instance using Docker, use our well maintained +docker container** + +- `searxng/searxng @dockerhub`_. + +.. sidebar:: hint + + The rest of this article is of interest only to those who want to create and + maintain their own Docker images. + +The sources are hosted at searxng-docker_ and the container includes: + +- a HTTPS reverse proxy `[caddy]`_ and +- a Redis_ DB + +The `default SearXNG setup <https://github.com/searxng/searxng-docker/blob/master/searxng/settings.yml>`_ +of this container: + +- enables :ref:`limiter <limiter>` to protect against bots +- enables :ref:`image proxy <image_proxy>` for better privacy +- enables :ref:`cache busting <static_use_hash>` to save bandwidth + +---- + + +Get Docker +========== + +If you plan to build and maintain a docker image by yourself, make sure you have +`Docker installed <https://docs.docker.com/get-docker/>`_. On Linux don't +forget to add your user to the docker group (log out and log back in so that +your group membership is re-evaluated): + +.. code:: sh + + $ sudo usermod -a -G docker $USER + + +searxng/searxng +=============== + +.. sidebar:: ``docker run`` + + - `-\-rm <https://docs.docker.com/engine/reference/run/#clean-up---rm>`__ + automatically clean up when container exits + - `-d <https://docs.docker.com/engine/reference/run/#detached--d>`__ start + detached container + - `-v <https://docs.docker.com/engine/reference/run/#volume-shared-filesystems>`__ + mount volume ``HOST:CONTAINER`` + +The docker image is based on :origin:`Dockerfile` and available from +`searxng/searxng @dockerhub`_. Using the docker image is quite easy, for +instance you can pull the `searxng/searxng @dockerhub`_ image and deploy a local +instance using `docker run <https://docs.docker.com/engine/reference/run/>`_: + +.. code:: sh + + $ mkdir my-instance + $ cd my-instance + $ export PORT=8080 + $ docker pull searxng/searxng + $ docker run --rm \ + -d -p ${PORT}:8080 \ + -v "${PWD}/searxng:/etc/searxng" \ + -e "BASE_URL=http://localhost:$PORT/" \ + -e "INSTANCE_NAME=my-instance" \ + searxng/searxng + 2f998.... # container's ID + +The environment variables UWSGI_WORKERS and UWSGI_THREADS overwrite the default +number of UWSGI processes and UWSGI threads specified in `/etc/searxng/uwsgi.ini`. + +Open your WEB browser and visit the URL: + +.. code:: sh + + $ xdg-open "http://localhost:$PORT" + +Inside ``${PWD}/searxng``, you will find ``settings.yml`` and ``uwsgi.ini``. You +can modify these files according to your needs and restart the Docker image. + +.. code:: sh + + $ docker container restart 2f998 + +Use command ``container ls`` to list running containers, add flag `-a +<https://docs.docker.com/engine/reference/commandline/container_ls>`__ to list +exited containers also. With ``container stop`` a running container can be +stopped. To get rid of a container use ``container rm``: + +.. code:: sh + + $ docker container ls + CONTAINER ID IMAGE COMMAND CREATED ... + 2f998d725993 searxng/searxng "/sbin/tini -- /usr/…" 7 minutes ago ... + + $ docker container stop 2f998 + $ docker container rm 2f998 + +.. sidebar:: Warning + + This might remove all docker items, not only those from SearXNG. + +If you won't use docker anymore and want to get rid of all containers & images +use the following *prune* command: + +.. code:: sh + + $ docker stop $(docker ps -aq) # stop all containers + $ docker system prune # make some housekeeping + $ docker rmi -f $(docker images -q) # drop all images + + +shell inside container +---------------------- + +.. sidebar:: Bashism + + - `A tale of two shells: bash or dash <https://lwn.net/Articles/343924/>`_ + - `How to make bash scripts work in dash <http://mywiki.wooledge.org/Bashism>`_ + - `Checking for Bashisms <https://dev.to/bowmanjd/writing-bash-scripts-that-are-not-only-bash-checking-for-bashisms-and-testing-with-dash-1bli>`_ + +Like in many other distributions, Alpine's `/bin/sh +<https://wiki.ubuntu.com/DashAsBinSh>`__ is :man:`dash`. Dash is meant to be +`POSIX-compliant <https://pubs.opengroup.org/onlinepubs/9699919799>`__. +Compared to debian, in the Alpine image :man:`bash` is not installed. The +:origin:`dockerfiles/docker-entrypoint.sh` script is checked *against dash* +(``make tests.shell``). + +To open a shell inside the container: + +.. code:: sh + + $ docker exec -it 2f998 sh + + +Build the image +=============== + +It's also possible to build SearXNG from the embedded :origin:`Dockerfile`:: + + $ git clone https://github.com/searxng/searxng.git + $ cd searxng + $ make docker.build + ... + Successfully built 49586c016434 + Successfully tagged searxng/searxng:latest + Successfully tagged searxng/searxng:1.0.0-209-9c823800-dirty + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + searxng/searxng 1.0.0-209-9c823800-dirty 49586c016434 13 minutes ago 308MB + searxng/searxng latest 49586c016434 13 minutes ago 308MB + alpine 3.13 6dbb9cc54074 3 weeks ago 5.61MB + + +Command line +============ + +.. sidebar:: docker run + + Use flags ``-it`` for `interactive processes + <https://docs.docker.com/engine/reference/run/#foreground>`__. + +In the :origin:`Dockerfile` the ENTRYPOINT_ is defined as +:origin:`dockerfiles/docker-entrypoint.sh` + +.. code:: sh + + docker run --rm -it searxng/searxng -h + +.. program-output:: ../dockerfiles/docker-entrypoint.sh -h diff --git a/_sources/admin/installation-nginx.rst.txt b/_sources/admin/installation-nginx.rst.txt new file mode 100644 index 000000000..f95354b53 --- /dev/null +++ b/_sources/admin/installation-nginx.rst.txt @@ -0,0 +1,252 @@ +.. _installation nginx: + +===== +NGINX +===== + +.. _nginx: + https://docs.nginx.com/nginx/admin-guide/ +.. _nginx server configuration: + https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers +.. _nginx beginners guide: + https://nginx.org/en/docs/beginners_guide.html +.. _Getting Started wiki: + https://www.nginx.com/resources/wiki/start/ +.. _uWSGI support from nginx: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html +.. _uwsgi_params: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#configuring-nginx +.. _SCRIPT_NAME: + https://werkzeug.palletsprojects.com/en/1.0.x/wsgi/#werkzeug.wsgi.get_script_name + +This section explains how to set up a SearXNG instance using the HTTP server nginx_. +If you have used the :ref:`installation scripts` and do not have any special preferences +you can install the :ref:`SearXNG site <nginx searxng site>` using +:ref:`searxng.sh <searxng.sh overview>`: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install nginx + +If you have special interests or problems with setting up nginx, the following +section might give you some guidance. + + +.. sidebar:: further reading + + - nginx_ + - `nginx beginners guide`_ + - `nginx server configuration`_ + - `Getting Started wiki`_ + - `uWSGI support from nginx`_ + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +The nginx HTTP server +===================== + +If nginx_ is not installed, install it now. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H apt-get install nginx + + .. group-tab:: Arch Linux + + .. code-block:: sh + + sudo -H pacman -S nginx-mainline + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + sudo -H dnf install nginx + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + +Now at http://localhost you should see a *Welcome to nginx!* page, on Fedora you +see a *Fedora Webserver - Test Page*. The test page comes from the default +`nginx server configuration`_. How this default site is configured, +depends on the linux distribution: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + less /etc/nginx/nginx.conf + + There is one line that includes site configurations from: + + .. code:: nginx + + include /etc/nginx/sites-enabled/*; + + .. group-tab:: Arch Linux + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + There is a configuration section named ``server``: + + .. code-block:: nginx + + server { + listen 80; + server_name localhost; + # ... + } + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + There is one line that includes site configurations from: + + .. code:: nginx + + include /etc/nginx/conf.d/*.conf; + + +.. _nginx searxng site: + +NGINX's SearXNG site +==================== + +Now you have to create a configuration file (``searxng.conf``) for the SearXNG +site. If nginx_ is new to you, the `nginx beginners guide`_ is a good starting +point and the `Getting Started wiki`_ is always a good resource *to keep in the +pocket*. + +Depending on what your SearXNG installation is listening on, you need a http or socket +communication to upstream. + +.. tabs:: + + .. group-tab:: socket + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START nginx socket + :end-before: END nginx socket + + .. group-tab:: http + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START nginx http + :end-before: END nginx http + +The :ref:`installation scripts` installs the :ref:`reference setup +<use_default_settings.yml>` and a :ref:`uwsgi setup` that listens on a socket by default. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + Create configuration at ``/etc/nginx/sites-available/`` and place a + symlink to ``sites-enabled``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/sites-available/searxng.conf \ + /etc/nginx/sites-enabled/searxng.conf + + .. group-tab:: Arch Linux + + In the ``/etc/nginx/nginx.conf`` file, in the ``server`` section add a + `include <https://nginx.org/en/docs/ngx_core_module.html#include>`_ + directive: + + .. code:: nginx + + server { + # ... + include /etc/nginx/default.d/*.conf; + # ... + } + + Create two folders, one for the *available sites* and one for the *enabled sites*: + + .. code:: bash + + mkdir -p /etc/nginx/default.d + mkdir -p /etc/nginx/default.apps-available + + Create configuration at ``/etc/nginx/default.apps-available`` and place a + symlink to ``default.d``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \ + /etc/nginx/default.d/searxng.conf + + .. group-tab:: Fedora / RHEL + + Create a folder for the *available sites*: + + .. code:: bash + + mkdir -p /etc/nginx/default.apps-available + + Create configuration at ``/etc/nginx/default.apps-available`` and place a + symlink to ``conf.d``: + + .. code:: bash + + sudo -H ln -s /etc/nginx/default.apps-available/searxng.conf \ + /etc/nginx/conf.d/searxng.conf + +Restart services: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H service uwsgi restart searxng + + .. group-tab:: Arch Linux + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H systemctl restart uwsgi@searxng + + .. group-tab:: Fedora / RHEL + + .. code:: bash + + sudo -H systemctl restart nginx + sudo -H touch /etc/uwsgi.d/searxng.ini + + +Disable logs +============ + +For better privacy you can disable nginx logs in ``/etc/nginx/nginx.conf``. + +.. code:: nginx + + http { + # ... + access_log /dev/null; + error_log /dev/null; + # ... + } diff --git a/_sources/admin/installation-scripts.rst.txt b/_sources/admin/installation-scripts.rst.txt new file mode 100644 index 000000000..2d43f5e37 --- /dev/null +++ b/_sources/admin/installation-scripts.rst.txt @@ -0,0 +1,62 @@ +.. _installation scripts: + +=================== +Installation Script +=================== + +.. sidebar:: Update the OS first! + + To avoid unwanted side effects, update your OS before installing SearXNG. + +The following will install a setup as shown in :ref:`the reference architecture +<arch public>`. First you need to get a clone of the repository. The clone is only needed for +the installation procedure and some maintenance tasks. + +.. sidebar:: further read + + - :ref:`toolboxing` + +Jump to a folder that is readable by *others* and start to clone SearXNG, +alternatively you can create your own fork and clone from there. + +.. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: further read + + - :ref:`inspect searxng` + +To install a SearXNG :ref:`reference setup <use_default_settings.yml>` +including a :ref:`uWSGI setup <architecture uWSGI>` as described in the +:ref:`installation basic` and in the :ref:`searxng uwsgi` section type: + +.. code:: bash + + $ sudo -H ./utils/searxng.sh install all + +.. attention:: + + For the installation procedure, use a *sudoer* login to run the scripts. If + you install from ``root``, take into account that the scripts are creating a + ``searxng`` user. In the installation procedure this new created user does + need to have read access to the cloned SearXNG repository, which is not the case if you clone + it into a folder below ``/root``! + +.. sidebar:: further read + + - :ref:`update searxng` + +.. _caddy: https://hub.docker.com/_/caddy + +When all services are installed and running fine, you can add SearXNG to your +HTTP server. We do not have any preferences regarding the HTTP server, you can use +whatever you prefer. + +We use caddy in our :ref:`docker image <installation docker>` and we have +implemented installation procedures for: + +- :ref:`installation nginx` +- :ref:`installation apache` diff --git a/_sources/admin/installation-searxng.rst.txt b/_sources/admin/installation-searxng.rst.txt new file mode 100644 index 000000000..446f45cd2 --- /dev/null +++ b/_sources/admin/installation-searxng.rst.txt @@ -0,0 +1,132 @@ +.. _installation basic: + +========================= +Step by step installation +========================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +In this section we show the setup of a SearXNG instance that will be installed +by the :ref:`installation scripts`. + +.. _install packages: + +Install packages +================ + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START distro-packages + :end-before: END distro-packages + +.. hint:: + + This installs also the packages needed by :ref:`searxng uwsgi` + +.. _create searxng user: + +Create user +=========== + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START create user + :end-before: END create user + +.. _searxng-src: + +Install SearXNG & dependencies +============================== + +Start a interactive shell from new created user and clone SearXNG: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START clone searxng + :end-before: END clone searxng + +In the same shell create *virtualenv*: + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START create virtualenv + :end-before: END create virtualenv + +To install SearXNG's dependencies, exit the SearXNG *bash* session you opened above +and start a new one. Before installing, check if your *virtualenv* was sourced +from the login (*~/.profile*): + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START manage.sh update_packages + :end-before: END manage.sh update_packages + +.. tip:: + + Open a second terminal for the configuration tasks and leave the ``(searx)$`` + terminal open for the tasks below. + + +.. _use_default_settings.yml: + +Configuration +============= + +.. sidebar:: ``use_default_settings: True`` + + - :ref:`settings.yml` + - :ref:`settings location` + - :ref:`settings use_default_settings` + - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` + +To create a initial ``/etc/searxng/settings.yml`` we recommend to start with a +copy of the file :origin:`utils/templates/etc/searxng/settings.yml`. This setup +:ref:`use default settings <settings use_default_settings>` from +:origin:`searx/settings.yml` and is shown in the tab *"Use default settings"* +below. This setup: + +- enables :ref:`limiter <limiter>` to protect against bots +- enables :ref:`image proxy <image_proxy>` for better privacy +- enables :ref:`cache busting <static_use_hash>` to save bandwidth + +Modify the ``/etc/searxng/settings.yml`` to your needs: + +.. tabs:: + + .. group-tab:: Use default settings + + .. literalinclude:: ../../utils/templates/etc/searxng/settings.yml + :language: yaml + :end-before: # hostnames: + + To see the entire file jump to :origin:`utils/templates/etc/searxng/settings.yml` + + .. group-tab:: searx/settings.yml + + .. literalinclude:: ../../searx/settings.yml + :language: yaml + :end-before: # hostnames: + + To see the entire file jump to :origin:`searx/settings.yml` + +For a *minimal setup* you need to set ``server:secret_key``. + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng config + :end-before: END searxng config + + +Check +===== + +To check your SearXNG setup, optional enable debugging and start the *webapp*. +SearXNG looks at the exported environment ``$SEARXNG_SETTINGS_PATH`` for a +configuration file. + +.. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START check searxng installation + :end-before: END check searxng installation + +If everything works fine, hit ``[CTRL-C]`` to stop the *webapp* and disable the +debug option in ``settings.yml``. You can now exit SearXNG user bash session (enter exit +command twice). At this point SearXNG is not demonized; uwsgi allows this. + diff --git a/_sources/admin/installation-uwsgi.rst.txt b/_sources/admin/installation-uwsgi.rst.txt new file mode 100644 index 000000000..78da22f45 --- /dev/null +++ b/_sources/admin/installation-uwsgi.rst.txt @@ -0,0 +1,268 @@ +.. _searxng uwsgi: + +===== +uWSGI +===== + +.. sidebar:: further reading + + - `systemd.unit`_ + - `uWSGI Emperor`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _systemd.unit: https://www.freedesktop.org/software/systemd/man/systemd.unit.html +.. _One service per app in systemd: + https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd +.. _uWSGI Emperor: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html +.. _uwsgi ini file: + https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files +.. _systemd unit template: + http://0pointer.de/blog/projects/instances.html + + +Origin uWSGI +============ + +.. _Tyrant mode: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html#tyrant-mode-secure-multi-user-hosting + +How uWSGI is implemented by distributors varies. The uWSGI project itself +recommends two methods: + +1. `systemd.unit`_ template file as described here `One service per app in systemd`_: + + There is one `systemd unit template`_ on the system installed and one `uwsgi + ini file`_ per uWSGI-app placed at dedicated locations. Take archlinux and a + ``searxng.ini`` as example:: + + systemd template unit: /usr/lib/systemd/system/uwsgi@.service + contains: [Service] + ExecStart=/usr/bin/uwsgi --ini /etc/uwsgi/%I.ini + + SearXNG application: /etc/uwsgi/searxng.ini + links to: /etc/uwsgi/apps-available/searxng.ini + + The SearXNG app (template ``/etc/uwsgi/%I.ini``) can be maintained as known + from common systemd units: + + .. code:: sh + + $ systemctl enable uwsgi@searxng + $ systemctl start uwsgi@searxng + $ systemctl restart uwsgi@searxng + $ systemctl stop uwsgi@searxng + +2. The `uWSGI Emperor`_ which fits for maintaining a large range of uwsgi + apps and there is a `Tyrant mode`_ to secure multi-user hosting. + + The Emperor mode is a special uWSGI instance that will monitor specific + events. The Emperor mode (the service) is started by a (common, not template) + systemd unit. + + The Emperor service will scan specific directories for `uwsgi ini file`_\s + (also know as *vassals*). If a *vassal* is added, removed or the timestamp is + modified, a corresponding action takes place: a new uWSGI instance is started, + reload or stopped. Take Fedora and a ``searxng.ini`` as example:: + + to install & start SearXNG instance create --> /etc/uwsgi.d/searxng.ini + to reload the instance edit timestamp --> touch /etc/uwsgi.d/searxng.ini + to stop instance remove ini --> rm /etc/uwsgi.d/searxng.ini + + +Distributors +============ + +The `uWSGI Emperor`_ mode and `systemd unit template`_ is what the distributors +mostly offer their users, even if they differ in the way they implement both +modes and their defaults. Another point they might differ in is the packaging of +plugins (if so, compare :ref:`install packages`) and what the default python +interpreter is (python2 vs. python3). + +While archlinux does not start a uWSGI service by default, Fedora (RHEL) starts +a Emperor in `Tyrant mode`_ by default (you should have read :ref:`uWSGI Tyrant +mode pitfalls`). Worth to know; debian (ubuntu) follow a complete different +approach, read see :ref:`Debian's uWSGI layout`. + +.. _Debian's uWSGI layout: + +Debian's uWSGI layout +--------------------- + +.. _uwsgi.README.Debian: + https://salsa.debian.org/uwsgi-team/uwsgi/-/raw/debian/latest/debian/uwsgi.README.Debian + +Be aware, Debian's uWSGI layout is quite different from the standard uWSGI +configuration. Your are familiar with :ref:`Debian's Apache layout`? .. they do a +similar thing for the uWSGI infrastructure. The folders are:: + + /etc/uwsgi/apps-available/ + /etc/uwsgi/apps-enabled/ + +The `uwsgi ini file`_ is enabled by a symbolic link:: + + ln -s /etc/uwsgi/apps-available/searxng.ini /etc/uwsgi/apps-enabled/ + +More details can be found in the uwsgi.README.Debian_ +(``/usr/share/doc/uwsgi/README.Debian.gz``). Some commands you should know on +Debian: + +.. code:: none + + Commands recognized by init.d script + ==================================== + + You can issue to init.d script following commands: + * start | starts daemon + * stop | stops daemon + * reload | sends to daemon SIGHUP signal + * force-reload | sends to daemon SIGTERM signal + * restart | issues 'stop', then 'start' commands + * status | shows status of daemon instance (running/not running) + + 'status' command must be issued with exactly one argument: '<confname>'. + + Controlling specific instances of uWSGI + ======================================= + + You could control specific instance(s) by issuing: + + SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi <command> <confname> <confname>... + + where: + * <command> is one of 'start', 'stop' etc. + * <confname> is the name of configuration file (without extension) + + For example, this is how instance for /etc/uwsgi/apps-enabled/hello.xml is + started: + + SYSTEMCTL_SKIP_REDIRECT=1 service uwsgi start hello + + +.. _uWSGI maintenance: + +uWSGI maintenance +================= + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description ubuntu-20.04 + :end-before: END searxng uwsgi-description ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description arch + :end-before: END searxng uwsgi-description arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-description fedora + :end-before: END searxng uwsgi-description fedora + + +.. _uwsgi setup: + +uWSGI setup +=========== + +Create the configuration ini-file according to your distribution and restart the +uwsgi application. As shown below, the :ref:`installation scripts` installs by +default: + +- a uWSGI setup that listens on a socket and +- enables :ref:`cache busting <static_use_hash>`. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini ubuntu-20.04 + :end-before: END searxng uwsgi-appini ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini arch + :end-before: END searxng uwsgi-appini arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searxng.rst + :start-after: START searxng uwsgi-appini fedora + :end-before: END searxng uwsgi-appini fedora + + +.. _uWSGI Tyrant mode pitfalls: + +Pitfalls of the Tyrant mode +=========================== + +The implementation of the process owners and groups in the `Tyrant mode`_ is +somewhat unusual and requires special consideration. In `Tyrant mode`_ mode the +Emperor will run the vassal using the UID/GID of the vassal configuration file +(user and group of the app ``.ini`` file). + +.. _#2099@uWSGI: https://github.com/unbit/uwsgi/issues/2099 +.. _#752@uWSGI: https://github.com/unbit/uwsgi/pull/752 +.. _#2425uWSGI: https://github.com/unbit/uwsgi/issues/2425 + +Without option ``emperor-tyrant-initgroups=true`` in ``/etc/uwsgi.ini`` the +process won't get the additional groups, but this option is not available in +2.0.x branch (see `#2099@uWSGI`_) the feature `#752@uWSGI`_ has been merged (on +Oct. 2014) to the master branch of uWSGI but had never been released; the last +major release is from Dec. 2013, since the there had been only bugfix releases +(see `#2425uWSGI`_). To shorten up: + + **In Tyrant mode, there is no way to get additional groups, and the uWSGI + process misses additional permissions that may be needed.** + +For example on Fedora (RHEL): If you try to install a redis DB with socket +communication and you want to connect to it from the SearXNG uWSGI, you will see a +*Permission denied* in the log of your instance:: + + ERROR:searx.redisdb: [searxng (993)] can't connect redis DB ... + ERROR:searx.redisdb: Error 13 connecting to unix socket: /usr/local/searxng-redis/run/redis.sock. Permission denied. + ERROR:searx.plugins.limiter: init limiter DB failed!!! + +Even if your *searxng* user of the uWSGI process is added to additional groups +to give access to the socket from the redis DB:: + + $ groups searxng + searxng : searxng searxng-redis + +To see the effective groups of the uwsgi process, you have to look at the status +of the process, by example:: + + $ ps -aef | grep '/usr/sbin/uwsgi --ini searxng.ini' + searxng 93 92 0 12:43 ? 00:00:00 /usr/sbin/uwsgi --ini searxng.ini + searxng 186 93 0 12:44 ? 00:00:01 /usr/sbin/uwsgi --ini searxng.ini + +Here you can see that the additional "Groups" of PID 186 are unset (missing gid +of ``searxng-redis``):: + + $ cat /proc/186/task/186/status + ... + Uid: 993 993 993 993 + Gid: 993 993 993 993 + FDSize: 128 + Groups: + ... diff --git a/_sources/admin/installation.rst.txt b/_sources/admin/installation.rst.txt new file mode 100644 index 000000000..54d901f76 --- /dev/null +++ b/_sources/admin/installation.rst.txt @@ -0,0 +1,22 @@ +.. _installation: + +============ +Installation +============ + +*You're spoilt for choice*, choose your preferred method of installation. + +- :ref:`installation docker` +- :ref:`installation scripts` +- :ref:`installation basic` + +The :ref:`installation basic` is an excellent illustration of *how a SearXNG +instance is build up* (see :ref:`architecture uWSGI`). If you do not have any +special preferences, it's recommended to use the :ref:`installation docker` or the +:ref:`installation scripts`. + +.. attention:: + + SearXNG is growing rapidly, you should regularly read our :ref:`migrate and + stay tuned` section. If you want to upgrade an existing instance or migrate + from searx to SearXNG, you should read this section first! diff --git a/_sources/admin/plugins.rst.txt b/_sources/admin/plugins.rst.txt new file mode 100644 index 000000000..d97b3dada --- /dev/null +++ b/_sources/admin/plugins.rst.txt @@ -0,0 +1,39 @@ +.. _plugins generic: + +=============== +Plugins builtin +=============== + +.. sidebar:: Further reading .. + + - :ref:`dev plugin` + +Configuration defaults (at built time): + +:DO: Default on + +.. _configured plugins: + +.. jinja:: searx + + .. flat-table:: Plugins configured at built time (defaults) + :header-rows: 1 + :stub-columns: 1 + :widths: 3 1 9 + + * - Name + - DO + - Description + + JS & CSS dependencies + + {% for plgin in plugins %} + + * - {{plgin.name}} + - {{(plgin.default_on and "y") or ""}} + - {{plgin.description}} + + {% for dep in (plgin.js_dependencies + plgin.css_dependencies) %} + | ``{{dep}}`` {% endfor %} + + {% endfor %} diff --git a/_sources/admin/searx.favicons.rst.txt b/_sources/admin/searx.favicons.rst.txt new file mode 100644 index 000000000..b05b3458b --- /dev/null +++ b/_sources/admin/searx.favicons.rst.txt @@ -0,0 +1,251 @@ +.. _favicons: + +======== +Favicons +======== + +.. sidebar:: warning + + Don't activate the favicons before reading the documentation. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Activating the favicons in SearXNG is very easy, but this **generates a +significantly higher load** in the client/server communication and increases +resources needed on the server. + +To mitigate these disadvantages, various methods have been implemented, +including a *cache*. The cache must be parameterized according to your own +requirements and maintained regularly. + +To activate favicons in SearXNG's result list, set a default +``favicon_resolver`` in the :ref:`search <settings search>` settings: + +.. code:: yaml + + search: + favicon_resolver: "duckduckgo" + +By default and without any extensions, SearXNG serves these resolvers: + +- ``duckduckgo`` +- ``allesedv`` +- ``google`` +- ``yandex`` + +With the above setting favicons are displayed, the user has the option to +deactivate this feature in his settings. If the user is to have the option of +selecting from several *resolvers*, a further setting is required / but this +setting will be discussed :ref:`later <register resolvers>` in this article, +first we have to setup the favicons cache. + +Infrastructure +============== + +The infrastructure for providing the favicons essentially consists of three +parts: + +- :py:obj:`Favicons-Proxy <.favicons.proxy>` (aka *proxy*) +- :py:obj:`Favicons-Resolvers <.favicons.resolvers>` (aka *resolver*) +- :py:obj:`Favicons-Cache <.favicons.cache>` (aka *cache*) + +To protect the privacy of users, the favicons are provided via a *proxy*. This +*proxy* is automatically activated with the above activation of a *resolver*. +Additional requests are required to provide the favicons: firstly, the *proxy* +must process the incoming requests and secondly, the *resolver* must make +outgoing requests to obtain the favicons from external sources. + +A *cache* has been developed to massively reduce both, incoming and outgoing +requests. This *cache* is also activated automatically with the above +activation of a *resolver*. In its defaults, however, the *cache* is minimal +and not well suitable for a production environment! + +.. _favicon cache setup: + +Setting up the cache +==================== + +To parameterize the *cache* and more settings of the favicons infrastructure, a +TOML_ configuration is created in the file ``/etc/searxng/favicons.toml``. + +.. code:: toml + + [favicons] + + cfg_schema = 1 # config's schema version no. + + [favicons.cache] + + db_url = "/var/cache/searxng/faviconcache.db" # default: "/tmp/faviconcache.db" + LIMIT_TOTAL_BYTES = 2147483648 # 2 GB / default: 50 MB + # HOLD_TIME = 5184000 # 60 days / default: 30 days + # BLOB_MAX_BYTES = 40960 # 40 KB / default 20 KB + # MAINTENANCE_MODE = "off" # default: "auto" + # MAINTENANCE_PERIOD = 600 # 10min / default: 1h + +:py:obj:`cfg_schema <.FaviconConfig.cfg_schema>`: + Is required to trigger any processes required for future upgrades / don't + change it. + +:py:obj:`cache.db_url <.FaviconCacheConfig.db_url>`: + The path to the (SQLite_) database file. The default path is in the `/tmp`_ + folder, which is deleted on every reboot and is therefore unsuitable for a + production environment. The FHS_ provides the folder for the + application cache + + The FHS_ provides the folder `/var/cache`_ for the cache of applications, so a + suitable storage location of SearXNG's caches is folder ``/var/cache/searxng``. + In container systems, a volume should be mounted for this folder and in a + standard installation (compare :ref:`create searxng user`), the folder must be + created and the user under which the SearXNG process is running must be given + write permission to this folder. + + .. code:: bash + + $ sudo mkdir /var/cache/searxng + $ sudo chown root:searxng /var/cache/searxng/ + $ sudo chmod g+w /var/cache/searxng/ + +:py:obj:`cache.LIMIT_TOTAL_BYTES <.FaviconCacheConfig.LIMIT_TOTAL_BYTES>`: + Maximum of bytes stored in the cache of all blobs. The limit is only reached + at each maintenance interval after which the oldest BLOBs are deleted; the + limit is exceeded during the maintenance period. + + .. attention:: + + If the maintenance period is too long or maintenance is switched + off completely, the cache grows uncontrollably. + +SearXNG hosters can change other parameters of the cache as required: + +- :py:obj:`cache.HOLD_TIME <.FaviconCacheConfig.HOLD_TIME>` +- :py:obj:`cache.BLOB_MAX_BYTES <.FaviconCacheConfig.BLOB_MAX_BYTES>` + + +Maintenance of the cache +------------------------ + +Regular maintenance of the cache is required! By default, regular maintenance +is triggered automatically as part of the client requests: + +- :py:obj:`cache.MAINTENANCE_MODE <.FaviconCacheConfig.MAINTENANCE_MODE>` (default ``auto``) +- :py:obj:`cache.MAINTENANCE_PERIOD <.FaviconCacheConfig.MAINTENANCE_PERIOD>` (default ``6000`` / 1h) + +As an alternative to maintenance as part of the client request process, it is +also possible to carry out maintenance using an external process. For example, +by creating a :man:`crontab` entry for maintenance: + +.. code:: bash + + $ python -m searx.favicons cache maintenance + +The following command can be used to display the state of the cache: + +.. code:: bash + + $ python -m searx.favicons cache state + + +.. _favicon proxy setup: + +Proxy configuration +=================== + +Most of the options of the :py:obj:`Favicons-Proxy <.favicons.proxy>` are +already set sensibly with settings from the :ref:`settings.yml <searxng +settings.yml>` and should not normally be adjusted. + +.. code:: toml + + [favicons.proxy] + + max_age = 5184000 # 60 days / default: 7 days (604800 sec) + + +:py:obj:`max_age <.FaviconProxyConfig.max_age>`: + The `HTTP Cache-Control max-age`_ response directive indicates that the + response remains fresh until N seconds after the response is generated. This + setting therefore determines how long a favicon remains in the client's cache. + As a rule, in the favicons infrastructure of SearXNG's this setting only + affects favicons whose byte size exceeds :ref:`BLOB_MAX_BYTES <favicon cache + setup>` (the other favicons that are already in the cache are embedded as + `data URL`_ in the :py:obj:`generated HTML <.favicons.proxy.favicon_url>`, + which can greatly reduce the number of additional requests). + +.. _register resolvers: + +Register resolvers +------------------ + +A :py:obj:`resolver <.favicon.resolvers>` is a function that obtains the favicon +from an external source. The resolver functions available to the user are +registered with their fully qualified name (FQN_) in a ``resolver_map``. + +If no ``resolver_map`` is defined in the ``favicon.toml``, the favicon +infrastructure of SearXNG generates this ``resolver_map`` automatically +depending on the ``settings.yml``. SearXNG would automatically generate the +following TOML configuration from the following YAML configuration: + +.. code:: yaml + + search: + favicon_resolver: "duckduckgo" + +.. code:: toml + + [favicons.proxy.resolver_map] + + "duckduckgo" = "searx.favicons.resolvers.duckduckgo" + +If this automatism is not desired, then (and only then) a separate +``resolver_map`` must be created. For example, to give the user two resolvers to +choose from, the following configuration could be used: + +.. code:: toml + + [favicons.proxy.resolver_map] + + "duckduckgo" = "searx.favicons.resolvers.duckduckgo" + "allesedv" = "searx.favicons.resolvers.allesedv" + # "google" = "searx.favicons.resolvers.google" + # "yandex" = "searx.favicons.resolvers.yandex" + +.. note:: + + With each resolver, the resource requirement increases significantly. + +The number of resolvers increases: + +- the number of incoming/outgoing requests and +- the number of favicons to be stored in the cache. + +In the following we list the resolvers available in the core of SearXNG, but via +the FQN_ it is also possible to implement your own resolvers and integrate them +into the *proxy*: + +- :py:obj:`searx.favicons.resolvers.duckduckgo` +- :py:obj:`searx.favicons.resolvers.allesedv` +- :py:obj:`searx.favicons.resolvers.google` +- :py:obj:`searx.favicons.resolvers.yandex` + + + +.. _SQLite: + https://www.sqlite.org/ +.. _FHS: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html +.. _`/var/cache`: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html +.. _`/tmp`: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html +.. _TOML: + https://toml.io/en/ +.. _HTTP Cache-Control max-age: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#response_directives +.. _data URL: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs +.. _FQN: https://en.wikipedia.org/wiki/Fully_qualified_name + diff --git a/_sources/admin/searx.limiter.rst.txt b/_sources/admin/searx.limiter.rst.txt new file mode 100644 index 000000000..c23635571 --- /dev/null +++ b/_sources/admin/searx.limiter.rst.txt @@ -0,0 +1,17 @@ +.. _limiter: + +======= +Limiter +======= + +.. sidebar:: info + + The limiter requires a :ref:`Redis <settings redis>` database. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.limiter + :members: diff --git a/_sources/admin/settings/index.rst.txt b/_sources/admin/settings/index.rst.txt new file mode 100644 index 000000000..acc91dbdd --- /dev/null +++ b/_sources/admin/settings/index.rst.txt @@ -0,0 +1,27 @@ +.. _searxng settings.yml: + +======== +Settings +======== + +.. sidebar:: Further reading .. + + - :ref:`engine settings` + - :ref:`engine file` + +.. toctree:: + :maxdepth: 2 + + settings + settings_engine + settings_brand + settings_general + settings_search + settings_server + settings_ui + settings_redis + settings_outgoing + settings_categories_as_tabs + + + diff --git a/_sources/admin/settings/settings.rst.txt b/_sources/admin/settings/settings.rst.txt new file mode 100644 index 000000000..9c6fb01be --- /dev/null +++ b/_sources/admin/settings/settings.rst.txt @@ -0,0 +1,117 @@ +.. _settings.yml: + +================ +``settings.yml`` +================ + +This page describe the options possibilities of the :origin:`searx/settings.yml` +file. + +.. sidebar:: Further reading .. + + - :ref:`use_default_settings.yml` + - :ref:`search API` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. _settings location: + +settings.yml location +===================== + +The initial ``settings.yml`` we be load from these locations: + +1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable. +2. ``/etc/searxng/settings.yml`` + +If these files don't exist (or are empty or can't be read), SearXNG uses the +:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to +see how you can simplify your *user defined* ``settings.yml``. + + + +.. _settings use_default_settings: + +use_default_settings +==================== + +.. sidebar:: ``use_default_settings: true`` + + - :ref:`settings location` + - :ref:`use_default_settings.yml` + - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` + +The user defined ``settings.yml`` is loaded from the :ref:`settings location` +and can relied on the default configuration :origin:`searx/settings.yml` using: + + ``use_default_settings: true`` + +``server:`` + In the following example, the actual settings are the default settings defined + in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and + the ``bind_address``: + + .. code:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + bind_address: "0.0.0.0" + +``engines:`` + With ``use_default_settings: true``, each settings can be override in a + similar way, the ``engines`` section is merged according to the engine + ``name``. In this example, SearXNG will load all the default engines, will + enable the ``bing`` engine and define a :ref:`token <private engines>` for + the arch linux engine: + + .. code:: yaml + + use_default_settings: true + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + - name: bing + disabled: false + + +``engines:`` / ``remove:`` + It is possible to remove some engines from the default settings. The following + example is similar to the above one, but SearXNG doesn't load the the google + engine: + + .. code:: yaml + + use_default_settings: + engines: + remove: + - google + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``keep_only:`` + As an alternative, it is possible to specify the engines to keep. In the + following example, SearXNG has only two engines: + + .. code:: yaml + + use_default_settings: + engines: + keep_only: + - google + - duckduckgo + server: + secret_key: "ultrasecretkey" # change this! + engines: + - name: google + tokens: ['$ecretValue'] + - name: duckduckgo + tokens: ['$ecretValue'] diff --git a/_sources/admin/settings/settings_brand.rst.txt b/_sources/admin/settings/settings_brand.rst.txt new file mode 100644 index 000000000..0f1a0d9a9 --- /dev/null +++ b/_sources/admin/settings/settings_brand.rst.txt @@ -0,0 +1,25 @@ +.. _settings brand: + +========== +``brand:`` +========== + +.. code:: yaml + + brand: + issue_url: https://github.com/searxng/searxng/issues + docs_url: https://docs.searxng.org + public_instances: https://searx.space + wiki_url: https://github.com/searxng/searxng/wiki + +``issue_url`` : + If you host your own issue tracker change this URL. + +``docs_url`` : + If you host your own documentation change this URL. + +``public_instances`` : + If you host your own https://searx.space change this URL. + +``wiki_url`` : + Link to your wiki (or ``false``) diff --git a/_sources/admin/settings/settings_categories_as_tabs.rst.txt b/_sources/admin/settings/settings_categories_as_tabs.rst.txt new file mode 100644 index 000000000..732d04678 --- /dev/null +++ b/_sources/admin/settings/settings_categories_as_tabs.rst.txt @@ -0,0 +1,31 @@ +.. _settings categories_as_tabs: + +======================= +``categories_as_tabs:`` +======================= + +A list of the categories that are displayed as tabs in the user interface. +Categories not listed here can still be searched with the :ref:`search-syntax`. + +.. code:: yaml + + categories_as_tabs: + general: + images: + videos: + news: + map: + music: + it: + science: + files: + social media: + +Engines are added to ``categories:`` (compare :ref:`engine categories`), the +categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If +there are no active engines in a category, the tab is not displayed (e.g. if a +user disables all engines in a category). + +On the preferences page (``/preferences``) -- under *engines* -- there is an +additional tab, called *other*. In this tab are all engines listed that are not +in one of the UI tabs (not included in ``categories_as_tabs``). diff --git a/_sources/admin/settings/settings_engine.rst.txt b/_sources/admin/settings/settings_engine.rst.txt new file mode 100644 index 000000000..454fcd241 --- /dev/null +++ b/_sources/admin/settings/settings_engine.rst.txt @@ -0,0 +1,244 @@ +.. _settings engine: + +=========== +``engine:`` +=========== + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`engines-dev` + +In the code example below a *full fledged* example of a YAML setup from a dummy +engine is shown. Most of the options have a default value or even are optional. + +.. hint:: + + A few more options are possible, but they are pretty specific to some + engines (:ref:`engine implementations`). + +.. code:: yaml + + - name: example engine + engine: example + shortcut: demo + base_url: 'https://{language}.example.com/' + send_accept_language_header: false + categories: general + timeout: 3.0 + api_key: 'apikey' + disabled: false + language: en_US + tokens: [ 'my-secret-token' ] + weight: 1 + display_error_messages: true + about: + website: https://example.com + wikidata_id: Q306656 + official_api_documentation: https://example.com/api-doc + use_official_api: true + require_api_key: true + results: HTML + + # overwrite values from section 'outgoing:' + enable_http2: false + retries: 1 + max_connections: 100 + max_keepalive_connections: 10 + keepalive_expiry: 5.0 + using_tor_proxy: false + proxies: + http: + - http://proxy1:8080 + - http://proxy2:8080 + https: + - http://proxy1:8080 + - http://proxy2:8080 + - socks5://user:password@proxy3:1080 + - socks5h://user:password@proxy4:1080 + + # other network settings + enable_http: false + retry_on_http_error: true # or 403 or [404, 429] + + +``name`` : + Name that will be used across SearXNG to define this engine. In settings, on + the result page... + +``engine`` : + Name of the python file used to handle requests and responses to and from this + search engine. + +``shortcut`` : + Code used to execute bang requests (in this case using ``!bi``) + +``base_url`` : optional + Part of the URL that should be stable across every request. Can be useful to + use multiple sites using only one engine, or updating the site URL without + touching at the code. + +``send_accept_language_header`` : + Several engines that support languages (or regions) deal with the HTTP header + ``Accept-Language`` to build a response that fits to the locale. When this + option is activated, the language (locale) that is selected by the user is used + to build and send a ``Accept-Language`` header in the request to the origin + search engine. + +.. _engine categories: + +``categories`` : optional + Specifies to which categories the engine should be added. Engines can be + assigned to multiple categories. + + Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the + UI. A search in a tab (in the UI) will query all engines that are active in + this tab. In the preferences page (``/preferences``) -- under *engines* -- + users can select what engine should be active when querying in this tab. + + Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines + in a category, regardless of whether they are active or not, or whether they + are in a tab of the UI or not. For example, ``!dictionaries`` can be used to + query all search engines in that category (group). + +``timeout`` : optional + Timeout of the search with the current search engine. Overwrites + ``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will + modify the global timeout of SearXNG.** + +``api_key`` : optional + In a few cases, using an API needs the use of a secret key. How to obtain them + is described in the file. + +``disabled`` : optional + To disable by default the engine, but not deleting it. It will allow the user + to manually activate it in the settings. + +``inactive``: optional + Remove the engine from the settings (*disabled & removed*). + +``language`` : optional + If you want to use another language for a specific engine, you can define it + by using the ISO code of language (and region), like ``fr``, ``en-US``, + ``de-DE``. + +``tokens`` : optional + A list of secret tokens to make this engine *private*, more details see + :ref:`private engines`. + +``weight`` : default ``1`` + Weighting of the results of this engine. + +``display_error_messages`` : default ``true`` + When an engine returns an error, the message is displayed on the user interface. + +``network`` : optional + Use the network configuration from another engine. + In addition, there are two default networks: + + - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses) + - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses) + +``enable_http`` : optional + Enable HTTP for this engine (by default only HTTPS is enabled). + +``retry_on_http_error`` : optional + Retry request on some HTTP status code. + + Example: + + * ``true`` : on HTTP status code between 400 and 599. + * ``403`` : on HTTP status code 403. + * ``[403, 429]``: on HTTP status code 403 and 429. + +``proxies`` : + Overwrites proxy settings from :ref:`settings outgoing`. + +``using_tor_proxy`` : + Using tor proxy (``true``) or not (``false``) for this engine. The default is + taken from ``using_tor_proxy`` of the :ref:`settings outgoing`. + +.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration + +``max_keepalive_connection#s`` : + `Pool limit configuration`_, overwrites value ``pool_maxsize`` from + :ref:`settings outgoing` for this engine. + +``max_connections`` : + `Pool limit configuration`_, overwrites value ``pool_connections`` from + :ref:`settings outgoing` for this engine. + +``keepalive_expiry`` : + `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from + :ref:`settings outgoing` for this engine. + + +.. _private engines: + +Private Engines (``tokens``) +============================ + +Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through :ref:`offline engines`. Or they would +rather share engines only with their trusted friends or colleagues. + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +To solve this issue the concept of *private engines* exists. + +A new option was added to engines named `tokens`. It expects a list of strings. +If the user making a request presents one of the tokens of an engine, they can +access information about the engine and make search requests. + +Example configuration to restrict access to the Arch Linux Wiki engine: + +.. code:: yaml + + - name: arch linux wiki + engine: archlinux + shortcut: al + tokens: [ 'my-secret-token' ] + +Unless a user has configured the right token, the engine is going to be hidden +from them. It is not going to be included in the list of engines on the +Preferences page and in the output of `/config` REST API call. + +Tokens can be added to one's configuration on the Preferences page under "Engine +tokens". The input expects a comma separated list of strings. + +The distribution of the tokens from the administrator to the users is not carved +in stone. As providing access to such engines implies that the admin knows and +trusts the user, we do not see necessary to come up with a strict process. +Instead, we would like to add guidelines to the documentation of the feature. + + +Example: Multilingual Search +============================ + +SearXNG does not support true multilingual search. You have to use the language +prefix in your search query when searching in a different language. + +But there is a workaround: By adding a new search engine with a different +language, SearXNG will search in your default and other language. + +Example configuration in settings.yml for a German and English speaker: + +.. code-block:: yaml + + search: + default_lang : "de" + ... + + engines: + - name : google english + engine : google + language : en + ... + +When searching, the default google engine will return German results and +"google english" will return English results. + diff --git a/_sources/admin/settings/settings_general.rst.txt b/_sources/admin/settings/settings_general.rst.txt new file mode 100644 index 000000000..75acb4f6d --- /dev/null +++ b/_sources/admin/settings/settings_general.rst.txt @@ -0,0 +1,42 @@ +.. _settings general: + +============ +``general:`` +============ + +.. code:: yaml + + general: + debug: false + instance_name: "SearXNG" + privacypolicy_url: false + donation_url: false + contact_url: false + enable_metrics: true + open_metrics: '' + +``debug`` : ``$SEARXNG_DEBUG`` + Allow a more detailed log if you run SearXNG directly. Display *detailed* error + messages in the browser too, so this must be deactivated in production. + +``donation_url`` : + Set value to ``true`` to use your own donation page written in the + :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable + the donation link altogether. + +``privacypolicy_url``: + Link to privacy policy. + +``contact_url``: + Contact ``mailto:`` address or WEB form. + +``enable_metrics``: + Enabled by default. Record various anonymous metrics available at ``/stats``, + ``/stats/errors`` and ``/preferences``. + +``open_metrics``: + Disabled by default. Set to a secret password to expose an + `OpenMetrics API <https://github.com/prometheus/OpenMetrics>`_ at ``/metrics``, + e.g. for usage with Prometheus. The ``/metrics`` endpoint is using HTTP Basic Auth, + where the password is the value of ``open_metrics`` set above. The username used for + Basic Auth can be randomly chosen as only the password is being validated. diff --git a/_sources/admin/settings/settings_outgoing.rst.txt b/_sources/admin/settings/settings_outgoing.rst.txt new file mode 100644 index 000000000..7d49ab789 --- /dev/null +++ b/_sources/admin/settings/settings_outgoing.rst.txt @@ -0,0 +1,110 @@ +.. _settings outgoing: + +============= +``outgoing:`` +============= + +Communication with search engines. + +.. code:: yaml + + outgoing: + request_timeout: 2.0 # default timeout in seconds, can be override by engine + max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix: "" # information like an email address to the administrator + pool_connections: 100 # Maximum number of allowable connections, or null + # for no limits. The default is 100. + pool_maxsize: 10 # Number of allowable keep-alive connections, or null + # to always allow. The default is 10. + enable_http2: true # See https://www.python-httpx.org/http2/ + # uncomment below section if you want to use a custom server certificate + # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults + # and https://www.python-httpx.org/compatibility/#ssl-configuration + # verify: ~/.mitmproxy/mitmproxy-ca-cert.cer + # + # uncomment below section if you want to use a proxyq see: SOCKS proxies + # https://2.python-requests.org/en/latest/user/advanced/#proxies + # are also supported: see + # https://2.python-requests.org/en/latest/user/advanced/#socks + # + # proxies: + # all://: + # - http://proxy1:8080 + # - http://proxy2:8080 + # + # using_tor_proxy: true + # + # Extra seconds to add in order to account for the time taken by the proxy + # + # extra_proxy_timeout: 10.0 + # + +``request_timeout`` : + Global timeout of the requests made to others engines in seconds. A bigger + timeout will allow to wait for answers from slow engines, but in consequence + will slow SearXNG reactivity (the result page may take the time specified in the + timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`. + +``useragent_suffix`` : + Suffix to the user-agent SearXNG uses to send requests to others engines. If an + engine wish to block you, a contact info here may be useful to avoid that. + +.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration + +``pool_maxsize``: + Number of allowable keep-alive connections, or ``null`` to always allow. The + default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_. + +``pool_connections`` : + Maximum number of allowable connections, or ``null`` # for no limits. The + default is 100. See ``max_connections`` `Pool limit configuration`_. + +``keepalive_expiry`` : + Number of seconds to keep a connection in the pool. By default 5.0 seconds. + See ``keepalive_expiry`` `Pool limit configuration`_. + +.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying + +``proxies`` : + Define one or more proxies you wish to use, see `httpx proxies`_. + If there are more than one proxy for one protocol (http, https), + requests to the engines are distributed in a round-robin fashion. + +``source_ips`` : + If you use multiple network interfaces, define from which IP the requests must + be made. Example: + + * ``0.0.0.0`` any local IPv4 address. + * ``::`` any local IPv6 address. + * ``192.168.0.1`` + * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses + * ``fe80::60a2:1691:e5a2:ee1f`` + * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network. + * ``[ 192.168.0.1, fe80::/126 ]`` + +``retries`` : + Number of retry in case of an HTTP error. On each retry, SearXNG uses an + different proxy and source ip. + +``enable_http2`` : + Enable by default. Set to ``false`` to disable HTTP/2. + +.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults +.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration + +``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR`` + Allow to specify a path to certificate. + see `httpx verification defaults`_. + + In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and + ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables. + see `httpx ssl configuration`_. + +``max_redirects`` : + 30 by default. Maximum redirect before it is an error. + +``using_tor_proxy`` : + Using tor proxy (``true``) or not (``false``) for all engines. The default is + ``false`` and can be overwritten in the :ref:`settings engine` + + diff --git a/_sources/admin/settings/settings_redis.rst.txt b/_sources/admin/settings/settings_redis.rst.txt new file mode 100644 index 000000000..9fb067553 --- /dev/null +++ b/_sources/admin/settings/settings_redis.rst.txt @@ -0,0 +1,49 @@ +.. _settings redis: + +========== +``redis:`` +========== + +.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url + +A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you +will find a description to test your redis connection in SearXNG. When using +sockets, don't forget to check the access rights on the socket:: + + ls -la /usr/local/searxng-redis/run/redis.sock + srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock + +In this example read/write access is given to the *searxng-redis* group. To get +access rights to redis instance (the socket), your SearXNG (or even your +developer) account needs to be added to the *searxng-redis* group. + +``url`` : ``$SEARXNG_REDIS_URL`` + URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`:: + + redis://[[username]:[password]]@localhost:6379/0 + rediss://[[username]:[password]]@localhost:6379/0 + unix://[[username]:[password]]@/path/to/socket.sock?db=0 + +.. _Redis Developer Notes: + +Redis Developer Notes +===================== + +To set up a local redis instance, first set the socket path of the Redis DB +in your YAML setting: + +.. code:: yaml + + redis: + url: unix:///usr/local/searxng-redis/run/redis.sock?db=0 + +Then use the following commands to install the redis instance (:ref:`manage +redis.help`): + +.. code:: sh + + $ ./manage redis.build + $ sudo -H ./manage redis.install + $ sudo -H ./manage redis.addgrp "${USER}" + # don't forget to logout & login to get member of group + diff --git a/_sources/admin/settings/settings_search.rst.txt b/_sources/admin/settings/settings_search.rst.txt new file mode 100644 index 000000000..b8f37b423 --- /dev/null +++ b/_sources/admin/settings/settings_search.rst.txt @@ -0,0 +1,104 @@ +.. _settings search: + +=========== +``search:`` +=========== + +.. code:: yaml + + search: + safe_search: 0 + autocomplete: "" + favicon_resolver: "" + default_lang: "" + ban_time_on_fail: 5 + max_ban_time_on_fail: 120 + suspended_times: + SearxEngineAccessDenied: 86400 + SearxEngineCaptcha: 86400 + SearxEngineTooManyRequests: 3600 + cf_SearxEngineCaptcha: 1296000 + cf_SearxEngineAccessDenied: 86400 + recaptcha_SearxEngineCaptcha: 604800 + formats: + - html + +``safe_search``: + Filter results. + + - ``0``: None + - ``1``: Moderate + - ``2``: Strict + +``autocomplete``: + Existing autocomplete backends, leave blank to turn it off. + + - ``dbpedia`` + - ``duckduckgo`` + - ``google`` + - ``mwmbl`` + - ``startpage`` + - ``swisscows`` + - ``qwant`` + - ``wikipedia`` + +``favicon_resolver``: + To activate favicons in SearXNG's result list select a default + favicon-resolver, leave blank to turn off the feature. Don't activate the + favicons before reading the :ref:`Favicons documentation <favicons>`. + +``default_lang``: + Default search language - leave blank to detect from browser information or + use codes from :origin:`searx/languages.py`. + +``languages``: + List of available languages - leave unset to use all codes from + :origin:`searx/languages.py`. Otherwise list codes of available languages. + The ``all`` value is shown as the ``Default language`` in the user interface + (in most cases, it is meant to send the query without a language parameter ; + in some cases, it means the English language) Example: + + .. code:: yaml + + languages: + - all + - en + - en-US + - de + - it-IT + - fr + - fr-BE + +``ban_time_on_fail``: + Ban time in seconds after engine errors. + +``max_ban_time_on_fail``: + Max ban time in seconds after engine errors. + +``suspended_times``: + Engine suspension time after error (in seconds; set to 0 to disable) + + ``SearxEngineAccessDenied``: 86400 + For error "Access denied" and "HTTP error [402, 403]" + + ``SearxEngineCaptcha``: 86400 + For error "CAPTCHA" + + ``SearxEngineTooManyRequests``: 3600 + For error "Too many request" and "HTTP error 429" + + Cloudflare CAPTCHA: + - ``cf_SearxEngineCaptcha``: 1296000 + - ``cf_SearxEngineAccessDenied``: 86400 + + Google CAPTCHA: + - ``recaptcha_SearxEngineCaptcha``: 604800 + +``formats``: + Result formats available from web, remove format to deny access (use lower + case). + + - ``html`` + - ``csv`` + - ``json`` + - ``rss`` diff --git a/_sources/admin/settings/settings_server.rst.txt b/_sources/admin/settings/settings_server.rst.txt new file mode 100644 index 000000000..cf480683e --- /dev/null +++ b/_sources/admin/settings/settings_server.rst.txt @@ -0,0 +1,57 @@ +.. _settings server: + +=========== +``server:`` +=========== + +.. code:: yaml + + server: + base_url: http://example.org/location # change this! + port: 8888 + bind_address: "127.0.0.1" + secret_key: "ultrasecretkey" # change this! + limiter: false + public_instance: false + image_proxy: false + default_http_headers: + X-Content-Type-Options : nosniff + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer + +``base_url`` : ``$SEARXNG_URL`` + The base URL where SearXNG is deployed. Used to create correct inbound links. + +``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` + Port number and *bind address* of the SearXNG web application if you run it + directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG + services running behind a proxy and using socket communications. + +``secret_key`` : ``$SEARXNG_SECRET`` + Used for cryptography purpose. + +``limiter`` : ``$SEARXNG_LIMITER`` + Rate limit the number of request on the instance, block some bots. The + :ref:`limiter` requires a :ref:`settings redis` database. + +.. _public_instance: + +``public_instance`` : ``$SEARXNG_PUBLIC_INSTANCE`` + + Setting that allows to enable features specifically for public instances (not + needed for local usage). By set to ``true`` the following features are + activated: + + - :py:obj:`searx.botdetection.link_token` in the :ref:`limiter` + +.. _image_proxy: + +``image_proxy`` : ``$SEARXNG_IMAGE_PROXY`` + Allow your instance of SearXNG of being able to proxy images. Uses memory space. + +.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers + +``default_http_headers`` : + Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__ + diff --git a/_sources/admin/settings/settings_ui.rst.txt b/_sources/admin/settings/settings_ui.rst.txt new file mode 100644 index 000000000..75f617683 --- /dev/null +++ b/_sources/admin/settings/settings_ui.rst.txt @@ -0,0 +1,74 @@ +.. _settings ui: + +======= +``ui:`` +======= + +.. _cache busting: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting + +.. code:: yaml + + ui: + static_use_hash: false + default_locale: "" + query_in_title: false + infinite_scroll: false + center_alignment: false + cache_url: https://web.archive.org/web/ + default_theme: simple + theme_args: + simple_style: auto + search_on_category_select: true + hotkeys: default + url_formatting: pretty + +.. _static_use_hash: + +``static_use_hash`` : ``$SEARXNG_STATIC_USE_HASH`` + Enables `cache busting`_ of static files. + +``default_locale`` : + SearXNG interface language. If blank, the locale is detected by using the + browser language. If it doesn't work, or you are deploying a language + specific instance of searx, a locale can be defined using an ISO language + code, like ``fr``, ``en``, ``de``. + +``query_in_title`` : + When true, the result page's titles contains the query it decreases the + privacy, since the browser can records the page titles. + +``infinite_scroll``: + When true, automatically loads the next page when scrolling to bottom of the current page. + +``center_alignment`` : default ``false`` + When enabled, the results are centered instead of being in the left (or RTL) + side of the screen. This setting only affects the *desktop layout* + (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`) + +.. cache_url: + +``cache_url`` : ``https://web.archive.org/web/`` + URL prefix of the internet archive or cache, don't forget trailing slash (if + needed). The default is https://web.archive.org/web/ alternatives are: + + - https://webcache.googleusercontent.com/search?q=cache: + - https://archive.today/ + +``default_theme`` : + Name of the theme you want to use by default on your SearXNG instance. + +``theme_args.simple_style``: + Style of simple theme: ``auto``, ``light``, ``dark``, ``black`` + +``results_on_new_tab``: + Open result links in a new tab by default. + +``search_on_category_select``: + Perform search immediately if a category selected. Disable to select multiple categories. + +``hotkeys``: + Hotkeys to use in the search interface: ``default``, ``vim`` (Vim-like). + +``url_formatting``: + Formatting type to use for result URLs: ``pretty``, ``full`` or ``host``. diff --git a/_sources/admin/update-searxng.rst.txt b/_sources/admin/update-searxng.rst.txt new file mode 100644 index 000000000..b9d15c3f7 --- /dev/null +++ b/_sources/admin/update-searxng.rst.txt @@ -0,0 +1,138 @@ +.. _searxng maintenance: + +=================== +SearXNG maintenance +=================== + +.. sidebar:: further read + + - :ref:`toolboxing` + - :ref:`uWSGI maintenance` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. _update searxng: + +How to update +============= + +How to update depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use the ``update`` command from the :ref:`searxng.sh` +script. + +.. code:: sh + + sudo -H ./utils/searxng.sh instance update + +.. _inspect searxng: + +How to inspect & debug +====================== + +How to debug depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use the ``inspect`` command from the :ref:`searxng.sh` +script. + +.. code:: sh + + sudo -H ./utils/searxng.sh instance inspect + +.. _migrate and stay tuned: + +Migrate and stay tuned! +======================= + +.. sidebar:: info + + - :pull:`1332` + - :pull:`456` + - :pull:`A comment about rolling release <446#issuecomment-954730358>` + +SearXNG is a *rolling release*; each commit to the master branch is a release. +SearXNG is growing rapidly, the services and opportunities are change every now +and then, to name just a few: + +- Bot protection has been switched from filtron to SearXNG's :ref:`limiter + <limiter>`, this requires a :ref:`Redis <settings redis>` database. + +- The image proxy morty is no longer needed, it has been replaced by the + :ref:`image proxy <image_proxy>` from SearXNG. + +- To save bandwidth :ref:`cache busting <static_use_hash>` has been implemented. + To get in use, the ``static-expires`` needs to be set in the :ref:`uwsgi + setup`. + +To stay tuned and get in use of the new features, instance maintainers have to +update the SearXNG code regularly (see :ref:`update searxng`). As the above +examples show, this is not always enough, sometimes services have to be set up +or reconfigured and sometimes services that are no longer needed should be +uninstalled. + +.. hint:: + + First of all: SearXNG is installed by the script :ref:`searxng.sh`. If you + have old filtron, morty or searx setup you should consider complete + uninstall/reinstall. + +Here you will find a list of changes that affect the infrastructure. Please +check to what extent it is necessary to update your installations: + +:pull:`1595`: ``[fix] uWSGI: increase buffer-size`` + Re-install uWSGI (:ref:`searxng.sh`) or fix your uWSGI ``searxng.ini`` + file manually. + + +remove obsolete services +------------------------ + +If your searx instance was installed *"Step by step"* or by the *"Installation +scripts"*, you need to undo the installation procedure completely. If you have +morty & filtron installed, it is recommended to uninstall these services also. +In case of scripts, to uninstall use the scripts from the origin you installed +searx from or try:: + + $ sudo -H ./utils/filtron.sh remove all + $ sudo -H ./utils/morty.sh remove all + $ sudo -H ./utils/searx.sh remove all + +.. hint:: + + If you are migrate from searx take into account that the ``.config.sh`` is no + longer used. + +If you upgrade from searx or from before :pull:`1332` has been merged and you +have filtron and/or morty installed, don't forget to remove HTTP sites. + +Apache:: + + $ sudo -H ./utils/filtron.sh apache remove + $ sudo -H ./utils/morty.sh apache remove + +nginx:: + + $ sudo -H ./utils/filtron.sh nginx remove + $ sudo -H ./utils/morty.sh nginx remove + + + +Check after Installation +------------------------ + +Once you have done your installation, you can run a SearXNG *check* procedure, +to see if there are some left overs. In this example there exists a *old* +``/etc/searx/settings.yml``:: + + $ sudo -H ./utils/searxng.sh instance check + + SearXNG checks + -------------- + ERROR: settings.yml in /etc/searx/ is deprecated, move file to folder /etc/searxng/ + INFO: [OK] (old) account 'searx' does not exists + INFO: [OK] (old) account 'filtron' does not exists + INFO: [OK] (old) account 'morty' does not exists + ... + INFO searx.redisdb : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock' + INFO searx.redisdb : connected to Redis diff --git a/_sources/dev/contribution_guide.rst.txt b/_sources/dev/contribution_guide.rst.txt new file mode 100644 index 000000000..df5200637 --- /dev/null +++ b/_sources/dev/contribution_guide.rst.txt @@ -0,0 +1,190 @@ +.. _how to contribute: + +================= +How to contribute +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Prime directives: Privacy, Hackability +====================================== + +SearXNG has two prime directives, **privacy-by-design and hackability** . The +hackability comes in three levels: + +- support of search engines +- plugins to alter search behaviour +- hacking SearXNG itself + +Note the lack of "world domination" among the directives. SearXNG has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +"privacy" deserves a separate chapter, as it's quite uncommon unfortunately. + +Privacy-by-design +----------------- + +SearXNG was born out of the need for a **privacy-respecting** search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities. + +A few widely used features work differently or turned off by default or not +implemented at all **as a consequence of privacy-by-design**. + +If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained. + +Also, if you think that something works weird with searx, it's might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST`` +requests in various browsers). + +Remember the other prime directive of SearXNG is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it. + + *Happy hacking.* + +Code +==== + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ +.. _Conventional Commits: https://www.conventionalcommits.org/ +.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages +.. _Structural split of changes: + https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes +.. _gitmoji: https://gitmoji.carloscuesta.me/ +.. _Semantic PR: https://github.com/zeke/semantic-pull-requests + +.. sidebar:: Create good commits! + + - `Structural split of changes`_ + - `Conventional Commits`_ + - `Git Commit Good Practice`_ + - some like to use: gitmoji_ + - not yet active: `Semantic PR`_ + +In order to submit a patch, please follow the steps below: + +- Follow coding conventions. + + - PEP8_ standards apply, except the convention of line length + - Maximum line length is 120 characters + +- The cardinal rule for creating good commits is to ensure there is only one + *logical change* per commit / read `Structural split of changes`_ + +- Check if your code breaks existing tests. If so, update the tests or fix your + code. + +- If your code can be unit-tested, add unit tests. + +- Add yourself to the :origin:`AUTHORS.rst` file. + +- Choose meaningful commit messages, read `Conventional Commits`_ + + .. code:: + + <type>[optional scope]: <description> + + [optional body] + + [optional footer(s)] + +- Create a pull request. + +For more help on getting started with SearXNG development, see :ref:`devquickstart`. + + +Translation +=========== + +Translation currently takes place on :ref:`weblate <translation>`. + + +.. _contrib docs: + +Documentation +============= + +.. _Sphinx: https://www.sphinx-doc.org +.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html + +.. sidebar:: The reST sources + + has been moved from ``gh-branch`` into ``master`` (:origin:`docs`). + +The documentation is built using Sphinx_. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +our :ref:`makefile`. + +Here is an example which makes a complete rebuild: + +.. code:: sh + + $ make docs.clean docs.html + ... + The HTML pages are in dist/docs. + +.. _make docs.live: + +live build +---------- + +.. _sphinx-autobuild: + https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md + +.. sidebar:: docs.clean + + It is recommended to assert a complete rebuild before deploying (use + ``docs.clean``). + +Live build is like WYSIWYG. If you want to edit the documentation, its +recommended to use. The Makefile target ``docs.live`` builds the docs, opens +URL in your favorite browser and rebuilds every time a reST file has been +changed (:ref:`make docs.clean`). + +.. code:: sh + + $ make docs.live + ... + The HTML pages are in dist/docs. + ... Serving on http://0.0.0.0:8000 + ... Start watching changes + +Live builds are implemented by sphinx-autobuild_. Use environment +``$(SPHINXOPTS)`` to pass arguments to the sphinx-autobuild_ command. Except +option ``--host`` (which is always set to ``0.0.0.0``) you can pass any +argument. E.g to find and use a free port, use: + +.. code:: sh + + $ SPHINXOPTS="--port 0" make docs.live + ... + ... Serving on http://0.0.0.0:50593 + ... + + +.. _deploy on github.io: + +deploy on github.io +------------------- + +To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make +docs.gh-pages`, which builds the documentation and runs all the needed git add, +commit and push: + +.. code:: sh + + $ make docs.clean docs.gh-pages + +.. attention:: + + If you are working in your own brand, don't forget to adjust your + :ref:`settings brand`. diff --git a/_sources/dev/engines/demo/demo_offline.rst.txt b/_sources/dev/engines/demo/demo_offline.rst.txt new file mode 100644 index 000000000..1b4cb887f --- /dev/null +++ b/_sources/dev/engines/demo/demo_offline.rst.txt @@ -0,0 +1,14 @@ +.. _demo offline engine: + +=================== +Demo Offline Engine +=================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_offline + :members: + diff --git a/_sources/dev/engines/demo/demo_online.rst.txt b/_sources/dev/engines/demo/demo_online.rst.txt new file mode 100644 index 000000000..9b94207f7 --- /dev/null +++ b/_sources/dev/engines/demo/demo_online.rst.txt @@ -0,0 +1,14 @@ +.. _demo online engine: + +================== +Demo Online Engine +================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_online + :members: + diff --git a/_sources/dev/engines/engine_overview.rst.txt b/_sources/dev/engines/engine_overview.rst.txt new file mode 100644 index 000000000..1a6270a3b --- /dev/null +++ b/_sources/dev/engines/engine_overview.rst.txt @@ -0,0 +1,571 @@ +.. _engines-dev: + +=============== +Engine Overview +=============== + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`settings engine` + +SearXNG is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between SearXNG and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. + +.. _general engine configuration: + +General Engine Configuration +============================ + +It is required to tell SearXNG the type of results the engine provides. The +arguments can be set in the engine file or in the settings file (normally +``settings.yml``). The arguments in the settings file override the ones in the +engine file. + +It does not matter if an option is stored in the engine file or in the settings. +However, the standard way is the following: + +.. _engine file: + +Engine File +----------- + +.. table:: Common options in the engine module + :width: 100% + + ======================= =========== ======================================================== + argument type information + ======================= =========== ======================================================== + categories list categories, in which the engine is working + paging boolean support multiple pages + time_range_support boolean support search time range + engine_type str - ``online`` :ref:`[ref] <online engines>` by + default, other possibles values are: + - ``offline`` :ref:`[ref] <offline engines>` + - ``online_dictionary`` :ref:`[ref] <online dictionary>` + - ``online_currency`` :ref:`[ref] <online currency>` + - ``online_url_search`` :ref:`[ref] <online url search>` + ======================= =========== ======================================================== + +.. _engine settings: + +Engine ``settings.yml`` +----------------------- + +For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`. + +.. table:: Common options in the engine setup (``settings.yml``) + :width: 100% + + ======================= =========== ================================================== + argument type information + ======================= =========== ================================================== + name string name of search-engine + engine string name of searxng-engine (file name without ``.py``) + enable_http bool enable HTTP (by default only HTTPS is enabled). + shortcut string shortcut of search-engine + timeout string specific timeout for search-engine + display_error_messages boolean display error messages on the web UI + proxies dict set proxies for a specific engine + (e.g. ``proxies : {http: socks5://proxy:port, + https: socks5://proxy:port}``) + ======================= =========== ================================================== + +.. _engine overrides: + +Overrides +--------- + +A few of the options have default values in the namespace of the engine's python +module, but are often overwritten by the settings. If ``None`` is assigned to an +option in the engine file, it has to be redefined in the settings, otherwise +SearXNG will not start with that engine (global names with a leading underline can +be ``None``). + +Here is an very simple example of the global names in the namespace of engine's +module: + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + _non_overwritten_global = 'foo' + + +.. table:: The naming of overrides is arbitrary / recommended overrides are: + :width: 100% + + ======================= =========== =========================================== + argument type information + ======================= =========== =========================================== + base_url string base-url, can be overwritten to use same + engine on other URL + number_of_results int maximum number of results per request + language string ISO code of language and country like en_US + api_key string api-key if required by engine + ======================= =========== =========================================== + +.. _engine request: + +Making a Request +================ + +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. + +.. _engine request arguments: + +Passed Arguments (request) +-------------------------- + +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. + +.. _engine request online: + +.. table:: If the ``engine_type`` is :py:obj:`online + <searx.search.processors.online.OnlineProcessor.get_params>` + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + url str ``''`` + method str ``'GET'`` + headers set ``{}`` + data set ``{}`` + cookies set ``{}`` + verify bool ``True`` + headers.User-Agent str a random User-Agent + category str current category, like ``'general'`` + safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) + time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` + pageno int current pagenumber + searxng_locale str SearXNG's locale selected by user. Specific language code like + ``'en'``, ``'en-US'``, or ``'all'`` if unspecified. + ====================== ============== ======================================================================== + + +.. _engine request online_dictionary: + +.. table:: If the ``engine_type`` is :py:obj:`online_dictionary + <searx.search.processors.online_dictionary.OnlineDictionaryProcessor.get_params>`, + in addition to the :ref:`online <engine request online>` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + from_lang str specific language code like ``'en_US'`` + to_lang str specific language code like ``'en_US'`` + query str the text query without the languages + ====================== ============== ======================================================================== + +.. _engine request online_currency: + +.. table:: If the ``engine_type`` is :py:obj:`online_currency + <searx.search.processors.online_currency.OnlineCurrencyProcessor.get_params>`, + in addition to the :ref:`online <engine request online>` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + amount float the amount to convert + from str ISO 4217 code + to str ISO 4217 code + from_name str currency name + to_name str currency name + ====================== ============== ======================================================================== + +.. _engine request online_url_search: + +.. table:: If the ``engine_type`` is :py:obj:`online_url_search + <searx.search.processors.online_url_search.OnlineUrlSearchProcessor.get_params>`, + in addition to the :ref:`online <engine request online>` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + search_url dict URLs from the search query: + + .. code:: python + + { + 'http': str, + 'ftp': str, + 'data:image': str + } + ====================== ============== ======================================================================== + +Specify Request +--------------- + +The function :py:func:`def request(query, params): +<searx.engines.demo_online.request>` always returns the ``params`` variable, the +following parameters can be used to specify a search request: + +.. table:: + :width: 100% + + =================== =========== ========================================================================== + argument type information + =================== =========== ========================================================================== + url str requested url + method str HTTP request method + headers set HTTP header information + data set HTTP data information + cookies set HTTP cookies + verify bool Performing SSL-Validity check + allow_redirects bool Follow redirects + max_redirects int maximum redirects, hard limit + soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine + raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 + =================== =========== ========================================================================== + + +.. _engine results: +.. _engine media types: + +Result Types (``template``) +=========================== + +Each result item of an engine can be of different media-types. Currently the +following media-types are supported. To set another media-type as +:ref:`template default`, the parameter ``template`` must be set to the desired +type. + +.. _template default: + +``default`` +----------- + +.. table:: Parameter of the **default** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + ========================= ===================================================== + + +.. _template images: + +``images`` +---------- + +.. list-table:: Parameter of the **images** media type + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``images.html`` + + * - url + - :py:class:`str` + - url to the result site + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - description of the image + + * - publishedDate + - :py:class:`datetime <datetime.datetime>` + - time of publish + + * - img_src + - :py:class:`str` + - url to the result image + + * - thumbnail_src + - :py:class:`str` + - url to a small-preview image + + * - resolution + - :py:class:`str` + - the resolution of the image (e.g. ``1920 x 1080`` pixel) + + * - img_format + - :py:class:`str` + - the format of the image (e.g. ``png``) + + * - filesize + - :py:class:`str` + - size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation + (e.g. ``MB`` for 1024 \* 1024 Bytes filesize). + + +.. _template videos: + +``videos`` +---------- + +.. table:: Parameter of the **videos** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``videos.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content *(not implemented yet)* + publishedDate :py:class:`datetime.datetime`, time of publish + thumbnail string, url to a small-preview image + length :py:class:`datetime.timedelta`, duration of result + views string, view count in humanized number format + ========================= ===================================================== + + +.. _template torrent: + +``torrent`` +----------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +.. table:: Parameter of the **torrent** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``torrent.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* + seed int, number of seeder + leech int, number of leecher + filesize int, size of file in bytes + files int, number of files + magnetlink string, magnetlink_ of the result + torrentfile string, torrentfile of the result + ========================= ===================================================== + + +.. _template map: + +``map`` +------- + +.. table:: Parameter of the **map** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``map.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + latitude latitude of result (in decimal format) + longitude longitude of result (in decimal format) + boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) + geojson geojson of result (https://geojson.org/) + osm.type type of osm-object (if OSM-Result) + osm.id id of osm-object (if OSM-Result) + address.name name of object + address.road street name of object + address.house_number house number of object + address.locality city, place of object + address.postcode postcode of object + address.country country of object + ========================= ===================================================== + + +.. _template paper: + +``paper`` +--------- + +.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ +.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types + +.. list-table:: Parameter of the **paper** media type / + see `BibTeX field types`_ and `BibTeX format`_ + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``paper.html`` + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - abstract + + * - comments + - :py:class:`str` + - free text display in italic below the content + + * - tags + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - free tag list + + * - publishedDate + - :py:class:`datetime <datetime.datetime>` + - last publication date + + * - type + - :py:class:`str` + - short description of medium type, e.g. *book*, *pdf* or *html* ... + + * - authors + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - list of authors of the work (authors with a "s") + + * - editor + - :py:class:`str` + - list of editors of a book + + * - publisher + - :py:class:`str` + - name of the publisher + + * - journal + - :py:class:`str` + - name of the journal or magazine the article was + published in + + * - volume + - :py:class:`str` + - volume number + + * - pages + - :py:class:`str` + - page range where the article is + + * - number + - :py:class:`str` + - number of the report or the issue number for a journal article + + * - doi + - :py:class:`str` + - DOI number (like ``10.1038/d41586-018-07848-2``) + + * - issn + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - ISSN number like ``1476-4687`` + + * - isbn + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - ISBN number like ``9780201896831`` + + * - pdf_url + - :py:class:`str` + - URL to the full article, the PDF version + + * - html_url + - :py:class:`str` + - URL to full article, HTML version + + +.. _template packages: + +``packages`` +------------ + +.. list-table:: Parameter of the **packages** media type + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``packages.html`` + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - abstract + + * - package_name + - :py:class:`str` + - the name of the package + + * - version + - :py:class:`str` + - the current version of the package + + * - maintainer + - :py:class:`str` + - the maintainer or author of the project + + * - publishedDate + - :py:class:`datetime <datetime.datetime>` + - date of latest update or release + + * - tags + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - free tag list + + * - popularity + - :py:class:`str` + - the popularity of the package, e.g. rating or download count + + * - license_name + - :py:class:`str` + - the name of the license + + * - license_url + - :py:class:`str` + - the web location of a license copy + + * - homepage + - :py:class:`str` + - the url of the project's homepage + + * - source_code_url + - :py:class:`str` + - the location of the project's source code + + * - links + - :py:class:`dict` + - additional links in the form of ``{'link_name': 'http://example.com'}`` diff --git a/_sources/dev/engines/enginelib.rst.txt b/_sources/dev/engines/enginelib.rst.txt new file mode 100644 index 000000000..34e3250da --- /dev/null +++ b/_sources/dev/engines/enginelib.rst.txt @@ -0,0 +1,22 @@ +.. _searx.enginelib: + +============== +Engine Library +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.enginelib + :members: + +.. _searx.enginelib.traits: + + +Engine traits +============= + +.. automodule:: searx.enginelib.traits + :members: diff --git a/_sources/dev/engines/engines.rst.txt b/_sources/dev/engines/engines.rst.txt new file mode 100644 index 000000000..0f2cb1f22 --- /dev/null +++ b/_sources/dev/engines/engines.rst.txt @@ -0,0 +1,9 @@ +.. _searx.engines loader: + +======================== +SearXNG's engines loader +======================== + +.. automodule:: searx.engines + :members: + diff --git a/_sources/dev/engines/index.rst.txt b/_sources/dev/engines/index.rst.txt new file mode 100644 index 000000000..6558dc2c2 --- /dev/null +++ b/_sources/dev/engines/index.rst.txt @@ -0,0 +1,107 @@ +.. _engine implementations: + +====================== +Engine Implementations +====================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. toctree:: + :caption: Framework Components + :maxdepth: 2 + + enginelib + engines + engine_overview + + +Engine Types +============ + +The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine +determines which :ref:`search processor <searx.search.processors>` is used by +the engine. + +In this section a list of the engines that are documented is given, a complete +list of the engines can be found in the source under: :origin:`searx/engines`. + +.. _online engines: + +Online Engines +-------------- + +.. sidebar:: info + + - :py:obj:`processors.online <searx.search.processors.online>` + +.. toctree:: + :maxdepth: 1 + :glob: + + demo/demo_online + xpath + mediawiki + +.. toctree:: + :maxdepth: 1 + :glob: + + online/* + +.. _offline engines: + +Offline Engines +--------------- + +.. sidebar:: info + + - :py:obj:`processors.offline <searx.search.processors.offline>` + +.. toctree:: + :maxdepth: 1 + :glob: + + offline_concept + demo/demo_offline + offline/* + +.. _online url search: + +Online URL Search +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>` + +.. toctree:: + :maxdepth: 1 + :glob: + + online_url_search/* + +.. _online currency: + +Online Currency +--------------- + +.. sidebar:: info + + - :py:obj:`processors.online_currency <searx.search.processors.online_currency>` + +*no engine of this type is documented yet / coming soon* + +.. _online dictionary: + +Online Dictionary +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>` + +*no engine of this type is documented yet / coming soon* diff --git a/_sources/dev/engines/mediawiki.rst.txt b/_sources/dev/engines/mediawiki.rst.txt new file mode 100644 index 000000000..ce708f95b --- /dev/null +++ b/_sources/dev/engines/mediawiki.rst.txt @@ -0,0 +1,13 @@ +.. _mediawiki engine: + +================ +MediaWiki Engine +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mediawiki + :members: diff --git a/_sources/dev/engines/offline/command-line-engines.rst.txt b/_sources/dev/engines/offline/command-line-engines.rst.txt new file mode 100644 index 000000000..0a80d698e --- /dev/null +++ b/_sources/dev/engines/offline/command-line-engines.rst.txt @@ -0,0 +1,23 @@ +.. _engine command: + +==================== +Command Line Engines +==================== + +.. sidebar:: info + + - :origin:`command.py <searx/engines/command.py>` + - :ref:`offline engines` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +.. automodule:: searx.engines.command + :members: diff --git a/_sources/dev/engines/offline/nosql-engines.rst.txt b/_sources/dev/engines/offline/nosql-engines.rst.txt new file mode 100644 index 000000000..76f5cfb61 --- /dev/null +++ b/_sources/dev/engines/offline/nosql-engines.rst.txt @@ -0,0 +1,97 @@ +.. _nosql engines: + +=============== +NoSQL databases +=============== + +.. sidebar:: further read + + - `NoSQL databases <https://en.wikipedia.org/wiki/NoSQL>`_ + - `redis.io <https://redis.io/>`_ + - `MongoDB <https://www.mongodb.com>`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +The following `NoSQL databases`_ are supported: + +- :ref:`engine redis_server` +- :ref:`engine mongodb` + +All of the engines above are just commented out in the :origin:`settings.yml +<searx/settings.yml>`, as you have to set various options and install +dependencies before using them. + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple <searx/templates/simple/result_templates/key-value.html>` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +Furthermore, if you do not wish to expose these engines on a public instance, you +can still add them and limit the access by setting ``tokens`` as described in +section :ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +`NoSQL databases`_ are used for storing arbitrary data without first defining +their structure. + + +.. _engine redis_server: + +Redis Server +------------ + +.. _redis: https://github.com/andymccurdy/redis-py#installation + +.. sidebar:: info + + - ``pip install`` redis_ + - redis.io_ + - :origin:`redis_server.py <searx/engines/redis_server.py>` + +.. automodule:: searx.engines.redis_server + :members: + + +.. _engine mongodb: + +MongoDB +------- + +.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation + +.. sidebar:: info + + - ``pip install`` pymongo_ + - MongoDB_ + - :origin:`mongodb.py <searx/engines/mongodb.py>` + + +.. automodule:: searx.engines.mongodb + :members: + diff --git a/_sources/dev/engines/offline/search-indexer-engines.rst.txt b/_sources/dev/engines/offline/search-indexer-engines.rst.txt new file mode 100644 index 000000000..fa92798cb --- /dev/null +++ b/_sources/dev/engines/offline/search-indexer-engines.rst.txt @@ -0,0 +1,62 @@ +================= +Local Search APIs +================= + +.. sidebar:: further read + + - `Comparison to alternatives + <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_ + +.. contents:: + :depth: 1 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +Each search engine is powerful, capable of full-text search. All of the engines +above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to +enable HTTP requests by setting ``enable_http`` to ``True``. + +Furthermore, if you do not want to expose these engines on a public instance, +you can still add them and limit the access by setting ``tokens`` as described +in section :ref:`private engines`. + +.. _engine meilisearch: + +MeiliSearch +=========== + +.. automodule:: searx.engines.meilisearch + :members: + + +.. _engine elasticsearch: + +Elasticsearch +============= + +.. automodule:: searx.engines.elasticsearch + :members: + +.. _engine solr: + +Solr +==== + +.. automodule:: searx.engines.solr + :members: + diff --git a/_sources/dev/engines/offline/sql-engines.rst.txt b/_sources/dev/engines/offline/sql-engines.rst.txt new file mode 100644 index 000000000..f642f1f29 --- /dev/null +++ b/_sources/dev/engines/offline/sql-engines.rst.txt @@ -0,0 +1,134 @@ +.. _sql engines: + +=========== +SQL Engines +=========== + +.. sidebar:: further read + + - `SQLite <https://www.sqlite.org/index.html>`_ + - `PostgreSQL <https://www.postgresql.org>`_ + - `MySQL <https://www.mysql.com>`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +With the *SQL engines* you can bind SQL databases into SearXNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` & :ref:`engine mariadb_server` + +All of the engines above are just commented out in the :origin:`settings.yml +<searx/settings.yml>`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple <searx/templates/simple/result_templates/key-value.html>` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting ``tokens`` as described in section +:ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. sidebar:: info + + - :origin:`sqlite.py <searx/engines/sqlite.py>` + +.. automodule:: searx.engines.sqlite + :members: + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: info + + - :origin:`postgresql.py <searx/engines/postgresql.py>` + - ``pip install`` `psycopg2-binary <psycopg2>`_ + +.. automodule:: searx.engines.postgresql + :members: + +.. _engine mysql_server: + +MySQL +----- + +.. sidebar:: info + + - :origin:`mysql_server.py <searx/engines/mysql_server.py>` + - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>` + + +.. automodule:: searx.engines.mysql_server + :members: + +.. _engine mariadb_server: + +MariaDB +-------- + +.. sidebar:: info + + - :origin:`mariadb_server.py <searx/engines/mariadb_server.py>` + - ``pip install`` :pypi:`mariadb <mariadb>` + + +.. automodule:: searx.engines.mariadb_server + :members: diff --git a/_sources/dev/engines/offline_concept.rst.txt b/_sources/dev/engines/offline_concept.rst.txt new file mode 100644 index 000000000..ddb34fc60 --- /dev/null +++ b/_sources/dev/engines/offline_concept.rst.txt @@ -0,0 +1,69 @@ +=============== +Offline Concept +=============== + +.. sidebar:: offline engines + + - :ref:`demo offline engine` + - :ref:`engine command` + - :ref:`sql engines` + - :ref:`nosql engines` + - :py:obj:`searx.search.processors.offline` + +To extend the functionality of SearXNG, offline engines are going to be +introduced. An offline engine is an engine which does not need Internet +connection to perform a search and does not use HTTP to communicate. + +Offline engines can be configured, by adding those to the `engines` list of +:origin:`settings.yml <searx/settings.yml>`. An example skeleton for offline +engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py +<searx/engines/demo_offline.py>`). + + +Programming Interface +===================== + +:py:func:`init(engine_settings=None) <searx.engines.demo_offline.init>` + All offline engines can have their own init function to setup the engine before + accepting requests. The function gets the settings from settings.yml as a + parameter. This function can be omitted, if there is no need to setup anything + in advance. + +:py:func:`search(query, params) <searx.engines.demo_offline.searc>` + Each offline engine has a function named ``search``. This function is + responsible to perform a search and return the results in a presentable + format. (Where *presentable* means presentable by the selected result + template.) + + The return value is a list of results retrieved by the engine. + +Engine representation in ``/config`` + If an engine is offline, the attribute ``offline`` is set to ``True``. + +.. _offline requirements: + +Extra Dependencies +================== + +If an offline engine depends on an external tool, SearXNG does not install it by +default. When an administrator configures such engine and starts the instance, +the process returns an error with the list of missing dependencies. Also, +required dependencies will be added to the comment/description of the engine, so +admins can install packages in advance. + +If there is a need to install additional packages in *Python's Virtual +Environment* of your SearXNG instance you need to switch into the environment +(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Private engines (Security) +========================== + +To limit the access to offline engines, if an instance is available publicly, +administrators can set token(s) for each of the :ref:`private engines`. If a +query contains a valid token, then SearXNG performs the requested private +search. If not, requests from an offline engines return errors. + diff --git a/_sources/dev/engines/online/adobe_stock.rst.txt b/_sources/dev/engines/online/adobe_stock.rst.txt new file mode 100644 index 000000000..48a6511c0 --- /dev/null +++ b/_sources/dev/engines/online/adobe_stock.rst.txt @@ -0,0 +1,13 @@ +.. _adobe stock engine: + +=========== +Adobe Stock +=========== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.adobe_stock + :members: diff --git a/_sources/dev/engines/online/alpinelinux.rst.txt b/_sources/dev/engines/online/alpinelinux.rst.txt new file mode 100644 index 000000000..071a1985e --- /dev/null +++ b/_sources/dev/engines/online/alpinelinux.rst.txt @@ -0,0 +1,13 @@ +.. _alpinelinux engine: + +===================== +Alpine Linux Packages +===================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.alpinelinux + :members: diff --git a/_sources/dev/engines/online/annas_archive.rst.txt b/_sources/dev/engines/online/annas_archive.rst.txt new file mode 100644 index 000000000..db88e5069 --- /dev/null +++ b/_sources/dev/engines/online/annas_archive.rst.txt @@ -0,0 +1,13 @@ +.. _annas_archive engine: + +============== +Anna's Archive +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.annas_archive + :members: diff --git a/_sources/dev/engines/online/archlinux.rst.txt b/_sources/dev/engines/online/archlinux.rst.txt new file mode 100644 index 000000000..834fffa43 --- /dev/null +++ b/_sources/dev/engines/online/archlinux.rst.txt @@ -0,0 +1,14 @@ +.. _archlinux engine: + +========== +Arch Linux +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.archlinux + :members: + diff --git a/_sources/dev/engines/online/bing.rst.txt b/_sources/dev/engines/online/bing.rst.txt new file mode 100644 index 000000000..19c31aa80 --- /dev/null +++ b/_sources/dev/engines/online/bing.rst.txt @@ -0,0 +1,43 @@ +.. _bing engines: + +============ +Bing Engines +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _bing web engine: + +Bing WEB +======== + +.. automodule:: searx.engines.bing + :members: + +.. _bing images engine: + +Bing Images +=========== + +.. automodule:: searx.engines.bing_images + :members: + +.. _bing videos engine: + +Bing Videos +=========== + +.. automodule:: searx.engines.bing_videos + :members: + +.. _bing news engine: + +Bing News +========= + +.. automodule:: searx.engines.bing_news + :members: diff --git a/_sources/dev/engines/online/bpb.rst.txt b/_sources/dev/engines/online/bpb.rst.txt new file mode 100644 index 000000000..f545dba48 --- /dev/null +++ b/_sources/dev/engines/online/bpb.rst.txt @@ -0,0 +1,13 @@ +.. _bpb engine: + +=== +Bpb +=== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.bpb + :members: diff --git a/_sources/dev/engines/online/brave.rst.txt b/_sources/dev/engines/online/brave.rst.txt new file mode 100644 index 000000000..a1c589b9d --- /dev/null +++ b/_sources/dev/engines/online/brave.rst.txt @@ -0,0 +1,13 @@ +.. _brave engine: + +============= +Brave Engines +============= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.brave + :members: diff --git a/_sources/dev/engines/online/bt4g.rst.txt b/_sources/dev/engines/online/bt4g.rst.txt new file mode 100644 index 000000000..980665204 --- /dev/null +++ b/_sources/dev/engines/online/bt4g.rst.txt @@ -0,0 +1,14 @@ +.. _bt4g engine: + +==== +BT4G +==== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.bt4g + :members: + diff --git a/_sources/dev/engines/online/dailymotion.rst.txt b/_sources/dev/engines/online/dailymotion.rst.txt new file mode 100644 index 000000000..c661172e5 --- /dev/null +++ b/_sources/dev/engines/online/dailymotion.rst.txt @@ -0,0 +1,13 @@ +.. _dailymotion engine: + +=========== +Dailymotion +=========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.dailymotion + :members: diff --git a/_sources/dev/engines/online/discourse.rst.txt b/_sources/dev/engines/online/discourse.rst.txt new file mode 100644 index 000000000..1fab9e34c --- /dev/null +++ b/_sources/dev/engines/online/discourse.rst.txt @@ -0,0 +1,8 @@ +.. _discourse engine: + +================ +Discourse Forums +================ + +.. automodule:: searx.engines.discourse + :members: diff --git a/_sources/dev/engines/online/duckduckgo.rst.txt b/_sources/dev/engines/online/duckduckgo.rst.txt new file mode 100644 index 000000000..0f1258ff9 --- /dev/null +++ b/_sources/dev/engines/online/duckduckgo.rst.txt @@ -0,0 +1,22 @@ +.. _duckduckgo engines: + +================== +DuckDuckGo Engines +================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.duckduckgo + :members: + +.. automodule:: searx.engines.duckduckgo_extra + :members: + +.. automodule:: searx.engines.duckduckgo_definitions + :members: + +.. automodule:: searx.engines.duckduckgo_weather + :members: diff --git a/_sources/dev/engines/online/geizhals.rst.txt b/_sources/dev/engines/online/geizhals.rst.txt new file mode 100644 index 000000000..766eb5f59 --- /dev/null +++ b/_sources/dev/engines/online/geizhals.rst.txt @@ -0,0 +1,8 @@ +.. _gitea geizhals: + +======== +Geizhals +======== + +.. automodule:: searx.engines.geizhals + :members: diff --git a/_sources/dev/engines/online/gitea.rst.txt b/_sources/dev/engines/online/gitea.rst.txt new file mode 100644 index 000000000..745504e4d --- /dev/null +++ b/_sources/dev/engines/online/gitea.rst.txt @@ -0,0 +1,8 @@ +.. _gitea engine: + +===== +Gitea +===== + +.. automodule:: searx.engines.gitea + :members: diff --git a/_sources/dev/engines/online/gitlab.rst.txt b/_sources/dev/engines/online/gitlab.rst.txt new file mode 100644 index 000000000..5f0d3e3d1 --- /dev/null +++ b/_sources/dev/engines/online/gitlab.rst.txt @@ -0,0 +1,8 @@ +.. _gitlab engine: + +====== +GitLab +====== + +.. automodule:: searx.engines.gitlab + :members: diff --git a/_sources/dev/engines/online/google.rst.txt b/_sources/dev/engines/online/google.rst.txt new file mode 100644 index 000000000..9085070bd --- /dev/null +++ b/_sources/dev/engines/online/google.rst.txt @@ -0,0 +1,76 @@ +.. _google engines: + +============== +Google Engines +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _google API: + +Google API +========== + +.. _Query Parameter Definitions: + https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions + +SearXNG's implementation of the Google API is mainly done in +:py:obj:`get_google_info <searx.engines.google.get_google_info>`. + +For detailed description of the *REST-full* API see: `Query Parameter +Definitions`_. The linked API documentation can sometimes be helpful during +reverse engineering. However, we cannot use it in the freely accessible WEB +services; not all parameters can be applied and some engines are more *special* +than other (e.g. :ref:`google news engine`). + + +.. _google web engine: + +Google WEB +========== + +.. automodule:: searx.engines.google + :members: + +.. _google autocomplete: + +Google Autocomplete +==================== + +.. autofunction:: searx.autocomplete.google_complete + +.. _google images engine: + +Google Images +============= + +.. automodule:: searx.engines.google_images + :members: + +.. _google videos engine: + +Google Videos +============= + +.. automodule:: searx.engines.google_videos + :members: + +.. _google news engine: + +Google News +=========== + +.. automodule:: searx.engines.google_news + :members: + +.. _google scholar engine: + +Google Scholar +============== + +.. automodule:: searx.engines.google_scholar + :members: diff --git a/_sources/dev/engines/online/lemmy.rst.txt b/_sources/dev/engines/online/lemmy.rst.txt new file mode 100644 index 000000000..584246fd1 --- /dev/null +++ b/_sources/dev/engines/online/lemmy.rst.txt @@ -0,0 +1,13 @@ +.. _lemmy engine: + +===== +Lemmy +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.lemmy + :members: diff --git a/_sources/dev/engines/online/loc.rst.txt b/_sources/dev/engines/online/loc.rst.txt new file mode 100644 index 000000000..2ed76cd81 --- /dev/null +++ b/_sources/dev/engines/online/loc.rst.txt @@ -0,0 +1,13 @@ +.. _loc engine: + +=================== +Library of Congress +=================== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.loc + :members: diff --git a/_sources/dev/engines/online/mastodon.rst.txt b/_sources/dev/engines/online/mastodon.rst.txt new file mode 100644 index 000000000..dc372f121 --- /dev/null +++ b/_sources/dev/engines/online/mastodon.rst.txt @@ -0,0 +1,13 @@ +.. _mastodon engine: + +======== +Mastodon +======== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mastodon + :members: diff --git a/_sources/dev/engines/online/moviepilot.rst.txt b/_sources/dev/engines/online/moviepilot.rst.txt new file mode 100644 index 000000000..ba35574e5 --- /dev/null +++ b/_sources/dev/engines/online/moviepilot.rst.txt @@ -0,0 +1,13 @@ +.. _moviepilot engine: + +========== +Moviepilot +========== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.moviepilot + :members: diff --git a/_sources/dev/engines/online/mrs.rst.txt b/_sources/dev/engines/online/mrs.rst.txt new file mode 100644 index 000000000..671f35ea5 --- /dev/null +++ b/_sources/dev/engines/online/mrs.rst.txt @@ -0,0 +1,13 @@ +.. _mrs engine: + +========================= +Matrix Rooms Search (MRS) +========================= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mrs + :members: diff --git a/_sources/dev/engines/online/mullvad_leta.rst.txt b/_sources/dev/engines/online/mullvad_leta.rst.txt new file mode 100644 index 000000000..63aef019b --- /dev/null +++ b/_sources/dev/engines/online/mullvad_leta.rst.txt @@ -0,0 +1,13 @@ +.. _voidlinux mullvad_leta: + +============ +Mullvad-Leta +============ + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mullvad_leta + :members: diff --git a/_sources/dev/engines/online/mwmbl.rst.txt b/_sources/dev/engines/online/mwmbl.rst.txt new file mode 100644 index 000000000..8eac7d7c5 --- /dev/null +++ b/_sources/dev/engines/online/mwmbl.rst.txt @@ -0,0 +1,27 @@ +.. _Mwmbl engine: + +============ +Mwmbl Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _mwmbl web engine: + +Mwmbl WEB +========= + +.. automodule:: searx.engines.mwmbl + :members: + + +.. _mwmbl autocomplete: + +Mwmbl Autocomplete +================== + +.. autofunction:: searx.autocomplete.mwmbl diff --git a/_sources/dev/engines/online/odysee.rst.txt b/_sources/dev/engines/online/odysee.rst.txt new file mode 100644 index 000000000..75be1ad11 --- /dev/null +++ b/_sources/dev/engines/online/odysee.rst.txt @@ -0,0 +1,13 @@ +.. _odysee engine: + +====== +Odysee +====== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.odysee + :members: diff --git a/_sources/dev/engines/online/peertube.rst.txt b/_sources/dev/engines/online/peertube.rst.txt new file mode 100644 index 000000000..bedf055fb --- /dev/null +++ b/_sources/dev/engines/online/peertube.rst.txt @@ -0,0 +1,27 @@ +.. _peertube engines: + +================ +Peertube Engines +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _peertube video engine: + +Peertube Video +============== + +.. automodule:: searx.engines.peertube + :members: + +.. _sepiasearch engine: + +SepiaSearch +=========== + +.. automodule:: searx.engines.sepiasearch + :members: diff --git a/_sources/dev/engines/online/piped.rst.txt b/_sources/dev/engines/online/piped.rst.txt new file mode 100644 index 000000000..822981056 --- /dev/null +++ b/_sources/dev/engines/online/piped.rst.txt @@ -0,0 +1,13 @@ +.. _piped engine: + +===== +Piped +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.piped + :members: diff --git a/_sources/dev/engines/online/presearch.rst.txt b/_sources/dev/engines/online/presearch.rst.txt new file mode 100644 index 000000000..59332c354 --- /dev/null +++ b/_sources/dev/engines/online/presearch.rst.txt @@ -0,0 +1,13 @@ +.. _engine presearch: + +================ +Presearch Engine +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.presearch + :members: diff --git a/_sources/dev/engines/online/qwant.rst.txt b/_sources/dev/engines/online/qwant.rst.txt new file mode 100644 index 000000000..66ad302d1 --- /dev/null +++ b/_sources/dev/engines/online/qwant.rst.txt @@ -0,0 +1,13 @@ +.. _qwant engine: + +===== +Qwant +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.qwant + :members: diff --git a/_sources/dev/engines/online/radio_browser.rst.txt b/_sources/dev/engines/online/radio_browser.rst.txt new file mode 100644 index 000000000..a150e59c5 --- /dev/null +++ b/_sources/dev/engines/online/radio_browser.rst.txt @@ -0,0 +1,13 @@ +.. _RadioBrowser engine: + +============ +RadioBrowser +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.radio_browser + :members: diff --git a/_sources/dev/engines/online/recoll.rst.txt b/_sources/dev/engines/online/recoll.rst.txt new file mode 100644 index 000000000..2f1a1e4df --- /dev/null +++ b/_sources/dev/engines/online/recoll.rst.txt @@ -0,0 +1,13 @@ +.. _engine recoll: + +============= +Recoll Engine +============= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.recoll + :members: diff --git a/_sources/dev/engines/online/seekr.rst.txt b/_sources/dev/engines/online/seekr.rst.txt new file mode 100644 index 000000000..fcbc7bf82 --- /dev/null +++ b/_sources/dev/engines/online/seekr.rst.txt @@ -0,0 +1,13 @@ +.. _seekr engine: + +============= +Seekr Engines +============= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.seekr + :members: diff --git a/_sources/dev/engines/online/startpage.rst.txt b/_sources/dev/engines/online/startpage.rst.txt new file mode 100644 index 000000000..89e3ad10b --- /dev/null +++ b/_sources/dev/engines/online/startpage.rst.txt @@ -0,0 +1,13 @@ +.. _startpage engines: + +================= +Startpage Engines +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.startpage + :members: diff --git a/_sources/dev/engines/online/tagesschau.rst.txt b/_sources/dev/engines/online/tagesschau.rst.txt new file mode 100644 index 000000000..f850bf99b --- /dev/null +++ b/_sources/dev/engines/online/tagesschau.rst.txt @@ -0,0 +1,13 @@ +.. _tagesschau engine: + +============== +Tagesschau API +============== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tagesschau + :members: diff --git a/_sources/dev/engines/online/torznab.rst.txt b/_sources/dev/engines/online/torznab.rst.txt new file mode 100644 index 000000000..9056b60d0 --- /dev/null +++ b/_sources/dev/engines/online/torznab.rst.txt @@ -0,0 +1,13 @@ +.. _torznab engine: + +============== +Torznab WebAPI +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.torznab + :members: diff --git a/_sources/dev/engines/online/void.rst.txt b/_sources/dev/engines/online/void.rst.txt new file mode 100644 index 000000000..a1839d4f2 --- /dev/null +++ b/_sources/dev/engines/online/void.rst.txt @@ -0,0 +1,13 @@ +.. _voidlinux engine: + +========================== +Void Linux binary packages +========================== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.voidlinux + :members: diff --git a/_sources/dev/engines/online/wallhaven.rst.txt b/_sources/dev/engines/online/wallhaven.rst.txt new file mode 100644 index 000000000..b473293ca --- /dev/null +++ b/_sources/dev/engines/online/wallhaven.rst.txt @@ -0,0 +1,13 @@ +.. _wallhaven engine: + +========= +Wallhaven +========= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.wallhaven + :members: diff --git a/_sources/dev/engines/online/wikipedia.rst.txt b/_sources/dev/engines/online/wikipedia.rst.txt new file mode 100644 index 000000000..d4920f0f6 --- /dev/null +++ b/_sources/dev/engines/online/wikipedia.rst.txt @@ -0,0 +1,27 @@ +.. _wikimedia engines: + +========= +Wikimedia +========= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _wikipedia engine: + +Wikipedia +========= + +.. automodule:: searx.engines.wikipedia + :members: + +.. _wikidata engine: + +Wikidata +========= + +.. automodule:: searx.engines.wikidata + :members: diff --git a/_sources/dev/engines/online/yacy.rst.txt b/_sources/dev/engines/online/yacy.rst.txt new file mode 100644 index 000000000..9407aca80 --- /dev/null +++ b/_sources/dev/engines/online/yacy.rst.txt @@ -0,0 +1,13 @@ +.. _yacy engine: + +==== +Yacy +==== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yacy + :members: diff --git a/_sources/dev/engines/online/yahoo.rst.txt b/_sources/dev/engines/online/yahoo.rst.txt new file mode 100644 index 000000000..96c1e2774 --- /dev/null +++ b/_sources/dev/engines/online/yahoo.rst.txt @@ -0,0 +1,13 @@ +.. _yahoo engine: + +============ +Yahoo Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yahoo + :members: diff --git a/_sources/dev/engines/online/zlibrary.rst.txt b/_sources/dev/engines/online/zlibrary.rst.txt new file mode 100644 index 000000000..fb197abff --- /dev/null +++ b/_sources/dev/engines/online/zlibrary.rst.txt @@ -0,0 +1,13 @@ +.. _zlibrary engine: + +========= +Z-Library +========= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.zlibrary + :members: diff --git a/_sources/dev/engines/online_url_search/tineye.rst.txt b/_sources/dev/engines/online_url_search/tineye.rst.txt new file mode 100644 index 000000000..3f4db7e0e --- /dev/null +++ b/_sources/dev/engines/online_url_search/tineye.rst.txt @@ -0,0 +1,14 @@ +.. _tineye engine: + +====== +Tineye +====== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tineye + :members: + diff --git a/_sources/dev/engines/xpath.rst.txt b/_sources/dev/engines/xpath.rst.txt new file mode 100644 index 000000000..42c4d47b6 --- /dev/null +++ b/_sources/dev/engines/xpath.rst.txt @@ -0,0 +1,14 @@ +.. _xpath engine: + +============ +XPath Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.xpath + :members: + diff --git a/_sources/dev/index.rst.txt b/_sources/dev/index.rst.txt new file mode 100644 index 000000000..09be9de5e --- /dev/null +++ b/_sources/dev/index.rst.txt @@ -0,0 +1,18 @@ +======================= +Developer documentation +======================= + +.. toctree:: + :maxdepth: 2 + + quickstart + rtm_asdf + contribution_guide + engines/index + search_api + plugins + translation + lxcdev + makefile + reST + searxng_extra/index diff --git a/_sources/dev/lxcdev.rst.txt b/_sources/dev/lxcdev.rst.txt new file mode 100644 index 000000000..79716ae57 --- /dev/null +++ b/_sources/dev/lxcdev.rst.txt @@ -0,0 +1,438 @@ +.. _lxcdev: + +============================== +Developing in Linux Containers +============================== + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +In this article we will show, how you can make use of Linux Containers (LXC_) in +*distributed and heterogeneous development cycles* (TL;DR; jump to the +:ref:`lxcdev summary`). + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers. Readers + should have a serious meaning about the terms: *distributed*, *merge* and + *linux container*. + + **hint** + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Motivation +========== + +Most often in our development cycle, we edit the sources and run some test +and/or builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook. + + **The environment in which we run all our development processes matters!** + +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer's desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer's desktop that would never +be set on a *production* system. + + **Linux Containers are isolate environments**, we use them to not mix up all + the prerequisites from various projects on developer's desktop. + +The scripts from :ref:`searx_utils` can divide in those to install and maintain +software + +- :ref:`searxng.sh` + +and the script + +- :ref:`lxc.sh` + +with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the: + +- :ref:`searxng lxc suite` + +.. _lxcdev install searxng: + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: The ``searxng-archlinux`` container + + is the base of all our exercises here. + +The :ref:`lxc-searxng.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. +For this blog post we exercise on a archlinux_ image. The container of this +image is named ``searxng-archlinux``. + +Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh remove searxng-archlinux + $ sudo -H ./utils/lxc.sh build searxng-archlinux + + +.. sidebar:: further read + + - :ref:`lxc.sh install suite` + - :ref:`installation nginx` + +To install the complete :ref:`SearXNG suite <searxng lxc suite>` and the HTTP +proxy :ref:`installation nginx` into the archlinux container run: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL + ... + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng + +.. sidebar:: Fully functional SearXNG suite + + From here on you have a fully functional SearXNG suite (including a + :ref:`redis db`). + +In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy. + +In the example above the SearXNG instance in the container is wrapped to +``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the +IP of your ``searxng-archlinux`` container is different to this example. To +test the instance in the container from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container is done by the ``./utils/lxc.sh cmd +searxng-archlinux`` command: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux SearXNG]$ + +The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + +Back in the browser on your desktop open the service http://n.n.n.140/searxng +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. + +Another point we have to notice is that the service :ref:`SearXNG <searxng.sh>` +runs under dedicated system user account with the same name (compare +:ref:`create searxng user`). To get a login shell from these accounts, simply +call: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l + INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + +The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system +user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv* +``(searxng-pyenv)`` environment is activated. + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional SearXNG suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- SearXNG setup in: ``/etc/searxng/settings.yml`` +- SearXNG user's home: ``/usr/local/searxng`` +- virtualenv in: ``/usr/local/searxng/searxng-pyenv`` +- SearXNG software in: ``/usr/local/searxng/searxng-src`` + +With the use of the :ref:`searxng.sh` the SearXNG service was installed as +:ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use +``systemctl`` (compare :ref:`uWSGI maintenance`). + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng + +With the command above, we stopped the SearXNG uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux distros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searxng/searxng-src/searx + virtualenv = /usr/local/searxng/searxng-pyenv + pythonpath = /usr/local/searxng/searxng-src + +If you have read the :ref:`Good to know` you remember, that each container +shares the root folder of the repository and the command ``utils/lxc.sh cmd`` +handles relative path names **transparent**. + +To wrap the SearXNG installation in the container into a developer one, we +simple have to create a symlink to the **transparent** repository from the +desktop. Now lets replace the repository at ``searxng-src`` in the container +with the working tree from outside of the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the SearXNG service (in the +container): + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +To *inspect* the SearXNG instance (already described above): + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searx.sh inspect service + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service + +Run :ref:`makefile`, e.g. to test inside the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make test + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test + + + +To install all prerequisites needed for a :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh install buildhost + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost + + +To build the docs on a buildhost :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make docs.html + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html + + +.. _lxcdev summary: + +Summary +======= + +We build up a fully functional SearXNG suite in a archlinux container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + ... + Developer install? (wraps source from HOST into the running instance) [YES/no] + +To wrap the suite into a developer one answer ``YES`` (or press Enter). + +.. code:: text + + link SearXNG's sources to: /share/SearXNG + ========================================= + + mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" + ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" + ls -ld /usr/local/searxng/searxng-src + |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG + +On code modification the instance has to be restarted (see :ref:`uWSGI +maintenance`): + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + +To access HTTP from the desktop we installed nginx for the services inside the +container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +To get information about the SearxNG suite in the archlinux container we can +use: + +.. code:: text + + $ sudo -H ./utils/lxc.sh show suite searxng-archlinux + [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ + [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] + [searxng-archlinux] uWSGI: + [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket + [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: + [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng + [searxng-archlinux] GIT_BRANCH : master + [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng + [searxng-archlinux] SEARXNG_PORT : 8888 + [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1 + diff --git a/_sources/dev/makefile.rst.txt b/_sources/dev/makefile.rst.txt new file mode 100644 index 000000000..383113bae --- /dev/null +++ b/_sources/dev/makefile.rst.txt @@ -0,0 +1,383 @@ +.. _makefile: + +======================= +Makefile & ``./manage`` +======================= + +.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction + +All relevant build and development tasks are implemented in the +:origin:`./manage <manage>` script and for CI or IDE integration a small +:origin:`Makefile` wrapper is available. If you are not familiar with +Makefiles, we recommend to read gnu-make_ introduction. + +.. sidebar:: build environment + + Before looking deeper at the targets, first read about :ref:`make + install`. + + To install developer requirements follow :ref:`buildhosts`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +The usage is simple, just type ``make {target-name}`` to *build* a target. +Calling the ``help`` target gives a first overview (``make help``): + +.. tabs:: + + .. group-tab:: ``make`` + + .. program-output:: bash -c "cd ..; make --no-print-directory help" + + + .. group-tab:: ``./manage`` + + The Makefile targets are implemented for comfort, if you can do without + tab-completion and need to have a more granular control, use + :origin:`manage` without the Makefile wrappers. + + .. code:: sh + + $ ./manage help + +.. _make install: + +Python environment (``make install``) +===================================== + +.. sidebar:: activate environment + + ``source ./local/py3/bin/activate`` + +We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a ``make install`` to get a virtualenv with a +*developer install* of SearXNG (:origin:`setup.py`). :: + + $ cd ~/searxng-clone + $ make install + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]' + ... + Successfully installed searxng-2023.7.19+a446dea1b + +If you release ``make install`` multiple times the installation will only +rebuild if the sha256 sum of the *requirement files* fails. With other words: +the check fails if you edit the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`). :: + + $ make install + PYENV OK + PYENV [virtualenv] requirements.sha256 failed + [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt + [virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt + ... + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]' + ... + Successfully installed searxng-2023.7.19+a446dea1b + +.. sidebar:: drop environment + + To get rid of the existing environment before re-build use :ref:`clean target + <make clean>` first. + +If you think, something goes wrong with your ./local environment or you change +the :origin:`setup.py` file, you have to call :ref:`make clean`. + +.. _make node.env: + +Node.js environment (``make node.env``) +======================================= + +.. _Node.js: https://nodejs.org/ +.. _nvm: https://github.com/nvm-sh +.. _npm: https://www.npmjs.com/ + +.. jinja:: searx + + Node.js_ version {{version.node}} or higher is required to build the themes. + If the requirement is not met, the build chain uses nvm_ (Node Version + Manager) to install latest LTS of Node.js_ locally: there is no need to + install nvm_ or npm_ on your system. + +To install NVM_ and Node.js_ in once you can use :ref:`make nvm.nodejs`. + +.. _make nvm: + +NVM ``make nvm.install nvm.status`` +----------------------------------- + +Use ``make nvm.status`` to get the current status of your Node.js_ and nvm_ +setup. + +.. tabs:: + + .. group-tab:: nvm.install + + .. code:: sh + + $ LANG=C make nvm.install + INFO: install (update) NVM at ./searxng/.nvm + INFO: clone: https://github.com/nvm-sh/nvm.git + || Cloning into './searxng/.nvm'... + INFO: checkout v0.39.4 + || HEAD is now at 8fbf8ab v0.39.4 + + .. group-tab:: nvm.status (ubu2004) + + Here is the output you will typically get on a Ubuntu 20.04 system which + serves only a `no longer active <https://nodejs.org/en/about/releases/>`_ + Release `Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_. + + .. code:: sh + + $ make nvm.status + INFO: Node.js is installed at /usr/bin/node + INFO: Node.js is version v10.19.0 + WARN: minimal Node.js version is 16.13.0 + INFO: npm is installed at /usr/bin/npm + INFO: npm is version 6.14.4 + WARN: NVM is not installed + +.. _make nvm.nodejs: + +``make nvm.nodejs`` +------------------- + +Install latest Node.js_ LTS locally (uses nvm_):: + + $ make nvm.nodejs + INFO: install (update) NVM at /share/searxng/.nvm + INFO: clone: https://github.com/nvm-sh/nvm.git + ... + Downloading and installing node v16.13.0... + ... + INFO: Node.js is installed at searxng/.nvm/versions/node/v16.13.0/bin/node + INFO: Node.js is version v16.13.0 + INFO: npm is installed at searxng/.nvm/versions/node/v16.13.0/bin/npm + INFO: npm is version 8.1.0 + INFO: NVM is installed at searxng/.nvm + +.. _make run: + +``make run`` +============ + +To get up a running a developer instance simply call ``make run``. This enables +*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` +instance and opens the URL in your favorite WEB browser (:man:`xdg-open`):: + + $ make run + +Changes to theme's HTML templates (jinja2) are instant. Changes to the CSS & JS +sources of the theme need to be rebuild. You can do that by running:: + + $ make themes.all + +Alternatively to ``themes.all`` you can run *live builds* of the theme you are +modify (:ref:`make themes`):: + + $ LIVE_THEME=simple make run + +.. _make format.python: + +``make format.python`` +====================== + +Format Python source code using `Black code style`_. See ``$BLACK_OPTIONS`` +and ``$BLACK_TARGETS`` in :origin:`Makefile`. + +.. attention:: + + We stuck at Black 22.12.0, please read comment in PR `Bump black from 22.12.0 + to 23.1.0`_ + +.. _Bump black from 22.12.0 to 23.1.0: + https://github.com/searxng/searxng/pull/2159#pullrequestreview-1284094735 + +.. _Black code style: + https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html + +.. _make clean: + +``make clean`` +============== + +Drops all intermediate files, all builds, but keep sources untouched. Before +calling ``make clean`` stop all processes using the :ref:`make install` or +:ref:`make node.env`. :: + + $ make clean + CLEAN pyenv + PYENV [virtualenv] drop local/py3 + CLEAN docs -- build/docs dist/docs + CLEAN themes -- locally installed npm dependencies + ... + CLEAN test stuff + CLEAN common files + +.. _make docs: + +``make docs`` +============= + +Target ``docs`` builds the documentation: + +.. code:: bash + + $ make docs + HTML ./docs --> file:// + DOCS build build/docs/includes + ... + The HTML pages are in dist/docs. + +.. _make docs.clean: + +``make docs.clean docs.live`` +---------------------------------- + +We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / +Documentation <contrib docs>` section. If you want to edit the documentation +read our :ref:`make docs.live` section. If you are working in your own brand, +adjust your :ref:`settings brand`. + + +.. _make docs.gh-pages: + +``make docs.gh-pages`` +---------------------- + +To deploy on github.io first adjust your :ref:`settings brand`. For any +further read :ref:`deploy on github.io`. + +.. _make test: + +``make test`` +============= + +Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit`` +and ``test.robot``. You can run tests selective, e.g.:: + + $ make test.pep8 test.unit test.shell + TEST test.pep8 OK + ... + TEST test.unit OK + ... + TEST test.shell OK + +.. _make test.shell: + +``make test.shell`` +------------------- + +:ref:`sh lint` / if you have changed some bash scripting run this test before +commit. + +.. _make test.pylint: + +``make test.pylint`` +-------------------- + +.. _Pylint: https://www.pylint.org/ + +Pylint_ is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile used in the SearXNG project is +found in project's root folder :origin:`.pylintrc`. + +.. _make search.checker: + +``make search.checker.{engine name}`` +===================================== + +To check all engines:: + + make search.checker + +To check a engine with whitespace in the name like *google news* replace space +by underline:: + + make search.checker.google_news + +To see HTTP requests and more use SEARXNG_DEBUG:: + + make SEARXNG_DEBUG=1 search.checker.google_news + +.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection + +To filter out HTTP redirects (3xx_):: + + make SEARXNG_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]" + ... + Engine google news Checking + https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + +.. _make themes: + +``make themes.*`` +================= + +.. sidebar:: further read + + - :ref:`devquickstart` + +The :origin:`Makefile` targets ``make theme.*`` cover common tasks to build the +theme(s). The ``./manage themes.*`` command line can be used to convenient run +common theme build tasks. + +.. program-output:: bash -c "cd ..; ./manage themes.help" + +To get live builds while modifying CSS & JS use (:ref:`make run`): + +.. code:: sh + + $ LIVE_THEME=simple make run + +.. _make static.build: + +``make static.build.*`` +======================= + +.. sidebar:: further read + + - :ref:`devquickstart` + +The :origin:`Makefile` targets ``static.build.*`` cover common tasks to build (a +commit of) the static files. The ``./manage static.build..*`` command line +can be used to convenient run common build tasks of the static files. + +.. program-output:: bash -c "cd ..; ./manage static.help" + + +.. _manage redis.help: + +``./manage redis.help`` +======================= + +The ``./manage redis.*`` command line can be used to convenient run common Redis +tasks (:ref:`Redis developer notes`). + +.. program-output:: bash -c "cd ..; ./manage redis.help" + + +.. _manage go.help: + +``./manage go.help`` +==================== + +The ``./manage go.*`` command line can be used to convenient run common `go +(wiki)`_ tasks. + +.. _go (wiki): https://en.wikipedia.org/wiki/Go_(programming_language) + +.. program-output:: bash -c "cd ..; ./manage go.help" diff --git a/_sources/dev/plugins.rst.txt b/_sources/dev/plugins.rst.txt new file mode 100644 index 000000000..fb3201e66 --- /dev/null +++ b/_sources/dev/plugins.rst.txt @@ -0,0 +1,106 @@ +.. _dev plugin: + +======= +Plugins +======= + +.. sidebar:: Further reading .. + + - :ref:`plugins generic` + +Plugins can extend or replace functionality of various components of searx. + +Example plugin +============== + +.. code:: python + + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disabled by default + + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, search): + search.result_container.suggestions.add('example') + return True + +External plugins +================ + +SearXNG supports *external plugins* / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from *The Green Web Foundation* `[ref] +<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__: + +.. code:: bash + + $ sudo utils/searxng.sh instance cmd bash -c + (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins + +In the :ref:`settings.yml` activate the ``plugins:`` section and add module +``only_show_green_results`` from ``tgwf-searx-plugins``. + +.. code:: yaml + + plugins: + ... + - only_show_green_results + ... + + +Plugin entry points +=================== + +Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. A plugin doesn't need to implement all the hooks. + + +.. py:function:: pre_search(request, search) -> bool + + Runs BEFORE the search request. + + `search.result_container` can be changed. + + Return a boolean: + + * True to continue the search + * False to stop the search + + :param flask.request request: + :param searx.search.SearchWithPlugins search: + :return: False to stop the search + :rtype: bool + + +.. py:function:: post_search(request, search) -> None + + Runs AFTER the search request. + + :param flask.request request: Flask request. + :param searx.search.SearchWithPlugins search: Context. + + +.. py:function:: on_result(request, search, result) -> bool + + Runs for each result of each engine. + + `result` can be changed. + + If `result["url"]` is defined, then `result["parsed_url"] = urlparse(result['url'])` + + .. warning:: + `result["url"]` can be changed, but `result["parsed_url"]` must be updated too. + + Return a boolean: + + * True to keep the result + * False to remove the result + + :param flask.request request: + :param searx.search.SearchWithPlugins search: + :param typing.Dict result: Result, see - :ref:`engine results` + :return: True to keep the result + :rtype: bool diff --git a/_sources/dev/quickstart.rst.txt b/_sources/dev/quickstart.rst.txt new file mode 100644 index 000000000..c45c24491 --- /dev/null +++ b/_sources/dev/quickstart.rst.txt @@ -0,0 +1,82 @@ +.. _devquickstart: + +====================== +Development Quickstart +====================== + +.. _npm: https://www.npmjs.com/ +.. _Node.js: https://nodejs.org/ + + +.. sidebar:: further read + + - :ref:`makefile` + - :ref:`buildhosts` + +SearXNG loves developers; Developers do not need to worry about tool chains, the +usual developer tasks can be comfortably executed via :ref:`make <makefile>`. + +Don't hesitate, just clone SearXNG's sources and start hacking right now .. + +.. code:: bash + + git clone https://github.com/searxng/searxng.git searxng + +Here is how a minimal workflow looks like: + +1. *start* hacking +2. *run* your code: :ref:`make run` +3. *format & test* your code: :ref:`make format.python` and :ref:`make test` + +If you think at some point something fails, go back to *start*. Otherwise, +choose a meaningful commit message and we are happy to receive your pull +request. To not end in *wild west* we have some directives, please pay attention +to our ":ref:`how to contribute`" guideline. + +.. sidebar:: further read + + - :ref:`make nvm` + - :ref:`make themes` + +If you implement themes, you will need to setup a :ref:`Node.js environment +<make node.env>`: ``make node.env`` + +Before you call *make run* (2.), you need to compile the modified styles and +JavaScript: ``make themes.all`` + +Alternatively you can also compile selective the theme you have modified, +e.g. the *simple* theme. + +.. code:: bash + + make themes.simple + +.. tip:: + + To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run`` + +.. sidebar:: further read + + - :ref:`make static.build` + +If you finished your *tests* you can start to commit your changes. To separate +the modified source code from the build products first run: + +.. code:: bash + + make static.build.restore + +This will restore the old build products and only your changes of the code +remain in the working tree which can now be added & committed. When all sources +are committed, you can commit the build products simply by: + +.. code:: bash + + make static.build.commit + +Committing the build products should be the last step, just before you send us +your PR. There is also a make target to rewind this last build commit: + +.. code:: bash + + make static.build.drop diff --git a/_sources/dev/reST.rst.txt b/_sources/dev/reST.rst.txt new file mode 100644 index 000000000..e539617b7 --- /dev/null +++ b/_sources/dev/reST.rst.txt @@ -0,0 +1,1438 @@ +.. _reST primer: + +=========== +reST primer +=========== + +.. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +We at SearXNG are using reStructuredText (aka reST_) markup for all kind of +documentation. With the builders from the Sphinx_ project a HTML output is +generated and deployed at docs.searxng.org_. For build prerequisites read +:ref:`docs build`. + +.. _docs.searxng.org: https://docs.searxng.org/ + +The source files of SearXNG's documentation are located at :origin:`docs`. +Sphinx assumes source files to be encoded in UTF-8 by default. Run :ref:`make +docs.live <make docs.live>` to build HTML while editing. + +.. sidebar:: Further reading + + - Sphinx-Primer_ + - `Sphinx markup constructs`_ + - reST_, docutils_, `docutils FAQ`_ + - Sphinx_, `sphinx-doc FAQ`_ + - `sphinx config`_, doctree_ + - `sphinx cross references`_ + - linuxdoc_ + - intersphinx_ + - sphinx-jinja_ + - `Sphinx's autodoc`_ + - `Sphinx's Python domain`_, `Sphinx's C domain`_ + - SVG_, ImageMagick_ + - DOT_, `Graphviz's dot`_, Graphviz_ + + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +Sphinx_ and reST_ have their place in the python ecosystem. Over that reST is +used in popular projects, e.g the Linux kernel documentation `[kernel doc]`_. + +.. _[kernel doc]: https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html + +.. sidebar:: Content matters + + The readability_ of the reST sources has its value, therefore we recommend to + make sparse usage of reST markup / .. content matters! + +**reST** is a plaintext markup language, its markup is *mostly* intuitive and +you will not need to learn much to produce well formed articles with. I use the +word *mostly*: like everything in live, reST has its advantages and +disadvantages, some markups feel a bit grumpy (especially if you are used to +other plaintext markups). + +Soft skills +=========== + +Before going any deeper into the markup let's face on some **soft skills** a +trained author brings with, to reach a well feedback from readers: + +- Documentation is dedicated to an audience and answers questions from the + audience point of view. +- Don't detail things which are general knowledge from the audience point of + view. +- Limit the subject, use cross links for any further reading. + +To be more concrete what a *point of view* means. In the (:origin:`docs`) +folder we have three sections (and the *blog* folder), each dedicate to a +different group of audience. + +User's POV: :origin:`docs/user` + A typical user knows about search engines and might have heard about + meta crawlers and privacy. + +Admin's POV: :origin:`docs/admin` + A typical Admin knows about setting up services on a linux system, but he does + not know all the pros and cons of a SearXNG setup. + +Developer's POV: :origin:`docs/dev` + Depending on the readability_ of code, a typical developer is able to read and + understand source code. Describe what a item aims to do (e.g. a function). + If the chronological order matters, describe it. Name the *out-of-limits + conditions* and all the side effects a external developer will not know. + +.. _reST inline markup: + +Basic inline markup +=================== + +.. sidebar:: Inline markup + + - :ref:`reST roles` + - :ref:`reST smart ref` + +Basic inline markup is done with asterisks and backquotes. If asterisks or +backquotes appear in running text and could be confused with inline markup +delimiters, they have to be escaped with a backslash (``\*pointer``). + +.. table:: basic inline markup + :widths: 4 3 7 + + ================================================ ==================== ======================== + description rendered markup + ================================================ ==================== ======================== + one asterisk for emphasis *italics* ``*italics*`` + two asterisks for strong emphasis **boldface** ``**boldface**`` + backquotes for code samples and literals ``foo()`` ````foo()```` + quote asterisks or backquotes \*foo is a pointer ``\*foo is a pointer`` + ================================================ ==================== ======================== + +.. _reST basic structure: + +Basic article structure +======================= + +The basic structure of an article makes use of heading adornments to markup +chapter, sections and subsections. + +.. _reST template: + +reST template +------------- + +reST template for an simple article: + +.. code:: reST + + .. _doc refname: + + ============== + Document title + ============== + + Lorem ipsum dolor sit amet, consectetur adipisici elit .. Further read + :ref:`chapter refname`. + + .. _chapter refname: + + Chapter + ======= + + Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut + aliquid ex ea commodi consequat ... + + .. _section refname: + + Section + ------- + + lorem .. + + .. _subsection refname: + + Subsection + ~~~~~~~~~~ + + lorem .. + + +Headings +-------- + +#. title - with overline for document title: + + .. code:: reST + + ============== + Document title + ============== + + +#. chapter - with anchor named ``anchor name``: + + .. code:: reST + + .. _anchor name: + + Chapter + ======= + +#. section + + .. code:: reST + + Section + ------- + +#. subsection + + .. code:: reST + + Subsection + ~~~~~~~~~~ + + + +Anchors & Links +=============== + +.. _reST anchor: + +Anchors +------- + +.. _ref role: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref + +To refer a point in the documentation a anchor is needed. The :ref:`reST +template <reST template>` shows an example where a chapter titled *"Chapters"* +gets an anchor named ``chapter title``. Another example from *this* document, +where the anchor named ``reST anchor``: + +.. code:: reST + + .. _reST anchor: + + Anchors + ------- + + To refer a point in the documentation a anchor is needed ... + +To refer anchors use the `ref role`_ markup: + +.. code:: reST + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar <reST anchor>`. + +.. admonition:: ``:ref:`` role + :class: rst-example + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar <reST anchor>`. + +.. _reST ordinary ref: + +Link ordinary URL +----------------- + +If you need to reference external URLs use *named* hyperlinks to maintain +readability of reST sources. Here is a example taken from *this* article: + +.. code:: reST + + .. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + <https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>`__. + +.. admonition:: Named hyperlink + :class: rst-example + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + <https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>`__. + + +.. _reST smart ref: + +Smart refs +---------- + +With the power of sphinx.ext.extlinks_ and intersphinx_ referencing external +content becomes smart. + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + refer ... rendered example markup + ========================== ================================== ==================================== + :rst:role:`rfc` :rfc:`822` ``:rfc:`822``` + :rst:role:`pep` :pep:`8` ``:pep:`8``` + sphinx.ext.extlinks_ + -------------------------------------------------------------------------------------------------- + project's wiki article :wiki:`Offline-engines` ``:wiki:`Offline-engines``` + to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html``` + files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` + pull request :pull:`4` ``:pull:`4``` + patch :patch:`af2cae6` ``:patch:`af2cae6``` + PyPi package :pypi:`httpx` ``:pypi:`httpx``` + manual page man :man:`bash` ``:man:`bash``` + intersphinx_ + -------------------------------------------------------------------------------------------------- + external anchor :ref:`python:and` ``:ref:`python:and``` + external doc anchor :doc:`jinja:templates` ``:doc:`jinja:templates``` + python code object :py:obj:`datetime.datetime` ``:py:obj:`datetime.datetime``` + flask code object :py:obj:`flask.Flask` ``:py:obj:`flask.Flask``` + ========================== ================================== ==================================== + + +Intersphinx is configured in :origin:`docs/conf.py`: + +.. code:: python + + intersphinx_mapping = { + "python": ("https://docs.python.org/3/", None), + "flask": ("https://flask.palletsprojects.com/", None), + "jinja": ("https://jinja.palletsprojects.com/", None), + "linuxdoc" : ("https://return42.github.io/linuxdoc/", None), + "sphinx" : ("https://www.sphinx-doc.org/en/master/", None), + } + + +To list all anchors of the inventory (e.g. ``python``) use: + +.. code:: sh + + $ python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv + ... + $ python -m sphinx.ext.intersphinx https://docs.searxng.org/objects.inv + ... + +Literal blocks +============== + +The simplest form of :duref:`literal-blocks` is a indented block introduced by +two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST +code` directive. To include literals from external files use +:rst:dir:`literalinclude` or :ref:`kernel-include <kernel-include-directive>` +directive (latter one expands environment variables in the path name). + +.. _reST literal: + +``::`` +------ + +.. code:: reST + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + +.. admonition:: Literal block + :class: rst-example + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + + +.. _reST code: + +``code-block`` +-------------- + +.. _pygments: https://pygments.org/languages/ + +.. sidebar:: Syntax highlighting + + is handled by pygments_. + +The :rst:dir:`code-block` directive is a variant of the :dudir:`code` directive +with additional options. To learn more about code literals visit +:ref:`sphinx:code-examples`. + +.. code-block:: reST + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +.. code-block:: reST + +.. admonition:: Code block + :class: rst-example + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +Unicode substitution +==================== + +The :dudir:`unicode directive <unicode-character-codes>` converts Unicode +character codes (numerical values) to characters. This directive can only be +used within a substitution definition. + +.. code-block:: reST + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + +.. admonition:: Unicode + :class: rst-example + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + + +.. _reST roles: + +Roles +===== + +.. sidebar:: Further reading + + - `Sphinx Roles`_ + - :doc:`sphinx:usage/restructuredtext/domains` + + +A *custom interpreted text role* (:duref:`ref <roles>`) is an inline piece of +explicit markup. It signifies that that the enclosed text should be interpreted +in a specific way. + +The general markup is one of: + +.. code:: reST + + :rolename:`ref-name` + :rolename:`ref text <ref-name>` + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + role rendered example markup + ========================== ================================== ==================================== + :rst:role:`guilabel` :guilabel:`&Cancel` ``:guilabel:`&Cancel``` + :rst:role:`kbd` :kbd:`C-x C-f` ``:kbd:`C-x C-f``` + :rst:role:`menuselection` :menuselection:`Open --> File` ``:menuselection:`Open --> File``` + :rst:role:`download` :download:`this file <reST.rst>` ``:download:`this file <reST.rst>``` + math_ :math:`a^2 + b^2 = c^2` ``:math:`a^2 + b^2 = c^2``` + :rst:role:`ref` :ref:`svg image example` ``:ref:`svg image example``` + :rst:role:`command` :command:`ls -la` ``:command:`ls -la``` + :durole:`emphasis` :emphasis:`italic` ``:emphasis:`italic``` + :durole:`strong` :strong:`bold` ``:strong:`bold``` + :durole:`literal` :literal:`foo()` ``:literal:`foo()``` + :durole:`subscript` H\ :sub:`2`\ O ``H\ :sub:`2`\ O`` + :durole:`superscript` E = mc\ :sup:`2` ``E = mc\ :sup:`2``` + :durole:`title-reference` :title:`Time` ``:title:`Time``` + ========================== ================================== ==================================== + +Figures & Images +================ + +.. sidebar:: Image processing + + With the directives from :ref:`linuxdoc <linuxdoc:kfigure>` the build process + is flexible. To get best results in the generated output format, install + ImageMagick_ and Graphviz_. + +SearXNG's sphinx setup includes: :ref:`linuxdoc:kfigure`. Scalable here means; +scalable in sense of the build process. Normally in absence of a converter +tool, the build process will break. From the authors POV it’s annoying to care +about the build process when handling with images, especially since he has no +access to the build process. With :ref:`linuxdoc:kfigure` the build process +continues and scales output quality in dependence of installed image processors. + +If you want to add an image, you should use the ``kernel-figure`` (inheritance +of :dudir:`figure`) and ``kernel-image`` (inheritance of :dudir:`image`) +directives. E.g. to insert a figure with a scalable image format use SVG +(:ref:`svg image example`): + +.. code:: reST + + .. _svg image example: + + .. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image + + To refer the figure, a caption block is needed: :ref:`svg image example`. + +.. _svg image example: + +.. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image. + +To refer the figure, a caption block is needed: :ref:`svg image example`. + +DOT files (aka Graphviz) +------------------------ + +With :ref:`linuxdoc:kernel-figure` reST support for **DOT** formatted files is +given. + +- `Graphviz's dot`_ +- DOT_ +- Graphviz_ + +A simple example is shown in :ref:`dot file example`: + +.. code:: reST + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +.. admonition:: hello.dot + :class: rst-example + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +``kernel-render`` DOT +--------------------- + +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the +:ref:`linuxdoc:kernel-render` directive. A simple example of embedded DOT_ is +shown in figure :ref:`dot render example`: + +.. code:: reST + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +Please note :ref:`build tools <linuxdoc:kfigure_build_tools>`. If Graphviz_ is +installed, you will see an vector image. If not, the raw markup is inserted as +*literal-block*. + +.. admonition:: kernel-render DOT + :class: rst-example + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +``kernel-render`` SVG +--------------------- + +A simple example of embedded SVG_ is shown in figure :ref:`svg render example`: + +.. code:: reST + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow +.. + + .. code:: xml + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" + baseProfile="full" width="70px" height="40px" + viewBox="0 0 700 400" + > + <line x1="180" y1="370" + x2="500" y2="50" + stroke="black" stroke-width="15px" + /> + <polygon points="585 0 525 25 585 50" + transform="rotate(135 525 25)" + /> + </svg> + +.. admonition:: kernel-render SVG + :class: rst-example + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" + baseProfile="full" width="70px" height="40px" + viewBox="0 0 700 400" + > + <line x1="180" y1="370" + x2="500" y2="50" + stroke="black" stroke-width="15px" + /> + <polygon points="585 0 525 25 585 50" + transform="rotate(135 525 25)" + /> + </svg> + + + + +.. _reST lists: + +List markups +============ + +Bullet list +----------- + +List markup (:duref:`ref <bullet-lists>`) is simple: + +.. code:: reST + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + +.. admonition:: bullet list + :class: rst-example + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + + +Horizontal list +--------------- + +The :rst:dir:`.. hlist:: <hlist>` transforms a bullet list into a more compact +list. + +.. code:: reST + + .. hlist:: + + - first list item + - second list item + - third list item + ... + +.. admonition:: hlist + :class: rst-example + + .. hlist:: + + - first list item + - second list item + - third list item + - next list item + - next list item xxxx + - next list item yyyy + - next list item zzzz + + +Definition list +--------------- + +.. sidebar:: Note .. + + - the term cannot have more than one line of text + + - there is **no blank line between term and definition block** // this + distinguishes definition lists (:duref:`ref <definition-lists>`) from block + quotes (:duref:`ref <block-quotes>`). + +Each definition list (:duref:`ref <definition-lists>`) item contains a term, +optional classifiers and a definition. A term is a simple one-line word or +phrase. Optional classifiers may follow the term on the same line, each after +an inline ' : ' (**space, colon, space**). A definition is a block indented +relative to the term, and may contain multiple paragraphs and other body +elements. There may be no blank line between a term line and a definition block +(*this distinguishes definition lists from block quotes*). Blank lines are +required before the first and after the last definition list item, but are +optional in-between. + +Definition lists are created as follows: + +.. code:: reST + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref <block-quotes>`) and not as term's definition! + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + Definition 4. + +.. admonition:: definition list + :class: rst-example + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref <block-quotes>`) and not as term's definition! + + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + + +Quoted paragraphs +----------------- + +Quoted paragraphs (:duref:`ref <block-quotes>`) are created by just indenting +them more than the surrounding paragraphs. Line blocks (:duref:`ref +<line-blocks>`) are a way of preserving line breaks: + +.. code:: reST + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. admonition:: Quoted paragraph and line block + :class: rst-example + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. _reST field list: + +Field Lists +----------- + +.. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + +.. sidebar:: bibliographic fields + + First lines fields are bibliographic fields, see `Sphinx Field Lists`_. + +Field lists are used as part of an extension syntax, such as options for +directives, or database-like records meant for further processing. Field lists +are mappings from field names to field bodies. They marked up like this: + +.. code:: reST + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + +.. admonition:: Field List + :class: rst-example + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + + +They are commonly used in Python documentation: + +.. code:: python + + def my_function(my_arg, my_other_arg): + """A function just for me. + + :param my_arg: The first of my arguments. + :param my_other_arg: The second of my arguments. + + :returns: A message (just for me, of course). + """ + +Further list blocks +------------------- + +- field lists (:duref:`ref <field-lists>`, with caveats noted in + :ref:`reST field list`) +- option lists (:duref:`ref <option-lists>`) +- quoted literal blocks (:duref:`ref <quoted-literal-blocks>`) +- doctest blocks (:duref:`ref <doctest-blocks>`) + + +Admonitions +=========== + +Sidebar +------- + +Sidebar is an eye catcher, often used for admonitions pointing further stuff or +site effects. Here is the source of the sidebar :ref:`on top of this page <reST +primer>`. + +.. code:: reST + + .. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +Generic admonition +------------------ + +The generic :dudir:`admonition <admonitions>` needs a title: + +.. code:: reST + + .. admonition:: generic admonition title + + lorem ipsum .. + + +.. admonition:: generic admonition title + + lorem ipsum .. + + +Specific admonitions +-------------------- + +Specific admonitions: :dudir:`hint`, :dudir:`note`, :dudir:`tip` :dudir:`attention`, +:dudir:`caution`, :dudir:`danger`, :dudir:`error`, , :dudir:`important`, and +:dudir:`warning` . + +.. code:: reST + + .. hint:: + + lorem ipsum .. + + .. note:: + + lorem ipsum .. + + .. warning:: + + lorem ipsum .. + + +.. hint:: + + lorem ipsum .. + +.. note:: + + lorem ipsum .. + +.. tip:: + + lorem ipsum .. + +.. attention:: + + lorem ipsum .. + +.. caution:: + + lorem ipsum .. + +.. danger:: + + lorem ipsum .. + +.. important:: + + lorem ipsum .. + +.. error:: + + lorem ipsum .. + +.. warning:: + + lorem ipsum .. + + +Tables +====== + +.. sidebar:: Nested tables + + Nested tables are ugly! Not all builder support nested tables, don't use + them! + +ASCII-art tables like :ref:`reST simple table` and :ref:`reST grid table` might +be comfortable for readers of the text-files, but they have huge disadvantages +in the creation and modifying. First, they are hard to edit. Think about +adding a row or a column to a ASCII-art table or adding a paragraph in a cell, +it is a nightmare on big tables. + + +.. sidebar:: List tables + + For meaningful patch and diff use :ref:`reST flat table`. + +Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a +cell generates a diff in which also changes are included, which are only +ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, +here are some helpers: + +* `Emacs Table Mode`_ +* `Online Tables Generator`_ + +.. _reST simple table: + +Simple tables +------------- + +:duref:`Simple tables <simple-tables>` allow *colspan* but not *rowspan*. If +your table need some metadata (e.g. a title) you need to add the ``.. table:: +directive`` :dudir:`(ref) <table>` in front and place the table in its body: + +.. code:: reST + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + +.. admonition:: Simple ASCII table + :class: rst-example + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + + + +.. _reST grid table: + +Grid tables +----------- + +:duref:`Grid tables <grid-tables>` allow colspan *colspan* and *rowspan*: + +.. code:: reST + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + +.. admonition:: ASCII grid table + :class: rst-example + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + + +.. _reST flat table: + +flat-table +---------- + +The ``flat-table`` is a further developed variant of the :ref:`list tables +<linuxdoc:list-table-directives>`. It is a double-stage list similar to the +:dudir:`list-table` with some additional features: + +column-span: ``cspan`` + with the role ``cspan`` a cell can be extended through additional columns + +row-span: ``rspan`` + with the role ``rspan`` a cell can be extended through additional rows + +auto-span: + spans rightmost cell of a table row over the missing cells on the right side + of that table-row. With Option ``:fill-cells:`` this behavior can changed + from *auto span* to *auto fill*, which automatically inserts (empty) cells + instead of spanning the last cell. + +options: + :header-rows: [int] count of header rows + :stub-columns: [int] count of stub columns + :widths: [[int] [int] ... ] widths of columns + :fill-cells: instead of auto-span missing cells, insert missing cells + +roles: + :cspan: [int] additional columns (*morecols*) + :rspan: [int] additional rows (*morerows*) + +The example below shows how to use this markup. The first level of the staged +list is the *table-row*. In the *table-row* there is only one markup allowed, +the list of the cells in this *table-row*. Exception are *comments* ( ``..`` ) +and *targets* (e.g. a ref to :ref:`row 2 of table's body <row body 2>`). + +.. code:: reST + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +.. admonition:: List table + :class: rst-example + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +CSV table +--------- + +CSV table might be the choice if you want to include CSV-data from a outstanding +(build) process into your documentation. + +.. code:: reST + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 2 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Content of file ``csv_table.txt``: + +.. literalinclude:: csv_table.txt + +.. admonition:: CSV table + :class: rst-example + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 3 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Templating +========== + +.. sidebar:: Build environment + + All *generic-doc* tasks are running in the :ref:`make install`. + +Templating is suitable for documentation which is created generic at the build +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with SearXNG modules installed). We use this e.g. to build chapter: +:ref:`configured engines`. Below the jinja directive from the +:origin:`docs/admin/engines.rst` is shown: + +.. literalinclude:: ../user/configured_engines.rst + :language: reST + :start-after: .. _configured engines: + +The context for the template is selected in the line ``.. jinja:: searx``. In +sphinx's build configuration (:origin:`docs/conf.py`) the ``searx`` context +contains the ``engines`` and ``plugins``. + +.. code:: py + + import searx.search + import searx.engines + import searx.plugins + searx.search.initialize() + jinja_contexts = { + 'searx': { + 'engines': searx.engines.engines, + 'plugins': searx.plugins.plugins + }, + } + + +Tabbed views +============ + +.. _sphinx-tabs: https://github.com/djungelorm/sphinx-tabs +.. _basic-tabs: https://github.com/djungelorm/sphinx-tabs#basic-tabs +.. _group-tabs: https://github.com/djungelorm/sphinx-tabs#group-tabs +.. _code-tabs: https://github.com/djungelorm/sphinx-tabs#code-tabs + +With `sphinx-tabs`_ extension we have *tabbed views*. To provide installation +instructions with one tab per distribution we use the `group-tabs`_ directive, +others are basic-tabs_ and code-tabs_. Below a *group-tab* example from +:ref:`docs build` is shown: + +.. literalinclude:: ../admin/buildhosts.rst + :language: reST + :start-after: .. SNIP sh lint requirements + :end-before: .. SNAP sh lint requirements + +.. _math: + +Math equations +============== + +.. _Mathematics: https://en.wikibooks.org/wiki/LaTeX/Mathematics +.. _amsmath user guide: + http://vesta.informatik.rwth-aachen.de/ftp/pub/mirror/ctan/macros/latex/required/amsmath/amsldoc.pdf + +.. sidebar:: About LaTeX + + - `amsmath user guide`_ + - Mathematics_ + - :ref:`docs build` + +The input language for mathematics is LaTeX markup using the :ctan:`amsmath` +package. + +To embed LaTeX markup in reST documents, use role :rst:role:`:math: <math>` for +inline and directive :rst:dir:`.. math:: <math>` for block markup. + +.. code:: reST + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + +.. admonition:: LaTeX math equation + :class: rst-example + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + + +The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` +(*displaystyle*) used in a inline markup or another fraction. + +.. code:: reST + + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + ``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + +.. admonition:: Line spacing + :class: rst-example + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + +.. _KISS: https://en.wikipedia.org/wiki/KISS_principle + +.. _readability: https://docs.python-guide.org/writing/style/ +.. _Sphinx-Primer: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +.. _reST: https://docutils.sourceforge.io/rst.html +.. _Sphinx Roles: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html +.. _Sphinx: https://www.sphinx-doc.org +.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html +.. _Sphinx markup constructs: + https://www.sphinx-doc.org/en/stable/markup/index.html +.. _`sphinx cross references`: + https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations +.. _sphinx.ext.extlinks: + https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html +.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html +.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html +.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html +.. _Sphinx's Python domain: + https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain +.. _Sphinx's C domain: + https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs +.. _doctree: + https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases +.. _docutils: http://docutils.sourceforge.net/docs/index.html +.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html +.. _linuxdoc: https://return42.github.io/linuxdoc +.. _jinja: https://jinja.palletsprojects.com/ +.. _sphinx-jinja: https://github.com/tardyp/sphinx-jinja +.. _SVG: https://www.w3.org/TR/SVG11/expanded-toc.html +.. _DOT: https://graphviz.gitlab.io/_pages/doc/info/lang.html +.. _`Graphviz's dot`: https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org + +.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode +.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables +.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html diff --git a/_sources/dev/rtm_asdf.rst.txt b/_sources/dev/rtm_asdf.rst.txt new file mode 100644 index 000000000..69180ef4e --- /dev/null +++ b/_sources/dev/rtm_asdf.rst.txt @@ -0,0 +1,121 @@ +================== +Runtime Management +================== + +The runtimes are managed with asdf and are activated in this project via the +`.tool-versions <.tool-versions>`_. If you have not yet installed asdf_, then +chapter :ref:`introduce asdf` may be of help to you. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Get started +=========== + +If you have asdf installed you can install the runtimes of this project by: + +.. code:: bash + + $ cd /path/to/searxng + $ asdf install # will install runtimes listed in .tool-versions + ... + +Manage Versions +=============== + +If you want to perform a ``test`` with special runtime versions of nodejs, +python or shellcheck, you can patch the ``.tool-versions``: + +.. code:: diff + + --- a/.tool-versions + +++ b/.tool-versions + @@ -1,2 +1,2 @@ + -python 3.12.0 + -shellcheck 0.9.0 + +python 3.11.6 + +shellcheck 0.8.0 + +To install use ``asdf install`` again. If the runtime tools have changed, any +existing (nodejs and python) environments should be cleaned up with a ``make +clean``. + +.. code:: bash + + $ asdf install + ... + $ make clean test + + +.. _introduce asdf: + +Introduce asdf +============== + +To `download asdf`_ and `install asdf`_: + +.. code:: bash + + $ git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch <version> + $ echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc + $ echo '. "$HOME/.asdf/completions/asdf.bash"' >> ~/.bashrc + +Start a new shell and try to `install plugins`_: + +.. code:: bash + + $ asdf plugin-list-all | grep -E '(golang|python|nodejs|shellcheck).git' + golang https://github.com/asdf-community/asdf-golang.git + nodejs https://github.com/asdf-vm/asdf-nodejs.git + python https://github.com/danhper/asdf-python.git + shellcheck https://github.com/luizm/asdf-shellcheck.git + + $ asdf plugin add golang https://github.com/asdf-community/asdf-golang.git + $ asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git + $ asdf plugin add python https://github.com/danhper/asdf-python.git + $ asdf plugin add shellcheck https://github.com/luizm/asdf-shellcheck.git + +Each plugin has dependencies, to compile runtimes visit the URLs from above and +look out for the dependencies you need to install on your OS, on Debian for the +runtimes listed above you will need: + +.. code:: bash + + $ sudo apt update + $ sudo apt install \ + dirmngr gpg curl gawk coreutils build-essential libssl-dev zlib1g-dev \ + libbz2-dev libreadline-dev libsqlite3-dev \ + libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev + +With dependencies installed you can install/compile runtimes: + +.. code:: bash + + $ asdf install golang latest + $ asdf install nodejs latest + $ asdf install python latest + $ asdf install shellcheck latest + +Python will be compiled and will take a while. + +In the repository the version is defined in `.tool-versions`_. Outside the +repository, its recommended that the runtime should use the versions of the OS +(`Fallback to System Version`_) / if not already done register the system +versions global: + +.. code:: bash + + $ cd / + $ asdf global golang system + $ asdf global nodejs system + $ asdf global python system + $ asdf global shellcheck system + +.. _asdf: https://asdf-vm.com/ +.. _download asdf: https://asdf-vm.com/guide/getting-started.html#_2-download-asdf +.. _install asdf: https://asdf-vm.com/guide/getting-started.html#_3-install-asdf +.. _install plugins: https://asdf-vm.com/guide/getting-started.html#install-the-plugin +.. _Fallback to System Version: https://asdf-vm.com/manage/versions.html#fallback-to-system-version diff --git a/_sources/dev/search_api.rst.txt b/_sources/dev/search_api.rst.txt new file mode 100644 index 000000000..aa5f847ea --- /dev/null +++ b/_sources/dev/search_api.rst.txt @@ -0,0 +1,124 @@ +.. _search API: + +========== +Search API +========== + +The search supports both ``GET`` and ``POST``. + +Furthermore, two endpoints ``/`` and ``/search`` are available for querying. + + +``GET /`` + +``GET /search`` + +Parameters +========== + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings.yml` + - :ref:`configured engines` + +``q`` : required + The search query. This string is passed to external search services. Thus, + SearXNG supports syntax of each search service. For example, ``site:github.com + SearXNG`` is a valid query for Google. However, if simply the query above is + passed to any search engine which does not filter its results based on this + syntax, you might not get the results you wanted. + + See more at :ref:`search-syntax` + +``categories`` : optional + Comma separated list, specifies the active search categories (see + :ref:`configured engines`) + +``engines`` : optional + Comma separated list, specifies the active search engines (see + :ref:`configured engines`). + +``language`` : default from :ref:`settings search` + Code of the language. + +``pageno`` : default ``1`` + Search page number. + +``time_range`` : optional + [ ``day``, ``month``, ``year`` ] + + Time range of search for engines which support it. See if an engine supports + time range search in the preferences page of an instance. + +``format`` : optional + [ ``json``, ``csv``, ``rss`` ] + + Output format of results. Format needs to be activated in :ref:`settings + search`. + +``results_on_new_tab`` : default ``0`` + [ ``0``, ``1`` ] + + Open search results on new tab. + +``image_proxy`` : default from :ref:`settings server` + [ ``True``, ``False`` ] + + Proxy image results through SearXNG. + +``autocomplete`` : default from :ref:`settings search` + [ ``google``, ``dbpedia``, ``duckduckgo``, ``mwmbl``, ``startpage``, + ``wikipedia``, ``stract``, ``swisscows``, ``qwant`` ] + + Service which completes words as you type. + +``safesearch`` : default from :ref:`settings search` + [ ``0``, ``1``, ``2`` ] + + Filter search results of engines which support safe search. See if an engine + supports safe search in the preferences page of an instance. + +``theme`` : default ``simple`` + [ ``simple`` ] + + Theme of instance. + + Please note, available themes depend on an instance. It is possible that an + instance administrator deleted, created or renamed themes on their instance. + See the available options in the preferences page of the instance. + +``enabled_plugins`` : optional + List of enabled plugins. + + :default: + ``Hash_plugin``, ``Self_Information``, + ``Tracker_URL_remover``, ``Ahmia_blacklist`` + + :values: + .. enabled by default + + ``Hash_plugin``, ``Self_Information``, + ``Tracker_URL_remover``, ``Ahmia_blacklist``, + + .. disabled by default + + ``Hostnames_plugin``, ``Open_Access_DOI_rewrite``, + ``Vim-like_hotkeys``, ``Tor_check_plugin`` + +``disabled_plugins``: optional + List of disabled plugins. + + :default: + ``Hostnames_plugin``, ``Open_Access_DOI_rewrite``, + ``Vim-like_hotkeys``, ``Tor_check_plugin`` + + :values: + see values from ``enabled_plugins`` + +``enabled_engines`` : optional : *all* :origin:`engines <searx/engines>` + List of enabled engines. + +``disabled_engines`` : optional : *all* :origin:`engines <searx/engines>` + List of disabled engines. + diff --git a/_sources/dev/searxng_extra/index.rst.txt b/_sources/dev/searxng_extra/index.rst.txt new file mode 100644 index 000000000..7bca3c0d4 --- /dev/null +++ b/_sources/dev/searxng_extra/index.rst.txt @@ -0,0 +1,14 @@ +.. _searxng_extra: + +============================= +Tooling box ``searxng_extra`` +============================= + +In the folder :origin:`searxng_extra/` we maintain some tools useful for CI and +developers. + +.. toctree:: + :maxdepth: 2 + + update + standalone_searx.py diff --git a/_sources/dev/searxng_extra/standalone_searx.py.rst.txt b/_sources/dev/searxng_extra/standalone_searx.py.rst.txt new file mode 100644 index 000000000..7cbbccfde --- /dev/null +++ b/_sources/dev/searxng_extra/standalone_searx.py.rst.txt @@ -0,0 +1,9 @@ + +.. _standalone_searx.py: + +===================================== +``searxng_extra/standalone_searx.py`` +===================================== + +.. automodule:: searxng_extra.standalone_searx + :members: diff --git a/_sources/dev/searxng_extra/update.rst.txt b/_sources/dev/searxng_extra/update.rst.txt new file mode 100644 index 000000000..dc3b06744 --- /dev/null +++ b/_sources/dev/searxng_extra/update.rst.txt @@ -0,0 +1,98 @@ +========================= +``searxng_extra/update/`` +========================= + +:origin:`[source] <searxng_extra/update/__init__.py>` + +Scripts to update static data in :origin:`searx/data/` + +.. _update_ahmia_blacklist.py: + +``update_ahmia_blacklist.py`` +============================= + +:origin:`[source] <searxng_extra/update/update_ahmia_blacklist.py>` + +.. automodule:: searxng_extra.update.update_ahmia_blacklist + :members: + + +``update_currencies.py`` +======================== + +:origin:`[source] <searxng_extra/update/update_currencies.py>` + +.. automodule:: searxng_extra.update.update_currencies + :members: + +``update_engine_descriptions.py`` +================================= + +:origin:`[source] <searxng_extra/update/update_engine_descriptions.py>` + +.. automodule:: searxng_extra.update.update_engine_descriptions + :members: + + +``update_external_bangs.py`` +============================ + +:origin:`[source] <searxng_extra/update/update_external_bangs.py>` + +.. automodule:: searxng_extra.update.update_external_bangs + :members: + + +``update_firefox_version.py`` +============================= + +:origin:`[source] <searxng_extra/update/update_firefox_version.py>` + +.. automodule:: searxng_extra.update.update_firefox_version + :members: + + +``update_engine_traits.py`` +=========================== + +:origin:`[source] <searxng_extra/update/update_engine_traits.py>` + +.. automodule:: searxng_extra.update.update_engine_traits + :members: + + +``update_osm_keys_tags.py`` +=========================== + +:origin:`[source] <searxng_extra/update/update_osm_keys_tags.py>` + +.. automodule:: searxng_extra.update.update_osm_keys_tags + :members: + + +``update_pygments.py`` +====================== + +:origin:`[source] <searxng_extra/update/update_pygments.py>` + +.. automodule:: searxng_extra.update.update_pygments + :members: + +.. _update_locales.py: + +``update_locales.py`` +===================== + +:origin:`[source] <searxng_extra/update/update_locales.py>` + +.. automodule:: searxng_extra.update.update_locales + :members: + + +``update_wikidata_units.py`` +============================ + +:origin:`[source] <searxng_extra/update/update_wikidata_units.py>` + +.. automodule:: searxng_extra.update.update_wikidata_units + :members: diff --git a/_sources/dev/translation.rst.txt b/_sources/dev/translation.rst.txt new file mode 100644 index 000000000..57c76a0c1 --- /dev/null +++ b/_sources/dev/translation.rst.txt @@ -0,0 +1,81 @@ +.. _translation: + +=========== +Translation +=========== + +.. _translate.codeberg.org: https://translate.codeberg.org/projects/searxng/ +.. _Weblate: https://docs.weblate.org +.. _translations branch: https://github.com/searxng/searxng/tree/translations +.. _orphan branch: https://git-scm.com/docs/git-checkout#Documentation/git-checkout.txt---orphanltnewbranchgt +.. _Weblate repository: https://translate.codeberg.org/projects/searxng/searxng/#repository +.. _wlc: https://docs.weblate.org/en/latest/wlc.html + +.. |translated| image:: https://translate.codeberg.org/widgets/searxng/-/searxng/svg-badge.svg + :target: https://translate.codeberg.org/projects/searxng/ + +.. sidebar:: |translated| + + - :ref:`searx.babel_extract` + - Weblate_ + - SearXNG `translations branch`_ + - SearXNG `Weblate repository`_ + - Weblate Client: wlc_ + - Babel Command-Line: `pybabel <http://babel.pocoo.org/en/latest/cmdline.html>`_ + - `weblate workflow <https://docs.weblate.org/en/latest/workflows.html>`_ + +Translation takes place on translate.codeberg.org_. + +Translations which has been added by translators on the translate.codeberg.org_ UI are +committed to Weblate's counterpart of the SearXNG *origin* repository which is +located at ``https://translate.codeberg.org/git/searxng/searxng``. + +There is no need to clone this repository, :ref:`SearXNG Weblate workflow` take +care of the synchronization with the *origin*. To avoid merging commits from +the counterpart directly on the ``master`` branch of *SearXNG origin*, a *pull +request* (PR) is created by this workflow. + +Weblate monitors the `translations branch`_, not the ``master`` branch. This +branch is an `orphan branch`_, decoupled from the master branch (we already know +orphan branches from the ``gh-pages``). The `translations branch`_ contains +only the + +- ``translation/messages.pot`` and the +- ``translation/*/messages.po`` files, nothing else. + + +.. _SearXNG Weblate workflow: + +.. figure:: translation.svg + + SearXNG's PR workflow to be in sync with Weblate + +Sync from *origin* to *weblate*: using ``make weblate.push.translations`` + For each commit on the ``master`` branch of SearXNG *origin* the GitHub job + :origin:`babel / Update translations branch + <.github/workflows/integration.yml>` checks for updated translations. + +Sync from *weblate* to *origin*: using ``make weblate.translations.commit`` + Every Friday, the GitHub workflow :origin:`babel / create PR for additions from + weblate <.github/workflows/translations-update.yml>` creates a PR with the + updated translation files: + + - ``translation/messages.pot``, + - ``translation/*/messages.po`` and + - ``translation/*/messages.mo`` + +wlc +=== + +.. _wlc configuration: https://docs.weblate.org/en/latest/wlc.html#wlc-config +.. _API key: https://translate.codeberg.org/accounts/profile/#api + +All weblate integration is done by GitHub workflows, but if you want to use wlc_, +copy this content into `wlc configuration`_ in your HOME ``~/.config/weblate`` + +.. code-block:: ini + + [keys] + https://translate.codeberg.org/api/ = APIKEY + +Replace ``APIKEY`` by your `API key`_. diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt new file mode 100644 index 000000000..4c5a0c2a5 --- /dev/null +++ b/_sources/index.rst.txt @@ -0,0 +1,59 @@ +================== +Welcome to SearXNG +================== + + *Search without being tracked.* + +.. jinja:: searx + + SearXNG is a free internet metasearch engine which aggregates results from up + to {{engines | length}} :ref:`search services <configured engines>`. Users + are neither tracked nor profiled. Additionally, SearXNG can be used over Tor + for online anonymity. + +Get started with SearXNG by using one of the instances listed at searx.space_. +If you don't trust anyone, you can set up your own, see :ref:`installation`. + +.. jinja:: searx + + .. sidebar:: features + + - :ref:`self hosted <installation>` + - :ref:`no user tracking / no profiling <SearXNG protect privacy>` + - script & cookies are optional + - secure, encrypted connections + - :ref:`{{engines | length}} search engines <configured engines>` + - `58 translations <https://translate.codeberg.org/projects/searxng/searxng/>`_ + - about 70 `well maintained <https://uptime.searxng.org/>`__ instances on searx.space_ + - :ref:`easy integration of search engines <demo online engine>` + - professional development: `CI <https://github.com/searxng/searxng/actions>`_, + `quality assurance <https://dev.searxng.org/>`_ & + `automated tested UI <https://dev.searxng.org/screenshots.html>`_ + +.. sidebar:: be a part + + SearXNG is driven by an open community, come join us! Don't hesitate, no + need to be an *expert*, everyone can contribute: + + - `help to improve translations <https://translate.codeberg.org/projects/searxng/searxng/>`_ + - `discuss with the community <https://matrix.to/#/#searxng:matrix.org>`_ + - report bugs & suggestions + - ... + +.. sidebar:: the origin + + SearXNG development has been started in the middle of 2021 as a fork of the + searx project. + + +.. toctree:: + :maxdepth: 2 + + user/index + own-instance + admin/index + dev/index + utils/index + src/index + +.. _searx.space: https://searx.space diff --git a/_sources/own-instance.rst.txt b/_sources/own-instance.rst.txt new file mode 100644 index 000000000..7d79f26c2 --- /dev/null +++ b/_sources/own-instance.rst.txt @@ -0,0 +1,86 @@ +=========================== +Why use a private instance? +=========================== + +.. sidebar:: Is running my own instance worth it? + + \.\.\.is a common question among SearXNG users. Before answering this + question, see what options a SearXNG user has. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Public instances are open to everyone who has access to their URL. Usually, they +are operated by unknown parties (from the users' point of view). Private +instances can be used by a select group of people, such as a SearXNG instance for a +group of friends, or a company which can be accessed through a VPN. Instances can also be +single-user instances, which run locally on the user's machine. + +To gain more insight on how these instances work, let's dive into how SearXNG +protects its users. + +.. _SearXNG protect privacy: + +How does SearXNG protect privacy? +================================= + +SearXNG protects the privacy of its users in multiple ways, regardless of the type +of the instance (private or public). Removal of private data from search requests +comes in three forms: + + 1. Removing private data from requests going to search services + 2. Not forwarding anything from third party services through search services + (e.g. advertisement) + 3. Removing private data from requests going to the results pages + +Removing private data means not sending cookies to external search engines and +generating a random browser profile for every request. Thus, it does not matter +if a public or private instance handles the request, because it is anonymized in +both cases. The IP address used will be the IP of the instance, but SearXNG can also be +configured to use proxy or Tor. `Result proxy +<https://github.com/asciimoo/morty>`__ is supported, too. + +SearXNG does not serve ads or tracking content, unlike most search services. Therefore, +private data is not forwarded to third parties who might monetize it. Besides +protecting users from search services, both the referring page and search query are +hidden from the results pages being visited. + + +What are the consequences of using public instances? +---------------------------------------------------- + +If someone uses a public instance, they have to trust the administrator of that +instance. This means that the user of the public instance does not know whether +their requests are logged, aggregated, and sent or sold to a third party. + +Also, public instances without proper protection are more vulnerable to abuse of +the search service, which may cause the external service to enforce +CAPTCHAs or to ban the IP address of the instance. Thus, search requests would return less +results. + +I see. What about private instances? +------------------------------------ + +If users run their :ref:`own instances <installation>`, everything is in their +control: the source code, logging settings and private data. Unknown instance +administrators do not have to be trusted. + +Furthermore, as the default settings of their instance are editable, there is no +need to use cookies to tailor SearXNG to their needs and preferences will not +reset to defaults when clearing browser cookies. As settings are stored on +the user's computer, they will not be accessible to others as long as their computer is +not compromised. + +Conclusion +========== + +Always use an instance which is operated by people you trust. The privacy +features of SearXNG are available to users no matter what kind of instance they +use. + +For those on the go, or just wanting to try SearXNG for the first time, public +instances are the best choice. Public instances are also making the +world a better place by giving those who cannot, or do not want to, run an +instance access to a privacy-respecting search service. diff --git a/_sources/src/index.rst.txt b/_sources/src/index.rst.txt new file mode 100644 index 000000000..fd16e7635 --- /dev/null +++ b/_sources/src/index.rst.txt @@ -0,0 +1,13 @@ +=========== +Source-Code +=========== + +This is a partial documentation of our source code. We are not aiming to document +every item from the source code, but we will add documentation when requested. + + +.. toctree:: + :maxdepth: 2 + :glob: + + searx.* diff --git a/_sources/src/searx.babel_extract.rst.txt b/_sources/src/searx.babel_extract.rst.txt new file mode 100644 index 000000000..741d67fc1 --- /dev/null +++ b/_sources/src/searx.babel_extract.rst.txt @@ -0,0 +1,8 @@ +.. _searx.babel_extract: + +=============================== +Custom message extractor (i18n) +=============================== + +.. automodule:: searx.babel_extract + :members: diff --git a/_sources/src/searx.botdetection.rst.txt b/_sources/src/searx.botdetection.rst.txt new file mode 100644 index 000000000..04cb81dfd --- /dev/null +++ b/_sources/src/searx.botdetection.rst.txt @@ -0,0 +1,62 @@ +.. _botdetection: + +============= +Bot Detection +============= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.botdetection + :members: + +.. _botdetection ip_lists: + +IP lists +======== + +.. automodule:: searx.botdetection.ip_lists + :members: + + +.. _botdetection rate limit: + +Rate limit +========== + +.. automodule:: searx.botdetection.ip_limit + :members: + +.. automodule:: searx.botdetection.link_token + :members: + + +.. _botdetection probe headers: + +Probe HTTP headers +================== + +.. automodule:: searx.botdetection.http_accept + :members: + +.. automodule:: searx.botdetection.http_accept_encoding + :members: + +.. automodule:: searx.botdetection.http_accept_language + :members: + +.. automodule:: searx.botdetection.http_connection + :members: + +.. automodule:: searx.botdetection.http_user_agent + :members: + +.. _botdetection config: + +Config +====== + +.. automodule:: searx.botdetection.config + :members: diff --git a/_sources/src/searx.exceptions.rst.txt b/_sources/src/searx.exceptions.rst.txt new file mode 100644 index 000000000..72117e148 --- /dev/null +++ b/_sources/src/searx.exceptions.rst.txt @@ -0,0 +1,8 @@ +.. _searx.exceptions: + +================== +SearXNG Exceptions +================== + +.. automodule:: searx.exceptions + :members: diff --git a/_sources/src/searx.favicons.rst.txt b/_sources/src/searx.favicons.rst.txt new file mode 100644 index 000000000..c1c6a500b --- /dev/null +++ b/_sources/src/searx.favicons.rst.txt @@ -0,0 +1,48 @@ +.. _favicons source: + +================= +Favicons (source) +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.favicons + :members: + +.. _favicons.config: + +Favicons Config +=============== + +.. automodule:: searx.favicons.config + :members: + +.. _favicons.proxy: + +Favicons Proxy +============== + +.. automodule:: searx.favicons.proxy + :members: + +.. _favicons.resolver: + +Favicons Resolver +================= + +.. automodule:: searx.favicons.resolvers + :members: + +.. _favicons.cache: + +Favicons Cache +============== + +.. automodule:: searx.favicons.cache + :members: + + + diff --git a/_sources/src/searx.infopage.rst.txt b/_sources/src/searx.infopage.rst.txt new file mode 100644 index 000000000..243e5b0bd --- /dev/null +++ b/_sources/src/searx.infopage.rst.txt @@ -0,0 +1,8 @@ +.. _searx.infopage: + +================ +Online ``/info`` +================ + +.. automodule:: searx.infopage + :members: diff --git a/_sources/src/searx.locales.rst.txt b/_sources/src/searx.locales.rst.txt new file mode 100644 index 000000000..9882e7890 --- /dev/null +++ b/_sources/src/searx.locales.rst.txt @@ -0,0 +1,15 @@ +.. _searx.locales: + +======= +Locales +======= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.locales + :members: + + diff --git a/_sources/src/searx.plugins.hostnames.rst.txt b/_sources/src/searx.plugins.hostnames.rst.txt new file mode 100644 index 000000000..5d0365536 --- /dev/null +++ b/_sources/src/searx.plugins.hostnames.rst.txt @@ -0,0 +1,9 @@ +.. _hostnames plugin: + +================ +Hostnames plugin +================ + +.. automodule:: searx.plugins.hostnames + :members: + diff --git a/_sources/src/searx.plugins.tor_check.rst.txt b/_sources/src/searx.plugins.tor_check.rst.txt new file mode 100644 index 000000000..905328ebf --- /dev/null +++ b/_sources/src/searx.plugins.tor_check.rst.txt @@ -0,0 +1,9 @@ +.. _tor check plugin: + +================ +Tor check plugin +================ + +.. automodule:: searx.plugins.tor_check + :members: + diff --git a/_sources/src/searx.plugins.unit_converter.rst.txt b/_sources/src/searx.plugins.unit_converter.rst.txt new file mode 100644 index 000000000..48d495d80 --- /dev/null +++ b/_sources/src/searx.plugins.unit_converter.rst.txt @@ -0,0 +1,9 @@ +.. _unit converter plugin: + +===================== +Unit converter plugin +===================== + +.. automodule:: searx.plugins.unit_converter + :members: + diff --git a/_sources/src/searx.redisdb.rst.txt b/_sources/src/searx.redisdb.rst.txt new file mode 100644 index 000000000..625378c91 --- /dev/null +++ b/_sources/src/searx.redisdb.rst.txt @@ -0,0 +1,8 @@ +.. _redis db: + +======== +Redis DB +======== + +.. automodule:: searx.redisdb + :members: diff --git a/_sources/src/searx.redislib.rst.txt b/_sources/src/searx.redislib.rst.txt new file mode 100644 index 000000000..b4604574c --- /dev/null +++ b/_sources/src/searx.redislib.rst.txt @@ -0,0 +1,8 @@ +.. _searx.redis: + +============= +Redis Library +============= + +.. automodule:: searx.redislib + :members: diff --git a/_sources/src/searx.search.processors.rst.txt b/_sources/src/searx.search.processors.rst.txt new file mode 100644 index 000000000..3bb4c44bc --- /dev/null +++ b/_sources/src/searx.search.processors.rst.txt @@ -0,0 +1,47 @@ +.. _searx.search.processors: + +================= +Search processors +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Abstract processor class +======================== + +.. automodule:: searx.search.processors.abstract + :members: + +Offline processor +================= + +.. automodule:: searx.search.processors.offline + :members: + +Online processor +================ + +.. automodule:: searx.search.processors.online + :members: + +Online currency processor +========================= + +.. automodule:: searx.search.processors.online_currency + :members: + +Online dictionary processor +=========================== + +.. automodule:: searx.search.processors.online_dictionary + :members: + +Online URL search processor +=========================== + +.. automodule:: searx.search.processors.online_url_search + :members: diff --git a/_sources/src/searx.search.rst.txt b/_sources/src/searx.search.rst.txt new file mode 100644 index 000000000..ad76d4183 --- /dev/null +++ b/_sources/src/searx.search.rst.txt @@ -0,0 +1,38 @@ +.. _searx.search: + +====== +Search +====== + +.. autoclass:: searx.search.EngineRef + :members: + +.. autoclass:: searx.search.SearchQuery + :members: + +.. autoclass:: searx.search.Search + + .. attribute:: search_query + :type: searx.search.SearchQuery + + .. attribute:: result_container + :type: searx.results.ResultContainer + + .. automethod:: search() -> searx.results.ResultContainer + +.. autoclass:: searx.search.SearchWithPlugins + :members: + + .. attribute:: search_query + :type: searx.search.SearchQuery + + .. attribute:: result_container + :type: searx.results.ResultContainer + + .. attribute:: ordered_plugin_list + :type: typing.List + + .. attribute:: request + :type: flask.request + + .. automethod:: search() -> searx.results.ResultContainer diff --git a/_sources/src/searx.settings.rst.txt b/_sources/src/searx.settings.rst.txt new file mode 100644 index 000000000..1496c407c --- /dev/null +++ b/_sources/src/searx.settings.rst.txt @@ -0,0 +1,8 @@ +.. _searx.settings_loader: + +=============== +Settings Loader +=============== + +.. automodule:: searx.settings_loader + :members: diff --git a/_sources/src/searx.sqlitedb.rst.txt b/_sources/src/searx.sqlitedb.rst.txt new file mode 100644 index 000000000..f984ee286 --- /dev/null +++ b/_sources/src/searx.sqlitedb.rst.txt @@ -0,0 +1,8 @@ +.. _sqlite db: + +========= +SQLite DB +========= + +.. automodule:: searx.sqlitedb + :members: diff --git a/_sources/src/searx.utils.rst.txt b/_sources/src/searx.utils.rst.txt new file mode 100644 index 000000000..6496700c1 --- /dev/null +++ b/_sources/src/searx.utils.rst.txt @@ -0,0 +1,8 @@ +.. _searx.utils: + +================================= +Utility functions for the engines +================================= + +.. automodule:: searx.utils + :members: diff --git a/_sources/user/about.rst.txt b/_sources/user/about.rst.txt new file mode 100644 index 000000000..08f1a068e --- /dev/null +++ b/_sources/user/about.rst.txt @@ -0,0 +1,4 @@ +.. _about SearXNG: + +.. include:: about.md + :parser: myst_parser.sphinx_ diff --git a/_sources/user/configured_engines.rst.txt b/_sources/user/configured_engines.rst.txt new file mode 100644 index 000000000..c32a264ed --- /dev/null +++ b/_sources/user/configured_engines.rst.txt @@ -0,0 +1,89 @@ +.. _configured engines: + +================== +Configured Engines +================== + +.. sidebar:: Further reading .. + + - :ref:`settings categories_as_tabs` + - :ref:`engines-dev` + - :ref:`settings engine` + - :ref:`general engine configuration` + +.. jinja:: searx + + SearXNG supports {{engines | length}} search engines of which + {{enabled_engine_count}} are enabled by default. + + Engines can be assigned to multiple :ref:`categories <engine categories>`. + The UI displays the tabs that are configured in :ref:`categories_as_tabs + <settings categories_as_tabs>`. In addition to these UI categories (also + called *tabs*), engines can be queried by their name or the categories they + belong to, by using a :ref:`\!bing syntax <search-syntax>`. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. jinja:: searx + + {% for category, engines in categories_as_tabs.items() %} + + tab ``!{{category.replace(' ', '_')}}`` + --------------------------------------- + + {% for group, group_bang, engines in engines | group_engines_in_tab %} + + {% if loop.length > 1 %} + {% if group_bang %}group ``{{group_bang}}``{% else %}{{group}}{% endif %} + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + {% endif %} + + .. flat-table:: + :header-rows: 2 + :stub-columns: 1 + :widths: 10 1 10 1 1 1 1 1 1 1 + + * - :cspan:`5` Engines configured by default (in :ref:`settings.yml <engine settings>`) + - :cspan:`3` :ref:`Supported features <engine file>` + + * - Name + - !bang + - Module + - Disabled + - Timeout + - Weight + - Paging + - Locale + - Safe search + - Time range + + {% for mod in engines %} + + * - `{{mod.name}} <{{mod.about and mod.about.website}}>`_ + {%- if mod.about and mod.about.language %} + ({{mod.about.language | upper}}) + {%- endif %} + - ``!{{mod.shortcut}}`` + - {%- if 'searx.engines.' + mod.__name__ in documented_modules %} + :py:mod:`~searx.engines.{{mod.__name__}}` + {%- else %} + :origin:`{{mod.__name__}} <searx/engines/{{mod.__name__}}.py>` + {%- endif %} + - {{(mod.disabled and "y") or ""}} + - {{mod.timeout}} + - {{mod.weight or 1 }} + {% if mod.engine_type == 'online' %} + - {{(mod.paging and "y") or ""}} + - {{(mod.language_support and "y") or ""}} + - {{(mod.safesearch and "y") or ""}} + - {{(mod.time_range_support and "y") or ""}} + {% else %} + - :cspan:`3` not applicable ({{mod.engine_type}}) + {% endif %} + + {% endfor %} + {% endfor %} + {% endfor %} diff --git a/_sources/user/index.rst.txt b/_sources/user/index.rst.txt new file mode 100644 index 000000000..705fd3f9e --- /dev/null +++ b/_sources/user/index.rst.txt @@ -0,0 +1,15 @@ +================ +User information +================ + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +.. toctree:: + :maxdepth: 2 + + search-syntax + configured_engines + about diff --git a/_sources/user/search-syntax.rst.txt b/_sources/user/search-syntax.rst.txt new file mode 100644 index 000000000..3acddd6e3 --- /dev/null +++ b/_sources/user/search-syntax.rst.txt @@ -0,0 +1,4 @@ +.. _search-syntax: + +.. include:: search-syntax.md + :parser: myst_parser.sphinx_ diff --git a/_sources/utils/index.rst.txt b/_sources/utils/index.rst.txt new file mode 100644 index 000000000..b570b07e6 --- /dev/null +++ b/_sources/utils/index.rst.txt @@ -0,0 +1,31 @@ +.. _searx_utils: +.. _toolboxing: + +================== +DevOps tooling box +================== + +In the folder :origin:`utils/` we maintain some tools useful for administrators +and developers. + +.. toctree:: + :maxdepth: 2 + + searxng.sh + lxc.sh + +Common command environments +=========================== + +The scripts in our tooling box often dispose of common environments: + +.. _FORCE_TIMEOUT: + +``FORCE_TIMEOUT`` : environment + Sets timeout for interactive prompts. If you want to run a script in batch + job, with defaults choices, set ``FORCE_TIMEOUT=0``. By example; to install a + SearXNG server and nginx proxy on all containers of the :ref:`SearXNG suite + <lxc-searxng.env>` use:: + + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install all + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx diff --git a/_sources/utils/lxc.sh.rst.txt b/_sources/utils/lxc.sh.rst.txt new file mode 100644 index 000000000..fae302c90 --- /dev/null +++ b/_sources/utils/lxc.sh.rst.txt @@ -0,0 +1,295 @@ + +.. _snap: https://snapcraft.io +.. _snapcraft LXD: https://snapcraft.io/lxd +.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/ +.. _LXC: https://linuxcontainers.org/lxc/introduction/ +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _`LXD@github`: https://github.com/lxc/lxd + +.. _archlinux: https://www.archlinux.org/ + +.. _lxc.sh: + +================ +``utils/lxc.sh`` +================ + +With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of +containers, what we call the: *lxc suite*. The :ref:`lxc-searxng.env` is +loaded by default, every time you start the ``lxc.sh`` script (*you do not need +to care about*). + +.. sidebar:: further reading + + - snap_, `snapcraft LXD`_ + - LXC_, LXD_ + - `LXC/LXD Image Server`_ + - `LXD@github`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _lxd install: + +Install LXD +=========== + +Before you can start with containers, you need to install and initiate LXD_ +once:: + + $ snap install lxd + $ lxd init --auto + +To make use of the containers from the *SearXNG suite*, you have to build the +:ref:`LXC suite containers <lxc.sh help>` initial. But be warned, **this might +take some time**:: + + $ sudo -H ./utils/lxc.sh build + +.. sidebar:: hint + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + +A cup of coffee later, your LXC suite is build up and you can run whatever task +you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh +help>`. + +.. _internet connectivity docker: + +Internet Connectivity & Docker +------------------------------ + +.. sidebar:: further read + + - `Docker blocking network of existing LXC containers <https://github.com/docker/for-linux/issues/103>`__ + - `Docker and IPtables (fralef.me) <https://fralef.me/docker-and-iptables.html>`__ + - `Docker and iptables (docker.com) <https://docs.docker.com/network/iptables/#docker-on-a-router/>`__ + +There is a conflict in the ``iptables`` setup of Docker & LXC. If you have +docker installed, you may find that the internet connectivity of your LXD +containers no longer work. + +Whenever docker is started (reboot) it sets the iptables policy for the +``FORWARD`` chain to ``DROP`` `[ref] +<https://docs.docker.com/network/iptables/#docker-on-a-router>`__:: + + $ sudo -H iptables-save | grep FORWARD + :FORWARD ACCEPT [7048:7851230] + :FORWARD DROP [7048:7851230] + +A handy solution of this problem might be to reset the policy for the +``FORWARD`` chain after the network has been initialized. For this create a +file in the ``if-up`` section of the network (``/etc/network/if-up.d/iptable``) +and insert the following lines:: + + #!/bin/sh + iptables -F FORWARD + iptables -P FORWARD ACCEPT + +Don't forget to set the execution bit:: + + sudo chmod ugo+x /etc/network/if-up.d/iptable + +Reboot your system and check the iptables rules:: + + $ sudo -H iptables-save | grep FORWARD + :FORWARD ACCEPT [7048:7851230] + :FORWARD ACCEPT [7048:7851230] + + +.. _searxng lxc suite: + +SearXNG LXC suite +================= + +The intention of the *SearXNG LXC suite* is to build up a suite of containers +for development tasks or :ref:`buildhosts <Setup SearXNG buildhost>` with a very +small set of simple commands. At the end of the ``--help`` output the SearXNG +suite from the :ref:`lxc-searxng.env` is introduced:: + + $ sudo -H ./utils/lxc.sh --help + ... + LXC suite: searxng + Suite includes installation of SearXNG + images: ubu2004 ubu2204 fedora35 archlinux + containers: searxng-ubu2004 searxng-ubu2204 searxng-fedora35 searxng-archlinux + +As shown above there are images and containers build up on this images. To show +more info about the containers in the *SearXNG LXC suite* call ``show suite``. +If this is the first time you make use of the SearXNG LXC suite, no containers +are installed and the output is:: + + $ sudo -H ./utils/lxc.sh show suite + + LXC suite (searxng-*) + ===================== + + +------+-------+------+------+------+-----------+ + | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | + +------+-------+------+------+------+-----------+ + + WARN: container searxng-ubu2004 does not yet exists + WARN: container searxng-ubu2204 does not yet exists + WARN: container searxng-fedora35 does not yet exists + WARN: container searxng-archlinux does not yet exists + +If you do not want to run a command or a build in all containers, **you can +build just one**. Here by example in the container that is build upon the +*archlinux* image:: + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux pwd + +Otherwise, to apply a command to all containers you can use:: + + $ sudo -H ./utils/lxc.sh build + $ sudo -H ./utils/lxc.sh cmd -- ls -la . + +Running commands +---------------- + +**Inside containers, you can run scripts** from the :ref:`toolboxing` or run +what ever command you need. By example, to start a bash use:: + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux SearXNG]# + +.. _Good to know: + +Good to know +------------ + +Each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` **handle relative path names transparent**:: + + $ pwd + /share/SearXNG + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux pwd + INFO: [searxng-archlinux] pwd + /share/SearXNG + +The path ``/share/SearXNG`` will be different on your HOST system. The commands +in the container are executed by the ``root`` inside of the container. Compare +output of:: + + $ ls -li Makefile + 47712402 -rw-rw-r-- 1 markus markus 2923 Apr 19 13:52 Makefile + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ls -li Makefile + INFO: [searxng-archlinux] ls -li Makefile + 47712402 -rw-rw-r-- 1 root root 2923 Apr 19 11:52 Makefile + ... + +Since the path ``/share/SearXNG`` of the HOST system is wrapped into the +container under the same name, the shown ``Makefile`` (inode ``47712402``) in +the output is always the identical ``/share/SearXNG/Makefile`` from the HOST +system. In the example shown above the owner of the path in the container is +the ``root`` user of the container (and the timezone in the container is +different to HOST system). + + +.. _lxc.sh install suite: + +Install suite +------------- + +.. sidebar:: further read + + - :ref:`working in containers` + - :ref:`FORCE_TIMEOUT <FORCE_TIMEOUT>` + +To install the complete :ref:`SearXNG suite <lxc-searxng.env>` into **all** LXC_ +containers leave the container argument empty and run:: + + $ sudo -H ./utils/lxc.sh build + $ sudo -H ./utils/lxc.sh install suite + +To *build & install* suite only in one container you can use by example:: + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + +The command above installs a SearXNG suite (see :ref:`installation scripts`). +To :ref:`install a nginx <installation nginx>` reverse proxy (or alternatively +use :ref:`apache <installation apache>`):: + + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +Same operation just in one container of the suite:: + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +The :ref:`FORCE_TIMEOUT <FORCE_TIMEOUT>` environment is set to zero to run the +script without user interaction. + +To get the IP (URL) of the SearXNG service in the containers use ``show suite`` +command. To test instances from containers just open the URLs in your +WEB-Browser:: + + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL + + [searxng-ubu2110] SEARXNG_URL : http://n.n.n.170/searxng + [searxng-ubu2004] SEARXNG_URL : http://n.n.n.160/searxng + [searxnggfedora35] SEARXNG_URL : http://n.n.n.150/searxng + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng + +Clean up +-------- + +If there comes the time you want to **get rid off all** the containers and +**clean up local images** just type:: + + $ sudo -H ./utils/lxc.sh remove + $ sudo -H ./utils/lxc.sh remove images + + +.. _Setup SearXNG buildhost: + +Setup SearXNG buildhost +======================= + +You can **install the SearXNG buildhost environment** into one or all containers. +The installation procedure to set up a :ref:`build host<buildhosts>` takes its +time. Installation in all containers will take more time (time for another cup +of coffee). :: + + sudo -H ./utils/lxc.sh cmd -- ./utils/searxng.sh install buildhost + +To build (live) documentation inside a archlinux_ container:: + + sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.clean docs.live + ... + [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080 + +To get IP of the container and the port number *live docs* is listening:: + + $ sudo ./utils/lxc.sh show suite | grep docs.live + ... + [searxng-archlinux] INFO: (eth0) docs.live: http://n.n.n.140:8080/ + + +.. _lxc.sh help: + +Command Help +============ + +The ``--help`` output of the script is largely self-explanatory: + +.. program-output:: ../utils/lxc.sh --help + + +.. _lxc-searxng.env: + +SearXNG suite config +==================== + +The SearXNG suite is defined in the file :origin:`utils/lxc-searxng.env`: + +.. literalinclude:: ../../utils/lxc-searxng.env + :language: bash diff --git a/_sources/utils/searxng.sh.rst.txt b/_sources/utils/searxng.sh.rst.txt new file mode 100644 index 000000000..bedc1ba4c --- /dev/null +++ b/_sources/utils/searxng.sh.rst.txt @@ -0,0 +1,42 @@ + +.. _searxng.sh: + +==================== +``utils/searxng.sh`` +==================== + +To simplify the installation and maintenance of a SearXNG instance you can use the +script :origin:`utils/searxng.sh`. + +.. sidebar:: further reading + + - :ref:`architecture` + - :ref:`installation` + - :ref:`installation nginx` + - :ref:`installation apache` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Install +======= + +In most cases you will install SearXNG simply by running the command: + +.. code:: bash + + sudo -H ./utils/searx.sh install all + +The installation is described in chapter :ref:`installation basic`. + +.. _searxng.sh overview: + +Command Help +============ + +The ``--help`` output of the script is largely self-explanatory: + +.. program-output:: ../utils/searxng.sh --help |