diff options
Diffstat (limited to '_sources/admin/searx.favicons.rst.txt')
| -rw-r--r-- | _sources/admin/searx.favicons.rst.txt | 251 |
1 files changed, 251 insertions, 0 deletions
diff --git a/_sources/admin/searx.favicons.rst.txt b/_sources/admin/searx.favicons.rst.txt new file mode 100644 index 000000000..b05b3458b --- /dev/null +++ b/_sources/admin/searx.favicons.rst.txt @@ -0,0 +1,251 @@ +.. _favicons: + +======== +Favicons +======== + +.. sidebar:: warning + + Don't activate the favicons before reading the documentation. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Activating the favicons in SearXNG is very easy, but this **generates a +significantly higher load** in the client/server communication and increases +resources needed on the server. + +To mitigate these disadvantages, various methods have been implemented, +including a *cache*. The cache must be parameterized according to your own +requirements and maintained regularly. + +To activate favicons in SearXNG's result list, set a default +``favicon_resolver`` in the :ref:`search <settings search>` settings: + +.. code:: yaml + + search: + favicon_resolver: "duckduckgo" + +By default and without any extensions, SearXNG serves these resolvers: + +- ``duckduckgo`` +- ``allesedv`` +- ``google`` +- ``yandex`` + +With the above setting favicons are displayed, the user has the option to +deactivate this feature in his settings. If the user is to have the option of +selecting from several *resolvers*, a further setting is required / but this +setting will be discussed :ref:`later <register resolvers>` in this article, +first we have to setup the favicons cache. + +Infrastructure +============== + +The infrastructure for providing the favicons essentially consists of three +parts: + +- :py:obj:`Favicons-Proxy <.favicons.proxy>` (aka *proxy*) +- :py:obj:`Favicons-Resolvers <.favicons.resolvers>` (aka *resolver*) +- :py:obj:`Favicons-Cache <.favicons.cache>` (aka *cache*) + +To protect the privacy of users, the favicons are provided via a *proxy*. This +*proxy* is automatically activated with the above activation of a *resolver*. +Additional requests are required to provide the favicons: firstly, the *proxy* +must process the incoming requests and secondly, the *resolver* must make +outgoing requests to obtain the favicons from external sources. + +A *cache* has been developed to massively reduce both, incoming and outgoing +requests. This *cache* is also activated automatically with the above +activation of a *resolver*. In its defaults, however, the *cache* is minimal +and not well suitable for a production environment! + +.. _favicon cache setup: + +Setting up the cache +==================== + +To parameterize the *cache* and more settings of the favicons infrastructure, a +TOML_ configuration is created in the file ``/etc/searxng/favicons.toml``. + +.. code:: toml + + [favicons] + + cfg_schema = 1 # config's schema version no. + + [favicons.cache] + + db_url = "/var/cache/searxng/faviconcache.db" # default: "/tmp/faviconcache.db" + LIMIT_TOTAL_BYTES = 2147483648 # 2 GB / default: 50 MB + # HOLD_TIME = 5184000 # 60 days / default: 30 days + # BLOB_MAX_BYTES = 40960 # 40 KB / default 20 KB + # MAINTENANCE_MODE = "off" # default: "auto" + # MAINTENANCE_PERIOD = 600 # 10min / default: 1h + +:py:obj:`cfg_schema <.FaviconConfig.cfg_schema>`: + Is required to trigger any processes required for future upgrades / don't + change it. + +:py:obj:`cache.db_url <.FaviconCacheConfig.db_url>`: + The path to the (SQLite_) database file. The default path is in the `/tmp`_ + folder, which is deleted on every reboot and is therefore unsuitable for a + production environment. The FHS_ provides the folder for the + application cache + + The FHS_ provides the folder `/var/cache`_ for the cache of applications, so a + suitable storage location of SearXNG's caches is folder ``/var/cache/searxng``. + In container systems, a volume should be mounted for this folder and in a + standard installation (compare :ref:`create searxng user`), the folder must be + created and the user under which the SearXNG process is running must be given + write permission to this folder. + + .. code:: bash + + $ sudo mkdir /var/cache/searxng + $ sudo chown root:searxng /var/cache/searxng/ + $ sudo chmod g+w /var/cache/searxng/ + +:py:obj:`cache.LIMIT_TOTAL_BYTES <.FaviconCacheConfig.LIMIT_TOTAL_BYTES>`: + Maximum of bytes stored in the cache of all blobs. The limit is only reached + at each maintenance interval after which the oldest BLOBs are deleted; the + limit is exceeded during the maintenance period. + + .. attention:: + + If the maintenance period is too long or maintenance is switched + off completely, the cache grows uncontrollably. + +SearXNG hosters can change other parameters of the cache as required: + +- :py:obj:`cache.HOLD_TIME <.FaviconCacheConfig.HOLD_TIME>` +- :py:obj:`cache.BLOB_MAX_BYTES <.FaviconCacheConfig.BLOB_MAX_BYTES>` + + +Maintenance of the cache +------------------------ + +Regular maintenance of the cache is required! By default, regular maintenance +is triggered automatically as part of the client requests: + +- :py:obj:`cache.MAINTENANCE_MODE <.FaviconCacheConfig.MAINTENANCE_MODE>` (default ``auto``) +- :py:obj:`cache.MAINTENANCE_PERIOD <.FaviconCacheConfig.MAINTENANCE_PERIOD>` (default ``6000`` / 1h) + +As an alternative to maintenance as part of the client request process, it is +also possible to carry out maintenance using an external process. For example, +by creating a :man:`crontab` entry for maintenance: + +.. code:: bash + + $ python -m searx.favicons cache maintenance + +The following command can be used to display the state of the cache: + +.. code:: bash + + $ python -m searx.favicons cache state + + +.. _favicon proxy setup: + +Proxy configuration +=================== + +Most of the options of the :py:obj:`Favicons-Proxy <.favicons.proxy>` are +already set sensibly with settings from the :ref:`settings.yml <searxng +settings.yml>` and should not normally be adjusted. + +.. code:: toml + + [favicons.proxy] + + max_age = 5184000 # 60 days / default: 7 days (604800 sec) + + +:py:obj:`max_age <.FaviconProxyConfig.max_age>`: + The `HTTP Cache-Control max-age`_ response directive indicates that the + response remains fresh until N seconds after the response is generated. This + setting therefore determines how long a favicon remains in the client's cache. + As a rule, in the favicons infrastructure of SearXNG's this setting only + affects favicons whose byte size exceeds :ref:`BLOB_MAX_BYTES <favicon cache + setup>` (the other favicons that are already in the cache are embedded as + `data URL`_ in the :py:obj:`generated HTML <.favicons.proxy.favicon_url>`, + which can greatly reduce the number of additional requests). + +.. _register resolvers: + +Register resolvers +------------------ + +A :py:obj:`resolver <.favicon.resolvers>` is a function that obtains the favicon +from an external source. The resolver functions available to the user are +registered with their fully qualified name (FQN_) in a ``resolver_map``. + +If no ``resolver_map`` is defined in the ``favicon.toml``, the favicon +infrastructure of SearXNG generates this ``resolver_map`` automatically +depending on the ``settings.yml``. SearXNG would automatically generate the +following TOML configuration from the following YAML configuration: + +.. code:: yaml + + search: + favicon_resolver: "duckduckgo" + +.. code:: toml + + [favicons.proxy.resolver_map] + + "duckduckgo" = "searx.favicons.resolvers.duckduckgo" + +If this automatism is not desired, then (and only then) a separate +``resolver_map`` must be created. For example, to give the user two resolvers to +choose from, the following configuration could be used: + +.. code:: toml + + [favicons.proxy.resolver_map] + + "duckduckgo" = "searx.favicons.resolvers.duckduckgo" + "allesedv" = "searx.favicons.resolvers.allesedv" + # "google" = "searx.favicons.resolvers.google" + # "yandex" = "searx.favicons.resolvers.yandex" + +.. note:: + + With each resolver, the resource requirement increases significantly. + +The number of resolvers increases: + +- the number of incoming/outgoing requests and +- the number of favicons to be stored in the cache. + +In the following we list the resolvers available in the core of SearXNG, but via +the FQN_ it is also possible to implement your own resolvers and integrate them +into the *proxy*: + +- :py:obj:`searx.favicons.resolvers.duckduckgo` +- :py:obj:`searx.favicons.resolvers.allesedv` +- :py:obj:`searx.favicons.resolvers.google` +- :py:obj:`searx.favicons.resolvers.yandex` + + + +.. _SQLite: + https://www.sqlite.org/ +.. _FHS: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html +.. _`/var/cache`: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s05.html +.. _`/tmp`: + https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s18.html +.. _TOML: + https://toml.io/en/ +.. _HTTP Cache-Control max-age: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#response_directives +.. _data URL: + https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs +.. _FQN: https://en.wikipedia.org/wiki/Fully_qualified_name + |