diff options
Diffstat (limited to 'docs/admin/engines')
| -rw-r--r-- | docs/admin/engines/command-line-engines.rst | 79 | ||||
| -rw-r--r-- | docs/admin/engines/index.rst | 26 | ||||
| -rw-r--r-- | docs/admin/engines/nosql-engines.rst | 135 | ||||
| -rw-r--r-- | docs/admin/engines/private-engines.rst | 49 | ||||
| -rw-r--r-- | docs/admin/engines/recoll.rst | 50 | ||||
| -rw-r--r-- | docs/admin/engines/search-indexer-engines.rst | 136 | ||||
| -rw-r--r-- | docs/admin/engines/searx.engines.xpath.rst | 9 | ||||
| -rw-r--r-- | docs/admin/engines/settings.rst | 778 | ||||
| -rw-r--r-- | docs/admin/engines/sql-engines.rst | 166 |
9 files changed, 0 insertions, 1428 deletions
diff --git a/docs/admin/engines/command-line-engines.rst b/docs/admin/engines/command-line-engines.rst deleted file mode 100644 index e9535e74f..000000000 --- a/docs/admin/engines/command-line-engines.rst +++ /dev/null @@ -1,79 +0,0 @@ -.. _engine command: - -==================== -Command Line Engines -==================== - -.. sidebar:: info - - - :origin:`command.py <searx/engines/command.py>` - - :ref:`offline engines` - -With *command engines* administrators can run engines to integrate arbitrary -shell commands. - -When creating and enabling a ``command`` engine on a public instance, you must -be careful to avoid leaking private data. The easiest solution is to limit the -access by setting ``tokens`` as described in section :ref:`private engines`. - -The engine base is flexible. Only your imagination can limit the power of this -engine (and maybe security concerns). The following options are available: - -``command``: - A comma separated list of the elements of the command. A special token - ``{{QUERY}}`` tells where to put the search terms of the user. Example: - - .. code:: yaml - - ['ls', '-l', '-h', '{{QUERY}}'] - -``delimiter``: - A mapping containing a delimiter ``char`` and the *titles* of each element in - ``keys``. - -``parse_regex``: - A dict containing the regular expressions for each result key. - -``query_type``: - - The expected type of user search terms. Possible values: ``path`` and - ``enum``. - - ``path``: - Checks if the user provided path is inside the working directory. If not, - the query is not executed. - - ``enum``: - Is a list of allowed search terms. If the user submits something which is - not included in the list, the query returns an error. - -``query_enum``: - A list containing allowed search terms if ``query_type`` is set to ``enum``. - -``working_dir``: - - The directory where the command has to be executed. Default: ``./`` - -``result_separator``: - The character that separates results. Default: ``\n`` - -The example engine below can be used to find files with a specific name in the -configured working directory: - -.. code:: yaml - - - name: find - engine: command - command: ['find', '.', '-name', '{{QUERY}}'] - query_type: path - shortcut: fnd - delimiter: - chars: ' ' - keys: ['line'] - - -Acknowledgment -============== - -This development was sponsored by `Search and Discovery Fund -<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. diff --git a/docs/admin/engines/index.rst b/docs/admin/engines/index.rst deleted file mode 100644 index 92c723817..000000000 --- a/docs/admin/engines/index.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. _engines and settings: - -================== -Engines & Settings -================== - -.. sidebar:: Further reading .. - - - :ref:`settings engine` - - :ref:`engine settings` & :ref:`engine file` - -.. toctree:: - :maxdepth: 3 - - settings - -.. toctree:: - :maxdepth: 1 - - private-engines - recoll - sql-engines - nosql-engines - search-indexer-engines - command-line-engines - searx.engines.xpath diff --git a/docs/admin/engines/nosql-engines.rst b/docs/admin/engines/nosql-engines.rst deleted file mode 100644 index 93f1dd58e..000000000 --- a/docs/admin/engines/nosql-engines.rst +++ /dev/null @@ -1,135 +0,0 @@ -=============== -NoSQL databases -=============== - -.. sidebar:: further read - - - `NoSQL databases <https://en.wikipedia.org/wiki/NoSQL>`_ - - `redis.io <https://redis.io/>`_ - - `MongoDB <https://www.mongodb.com>`_ - -The following `NoSQL databases`_ are supported: - -- :ref:`engine redis_server` -- :ref:`engine mongodb` - -All of the engines above are just commented out in the :origin:`settings.yml -<searx/settings.yml>`, as you have to set various options and install -dependencies before using them. - -By default, the engines use the ``key-value`` template for displaying results / -see :origin:`simple <searx/templates/simple/result_templates/key-value.html>` -theme. If you are not satisfied with the original result layout, you can use -your own template, set ``result_template`` attribute to ``{template_name}`` and -place the templates at:: - - searx/templates/{theme_name}/result_templates/{template_name} - -Furthermore, if you do not wish to expose these engines on a public instance, you -can still add them and limit the access by setting ``tokens`` as described in -section :ref:`private engines`. - - -Configure the engines -===================== - -`NoSQL databases`_ are used for storing arbitrary data without first defining -their structure. - - -Extra Dependencies ------------------- - -For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to -install additional packages in Python's Virtual Environment of your SearXNG -instance. To switch into the environment (:ref:`searxng-src`) you can use -:ref:`searxng.sh`:: - - $ sudo utils/searxng.sh instance cmd bash - (searxng-pyenv)$ pip install ... - - -.. _engine redis_server: - -Redis Server ------------- - -.. _redis: https://github.com/andymccurdy/redis-py#installation - -.. sidebar:: info - - - ``pip install`` redis_ - - redis.io_ - - :origin:`redis_server.py <searx/engines/redis_server.py>` - - -Redis is an open source (BSD licensed), in-memory data structure (key value -based) store. Before configuring the ``redis_server`` engine, you must install -the dependency redis_. - -Select a database to search in and set its index in the option ``db``. You can -either look for exact matches or use partial keywords to find what you are -looking for by configuring ``exact_match_only``. You find an example -configuration below: - -.. code:: yaml - - # Required dependency: redis - - - name: myredis - shortcut : rds - engine: redis_server - exact_match_only: false - host: '127.0.0.1' - port: 6379 - enable_http: true - password: '' - db: 0 - -.. _engine mongodb: - -MongoDB -------- - -.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation - -.. sidebar:: info - - - ``pip install`` pymongo_ - - MongoDB_ - - :origin:`mongodb.py <searx/engines/mongodb.py>` - -MongoDB_ is a document based database program that handles JSON like data. -Before configuring the ``mongodb`` engine, you must install the dependency -redis_. - -In order to query MongoDB_, you have to select a ``database`` and a -``collection``. Furthermore, you have to select a ``key`` that is going to be -searched. MongoDB_ also supports the option ``exact_match_only``, so configure -it as you wish. Below is an example configuration for using a MongoDB -collection: - -.. code:: yaml - - # MongoDB engine - # Required dependency: pymongo - - - name: mymongo - engine: mongodb - shortcut: md - exact_match_only: false - host: '127.0.0.1' - port: 27017 - enable_http: true - results_per_page: 20 - database: 'business' - collection: 'reviews' # name of the db collection - key: 'name' # key in the collection to search for - - -Acknowledgment -============== - -This development was sponsored by `Search and Discovery Fund -<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. - diff --git a/docs/admin/engines/private-engines.rst b/docs/admin/engines/private-engines.rst deleted file mode 100644 index cc6ab2565..000000000 --- a/docs/admin/engines/private-engines.rst +++ /dev/null @@ -1,49 +0,0 @@ -.. _private engines: - -============================ -Private Engines (``tokens``) -============================ - -Administrators might find themselves wanting to limit access to some of the -enabled engines on their instances. It might be because they do not want to -expose some private information through :ref:`offline engines`. Or they would -rather share engines only with their trusted friends or colleagues. - -To solve this issue the concept of *private engines* exists. - - -A new option was added to engines named `tokens`. It expects a list of -strings. If the user making a request presents one of the tokens of an engine, -they can access information about the engine and make search requests. - -Example configuration to restrict access to the Arch Linux Wiki engine: - -.. code:: yaml - - - name: arch linux wiki - engine: archlinux - shortcut: al - tokens: [ 'my-secret-token' ] - - -Unless a user has configured the right token, the engine is going -to be hidden from him/her. It is not going to be included in the -list of engines on the Preferences page and in the output of -`/config` REST API call. - -Tokens can be added to one's configuration on the Preferences page -under "Engine tokens". The input expects a comma separated list of -strings. - -The distribution of the tokens from the administrator to the users -is not carved in stone. As providing access to such engines -implies that the admin knows and trusts the user, we do not see -necessary to come up with a strict process. Instead, -we would like to add guidelines to the documentation of the feature. - - -Acknowledgment -============== - -This development was sponsored by `Search and Discovery Fund -<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. diff --git a/docs/admin/engines/recoll.rst b/docs/admin/engines/recoll.rst deleted file mode 100644 index ab8530044..000000000 --- a/docs/admin/engines/recoll.rst +++ /dev/null @@ -1,50 +0,0 @@ -.. _engine recoll: - -============= -Recoll Engine -============= - -.. sidebar:: info - - - `Recoll <https://www.lesbonscomptes.com/recoll/>`_ - - `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_ - - :origin:`searx/engines/recoll.py` - -Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_ -does not offer WEB or API access, this can be achieved using recoll-webui_ - - -Configuration -============= - -You must configure the following settings: - -``base_url``: - Location where recoll-webui can be reached. - -``mount_prefix``: - Location where the file hierarchy is mounted on your *local* filesystem. - -``dl_prefix``: - Location where the file hierarchy as indexed by recoll can be reached. - -``search_dir``: - Part of the indexed file hierarchy to be search, if empty the full domain is - searched. - - -Example -======= - -Scenario: - -#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``, -#. the Recoll search interface can be reached at https://recoll.example.org/ and -#. the contents of this filesystem can be reached though https://download.example.org/reference - -.. code:: yaml - - base_url: https://recoll.example.org/ - mount_prefix: /export/documents - dl_prefix: https://download.example.org - search_dir: '' diff --git a/docs/admin/engines/search-indexer-engines.rst b/docs/admin/engines/search-indexer-engines.rst deleted file mode 100644 index 51a661896..000000000 --- a/docs/admin/engines/search-indexer-engines.rst +++ /dev/null @@ -1,136 +0,0 @@ -==================== -Local Search Engines -==================== - -.. sidebar:: further read - - - `Comparison to alternatives - <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_ - -Administrators might find themselves wanting to integrate locally running search -engines. The following ones are supported for now: - -* `Elasticsearch`_ -* `Meilisearch`_ -* `Solr`_ - -Each search engine is powerful, capable of full-text search. All of the engines -above are added to ``settings.yml`` just commented out, as you have to -``base_url`` for all them. - -Please note that if you are not using HTTPS to access these engines, you have to enable -HTTP requests by setting ``enable_http`` to ``True``. - -Furthermore, if you do not want to expose these engines on a public instance, you -can still add them and limit the access by setting ``tokens`` as described in -section :ref:`private engines`. - -.. _engine meilisearch: - -MeiliSearch -=========== - -.. sidebar:: info - - - :origin:`meilisearch.py <searx/engines/meilisearch.py>` - - `MeiliSearch <https://www.meilisearch.com>`_ - - `MeiliSearch Documentation <https://docs.meilisearch.com/>`_ - - `Install MeiliSearch - <https://docs.meilisearch.com/learn/getting_started/installation.html>`_ - -MeiliSearch_ is aimed at individuals and small companies. It is designed for -small-scale (less than 10 million documents) data collections. E.g. it is great -for storing web pages you have visited and searching in the contents later. - -The engine supports faceted search, so you can search in a subset of documents -of the collection. Furthermore, you can search in MeiliSearch_ instances that -require authentication by setting ``auth_token``. - -Here is a simple example to query a Meilisearch instance: - -.. code:: yaml - - - name: meilisearch - engine: meilisearch - shortcut: mes - base_url: http://localhost:7700 - index: my-index - enable_http: true - - -.. _engine elasticsearch: - -Elasticsearch -============= - -.. sidebar:: info - - - :origin:`elasticsearch.py <searx/engines/elasticsearch.py>` - - `Elasticsearch <https://www.elastic.co/elasticsearch/>`_ - - `Elasticsearch Guide - <https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html>`_ - - `Install Elasticsearch - <https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html>`_ - -Elasticsearch_ supports numerous ways to query the data it is storing. At the -moment the engine supports the most popular search methods (``query_type``): - -- ``match``, -- ``simple_query_string``, -- ``term`` and -- ``terms``. - -If none of the methods fit your use case, you can select ``custom`` query type -and provide the JSON payload to submit to Elasticsearch in -``custom_query_json``. - -The following is an example configuration for an Elasticsearch_ instance with -authentication configured to read from ``my-index`` index. - -.. code:: yaml - - - name: elasticsearch - shortcut: es - engine: elasticsearch - base_url: http://localhost:9200 - username: elastic - password: changeme - index: my-index - query_type: match - # custom_query_json: '{ ... }' - enable_http: true - -.. _engine solr: - -Solr -==== - -.. sidebar:: info - - - :origin:`solr.py <searx/engines/solr.py>` - - `Solr <https://solr.apache.org>`_ - - `Solr Resources <https://solr.apache.org/resources.html>`_ - - `Install Solr <https://solr.apache.org/guide/installing-solr.html>`_ - -Solr_ is a popular search engine based on Lucene, just like Elasticsearch_. But -instead of searching in indices, you can search in collections. - -This is an example configuration for searching in the collection -``my-collection`` and get the results in ascending order. - -.. code:: yaml - - - name: solr - engine: solr - shortcut: slr - base_url: http://localhost:8983 - collection: my-collection - sort: asc - enable_http: true - - -Acknowledgment -============== - -This development was sponsored by `Search and Discovery Fund -<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. diff --git a/docs/admin/engines/searx.engines.xpath.rst b/docs/admin/engines/searx.engines.xpath.rst deleted file mode 100644 index 695aa5224..000000000 --- a/docs/admin/engines/searx.engines.xpath.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _xpath engine: - -============ -XPath Engine -============ - -.. automodule:: searx.engines.xpath - :members: - diff --git a/docs/admin/engines/settings.rst b/docs/admin/engines/settings.rst deleted file mode 100644 index 250a27461..000000000 --- a/docs/admin/engines/settings.rst +++ /dev/null @@ -1,778 +0,0 @@ -.. _settings.yml: - -================ -``settings.yml`` -================ - -This page describe the options possibilities of the :origin:`searx/settings.yml` -file. - -.. sidebar:: Further reading .. - - - :ref:`use_default_settings.yml` - - :ref:`search API` - -.. contents:: Contents - :depth: 2 - :local: - :backlinks: entry - -.. _settings location: - -settings.yml location -===================== - -The initial ``settings.yml`` we be load from these locations: - -1. the full path specified in the ``SEARXNG_SETTINGS_PATH`` environment variable. -2. ``/etc/searxng/settings.yml`` - -If these files don't exist (or are empty or can't be read), SearXNG uses the -:origin:`searx/settings.yml` file. Read :ref:`settings use_default_settings` to -see how you can simplify your *user defined* ``settings.yml``. - - -.. _settings global: - -Global Settings -=============== - -.. _settings brand: - -``brand:`` ----------- - -.. code:: yaml - - brand: - issue_url: https://github.com/searxng/searxng/issues - docs_url: https://docs.searxng.org - public_instances: https://searx.space - wiki_url: https://github.com/searxng/searxng/wiki - -``issue_url`` : - If you host your own issue tracker change this URL. - -``docs_url`` : - If you host your own documentation change this URL. - -``public_instances`` : - If you host your own https://searx.space change this URL. - -``wiki_url`` : - Link to your wiki (or ``false``) - -.. _settings general: - -``general:`` ------------- - -.. code:: yaml - - general: - debug: false - instance_name: "SearXNG" - privacypolicy_url: false - donation_url: false - contact_url: false - enable_metrics: true - -``debug`` : ``$SEARXNG_DEBUG`` - Allow a more detailed log if you run SearXNG directly. Display *detailed* error - messages in the browser too, so this must be deactivated in production. - -``donation_url`` : - Set value to ``true`` to use your own donation page written in the - :ref:`searx/info/en/donate.md <searx.infopage>` and use ``false`` to disable - the donation link altogether. - -``privacypolicy_url``: - Link to privacy policy. - -``contact_url``: - Contact ``mailto:`` address or WEB form. - -``enable_metrics``: - Enabled by default. Record various anonymous metrics availabled at ``/stats``, - ``/stats/errors`` and ``/preferences``. - -.. _settings search: - -``search:`` ------------ - -.. code:: yaml - - search: - safe_search: 0 - autocomplete: "" - default_lang: "" - ban_time_on_fail: 5 - max_ban_time_on_fail: 120 - suspended_times: - SearxEngineAccessDenied: 86400 - SearxEngineCaptcha: 86400 - SearxEngineTooManyRequests: 3600 - cf_SearxEngineCaptcha: 1296000 - cf_SearxEngineAccessDenied: 86400 - recaptcha_SearxEngineCaptcha: 604800 - formats: - - html - -``safe_search``: - Filter results. - - - ``0``: None - - ``1``: Moderate - - ``2``: Strict - -``autocomplete``: - Existing autocomplete backends, leave blank to turn it off. - - - ``dbpedia`` - - ``duckduckgo`` - - ``google`` - - ``startpage`` - - ``swisscows`` - - ``qwant`` - - ``wikipedia`` - -``default_lang``: - Default search language - leave blank to detect from browser information or - use codes from :origin:`searx/languages.py`. - -``languages``: - List of available languages - leave unset to use all codes from - :origin:`searx/languages.py`. Otherwise list codes of available languages. - The ``all`` value is shown as the ``Default language`` in the user interface - (in most cases, it is meant to send the query without a language parameter ; - in some cases, it means the English language) Example: - - .. code:: yaml - - languages: - - all - - en - - en-US - - de - - it-IT - - fr - - fr-BE - -``ban_time_on_fail``: - Ban time in seconds after engine errors. - -``max_ban_time_on_fail``: - Max ban time in seconds after engine errors. - -``suspended_times``: - Engine suspension time after error (in seconds; set to 0 to disable) - - ``SearxEngineAccessDenied``: 86400 - For error "Access denied" and "HTTP error [402, 403]" - - ``SearxEngineCaptcha``: 86400 - For error "CAPTCHA" - - ``SearxEngineTooManyRequests``: 3600 - For error "Too many request" and "HTTP error 429" - - Cloudflare CAPTCHA: - - ``cf_SearxEngineCaptcha``: 1296000 - - ``cf_SearxEngineAccessDenied``: 86400 - - Google CAPTCHA: - - ``recaptcha_SearxEngineCaptcha``: 604800 - -``formats``: - Result formats available from web, remove format to deny access (use lower - case). - - - ``html`` - - ``csv`` - - ``json`` - - ``rss`` - - -.. _settings server: - -``server:`` ------------ - -.. code:: yaml - - server: - base_url: http://example.org/location # change this! - port: 8888 - bind_address: "127.0.0.1" - secret_key: "ultrasecretkey" # change this! - limiter: false - image_proxy: false - default_http_headers: - X-Content-Type-Options : nosniff - X-XSS-Protection : 1; mode=block - X-Download-Options : noopen - X-Robots-Tag : noindex, nofollow - Referrer-Policy : no-referrer - - -``base_url`` : ``$SEARXNG_URL`` :ref:`buildenv <make buildenv>` - The base URL where SearXNG is deployed. Used to create correct inbound links. - If you change the value, don't forget to rebuild instance's environment - (:ref:`utils/brand.env <make buildenv>`) - -``port`` & ``bind_address``: ``$SEARXNG_PORT`` & ``$SEARXNG_BIND_ADDRESS`` :ref:`buildenv <make buildenv>` - Port number and *bind address* of the SearXNG web application if you run it - directly using ``python searx/webapp.py``. Doesn't apply to a SearXNG - services running behind a proxy and using socket communications. If you - change the value, don't forget to rebuild instance's environment - (:ref:`utils/brand.env <make buildenv>`) - -``secret_key`` : ``$SEARXNG_SECRET`` - Used for cryptography purpose. - -.. _limiter: - -``limiter`` : - Rate limit the number of request on the instance, block some bots. The - :ref:`limiter src` requires a :ref:`settings redis` database. - -.. _image_proxy: - -``image_proxy`` : - Allow your instance of SearXNG of being able to proxy images. Uses memory space. - -.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers - -``default_http_headers`` : - Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__ - - -.. _settings ui: - -``ui:`` -------- - -.. _cache busting: - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control#caching_static_assets_with_cache_busting - -.. code:: yaml - - ui: - static_use_hash: false - default_locale: "" - query_in_title: false - infinite_scroll: false - center_alignment: false - cache_url: https://web.archive.org/web/ - default_theme: simple - theme_args: - simple_style: auto - -.. _static_use_hash: - -``static_use_hash`` : - Enables `cache busting`_ of static files. - -``default_locale`` : - SearXNG interface language. If blank, the locale is detected by using the - browser language. If it doesn't work, or you are deploying a language - specific instance of searx, a locale can be defined using an ISO language - code, like ``fr``, ``en``, ``de``. - -``query_in_title`` : - When true, the result page's titles contains the query it decreases the - privacy, since the browser can records the page titles. - -``infinite_scroll``: - When true, automatically loads the next page when scrolling to bottom of the current page. - -``center_alignment`` : default ``false`` - When enabled, the results are centered instead of being in the left (or RTL) - side of the screen. This setting only affects the *desktop layout* - (:origin:`min-width: @tablet <searx/static/themes/simple/src/less/definitions.less>`) - -.. cache_url: - -``cache_url`` : ``https://web.archive.org/web/`` - URL prefix of the internet archive or cache, don't forgett trailing slash (if - needed). The default is https://web.archive.org/web/ alternatives are: - - - https://webcache.googleusercontent.com/search?q=cache: - - https://archive.today/ - -``default_theme`` : - Name of the theme you want to use by default on your SearXNG instance. - -``theme_args.simple_style``: - Style of simple theme: ``auto``, ``light``, ``dark`` - -``results_on_new_tab``: - Open result links in a new tab by default. - -.. _settings redis: - -``redis:`` ----------- - -.. _Redis.from_url(url): https://redis-py.readthedocs.io/en/stable/connections.html#redis.client.Redis.from_url - -A redis DB can be connected by an URL, in :py:obj:`searx.redisdb` you -will find a description to test your redis connection in SerXNG. When using -sockets, don't forget to check the access rights on the socket:: - - ls -la /usr/local/searxng-redis/run/redis.sock - srwxrwx--- 1 searxng-redis searxng-redis ... /usr/local/searxng-redis/run/redis.sock - -In this example read/write access is given to the *searxng-redis* group. To get -access rights to redis instance (the socket), your SearXNG (or even your -developer) account needs to be added to the *searxng-redis* group. - -``url`` : ``$SEARXNG_REDIS_URL`` - URL to connect redis database, see `Redis.from_url(url)`_ & :ref:`redis db`:: - - redis://[[username]:[password]]@localhost:6379/0 - rediss://[[username]:[password]]@localhost:6379/0 - unix://[[username]:[password]]@/path/to/socket.sock?db=0 - -.. admonition:: Tip for developers - - To set up a local redis instance, first set the socket path of the Redis DB - in your YAML setting: - - .. code:: yaml - - redis: - url: unix:///usr/local/searxng-redis/run/redis.sock?db=0 - - Then use the following commands to install the redis instance :: - - $ ./manage redis.build - $ sudo -H ./manage redis.install - $ sudo -H ./manage redis.addgrp "${USER}" - # don't forget to logout & login to get member of group - - -.. _settings outgoing: - -``outgoing:`` -------------- - -Communication with search engines. - -.. code:: yaml - - outgoing: - request_timeout: 2.0 # default timeout in seconds, can be override by engine - max_request_timeout: 10.0 # the maximum timeout in seconds - useragent_suffix: "" # information like an email address to the administrator - pool_connections: 100 # Maximum number of allowable connections, or null - # for no limits. The default is 100. - pool_maxsize: 10 # Number of allowable keep-alive connections, or null - # to always allow. The default is 10. - enable_http2: true # See https://www.python-httpx.org/http2/ - # uncomment below section if you want to use a custom server certificate - # see https://www.python-httpx.org/advanced/#changing-the-verification-defaults - # and https://www.python-httpx.org/compatibility/#ssl-configuration - # verify: ~/.mitmproxy/mitmproxy-ca-cert.cer - # - # uncomment below section if you want to use a proxyq see: SOCKS proxies - # https://2.python-requests.org/en/latest/user/advanced/#proxies - # are also supported: see - # https://2.python-requests.org/en/latest/user/advanced/#socks - # - # proxies: - # all://: - # - http://proxy1:8080 - # - http://proxy2:8080 - # - # using_tor_proxy: true - # - # Extra seconds to add in order to account for the time taken by the proxy - # - # extra_proxy_timeout: 10.0 - # - -``request_timeout`` : - Global timeout of the requests made to others engines in seconds. A bigger - timeout will allow to wait for answers from slow engines, but in consequence - will slow SearXNG reactivity (the result page may take the time specified in the - timeout to load). Can be override by ``timeout`` in the :ref:`settings engine`. - -``useragent_suffix`` : - Suffix to the user-agent SearXNG uses to send requests to others engines. If an - engine wish to block you, a contact info here may be useful to avoid that. - -.. _Pool limit configuration: https://www.python-httpx.org/advanced/#pool-limit-configuration - -``pool_maxsize``: - Number of allowable keep-alive connections, or ``null`` to always allow. The - default is 10. See ``max_keepalive_connections`` `Pool limit configuration`_. - -``pool_connections`` : - Maximum number of allowable connections, or ``null`` # for no limits. The - default is 100. See ``max_connections`` `Pool limit configuration`_. - -``keepalive_expiry`` : - Number of seconds to keep a connection in the pool. By default 5.0 seconds. - See ``keepalive_expiry`` `Pool limit configuration`_. - - -.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying - -``proxies`` : - Define one or more proxies you wish to use, see `httpx proxies`_. - If there are more than one proxy for one protocol (http, https), - requests to the engines are distributed in a round-robin fashion. - -``source_ips`` : - If you use multiple network interfaces, define from which IP the requests must - be made. Example: - - * ``0.0.0.0`` any local IPv4 address. - * ``::`` any local IPv6 address. - * ``192.168.0.1`` - * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses - * ``fe80::60a2:1691:e5a2:ee1f`` - * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network. - * ``[ 192.168.0.1, fe80::/126 ]`` - -``retries`` : - Number of retry in case of an HTTP error. On each retry, SearXNG uses an - different proxy and source ip. - -``enable_http2`` : - Enable by default. Set to ``false`` to disable HTTP/2. - -.. _httpx verification defaults: https://www.python-httpx.org/advanced/#changing-the-verification-defaults -.. _httpx ssl configuration: https://www.python-httpx.org/compatibility/#ssl-configuration - -``verify``: : ``$SSL_CERT_FILE``, ``$SSL_CERT_DIR`` - Allow to specify a path to certificate. - see `httpx verification defaults`_. - - In addition to ``verify``, SearXNG supports the ``$SSL_CERT_FILE`` (for a file) and - ``$SSL_CERT_DIR`` (for a directory) OpenSSL variables. - see `httpx ssl configuration`_. - -``max_redirects`` : - 30 by default. Maximum redirect before it is an error. - -``using_tor_proxy`` : - Using tor proxy (``true``) or not (``false``) for all engines. The default is - ``false`` and can be overwritten in the :ref:`settings engine` - - - -.. _settings categories_as_tabs: - -``categories_as_tabs:`` ------------------------ - -A list of the categories that are displayed as tabs in the user interface. -Categories not listed here can still be searched with the :ref:`search-syntax`. - -.. code-block:: yaml - - categories_as_tabs: - general: - images: - videos: - news: - map: - music: - it: - science: - files: - social media: - -Engines are added to ``categories:`` (compare :ref:`engine categories`), the -categories listed in ``categories_as_tabs`` are shown as tabs in the UI. If -there are no active engines in a category, the tab is not displayed (e.g. if a -user disables all engines in a category). - -On the preferences page (``/preferences``) -- under *engines* -- there is an -additional tab, called *other*. In this tab are all engines listed that are not -in one of the UI tabs (not included in ``categories_as_tabs``). - -.. _settings engine: - -Engine settings -=============== - -.. sidebar:: Further reading .. - - - :ref:`configured engines` - - :ref:`engines-dev` - -In the code example below a *full fledged* example of a YAML setup from a dummy -engine is shown. Most of the options have a default value or even are optional. - -.. code:: yaml - - - name: example engine - engine: example - shortcut: demo - base_url: 'https://{language}.example.com/' - send_accept_language_header: false - categories: general - timeout: 3.0 - api_key: 'apikey' - disabled: false - language: en_US - tokens: [ 'my-secret-token' ] - weight: 1 - display_error_messages: true - about: - website: https://example.com - wikidata_id: Q306656 - official_api_documentation: https://example.com/api-doc - use_official_api: true - require_api_key: true - results: HTML - - # overwrite values from section 'outgoing:' - enable_http2: false - retries: 1 - max_connections: 100 - max_keepalive_connections: 10 - keepalive_expiry: 5.0 - using_tor_proxy: false - proxies: - http: - - http://proxy1:8080 - - http://proxy2:8080 - https: - - http://proxy1:8080 - - http://proxy2:8080 - - socks5://user:password@proxy3:1080 - - socks5h://user:password@proxy4:1080 - - # other network settings - enable_http: false - retry_on_http_error: true # or 403 or [404, 429] - - -``name`` : - Name that will be used across SearXNG to define this engine. In settings, on - the result page... - -``engine`` : - Name of the python file used to handle requests and responses to and from this - search engine. - -``shortcut`` : - Code used to execute bang requests (in this case using ``!bi``) - -``base_url`` : optional - Part of the URL that should be stable across every request. Can be useful to - use multiple sites using only one engine, or updating the site URL without - touching at the code. - -``send_accept_language_header`` : - Several engines that support languages (or regions) deal with the HTTP header - ``Accept-Language`` to build a response that fits to the locale. When this - option is activated, the language (locale) that is selected by the user is used - to build and send a ``Accept-Language`` header in the request to the origin - search engine. - -.. _engine categories: - -``categories`` : optional - Specifies to which categories the engine should be added. Engines can be - assigned to multiple categories. - - Categories can be shown as tabs (:ref:`settings categories_as_tabs`) in the - UI. A search in a tab (in the UI) will query all engines that are active in - this tab. In the preferences page (``/preferences``) -- under *engines* -- - users can select what engine should be active when querying in this tab. - - Alternatively, :ref:`\!bang <search-syntax>` can be used to search all engines - in a category, regardless of whether they are active or not, or whether they - are in a tab of the UI or not. For example, ``!dictionaries`` can be used to - query all search engines in that category (group). - -``timeout`` : optional - Timeout of the search with the current search engine. Overwrites - ``request_timeout`` from :ref:`settings outgoing`. **Be careful, it will - modify the global timeout of SearXNG.** - -``api_key`` : optional - In a few cases, using an API needs the use of a secret key. How to obtain them - is described in the file. - -``disabled`` : optional - To disable by default the engine, but not deleting it. It will allow the user - to manually activate it in the settings. - -``inactive``: optional - Remove the engine from the settings (*disabled & removed*). - -``language`` : optional - If you want to use another language for a specific engine, you can define it - by using the ISO code of language (and region), like ``fr``, ``en-US``, - ``de-DE``. - -``tokens`` : optional - A list of secret tokens to make this engine *private*, more details see - :ref:`private engines`. - -``weight`` : default ``1`` - Weighting of the results of this engine. - -``display_error_messages`` : default ``true`` - When an engine returns an error, the message is displayed on the user interface. - -``network`` : optional - Use the network configuration from another engine. - In addition, there are two default networks: - - - ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses) - - ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses) - -``enable_http`` : optional - Enable HTTP for this engine (by default only HTTPS is enabled). - -``retry_on_http_error`` : optional - Retry request on some HTTP status code. - - Example: - - * ``true`` : on HTTP status code between 400 and 599. - * ``403`` : on HTTP status code 403. - * ``[403, 429]``: on HTTP status code 403 and 429. - -``proxies`` : - Overwrites proxy settings from :ref:`settings outgoing`. - -``using_tor_proxy`` : - Using tor proxy (``true``) or not (``false``) for this engine. The default is - taken from ``using_tor_proxy`` of the :ref:`settings outgoing`. - -``max_keepalive_connection#s`` : - `Pool limit configuration`_, overwrites value ``pool_maxsize`` from - :ref:`settings outgoing` for this engine. - -``max_connections`` : - `Pool limit configuration`_, overwrites value ``pool_connections`` from - :ref:`settings outgoing` for this engine. - -``keepalive_expiry`` : - `Pool limit configuration`_, overwrites value ``keepalive_expiry`` from - :ref:`settings outgoing` for this engine. - -.. note:: - - A few more options are possible, but they are pretty specific to some - engines, and so won't be described here. - - -Example: Multilingual Search ----------------------------- - -SearXNG does not support true multilingual search. You have to use the language -prefix in your search query when searching in a different language. - -But there is a workaround: By adding a new search engine with a different -language, SearXNG will search in your default and other language. - -Example configuration in settings.yml for a German and English speaker: - -.. code-block:: yaml - - search: - default_lang : "de" - ... - - engines: - - name : google english - engine : google - language : en - ... - -When searching, the default google engine will return German results and -"google english" will return English results. - - -.. _settings use_default_settings: - -use_default_settings -==================== - -.. sidebar:: ``use_default_settings: true`` - - - :ref:`settings location` - - :ref:`use_default_settings.yml` - - :origin:`/etc/searxng/settings.yml <utils/templates/etc/searxng/settings.yml>` - -The user defined ``settings.yml`` is loaded from the :ref:`settings location` -and can relied on the default configuration :origin:`searx/settings.yml` using: - - ``use_default_settings: true`` - -``server:`` - In the following example, the actual settings are the default settings defined - in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and - the ``bind_address``: - - .. code-block:: yaml - - use_default_settings: true - server: - secret_key: "ultrasecretkey" # change this! - bind_address: "0.0.0.0" - -``engines:`` - With ``use_default_settings: true``, each settings can be override in a - similar way, the ``engines`` section is merged according to the engine - ``name``. In this example, SearXNG will load all the default engines, will - enable the ``bing`` engine and define a :ref:`token <private engines>` for - the arch linux engine: - - .. code-block:: yaml - - use_default_settings: true - server: - secret_key: "ultrasecretkey" # change this! - engines: - - name: arch linux wiki - tokens: ['$ecretValue'] - - name: bing - disabled: false - - -``engines:`` / ``remove:`` - It is possible to remove some engines from the default settings. The following - example is similar to the above one, but SearXNG doesn't load the the google - engine: - - .. code-block:: yaml - - use_default_settings: - engines: - remove: - - google - server: - secret_key: "ultrasecretkey" # change this! - engines: - - name: arch linux wiki - tokens: ['$ecretValue'] - -``engines:`` / ``keep_only:`` - As an alternative, it is possible to specify the engines to keep. In the - following example, SearXNG has only two engines: - - .. code-block:: yaml - - use_default_settings: - engines: - keep_only: - - google - - duckduckgo - server: - secret_key: "ultrasecretkey" # change this! - engines: - - name: google - tokens: ['$ecretValue'] - - name: duckduckgo - tokens: ['$ecretValue'] diff --git a/docs/admin/engines/sql-engines.rst b/docs/admin/engines/sql-engines.rst deleted file mode 100644 index 3cd3c8f73..000000000 --- a/docs/admin/engines/sql-engines.rst +++ /dev/null @@ -1,166 +0,0 @@ -.. _sql engines: - -=========== -SQL Engines -=========== - -.. sidebar:: further read - - - `SQLite <https://www.sqlite.org/index.html>`_ - - `PostgreSQL <https://www.postgresql.org>`_ - - `MySQL <https://www.mysql.com>`_ - -With the *SQL engines* you can bind SQL databases into SearXNG. The following -Relational Database Management System (RDBMS) are supported: - -- :ref:`engine sqlite` -- :ref:`engine postgresql` -- :ref:`engine mysql_server` - -All of the engines above are just commented out in the :origin:`settings.yml -<searx/settings.yml>`, as you have to set the required attributes for the -engines, e.g. ``database:`` ... - -.. code:: yaml - - - name: ... - engine: {sqlite|postgresql|mysql_server} - database: ... - result_template: {template_name} - query_str: ... - -By default, the engines use the ``key-value`` template for displaying results / -see :origin:`simple <searx/templates/simple/result_templates/key-value.html>` -theme. If you are not satisfied with the original result layout, you can use -your own template, set ``result_template`` attribute to ``{template_name}`` and -place the templates at:: - - searx/templates/{theme_name}/result_templates/{template_name} - -If you do not wish to expose these engines on a public instance, you can still -add them and limit the access by setting ``tokens`` as described in section -:ref:`private engines`. - - -Configure the engines -===================== - -The configuration of the new database engines are similar. You must put a valid -SQL-SELECT query in ``query_str``. At the moment you can only bind at most one -parameter in your query. By setting the attribute ``limit`` you can define how -many results you want from the SQL server. Basically, it is the same as the -``LIMIT`` keyword in SQL. - -Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines -rely on these keywords during paging. If you want to configure the number of -returned results use the option ``limit``. - -.. _engine sqlite: - -SQLite ------- - -.. sidebar:: info - - - :origin:`sqlite.py <searx/engines/sqlite.py>` - -.. _MediathekView: https://mediathekview.de/ - -SQLite is a small, fast and reliable SQL database engine. It does not require -any extra dependency. To demonstrate the power of database engines, here is a -more complex example which reads from a MediathekView_ (DE) movie database. For -this example of the SQlite engine download the database: - -- https://liste.mediathekview.de/filmliste-v2.db.bz2 - -and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g -Query to test: ``!mediathekview concert`` - -.. code:: yaml - - - name: mediathekview - engine: sqlite - disabled: False - categories: general - result_template: default.html - database: searx/data/filmliste-v2.db - query_str: >- - SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, - COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, - description AS content - FROM film - WHERE title LIKE :wildcard OR description LIKE :wildcard - ORDER BY duration DESC - - -Extra Dependencies ------------------- - -For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to -install additional packages in Python's Virtual Environment of your SearXNG -instance. To switch into the environment (:ref:`searxng-src`) you can use -:ref:`searxng.sh`:: - - $ sudo utils/searxng.sh instance cmd bash - (searxng-pyenv)$ pip install ... - - -.. _engine postgresql: - -PostgreSQL ----------- - -.. _psycopg2: https://www.psycopg.org/install - -.. sidebar:: info - - - :origin:`postgresql.py <searx/engines/postgresql.py>` - - ``pip install`` psycopg2_ - -PostgreSQL is a powerful and robust open source database. Before configuring -the PostgreSQL engine, you must install the dependency ``psychopg2``. You can -find an example configuration below: - -.. code:: yaml - - - name: my_database - engine: postgresql - database: my_database - username: searxng - password: password - query_str: 'SELECT * from my_table WHERE my_column = %(query)s' - -.. _engine mysql_server: - -MySQL ------ - -.. sidebar:: info - - - :origin:`mysql_server.py <searx/engines/mysql_server.py>` - - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>` - -MySQL is said to be the most popular open source database. Before enabling MySQL -engine, you must install the package ``mysql-connector-python``. - -The authentication plugin is configurable by setting ``auth_plugin`` in the -attributes. By default it is set to ``caching_sha2_password``. This is an -example configuration for querying a MySQL server: - -.. code:: yaml - - - name: my_database - engine: mysql_server - database: my_database - username: searxng - password: password - limit: 5 - query_str: 'SELECT * from my_table WHERE my_column=%(query)s' - - -Acknowledgment -============== - -This development was sponsored by `Search and Discovery Fund -<https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. - |