diff options
Diffstat (limited to '_sources/dev')
66 files changed, 4898 insertions, 0 deletions
diff --git a/_sources/dev/contribution_guide.rst.txt b/_sources/dev/contribution_guide.rst.txt new file mode 100644 index 000000000..df5200637 --- /dev/null +++ b/_sources/dev/contribution_guide.rst.txt @@ -0,0 +1,190 @@ +.. _how to contribute: + +================= +How to contribute +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +Prime directives: Privacy, Hackability +====================================== + +SearXNG has two prime directives, **privacy-by-design and hackability** . The +hackability comes in three levels: + +- support of search engines +- plugins to alter search behaviour +- hacking SearXNG itself + +Note the lack of "world domination" among the directives. SearXNG has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +"privacy" deserves a separate chapter, as it's quite uncommon unfortunately. + +Privacy-by-design +----------------- + +SearXNG was born out of the need for a **privacy-respecting** search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities. + +A few widely used features work differently or turned off by default or not +implemented at all **as a consequence of privacy-by-design**. + +If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained. + +Also, if you think that something works weird with searx, it's might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST`` +requests in various browsers). + +Remember the other prime directive of SearXNG is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it. + + *Happy hacking.* + +Code +==== + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ +.. _Conventional Commits: https://www.conventionalcommits.org/ +.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages +.. _Structural split of changes: + https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes +.. _gitmoji: https://gitmoji.carloscuesta.me/ +.. _Semantic PR: https://github.com/zeke/semantic-pull-requests + +.. sidebar:: Create good commits! + + - `Structural split of changes`_ + - `Conventional Commits`_ + - `Git Commit Good Practice`_ + - some like to use: gitmoji_ + - not yet active: `Semantic PR`_ + +In order to submit a patch, please follow the steps below: + +- Follow coding conventions. + + - PEP8_ standards apply, except the convention of line length + - Maximum line length is 120 characters + +- The cardinal rule for creating good commits is to ensure there is only one + *logical change* per commit / read `Structural split of changes`_ + +- Check if your code breaks existing tests. If so, update the tests or fix your + code. + +- If your code can be unit-tested, add unit tests. + +- Add yourself to the :origin:`AUTHORS.rst` file. + +- Choose meaningful commit messages, read `Conventional Commits`_ + + .. code:: + + <type>[optional scope]: <description> + + [optional body] + + [optional footer(s)] + +- Create a pull request. + +For more help on getting started with SearXNG development, see :ref:`devquickstart`. + + +Translation +=========== + +Translation currently takes place on :ref:`weblate <translation>`. + + +.. _contrib docs: + +Documentation +============= + +.. _Sphinx: https://www.sphinx-doc.org +.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html + +.. sidebar:: The reST sources + + has been moved from ``gh-branch`` into ``master`` (:origin:`docs`). + +The documentation is built using Sphinx_. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +our :ref:`makefile`. + +Here is an example which makes a complete rebuild: + +.. code:: sh + + $ make docs.clean docs.html + ... + The HTML pages are in dist/docs. + +.. _make docs.live: + +live build +---------- + +.. _sphinx-autobuild: + https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md + +.. sidebar:: docs.clean + + It is recommended to assert a complete rebuild before deploying (use + ``docs.clean``). + +Live build is like WYSIWYG. If you want to edit the documentation, its +recommended to use. The Makefile target ``docs.live`` builds the docs, opens +URL in your favorite browser and rebuilds every time a reST file has been +changed (:ref:`make docs.clean`). + +.. code:: sh + + $ make docs.live + ... + The HTML pages are in dist/docs. + ... Serving on http://0.0.0.0:8000 + ... Start watching changes + +Live builds are implemented by sphinx-autobuild_. Use environment +``$(SPHINXOPTS)`` to pass arguments to the sphinx-autobuild_ command. Except +option ``--host`` (which is always set to ``0.0.0.0``) you can pass any +argument. E.g to find and use a free port, use: + +.. code:: sh + + $ SPHINXOPTS="--port 0" make docs.live + ... + ... Serving on http://0.0.0.0:50593 + ... + + +.. _deploy on github.io: + +deploy on github.io +------------------- + +To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make +docs.gh-pages`, which builds the documentation and runs all the needed git add, +commit and push: + +.. code:: sh + + $ make docs.clean docs.gh-pages + +.. attention:: + + If you are working in your own brand, don't forget to adjust your + :ref:`settings brand`. diff --git a/_sources/dev/engines/demo/demo_offline.rst.txt b/_sources/dev/engines/demo/demo_offline.rst.txt new file mode 100644 index 000000000..1b4cb887f --- /dev/null +++ b/_sources/dev/engines/demo/demo_offline.rst.txt @@ -0,0 +1,14 @@ +.. _demo offline engine: + +=================== +Demo Offline Engine +=================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_offline + :members: + diff --git a/_sources/dev/engines/demo/demo_online.rst.txt b/_sources/dev/engines/demo/demo_online.rst.txt new file mode 100644 index 000000000..9b94207f7 --- /dev/null +++ b/_sources/dev/engines/demo/demo_online.rst.txt @@ -0,0 +1,14 @@ +.. _demo online engine: + +================== +Demo Online Engine +================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.demo_online + :members: + diff --git a/_sources/dev/engines/engine_overview.rst.txt b/_sources/dev/engines/engine_overview.rst.txt new file mode 100644 index 000000000..1a6270a3b --- /dev/null +++ b/_sources/dev/engines/engine_overview.rst.txt @@ -0,0 +1,571 @@ +.. _engines-dev: + +=============== +Engine Overview +=============== + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine + +.. sidebar:: Further reading .. + + - :ref:`configured engines` + - :ref:`settings engine` + +SearXNG is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between SearXNG and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. + +.. _general engine configuration: + +General Engine Configuration +============================ + +It is required to tell SearXNG the type of results the engine provides. The +arguments can be set in the engine file or in the settings file (normally +``settings.yml``). The arguments in the settings file override the ones in the +engine file. + +It does not matter if an option is stored in the engine file or in the settings. +However, the standard way is the following: + +.. _engine file: + +Engine File +----------- + +.. table:: Common options in the engine module + :width: 100% + + ======================= =========== ======================================================== + argument type information + ======================= =========== ======================================================== + categories list categories, in which the engine is working + paging boolean support multiple pages + time_range_support boolean support search time range + engine_type str - ``online`` :ref:`[ref] <online engines>` by + default, other possibles values are: + - ``offline`` :ref:`[ref] <offline engines>` + - ``online_dictionary`` :ref:`[ref] <online dictionary>` + - ``online_currency`` :ref:`[ref] <online currency>` + - ``online_url_search`` :ref:`[ref] <online url search>` + ======================= =========== ======================================================== + +.. _engine settings: + +Engine ``settings.yml`` +----------------------- + +For a more detailed description, see :ref:`settings engine` in the :ref:`settings.yml`. + +.. table:: Common options in the engine setup (``settings.yml``) + :width: 100% + + ======================= =========== ================================================== + argument type information + ======================= =========== ================================================== + name string name of search-engine + engine string name of searxng-engine (file name without ``.py``) + enable_http bool enable HTTP (by default only HTTPS is enabled). + shortcut string shortcut of search-engine + timeout string specific timeout for search-engine + display_error_messages boolean display error messages on the web UI + proxies dict set proxies for a specific engine + (e.g. ``proxies : {http: socks5://proxy:port, + https: socks5://proxy:port}``) + ======================= =========== ================================================== + +.. _engine overrides: + +Overrides +--------- + +A few of the options have default values in the namespace of the engine's python +module, but are often overwritten by the settings. If ``None`` is assigned to an +option in the engine file, it has to be redefined in the settings, otherwise +SearXNG will not start with that engine (global names with a leading underline can +be ``None``). + +Here is an very simple example of the global names in the namespace of engine's +module: + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + _non_overwritten_global = 'foo' + + +.. table:: The naming of overrides is arbitrary / recommended overrides are: + :width: 100% + + ======================= =========== =========================================== + argument type information + ======================= =========== =========================================== + base_url string base-url, can be overwritten to use same + engine on other URL + number_of_results int maximum number of results per request + language string ISO code of language and country like en_US + api_key string api-key if required by engine + ======================= =========== =========================================== + +.. _engine request: + +Making a Request +================ + +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. + +.. _engine request arguments: + +Passed Arguments (request) +-------------------------- + +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. + +.. _engine request online: + +.. table:: If the ``engine_type`` is :py:obj:`online + <searx.search.processors.online.OnlineProcessor.get_params>` + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + url str ``''`` + method str ``'GET'`` + headers set ``{}`` + data set ``{}`` + cookies set ``{}`` + verify bool ``True`` + headers.User-Agent str a random User-Agent + category str current category, like ``'general'`` + safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) + time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` + pageno int current pagenumber + searxng_locale str SearXNG's locale selected by user. Specific language code like + ``'en'``, ``'en-US'``, or ``'all'`` if unspecified. + ====================== ============== ======================================================================== + + +.. _engine request online_dictionary: + +.. table:: If the ``engine_type`` is :py:obj:`online_dictionary + <searx.search.processors.online_dictionary.OnlineDictionaryProcessor.get_params>`, + in addition to the :ref:`online <engine request online>` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + from_lang str specific language code like ``'en_US'`` + to_lang str specific language code like ``'en_US'`` + query str the text query without the languages + ====================== ============== ======================================================================== + +.. _engine request online_currency: + +.. table:: If the ``engine_type`` is :py:obj:`online_currency + <searx.search.processors.online_currency.OnlineCurrencyProcessor.get_params>`, + in addition to the :ref:`online <engine request online>` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + amount float the amount to convert + from str ISO 4217 code + to str ISO 4217 code + from_name str currency name + to_name str currency name + ====================== ============== ======================================================================== + +.. _engine request online_url_search: + +.. table:: If the ``engine_type`` is :py:obj:`online_url_search + <searx.search.processors.online_url_search.OnlineUrlSearchProcessor.get_params>`, + in addition to the :ref:`online <engine request online>` arguments: + :width: 100% + + ====================== ============== ======================================================================== + argument type default-value, information + ====================== ============== ======================================================================== + search_url dict URLs from the search query: + + .. code:: python + + { + 'http': str, + 'ftp': str, + 'data:image': str + } + ====================== ============== ======================================================================== + +Specify Request +--------------- + +The function :py:func:`def request(query, params): +<searx.engines.demo_online.request>` always returns the ``params`` variable, the +following parameters can be used to specify a search request: + +.. table:: + :width: 100% + + =================== =========== ========================================================================== + argument type information + =================== =========== ========================================================================== + url str requested url + method str HTTP request method + headers set HTTP header information + data set HTTP data information + cookies set HTTP cookies + verify bool Performing SSL-Validity check + allow_redirects bool Follow redirects + max_redirects int maximum redirects, hard limit + soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine + raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 + =================== =========== ========================================================================== + + +.. _engine results: +.. _engine media types: + +Result Types (``template``) +=========================== + +Each result item of an engine can be of different media-types. Currently the +following media-types are supported. To set another media-type as +:ref:`template default`, the parameter ``template`` must be set to the desired +type. + +.. _template default: + +``default`` +----------- + +.. table:: Parameter of the **default** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + ========================= ===================================================== + + +.. _template images: + +``images`` +---------- + +.. list-table:: Parameter of the **images** media type + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``images.html`` + + * - url + - :py:class:`str` + - url to the result site + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - description of the image + + * - publishedDate + - :py:class:`datetime <datetime.datetime>` + - time of publish + + * - img_src + - :py:class:`str` + - url to the result image + + * - thumbnail_src + - :py:class:`str` + - url to a small-preview image + + * - resolution + - :py:class:`str` + - the resolution of the image (e.g. ``1920 x 1080`` pixel) + + * - img_format + - :py:class:`str` + - the format of the image (e.g. ``png``) + + * - filesize + - :py:class:`str` + - size of bytes in :py:obj:`human readable <searx.humanize_bytes>` notation + (e.g. ``MB`` for 1024 \* 1024 Bytes filesize). + + +.. _template videos: + +``videos`` +---------- + +.. table:: Parameter of the **videos** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``videos.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content *(not implemented yet)* + publishedDate :py:class:`datetime.datetime`, time of publish + thumbnail string, url to a small-preview image + length :py:class:`datetime.timedelta`, duration of result + views string, view count in humanized number format + ========================= ===================================================== + + +.. _template torrent: + +``torrent`` +----------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +.. table:: Parameter of the **torrent** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``torrent.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* + seed int, number of seeder + leech int, number of leecher + filesize int, size of file in bytes + files int, number of files + magnetlink string, magnetlink_ of the result + torrentfile string, torrentfile of the result + ========================= ===================================================== + + +.. _template map: + +``map`` +------- + +.. table:: Parameter of the **map** media type: + :width: 100% + + ========================= ===================================================== + result-parameter information + ------------------------- ----------------------------------------------------- + template is set to ``map.html`` + ========================= ===================================================== + url string, url of the result + title string, title of the result + content string, general result-text + publishedDate :py:class:`datetime.datetime`, time of publish + latitude latitude of result (in decimal format) + longitude longitude of result (in decimal format) + boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) + geojson geojson of result (https://geojson.org/) + osm.type type of osm-object (if OSM-Result) + osm.id id of osm-object (if OSM-Result) + address.name name of object + address.road street name of object + address.house_number house number of object + address.locality city, place of object + address.postcode postcode of object + address.country country of object + ========================= ===================================================== + + +.. _template paper: + +``paper`` +--------- + +.. _BibTeX format: https://www.bibtex.com/g/bibtex-format/ +.. _BibTeX field types: https://en.wikipedia.org/wiki/BibTeX#Field_types + +.. list-table:: Parameter of the **paper** media type / + see `BibTeX field types`_ and `BibTeX format`_ + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``paper.html`` + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - abstract + + * - comments + - :py:class:`str` + - free text display in italic below the content + + * - tags + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - free tag list + + * - publishedDate + - :py:class:`datetime <datetime.datetime>` + - last publication date + + * - type + - :py:class:`str` + - short description of medium type, e.g. *book*, *pdf* or *html* ... + + * - authors + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - list of authors of the work (authors with a "s") + + * - editor + - :py:class:`str` + - list of editors of a book + + * - publisher + - :py:class:`str` + - name of the publisher + + * - journal + - :py:class:`str` + - name of the journal or magazine the article was + published in + + * - volume + - :py:class:`str` + - volume number + + * - pages + - :py:class:`str` + - page range where the article is + + * - number + - :py:class:`str` + - number of the report or the issue number for a journal article + + * - doi + - :py:class:`str` + - DOI number (like ``10.1038/d41586-018-07848-2``) + + * - issn + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - ISSN number like ``1476-4687`` + + * - isbn + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - ISBN number like ``9780201896831`` + + * - pdf_url + - :py:class:`str` + - URL to the full article, the PDF version + + * - html_url + - :py:class:`str` + - URL to full article, HTML version + + +.. _template packages: + +``packages`` +------------ + +.. list-table:: Parameter of the **packages** media type + :header-rows: 2 + :width: 100% + + * - result-parameter + - Python type + - information + + * - template + - :py:class:`str` + - is set to ``packages.html`` + + * - title + - :py:class:`str` + - title of the result + + * - content + - :py:class:`str` + - abstract + + * - package_name + - :py:class:`str` + - the name of the package + + * - version + - :py:class:`str` + - the current version of the package + + * - maintainer + - :py:class:`str` + - the maintainer or author of the project + + * - publishedDate + - :py:class:`datetime <datetime.datetime>` + - date of latest update or release + + * - tags + - :py:class:`List <list>`\ [\ :py:class:`str`\ ] + - free tag list + + * - popularity + - :py:class:`str` + - the popularity of the package, e.g. rating or download count + + * - license_name + - :py:class:`str` + - the name of the license + + * - license_url + - :py:class:`str` + - the web location of a license copy + + * - homepage + - :py:class:`str` + - the url of the project's homepage + + * - source_code_url + - :py:class:`str` + - the location of the project's source code + + * - links + - :py:class:`dict` + - additional links in the form of ``{'link_name': 'http://example.com'}`` diff --git a/_sources/dev/engines/enginelib.rst.txt b/_sources/dev/engines/enginelib.rst.txt new file mode 100644 index 000000000..34e3250da --- /dev/null +++ b/_sources/dev/engines/enginelib.rst.txt @@ -0,0 +1,22 @@ +.. _searx.enginelib: + +============== +Engine Library +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.enginelib + :members: + +.. _searx.enginelib.traits: + + +Engine traits +============= + +.. automodule:: searx.enginelib.traits + :members: diff --git a/_sources/dev/engines/engines.rst.txt b/_sources/dev/engines/engines.rst.txt new file mode 100644 index 000000000..0f2cb1f22 --- /dev/null +++ b/_sources/dev/engines/engines.rst.txt @@ -0,0 +1,9 @@ +.. _searx.engines loader: + +======================== +SearXNG's engines loader +======================== + +.. automodule:: searx.engines + :members: + diff --git a/_sources/dev/engines/index.rst.txt b/_sources/dev/engines/index.rst.txt new file mode 100644 index 000000000..6558dc2c2 --- /dev/null +++ b/_sources/dev/engines/index.rst.txt @@ -0,0 +1,107 @@ +.. _engine implementations: + +====================== +Engine Implementations +====================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. toctree:: + :caption: Framework Components + :maxdepth: 2 + + enginelib + engines + engine_overview + + +Engine Types +============ + +The :py:obj:`engine_type <searx.enginelib.Engine.engine_type>` of an engine +determines which :ref:`search processor <searx.search.processors>` is used by +the engine. + +In this section a list of the engines that are documented is given, a complete +list of the engines can be found in the source under: :origin:`searx/engines`. + +.. _online engines: + +Online Engines +-------------- + +.. sidebar:: info + + - :py:obj:`processors.online <searx.search.processors.online>` + +.. toctree:: + :maxdepth: 1 + :glob: + + demo/demo_online + xpath + mediawiki + +.. toctree:: + :maxdepth: 1 + :glob: + + online/* + +.. _offline engines: + +Offline Engines +--------------- + +.. sidebar:: info + + - :py:obj:`processors.offline <searx.search.processors.offline>` + +.. toctree:: + :maxdepth: 1 + :glob: + + offline_concept + demo/demo_offline + offline/* + +.. _online url search: + +Online URL Search +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_url_search <searx.search.processors.online_url_search>` + +.. toctree:: + :maxdepth: 1 + :glob: + + online_url_search/* + +.. _online currency: + +Online Currency +--------------- + +.. sidebar:: info + + - :py:obj:`processors.online_currency <searx.search.processors.online_currency>` + +*no engine of this type is documented yet / coming soon* + +.. _online dictionary: + +Online Dictionary +----------------- + +.. sidebar:: info + + - :py:obj:`processors.online_dictionary <searx.search.processors.online_dictionary>` + +*no engine of this type is documented yet / coming soon* diff --git a/_sources/dev/engines/mediawiki.rst.txt b/_sources/dev/engines/mediawiki.rst.txt new file mode 100644 index 000000000..ce708f95b --- /dev/null +++ b/_sources/dev/engines/mediawiki.rst.txt @@ -0,0 +1,13 @@ +.. _mediawiki engine: + +================ +MediaWiki Engine +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mediawiki + :members: diff --git a/_sources/dev/engines/offline/command-line-engines.rst.txt b/_sources/dev/engines/offline/command-line-engines.rst.txt new file mode 100644 index 000000000..0a80d698e --- /dev/null +++ b/_sources/dev/engines/offline/command-line-engines.rst.txt @@ -0,0 +1,23 @@ +.. _engine command: + +==================== +Command Line Engines +==================== + +.. sidebar:: info + + - :origin:`command.py <searx/engines/command.py>` + - :ref:`offline engines` + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +.. automodule:: searx.engines.command + :members: diff --git a/_sources/dev/engines/offline/nosql-engines.rst.txt b/_sources/dev/engines/offline/nosql-engines.rst.txt new file mode 100644 index 000000000..76f5cfb61 --- /dev/null +++ b/_sources/dev/engines/offline/nosql-engines.rst.txt @@ -0,0 +1,97 @@ +.. _nosql engines: + +=============== +NoSQL databases +=============== + +.. sidebar:: further read + + - `NoSQL databases <https://en.wikipedia.org/wiki/NoSQL>`_ + - `redis.io <https://redis.io/>`_ + - `MongoDB <https://www.mongodb.com>`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +The following `NoSQL databases`_ are supported: + +- :ref:`engine redis_server` +- :ref:`engine mongodb` + +All of the engines above are just commented out in the :origin:`settings.yml +<searx/settings.yml>`, as you have to set various options and install +dependencies before using them. + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple <searx/templates/simple/result_templates/key-value.html>` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +Furthermore, if you do not wish to expose these engines on a public instance, you +can still add them and limit the access by setting ``tokens`` as described in +section :ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine redis_server` or :ref:`engine mongodb` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +`NoSQL databases`_ are used for storing arbitrary data without first defining +their structure. + + +.. _engine redis_server: + +Redis Server +------------ + +.. _redis: https://github.com/andymccurdy/redis-py#installation + +.. sidebar:: info + + - ``pip install`` redis_ + - redis.io_ + - :origin:`redis_server.py <searx/engines/redis_server.py>` + +.. automodule:: searx.engines.redis_server + :members: + + +.. _engine mongodb: + +MongoDB +------- + +.. _pymongo: https://github.com/mongodb/mongo-python-driver#installation + +.. sidebar:: info + + - ``pip install`` pymongo_ + - MongoDB_ + - :origin:`mongodb.py <searx/engines/mongodb.py>` + + +.. automodule:: searx.engines.mongodb + :members: + diff --git a/_sources/dev/engines/offline/search-indexer-engines.rst.txt b/_sources/dev/engines/offline/search-indexer-engines.rst.txt new file mode 100644 index 000000000..fa92798cb --- /dev/null +++ b/_sources/dev/engines/offline/search-indexer-engines.rst.txt @@ -0,0 +1,62 @@ +================= +Local Search APIs +================= + +.. sidebar:: further read + + - `Comparison to alternatives + <https://docs.meilisearch.com/learn/what_is_meilisearch/comparison_to_alternatives.html>`_ + +.. contents:: + :depth: 1 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +Administrators might find themselves wanting to integrate locally running search +engines. The following ones are supported for now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +Each search engine is powerful, capable of full-text search. All of the engines +above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to +enable HTTP requests by setting ``enable_http`` to ``True``. + +Furthermore, if you do not want to expose these engines on a public instance, +you can still add them and limit the access by setting ``tokens`` as described +in section :ref:`private engines`. + +.. _engine meilisearch: + +MeiliSearch +=========== + +.. automodule:: searx.engines.meilisearch + :members: + + +.. _engine elasticsearch: + +Elasticsearch +============= + +.. automodule:: searx.engines.elasticsearch + :members: + +.. _engine solr: + +Solr +==== + +.. automodule:: searx.engines.solr + :members: + diff --git a/_sources/dev/engines/offline/sql-engines.rst.txt b/_sources/dev/engines/offline/sql-engines.rst.txt new file mode 100644 index 000000000..f642f1f29 --- /dev/null +++ b/_sources/dev/engines/offline/sql-engines.rst.txt @@ -0,0 +1,134 @@ +.. _sql engines: + +=========== +SQL Engines +=========== + +.. sidebar:: further read + + - `SQLite <https://www.sqlite.org/index.html>`_ + - `PostgreSQL <https://www.postgresql.org>`_ + - `MySQL <https://www.mysql.com>`_ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. sidebar:: info + + Initial sponsored by `Search and Discovery Fund + <https://nlnet.nl/discovery>`_ of `NLnet Foundation <https://nlnet.nl/>`_. + +With the *SQL engines* you can bind SQL databases into SearXNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` & :ref:`engine mariadb_server` + +All of the engines above are just commented out in the :origin:`settings.yml +<searx/settings.yml>`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`simple <searx/templates/simple/result_templates/key-value.html>` +theme. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +If you do not wish to expose these engines on a public instance, you can still +add them and limit the access by setting ``tokens`` as described in section +:ref:`private engines`. + + +Extra Dependencies +================== + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearXNG +instance. To switch into the environment (:ref:`searxng-src`) you can use +:ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. sidebar:: info + + - :origin:`sqlite.py <searx/engines/sqlite.py>` + +.. automodule:: searx.engines.sqlite + :members: + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: info + + - :origin:`postgresql.py <searx/engines/postgresql.py>` + - ``pip install`` `psycopg2-binary <psycopg2>`_ + +.. automodule:: searx.engines.postgresql + :members: + +.. _engine mysql_server: + +MySQL +----- + +.. sidebar:: info + + - :origin:`mysql_server.py <searx/engines/mysql_server.py>` + - ``pip install`` :pypi:`mysql-connector-python <mysql-connector-python>` + + +.. automodule:: searx.engines.mysql_server + :members: + +.. _engine mariadb_server: + +MariaDB +-------- + +.. sidebar:: info + + - :origin:`mariadb_server.py <searx/engines/mariadb_server.py>` + - ``pip install`` :pypi:`mariadb <mariadb>` + + +.. automodule:: searx.engines.mariadb_server + :members: diff --git a/_sources/dev/engines/offline_concept.rst.txt b/_sources/dev/engines/offline_concept.rst.txt new file mode 100644 index 000000000..ddb34fc60 --- /dev/null +++ b/_sources/dev/engines/offline_concept.rst.txt @@ -0,0 +1,69 @@ +=============== +Offline Concept +=============== + +.. sidebar:: offline engines + + - :ref:`demo offline engine` + - :ref:`engine command` + - :ref:`sql engines` + - :ref:`nosql engines` + - :py:obj:`searx.search.processors.offline` + +To extend the functionality of SearXNG, offline engines are going to be +introduced. An offline engine is an engine which does not need Internet +connection to perform a search and does not use HTTP to communicate. + +Offline engines can be configured, by adding those to the `engines` list of +:origin:`settings.yml <searx/settings.yml>`. An example skeleton for offline +engines can be found in :ref:`demo offline engine` (:origin:`demo_offline.py +<searx/engines/demo_offline.py>`). + + +Programming Interface +===================== + +:py:func:`init(engine_settings=None) <searx.engines.demo_offline.init>` + All offline engines can have their own init function to setup the engine before + accepting requests. The function gets the settings from settings.yml as a + parameter. This function can be omitted, if there is no need to setup anything + in advance. + +:py:func:`search(query, params) <searx.engines.demo_offline.searc>` + Each offline engine has a function named ``search``. This function is + responsible to perform a search and return the results in a presentable + format. (Where *presentable* means presentable by the selected result + template.) + + The return value is a list of results retrieved by the engine. + +Engine representation in ``/config`` + If an engine is offline, the attribute ``offline`` is set to ``True``. + +.. _offline requirements: + +Extra Dependencies +================== + +If an offline engine depends on an external tool, SearXNG does not install it by +default. When an administrator configures such engine and starts the instance, +the process returns an error with the list of missing dependencies. Also, +required dependencies will be added to the comment/description of the engine, so +admins can install packages in advance. + +If there is a need to install additional packages in *Python's Virtual +Environment* of your SearXNG instance you need to switch into the environment +(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`:: + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... + + +Private engines (Security) +========================== + +To limit the access to offline engines, if an instance is available publicly, +administrators can set token(s) for each of the :ref:`private engines`. If a +query contains a valid token, then SearXNG performs the requested private +search. If not, requests from an offline engines return errors. + diff --git a/_sources/dev/engines/online/adobe_stock.rst.txt b/_sources/dev/engines/online/adobe_stock.rst.txt new file mode 100644 index 000000000..48a6511c0 --- /dev/null +++ b/_sources/dev/engines/online/adobe_stock.rst.txt @@ -0,0 +1,13 @@ +.. _adobe stock engine: + +=========== +Adobe Stock +=========== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.adobe_stock + :members: diff --git a/_sources/dev/engines/online/alpinelinux.rst.txt b/_sources/dev/engines/online/alpinelinux.rst.txt new file mode 100644 index 000000000..071a1985e --- /dev/null +++ b/_sources/dev/engines/online/alpinelinux.rst.txt @@ -0,0 +1,13 @@ +.. _alpinelinux engine: + +===================== +Alpine Linux Packages +===================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.alpinelinux + :members: diff --git a/_sources/dev/engines/online/annas_archive.rst.txt b/_sources/dev/engines/online/annas_archive.rst.txt new file mode 100644 index 000000000..db88e5069 --- /dev/null +++ b/_sources/dev/engines/online/annas_archive.rst.txt @@ -0,0 +1,13 @@ +.. _annas_archive engine: + +============== +Anna's Archive +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.annas_archive + :members: diff --git a/_sources/dev/engines/online/archlinux.rst.txt b/_sources/dev/engines/online/archlinux.rst.txt new file mode 100644 index 000000000..834fffa43 --- /dev/null +++ b/_sources/dev/engines/online/archlinux.rst.txt @@ -0,0 +1,14 @@ +.. _archlinux engine: + +========== +Arch Linux +========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.archlinux + :members: + diff --git a/_sources/dev/engines/online/bing.rst.txt b/_sources/dev/engines/online/bing.rst.txt new file mode 100644 index 000000000..19c31aa80 --- /dev/null +++ b/_sources/dev/engines/online/bing.rst.txt @@ -0,0 +1,43 @@ +.. _bing engines: + +============ +Bing Engines +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _bing web engine: + +Bing WEB +======== + +.. automodule:: searx.engines.bing + :members: + +.. _bing images engine: + +Bing Images +=========== + +.. automodule:: searx.engines.bing_images + :members: + +.. _bing videos engine: + +Bing Videos +=========== + +.. automodule:: searx.engines.bing_videos + :members: + +.. _bing news engine: + +Bing News +========= + +.. automodule:: searx.engines.bing_news + :members: diff --git a/_sources/dev/engines/online/bpb.rst.txt b/_sources/dev/engines/online/bpb.rst.txt new file mode 100644 index 000000000..f545dba48 --- /dev/null +++ b/_sources/dev/engines/online/bpb.rst.txt @@ -0,0 +1,13 @@ +.. _bpb engine: + +=== +Bpb +=== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.bpb + :members: diff --git a/_sources/dev/engines/online/brave.rst.txt b/_sources/dev/engines/online/brave.rst.txt new file mode 100644 index 000000000..a1c589b9d --- /dev/null +++ b/_sources/dev/engines/online/brave.rst.txt @@ -0,0 +1,13 @@ +.. _brave engine: + +============= +Brave Engines +============= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.brave + :members: diff --git a/_sources/dev/engines/online/bt4g.rst.txt b/_sources/dev/engines/online/bt4g.rst.txt new file mode 100644 index 000000000..980665204 --- /dev/null +++ b/_sources/dev/engines/online/bt4g.rst.txt @@ -0,0 +1,14 @@ +.. _bt4g engine: + +==== +BT4G +==== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.bt4g + :members: + diff --git a/_sources/dev/engines/online/dailymotion.rst.txt b/_sources/dev/engines/online/dailymotion.rst.txt new file mode 100644 index 000000000..c661172e5 --- /dev/null +++ b/_sources/dev/engines/online/dailymotion.rst.txt @@ -0,0 +1,13 @@ +.. _dailymotion engine: + +=========== +Dailymotion +=========== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.dailymotion + :members: diff --git a/_sources/dev/engines/online/discourse.rst.txt b/_sources/dev/engines/online/discourse.rst.txt new file mode 100644 index 000000000..1fab9e34c --- /dev/null +++ b/_sources/dev/engines/online/discourse.rst.txt @@ -0,0 +1,8 @@ +.. _discourse engine: + +================ +Discourse Forums +================ + +.. automodule:: searx.engines.discourse + :members: diff --git a/_sources/dev/engines/online/duckduckgo.rst.txt b/_sources/dev/engines/online/duckduckgo.rst.txt new file mode 100644 index 000000000..0f1258ff9 --- /dev/null +++ b/_sources/dev/engines/online/duckduckgo.rst.txt @@ -0,0 +1,22 @@ +.. _duckduckgo engines: + +================== +DuckDuckGo Engines +================== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.duckduckgo + :members: + +.. automodule:: searx.engines.duckduckgo_extra + :members: + +.. automodule:: searx.engines.duckduckgo_definitions + :members: + +.. automodule:: searx.engines.duckduckgo_weather + :members: diff --git a/_sources/dev/engines/online/geizhals.rst.txt b/_sources/dev/engines/online/geizhals.rst.txt new file mode 100644 index 000000000..766eb5f59 --- /dev/null +++ b/_sources/dev/engines/online/geizhals.rst.txt @@ -0,0 +1,8 @@ +.. _gitea geizhals: + +======== +Geizhals +======== + +.. automodule:: searx.engines.geizhals + :members: diff --git a/_sources/dev/engines/online/gitea.rst.txt b/_sources/dev/engines/online/gitea.rst.txt new file mode 100644 index 000000000..745504e4d --- /dev/null +++ b/_sources/dev/engines/online/gitea.rst.txt @@ -0,0 +1,8 @@ +.. _gitea engine: + +===== +Gitea +===== + +.. automodule:: searx.engines.gitea + :members: diff --git a/_sources/dev/engines/online/gitlab.rst.txt b/_sources/dev/engines/online/gitlab.rst.txt new file mode 100644 index 000000000..5f0d3e3d1 --- /dev/null +++ b/_sources/dev/engines/online/gitlab.rst.txt @@ -0,0 +1,8 @@ +.. _gitlab engine: + +====== +GitLab +====== + +.. automodule:: searx.engines.gitlab + :members: diff --git a/_sources/dev/engines/online/google.rst.txt b/_sources/dev/engines/online/google.rst.txt new file mode 100644 index 000000000..9085070bd --- /dev/null +++ b/_sources/dev/engines/online/google.rst.txt @@ -0,0 +1,76 @@ +.. _google engines: + +============== +Google Engines +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _google API: + +Google API +========== + +.. _Query Parameter Definitions: + https://developers.google.com/custom-search/docs/xml_results#WebSearch_Query_Parameter_Definitions + +SearXNG's implementation of the Google API is mainly done in +:py:obj:`get_google_info <searx.engines.google.get_google_info>`. + +For detailed description of the *REST-full* API see: `Query Parameter +Definitions`_. The linked API documentation can sometimes be helpful during +reverse engineering. However, we cannot use it in the freely accessible WEB +services; not all parameters can be applied and some engines are more *special* +than other (e.g. :ref:`google news engine`). + + +.. _google web engine: + +Google WEB +========== + +.. automodule:: searx.engines.google + :members: + +.. _google autocomplete: + +Google Autocomplete +==================== + +.. autofunction:: searx.autocomplete.google_complete + +.. _google images engine: + +Google Images +============= + +.. automodule:: searx.engines.google_images + :members: + +.. _google videos engine: + +Google Videos +============= + +.. automodule:: searx.engines.google_videos + :members: + +.. _google news engine: + +Google News +=========== + +.. automodule:: searx.engines.google_news + :members: + +.. _google scholar engine: + +Google Scholar +============== + +.. automodule:: searx.engines.google_scholar + :members: diff --git a/_sources/dev/engines/online/lemmy.rst.txt b/_sources/dev/engines/online/lemmy.rst.txt new file mode 100644 index 000000000..584246fd1 --- /dev/null +++ b/_sources/dev/engines/online/lemmy.rst.txt @@ -0,0 +1,13 @@ +.. _lemmy engine: + +===== +Lemmy +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.lemmy + :members: diff --git a/_sources/dev/engines/online/loc.rst.txt b/_sources/dev/engines/online/loc.rst.txt new file mode 100644 index 000000000..2ed76cd81 --- /dev/null +++ b/_sources/dev/engines/online/loc.rst.txt @@ -0,0 +1,13 @@ +.. _loc engine: + +=================== +Library of Congress +=================== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.loc + :members: diff --git a/_sources/dev/engines/online/mastodon.rst.txt b/_sources/dev/engines/online/mastodon.rst.txt new file mode 100644 index 000000000..dc372f121 --- /dev/null +++ b/_sources/dev/engines/online/mastodon.rst.txt @@ -0,0 +1,13 @@ +.. _mastodon engine: + +======== +Mastodon +======== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mastodon + :members: diff --git a/_sources/dev/engines/online/moviepilot.rst.txt b/_sources/dev/engines/online/moviepilot.rst.txt new file mode 100644 index 000000000..ba35574e5 --- /dev/null +++ b/_sources/dev/engines/online/moviepilot.rst.txt @@ -0,0 +1,13 @@ +.. _moviepilot engine: + +========== +Moviepilot +========== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.moviepilot + :members: diff --git a/_sources/dev/engines/online/mrs.rst.txt b/_sources/dev/engines/online/mrs.rst.txt new file mode 100644 index 000000000..671f35ea5 --- /dev/null +++ b/_sources/dev/engines/online/mrs.rst.txt @@ -0,0 +1,13 @@ +.. _mrs engine: + +========================= +Matrix Rooms Search (MRS) +========================= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mrs + :members: diff --git a/_sources/dev/engines/online/mullvad_leta.rst.txt b/_sources/dev/engines/online/mullvad_leta.rst.txt new file mode 100644 index 000000000..63aef019b --- /dev/null +++ b/_sources/dev/engines/online/mullvad_leta.rst.txt @@ -0,0 +1,13 @@ +.. _voidlinux mullvad_leta: + +============ +Mullvad-Leta +============ + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.mullvad_leta + :members: diff --git a/_sources/dev/engines/online/mwmbl.rst.txt b/_sources/dev/engines/online/mwmbl.rst.txt new file mode 100644 index 000000000..8eac7d7c5 --- /dev/null +++ b/_sources/dev/engines/online/mwmbl.rst.txt @@ -0,0 +1,27 @@ +.. _Mwmbl engine: + +============ +Mwmbl Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _mwmbl web engine: + +Mwmbl WEB +========= + +.. automodule:: searx.engines.mwmbl + :members: + + +.. _mwmbl autocomplete: + +Mwmbl Autocomplete +================== + +.. autofunction:: searx.autocomplete.mwmbl diff --git a/_sources/dev/engines/online/odysee.rst.txt b/_sources/dev/engines/online/odysee.rst.txt new file mode 100644 index 000000000..75be1ad11 --- /dev/null +++ b/_sources/dev/engines/online/odysee.rst.txt @@ -0,0 +1,13 @@ +.. _odysee engine: + +====== +Odysee +====== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.odysee + :members: diff --git a/_sources/dev/engines/online/peertube.rst.txt b/_sources/dev/engines/online/peertube.rst.txt new file mode 100644 index 000000000..bedf055fb --- /dev/null +++ b/_sources/dev/engines/online/peertube.rst.txt @@ -0,0 +1,27 @@ +.. _peertube engines: + +================ +Peertube Engines +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _peertube video engine: + +Peertube Video +============== + +.. automodule:: searx.engines.peertube + :members: + +.. _sepiasearch engine: + +SepiaSearch +=========== + +.. automodule:: searx.engines.sepiasearch + :members: diff --git a/_sources/dev/engines/online/piped.rst.txt b/_sources/dev/engines/online/piped.rst.txt new file mode 100644 index 000000000..822981056 --- /dev/null +++ b/_sources/dev/engines/online/piped.rst.txt @@ -0,0 +1,13 @@ +.. _piped engine: + +===== +Piped +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.piped + :members: diff --git a/_sources/dev/engines/online/presearch.rst.txt b/_sources/dev/engines/online/presearch.rst.txt new file mode 100644 index 000000000..59332c354 --- /dev/null +++ b/_sources/dev/engines/online/presearch.rst.txt @@ -0,0 +1,13 @@ +.. _engine presearch: + +================ +Presearch Engine +================ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.presearch + :members: diff --git a/_sources/dev/engines/online/qwant.rst.txt b/_sources/dev/engines/online/qwant.rst.txt new file mode 100644 index 000000000..66ad302d1 --- /dev/null +++ b/_sources/dev/engines/online/qwant.rst.txt @@ -0,0 +1,13 @@ +.. _qwant engine: + +===== +Qwant +===== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.qwant + :members: diff --git a/_sources/dev/engines/online/radio_browser.rst.txt b/_sources/dev/engines/online/radio_browser.rst.txt new file mode 100644 index 000000000..a150e59c5 --- /dev/null +++ b/_sources/dev/engines/online/radio_browser.rst.txt @@ -0,0 +1,13 @@ +.. _RadioBrowser engine: + +============ +RadioBrowser +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.radio_browser + :members: diff --git a/_sources/dev/engines/online/recoll.rst.txt b/_sources/dev/engines/online/recoll.rst.txt new file mode 100644 index 000000000..2f1a1e4df --- /dev/null +++ b/_sources/dev/engines/online/recoll.rst.txt @@ -0,0 +1,13 @@ +.. _engine recoll: + +============= +Recoll Engine +============= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.recoll + :members: diff --git a/_sources/dev/engines/online/seekr.rst.txt b/_sources/dev/engines/online/seekr.rst.txt new file mode 100644 index 000000000..fcbc7bf82 --- /dev/null +++ b/_sources/dev/engines/online/seekr.rst.txt @@ -0,0 +1,13 @@ +.. _seekr engine: + +============= +Seekr Engines +============= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.seekr + :members: diff --git a/_sources/dev/engines/online/startpage.rst.txt b/_sources/dev/engines/online/startpage.rst.txt new file mode 100644 index 000000000..89e3ad10b --- /dev/null +++ b/_sources/dev/engines/online/startpage.rst.txt @@ -0,0 +1,13 @@ +.. _startpage engines: + +================= +Startpage Engines +================= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.startpage + :members: diff --git a/_sources/dev/engines/online/tagesschau.rst.txt b/_sources/dev/engines/online/tagesschau.rst.txt new file mode 100644 index 000000000..f850bf99b --- /dev/null +++ b/_sources/dev/engines/online/tagesschau.rst.txt @@ -0,0 +1,13 @@ +.. _tagesschau engine: + +============== +Tagesschau API +============== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tagesschau + :members: diff --git a/_sources/dev/engines/online/torznab.rst.txt b/_sources/dev/engines/online/torznab.rst.txt new file mode 100644 index 000000000..9056b60d0 --- /dev/null +++ b/_sources/dev/engines/online/torznab.rst.txt @@ -0,0 +1,13 @@ +.. _torznab engine: + +============== +Torznab WebAPI +============== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.torznab + :members: diff --git a/_sources/dev/engines/online/void.rst.txt b/_sources/dev/engines/online/void.rst.txt new file mode 100644 index 000000000..a1839d4f2 --- /dev/null +++ b/_sources/dev/engines/online/void.rst.txt @@ -0,0 +1,13 @@ +.. _voidlinux engine: + +========================== +Void Linux binary packages +========================== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.voidlinux + :members: diff --git a/_sources/dev/engines/online/wallhaven.rst.txt b/_sources/dev/engines/online/wallhaven.rst.txt new file mode 100644 index 000000000..b473293ca --- /dev/null +++ b/_sources/dev/engines/online/wallhaven.rst.txt @@ -0,0 +1,13 @@ +.. _wallhaven engine: + +========= +Wallhaven +========= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.wallhaven + :members: diff --git a/_sources/dev/engines/online/wikipedia.rst.txt b/_sources/dev/engines/online/wikipedia.rst.txt new file mode 100644 index 000000000..d4920f0f6 --- /dev/null +++ b/_sources/dev/engines/online/wikipedia.rst.txt @@ -0,0 +1,27 @@ +.. _wikimedia engines: + +========= +Wikimedia +========= + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +.. _wikipedia engine: + +Wikipedia +========= + +.. automodule:: searx.engines.wikipedia + :members: + +.. _wikidata engine: + +Wikidata +========= + +.. automodule:: searx.engines.wikidata + :members: diff --git a/_sources/dev/engines/online/yacy.rst.txt b/_sources/dev/engines/online/yacy.rst.txt new file mode 100644 index 000000000..9407aca80 --- /dev/null +++ b/_sources/dev/engines/online/yacy.rst.txt @@ -0,0 +1,13 @@ +.. _yacy engine: + +==== +Yacy +==== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yacy + :members: diff --git a/_sources/dev/engines/online/yahoo.rst.txt b/_sources/dev/engines/online/yahoo.rst.txt new file mode 100644 index 000000000..96c1e2774 --- /dev/null +++ b/_sources/dev/engines/online/yahoo.rst.txt @@ -0,0 +1,13 @@ +.. _yahoo engine: + +============ +Yahoo Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.yahoo + :members: diff --git a/_sources/dev/engines/online/zlibrary.rst.txt b/_sources/dev/engines/online/zlibrary.rst.txt new file mode 100644 index 000000000..fb197abff --- /dev/null +++ b/_sources/dev/engines/online/zlibrary.rst.txt @@ -0,0 +1,13 @@ +.. _zlibrary engine: + +========= +Z-Library +========= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.zlibrary + :members: diff --git a/_sources/dev/engines/online_url_search/tineye.rst.txt b/_sources/dev/engines/online_url_search/tineye.rst.txt new file mode 100644 index 000000000..3f4db7e0e --- /dev/null +++ b/_sources/dev/engines/online_url_search/tineye.rst.txt @@ -0,0 +1,14 @@ +.. _tineye engine: + +====== +Tineye +====== + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.tineye + :members: + diff --git a/_sources/dev/engines/xpath.rst.txt b/_sources/dev/engines/xpath.rst.txt new file mode 100644 index 000000000..42c4d47b6 --- /dev/null +++ b/_sources/dev/engines/xpath.rst.txt @@ -0,0 +1,14 @@ +.. _xpath engine: + +============ +XPath Engine +============ + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +.. automodule:: searx.engines.xpath + :members: + diff --git a/_sources/dev/index.rst.txt b/_sources/dev/index.rst.txt new file mode 100644 index 000000000..09be9de5e --- /dev/null +++ b/_sources/dev/index.rst.txt @@ -0,0 +1,18 @@ +======================= +Developer documentation +======================= + +.. toctree:: + :maxdepth: 2 + + quickstart + rtm_asdf + contribution_guide + engines/index + search_api + plugins + translation + lxcdev + makefile + reST + searxng_extra/index diff --git a/_sources/dev/lxcdev.rst.txt b/_sources/dev/lxcdev.rst.txt new file mode 100644 index 000000000..79716ae57 --- /dev/null +++ b/_sources/dev/lxcdev.rst.txt @@ -0,0 +1,438 @@ +.. _lxcdev: + +============================== +Developing in Linux Containers +============================== + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +In this article we will show, how you can make use of Linux Containers (LXC_) in +*distributed and heterogeneous development cycles* (TL;DR; jump to the +:ref:`lxcdev summary`). + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers. Readers + should have a serious meaning about the terms: *distributed*, *merge* and + *linux container*. + + **hint** + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Motivation +========== + +Most often in our development cycle, we edit the sources and run some test +and/or builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook. + + **The environment in which we run all our development processes matters!** + +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer's desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer's desktop that would never +be set on a *production* system. + + **Linux Containers are isolate environments**, we use them to not mix up all + the prerequisites from various projects on developer's desktop. + +The scripts from :ref:`searx_utils` can divide in those to install and maintain +software + +- :ref:`searxng.sh` + +and the script + +- :ref:`lxc.sh` + +with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the: + +- :ref:`searxng lxc suite` + +.. _lxcdev install searxng: + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: The ``searxng-archlinux`` container + + is the base of all our exercises here. + +The :ref:`lxc-searxng.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. +For this blog post we exercise on a archlinux_ image. The container of this +image is named ``searxng-archlinux``. + +Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh remove searxng-archlinux + $ sudo -H ./utils/lxc.sh build searxng-archlinux + + +.. sidebar:: further read + + - :ref:`lxc.sh install suite` + - :ref:`installation nginx` + +To install the complete :ref:`SearXNG suite <searxng lxc suite>` and the HTTP +proxy :ref:`installation nginx` into the archlinux container run: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL + ... + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng + +.. sidebar:: Fully functional SearXNG suite + + From here on you have a fully functional SearXNG suite (including a + :ref:`redis db`). + +In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy. + +In the example above the SearXNG instance in the container is wrapped to +``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the +IP of your ``searxng-archlinux`` container is different to this example. To +test the instance in the container from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container is done by the ``./utils/lxc.sh cmd +searxng-archlinux`` command: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux SearXNG]$ + +The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + +Back in the browser on your desktop open the service http://n.n.n.140/searxng +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. + +Another point we have to notice is that the service :ref:`SearXNG <searxng.sh>` +runs under dedicated system user account with the same name (compare +:ref:`create searxng user`). To get a login shell from these accounts, simply +call: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l + INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + +The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system +user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv* +``(searxng-pyenv)`` environment is activated. + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional SearXNG suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- SearXNG setup in: ``/etc/searxng/settings.yml`` +- SearXNG user's home: ``/usr/local/searxng`` +- virtualenv in: ``/usr/local/searxng/searxng-pyenv`` +- SearXNG software in: ``/usr/local/searxng/searxng-src`` + +With the use of the :ref:`searxng.sh` the SearXNG service was installed as +:ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use +``systemctl`` (compare :ref:`uWSGI maintenance`). + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng + +With the command above, we stopped the SearXNG uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux distros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searxng/searxng-src/searx + virtualenv = /usr/local/searxng/searxng-pyenv + pythonpath = /usr/local/searxng/searxng-src + +If you have read the :ref:`Good to know` you remember, that each container +shares the root folder of the repository and the command ``utils/lxc.sh cmd`` +handles relative path names **transparent**. + +To wrap the SearXNG installation in the container into a developer one, we +simple have to create a symlink to the **transparent** repository from the +desktop. Now lets replace the repository at ``searxng-src`` in the container +with the working tree from outside of the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the SearXNG service (in the +container): + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +To *inspect* the SearXNG instance (already described above): + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searx.sh inspect service + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service + +Run :ref:`makefile`, e.g. to test inside the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make test + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test + + + +To install all prerequisites needed for a :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh install buildhost + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost + + +To build the docs on a buildhost :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make docs.html + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html + + +.. _lxcdev summary: + +Summary +======= + +We build up a fully functional SearXNG suite in a archlinux container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + ... + Developer install? (wraps source from HOST into the running instance) [YES/no] + +To wrap the suite into a developer one answer ``YES`` (or press Enter). + +.. code:: text + + link SearXNG's sources to: /share/SearXNG + ========================================= + + mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" + ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" + ls -ld /usr/local/searxng/searxng-src + |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG + +On code modification the instance has to be restarted (see :ref:`uWSGI +maintenance`): + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + +To access HTTP from the desktop we installed nginx for the services inside the +container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +To get information about the SearxNG suite in the archlinux container we can +use: + +.. code:: text + + $ sudo -H ./utils/lxc.sh show suite searxng-archlinux + [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ + [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] + [searxng-archlinux] uWSGI: + [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket + [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: + [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng + [searxng-archlinux] GIT_BRANCH : master + [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng + [searxng-archlinux] SEARXNG_PORT : 8888 + [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1 + diff --git a/_sources/dev/makefile.rst.txt b/_sources/dev/makefile.rst.txt new file mode 100644 index 000000000..383113bae --- /dev/null +++ b/_sources/dev/makefile.rst.txt @@ -0,0 +1,383 @@ +.. _makefile: + +======================= +Makefile & ``./manage`` +======================= + +.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction + +All relevant build and development tasks are implemented in the +:origin:`./manage <manage>` script and for CI or IDE integration a small +:origin:`Makefile` wrapper is available. If you are not familiar with +Makefiles, we recommend to read gnu-make_ introduction. + +.. sidebar:: build environment + + Before looking deeper at the targets, first read about :ref:`make + install`. + + To install developer requirements follow :ref:`buildhosts`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + +The usage is simple, just type ``make {target-name}`` to *build* a target. +Calling the ``help`` target gives a first overview (``make help``): + +.. tabs:: + + .. group-tab:: ``make`` + + .. program-output:: bash -c "cd ..; make --no-print-directory help" + + + .. group-tab:: ``./manage`` + + The Makefile targets are implemented for comfort, if you can do without + tab-completion and need to have a more granular control, use + :origin:`manage` without the Makefile wrappers. + + .. code:: sh + + $ ./manage help + +.. _make install: + +Python environment (``make install``) +===================================== + +.. sidebar:: activate environment + + ``source ./local/py3/bin/activate`` + +We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a ``make install`` to get a virtualenv with a +*developer install* of SearXNG (:origin:`setup.py`). :: + + $ cd ~/searxng-clone + $ make install + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]' + ... + Successfully installed searxng-2023.7.19+a446dea1b + +If you release ``make install`` multiple times the installation will only +rebuild if the sha256 sum of the *requirement files* fails. With other words: +the check fails if you edit the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`). :: + + $ make install + PYENV OK + PYENV [virtualenv] requirements.sha256 failed + [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt + [virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt + ... + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV [install] pip install --use-pep517 --no-build-isolation -e 'searx[test]' + ... + Successfully installed searxng-2023.7.19+a446dea1b + +.. sidebar:: drop environment + + To get rid of the existing environment before re-build use :ref:`clean target + <make clean>` first. + +If you think, something goes wrong with your ./local environment or you change +the :origin:`setup.py` file, you have to call :ref:`make clean`. + +.. _make node.env: + +Node.js environment (``make node.env``) +======================================= + +.. _Node.js: https://nodejs.org/ +.. _nvm: https://github.com/nvm-sh +.. _npm: https://www.npmjs.com/ + +.. jinja:: searx + + Node.js_ version {{version.node}} or higher is required to build the themes. + If the requirement is not met, the build chain uses nvm_ (Node Version + Manager) to install latest LTS of Node.js_ locally: there is no need to + install nvm_ or npm_ on your system. + +To install NVM_ and Node.js_ in once you can use :ref:`make nvm.nodejs`. + +.. _make nvm: + +NVM ``make nvm.install nvm.status`` +----------------------------------- + +Use ``make nvm.status`` to get the current status of your Node.js_ and nvm_ +setup. + +.. tabs:: + + .. group-tab:: nvm.install + + .. code:: sh + + $ LANG=C make nvm.install + INFO: install (update) NVM at ./searxng/.nvm + INFO: clone: https://github.com/nvm-sh/nvm.git + || Cloning into './searxng/.nvm'... + INFO: checkout v0.39.4 + || HEAD is now at 8fbf8ab v0.39.4 + + .. group-tab:: nvm.status (ubu2004) + + Here is the output you will typically get on a Ubuntu 20.04 system which + serves only a `no longer active <https://nodejs.org/en/about/releases/>`_ + Release `Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_. + + .. code:: sh + + $ make nvm.status + INFO: Node.js is installed at /usr/bin/node + INFO: Node.js is version v10.19.0 + WARN: minimal Node.js version is 16.13.0 + INFO: npm is installed at /usr/bin/npm + INFO: npm is version 6.14.4 + WARN: NVM is not installed + +.. _make nvm.nodejs: + +``make nvm.nodejs`` +------------------- + +Install latest Node.js_ LTS locally (uses nvm_):: + + $ make nvm.nodejs + INFO: install (update) NVM at /share/searxng/.nvm + INFO: clone: https://github.com/nvm-sh/nvm.git + ... + Downloading and installing node v16.13.0... + ... + INFO: Node.js is installed at searxng/.nvm/versions/node/v16.13.0/bin/node + INFO: Node.js is version v16.13.0 + INFO: npm is installed at searxng/.nvm/versions/node/v16.13.0/bin/npm + INFO: npm is version 8.1.0 + INFO: NVM is installed at searxng/.nvm + +.. _make run: + +``make run`` +============ + +To get up a running a developer instance simply call ``make run``. This enables +*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` +instance and opens the URL in your favorite WEB browser (:man:`xdg-open`):: + + $ make run + +Changes to theme's HTML templates (jinja2) are instant. Changes to the CSS & JS +sources of the theme need to be rebuild. You can do that by running:: + + $ make themes.all + +Alternatively to ``themes.all`` you can run *live builds* of the theme you are +modify (:ref:`make themes`):: + + $ LIVE_THEME=simple make run + +.. _make format.python: + +``make format.python`` +====================== + +Format Python source code using `Black code style`_. See ``$BLACK_OPTIONS`` +and ``$BLACK_TARGETS`` in :origin:`Makefile`. + +.. attention:: + + We stuck at Black 22.12.0, please read comment in PR `Bump black from 22.12.0 + to 23.1.0`_ + +.. _Bump black from 22.12.0 to 23.1.0: + https://github.com/searxng/searxng/pull/2159#pullrequestreview-1284094735 + +.. _Black code style: + https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html + +.. _make clean: + +``make clean`` +============== + +Drops all intermediate files, all builds, but keep sources untouched. Before +calling ``make clean`` stop all processes using the :ref:`make install` or +:ref:`make node.env`. :: + + $ make clean + CLEAN pyenv + PYENV [virtualenv] drop local/py3 + CLEAN docs -- build/docs dist/docs + CLEAN themes -- locally installed npm dependencies + ... + CLEAN test stuff + CLEAN common files + +.. _make docs: + +``make docs`` +============= + +Target ``docs`` builds the documentation: + +.. code:: bash + + $ make docs + HTML ./docs --> file:// + DOCS build build/docs/includes + ... + The HTML pages are in dist/docs. + +.. _make docs.clean: + +``make docs.clean docs.live`` +---------------------------------- + +We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / +Documentation <contrib docs>` section. If you want to edit the documentation +read our :ref:`make docs.live` section. If you are working in your own brand, +adjust your :ref:`settings brand`. + + +.. _make docs.gh-pages: + +``make docs.gh-pages`` +---------------------- + +To deploy on github.io first adjust your :ref:`settings brand`. For any +further read :ref:`deploy on github.io`. + +.. _make test: + +``make test`` +============= + +Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit`` +and ``test.robot``. You can run tests selective, e.g.:: + + $ make test.pep8 test.unit test.shell + TEST test.pep8 OK + ... + TEST test.unit OK + ... + TEST test.shell OK + +.. _make test.shell: + +``make test.shell`` +------------------- + +:ref:`sh lint` / if you have changed some bash scripting run this test before +commit. + +.. _make test.pylint: + +``make test.pylint`` +-------------------- + +.. _Pylint: https://www.pylint.org/ + +Pylint_ is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile used in the SearXNG project is +found in project's root folder :origin:`.pylintrc`. + +.. _make search.checker: + +``make search.checker.{engine name}`` +===================================== + +To check all engines:: + + make search.checker + +To check a engine with whitespace in the name like *google news* replace space +by underline:: + + make search.checker.google_news + +To see HTTP requests and more use SEARXNG_DEBUG:: + + make SEARXNG_DEBUG=1 search.checker.google_news + +.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection + +To filter out HTTP redirects (3xx_):: + + make SEARXNG_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]" + ... + Engine google news Checking + https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + +.. _make themes: + +``make themes.*`` +================= + +.. sidebar:: further read + + - :ref:`devquickstart` + +The :origin:`Makefile` targets ``make theme.*`` cover common tasks to build the +theme(s). The ``./manage themes.*`` command line can be used to convenient run +common theme build tasks. + +.. program-output:: bash -c "cd ..; ./manage themes.help" + +To get live builds while modifying CSS & JS use (:ref:`make run`): + +.. code:: sh + + $ LIVE_THEME=simple make run + +.. _make static.build: + +``make static.build.*`` +======================= + +.. sidebar:: further read + + - :ref:`devquickstart` + +The :origin:`Makefile` targets ``static.build.*`` cover common tasks to build (a +commit of) the static files. The ``./manage static.build..*`` command line +can be used to convenient run common build tasks of the static files. + +.. program-output:: bash -c "cd ..; ./manage static.help" + + +.. _manage redis.help: + +``./manage redis.help`` +======================= + +The ``./manage redis.*`` command line can be used to convenient run common Redis +tasks (:ref:`Redis developer notes`). + +.. program-output:: bash -c "cd ..; ./manage redis.help" + + +.. _manage go.help: + +``./manage go.help`` +==================== + +The ``./manage go.*`` command line can be used to convenient run common `go +(wiki)`_ tasks. + +.. _go (wiki): https://en.wikipedia.org/wiki/Go_(programming_language) + +.. program-output:: bash -c "cd ..; ./manage go.help" diff --git a/_sources/dev/plugins.rst.txt b/_sources/dev/plugins.rst.txt new file mode 100644 index 000000000..fb3201e66 --- /dev/null +++ b/_sources/dev/plugins.rst.txt @@ -0,0 +1,106 @@ +.. _dev plugin: + +======= +Plugins +======= + +.. sidebar:: Further reading .. + + - :ref:`plugins generic` + +Plugins can extend or replace functionality of various components of searx. + +Example plugin +============== + +.. code:: python + + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disabled by default + + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, search): + search.result_container.suggestions.add('example') + return True + +External plugins +================ + +SearXNG supports *external plugins* / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from *The Green Web Foundation* `[ref] +<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__: + +.. code:: bash + + $ sudo utils/searxng.sh instance cmd bash -c + (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins + +In the :ref:`settings.yml` activate the ``plugins:`` section and add module +``only_show_green_results`` from ``tgwf-searx-plugins``. + +.. code:: yaml + + plugins: + ... + - only_show_green_results + ... + + +Plugin entry points +=================== + +Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. A plugin doesn't need to implement all the hooks. + + +.. py:function:: pre_search(request, search) -> bool + + Runs BEFORE the search request. + + `search.result_container` can be changed. + + Return a boolean: + + * True to continue the search + * False to stop the search + + :param flask.request request: + :param searx.search.SearchWithPlugins search: + :return: False to stop the search + :rtype: bool + + +.. py:function:: post_search(request, search) -> None + + Runs AFTER the search request. + + :param flask.request request: Flask request. + :param searx.search.SearchWithPlugins search: Context. + + +.. py:function:: on_result(request, search, result) -> bool + + Runs for each result of each engine. + + `result` can be changed. + + If `result["url"]` is defined, then `result["parsed_url"] = urlparse(result['url'])` + + .. warning:: + `result["url"]` can be changed, but `result["parsed_url"]` must be updated too. + + Return a boolean: + + * True to keep the result + * False to remove the result + + :param flask.request request: + :param searx.search.SearchWithPlugins search: + :param typing.Dict result: Result, see - :ref:`engine results` + :return: True to keep the result + :rtype: bool diff --git a/_sources/dev/quickstart.rst.txt b/_sources/dev/quickstart.rst.txt new file mode 100644 index 000000000..c45c24491 --- /dev/null +++ b/_sources/dev/quickstart.rst.txt @@ -0,0 +1,82 @@ +.. _devquickstart: + +====================== +Development Quickstart +====================== + +.. _npm: https://www.npmjs.com/ +.. _Node.js: https://nodejs.org/ + + +.. sidebar:: further read + + - :ref:`makefile` + - :ref:`buildhosts` + +SearXNG loves developers; Developers do not need to worry about tool chains, the +usual developer tasks can be comfortably executed via :ref:`make <makefile>`. + +Don't hesitate, just clone SearXNG's sources and start hacking right now .. + +.. code:: bash + + git clone https://github.com/searxng/searxng.git searxng + +Here is how a minimal workflow looks like: + +1. *start* hacking +2. *run* your code: :ref:`make run` +3. *format & test* your code: :ref:`make format.python` and :ref:`make test` + +If you think at some point something fails, go back to *start*. Otherwise, +choose a meaningful commit message and we are happy to receive your pull +request. To not end in *wild west* we have some directives, please pay attention +to our ":ref:`how to contribute`" guideline. + +.. sidebar:: further read + + - :ref:`make nvm` + - :ref:`make themes` + +If you implement themes, you will need to setup a :ref:`Node.js environment +<make node.env>`: ``make node.env`` + +Before you call *make run* (2.), you need to compile the modified styles and +JavaScript: ``make themes.all`` + +Alternatively you can also compile selective the theme you have modified, +e.g. the *simple* theme. + +.. code:: bash + + make themes.simple + +.. tip:: + + To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run`` + +.. sidebar:: further read + + - :ref:`make static.build` + +If you finished your *tests* you can start to commit your changes. To separate +the modified source code from the build products first run: + +.. code:: bash + + make static.build.restore + +This will restore the old build products and only your changes of the code +remain in the working tree which can now be added & committed. When all sources +are committed, you can commit the build products simply by: + +.. code:: bash + + make static.build.commit + +Committing the build products should be the last step, just before you send us +your PR. There is also a make target to rewind this last build commit: + +.. code:: bash + + make static.build.drop diff --git a/_sources/dev/reST.rst.txt b/_sources/dev/reST.rst.txt new file mode 100644 index 000000000..e539617b7 --- /dev/null +++ b/_sources/dev/reST.rst.txt @@ -0,0 +1,1438 @@ +.. _reST primer: + +=========== +reST primer +=========== + +.. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +We at SearXNG are using reStructuredText (aka reST_) markup for all kind of +documentation. With the builders from the Sphinx_ project a HTML output is +generated and deployed at docs.searxng.org_. For build prerequisites read +:ref:`docs build`. + +.. _docs.searxng.org: https://docs.searxng.org/ + +The source files of SearXNG's documentation are located at :origin:`docs`. +Sphinx assumes source files to be encoded in UTF-8 by default. Run :ref:`make +docs.live <make docs.live>` to build HTML while editing. + +.. sidebar:: Further reading + + - Sphinx-Primer_ + - `Sphinx markup constructs`_ + - reST_, docutils_, `docutils FAQ`_ + - Sphinx_, `sphinx-doc FAQ`_ + - `sphinx config`_, doctree_ + - `sphinx cross references`_ + - linuxdoc_ + - intersphinx_ + - sphinx-jinja_ + - `Sphinx's autodoc`_ + - `Sphinx's Python domain`_, `Sphinx's C domain`_ + - SVG_, ImageMagick_ + - DOT_, `Graphviz's dot`_, Graphviz_ + + +.. contents:: + :depth: 3 + :local: + :backlinks: entry + +Sphinx_ and reST_ have their place in the python ecosystem. Over that reST is +used in popular projects, e.g the Linux kernel documentation `[kernel doc]`_. + +.. _[kernel doc]: https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html + +.. sidebar:: Content matters + + The readability_ of the reST sources has its value, therefore we recommend to + make sparse usage of reST markup / .. content matters! + +**reST** is a plaintext markup language, its markup is *mostly* intuitive and +you will not need to learn much to produce well formed articles with. I use the +word *mostly*: like everything in live, reST has its advantages and +disadvantages, some markups feel a bit grumpy (especially if you are used to +other plaintext markups). + +Soft skills +=========== + +Before going any deeper into the markup let's face on some **soft skills** a +trained author brings with, to reach a well feedback from readers: + +- Documentation is dedicated to an audience and answers questions from the + audience point of view. +- Don't detail things which are general knowledge from the audience point of + view. +- Limit the subject, use cross links for any further reading. + +To be more concrete what a *point of view* means. In the (:origin:`docs`) +folder we have three sections (and the *blog* folder), each dedicate to a +different group of audience. + +User's POV: :origin:`docs/user` + A typical user knows about search engines and might have heard about + meta crawlers and privacy. + +Admin's POV: :origin:`docs/admin` + A typical Admin knows about setting up services on a linux system, but he does + not know all the pros and cons of a SearXNG setup. + +Developer's POV: :origin:`docs/dev` + Depending on the readability_ of code, a typical developer is able to read and + understand source code. Describe what a item aims to do (e.g. a function). + If the chronological order matters, describe it. Name the *out-of-limits + conditions* and all the side effects a external developer will not know. + +.. _reST inline markup: + +Basic inline markup +=================== + +.. sidebar:: Inline markup + + - :ref:`reST roles` + - :ref:`reST smart ref` + +Basic inline markup is done with asterisks and backquotes. If asterisks or +backquotes appear in running text and could be confused with inline markup +delimiters, they have to be escaped with a backslash (``\*pointer``). + +.. table:: basic inline markup + :widths: 4 3 7 + + ================================================ ==================== ======================== + description rendered markup + ================================================ ==================== ======================== + one asterisk for emphasis *italics* ``*italics*`` + two asterisks for strong emphasis **boldface** ``**boldface**`` + backquotes for code samples and literals ``foo()`` ````foo()```` + quote asterisks or backquotes \*foo is a pointer ``\*foo is a pointer`` + ================================================ ==================== ======================== + +.. _reST basic structure: + +Basic article structure +======================= + +The basic structure of an article makes use of heading adornments to markup +chapter, sections and subsections. + +.. _reST template: + +reST template +------------- + +reST template for an simple article: + +.. code:: reST + + .. _doc refname: + + ============== + Document title + ============== + + Lorem ipsum dolor sit amet, consectetur adipisici elit .. Further read + :ref:`chapter refname`. + + .. _chapter refname: + + Chapter + ======= + + Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut + aliquid ex ea commodi consequat ... + + .. _section refname: + + Section + ------- + + lorem .. + + .. _subsection refname: + + Subsection + ~~~~~~~~~~ + + lorem .. + + +Headings +-------- + +#. title - with overline for document title: + + .. code:: reST + + ============== + Document title + ============== + + +#. chapter - with anchor named ``anchor name``: + + .. code:: reST + + .. _anchor name: + + Chapter + ======= + +#. section + + .. code:: reST + + Section + ------- + +#. subsection + + .. code:: reST + + Subsection + ~~~~~~~~~~ + + + +Anchors & Links +=============== + +.. _reST anchor: + +Anchors +------- + +.. _ref role: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref + +To refer a point in the documentation a anchor is needed. The :ref:`reST +template <reST template>` shows an example where a chapter titled *"Chapters"* +gets an anchor named ``chapter title``. Another example from *this* document, +where the anchor named ``reST anchor``: + +.. code:: reST + + .. _reST anchor: + + Anchors + ------- + + To refer a point in the documentation a anchor is needed ... + +To refer anchors use the `ref role`_ markup: + +.. code:: reST + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar <reST anchor>`. + +.. admonition:: ``:ref:`` role + :class: rst-example + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manually :ref:`foo + bar <reST anchor>`. + +.. _reST ordinary ref: + +Link ordinary URL +----------------- + +If you need to reference external URLs use *named* hyperlinks to maintain +readability of reST sources. Here is a example taken from *this* article: + +.. code:: reST + + .. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + <https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>`__. + +.. admonition:: Named hyperlink + :class: rst-example + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + <https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>`__. + + +.. _reST smart ref: + +Smart refs +---------- + +With the power of sphinx.ext.extlinks_ and intersphinx_ referencing external +content becomes smart. + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + refer ... rendered example markup + ========================== ================================== ==================================== + :rst:role:`rfc` :rfc:`822` ``:rfc:`822``` + :rst:role:`pep` :pep:`8` ``:pep:`8``` + sphinx.ext.extlinks_ + -------------------------------------------------------------------------------------------------- + project's wiki article :wiki:`Offline-engines` ``:wiki:`Offline-engines``` + to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html``` + files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` + pull request :pull:`4` ``:pull:`4``` + patch :patch:`af2cae6` ``:patch:`af2cae6``` + PyPi package :pypi:`httpx` ``:pypi:`httpx``` + manual page man :man:`bash` ``:man:`bash``` + intersphinx_ + -------------------------------------------------------------------------------------------------- + external anchor :ref:`python:and` ``:ref:`python:and``` + external doc anchor :doc:`jinja:templates` ``:doc:`jinja:templates``` + python code object :py:obj:`datetime.datetime` ``:py:obj:`datetime.datetime``` + flask code object :py:obj:`flask.Flask` ``:py:obj:`flask.Flask``` + ========================== ================================== ==================================== + + +Intersphinx is configured in :origin:`docs/conf.py`: + +.. code:: python + + intersphinx_mapping = { + "python": ("https://docs.python.org/3/", None), + "flask": ("https://flask.palletsprojects.com/", None), + "jinja": ("https://jinja.palletsprojects.com/", None), + "linuxdoc" : ("https://return42.github.io/linuxdoc/", None), + "sphinx" : ("https://www.sphinx-doc.org/en/master/", None), + } + + +To list all anchors of the inventory (e.g. ``python``) use: + +.. code:: sh + + $ python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv + ... + $ python -m sphinx.ext.intersphinx https://docs.searxng.org/objects.inv + ... + +Literal blocks +============== + +The simplest form of :duref:`literal-blocks` is a indented block introduced by +two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST +code` directive. To include literals from external files use +:rst:dir:`literalinclude` or :ref:`kernel-include <kernel-include-directive>` +directive (latter one expands environment variables in the path name). + +.. _reST literal: + +``::`` +------ + +.. code:: reST + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + +.. admonition:: Literal block + :class: rst-example + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + + +.. _reST code: + +``code-block`` +-------------- + +.. _pygments: https://pygments.org/languages/ + +.. sidebar:: Syntax highlighting + + is handled by pygments_. + +The :rst:dir:`code-block` directive is a variant of the :dudir:`code` directive +with additional options. To learn more about code literals visit +:ref:`sphinx:code-examples`. + +.. code-block:: reST + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +.. code-block:: reST + +.. admonition:: Code block + :class: rst-example + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +Unicode substitution +==================== + +The :dudir:`unicode directive <unicode-character-codes>` converts Unicode +character codes (numerical values) to characters. This directive can only be +used within a substitution definition. + +.. code-block:: reST + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + +.. admonition:: Unicode + :class: rst-example + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + + +.. _reST roles: + +Roles +===== + +.. sidebar:: Further reading + + - `Sphinx Roles`_ + - :doc:`sphinx:usage/restructuredtext/domains` + + +A *custom interpreted text role* (:duref:`ref <roles>`) is an inline piece of +explicit markup. It signifies that that the enclosed text should be interpreted +in a specific way. + +The general markup is one of: + +.. code:: reST + + :rolename:`ref-name` + :rolename:`ref text <ref-name>` + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + role rendered example markup + ========================== ================================== ==================================== + :rst:role:`guilabel` :guilabel:`&Cancel` ``:guilabel:`&Cancel``` + :rst:role:`kbd` :kbd:`C-x C-f` ``:kbd:`C-x C-f``` + :rst:role:`menuselection` :menuselection:`Open --> File` ``:menuselection:`Open --> File``` + :rst:role:`download` :download:`this file <reST.rst>` ``:download:`this file <reST.rst>``` + math_ :math:`a^2 + b^2 = c^2` ``:math:`a^2 + b^2 = c^2``` + :rst:role:`ref` :ref:`svg image example` ``:ref:`svg image example``` + :rst:role:`command` :command:`ls -la` ``:command:`ls -la``` + :durole:`emphasis` :emphasis:`italic` ``:emphasis:`italic``` + :durole:`strong` :strong:`bold` ``:strong:`bold``` + :durole:`literal` :literal:`foo()` ``:literal:`foo()``` + :durole:`subscript` H\ :sub:`2`\ O ``H\ :sub:`2`\ O`` + :durole:`superscript` E = mc\ :sup:`2` ``E = mc\ :sup:`2``` + :durole:`title-reference` :title:`Time` ``:title:`Time``` + ========================== ================================== ==================================== + +Figures & Images +================ + +.. sidebar:: Image processing + + With the directives from :ref:`linuxdoc <linuxdoc:kfigure>` the build process + is flexible. To get best results in the generated output format, install + ImageMagick_ and Graphviz_. + +SearXNG's sphinx setup includes: :ref:`linuxdoc:kfigure`. Scalable here means; +scalable in sense of the build process. Normally in absence of a converter +tool, the build process will break. From the authors POV it’s annoying to care +about the build process when handling with images, especially since he has no +access to the build process. With :ref:`linuxdoc:kfigure` the build process +continues and scales output quality in dependence of installed image processors. + +If you want to add an image, you should use the ``kernel-figure`` (inheritance +of :dudir:`figure`) and ``kernel-image`` (inheritance of :dudir:`image`) +directives. E.g. to insert a figure with a scalable image format use SVG +(:ref:`svg image example`): + +.. code:: reST + + .. _svg image example: + + .. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image + + To refer the figure, a caption block is needed: :ref:`svg image example`. + +.. _svg image example: + +.. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image. + +To refer the figure, a caption block is needed: :ref:`svg image example`. + +DOT files (aka Graphviz) +------------------------ + +With :ref:`linuxdoc:kernel-figure` reST support for **DOT** formatted files is +given. + +- `Graphviz's dot`_ +- DOT_ +- Graphviz_ + +A simple example is shown in :ref:`dot file example`: + +.. code:: reST + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +.. admonition:: hello.dot + :class: rst-example + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +``kernel-render`` DOT +--------------------- + +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the +:ref:`linuxdoc:kernel-render` directive. A simple example of embedded DOT_ is +shown in figure :ref:`dot render example`: + +.. code:: reST + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +Please note :ref:`build tools <linuxdoc:kfigure_build_tools>`. If Graphviz_ is +installed, you will see an vector image. If not, the raw markup is inserted as +*literal-block*. + +.. admonition:: kernel-render DOT + :class: rst-example + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +``kernel-render`` SVG +--------------------- + +A simple example of embedded SVG_ is shown in figure :ref:`svg render example`: + +.. code:: reST + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow +.. + + .. code:: xml + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" + baseProfile="full" width="70px" height="40px" + viewBox="0 0 700 400" + > + <line x1="180" y1="370" + x2="500" y2="50" + stroke="black" stroke-width="15px" + /> + <polygon points="585 0 525 25 585 50" + transform="rotate(135 525 25)" + /> + </svg> + +.. admonition:: kernel-render SVG + :class: rst-example + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" + baseProfile="full" width="70px" height="40px" + viewBox="0 0 700 400" + > + <line x1="180" y1="370" + x2="500" y2="50" + stroke="black" stroke-width="15px" + /> + <polygon points="585 0 525 25 585 50" + transform="rotate(135 525 25)" + /> + </svg> + + + + +.. _reST lists: + +List markups +============ + +Bullet list +----------- + +List markup (:duref:`ref <bullet-lists>`) is simple: + +.. code:: reST + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + +.. admonition:: bullet list + :class: rst-example + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + + +Horizontal list +--------------- + +The :rst:dir:`.. hlist:: <hlist>` transforms a bullet list into a more compact +list. + +.. code:: reST + + .. hlist:: + + - first list item + - second list item + - third list item + ... + +.. admonition:: hlist + :class: rst-example + + .. hlist:: + + - first list item + - second list item + - third list item + - next list item + - next list item xxxx + - next list item yyyy + - next list item zzzz + + +Definition list +--------------- + +.. sidebar:: Note .. + + - the term cannot have more than one line of text + + - there is **no blank line between term and definition block** // this + distinguishes definition lists (:duref:`ref <definition-lists>`) from block + quotes (:duref:`ref <block-quotes>`). + +Each definition list (:duref:`ref <definition-lists>`) item contains a term, +optional classifiers and a definition. A term is a simple one-line word or +phrase. Optional classifiers may follow the term on the same line, each after +an inline ' : ' (**space, colon, space**). A definition is a block indented +relative to the term, and may contain multiple paragraphs and other body +elements. There may be no blank line between a term line and a definition block +(*this distinguishes definition lists from block quotes*). Blank lines are +required before the first and after the last definition list item, but are +optional in-between. + +Definition lists are created as follows: + +.. code:: reST + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref <block-quotes>`) and not as term's definition! + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + Definition 4. + +.. admonition:: definition list + :class: rst-example + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref <block-quotes>`) and not as term's definition! + + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + + +Quoted paragraphs +----------------- + +Quoted paragraphs (:duref:`ref <block-quotes>`) are created by just indenting +them more than the surrounding paragraphs. Line blocks (:duref:`ref +<line-blocks>`) are a way of preserving line breaks: + +.. code:: reST + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. admonition:: Quoted paragraph and line block + :class: rst-example + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. _reST field list: + +Field Lists +----------- + +.. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + +.. sidebar:: bibliographic fields + + First lines fields are bibliographic fields, see `Sphinx Field Lists`_. + +Field lists are used as part of an extension syntax, such as options for +directives, or database-like records meant for further processing. Field lists +are mappings from field names to field bodies. They marked up like this: + +.. code:: reST + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + +.. admonition:: Field List + :class: rst-example + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + + +They are commonly used in Python documentation: + +.. code:: python + + def my_function(my_arg, my_other_arg): + """A function just for me. + + :param my_arg: The first of my arguments. + :param my_other_arg: The second of my arguments. + + :returns: A message (just for me, of course). + """ + +Further list blocks +------------------- + +- field lists (:duref:`ref <field-lists>`, with caveats noted in + :ref:`reST field list`) +- option lists (:duref:`ref <option-lists>`) +- quoted literal blocks (:duref:`ref <quoted-literal-blocks>`) +- doctest blocks (:duref:`ref <doctest-blocks>`) + + +Admonitions +=========== + +Sidebar +------- + +Sidebar is an eye catcher, often used for admonitions pointing further stuff or +site effects. Here is the source of the sidebar :ref:`on top of this page <reST +primer>`. + +.. code:: reST + + .. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at SearXNG encourage our + contributors to follow principles like KISS_ and readability_. + +Generic admonition +------------------ + +The generic :dudir:`admonition <admonitions>` needs a title: + +.. code:: reST + + .. admonition:: generic admonition title + + lorem ipsum .. + + +.. admonition:: generic admonition title + + lorem ipsum .. + + +Specific admonitions +-------------------- + +Specific admonitions: :dudir:`hint`, :dudir:`note`, :dudir:`tip` :dudir:`attention`, +:dudir:`caution`, :dudir:`danger`, :dudir:`error`, , :dudir:`important`, and +:dudir:`warning` . + +.. code:: reST + + .. hint:: + + lorem ipsum .. + + .. note:: + + lorem ipsum .. + + .. warning:: + + lorem ipsum .. + + +.. hint:: + + lorem ipsum .. + +.. note:: + + lorem ipsum .. + +.. tip:: + + lorem ipsum .. + +.. attention:: + + lorem ipsum .. + +.. caution:: + + lorem ipsum .. + +.. danger:: + + lorem ipsum .. + +.. important:: + + lorem ipsum .. + +.. error:: + + lorem ipsum .. + +.. warning:: + + lorem ipsum .. + + +Tables +====== + +.. sidebar:: Nested tables + + Nested tables are ugly! Not all builder support nested tables, don't use + them! + +ASCII-art tables like :ref:`reST simple table` and :ref:`reST grid table` might +be comfortable for readers of the text-files, but they have huge disadvantages +in the creation and modifying. First, they are hard to edit. Think about +adding a row or a column to a ASCII-art table or adding a paragraph in a cell, +it is a nightmare on big tables. + + +.. sidebar:: List tables + + For meaningful patch and diff use :ref:`reST flat table`. + +Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a +cell generates a diff in which also changes are included, which are only +ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, +here are some helpers: + +* `Emacs Table Mode`_ +* `Online Tables Generator`_ + +.. _reST simple table: + +Simple tables +------------- + +:duref:`Simple tables <simple-tables>` allow *colspan* but not *rowspan*. If +your table need some metadata (e.g. a title) you need to add the ``.. table:: +directive`` :dudir:`(ref) <table>` in front and place the table in its body: + +.. code:: reST + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + +.. admonition:: Simple ASCII table + :class: rst-example + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + + + +.. _reST grid table: + +Grid tables +----------- + +:duref:`Grid tables <grid-tables>` allow colspan *colspan* and *rowspan*: + +.. code:: reST + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + +.. admonition:: ASCII grid table + :class: rst-example + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + + +.. _reST flat table: + +flat-table +---------- + +The ``flat-table`` is a further developed variant of the :ref:`list tables +<linuxdoc:list-table-directives>`. It is a double-stage list similar to the +:dudir:`list-table` with some additional features: + +column-span: ``cspan`` + with the role ``cspan`` a cell can be extended through additional columns + +row-span: ``rspan`` + with the role ``rspan`` a cell can be extended through additional rows + +auto-span: + spans rightmost cell of a table row over the missing cells on the right side + of that table-row. With Option ``:fill-cells:`` this behavior can changed + from *auto span* to *auto fill*, which automatically inserts (empty) cells + instead of spanning the last cell. + +options: + :header-rows: [int] count of header rows + :stub-columns: [int] count of stub columns + :widths: [[int] [int] ... ] widths of columns + :fill-cells: instead of auto-span missing cells, insert missing cells + +roles: + :cspan: [int] additional columns (*morecols*) + :rspan: [int] additional rows (*morerows*) + +The example below shows how to use this markup. The first level of the staged +list is the *table-row*. In the *table-row* there is only one markup allowed, +the list of the cells in this *table-row*. Exception are *comments* ( ``..`` ) +and *targets* (e.g. a ref to :ref:`row 2 of table's body <row body 2>`). + +.. code:: reST + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +.. admonition:: List table + :class: rst-example + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to right end + + * - row 6 + - cell 6.1 + - .. + + +CSV table +--------- + +CSV table might be the choice if you want to include CSV-data from a outstanding +(build) process into your documentation. + +.. code:: reST + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 2 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Content of file ``csv_table.txt``: + +.. literalinclude:: csv_table.txt + +.. admonition:: CSV table + :class: rst-example + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 3 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Templating +========== + +.. sidebar:: Build environment + + All *generic-doc* tasks are running in the :ref:`make install`. + +Templating is suitable for documentation which is created generic at the build +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with SearXNG modules installed). We use this e.g. to build chapter: +:ref:`configured engines`. Below the jinja directive from the +:origin:`docs/admin/engines.rst` is shown: + +.. literalinclude:: ../user/configured_engines.rst + :language: reST + :start-after: .. _configured engines: + +The context for the template is selected in the line ``.. jinja:: searx``. In +sphinx's build configuration (:origin:`docs/conf.py`) the ``searx`` context +contains the ``engines`` and ``plugins``. + +.. code:: py + + import searx.search + import searx.engines + import searx.plugins + searx.search.initialize() + jinja_contexts = { + 'searx': { + 'engines': searx.engines.engines, + 'plugins': searx.plugins.plugins + }, + } + + +Tabbed views +============ + +.. _sphinx-tabs: https://github.com/djungelorm/sphinx-tabs +.. _basic-tabs: https://github.com/djungelorm/sphinx-tabs#basic-tabs +.. _group-tabs: https://github.com/djungelorm/sphinx-tabs#group-tabs +.. _code-tabs: https://github.com/djungelorm/sphinx-tabs#code-tabs + +With `sphinx-tabs`_ extension we have *tabbed views*. To provide installation +instructions with one tab per distribution we use the `group-tabs`_ directive, +others are basic-tabs_ and code-tabs_. Below a *group-tab* example from +:ref:`docs build` is shown: + +.. literalinclude:: ../admin/buildhosts.rst + :language: reST + :start-after: .. SNIP sh lint requirements + :end-before: .. SNAP sh lint requirements + +.. _math: + +Math equations +============== + +.. _Mathematics: https://en.wikibooks.org/wiki/LaTeX/Mathematics +.. _amsmath user guide: + http://vesta.informatik.rwth-aachen.de/ftp/pub/mirror/ctan/macros/latex/required/amsmath/amsldoc.pdf + +.. sidebar:: About LaTeX + + - `amsmath user guide`_ + - Mathematics_ + - :ref:`docs build` + +The input language for mathematics is LaTeX markup using the :ctan:`amsmath` +package. + +To embed LaTeX markup in reST documents, use role :rst:role:`:math: <math>` for +inline and directive :rst:dir:`.. math:: <math>` for block markup. + +.. code:: reST + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + +.. admonition:: LaTeX math equation + :class: rst-example + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + + +The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` +(*displaystyle*) used in a inline markup or another fraction. + +.. code:: reST + + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + ``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + +.. admonition:: Line spacing + :class: rst-example + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + +.. _KISS: https://en.wikipedia.org/wiki/KISS_principle + +.. _readability: https://docs.python-guide.org/writing/style/ +.. _Sphinx-Primer: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +.. _reST: https://docutils.sourceforge.io/rst.html +.. _Sphinx Roles: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html +.. _Sphinx: https://www.sphinx-doc.org +.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html +.. _Sphinx markup constructs: + https://www.sphinx-doc.org/en/stable/markup/index.html +.. _`sphinx cross references`: + https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations +.. _sphinx.ext.extlinks: + https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html +.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html +.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html +.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html +.. _Sphinx's Python domain: + https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain +.. _Sphinx's C domain: + https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs +.. _doctree: + https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases +.. _docutils: http://docutils.sourceforge.net/docs/index.html +.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html +.. _linuxdoc: https://return42.github.io/linuxdoc +.. _jinja: https://jinja.palletsprojects.com/ +.. _sphinx-jinja: https://github.com/tardyp/sphinx-jinja +.. _SVG: https://www.w3.org/TR/SVG11/expanded-toc.html +.. _DOT: https://graphviz.gitlab.io/_pages/doc/info/lang.html +.. _`Graphviz's dot`: https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org + +.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode +.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables +.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html diff --git a/_sources/dev/rtm_asdf.rst.txt b/_sources/dev/rtm_asdf.rst.txt new file mode 100644 index 000000000..69180ef4e --- /dev/null +++ b/_sources/dev/rtm_asdf.rst.txt @@ -0,0 +1,121 @@ +================== +Runtime Management +================== + +The runtimes are managed with asdf and are activated in this project via the +`.tool-versions <.tool-versions>`_. If you have not yet installed asdf_, then +chapter :ref:`introduce asdf` may be of help to you. + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Get started +=========== + +If you have asdf installed you can install the runtimes of this project by: + +.. code:: bash + + $ cd /path/to/searxng + $ asdf install # will install runtimes listed in .tool-versions + ... + +Manage Versions +=============== + +If you want to perform a ``test`` with special runtime versions of nodejs, +python or shellcheck, you can patch the ``.tool-versions``: + +.. code:: diff + + --- a/.tool-versions + +++ b/.tool-versions + @@ -1,2 +1,2 @@ + -python 3.12.0 + -shellcheck 0.9.0 + +python 3.11.6 + +shellcheck 0.8.0 + +To install use ``asdf install`` again. If the runtime tools have changed, any +existing (nodejs and python) environments should be cleaned up with a ``make +clean``. + +.. code:: bash + + $ asdf install + ... + $ make clean test + + +.. _introduce asdf: + +Introduce asdf +============== + +To `download asdf`_ and `install asdf`_: + +.. code:: bash + + $ git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch <version> + $ echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc + $ echo '. "$HOME/.asdf/completions/asdf.bash"' >> ~/.bashrc + +Start a new shell and try to `install plugins`_: + +.. code:: bash + + $ asdf plugin-list-all | grep -E '(golang|python|nodejs|shellcheck).git' + golang https://github.com/asdf-community/asdf-golang.git + nodejs https://github.com/asdf-vm/asdf-nodejs.git + python https://github.com/danhper/asdf-python.git + shellcheck https://github.com/luizm/asdf-shellcheck.git + + $ asdf plugin add golang https://github.com/asdf-community/asdf-golang.git + $ asdf plugin add nodejs https://github.com/asdf-vm/asdf-nodejs.git + $ asdf plugin add python https://github.com/danhper/asdf-python.git + $ asdf plugin add shellcheck https://github.com/luizm/asdf-shellcheck.git + +Each plugin has dependencies, to compile runtimes visit the URLs from above and +look out for the dependencies you need to install on your OS, on Debian for the +runtimes listed above you will need: + +.. code:: bash + + $ sudo apt update + $ sudo apt install \ + dirmngr gpg curl gawk coreutils build-essential libssl-dev zlib1g-dev \ + libbz2-dev libreadline-dev libsqlite3-dev \ + libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev + +With dependencies installed you can install/compile runtimes: + +.. code:: bash + + $ asdf install golang latest + $ asdf install nodejs latest + $ asdf install python latest + $ asdf install shellcheck latest + +Python will be compiled and will take a while. + +In the repository the version is defined in `.tool-versions`_. Outside the +repository, its recommended that the runtime should use the versions of the OS +(`Fallback to System Version`_) / if not already done register the system +versions global: + +.. code:: bash + + $ cd / + $ asdf global golang system + $ asdf global nodejs system + $ asdf global python system + $ asdf global shellcheck system + +.. _asdf: https://asdf-vm.com/ +.. _download asdf: https://asdf-vm.com/guide/getting-started.html#_2-download-asdf +.. _install asdf: https://asdf-vm.com/guide/getting-started.html#_3-install-asdf +.. _install plugins: https://asdf-vm.com/guide/getting-started.html#install-the-plugin +.. _Fallback to System Version: https://asdf-vm.com/manage/versions.html#fallback-to-system-version diff --git a/_sources/dev/search_api.rst.txt b/_sources/dev/search_api.rst.txt new file mode 100644 index 000000000..aa5f847ea --- /dev/null +++ b/_sources/dev/search_api.rst.txt @@ -0,0 +1,124 @@ +.. _search API: + +========== +Search API +========== + +The search supports both ``GET`` and ``POST``. + +Furthermore, two endpoints ``/`` and ``/search`` are available for querying. + + +``GET /`` + +``GET /search`` + +Parameters +========== + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings.yml` + - :ref:`configured engines` + +``q`` : required + The search query. This string is passed to external search services. Thus, + SearXNG supports syntax of each search service. For example, ``site:github.com + SearXNG`` is a valid query for Google. However, if simply the query above is + passed to any search engine which does not filter its results based on this + syntax, you might not get the results you wanted. + + See more at :ref:`search-syntax` + +``categories`` : optional + Comma separated list, specifies the active search categories (see + :ref:`configured engines`) + +``engines`` : optional + Comma separated list, specifies the active search engines (see + :ref:`configured engines`). + +``language`` : default from :ref:`settings search` + Code of the language. + +``pageno`` : default ``1`` + Search page number. + +``time_range`` : optional + [ ``day``, ``month``, ``year`` ] + + Time range of search for engines which support it. See if an engine supports + time range search in the preferences page of an instance. + +``format`` : optional + [ ``json``, ``csv``, ``rss`` ] + + Output format of results. Format needs to be activated in :ref:`settings + search`. + +``results_on_new_tab`` : default ``0`` + [ ``0``, ``1`` ] + + Open search results on new tab. + +``image_proxy`` : default from :ref:`settings server` + [ ``True``, ``False`` ] + + Proxy image results through SearXNG. + +``autocomplete`` : default from :ref:`settings search` + [ ``google``, ``dbpedia``, ``duckduckgo``, ``mwmbl``, ``startpage``, + ``wikipedia``, ``stract``, ``swisscows``, ``qwant`` ] + + Service which completes words as you type. + +``safesearch`` : default from :ref:`settings search` + [ ``0``, ``1``, ``2`` ] + + Filter search results of engines which support safe search. See if an engine + supports safe search in the preferences page of an instance. + +``theme`` : default ``simple`` + [ ``simple`` ] + + Theme of instance. + + Please note, available themes depend on an instance. It is possible that an + instance administrator deleted, created or renamed themes on their instance. + See the available options in the preferences page of the instance. + +``enabled_plugins`` : optional + List of enabled plugins. + + :default: + ``Hash_plugin``, ``Self_Information``, + ``Tracker_URL_remover``, ``Ahmia_blacklist`` + + :values: + .. enabled by default + + ``Hash_plugin``, ``Self_Information``, + ``Tracker_URL_remover``, ``Ahmia_blacklist``, + + .. disabled by default + + ``Hostnames_plugin``, ``Open_Access_DOI_rewrite``, + ``Vim-like_hotkeys``, ``Tor_check_plugin`` + +``disabled_plugins``: optional + List of disabled plugins. + + :default: + ``Hostnames_plugin``, ``Open_Access_DOI_rewrite``, + ``Vim-like_hotkeys``, ``Tor_check_plugin`` + + :values: + see values from ``enabled_plugins`` + +``enabled_engines`` : optional : *all* :origin:`engines <searx/engines>` + List of enabled engines. + +``disabled_engines`` : optional : *all* :origin:`engines <searx/engines>` + List of disabled engines. + diff --git a/_sources/dev/searxng_extra/index.rst.txt b/_sources/dev/searxng_extra/index.rst.txt new file mode 100644 index 000000000..7bca3c0d4 --- /dev/null +++ b/_sources/dev/searxng_extra/index.rst.txt @@ -0,0 +1,14 @@ +.. _searxng_extra: + +============================= +Tooling box ``searxng_extra`` +============================= + +In the folder :origin:`searxng_extra/` we maintain some tools useful for CI and +developers. + +.. toctree:: + :maxdepth: 2 + + update + standalone_searx.py diff --git a/_sources/dev/searxng_extra/standalone_searx.py.rst.txt b/_sources/dev/searxng_extra/standalone_searx.py.rst.txt new file mode 100644 index 000000000..7cbbccfde --- /dev/null +++ b/_sources/dev/searxng_extra/standalone_searx.py.rst.txt @@ -0,0 +1,9 @@ + +.. _standalone_searx.py: + +===================================== +``searxng_extra/standalone_searx.py`` +===================================== + +.. automodule:: searxng_extra.standalone_searx + :members: diff --git a/_sources/dev/searxng_extra/update.rst.txt b/_sources/dev/searxng_extra/update.rst.txt new file mode 100644 index 000000000..dc3b06744 --- /dev/null +++ b/_sources/dev/searxng_extra/update.rst.txt @@ -0,0 +1,98 @@ +========================= +``searxng_extra/update/`` +========================= + +:origin:`[source] <searxng_extra/update/__init__.py>` + +Scripts to update static data in :origin:`searx/data/` + +.. _update_ahmia_blacklist.py: + +``update_ahmia_blacklist.py`` +============================= + +:origin:`[source] <searxng_extra/update/update_ahmia_blacklist.py>` + +.. automodule:: searxng_extra.update.update_ahmia_blacklist + :members: + + +``update_currencies.py`` +======================== + +:origin:`[source] <searxng_extra/update/update_currencies.py>` + +.. automodule:: searxng_extra.update.update_currencies + :members: + +``update_engine_descriptions.py`` +================================= + +:origin:`[source] <searxng_extra/update/update_engine_descriptions.py>` + +.. automodule:: searxng_extra.update.update_engine_descriptions + :members: + + +``update_external_bangs.py`` +============================ + +:origin:`[source] <searxng_extra/update/update_external_bangs.py>` + +.. automodule:: searxng_extra.update.update_external_bangs + :members: + + +``update_firefox_version.py`` +============================= + +:origin:`[source] <searxng_extra/update/update_firefox_version.py>` + +.. automodule:: searxng_extra.update.update_firefox_version + :members: + + +``update_engine_traits.py`` +=========================== + +:origin:`[source] <searxng_extra/update/update_engine_traits.py>` + +.. automodule:: searxng_extra.update.update_engine_traits + :members: + + +``update_osm_keys_tags.py`` +=========================== + +:origin:`[source] <searxng_extra/update/update_osm_keys_tags.py>` + +.. automodule:: searxng_extra.update.update_osm_keys_tags + :members: + + +``update_pygments.py`` +====================== + +:origin:`[source] <searxng_extra/update/update_pygments.py>` + +.. automodule:: searxng_extra.update.update_pygments + :members: + +.. _update_locales.py: + +``update_locales.py`` +===================== + +:origin:`[source] <searxng_extra/update/update_locales.py>` + +.. automodule:: searxng_extra.update.update_locales + :members: + + +``update_wikidata_units.py`` +============================ + +:origin:`[source] <searxng_extra/update/update_wikidata_units.py>` + +.. automodule:: searxng_extra.update.update_wikidata_units + :members: diff --git a/_sources/dev/translation.rst.txt b/_sources/dev/translation.rst.txt new file mode 100644 index 000000000..57c76a0c1 --- /dev/null +++ b/_sources/dev/translation.rst.txt @@ -0,0 +1,81 @@ +.. _translation: + +=========== +Translation +=========== + +.. _translate.codeberg.org: https://translate.codeberg.org/projects/searxng/ +.. _Weblate: https://docs.weblate.org +.. _translations branch: https://github.com/searxng/searxng/tree/translations +.. _orphan branch: https://git-scm.com/docs/git-checkout#Documentation/git-checkout.txt---orphanltnewbranchgt +.. _Weblate repository: https://translate.codeberg.org/projects/searxng/searxng/#repository +.. _wlc: https://docs.weblate.org/en/latest/wlc.html + +.. |translated| image:: https://translate.codeberg.org/widgets/searxng/-/searxng/svg-badge.svg + :target: https://translate.codeberg.org/projects/searxng/ + +.. sidebar:: |translated| + + - :ref:`searx.babel_extract` + - Weblate_ + - SearXNG `translations branch`_ + - SearXNG `Weblate repository`_ + - Weblate Client: wlc_ + - Babel Command-Line: `pybabel <http://babel.pocoo.org/en/latest/cmdline.html>`_ + - `weblate workflow <https://docs.weblate.org/en/latest/workflows.html>`_ + +Translation takes place on translate.codeberg.org_. + +Translations which has been added by translators on the translate.codeberg.org_ UI are +committed to Weblate's counterpart of the SearXNG *origin* repository which is +located at ``https://translate.codeberg.org/git/searxng/searxng``. + +There is no need to clone this repository, :ref:`SearXNG Weblate workflow` take +care of the synchronization with the *origin*. To avoid merging commits from +the counterpart directly on the ``master`` branch of *SearXNG origin*, a *pull +request* (PR) is created by this workflow. + +Weblate monitors the `translations branch`_, not the ``master`` branch. This +branch is an `orphan branch`_, decoupled from the master branch (we already know +orphan branches from the ``gh-pages``). The `translations branch`_ contains +only the + +- ``translation/messages.pot`` and the +- ``translation/*/messages.po`` files, nothing else. + + +.. _SearXNG Weblate workflow: + +.. figure:: translation.svg + + SearXNG's PR workflow to be in sync with Weblate + +Sync from *origin* to *weblate*: using ``make weblate.push.translations`` + For each commit on the ``master`` branch of SearXNG *origin* the GitHub job + :origin:`babel / Update translations branch + <.github/workflows/integration.yml>` checks for updated translations. + +Sync from *weblate* to *origin*: using ``make weblate.translations.commit`` + Every Friday, the GitHub workflow :origin:`babel / create PR for additions from + weblate <.github/workflows/translations-update.yml>` creates a PR with the + updated translation files: + + - ``translation/messages.pot``, + - ``translation/*/messages.po`` and + - ``translation/*/messages.mo`` + +wlc +=== + +.. _wlc configuration: https://docs.weblate.org/en/latest/wlc.html#wlc-config +.. _API key: https://translate.codeberg.org/accounts/profile/#api + +All weblate integration is done by GitHub workflows, but if you want to use wlc_, +copy this content into `wlc configuration`_ in your HOME ``~/.config/weblate`` + +.. code-block:: ini + + [keys] + https://translate.codeberg.org/api/ = APIKEY + +Replace ``APIKEY`` by your `API key`_. |