diff options
Diffstat (limited to 'docs/dev')
| -rw-r--r-- | docs/dev/engine_overview.rst | 8 | ||||
| -rw-r--r-- | docs/dev/lxcdev.rst | 185 | ||||
| -rw-r--r-- | docs/dev/makefile.rst | 26 | ||||
| -rw-r--r-- | docs/dev/offline_engines.rst | 6 | ||||
| -rw-r--r-- | docs/dev/plugins.rst | 27 | ||||
| -rw-r--r-- | docs/dev/quickstart.rst | 14 |
6 files changed, 117 insertions, 149 deletions
diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 439875491..e950ae667 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -66,11 +66,11 @@ For a more detailed description, see :ref:`settings engine` in the :ref:`settin .. table:: Common options in the engine setup (``settings.yml``) :width: 100% - ======================= =========== =============================================== + ======================= =========== ================================================== argument type information - ======================= =========== =============================================== + ======================= =========== ================================================== name string name of search-engine - engine string name of searx-engine (filename without ``.py``) + engine string name of searxng-engine (file name without ``.py``) enable_http bool enable HTTP (by default only HTTPS is enabled). shortcut string shortcut of search-engine timeout string specific timeout for search-engine @@ -78,7 +78,7 @@ For a more detailed description, see :ref:`settings engine` in the :ref:`settin proxies dict set proxies for a specific engine (e.g. ``proxies : {http: socks5://proxy:port, https: socks5://proxy:port}``) - ======================= =========== =============================================== + ======================= =========== ================================================== .. _engine overrides: diff --git a/docs/dev/lxcdev.rst b/docs/dev/lxcdev.rst index 85b262212..6195a575d 100644 --- a/docs/dev/lxcdev.rst +++ b/docs/dev/lxcdev.rst @@ -45,9 +45,7 @@ be set on a *production* system. The scripts from :ref:`searx_utils` can divide in those to install and maintain software: -- :ref:`searx.sh` -- :ref:`filtron.sh` -- :ref:`morty.sh` +- :ref:`searxng.sh` and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or even development tasks over a stack of isolated containers / what we call the: @@ -73,7 +71,7 @@ once: .. group-tab:: desktop - .. code:: sh + .. code:: bash $ snap install lxd $ lxd init --auto @@ -85,7 +83,7 @@ fork: .. group-tab:: desktop - .. code:: sh + .. code:: bash $ cd ~/Downloads $ git clone https://github.com/searxng/searxng.git searxng @@ -94,19 +92,19 @@ fork: The :ref:`lxc-searxng.env` consists of several images, see ``export LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. For this blog post we exercise on a archlinux_ image. The container of this image is named -``searx-archlinux``. Lets build the container, but be sure that this container +``searxng-archlinux``. Lets build the container, but be sure that this container does not already exists, so first lets remove possible old one: .. tabs:: .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh remove searx-archlinux - $ sudo -H ./utils/lxc.sh build searx-archlinux + $ sudo -H ./utils/lxc.sh remove searxng-archlinux + $ sudo -H ./utils/lxc.sh build searxng-archlinux -.. sidebar:: The ``searx-archlinux`` container +.. sidebar:: The ``searxng-archlinux`` container is the base of all our exercises here. @@ -117,9 +115,9 @@ In this container we install all services :ref:`including searx, morty & filtron .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh install suite searx-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux To proxy HTTP from filtron and morty in the container to the outside of the container, install nginx into the container. Once for the bot blocker filtron: @@ -128,9 +126,9 @@ container, install nginx into the container. Once for the bot blocker filtron: .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ ./utils/filtron.sh nginx install ... INFO: got 429 from http://10.174.184.156/searx @@ -141,9 +139,9 @@ and once for the content sanitizer (content proxy morty): .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ ./utils/morty.sh nginx install ... INFO: got 200 from http://10.174.184.156/morty/ @@ -154,7 +152,7 @@ and once for the content sanitizer (content proxy morty): blocker (filtron) and WEB content sanitizer (content proxy morty), both are needed for a *privacy protecting* search engine. -On your system, the IP of your ``searx-archlinux`` container differs from +On your system, the IP of your ``searxng-archlinux`` container differs from http://10.174.184.156/searx, just open the URL reported in your installation protocol in your WEB browser from the desktop to test the instance from outside of the container. @@ -169,27 +167,27 @@ In containers, work as usual Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers open the root-bash in the container using ``./utils/lxc.sh cmd -searx-archlinux``: +searxng-archlinux``: .. tabs:: .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash - INFO: [searx-archlinux] bash - [root@searx-archlinux searx]# pwd + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux searx]# pwd /share/searxng -The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in -the searx-container. To debug the running SearXNG instance use: +The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user in +the searxng-container. To debug the running SearXNG instance use: .. tabs:: - .. group-tab:: root@searx-archlinux + .. group-tab:: root@searxng-archlinux - .. code:: sh + .. code:: bash $ ./utils/searx.sh inspect service ... @@ -202,56 +200,42 @@ above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. To debug services from filtron and morty analogous use: -.. tabs:: - - .. group-tab:: root@searx-archlinux - - .. code:: sh - - $ ./utils/filtron.sh inspect service - $ ./utils/morty.sh inspect service - -Another point we have to notice is that each service (:ref:`SearXNG <searx.sh>`, -:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated -system user account with the same name (compare :ref:`create searxng user`). To -get a shell from theses accounts, simply call one of the scripts: +Another point we have to notice is that the service (:ref:`SearXNG <searxng.sh>` +runs under dedicated system user account with the same name (compare +:ref:`create searxng user`). To get a shell from theses accounts, simply call: .. tabs:: - .. group-tab:: root@searx-archlinux + .. group-tab:: root@searxng-archlinux - .. code:: sh + .. code:: bash - $ ./utils/searx.sh shell - $ ./utils/filtron.sh shell - $ ./utils/morty.sh shell + $ ./utils/searxng.sh instance cmd bash -To get in touch, open a shell from the service user (searx@searx-archlinux): +To get in touch, open a shell from the service user (searxng@searxng-archlinux): .. tabs:: .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/searx.sh shell - // exit with [CTRL-D] - (searx-pyenv) [searx@searx-archlinux ~]$ ... + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash + INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash + [searxng@searxng-archlinux ~]$ -The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system -user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* -``(searx-pyenv)`` environment is activated. +The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system +user ``searx`` in the ``searxng-archlinux`` container and the python *virtualenv* +``(searxng-pyenv)`` environment is activated. .. tabs:: - .. group-tab:: searx@searx-archlinux - - .. code:: sh + .. group-tab:: searxng@searxng-archlinux - (searx-pyenv) [searx@searx-archlinux ~]$ pwd - /usr/local/searx + .. code:: bash + (searxng-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng Wrap production into developer suite @@ -262,23 +246,22 @@ from a LXC container (which is quite ready for production) into a developer suite. For this, we have to keep an eye on the :ref:`installation basic`: - SearXNG setup in: ``/etc/searxng/settings.yml`` -- SearXNG user's home: ``/usr/local/searx`` -- virtualenv in: ``/usr/local/searx/searx-pyenv`` -- SearXNG software in: ``/usr/local/searx/searx-src`` +- SearXNG user's home: ``/usr/local/searxng`` +- virtualenv in: ``/usr/local/searxng/searxng-pyenv`` +- SearXNG software in: ``/usr/local/searxng/searxng-src`` -With the use of the :ref:`searx.sh` the SearXNG service was installed as +With the use of the :ref:`searxng.sh` the SearXNG service was installed as :ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use -``systemctl`` (compare :ref:`service architectures on distributions <uwsgi -configuration>`). +``systemctl`` (compare :ref:`uWSGI maintenance`). .. tabs:: .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - systemctl stop uwsgi@searx + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + systemctl stop uwsgi@searxng With the command above, we stopped the SearXNG uWSGI-App in the archlinux container. @@ -291,29 +274,29 @@ least you should attend the settings of ``uid``, ``chdir``, ``env`` and env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml http = 127.0.0.1:8888 - chdir = /usr/local/searx/searx-src/searx - virtualenv = /usr/local/searx/searx-pyenv - pythonpath = /usr/local/searx/searx-src + chdir = /usr/local/searxng/searxng-src/searx + virtualenv = /usr/local/searxng/searxng-pyenv + pythonpath = /usr/local/searxng/searxng-src If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that each container shares the root folder of the repository and the command ``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the SearXNG installation into a developer one, we simple have to create a smylink to the **transparent** reposetory from the desktop. Now lets replace the -repository at ``searx-src`` in the container with the working tree from outside +repository at ``searxng-src`` in the container with the working tree from outside of the container: .. tabs:: .. group-tab:: container becomes a developer suite - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ln -s /share/searx/ /usr/local/searx/searx-src + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + ln -s /share/searx/ /usr/local/searxng/searxng-src Now we can develop as usual in the working tree of our desktop system. Every time the software was changed, you have to restart the SearXNG service (in the @@ -323,9 +306,9 @@ conatiner): .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ systemctl restart uwsgi@searx @@ -338,30 +321,30 @@ daily usage: To *inspect* the SearXNG instance (already described above): - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ ./utils/searx.sh inspect service Run :ref:`makefile`, e.g. to test inside the container: - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ make test To install all prerequisites needed for a :ref:`buildhosts`: - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ - ./utils/searx.sh install buildhost + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + ./utils/searxng.sh install buildhost To build the docs on a buildhost :ref:`buildhosts`: - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ make docs.html .. _lxcdev summary: @@ -371,18 +354,18 @@ Summary We build up a fully functional SearXNG suite in a archlinux container: -.. code:: sh +.. code:: bash - $ sudo -H ./utils/lxc.sh install suite searx-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux To access HTTP from the desktop we installed nginx for the services inside the conatiner: .. tabs:: - .. group-tab:: [root@searx-archlinux] + .. group-tab:: [root@searxng-archlinux] - .. code:: sh + .. code:: bash $ ./utils/filtron.sh nginx install $ ./utils/morty.sh nginx install @@ -393,12 +376,12 @@ the container : .. tabs:: - .. group-tab:: [root@searx-archlinux] + .. group-tab:: [root@searxng-archlinux] - .. code:: sh + .. code:: bash - $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old - $ ln -s /share/searx/ /usr/local/searx/searx-src + $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + $ ln -s /share/searx/ /usr/local/searxng/searxng-src $ systemctl restart uwsgi@searx To get information about the searxNG suite in the archlinux container we can @@ -408,13 +391,13 @@ use: .. group-tab:: desktop - .. code:: sh + .. code:: bash - $ sudo -H ./utils/lxc.sh show suite searx-archlinux + $ sudo -H ./utils/lxc.sh show suite searxng-archlinux ... - [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx - [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ - [searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ - [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] + [searxng-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx + [searxng-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ + [searxng-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ + [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] ... diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst index ceb76bf10..68c708a85 100644 --- a/docs/dev/makefile.rst +++ b/docs/dev/makefile.rst @@ -40,7 +40,7 @@ We do no longer need to build up the virtualenv manually. Jump into your git working tree and release a ``make install`` to get a virtualenv with a *developer install* of SearXNG (:origin:`setup.py`). :: - $ cd ~/searx-clone + $ cd ~/searxng-clone $ make install PYENV [virtualenv] installing ./requirements*.txt into local/py3 ... @@ -288,27 +288,3 @@ To filter out HTTP redirects (3xx_):: https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None -- - - -``make pybuild`` -================ - -.. _PyPi: https://pypi.org/ -.. _twine: https://twine.readthedocs.io/en/latest/ - -Build Python packages in ``./dist/py``:: - - $ make pybuild - ... - BUILD pybuild - running sdist - running egg_info - ... - running bdist_wheel - - $ ls ./dist - searx-0.18.0-py3-none-any.whl searx-0.18.0.tar.gz - -To upload packages to PyPi_, there is also a ``pypi.upload`` target (to test use -``pypi.upload.test``). Since you are not the owner of :pypi:`searx` you will -never need to upload. diff --git a/docs/dev/offline_engines.rst b/docs/dev/offline_engines.rst index 9320c6262..bfb2664f5 100644 --- a/docs/dev/offline_engines.rst +++ b/docs/dev/offline_engines.rst @@ -55,10 +55,10 @@ admins can install packages in advance. If there is a need to install additional packages in *Python's Virtual Environment* of your SearXNG instance you need to switch into the environment -(:ref:`searx-src`) first, for this you can use :ref:`searx.sh`:: +(:ref:`searxng-src`) first, for this you can use :ref:`searxng.sh`:: - $ sudo utils/searx.sh shell - (searx-pyenv)$ pip install ... + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install ... Private engines (Security) diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst index 44401e34f..36a443230 100644 --- a/docs/dev/plugins.rst +++ b/docs/dev/plugins.rst @@ -33,17 +33,26 @@ Example plugin External plugins ================ -External plugins are standard python modules implementing all the requirements of the standard plugins. -Plugins can be enabled by adding them to :ref:`settings.yml`'s ``plugins`` section. -Example external plugin can be found `here <https://github.com/asciimoo/searx_external_plugin_example>`_. +SearXNG supports *external plugins* / there is no need to install one, SearXNG +runs out of the box. But to demonstrate; in the example below we install the +SearXNG plugins from *The Green Web Foundation* `[ref] +<https://www.thegreenwebfoundation.org/news/searching-the-green-web-with-searx/>`__: -Register your plugin -==================== +.. code:: bash + + $ sudo utils/searxng.sh instance cmd bash + (searxng-pyenv)$ pip install git+https://github.com/return42/tgwf-searx-plugins + +In the :ref:`settings.yml` activate the ``plugins:`` section and add module +``only_show_green_results`` from ``tgwf-searx-plugins``. + +.. code:: yaml + + plugins: + ... + - only_show_green_results + ... -To enable your plugin register your plugin in -searx > plugin > __init__.py. -And at the bottom of the file add your plugin like. -``plugins.register(name_of_python_file)`` Plugin entry points =================== diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index db52a2d80..921384aab 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -10,7 +10,7 @@ Development Quickstart SearXNG loves developers, just clone and start hacking. All the rest is done for you simply by using :ref:`make <makefile>`. -.. code:: sh +.. code:: bash git clone https://github.com/searxng/searxng.git searxng @@ -27,21 +27,21 @@ to our ":ref:`how to contribute`" guideline. If you implement themes, you will need to setup a :ref:`make node.env` once: -.. code:: sh +.. code:: bash make node.env Before you call *make run* (2.), you need to compile the modified styles and JavaScript: -.. code:: sh +.. code:: bash make themes.all Alternatively you can also compile selective the theme you have modified, e.g. the *simple* theme. -.. code:: sh +.. code:: bash make themes.simple @@ -52,7 +52,7 @@ e.g. the *simple* theme. If you finished your *tests* you can start to commit your changes. To separate the modified source code from the build products first run: -.. code:: sh +.. code:: bash make static.build.restore @@ -60,13 +60,13 @@ This will restore the old build products and only your changes of the code remain in the working tree which can now be added & commited. When all sources are commited, you can commit the build products simply by: -.. code:: sh +.. code:: bash make static.build.commit Commiting the build products should be the last step, just before you send us your PR. There is also a make target to rewind this last build commit: -.. code:: sh +.. code:: bash make static.build.drop |