diff options
Diffstat (limited to 'docs/dev/lxcdev.rst')
| -rw-r--r-- | docs/dev/lxcdev.rst | 339 |
1 files changed, 187 insertions, 152 deletions
diff --git a/docs/dev/lxcdev.rst b/docs/dev/lxcdev.rst index 6688c21f0..ef603e9fd 100644 --- a/docs/dev/lxcdev.rst +++ b/docs/dev/lxcdev.rst @@ -16,6 +16,12 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in should have a serious meaning about the terms: *distributed*, *merge* and *linux container*. + **hint** + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + + .. contents:: Contents :depth: 2 :local: @@ -25,38 +31,38 @@ In this article we will show, how you can make use of Linux Containers (LXC_) in Motivation ========== -Usually in our development cycle, we edit the sources and run some test and/or -builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This cycle -is simple and perfect but might fail in some aspects we should not overlook. +Most often in our development cycle, we edit the sources and run some test +and/or builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook. **The environment in which we run all our development processes matters!** The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but -they do not have access to all prerequisites. For example, there may have -dependencies on packages that are installed on the developer's desktop, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer's desktop, but usually are not preinstalled on a server or client system. Another example is; settings have been made to the software on developer's desktop that would never be set on a *production* system. - **Linux Containers are isolate environments and not to mix up all the - prerequisites from various projects on developer's desktop is always a good - choice.** + **Linux Containers are isolate environments**, we use them to not mix up all + the prerequisites from various projects on developer's desktop. The scripts from :ref:`searx_utils` can divide in those to install and maintain -software: +software - :ref:`searxng.sh` -and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or -even development tasks over a stack of isolated containers / what we call the: +and the script - **SearXNG LXC suite** +- :ref:`lxc.sh` -.. hint:: +with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the: - If you see any problems with the internet connectivity of your - containers read section :ref:`internet connectivity docker`. +- :ref:`searxng lxc suite` +.. _lxcdev install searxng: Gentlemen, start your engines! ============================== @@ -69,7 +75,7 @@ once: .. tabs:: - .. group-tab:: desktop + .. group-tab:: desktop (HOST) .. code:: bash @@ -81,7 +87,7 @@ fork: .. tabs:: - .. group-tab:: desktop + .. group-tab:: desktop (HOST) .. code:: bash @@ -89,76 +95,61 @@ fork: $ git clone https://github.com/searxng/searxng.git searxng $ cd searxng -The :ref:`lxc-searxng.env` consists of several images, see ``export -LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. For this blog post -we exercise on a archlinux_ image. The container of this image is named -``searxng-archlinux``. Lets build the container, but be sure that this container -does not already exists, so first lets remove possible old one: - -.. tabs:: - - .. group-tab:: desktop - - .. code:: bash - - $ sudo -H ./utils/lxc.sh remove searxng-archlinux - $ sudo -H ./utils/lxc.sh build searxng-archlinux - .. sidebar:: The ``searxng-archlinux`` container is the base of all our exercises here. -In this container we install all services :ref:`including searx, morty & filtron -<lxc.sh install suite>` in once: +The :ref:`lxc-searxng.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. +For this blog post we exercise on a archlinux_ image. The container of this +image is named ``searxng-archlinux``. + +Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one: .. tabs:: - .. group-tab:: desktop + .. group-tab:: desktop (HOST) .. code:: bash - $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + $ sudo -H ./utils/lxc.sh remove searxng-archlinux + $ sudo -H ./utils/lxc.sh build searxng-archlinux -To proxy HTTP from filtron and morty in the container to the outside of the -container, install nginx into the container. Once for the bot blocker filtron: -.. tabs:: +.. sidebar:: further read - .. group-tab:: desktop + - :ref:`lxc.sh install suite` + - :ref:`installation nginx` - .. code:: bash - - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - ./utils/filtron.sh nginx install - ... - INFO: got 429 from http://10.174.184.156/searx - -and once for the content sanitizer (content proxy morty): +To install the complete :ref:`SearXNG suite <searxng lxc suite>` and the HTTP +proxy :ref:`installation nginx` into the archlinux container run: .. tabs:: - .. group-tab:: desktop + .. group-tab:: desktop (HOST) .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - ./utils/morty.sh nginx install + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL ... - INFO: got 200 from http://10.174.184.156/morty/ + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng .. sidebar:: Fully functional SearXNG suite - From here on you have a fully functional SearXNG suite running with bot - blocker (filtron) and WEB content sanitizer (content proxy morty), both are - needed for a *privacy protecting* search engine. + From here on you have a fully functional SearXNG suite (including a + :ref:`redis db`). -On your system, the IP of your ``searxng-archlinux`` container differs from -http://10.174.184.156/searx, just open the URL reported in your installation -protocol in your WEB browser from the desktop to test the instance from outside -of the container. +In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy. -In such a earXNG suite admins can maintain and access the debug log of the -different services quite easy. +In the example above the SearXNG instance in the container is wrapped to +``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the +IP of your ``searxng-archlinux`` container is different to this example. To +test the instance in the conatiner from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation .. _working in containers: @@ -166,77 +157,76 @@ In containers, work as usual ============================ Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers -open the root-bash in the container using ``./utils/lxc.sh cmd -searxng-archlinux``: +open the root-bash in the container is done by the ``./utils/lxc.sh cmd +searxng-archlinux`` command: .. tabs:: - .. group-tab:: desktop + .. group-tab:: desktop (HOST) .. code:: bash $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash INFO: [searxng-archlinux] bash - [root@searxng-archlinux searx]# pwd - /share/searxng + [root@searxng-archlinux SearXNG]$ -The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user in -the searxng-container. To debug the running SearXNG instance use: +The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use: .. tabs:: - .. group-tab:: root@searxng-archlinux + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) .. code:: bash - $ ./utils/searx.sh inspect service + $ ./utils/searxng.sh instance inspect ... use [CTRL-C] to stop monitoring the log ... -Back in the browser on your desktop open the service http://10.174.184.156/searx + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + +Back in the browser on your desktop open the service http://n.n.n.140/searxng and run your application tests while the debug log is shown in the terminal from above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. -To debug services from filtron and morty analogous use: -Another point we have to notice is that the service (:ref:`SearXNG <searxng.sh>` +Another point we have to notice is that the service :ref:`SearXNG <searxng.sh>` runs under dedicated system user account with the same name (compare -:ref:`create searxng user`). To get a shell from these accounts, simply call: +:ref:`create searxng user`). To get a login shell from these accounts, simply +call: .. tabs:: - .. group-tab:: root@searxng-archlinux + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) .. code:: bash - $ ./utils/searxng.sh instance cmd bash - -To get in touch, open a shell from the service user (searxng@searxng-archlinux): - -.. tabs:: + $ ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng - .. group-tab:: desktop + .. group-tab:: desktop (HOST) .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash - INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash - [searxng@searxng-archlinux ~]$ + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l + INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system -user ``searx`` in the ``searxng-archlinux`` container and the python *virtualenv* +user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv* ``(searxng-pyenv)`` environment is activated. -.. tabs:: - - .. group-tab:: searxng@searxng-archlinux - - .. code:: bash - - (searxng-pyenv) [searxng@searxng-archlinux ~]$ pwd - /usr/local/searxng - Wrap production into developer suite ==================================== @@ -256,12 +246,11 @@ With the use of the :ref:`searxng.sh` the SearXNG service was installed as .. tabs:: - .. group-tab:: desktop + .. group-tab:: uwsgi@searxng .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - systemctl stop uwsgi@searxng + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng With the command above, we stopped the SearXNG uWSGI-App in the archlinux container. @@ -278,17 +267,25 @@ least you should attend the settings of ``uid``, ``chdir``, ``env`` and virtualenv = /usr/local/searxng/searxng-pyenv pythonpath = /usr/local/searxng/searxng-src -If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that -each container shares the root folder of the repository and the command -``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the -SearXNG installation into a developer one, we simple have to create a smylink to -the **transparent** reposetory from the desktop. Now lets replace the -repository at ``searxng-src`` in the container with the working tree from outside -of the container: +If you have read the :ref:`Good to know` you remember, that each container +shares the root folder of the repository and the command ``utils/lxc.sh cmd`` +handles relative path names **transparent**. + +To wrap the SearXNG installation in the container into a developer one, we +simple have to create a smylink to the **transparent** reposetory from the +desktop. Now lets replace the repository at ``searxng-src`` in the container +with the working tree from outside of the container: .. tabs:: - .. group-tab:: container becomes a developer suite + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + + .. group-tab:: desktop (HOST) .. code:: bash @@ -296,7 +293,7 @@ of the container: mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - ln -s /share/searx/ /usr/local/searxng/searxng-src + ln -s /share/SearXNG/ /usr/local/searxng/searxng-src Now we can develop as usual in the working tree of our desktop system. Every time the software was changed, you have to restart the SearXNG service (in the @@ -304,48 +301,83 @@ container): .. tabs:: - .. group-tab:: desktop + .. group-tab:: uwsgi@searxng .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - systemctl restart uwsgi@searx + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng Remember: :ref:`working in containers` .. here are just some examples from my daily usage: +To *inspect* the SearXNG instance (already described above): + .. tabs:: - .. group-tab:: desktop + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) - To *inspect* the SearXNG instance (already described above): + .. code:: bash + + $ ./utils/searx.sh inspect service + + .. group-tab:: desktop (HOST) .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - ./utils/searx.sh inspect service + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service - Run :ref:`makefile`, e.g. to test inside the container: +Run :ref:`makefile`, e.g. to test inside the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - make test + $ make test - To install all prerequisites needed for a :ref:`buildhosts`: + .. group-tab:: desktop (HOST) .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - ./utils/searxng.sh install buildhost + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test + - To build the docs on a buildhost :ref:`buildhosts`: + +To install all prerequisites needed for a :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) .. code:: bash - $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ - make docs.html + $ ./utils/searxng.sh install buildhost + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost + + +To build the docs on a buildhost :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make docs.html + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html + .. _lxcdev summary: @@ -356,48 +388,51 @@ We build up a fully functional SearXNG suite in a archlinux container: .. code:: bash + $ sudo -H ./utils/lxc.sh build searxng-archlinux $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + ... + Developer install? (wraps source from HOST into the running instance) [YES/no] -To access HTTP from the desktop we installed nginx for the services inside the -container: +To wrap the suite into a developer one answer ``YES`` (or press Enter). -.. tabs:: +.. code:: text - .. group-tab:: [root@searxng-archlinux] + link SearXNG's sources to: /share/SearXNG + ========================================= - .. code:: bash + mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" + ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" + ls -ld /usr/local/searxng/searxng-src + |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG - $ ./utils/filtron.sh nginx install - $ ./utils/morty.sh nginx install +On code modification the instance has to be restarted (see :ref:`uWSGI +maintenance`): -To wrap the suite into a developer one, we created a symbolic link to the -repository which is shared **transparent** from the desktop's file system into -the container : +.. code:: bash -.. tabs:: + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng - .. group-tab:: [root@searxng-archlinux] +To access HTTP from the desktop we installed nginx for the services inside the +container: - .. code:: bash +.. code:: bash - $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old - $ ln -s /share/searx/ /usr/local/searxng/searxng-src - $ systemctl restart uwsgi@searx + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx -To get information about the searxNG suite in the archlinux container we can +To get information about the SearxNG suite in the archlinux container we can use: -.. tabs:: - - .. group-tab:: desktop - - .. code:: bash - - $ sudo -H ./utils/lxc.sh show suite searxng-archlinux - ... - [searxng-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx - [searxng-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ - [searxng-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ - [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] - ... +.. code:: text + + $ sudo -H ./utils/lxc.sh show suite searxng-archlinux + [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ + [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] + [searxng-archlinux] uWSGI: + [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket + [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: + [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng + [searxng-archlinux] GIT_BRANCH : master + [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng + [searxng-archlinux] SEARXNG_PORT : 8888 + [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1 |