diff options
Diffstat (limited to '_sources/dev/lxcdev.rst.txt')
| -rw-r--r-- | _sources/dev/lxcdev.rst.txt | 438 |
1 files changed, 438 insertions, 0 deletions
diff --git a/_sources/dev/lxcdev.rst.txt b/_sources/dev/lxcdev.rst.txt new file mode 100644 index 000000000..79716ae57 --- /dev/null +++ b/_sources/dev/lxcdev.rst.txt @@ -0,0 +1,438 @@ +.. _lxcdev: + +============================== +Developing in Linux Containers +============================== + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +In this article we will show, how you can make use of Linux Containers (LXC_) in +*distributed and heterogeneous development cycles* (TL;DR; jump to the +:ref:`lxcdev summary`). + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers. Readers + should have a serious meaning about the terms: *distributed*, *merge* and + *linux container*. + + **hint** + + If you have issues with the internet connectivity of your containers read + section :ref:`internet connectivity docker`. + + +.. contents:: + :depth: 2 + :local: + :backlinks: entry + + +Motivation +========== + +Most often in our development cycle, we edit the sources and run some test +and/or builds by using ``make`` :ref:`[ref] <makefile>` before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook. + + **The environment in which we run all our development processes matters!** + +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer's desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer's desktop that would never +be set on a *production* system. + + **Linux Containers are isolate environments**, we use them to not mix up all + the prerequisites from various projects on developer's desktop. + +The scripts from :ref:`searx_utils` can divide in those to install and maintain +software + +- :ref:`searxng.sh` + +and the script + +- :ref:`lxc.sh` + +with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the: + +- :ref:`searxng lxc suite` + +.. _lxcdev install searxng: + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ cd ~/Downloads + $ git clone https://github.com/searxng/searxng.git searxng + $ cd searxng + +.. sidebar:: The ``searxng-archlinux`` container + + is the base of all our exercises here. + +The :ref:`lxc-searxng.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searxng.env#L19`. +For this blog post we exercise on a archlinux_ image. The container of this +image is named ``searxng-archlinux``. + +Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh remove searxng-archlinux + $ sudo -H ./utils/lxc.sh build searxng-archlinux + + +.. sidebar:: further read + + - :ref:`lxc.sh install suite` + - :ref:`installation nginx` + +To install the complete :ref:`SearXNG suite <searxng lxc suite>` and the HTTP +proxy :ref:`installation nginx` into the archlinux container run: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + $ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL + ... + [searxng-archlinux] SEARXNG_URL : http://n.n.n.140/searxng + +.. sidebar:: Fully functional SearXNG suite + + From here on you have a fully functional SearXNG suite (including a + :ref:`redis db`). + +In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy. + +In the example above the SearXNG instance in the container is wrapped to +``http://n.n.n.140/searxng`` to the HOST system. Note, on your HOST system, the +IP of your ``searxng-archlinux`` container is different to this example. To +test the instance in the container from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container is done by the ``./utils/lxc.sh cmd +searxng-archlinux`` command: + +.. tabs:: + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash + INFO: [searxng-archlinux] bash + [root@searxng-archlinux SearXNG]$ + +The prompt ``[root@searxng-archlinux ...]`` signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance inspect + ... + use [CTRL-C] to stop monitoring the log + ... + + +Back in the browser on your desktop open the service http://n.n.n.140/searxng +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in SearXNG's settings file and restarts the SearXNG uwsgi application. + +Another point we have to notice is that the service :ref:`SearXNG <searxng.sh>` +runs under dedicated system user account with the same name (compare +:ref:`create searxng user`). To get a login shell from these accounts, simply +call: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh instance cmd bash -l + INFO: [searxng-archlinux] ./utils/searxng.sh instance cmd bash -l + (searx-pyenv) [searxng@searxng-archlinux ~]$ pwd + /usr/local/searxng + +The prompt ``[searxng@searxng-archlinux]`` signals that you are logged in as system +user ``searxng`` in the ``searxng-archlinux`` container and the python *virtualenv* +``(searxng-pyenv)`` environment is activated. + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional SearXNG suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- SearXNG setup in: ``/etc/searxng/settings.yml`` +- SearXNG user's home: ``/usr/local/searxng`` +- virtualenv in: ``/usr/local/searxng/searxng-pyenv`` +- SearXNG software in: ``/usr/local/searxng/searxng-src`` + +With the use of the :ref:`searxng.sh` the SearXNG service was installed as +:ref:`uWSGI application <searxng uwsgi>`. To maintain this service, we can use +``systemctl`` (compare :ref:`uWSGI maintenance`). + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng + +With the command above, we stopped the SearXNG uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux distros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searxng.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searxng/searxng-src/searx + virtualenv = /usr/local/searxng/searxng-pyenv + pythonpath = /usr/local/searxng/searxng-src + +If you have read the :ref:`Good to know` you remember, that each container +shares the root folder of the repository and the command ``utils/lxc.sh cmd`` +handles relative path names **transparent**. + +To wrap the SearXNG installation in the container into a developer one, we +simple have to create a symlink to the **transparent** repository from the +desktop. Now lets replace the repository at ``searxng-src`` in the container +with the working tree from outside of the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + $ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + mv /usr/local/searxng/searxng-src /usr/local/searxng/searxng-src.old + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux \ + ln -s /share/SearXNG/ /usr/local/searxng/searxng-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the SearXNG service (in the +container): + +.. tabs:: + + .. group-tab:: uwsgi@searxng + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +To *inspect* the SearXNG instance (already described above): + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searx.sh inspect service + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searx.sh inspect service + +Run :ref:`makefile`, e.g. to test inside the container: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make test + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make test + + + +To install all prerequisites needed for a :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ ./utils/searxng.sh install buildhost + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux ./utils/searxng.sh install buildhost + + +To build the docs on a buildhost :ref:`buildhosts`: + +.. tabs:: + + .. group-tab:: ``[root@searxng-archlinux SearXNG]`` (GUEST) + + .. code:: bash + + $ make docs.html + + .. group-tab:: desktop (HOST) + + .. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux make docs.html + + +.. _lxcdev summary: + +Summary +======= + +We build up a fully functional SearXNG suite in a archlinux container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh build searxng-archlinux + $ sudo -H ./utils/lxc.sh install suite searxng-archlinux + ... + Developer install? (wraps source from HOST into the running instance) [YES/no] + +To wrap the suite into a developer one answer ``YES`` (or press Enter). + +.. code:: text + + link SearXNG's sources to: /share/SearXNG + ========================================= + + mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup" + ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src" + ls -ld /usr/local/searxng/searxng-src + |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG + +On code modification the instance has to be restarted (see :ref:`uWSGI +maintenance`): + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng + +To access HTTP from the desktop we installed nginx for the services inside the +container: + +.. code:: bash + + $ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx + +To get information about the SearxNG suite in the archlinux container we can +use: + +.. code:: text + + $ sudo -H ./utils/lxc.sh show suite searxng-archlinux + [searxng-archlinux] INFO: (eth0) docs-live: http:///n.n.n.140:8080/ + [searxng-archlinux] INFO: (eth0) IPv6: http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744] + [searxng-archlinux] uWSGI: + [searxng-archlinux] SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket + [searxng-archlinux] environment /usr/local/searxng/searxng-src/utils/brand.env: + [searxng-archlinux] GIT_URL : https://github.com/searxng/searxng + [searxng-archlinux] GIT_BRANCH : master + [searxng-archlinux] SEARXNG_URL : http:///n.n.n.140/searxng + [searxng-archlinux] SEARXNG_PORT : 8888 + [searxng-archlinux] SEARXNG_BIND_ADDRESS : 127.0.0.1 + |