+ +
+

Developing in Linux Containers

+

In this article we will show, how you can make use of Linux Containers (LXC) in +distributed and heterogeneous development cycles (TL;DR; jump to the +Summary).

+ + +
+

Motivation

+

Most often in our development cycle, we edit the sources and run some test +and/or builds by using make [ref] before we commit. This +cycle is simple and perfect but might fail in some aspects we should not +overlook.

+
+

The environment in which we run all our development processes matters!

+
+

The Makefile & ./manage and the Python environment (make install) encapsulate a lot for us, but +these tools do not have access to all prerequisites. For example, there may +have dependencies on packages that are installed on developer’s desktop, but +usually are not preinstalled on a server or client system. Another example is; +settings have been made to the software on developer’s desktop that would never +be set on a production system.

+
+

Linux Containers are isolate environments, we use them to not mix up all +the prerequisites from various projects on developer’s desktop.

+
+

The scripts from DevOps tooling box can divide in those to install and maintain +software

+ +

and the script

+ +

with we can scale our installation, maintenance or even development tasks over a +stack of isolated containers / what we call the:

+ +
+
+

Gentlemen, start your engines!

+

Before you can start with containers, you need to install and initiate LXD +once:

+
+
$ snap install lxd
+$ lxd init --auto
+
+
+
+

And you need to clone from origin or if you have your own fork, clone from your +fork:

+
+
$ cd ~/Downloads
+$ git clone https://github.com/searxng/searxng.git searxng
+$ cd searxng
+
+
+
+ +

The SearXNG suite config consists of several images, see export +LXC_SUITE=(... near by git://utils/lxc-searxng.env#L19. +For this blog post we exercise on a archlinux image. The container of this +image is named searxng-archlinux.

+

Lets build the container, but be sure that this container does not already +exists, so first lets remove possible old one:

+
+
$ sudo -H ./utils/lxc.sh remove searxng-archlinux
+$ sudo -H ./utils/lxc.sh build searxng-archlinux
+
+
+
+ +

To install the complete SearXNG suite and the HTTP +proxy NGINX into the archlinux container run:

+
+
$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
+$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+$ sudo ./utils/lxc.sh show suite | grep SEARXNG_URL
+...
+[searxng-archlinux]    SEARXNG_URL          : http://n.n.n.140/searxng
+
+
+
+ +

In such a SearXNG suite admins can maintain and access the debug log of the +services quite easy.

+

In the example above the SearXNG instance in the container is wrapped to +http://n.n.n.140/searxng to the HOST system. Note, on your HOST system, the +IP of your searxng-archlinux container is different to this example. To +test the instance in the container from outside of the container, in your WEB +browser on your desktop just open the URL reported in your installation

+
+
+

In containers, work as usual

+

Usually you open a root-bash using sudo -H bash. In case of LXC containers +open the root-bash in the container is done by the ./utils/lxc.sh cmd +searxng-archlinux command:

+
+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux bash
+INFO:  [searxng-archlinux] bash
+[root@searxng-archlinux SearXNG]$
+
+
+
+

The prompt [root@searxng-archlinux ...] signals, that you are the root user +in the container (GUEST). To debug the running SearXNG instance use:

+
+
$ ./utils/searxng.sh instance inspect
+...
+use [CTRL-C] to stop monitoring the log
+...
+
+
+
+

Back in the browser on your desktop open the service http://n.n.n.140/searxng +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using CTRL-C, this also disables the “debug +option” in SearXNG’s settings file and restarts the SearXNG uwsgi application.

+

Another point we have to notice is that the service SearXNG +runs under dedicated system user account with the same name (compare +Create user). To get a login shell from these accounts, simply +call:

+
+
$ ./utils/searxng.sh instance cmd bash -l
+(searx-pyenv) [searxng@searxng-archlinux ~]$ pwd
+/usr/local/searxng
+
+
+
+

The prompt [searxng@searxng-archlinux] signals that you are logged in as system +user searxng in the searxng-archlinux container and the python virtualenv +(searxng-pyenv) environment is activated.

+
+
+

Wrap production into developer suite

+

In this section we will see how to change the “Fully functional SearXNG suite” +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the Step by step installation:

+
    +

  • SearXNG setup in: /etc/searxng/settings.yml

    • +

  • SearXNG user’s home: /usr/local/searxng

    • +

  • virtualenv in: /usr/local/searxng/searxng-pyenv

    • +

  • SearXNG software in: /usr/local/searxng/searxng-src

    • +
+

With the use of the utils/searxng.sh the SearXNG service was installed as +uWSGI application. To maintain this service, we can use +systemctl (compare uWSGI maintenance).

+
+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl stop uwsgi@searxng
+
+
+
+

With the command above, we stopped the SearXNG uWSGI-App in the archlinux +container.

+

The uWSGI-App for the archlinux distros is configured in +git://utils/templates/etc/uwsgi/apps-archlinux/searxng.ini, from where at +least you should attend the settings of uid, chdir, env and +http:

+
env = SEARXNG_SETTINGS_PATH=/etc/searxng/settings.yml
+http = 127.0.0.1:8888
+
+chdir = /usr/local/searxng/searxng-src/searx
+virtualenv = /usr/local/searxng/searxng-pyenv
+pythonpath = /usr/local/searxng/searxng-src
+
+
+

If you have read the Good to know you remember, that each container +shares the root folder of the repository and the command utils/lxc.sh cmd +handles relative path names transparent.

+

To wrap the SearXNG installation in the container into a developer one, we +simple have to create a symlink to the transparent repository from the +desktop. Now lets replace the repository at searxng-src in the container +with the working tree from outside of the container:

+
+
$ mv /usr/local/searxng/searxng-src  /usr/local/searxng/searxng-src.old
+$ ln -s /share/SearXNG/ /usr/local/searxng/searxng-src
+
+
+
+

Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the SearXNG service (in the +container):

+
+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
+
+
+
+

Remember: In containers, work as usual .. here are just some examples from my +daily usage:

+

To inspect the SearXNG instance (already described above):

+
+
$ ./utils/searx.sh inspect service
+
+
+
+

Run Makefile & ./manage, e.g. to test inside the container:

+
+
$ make test
+
+
+
+

To install all prerequisites needed for a Buildhosts:

+
+
$ ./utils/searxng.sh install buildhost
+
+
+
+

To build the docs on a buildhost Buildhosts:

+
+
$ make docs.html
+
+
+
+
+
+

Summary

+

We build up a fully functional SearXNG suite in a archlinux container:

+
$ sudo -H ./utils/lxc.sh build searxng-archlinux
+$ sudo -H ./utils/lxc.sh install suite searxng-archlinux
+...
+Developer install? (wraps source from HOST into the running instance) [YES/no]
+
+
+

To wrap the suite into a developer one answer YES (or press Enter).

+
link SearXNG's sources to: /share/SearXNG
+=========================================
+
+mv -f "/usr/local/searxng/searxng-src" "/usr/local/searxng/searxng-src.backup"
+ln -s "/share/SearXNG" "/usr/local/searxng/searxng-src"
+ls -ld /usr/local/searxng/searxng-src
+  |searxng| lrwxrwxrwx 1 searxng searxng ... /usr/local/searxng/searxng-src -> /share/SearXNG
+
+
+

On code modification the instance has to be restarted (see uWSGI maintenance):

+
$ sudo -H ./utils/lxc.sh cmd searxng-archlinux systemctl restart uwsgi@searxng
+
+
+

To access HTTP from the desktop we installed nginx for the services inside the +container:

+
$ sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx
+
+
+

To get information about the SearxNG suite in the archlinux container we can +use:

+
$ sudo -H ./utils/lxc.sh show suite searxng-archlinux
+[searxng-archlinux]  INFO:  (eth0) docs-live:  http:///n.n.n.140:8080/
+[searxng-archlinux]  INFO:  (eth0) IPv6:       http://[fd42:555b:2af9:e121:216:3eff:fe5b:1744]
+[searxng-archlinux]  uWSGI:
+[searxng-archlinux]    SEARXNG_UWSGI_SOCKET : /usr/local/searxng/run/socket
+[searxng-archlinux]  environment /usr/local/searxng/searxng-src/utils/brand.env:
+[searxng-archlinux]    GIT_URL              : https://github.com/searxng/searxng
+[searxng-archlinux]    GIT_BRANCH           : master
+[searxng-archlinux]    SEARXNG_URL          : http:///n.n.n.140/searxng
+[searxng-archlinux]    SEARXNG_PORT         : 8888
+[searxng-archlinux]    SEARXNG_BIND_ADDRESS : 127.0.0.1
+
+
+
+
+ + +
+