diff options
| -rw-r--r-- | docs/admin/buildhosts.rst | 20 | ||||
| -rw-r--r-- | docs/build-templates/searx.rst | 26 | ||||
| -rw-r--r-- | docs/utils/lxc.sh.rst | 103 | ||||
| -rw-r--r-- | utils/lxc-searx.env | 12 | ||||
| -rwxr-xr-x | utils/lxc.sh | 10 | ||||
| -rwxr-xr-x | utils/searx.sh | 16 |
6 files changed, 139 insertions, 48 deletions
diff --git a/docs/admin/buildhosts.rst b/docs/admin/buildhosts.rst index a6820aa61..04698d8d2 100644 --- a/docs/admin/buildhosts.rst +++ b/docs/admin/buildhosts.rst @@ -4,9 +4,6 @@ Buildhosts ========== -To get best results from build, its recommend to install additional packages -on build hosts. - .. sidebar:: This article needs some work If you have any contribution send us your :pull:`PR <../pulls>`, see @@ -17,6 +14,23 @@ on build hosts. :local: :backlinks: entry +To get best results from build, its recommend to install additional packages +on build hosts (see :ref:`searx.sh`).:: + + sudo -H ./utils/searx.sh buildhost + +This will install packages needed by searx: + +.. include:: ../../build/docs/includes/searx.rst + :start-after: START distro-packages + :end-before: END distro-packages + +and packages needed to build docuemtation and run tests: + +.. include:: ../../build/docs/includes/searx.rst + :start-after: START build-packages + :end-before: END build-packages + .. _docs build: Build docs diff --git a/docs/build-templates/searx.rst b/docs/build-templates/searx.rst index 9e33928ad..964deb23d 100644 --- a/docs/build-templates/searx.rst +++ b/docs/build-templates/searx.rst @@ -28,6 +28,32 @@ ${fedora} .. END distro-packages +.. START build-packages + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo -H apt-get install -y \\ +${debian_build} + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo -H pacman -S --noconfirm \\ +${arch_build} + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo -H dnf install -y \\ +${fedora_build} + +.. END build-packages .. START create user diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst index 68577ea13..20f387158 100644 --- a/docs/utils/lxc.sh.rst +++ b/docs/utils/lxc.sh.rst @@ -1,5 +1,4 @@ - .. _snap: https://snapcraft.io .. _snapcraft LXD: https://snapcraft.io/lxd .. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/ @@ -7,6 +6,8 @@ .. _LXD: https://linuxcontainers.org/lxd/introduction/ .. _`LXD@github`: https://github.com/lxc/lxd +.. _archlinux: https://www.archlinux.org/ + .. _lxc.sh: ================ @@ -21,24 +22,31 @@ - `LXD@github`_ With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of -containers, what we call the: *lxc suite*. Before you can start with -containers, you need to install and initiate LXD_ once:: +containers, what we call the: *lxc suite*. The *searx suite* +(:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by default, every time +you start the ``lxc.sh`` script (*you do not need to care about*). + +Before you can start with containers, you need to install and initiate LXD_ +once:: $ snap install lxd $ lxd init --auto -The *searx suite* (:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by -default, every time you start the ``lxc.sh`` script (you do not need to care -about). To make use of the containers from the *searx suite*, you have to build -the :ref:`LXC suite containers <lxc.sh --help>` first. But be warned, this -might take some time:: +To make use of the containers from the *searx suite*, you have to build the +:ref:`LXC suite containers <lxc.sh help>` initial. But be warned, this might +take some time:: $ sudo -H ./utils/lxc.sh build A cup of coffee later, your LXC suite is build up and you can run whatever task you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh ---help>`. Each container shares the root folder of the repository and the -command ``utils/lxc.sh cmd`` handles relative path names *transparent*:: +help>`. + +*Good to know ...* + +Eeach container shares the root folder of the repository and the +command ``utils/lxc.sh cmd`` **handles relative path names transparent**, +compare output of:: $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile ... @@ -46,29 +54,70 @@ command ``utils/lxc.sh cmd`` handles relative path names *transparent*:: [searx-fedora31] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile [searx-archlinux] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile -With this in mind, you can run :ref:`searx.sh` and install packages, needed by -searx:: +If there comes the time you want to **get rid off all** the containers and +**clean up local images** just type:: - $ sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install packages + $ sudo -H ./utils/lxc.sh remove + $ sudo -H ./utils/lxc.sh remove images -And run one of the :origin:`Makefile` targets:: - $ sudo -H ./utils/lxc.sh cmd -- make test.sh +Install suite +============= -You can install a *buildhost environment* into the containers (time for another -cup of coffee):: +To install the complete :ref:`searx suite (includes searx, morty & filtron) +<lxc-searx.env>` into all LXC_ use:: - $ sudo -H ./utils/lxc.sh install buildhost + $ sudo -H ./utils/lxc.sh install suite -If you want to get rid off all the containers, just type:: +The command above installs a searx suite (see :ref:`installation scripts`). To +get the IP (URL) of the filtron service in the containers use ``show suite`` +command. To test instances from containers just open the URLs in your +WEB-Browser:: - $ sudo -H ./utils/lxc.sh remove + $ sudo ./utils/lxc.sh show suite | grep filtron + [searx-ubu1604] INFO: (eth0) filtron: http://n.n.n.135:4004/ + [searx-ubu1804] INFO: (eth0) filtron: http://n.n.n.141:4004/ + [searx-ubu1910] INFO: (eth0) filtron: http://n.n.n.137:4004/ + [searx-ubu2004] INFO: (eth0) filtron: http://n.n.n.127:4004/ + [searx-fedora31] INFO: (eth0) filtron: http://n.n.n.18:4004/ + [searx-archlinux] INFO: (eth0) filtron: http://n.n.n.12:4004/ -To clean up your local images use:: + +Running commands +================ - $ sudo -H ./utils/lxc.sh remove images +**Inside containers, you can use make or run scripts** from the +:ref:`toolboxing`. By example: to setup a :ref:`buildhosts` and run the +Makefile target ``test`` in the archlinux_ container:: + + sudo -H ./utils/lxc.sh cmd searx-archlinux ./utils/searx.sh buildhost + sudo -H ./utils/lxc.sh cmd searx-archlinux make test + + +Setup searx buildhost +===================== -.. _lxc.sh --help: +You can **install the searx buildhost environment** into one or all containers. +The installation procedure to set up a :ref:`build host<buildhosts>` takes its +time. Installation in all containers will take more time (time for another cup +of coffee).:: + + sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh buildhost + +To build (live) documentation inside a archlinux_ container:: + + sudo -H ./utils/lxc.sh cmd searx-archlinux make docs-clean docs-live + ... + [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080 + +To get IP of the container and the port number *live docs* is listening:: + + $ sudo ./utils/lxc.sh show suite | grep docs-live + ... + [searx-archlinux] INFO: (eth0) docs-live: http://n.n.n.12:8080/ + + +.. _lxc.sh help: Overview ======== @@ -77,3 +126,11 @@ The ``--help`` output of the script is largely self-explanatory: .. program-output:: ../utils/lxc.sh --help + +.. _lxc-searx.env: + +searx suite +=========== + +.. literalinclude:: ../../utils/lxc-searx.env + :language: bash diff --git a/utils/lxc-searx.env b/utils/lxc-searx.env index a5de6c3de..41bbae468 100644 --- a/utils/lxc-searx.env +++ b/utils/lxc-searx.env @@ -49,18 +49,6 @@ lxc_suite_install() { ) } -lxc_suite_prepare_buildhost() { - ( - lxc_set_suite_env - FORCE_TIMEOUT=0 - export FORCE_TIMEOUT - # "${LXC_REPO_ROOT}/utils/searx.sh* install packages - "${LXC_REPO_ROOT}/utils/searx.sh" install buildhost - rst_title "buildhost installation finished ($(hostname))" part - echo - ) -} - lxc_suite_info() { ( lxc_set_suite_env diff --git a/utils/lxc.sh b/utils/lxc.sh index 2526a257d..612ca566d 100755 --- a/utils/lxc.sh +++ b/utils/lxc.sh @@ -82,7 +82,7 @@ usage:: $_cmd [start|stop] [containers|<name>] $_cmd show [info|config|suite|images] $_cmd cmd [--|<name>] '...' - $_cmd install [suite|base|buildhost] + $_cmd install [suite|base] build :containers: build, launch and 'install basic' packages on 'containers' @@ -106,7 +106,6 @@ cmd install :suite: install LXC suite; ${LXC_SUITE_INSTALL_INFO} :base: prepare LXC; install basic packages - :buildhost: prepare LXC; install buildhost packages EOF usage_images @@ -150,7 +149,9 @@ main() { # don't check prerequisite when in recursion if [[ ! $1 == __* ]]; then - ! required_commands lxc && lxd_info && exit 42 + if ! in_container; then + ! required_commands lxc && lxd_info && exit 42 + fi [[ -z $LXC_SUITE ]] && err_msg "missing LXC_SUITE" && exit 42 fi @@ -240,7 +241,7 @@ main() { install) sudo_or_exit case $2 in - suite|base|buildhost) + suite|base) lxc_exec "${LXC_REPO_ROOT}/utils/lxc.sh" __install "$2" ;; *) usage "$_usage"; exit 42 ;; @@ -252,7 +253,6 @@ main() { case $2 in suite) lxc_suite_install ;; base) FORCE_TIMEOUT=0 lxc_install_base_packages ;; - buildhost) lxc_suite_prepare_buildhost ;; esac ;; doc) diff --git a/utils/searx.sh b/utils/searx.sh index 71ab978a1..17e8db84e 100755 --- a/utils/searx.sh +++ b/utils/searx.sh @@ -48,7 +48,7 @@ git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev shellcheck" BUILD_PACKAGES_debian="\ -graphviz imagemagick texlive-xetex librsvg2-bin +firefox graphviz imagemagick texlive-xetex librsvg2-bin texlive-latex-recommended texlive-extra-utils ttf-dejavu" # pacman packages @@ -59,7 +59,7 @@ git base-devel libxml2 shellcheck" BUILD_PACKAGES_arch="\ -graphviz imagemagick texlive-bin extra/librsvg +firefox graphviz imagemagick texlive-bin extra/librsvg texlive-core texlive-latexextra ttf-dejavu" # dnf packages @@ -70,7 +70,7 @@ git @development-tools libxml2 ShellCheck" BUILD_PACKAGES_fedora="\ -graphviz graphviz-gd ImageMagick librsvg2-tools +firefox graphviz graphviz-gd ImageMagick librsvg2-tools texlive-xetex-bin texlive-collection-fontsrecommended texlive-collection-latex dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts" @@ -116,7 +116,7 @@ usage() { usage:: $(basename "$0") shell - $(basename "$0") install [all|user|searx-src|pyenv|uwsgi|apache|packages] + $(basename "$0") install [all|user|searx-src|pyenv|uwsgi|apache|packages|buildhost] $(basename "$0") update [searx] $(basename "$0") remove [all|user|pyenv|searx-src] $(basename "$0") activate [service] @@ -135,7 +135,7 @@ install / remove :uwsgi: install searx uWSGI application :settings: reinstall settings from ${REPO_ROOT}/searx/settings.yml :packages: install needed packages from OS package manager - :buildhost: install packages from OS package manager needed by buildhost + :buildhost: install packages from OS package manager needed by buildhosts update searx Update searx installation ($SERVICE_HOME) activate service @@ -688,9 +688,15 @@ rst-doc() { local debian="${SEARX_PACKAGES_debian}" local arch="${SEARX_PACKAGES_arch}" local fedora="${SEARX_PACKAGES_fedora}" + local debian_build="${BUILD_PACKAGES_debian}" + local arch_build="${BUILD_PACKAGES_arch}" + local fedora_build="${BUILD_PACKAGES_fedora}" debian="$(echo "${debian}" | sed 's/.*/ & \\/' | sed '$ s/.$//')" arch="$(echo "${arch}" | sed 's/.*/ & \\/' | sed '$ s/.$//')" fedora="$(echo "${fedora}" | sed 's/.*/ & \\/' | sed '$ s/.$//')" + debian_build="$(echo "${debian_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')" + arch_build="$(echo "${arch_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')" + fedora_build="$(echo "${fedora_build}" | sed 's/.*/ & \\/' | sed '$ s/.$//')" eval "echo \"$(< "${REPO_ROOT}/docs/build-templates/searx.rst")\"" |