diff options
author | Markus Heiser <markus.heiser@darmarIT.de> | 2020-06-29 16:09:59 +0000 |
---|---|---|
committer | GitHub <noreply@github.com> | 2020-06-29 16:09:59 +0000 |
commit | 8d318ee1425b55d71b219d46d46860defa9e832b (patch) | |
tree | 1c6892a6ecbd9ded0ec7e16294ab7688ce80a69f | |
parent | 5fac6cffa2fc974e12fc4c07530561c380fbb1f0 (diff) | |
parent | 460401293ca8c57444f7a2e5709f2575b98ecfc2 (diff) | |
download | searxng-8d318ee1425b55d71b219d46d46860defa9e832b.tar.gz searxng-8d318ee1425b55d71b219d46d46860defa9e832b.zip |
Merge branch 'master' into gigablast
-rw-r--r-- | README.rst | 96 | ||||
-rw-r--r-- | docs/admin/installation-uwsgi.rst | 11 | ||||
-rw-r--r-- | docs/blog/index.rst | 1 | ||||
-rw-r--r-- | docs/blog/lxcdev-202006.rst | 413 | ||||
-rw-r--r-- | docs/blog/private-engines.rst | 2 | ||||
-rw-r--r-- | docs/dev/search_api.rst | 4 | ||||
-rw-r--r-- | docs/user/own-instance.rst | 4 | ||||
-rw-r--r-- | docs/utils/lxc.sh.rst | 7 | ||||
-rw-r--r-- | searx/engines/duckduckgo.py | 6 | ||||
-rw-r--r-- | searx/engines/yacy.py | 2 | ||||
-rw-r--r-- | searx/templates/__common__/about.html | 14 | ||||
-rw-r--r-- | searx/templates/oscar/base.html | 4 | ||||
-rw-r--r-- | searx/templates/oscar/macros.html | 26 | ||||
-rw-r--r-- | searx/templates/oscar/result_templates/code.html | 4 | ||||
-rw-r--r-- | searx/templates/oscar/result_templates/default.html | 10 | ||||
-rw-r--r-- | searx/templates/oscar/result_templates/images.html | 4 | ||||
-rw-r--r-- | searx/templates/oscar/result_templates/map.html | 8 | ||||
-rw-r--r-- | searx/templates/oscar/result_templates/torrent.html | 8 | ||||
-rw-r--r-- | searx/templates/oscar/result_templates/videos.html | 12 | ||||
-rwxr-xr-x | utils/searx.sh | 70 |
20 files changed, 578 insertions, 128 deletions
diff --git a/README.rst b/README.rst index a7b75d6e2..55deb674e 100644 --- a/README.rst +++ b/README.rst @@ -1,63 +1,73 @@ -searx -===== +.. SPDX-License-Identifier: AGPL-3.0-or-later + +.. figure:: https://raw.githubusercontent.com/asciimoo/searx/master/searx/static/themes/oscar/img/logo_searx_a.png + :target: https://asciimoo.github.io/searx/ + :alt: searX + :width: 100% + :align: center + +------- + +|searx install| +|searx homepage| +|searx wiki| +|AGPL License| +|Issues| +|commits| +|OpenCollective searx backers| +|OpenCollective searx sponsors| -A privacy-respecting, hackable `metasearch -engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__. +Privacy-respecting, hackable `metasearch engine`_ / *pronunciation* **səːks**. -Pronunciation: səːks +.. _metasearch engine: https://en.wikipedia.org/wiki/Metasearch_engine -List of `running -instances <https://searx.space/>`__. +.. |searx install| image:: https://img.shields.io/badge/-install-blue + :target: https://asciimoo.github.io/searx/admin/installation.html -See the `documentation <https://asciimoo.github.io/searx>`__ and the `wiki <https://github.com/asciimoo/searx/wiki>`__ for more information. +.. |searx homepage| image:: https://img.shields.io/badge/-homepage-blue + :target: https://asciimoo.github.io/searx -|OpenCollective searx backers| -|OpenCollective searx sponsors| +.. |searx wiki| image:: https://img.shields.io/badge/-wiki-blue + :target: https://github.com/asciimoo/searx/wiki -Installation -~~~~~~~~~~~~ +.. |AGPL License| image:: https://img.shields.io/badge/license-AGPL-blue.svg + :target: https://github.com/asciimoo/searx/blob/master/LICENSE -With Docker ------------ -Go to the `searx-docker <https://github.com/searx/searx-docker>`__ project. +.. |Issues| image:: https://img.shields.io/github/issues/asciimoo/searx?color=yellow&label=issues + :target: https://github.com/asciimoo/searx/issues -Without Docker --------------- -For all of the details, follow this `step by step installation <https://asciimoo.github.io/searx/admin/installation.html>`__. +.. |PR| image:: https://img.shields.io/github/issues-pr-raw/asciimoo/searx?color=yellow&label=PR + :target: https://github.com/asciimoo/searx/pulls -Note: the documentation needs to be updated. +.. |commits| image:: https://img.shields.io/github/commit-activity/y/asciimoo/searx?color=yellow&label=commits + :target: https://github.com/asciimoo/searx/commits/master -If you are in a hurry ---------------------- -- clone the source: - ``git clone https://github.com/asciimoo/searx.git && cd searx`` -- install dependencies: ``./manage.sh update_packages`` -- edit your - `settings.yml <https://github.com/asciimoo/searx/blob/master/searx/settings.yml>`__ - (set your ``secret_key``!) -- run ``python searx/webapp.py`` to start the application +.. |OpenCollective searx backers| image:: https://opencollective.com/searx/backers/badge.svg + :target: https://opencollective.com/searx#backer +.. |OpenCollective searx sponsors| image:: https://opencollective.com/searx/sponsors/badge.svg + :target: https://opencollective.com/searx#sponsor -Bugs -~~~~ -Bugs or suggestions? Visit the `issue -tracker <https://github.com/asciimoo/searx/issues>`__. +If you are looking for running instances, ready to use, then visit searx.space_. -`License <https://github.com/asciimoo/searx/blob/master/LICENSE>`__ -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Otherwise jump to the user_, admin_ and developer_ handbooks you will find on +our homepage_. -More about searx -~~~~~~~~~~~~~~~~ +.. _searx.space: https://searx.space +.. _user: https://asciimoo.github.io/searx/user +.. _admin: https://asciimoo.github.io/searx/admin +.. _developer: https://asciimoo.github.io/searx/dev +.. _homepage: https://asciimoo.github.io/searx -- `openhub <https://www.openhub.net/p/searx/>`__ -- `twitter <https://twitter.com/Searx_engine>`__ -- IRC: #searx @ freenode +contact: + openhub_ // twitter_ // IRC: #searx @ freenode +.. _openhub: https://www.openhub.net/p/searx +.. _twitter: https://twitter.com/Searx_engine -.. |OpenCollective searx backers| image:: https://opencollective.com/searx/backers/badge.svg - :target: https://opencollective.com/searx#backer +------- +|gluten free| -.. |OpenCollective searx sponsors| image:: https://opencollective.com/searx/sponsors/badge.svg - :target: https://opencollective.com/searx#sponsor +.. |gluten free| image:: https://forthebadge.com/images/featured/featured-gluten-free.svg diff --git a/docs/admin/installation-uwsgi.rst b/docs/admin/installation-uwsgi.rst index ac4c463b9..7b482975d 100644 --- a/docs/admin/installation-uwsgi.rst +++ b/docs/admin/installation-uwsgi.rst @@ -108,6 +108,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description ubuntu-20.04 :end-before: END searx uwsgi-description ubuntu-20.04 + .. hotfix: a bug group-tab need this comment .. group-tab:: Arch Linux @@ -115,6 +116,7 @@ restart the uwsgi application. :start-after: START searx uwsgi-description arch :end-before: END searx uwsgi-description arch + .. hotfix: a bug group-tab need this comment .. group-tab:: Fedora / RHEL @@ -128,22 +130,21 @@ restart the uwsgi application. .. group-tab:: Ubuntu / debian .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini ubuntu-20.04 :end-before: END searx uwsgi-appini ubuntu-20.04 + .. hotfix: a bug group-tab need this comment + .. group-tab:: Arch Linux .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini arch :end-before: END searx uwsgi-appini arch + .. hotfix: a bug group-tab need this comment + .. group-tab:: Fedora / RHEL .. kernel-include:: $DOCS_BUILD/includes/searx.rst - :code: ini :start-after: START searx uwsgi-appini fedora :end-before: END searx uwsgi-appini fedora - - diff --git a/docs/blog/index.rst b/docs/blog/index.rst index 5a8135fdc..04d95350d 100644 --- a/docs/blog/index.rst +++ b/docs/blog/index.rst @@ -6,6 +6,7 @@ Blog :maxdepth: 2 :caption: Contents + lxcdev-202006 python3 admin intro-offline diff --git a/docs/blog/lxcdev-202006.rst b/docs/blog/lxcdev-202006.rst new file mode 100644 index 000000000..f9ca3c2cc --- /dev/null +++ b/docs/blog/lxcdev-202006.rst @@ -0,0 +1,413 @@ +.. _blog-lxcdev-202006: + +======================================= +Developing in Linux containers [202006] +======================================= + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers / readers + should have a serious meaning about: *distributed*, *merge* and *linux + container*. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog +post I will show you, how you can make use of them in *distributed and +heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 +abstract`). + +Motivation +========== + +Normally in our development cycle, we edit the sources and run some test and/or +builds by using ``make`` before we commit. This cycle is simple and perfect but +might fail in some aspects we should not overlook. + + The environment in which we run all our development processes matters! + +The :ref:`makefile` and the :ref:`make pyenv` encapsulate a lot for us, but they +do not have access to all prerequisites. For example, there may have +dependencies on packages that are installed on the developer's desktop, but +usually are not preinstalled on a server or client system. Another examples +are; settings have been made to the software on the developer's host that would +never be set on a *production* system. + +*Linux Containers* (LXC_) are isolate environments and not to mix up on +developer's all the prerequisites of all the projects he contribute to, is +always a good choice. + +The scripts from PR :PR:`1803` can divide in those to install and maintain +software: + +- :ref:`searx.sh` +- :ref:`filtron.sh` +- :ref:`morty.sh` + +and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or +even development tasks over a stack of containers, what we call: *Searx's lxc +suite*. + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ cd ~/Downloads + $ git clone https://github.com/asciimoo/searx.git + $ cd searx + +The :ref:`lxc-searx.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post +we exercise on a archlinux_ image. The container of this image is named +``searx-archlinux``. Lets build the container, but be sure that this container +does not already exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh remove searx-archlinux + $ sudo -H ./utils/lxc.sh build searx-archlinux + +.. sidebar:: The ``searx-archlinux`` container + + is the base of all our exercises here. + +In this container we install all services :ref:`including searx, morty & filtron +<lxc.sh install suite>` in once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To proxy HTTP from filtron and morty in the container to the outside of the +container, install nginx into the container. Once for the bot blocker filtron: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/filtron.sh nginx install + ... + INFO: got 429 from http://10.174.184.156/searx + +and once for the content sanitizer (content proxy morty): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/morty.sh nginx install + ... + INFO: got 200 from http://10.174.184.156/morty/ + +.. sidebar:: Fully functional searx suite + + From here on you have a fully functional searx suite running with bot blocker + (filtron) and Web content sanitizer (content proxy morty) needed for a + *privacy protecting* search engine. + +On your system, the IP of your ``searx-archlinux`` container differs from +http://10.174.184.156/searx, just open the URL reported in your installation +protocol in your WEB browser from the desktop to test the instance from outside +of the container. + +In such a searx suite admins can maintain and access the debug log of the +different services quite easy. + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container using ``./utils/lxc.sh cmd +searx-archlinux``: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash + INFO: [searx-archlinux] bash + [root@searx-archlinux searx]# pwd + /share/searx + +The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in +the searx-container. To debug the running searx instance use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh inspect service + ... + use [CTRL-C] to stop monitoring the log + ... + +Back in the browser on your desktop open the service http://10.174.184.156/searx +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in searx's settings file and restarts the searx uwsgi application. To +debug services from filtron and morty analogous use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/filtron.sh inspect service + $ ./utils/morty.sh inspect service + +Another point we have to notice is that each service (:ref:`searx <searx.sh>`, +:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated +system user account with the same name (compare :ref:`create searx user`). To +get a shell from theses accounts, simply call one of the scripts: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh shell + $ ./utils/filtron.sh shell + $ ./utils/morty.sh shell + +To get in touch, open a shell from the service user (searx@searx-archlinux): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh shell + // exit with [CTRL-D] + (searx-pyenv) [searx@searx-archlinux ~]$ ... + +The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system +user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* +``(searx-pyenv)`` environment is activated. + +.. tabs:: + + .. group-tab:: searx@searx-archlinux + + .. code:: sh + + (searx-pyenv) [searx@searx-archlinux ~]$ pwd + /usr/local/searx + + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional searx suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- searx setup in: ``/etc/searx/settings.yml`` +- searx user's home: ``/usr/local/searx`` +- virtualenv in: ``/usr/local/searx/searx-pyenv`` +- searx software in: ``/usr/local/searx/searx-src`` + +The searx software is a clone of the ``GIT_URL`` (see :ref:`makefile setup`) and +the working tree is checked out from the ``GIT_BRANCH``. With the use of the +:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application +<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare +:ref:`service architectures on distributions <uwsgi configuration>`). + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl stop uwsgi@searx + +With the command above, we stopped the searx uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux dsitros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searx/searx-src/searx + virtualenv = /usr/local/searx/searx-pyenv + pythonpath = /usr/local/searx/searx-src + +If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that +each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the +searx installation into a developer one, we simple have to create a smylink to +the **transparent** reposetory from the desktop. Now lets replace the +repository at ``searx-src`` in the container with the working tree from outside +of the container: + +.. tabs:: + + .. group-tab:: container becomes a developer suite + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ln -s /share/searx/ /usr/local/searx/searx-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the searx service (in the +conatiner): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl restart uwsgi@searx + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +.. tabs:: + + .. group-tab:: desktop + + To *inspect* the searx instance (already described above): + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh inspect service + + Run :ref:`makefile`, e.g. to test inside the container: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make test + + To install all prerequisites needed for a :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh install buildhost + + To build the docs on a buildhost :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make docs + +.. _blog-lxcdev-202006 abstract: + +Abstract +======== + +We build up a fully functional searx suite in a archlinux container: + +.. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To access HTTP from the desktop we installed nginx for the services inside the +conatiner: + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ ./utils/filtron.sh nginx install + $ ./utils/morty.sh nginx install + +To wrap the suite into a developer one, we created a symbolic link to the +repository which is shared **transparent** from the desktop's file system into +the container : + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + $ ln -s /share/searx/ /usr/local/searx/searx-src + $ systemctl restart uwsgi@searx + +To get remarks from the suite of the archlinux container we can use: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh show suite searx-archlinux + ... + [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx + [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ + [searx-archlinux] INFO: (eth0) docs-live: http://10.174.184.156:8080/ + [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] + ... + diff --git a/docs/blog/private-engines.rst b/docs/blog/private-engines.rst index c26b3ed1c..796f0fc69 100644 --- a/docs/blog/private-engines.rst +++ b/docs/blog/private-engines.rst @@ -13,7 +13,7 @@ Private engines To solve this issue private engines were introduced in :pull:`1823`. A new option was added to engines named `tokens`. It expects a list of strings. If the user making a request presents one of the tokens -of an engine, he/she is able to access information about the engine +of an engine, they can access information about the engine and make search requests. Example configuration to restrict access to the Arch Linux Wiki engine: diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 922548ffb..960d2fb37 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -81,7 +81,7 @@ Parameters Theme of instance. Please note, available themes depend on an instance. It is possible that an - instance administrator deleted, created or renamed themes on his/her instance. + instance administrator deleted, created or renamed themes on their instance. See the available options in the preferences page of the instance. ``oscar-style`` : default ``logicodev`` @@ -91,7 +91,7 @@ Parameters ``oscar``. Please note, available styles depend on an instance. It is possible that an - instance administrator deleted, created or renamed styles on his/her + instance administrator deleted, created or renamed styles on their instance. See the available options in the preferences page of the instance. ``enabled_plugins`` : optional diff --git a/docs/user/own-instance.rst b/docs/user/own-instance.rst index 4876abe99..af415b61d 100644 --- a/docs/user/own-instance.rst +++ b/docs/user/own-instance.rst @@ -44,9 +44,9 @@ hidden from visited result pages. What are the consequences of using public instances? ---------------------------------------------------- -If someone uses a public instance, he/she has to trust the administrator of that +If someone uses a public instance, they have to trust the administrator of that instance. This means that the user of the public instance does not know whether -his/her requests are logged, aggregated and sent or sold to a third party. +their requests are logged, aggregated and sent or sold to a third party. Also, public instances without proper protection are more vulnerable to abusing the search service, In this case the external service in exchange returns diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst index ae1412870..114d0911e 100644 --- a/docs/utils/lxc.sh.rst +++ b/docs/utils/lxc.sh.rst @@ -47,9 +47,9 @@ one**:: *Good to know ...* -Eeach container shares the root folder of the repository and the -command ``utils/lxc.sh cmd`` **handles relative path names transparent**, -compare output of:: +Each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output +of:: $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile ... @@ -66,6 +66,7 @@ If there comes the time you want to **get rid off all** the containers and $ sudo -H ./utils/lxc.sh remove $ sudo -H ./utils/lxc.sh remove images +.. _lxc.sh install suite: Install suite ============= diff --git a/searx/engines/duckduckgo.py b/searx/engines/duckduckgo.py index 0d2c0af2d..6e07b5021 100644 --- a/searx/engines/duckduckgo.py +++ b/searx/engines/duckduckgo.py @@ -50,6 +50,7 @@ result_xpath = '//div[@class="result results_links results_links_deep web-result url_xpath = './/a[@class="result__a"]/@href' title_xpath = './/a[@class="result__a"]' content_xpath = './/a[@class="result__snippet"]' +correction_xpath = '//div[@id="did_you_mean"]//a' # match query's language to a region code that duckduckgo will accept @@ -125,6 +126,11 @@ def response(resp): 'content': content, 'url': res_url}) + # parse correction + for correction in eval_xpath(doc, correction_xpath): + # append correction + results.append({'correction': extract_text(correction)}) + # return results return results diff --git a/searx/engines/yacy.py b/searx/engines/yacy.py index 25bc83687..f1d4c6abe 100644 --- a/searx/engines/yacy.py +++ b/searx/engines/yacy.py @@ -75,7 +75,7 @@ def response(resp): for result in search_results[0].get('items', []): # parse image results - if result.get('image'): + if result.get('image') and result.get('width') and result.get('height'): result_url = '' if 'url' in result: diff --git a/searx/templates/__common__/about.html b/searx/templates/__common__/about.html index d3e8d06a6..9741b5162 100644 --- a/searx/templates/__common__/about.html +++ b/searx/templates/__common__/about.html @@ -59,12 +59,16 @@ <h2 id='add to browser'>How to set as the default search engine?</h2> - <dt>Firefox</dt> + <p> + Searx supports <a href="https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md">OpenSearch</a>. + For more information on changing your default search engine, see your browser's documentation: + </p> - <dd> - <a href="#" onclick="window.external.AddSearchProvider(window.location.protocol + '//' + window.location.host + '{{ url_for('opensearch') }}');">Install</a> - searx as a search engine on any version of Firefox! (javascript required) - </dd> + <ul> + <li><a href="https://support.mozilla.org/en-US/kb/add-or-remove-search-engine-firefox">Firefox</a></li> + <li><a href="https://support.microsoft.com/en-us/help/4028574/microsoft-edge-change-the-default-search-engine" >Microsoft Egde</a></li> + <li>Chrome based browsers <a href="https://www.chromium.org/tab-to-search">only add websites that the user navigates to without a path.</a> + </ul> <h2>Where to find anonymous usage statistics of this instance ?</h2> diff --git a/searx/templates/oscar/base.html b/searx/templates/oscar/base.html index 2ccadd700..a3bfa5249 100644 --- a/searx/templates/oscar/base.html +++ b/searx/templates/oscar/base.html @@ -8,7 +8,7 @@ <meta http-equiv="X-UA-Compatible" content="IE=edge"> <meta name="generator" content="searx/{{ searx_version }}"> <meta name="referrer" content="no-referrer"> - <meta name="viewport" content="width=device-width, initial-scale=1 , maximum-scale=1.0, user-scalable=1" /> + <meta name="viewport" content="width=device-width, initial-scale=1 , maximum-scale=2.0, user-scalable=1" /> {% block meta %}{% endblock %} <script src="{{ url_for('js_translations') }}"></script> <title>{% block title %}{% endblock %}{{ instance_name }}</title> @@ -42,7 +42,7 @@ <style type="text/css"> .tab-content > .active_if_nojs, .active_if_nojs {display: block !important; visibility: visible !important;} .margin_top_if_nojs {margin-top: 20px;} - .hide_if_nojs {display: none !important;overflow:none !important;} + .hide_if_nojs {display: none !important;overflow: hidden !important;} .disabled_if_nojs {pointer-events: none; cursor: default; text-decoration: line-through;} </style> </noscript> diff --git a/searx/templates/oscar/macros.html b/searx/templates/oscar/macros.html index 809c9f595..58f966997 100644 --- a/searx/templates/oscar/macros.html +++ b/searx/templates/oscar/macros.html @@ -8,34 +8,34 @@ <img width="32" height="32" class="favicon" src="{{ url_for('static', filename='themes/oscar/img/icons/' + favicon + '.png') }}" alt="{{ favicon }}" /> {%- endmacro %} -{%- macro result_link(url, title, classes='') -%} -<a href="{{ url }}" {% if classes %}class="{{ classes }}" {% endif %}{% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %}>{{ title }}</a> +{%- macro result_link(url, title, classes='', id='') -%} +<a href="{{ url }}" {% if classes %}class="{{ classes }}" {% endif %}{% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %}{% if id %} aria-labelledby="result-{{id}}"{%endif%}>{{ title }}</a> {%- endmacro -%} <!-- Draw result header --> -{% macro result_header(result, favicons) -%} -<h4 class="result_header">{% if result.engine~".png" in favicons %}{{ draw_favicon(result.engine) }} {% endif %}{% if result.url %}{{ result_link(result.url, result.title|safe) }}{% else %}{{ result.title|safe}}{% endif %}</h4> +{% macro result_header(result, favicons, id) -%} +<h4 class="result_header" id="result-{{id}}">{% if result.engine~".png" in favicons %}{{ draw_favicon(result.engine) }} {% endif %}{% if result.url %}{{ result_link(result.url, result.title|safe, id=id) }}{% else %}{{ result.title|safe}}{% endif %}</h4> {%- endmacro %} <!-- Draw result sub header --> -{% macro result_sub_header(result) -%} +{% macro result_sub_header(result, id) -%} {% if result.publishedDate %}<time class="text-muted" datetime="{{ result.pubdate }}" >{{ result.publishedDate }}</time>{% endif %} - {% if result.magnetlink %}<small> • {{ result_link(result.magnetlink, icon('magnet') + _('magnet link'), "magnetlink") }}</small>{% endif %} - {% if result.torrentfile %}<small> • {{ result_link(result.torrentfile, icon('download-alt') + _('torrent file'), "torrentfile") }}</small>{% endif %} + {% if result.magnetlink %}<small> • {{ result_link(result.magnetlink, icon('magnet') + _('magnet link'), "magnetlink", id) }}</small>{% endif %} + {% if result.torrentfile %}<small> • {{ result_link(result.torrentfile, icon('download-alt') + _('torrent file'), "torrentfile", id) }}</small>{% endif %} {%- endmacro %} <!-- Draw result footer --> -{% macro result_footer(result) -%} +{% macro result_footer(result, id) -%} <div class="clearfix"></div>{{- "" -}} <div class="pull-right"> {%- for engine in result.engines -%} <span class="label label-default">{{ engine }}</span> {%- endfor -%} {%- if result.url -%} - <small>{{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info") }}</small> + <small>{{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info", id) }}</small> {%- endif -%} {%- if proxify -%} - <small>{{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info") }}</small> + <small>{{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info", id) }}</small> {%- endif -%} </div> {%- if result.pretty_url -%} @@ -44,16 +44,16 @@ {%- endmacro %} <!-- Draw result footer --> -{% macro result_footer_rtl(result) -%} +{% macro result_footer_rtl(result, id) -%} <div class="clearfix"></div>{{- "" -}} {% for engine in result.engines -%} <span class="label label-default">{{ engine }}</span> {%- endfor %} {%- if result.url -%} - <small>{{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info") }}</small> + <small>{{ result_link("https://web.archive.org/web/" + result.url, icon('link') + _('cached'), "text-info", id) }}</small> {%- endif -%} {% if proxify -%} - <small>{{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info") }}</small> + <small>{{ result_link(proxify(result.url), icon('sort') + _('proxied'), "text-info", id) }}</small> {%- endif %} {%- if result.pretty_url -%} <div class="external-link">{{ result.pretty_url }}</div> diff --git a/searx/templates/oscar/result_templates/code.html b/searx/templates/oscar/result_templates/code.html index a1c18a6b7..c7d93e2b3 100644 --- a/searx/templates/oscar/result_templates/code.html +++ b/searx/templates/oscar/result_templates/code.html @@ -1,11 +1,11 @@ -{% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon %} +{% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon with context%} {{ result_header(result, favicons) }} {{ result_sub_header(result) }} {% if result.content %}<p class="result-content">{{ result.content|safe }}</p>{% endif %} -{% if result.repository %}<p class="result-content">{{ icon('file') }} <a href="{{ result.repository }}" {% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %}>{{ result.repository }}</a></p>{% endif %} +{% if result.repository %}<p class="result-content">{{ icon('file') }} <a href="{{ result.repository }}" {% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %} aria-labelledby="result-{{loop.index}}">{{ result.repository }}</a></p>{% endif %} <div dir="ltr"> {{ result.codelines|code_highlighter(result.code_language)|safe }} diff --git a/searx/templates/oscar/result_templates/default.html b/searx/templates/oscar/result_templates/default.html index 57365f34e..bc71d190d 100644 --- a/searx/templates/oscar/result_templates/default.html +++ b/searx/templates/oscar/result_templates/default.html @@ -1,10 +1,10 @@ {% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon with context %} -{{- result_header(result, favicons) -}} -{{- result_sub_header(result) -}} +{{- result_header(result, favicons, loop.index) -}} +{{- result_sub_header(result, loop.index) -}} {%- if result.embedded -%} - <small> • <a class="text-info btn-collapse collapsed cursor-pointer media-loader disabled_if_nojs" data-toggle="collapse" data-target="#result-media-{{ index }}" data-btn-text-collapsed="{{ _('show media') }}" data-btn-text-not-collapsed="{{ _('hide media') }}">{{ icon('music') }} {{ _('show media') }}</a></small> + <small> • <a class="text-info btn-collapse collapsed cursor-pointer media-loader disabled_if_nojs" data-toggle="collapse" data-target="#result-media-{{ index }}" data-btn-text-collapsed="{{ _('show media') }}" data-btn-text-not-collapsed="{{ _('hide media') }}" aria-labelledby="result-{{loop.index}}">{{ icon('music') }} {{ _('show media') }}</a></small> {%- endif -%} {%- if result.embedded -%} @@ -25,7 +25,7 @@ {%- endif -%} {%- if rtl -%} -{{ result_footer_rtl(result) }} +{{ result_footer_rtl(result, loop.index) }} {%- else -%} -{{ result_footer(result) }} +{{ result_footer(result, loop.index) }} {%- endif -%} diff --git a/searx/templates/oscar/result_templates/images.html b/searx/templates/oscar/result_templates/images.html index d0a3b7b83..96cdd53bb 100644 --- a/searx/templates/oscar/result_templates/images.html +++ b/searx/templates/oscar/result_templates/images.html @@ -1,6 +1,6 @@ -{%- from 'oscar/macros.html' import draw_favicon -%} +{%- from 'oscar/macros.html' import draw_favicon with context -%} -<a href="{{ result.img_src }}" {% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %} data-toggle="modal" data-target="#modal-{{ index }}-{{pageno}}">{{- "" -}} +<a href="{{ result.img_src }}" {% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %} data-toggle="modal" data-target="#modal-{{ index }}-{{pageno}}" id="result-{{loop.index}}">{{- "" -}} <img src="{% if result.thumbnail_src %}{{ image_proxify(result.thumbnail_src) }}{% else %}{{ image_proxify(result.img_src) }}{% endif %}" alt="{{ result.title|striptags }}" title="{{ result.title|striptags }}" class="img-thumbnail">{{- "" -}} </a> <div class="modal fade" id="modal-{{ index }}-{{ pageno }}" tabindex="-1" role="dialog" aria-hidden="true">{{- "" -}} diff --git a/searx/templates/oscar/result_templates/map.html b/searx/templates/oscar/result_templates/map.html index 712375d7f..be34d65d0 100644 --- a/searx/templates/oscar/result_templates/map.html +++ b/searx/templates/oscar/result_templates/map.html @@ -1,7 +1,7 @@ {% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon %} -{{ result_header(result, favicons) }} -{{ result_sub_header(result) }} +{{ result_header(result, favicons, loop.index) }} +{{ result_sub_header(result, loop.index) }} {% if (result.latitude and result.longitude) or result.boundingbox %} <small> • <a class="text-info btn-collapse collapsed searx_init_map cursor-pointer disabled_if_nojs" data-toggle="collapse" data-target="#result-map-{{ index }}" data-leaflet-target="osm-map-{{ index }}" data-map-lon="{{ result.longitude }}" data-map-lat="{{ result.latitude }}" {% if result.boundingbox %}data-map-boundingbox='{{ result.boundingbox|tojson|safe }}'{% endif %} {% if result.geojson %}data-map-geojson='{{ result.geojson|tojson|safe }}'{% endif %} data-btn-text-collapsed="{{ _('show map') }}" data-btn-text-not-collapsed="{{ _('hide map') }}">{{ icon('globe') }} {{ _('show map') }}</a></small> @@ -66,7 +66,7 @@ {% endif %} {% if rtl %} -{{ result_footer_rtl(result) }} +{{ result_footer_rtl(result, loop.index) }} {% else %} -{{ result_footer(result) }} +{{ result_footer(result, loop.index) }} {% endif %} diff --git a/searx/templates/oscar/result_templates/torrent.html b/searx/templates/oscar/result_templates/torrent.html index 089367e36..0bba61417 100644 --- a/searx/templates/oscar/result_templates/torrent.html +++ b/searx/templates/oscar/result_templates/torrent.html @@ -1,7 +1,7 @@ {% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon %} -{{ result_header(result, favicons) }} -{{ result_sub_header(result) }} +{{ result_header(result, favicons, loop.index) }} +{{ result_sub_header(result, loop.index) }} {% if result.seed is defined %}<p class="result-content">{{ icon('transfer') }} {{ _('Seeder') }} <span class="badge">{{ result.seed }}</span> • {{ _('Leecher') }} <span class="badge">{{ result.leech }}</span>{% endif %} {% if result.filesize %}<br />{{ icon('floppy-disk') }} {{ _('Filesize') }} @@ -19,7 +19,7 @@ </p> {% if rtl %} -{{ result_footer_rtl(result) }} +{{ result_footer_rtl(result, loop.index) }} {% else %} -{{ result_footer(result) }} +{{ result_footer(result, loop.index) }} {% endif %} diff --git a/searx/templates/oscar/result_templates/videos.html b/searx/templates/oscar/result_templates/videos.html index 6d9101d2e..bf02a553a 100644 --- a/searx/templates/oscar/result_templates/videos.html +++ b/searx/templates/oscar/result_templates/videos.html @@ -1,10 +1,10 @@ {% from 'oscar/macros.html' import result_header, result_sub_header, result_footer, result_footer_rtl, icon %} -{{ result_header(result, favicons) }} -{{ result_sub_header(result) }} +{{ result_header(result, favicons, loop.index) }} +{{ result_sub_header(result, loop.index) }} {% if result.embedded %} - <small> • <a class="text-info btn-collapse collapsed cursor-pointer media-loader disabled_if_nojs" data-toggle="collapse" data-target="#result-video-{{ index }}" data-btn-text-collapsed="{{ _('show video') }}" data-btn-text-not-collapsed="{{ _('hide video') }}">{{ icon('film') }} {{ _('show video') }}</a></small> + <small> • <a class="text-info btn-collapse collapsed cursor-pointer media-loader disabled_if_nojs" data-toggle="collapse" data-target="#result-video-{{ index }}" data-btn-text-collapsed="{{ _('show video') }}" data-btn-text-not-collapsed="{{ _('hide video') }}" aria-labelledby="result-{{loop.index}}">{{ icon('film') }} {{ _('show video') }}</a></small> {% endif %} {% if result.embedded %} @@ -15,7 +15,7 @@ <div class="container-fluid"> <div class="row"> - <a href="{{ result.url }}" {% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %}><img class="thumbnail col-xs-6 col-sm-4 col-md-4 result-content" src="{{ image_proxify(result.thumbnail) }}" /></a> + <a href="{{ result.url }}" {% if results_on_new_tab %}target="_blank" rel="noopener noreferrer"{% else %}rel="noreferrer"{% endif %}><img class="thumbnail col-xs-6 col-sm-4 col-md-4 result-content" src="{{ image_proxify(result.thumbnail) }}" aria-labelledby="result-{{loop.index}}" /></a> {% if result.author %}<p class="col-xs-12 col-sm-8 col-md-8 result-content"><b>{{ _('Author') }}</b>: {{ result.author }}</p>{% endif %} {% if result.length %}<p class="col-xs-12 col-sm-8 col-md-8 result-content"><b>{{ _('Length') }}</b>: {{ result.length }}</p>{% endif %} {% if result.content %}<p class="col-xs-12 col-sm-8 col-md-8 result-content">{{ result.content|safe }}</p>{% endif %} @@ -23,7 +23,7 @@ </div> {% if rtl %} -{{ result_footer_rtl(result) }} +{{ result_footer_rtl(result, loop.index) }} {% else %} -{{ result_footer(result) }} +{{ result_footer(result, loop.index) }} {% endif %} diff --git a/utils/searx.sh b/utils/searx.sh index 4009fddb0..886b38f8c 100755 --- a/utils/searx.sh +++ b/utils/searx.sh @@ -816,47 +816,61 @@ rst-doc() { case $DIST_ID-$DIST_VERS in ubuntu-*|debian-*) cat <<EOF -# init.d --> /usr/share/doc/uwsgi/README.Debian.gz -# For uWSGI debian uses the LSB init process, this might be changed -# one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 - -create ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} -enable: sudo -H ln -s ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} ${uWSGI_APPS_ENABLED}/ -start: sudo -H service uwsgi start ${SEARX_UWSGI_APP%.*} -restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} -stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} -disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + +.. code:: bash + + # init.d --> /usr/share/doc/uwsgi/README.Debian.gz + # For uWSGI debian uses the LSB init process, this might be changed + # one day, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=833067 + + create ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} + enable: sudo -H ln -s ${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP} ${uWSGI_APPS_ENABLED}/ + start: sudo -H service uwsgi start ${SEARX_UWSGI_APP%.*} + restart: sudo -H service uwsgi restart ${SEARX_UWSGI_APP%.*} + stop: sudo -H service uwsgi stop ${SEARX_UWSGI_APP%.*} + disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + EOF ;; arch-*) cat <<EOF -# systemd --> /usr/lib/systemd/system/uwsgi@.service -# For uWSGI archlinux uses systemd template units, see -# - http://0pointer.de/blog/projects/instances.html -# - https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd - -create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} -enable: sudo -H systemctl enable uwsgi@${SEARX_UWSGI_APP%.*} -start: sudo -H systemctl start uwsgi@${SEARX_UWSGI_APP%.*} -restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} -stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} -disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} + +.. code:: bash + + # systemd --> /usr/lib/systemd/system/uwsgi@.service + # For uWSGI archlinux uses systemd template units, see + # - http://0pointer.de/blog/projects/instances.html + # - https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd + + create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + enable: sudo -H systemctl enable uwsgi@${SEARX_UWSGI_APP%.*} + start: sudo -H systemctl start uwsgi@${SEARX_UWSGI_APP%.*} + restart: sudo -H systemctl restart uwsgi@${SEARX_UWSGI_APP%.*} + stop: sudo -H systemctl stop uwsgi@${SEARX_UWSGI_APP%.*} + disable: sudo -H systemctl disable uwsgi@${SEARX_UWSGI_APP%.*} + EOF ;; fedora-*) cat <<EOF -# systemd --> /usr/lib/systemd/system/uwsgi.service -# The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see -# - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html -create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} -restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} -disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} +.. code:: bash + + # systemd --> /usr/lib/systemd/system/uwsgi.service + # The unit file starts uWSGI in emperor mode (/etc/uwsgi.ini), see + # - https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html + + create: ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + restart: sudo -H touch ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + disable: sudo -H rm ${uWSGI_APPS_ENABLED}/${SEARX_UWSGI_APP} + EOF ;; esac echo -e ".. END searx uwsgi-description $DIST_NAME" echo -e "\n.. START searx uwsgi-appini $DIST_NAME" - eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" + echo ".. code:: bash" + echo + eval "echo \"$(< "${TEMPLATES}/${uWSGI_APPS_AVAILABLE}/${SEARX_UWSGI_APP}")\"" | prefix_stdout " " echo -e "\n.. END searx uwsgi-appini $DIST_NAME" ) |