summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
Diffstat
-rw-r--r--.config.sh1
-rw-r--r--CONTRIBUTING.md4
-rwxr-xr-xdockerfiles/docker-entrypoint.sh2
-rw-r--r--docs/admin/buildhosts.rst2
-rw-r--r--docs/admin/installation-nginx.rst2
-rw-r--r--docs/admin/installation-searx.rst29
-rw-r--r--docs/admin/settings.rst187
-rw-r--r--docs/build-templates/searx.rst20
-rw-r--r--docs/conf.py6
-rw-r--r--docs/dev/contribution_guide.rst4
-rw-r--r--docs/dev/engine_overview.rst2
-rw-r--r--docs/dev/quickstart.rst2
-rw-r--r--docs/dev/reST.rst24
-rw-r--r--searx/autocomplete.py5
-rw-r--r--searx/engines/duckduckgo_definitions.py4
-rw-r--r--searx/engines/filecrop.py85
-rw-r--r--searx/engines/wikipedia.py2
-rw-r--r--searx/engines/www1x.py24
-rw-r--r--searx/engines/youtube_noapi.py2
-rw-r--r--searx/settings.yml1
-rw-r--r--utils/makefile.python2
-rwxr-xr-xutils/searx.sh13
-rw-r--r--utils/templates/etc/searx/use_default_settings.yml22
23 files changed, 219 insertions, 226 deletions
diff --git a/.config.sh b/.config.sh
index 4eff5f4c6..f9bac7383 100644
--- a/.config.sh
+++ b/.config.sh
@@ -26,6 +26,7 @@ fi
# ---------
# SEARX_INTERNAL_URL="127.0.0.1:8888"
+# SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"
# Only change, if you maintain a searx brand in your searx fork.
# GIT_BRANCH="${GIT_BRANCH:-master}"
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 6850ab405..300349f3b 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -2,8 +2,8 @@
## Resources in the documentation
-* [Development quickstart](http://searx.github.io/searx/dev/contribution_guide.html)
-* [Contribution guide](http://searx.github.io/searx/dev/contribution_guide.html)
+* [Development quickstart](https://searx.github.io/searx/dev/contribution_guide.html)
+* [Contribution guide](https://searx.github.io/searx/dev/contribution_guide.html)
## Submitting PRs
diff --git a/dockerfiles/docker-entrypoint.sh b/dockerfiles/docker-entrypoint.sh
index a8f18f05b..accc015f7 100755
--- a/dockerfiles/docker-entrypoint.sh
+++ b/dockerfiles/docker-entrypoint.sh
@@ -66,7 +66,7 @@ patch_searx_settings() {
CONF="$1"
# Make sure that there is trailing slash at the end of BASE_URL
- # see http://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion
+ # see https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion
export BASE_URL="${BASE_URL%/}/"
# update settings.yml
diff --git a/docs/admin/buildhosts.rst b/docs/admin/buildhosts.rst
index a727d25b9..1f6eb472e 100644
--- a/docs/admin/buildhosts.rst
+++ b/docs/admin/buildhosts.rst
@@ -67,7 +67,7 @@ to ``imgmath``:
If your docs build (``make docs``) shows warnings like this::
WARNING: dot(1) not found, for better output quality install \
- graphviz from http://www.graphviz.org
+ graphviz from https://www.graphviz.org
..
WARNING: LaTeX command 'latex' cannot be run (needed for math \
display), check the imgmath_latex setting
diff --git a/docs/admin/installation-nginx.rst b/docs/admin/installation-nginx.rst
index 65fd73573..589c40ada 100644
--- a/docs/admin/installation-nginx.rst
+++ b/docs/admin/installation-nginx.rst
@@ -9,7 +9,7 @@ Install with nginx
.. _nginx server configuration:
https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers
.. _nginx beginners guide:
- http://nginx.org/en/docs/beginners_guide.html
+ https://nginx.org/en/docs/beginners_guide.html
.. _Getting Started wiki:
https://www.nginx.com/resources/wiki/start/
.. _uWSGI support from nginx:
diff --git a/docs/admin/installation-searx.rst b/docs/admin/installation-searx.rst
index a368bfe8c..3f8904a1d 100644
--- a/docs/admin/installation-searx.rst
+++ b/docs/admin/installation-searx.rst
@@ -64,17 +64,38 @@ from the login (*~/.profile*):
Open a second terminal for the configuration tasks and left the ``(searx)$``
terminal open for the tasks below.
+
+.. _use_default_settings.yml:
+
Configuration
-==============
+=============
+
+To create a initial ``/etc/searx/settings.yml`` you can start with a copy of the
+file :origin:`utils/templates/etc/searx/use_default_settings.yml`. This setup
+:option:ref:`use default settings <settings use_default_settings>` from
+:origin:`searx/settings.yml` and is recommended since :pull:`2291` is merged.
-Create a copy of the :origin:`searx/settings.yml` configuration file in system's
-*/etc* folder. Configure like shown below -- replace ``searx@\$(uname -n)`` with
-a name of your choice -- *and/or* edit ``/etc/searx/settings.yml`` if necessary.
+For minimal Setup, configure like shown below – replace ``searx@\$(uname -n)``
+with a name of your choice, set ``ultrasecretkey`` -- *and/or* edit
+``/etc/searx/settings.yml`` to your needs.
.. kernel-include:: $DOCS_BUILD/includes/searx.rst
:start-after: START searx config
:end-before: END searx config
+.. tabs::
+
+ .. group-tab:: Use default settings
+
+ .. literalinclude:: ../../utils/templates/etc/searx/use_default_settings.yml
+ :language: yaml
+
+ .. group-tab:: searx/settings.yml
+
+ .. literalinclude:: ../../searx/settings.yml
+ :language: yaml
+
+
Check
=====
diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst
index 532b99752..985c16f85 100644
--- a/docs/admin/settings.rst
+++ b/docs/admin/settings.rst
@@ -9,6 +9,7 @@ file.
.. sidebar:: Further reading ..
+ - :ref:`use_default_settings.yml`
- :ref:`search API`
.. contents:: Contents
@@ -16,92 +17,61 @@ file.
:local:
:backlinks: entry
-.. _settings global:
+.. _settings location:
-Global Settings
-===============
+settings.yml location
+=====================
-.. code:: yaml
+First, searx will try to load settings.yml from these locations:
- server:
- port : 8888
- secret_key : "ultrasecretkey" # change this!
- debug : False # debug mode, only for development
- request_timeout : 2.0 # seconds
- base_url : False # set custom base_url (or False)
- themes_path : "" # custom ui themes path
- default_theme : oscar # ui theme
- useragent_suffix : "" # suffix of searx_useragent, could contain
- # informations like admins email address
- image_proxy : False # proxying image results through searx
- default_locale : "" # default interface locale
+1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable.
+2. ``/etc/searx/settings.yml``
- outgoing: # communication with search engines
- request_timeout : 2.0 # default timeout in seconds, can be override by engine
- # max_request_timeout: 10.0 # the maximum timeout in seconds
- useragent_suffix : "" # suffix of searx_useragent, could contain informations like an email address to the administrator
- pool_connections : 100 # Number of different hosts
- pool_maxsize : 10 # Number of simultaneous requests by host
-
- #proxies:
- # http:
- # - http://proxy1:8080
- # - http://proxy2:8080
- # https:
- # - http://proxy1:8080
- # - http://proxy2:8080
- # - socks5://user:password@proxy3:1080
- # - socks5h://user:password@proxy4:1080
-
- #source_ips:
- # - 1.1.1.1
- # - 1.1.1.2
+If these files don't exist (or are empty or can't be read), searx uses the :origin:`searx/settings.yml` file.
- locales:
- en : English
- de : Deutsch
- he : Hebrew
- hu : Magyar
- fr : Français
- es : Español
- it : Italiano
- nl : Nederlands
- ja : 日本語 (Japanese)
- tr : Türkçe
- ru : Russian
- ro : Romanian
+.. _settings global:
-``port`` :
- Port number of the searx web application if you run it directly using ``python
- searx/webapp.py``. Doesn't apply to searx running on Apache or Nginx.
+Global Settings
+===============
-``secret_key`` :
- Used for cryptography purpose.
+.. code:: yaml
+
+ general:
+ debug : False # Debug mode, only for development
+ instance_name : "searx" # displayed name
``debug`` :
Allow a more detailed log if you run searx directly. Display *detailed* error
messages in the browser too, so this must be deactivated in production.
-``request_timeout`` :
- Global timeout of the requests made to others engines in seconds. A bigger
- timeout will allow to wait for answers from slow engines, but in consequence
- will slow searx reactivity (the result page may take the time specified in the
- timeout to load)
-
-``base_url`` :
- The base URL where searx is deployed. Used to create correct inbound links.
+.. code:: yaml
-``themes_path`` :
- Path to where the themes are located. If you didn't develop anything, leave it
- blank.
+ server:
+ port : 8888
+ bind_address : "127.0.0.1" # address to listen on
+ secret_key : "ultrasecretkey" # change this!
+ base_url : False # set custom base_url (or False)
+ image_proxy : False # proxying image results through searx
+ default_locale : "" # default interface locale
+ default_theme : oscar # ui theme
+ default_http_headers:
+ X-Content-Type-Options : nosniff
+ X-XSS-Protection : 1; mode=block
+ X-Download-Options : noopen
+ X-Robots-Tag : noindex, nofollow
+ Referrer-Policy : no-referrer
+
+``port`` & ``bind_address``:
+ Port number and *bind address* of the searx web application if you run it
+ directly using ``python searx/webapp.py``. Doesn't apply to searx running on
+ Apache or Nginx.
-``default_theme`` :
- Name of the theme you want to use by default on your searx instance.
+``secret_key`` :
+ Used for cryptography purpose.
-``useragent_suffix`` :
- Suffix to the user-agent searx uses to send requests to others engines. If an
- engine wish to block you, a contact info here may be useful to avoid that.
+``base_url`` :
+ The base URL where searx is deployed. Used to create correct inbound links.
``image_proxy`` :
Allow your instance of searx of being able to proxy images. Uses memory space.
@@ -112,7 +82,49 @@ Global Settings
specific instance of searx, a locale can be defined using an ISO language
code, like ``fr``, ``en``, ``de``.
-.. _requests proxies: http://requests.readthedocs.io/en/latest/user/advanced/#proxies
+``default_theme`` :
+ Name of the theme you want to use by default on your searx instance.
+
+.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
+
+``default_http_headers``:
+ Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__
+
+
+.. code:: yaml
+
+ outgoing: # communication with search engines
+ request_timeout : 2.0 # default timeout in seconds, can be override by engine
+ # max_request_timeout: 10.0 # the maximum timeout in seconds
+ useragent_suffix : "" # informations like an email address to the administrator
+ pool_connections : 100 # Number of different hosts
+ pool_maxsize : 10 # Number of simultaneous requests by host
+ # uncomment below section if you want to use a proxy
+ # proxies:
+ # http:
+ # - http://proxy1:8080
+ # - http://proxy2:8080
+ # https:
+ # - http://proxy1:8080
+ # - http://proxy2:8080
+ # uncomment below section only if you have more than one network interface
+ # which can be the source of outgoing search requests
+ # source_ips:
+ # - 1.1.1.1
+ # - 1.1.1.2
+
+
+``request_timeout`` :
+ Global timeout of the requests made to others engines in seconds. A bigger
+ timeout will allow to wait for answers from slow engines, but in consequence
+ will slow searx reactivity (the result page may take the time specified in the
+ timeout to load). Can be override by :ref:`settings engine`
+
+``useragent_suffix`` :
+ Suffix to the user-agent searx uses to send requests to others engines. If an
+ engine wish to block you, a contact info here may be useful to avoid that.
+
+.. _requests proxies: https://requests.readthedocs.io/en/latest/user/advanced/#proxies
.. _PySocks: https://pypi.org/project/PySocks/
``proxies`` :
@@ -120,10 +132,29 @@ Global Settings
If there are more than one proxy for one protocol (http, https),
requests to the engines are distributed in a round-robin fashion.
+ - Proxy: `see <https://2.python-requests.org/en/latest/user/advanced/#proxies>`__.
+ - SOCKS proxies are also supported: `see <https://2.python-requests.org/en/latest/user/advanced/#socks>`__
+
``source_ips`` :
If you use multiple network interfaces, define from which IP the requests must
be made. This parameter is ignored when ``proxies`` is set.
+.. code:: yaml
+
+ locales:
+ en : English
+ de : Deutsch
+ he : Hebrew
+ hu : Magyar
+ fr : Français
+ es : Español
+ it : Italiano
+ nl : Nederlands
+ ja : 日本語 (Japanese)
+ tr : Türkçe
+ ru : Russian
+ ro : Romanian
+
``locales`` :
Locales codes and their names. Available translations of searx interface.
@@ -208,19 +239,7 @@ Engine settings
engines, and so won't be described here.
-.. _settings location:
-
-settings.yml location
-=====================
-
-First, searx will try to load settings.yml from these locations:
-
-1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable.
-2. ``/etc/searx/settings.yml``
-
-If these files don't exist (or are empty or can't be read), searx uses the :origin:`searx/settings.yml` file.
-
-.. _ settings use_default_settings:
+.. _settings use_default_settings:
use_default_settings
====================
diff --git a/docs/build-templates/searx.rst b/docs/build-templates/searx.rst
index 080de293f..fe82ec3d0 100644
--- a/docs/build-templates/searx.rst
+++ b/docs/build-templates/searx.rst
@@ -128,12 +128,28 @@ ${fedora_build}
.. tabs::
- .. group-tab:: bash
+ .. group-tab:: Use default settings
+
+ .. code-block:: sh
+
+ $ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\"
+ $ sudo -H cp \"$SEARX_SRC/utils/templates/etc/searx/use_default_settings.yml\" \\
+ \"${SEARX_SETTINGS_PATH}\"
+
+ .. group-tab:: searx/settings.yml
.. code-block:: sh
$ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\"
- $ sudo -H cp \"$SEARX_SRC/searx/settings.yml\" \"${SEARX_SETTINGS_PATH}\"
+ $ sudo -H cp \"$SEARX_SRC/searx/settings.yml\" \\
+ \"${SEARX_SETTINGS_PATH}\"
+
+.. tabs::
+
+ .. group-tab:: minimal setup
+
+ .. code-block:: sh
+
$ sudo -H sed -i -e \"s/ultrasecretkey/\$(openssl rand -hex 16)/g\" \"$SEARX_SETTINGS_PATH\"
$ sudo -H sed -i -e \"s/{instance_name}/searx@\$(uname -n)/g\" \"$SEARX_SETTINGS_PATH\"
diff --git a/docs/conf.py b/docs/conf.py
index 4b348ae0e..d6fde9bec 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -49,11 +49,11 @@ extlinks['man'] = ('https://manpages.debian.org/jump?q=%s', '')
#extlinks['role'] = (
# 'https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-%s', '')
extlinks['duref'] = (
- 'http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '')
+ 'https://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '')
extlinks['durole'] = (
- 'http://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '')
+ 'https://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '')
extlinks['dudir'] = (
- 'http://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '')
+ 'https://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '')
extlinks['ctan'] = (
'https://ctan.org/pkg/%s', 'CTAN: ')
diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst
index 26f8d2bb7..90b22670c 100644
--- a/docs/dev/contribution_guide.rst
+++ b/docs/dev/contribution_guide.rst
@@ -117,8 +117,8 @@ Translation currently takes place on :ref:`transifex <translation>`.
Documentation
=============
-.. _Sphinx: http://www.sphinx-doc.org
-.. _reST: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+.. _Sphinx: https://www.sphinx-doc.org
+.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
.. sidebar:: The reST sources
diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst
index 0b5f9857f..99726a456 100644
--- a/docs/dev/engine_overview.rst
+++ b/docs/dev/engine_overview.rst
@@ -265,7 +265,7 @@ latitude latitude of result (in decimal format)
longitude longitude of result (in decimal format)
boundingbox boundingbox of result (array of 4. values
``[lat-min, lat-max, lon-min, lon-max]``)
-geojson geojson of result (http://geojson.org)
+geojson geojson of result (https://geojson.org/)
osm.type type of osm-object (if OSM-Result)
osm.id id of osm-object (if OSM-Result)
address.name name of object
diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst
index 3e1a5e344..14af03fa6 100644
--- a/docs/dev/quickstart.rst
+++ b/docs/dev/quickstart.rst
@@ -60,7 +60,7 @@ read :ref:`make test`.
How to compile styles and javascript
====================================
-.. _less: http://lesscss.org/
+.. _less: https://lesscss.org/
.. _NodeJS: https://nodejs.org
How to build styles
diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst
index 906a0e9af..963378748 100644
--- a/docs/dev/reST.rst
+++ b/docs/dev/reST.rst
@@ -1391,27 +1391,27 @@ The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac``
.. _readability: https://docs.python-guide.org/writing/style/
.. _Sphinx-Primer:
- http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
+ https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
.. _reST: https://docutils.sourceforge.io/rst.html
.. _Sphinx Roles:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html
-.. _Sphinx: http://www.sphinx-doc.org
-.. _`sphinx-doc FAQ`: http://www.sphinx-doc.org/en/stable/faq.html
+.. _Sphinx: https://www.sphinx-doc.org
+.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html
.. _Sphinx markup constructs:
- http://www.sphinx-doc.org/en/stable/markup/index.html
+ https://www.sphinx-doc.org/en/stable/markup/index.html
.. _`sphinx cross references`:
- http://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations
+ https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations
.. _sphinx.ext.extlinks:
https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
-.. _intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html
-.. _sphinx config: http://www.sphinx-doc.org/en/stable/config.html
-.. _Sphinx's autodoc: http://www.sphinx-doc.org/en/stable/ext/autodoc.html
+.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html
+.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html
+.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html
.. _Sphinx's Python domain:
- http://www.sphinx-doc.org/en/stable/domains.html#the-python-domain
+ https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain
.. _Sphinx's C domain:
- http://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs
+ https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs
.. _doctree:
- http://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases
+ https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases
.. _docutils: http://docutils.sourceforge.net/docs/index.html
.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html
.. _linuxdoc: https://return42.github.io/linuxdoc
@@ -1424,5 +1424,5 @@ The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac``
.. _ImageMagick: https://www.imagemagick.org
.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode
-.. _`Online Tables Generator`: http://www.tablesgenerator.com/text_tables
+.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables
.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html
diff --git a/searx/autocomplete.py b/searx/autocomplete.py
index 420b8a461..fbe634a5b 100644
--- a/searx/autocomplete.py
+++ b/searx/autocomplete.py
@@ -113,7 +113,7 @@ def searx_bang(full_query):
def dbpedia(query, lang):
# dbpedia autocompleter, no HTTPS
- autocomplete_url = 'http://lookup.dbpedia.org/api/search.asmx/KeywordSearch?'
+ autocomplete_url = 'https://lookup.dbpedia.org/api/search.asmx/KeywordSearch?'
response = get(autocomplete_url + urlencode(dict(QueryString=query)))
@@ -121,8 +121,7 @@ def dbpedia(query, lang):
if response.ok:
dom = etree.fromstring(response.content)
- results = dom.xpath('//a:Result/a:Label//text()',
- namespaces={'a': 'http://lookup.dbpedia.org/'})
+ results = dom.xpath('//Result/Label//text()')
return results
diff --git a/searx/engines/duckduckgo_definitions.py b/searx/engines/duckduckgo_definitions.py
index 5a7649173..1d1c84b4b 100644
--- a/searx/engines/duckduckgo_definitions.py
+++ b/searx/engines/duckduckgo_definitions.py
@@ -10,7 +10,7 @@ DuckDuckGo (definitions)
"""
import json
-from urllib.parse import urlencode
+from urllib.parse import urlencode, urlparse, urljoin
from lxml import html
from searx import logger
@@ -102,6 +102,8 @@ def response(resp):
# image
image = search_res.get('Image')
image = None if image == '' else image
+ if image is not None and urlparse(image).netloc == '':
+ image = urljoin('https://duckduckgo.com', image)
# urls
# Official website, Wikipedia page
diff --git a/searx/engines/filecrop.py b/searx/engines/filecrop.py
deleted file mode 100644
index 0331e7b19..000000000
--- a/searx/engines/filecrop.py
+++ /dev/null
@@ -1,85 +0,0 @@
-from html.parser import HTMLParser
-from urllib.parse import urlencode
-
-
-url = 'http://www.filecrop.com/'
-search_url = url + '/search.php?{query}&size_i=0&size_f=100000000&engine_r=1&engine_d=1&engine_e=1&engine_4=1&engine_m=1&pos={index}' # noqa
-
-paging = True
-
-
-class FilecropResultParser(HTMLParser): # pylint: disable=W0223 # (see https://bugs.python.org/issue31844)
-
- def __init__(self):
- HTMLParser.__init__(self)
- self.__start_processing = False
-
- self.results = []
- self.result = {}
-
- self.tr_counter = 0
- self.data_counter = 0
-
- def handle_starttag(self, tag, attrs):
-
- if tag == 'tr':
- if ('bgcolor', '#edeff5') in attrs or\
- ('bgcolor', '#ffffff') in attrs:
- self.__start_processing = True
-
- if not self.__start_processing:
- return
-
- if tag == 'label':
- self.result['title'] = [attr[1] for attr in attrs
- if attr[0] == 'title'][0]
- elif tag == 'a' and ('rel', 'nofollow') in attrs\
- and ('class', 'sourcelink') in attrs:
- if 'content' in self.result:
- self.result['content'] += [attr[1] for attr in attrs
- if attr[0] == 'title'][0]
- else:
- self.result['content'] = [attr[1] for attr in attrs
- if attr[0] == 'title'][0]
- self.result['content'] += ' '
- elif tag == 'a':
- self.result['url'] = url + [attr[1] for attr in attrs
- if attr[0] == 'href'][0]
-
- def handle_endtag(self, tag):
- if self.__start_processing is False:
- return
-
- if tag == 'tr':
- self.tr_counter += 1
-
- if self.tr_counter == 2:
- self.__start_processing = False
- self.tr_counter = 0
- self.data_counter = 0
- self.results.append(self.result)
- self.result = {}
-
- def handle_data(self, data):
- if not self.__start_processing:
- return
-
- if 'content' in self.result:
- self.result['content'] += data + ' '
- else:
- self.result['content'] = data + ' '
-
- self.data_counter += 1
-
-
-def request(query, params):
- index = 1 + (params['pageno'] - 1) * 30
- params['url'] = search_url.format(query=urlencode({'w': query}), index=index)
- return params
-
-
-def response(resp):
- parser = FilecropResultParser()
- parser.feed(resp.text)
-
- return parser.results
diff --git a/searx/engines/wikipedia.py b/searx/engines/wikipedia.py
index 9fce170eb..000e1af76 100644
--- a/searx/engines/wikipedia.py
+++ b/searx/engines/wikipedia.py
@@ -52,7 +52,7 @@ def response(resp):
api_result = loads(resp.text)
# skip disambiguation pages
- if api_result['type'] != 'standard':
+ if api_result.get('type') != 'standard':
return []
title = api_result['title']
diff --git a/searx/engines/www1x.py b/searx/engines/www1x.py
index 8d691c852..b8f111a50 100644
--- a/searx/engines/www1x.py
+++ b/searx/engines/www1x.py
@@ -7,12 +7,12 @@
@using-api no
@results HTML
@stable no (HTML can change)
- @parse url, title, thumbnail, img_src, content
+ @parse url, title, thumbnail
"""
-from lxml import html
+from lxml import html, etree
from urllib.parse import urlencode, urljoin
-from searx.utils import extract_text
+from searx.utils import extract_text, eval_xpath_list, eval_xpath_getindex
# engine dependent config
categories = ['images']
@@ -21,6 +21,7 @@ paging = False
# search-url
base_url = 'https://1x.com'
search_url = base_url + '/backend/search.php?{query}'
+gallery_url = 'https://gallery.1x.com/'
# do search-request
@@ -33,23 +34,18 @@ def request(query, params):
# get response from search-request
def response(resp):
results = []
-
- dom = html.fromstring(resp.text)
- for res in dom.xpath('//div[@class="List-item MainListing"]'):
- # processed start and end of link
- link = res.xpath('//a')[0]
-
+ xmldom = etree.fromstring(resp.content)
+ xmlsearchresult = eval_xpath_getindex(xmldom, '//searchresult', 0)
+ dom = html.fragment_fromstring(xmlsearchresult.text, create_parent='div')
+ for link in eval_xpath_list(dom, '/div/table/tr/td/div[2]//a'):
url = urljoin(base_url, link.attrib.get('href'))
title = extract_text(link)
-
- thumbnail_src = urljoin(base_url, res.xpath('.//img')[0].attrib['src'])
- # TODO: get image with higher resolution
- img_src = thumbnail_src
+ thumbnail_src = urljoin(gallery_url, eval_xpath_getindex(link, './/img', 0).attrib['src'])
# append result
results.append({'url': url,
'title': title,
- 'img_src': img_src,
+ 'img_src': thumbnail_src,
'content': '',
'thumbnail_src': thumbnail_src,
'template': 'images.html'})
diff --git a/searx/engines/youtube_noapi.py b/searx/engines/youtube_noapi.py
index 5f7d2ceab..36fc72e36 100644
--- a/searx/engines/youtube_noapi.py
+++ b/searx/engines/youtube_noapi.py
@@ -49,7 +49,7 @@ def response(resp):
results = []
results_data = resp.text[resp.text.find('ytInitialData'):]
- results_data = results_data[results_data.find('{'):results_data.find(';\n')]
+ results_data = results_data[results_data.find('{'):results_data.find(';</script>')]
results_json = loads(results_data) if results_data else {}
sections = results_json.get('contents', {})\
diff --git a/searx/settings.yml b/searx/settings.yml
index 432592fc2..412d0fd1f 100644
--- a/searx/settings.yml
+++ b/searx/settings.yml
@@ -289,6 +289,7 @@ engines:
- name : 1x
engine : www1x
shortcut : 1x
+ timeout : 3.0
disabled : True
- name : fdroid
diff --git a/utils/makefile.python b/utils/makefile.python
index 6c6696964..668b0894b 100644
--- a/utils/makefile.python
+++ b/utils/makefile.python
@@ -252,7 +252,7 @@ pyenv-python: pyenv-install
# PyPi is required and since uploads via setuptools is not recommended, we have
# to imstall / use twine ... its really a mess.
#
-# [1] http://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi
+# [1] https://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi
# [2] https://github.com/pypa/pip/pull/1519
# https://github.com/pypa/twine
diff --git a/utils/searx.sh b/utils/searx.sh
index 386b2861f..06b3c2dfc 100755
--- a/utils/searx.sh
+++ b/utils/searx.sh
@@ -36,6 +36,7 @@ GIT_BRANCH="${GIT_BRANCH:-master}"
SEARX_PYENV="${SERVICE_HOME}/searx-pyenv"
SEARX_SRC="${SERVICE_HOME}/searx-src"
SEARX_SETTINGS_PATH="/etc/searx/settings.yml"
+SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"
SEARX_UWSGI_APP="searx.ini"
# shellcheck disable=SC2034
SEARX_UWSGI_SOCKET="/run/uwsgi/app/searx/socket"
@@ -139,7 +140,7 @@ usage() {
cat <<EOF
usage::
$(basename "$0") shell
- $(basename "$0") install [all|user|searx-src|pyenv|uwsgi|packages|buildhost]
+ $(basename "$0") install [all|user|searx-src|pyenv|uwsgi|packages|settings|buildhost]
$(basename "$0") update [searx]
$(basename "$0") remove [all|user|pyenv|searx-src]
$(basename "$0") activate [service]
@@ -413,14 +414,14 @@ install_settings() {
if [[ ! -f ${SEARX_SETTINGS_PATH} ]]; then
info_msg "install settings ${REPO_ROOT}/searx/settings.yml"
info_msg " --> ${SEARX_SETTINGS_PATH}"
- cp "${REPO_ROOT}/searx/settings.yml" "${SEARX_SETTINGS_PATH}"
+ cp "${SEARX_SETTINGS_TEMPLATE}" "${SEARX_SETTINGS_PATH}"
configure_searx
return
fi
rst_para "Diff between origin's setting file (+) and current (-):"
- echo
- $DIFF_CMD "${SEARX_SETTINGS_PATH}" "${SEARX_SRC}/searx/settings.yml"
+ echo "${SEARX_SETTINGS_PATH}" "${SEARX_SETTINGS_TEMPLATE}"
+ $DIFF_CMD "${SEARX_SETTINGS_PATH}" "${SEARX_SETTINGS_TEMPLATE}"
local action
choose_one action "What should happen to the settings file? " \
@@ -434,7 +435,7 @@ install_settings() {
"use origin settings")
backup_file "${SEARX_SETTINGS_PATH}"
info_msg "install origin settings"
- cp "${SEARX_SRC}/searx/settings.yml" "${SEARX_SETTINGS_PATH}"
+ cp "${SEARX_SETTINGS_TEMPLATE}" "${SEARX_SETTINGS_PATH}"
;;
"start interactiv shell")
backup_file "${SEARX_SETTINGS_PATH}"
@@ -442,7 +443,7 @@ install_settings() {
sudo -H -i
rst_para 'Diff between new setting file (-) and current (+):'
echo
- $DIFF_CMD "${SEARX_SRC}/searx/settings.yml" "${SEARX_SETTINGS_PATH}"
+ $DIFF_CMD "${SEARX_SETTINGS_TEMPLATE}" "${SEARX_SETTINGS_PATH}"
wait_key
;;
esac
diff --git a/utils/templates/etc/searx/use_default_settings.yml b/utils/templates/etc/searx/use_default_settings.yml
new file mode 100644
index 000000000..e019a25bb
--- /dev/null
+++ b/utils/templates/etc/searx/use_default_settings.yml
@@ -0,0 +1,22 @@
+use_default_settings: True
+
+general:
+ debug : False # Debug mode, only for development
+ instance_name : "searx" # displayed name
+
+search:
+ safe_search : 0 # Filter results. 0: None, 1: Moderate, 2: Strict
+ autocomplete : "" # Existing autocomplete backends: "dbpedia", "duckduckgo", "google", "startpage", "swisscows", "qwant", "wikipedia" - leave blank to turn it off by default
+ default_lang : "" # Default search language - leave blank to detect from browser information or use codes from 'languages.py'
+
+server:
+ port : 8888
+ bind_address : "127.0.0.1" # address to listen on
+ secret_key : "ultrasecretkey" # change this!
+ base_url : False # Set custom base_url. Possible values: False or "https://your.custom.host/location/"
+ image_proxy : False # Proxying image results through searx
+
+# uncomment below section if you have running morty proxy
+#result_proxy:
+# url : http://127.0.0.1:3000/
+# key : !!binary "your_morty_proxy_key"
7apu2bq3rhoylwmucxizk54afw5jyda7pho4j5zdcqpwkufke7zdzwad.onion