diff options
Diffstat (limited to 'docs/dev')
| -rw-r--r-- | docs/dev/contribution_guide.rst | 147 | ||||
| -rw-r--r-- | docs/dev/engine_overview.rst | 265 | ||||
| -rw-r--r-- | docs/dev/index.rst | 13 | ||||
| -rw-r--r-- | docs/dev/plugins.rst | 48 | ||||
| -rw-r--r-- | docs/dev/quickstart.rst | 110 | ||||
| -rw-r--r-- | docs/dev/search_api.rst | 112 | ||||
| -rw-r--r-- | docs/dev/translation.rst | 71 |
7 files changed, 766 insertions, 0 deletions
diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst new file mode 100644 index 000000000..064f28e63 --- /dev/null +++ b/docs/dev/contribution_guide.rst @@ -0,0 +1,147 @@ +================= +How to contribute +================= + +Prime directives: Privacy, Hackability +====================================== + +Searx has two prime directives, **privacy-by-design and hackability** . The +hackability comes in three levels: + +- support of search engines +- plugins to alter search behaviour +- hacking searx itself + +Note the lack of "world domination" among the directives. Searx has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +"privacy" deserves a separate chapter, as it's quite uncommon unfortunately. + +Privacy-by-design +----------------- + +Searx was born out of the need for a **privacy-respecting** search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities. + +A few widely used features work differently or turned off by default or not +implemented at all **as a consequence of privacy-by-design**. + +If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained. + +Also, if you think that something works weird with searx, it's might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST`` +requests in various browsers). + +Remember the other prime directive of searx is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it. + + *Happy hacking.* + +Code +==== + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ + + +In order to submit a patch, please follow the steps below: + +- Follow coding conventions. + + - PEP8_ standards apply, except the convention of line length + - Maximum line length is 120 characters + +- Check if your code breaks existing tests. If so, update the tests or fix your + code. + +- If your code can be unit-tested, add unit tests. + +- Add yourself to the :origin:`AUTHORS.rst` file. + +- Create a pull request. + +For more help on getting started with searx development, see :ref:`devquickstart`. + + +Translation +=========== + +Translation currently takes place on :ref:`transifex <translation>`. + +.. caution:: + + Please, do not update translation files in the repo. + + +Documentation +============= + +.. _Sphinx: http://www.sphinx-doc.org +.. _reST: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html + +.. sidebar:: The reST sources + + has been moved from ``gh-branch`` into ``master`` (:origin:`docs`). + +The documentation is built using Sphinx_. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +Makefile our targets. + +Here is an example which makes a complete rebuild: + +.. code:: sh + + $ make docs-clean docs + ... + The HTML pages are in dist/docs. + + +live build +---------- + +.. sidebar:: docs-clean + + It is recommended to assert a complete rebuild before deploying (use + ``docs-clean``). + +Live build is like WYSIWYG, If you want to edit the documentation, its +recommended to use. The Makefile target ``docs-live`` builds the docs, opens URL +in your favorite browser and rebuilds every time a reST file has been changed. + +.. code:: sh + + $ make docs-live + ... + The HTML pages are in dist/docs. + ... Serving on http://0.0.0.0:8080 + ... Start watching changes + + + +deploy on github.io +------------------- + +To deploy documentation at :docs:`github.io <.>` use Makefile target +``gh-pages``, which will builds the documentation, clones searx into a sub +folder ``gh-pages``, cleans it, copies the doc build into and runs all the +needed git add, commit and push: + +.. code:: sh + + $ make docs-clean gh-pages + ... + SPHINX docs --> file://<...>/dist/docs + The HTML pages are in dist/docs. + ... + Cloning into 'gh-pages' ... + ... + cd gh-pages; git checkout gh-pages >/dev/null + Switched to a new branch 'gh-pages' + ... + doc available at --> https://asciimoo.github.io/searx diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst new file mode 100644 index 000000000..92405dc64 --- /dev/null +++ b/docs/dev/engine_overview.rst @@ -0,0 +1,265 @@ + +.. _engines-dev: + +=============== +Engine overview +=============== + +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine + +searx is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between searx and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. + +.. contents:: + :depth: 3 + :backlinks: entry + +general engine configuration +============================ + +It is required to tell searx the type of results the engine provides. The +arguments can be set in the engine file or in the settings file +(normally ``settings.yml``). The arguments in the settings file override +the ones in the engine file. + +It does not matter if an option is stored in the engine file or in the +settings. However, the standard way is the following: + + +engine file +----------- + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +categories list pages, in which the engine is working +paging boolean support multible pages +language_support boolean support language choosing +time_range_support boolean support search time range +offline boolean engine runs offline +======================= =========== =========================================== + + +settings.yml +------------ + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +name string name of search-engine +engine string name of searx-engine + (filename without ``.py``) +shortcut string shortcut of search-engine +timeout string specific timeout for search-engine +======================= =========== =========================================== + + +overrides +--------- + +A few of the options have default values in the engine, but are often +overwritten by the settings. If ``None`` is assigned to an option in the engine +file, it has to be redefined in the settings, otherwise searx will not start +with that engine. + +The naming of overrides is arbitrary. But the recommended overrides are the +following: + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +base_url string base-url, can be overwritten to use same + engine on other URL +number_of_results int maximum number of results per request +language string ISO code of language and country like en_US +api_key string api-key if required by engine +======================= =========== =========================================== + +example code +------------ + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + language_support = True + + +making a request +================ + +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. + +passed arguments +---------------- + +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. + +====================== ============ ======================================================================== +argument type default-value, information +====================== ============ ======================================================================== +url string ``''`` +method string ``'GET'`` +headers set ``{}`` +data set ``{}`` +cookies set ``{}`` +verify boolean ``True`` +headers.User-Agent string a random User-Agent +category string current category, like ``'general'`` +started datetime current date-time +pageno int current pagenumber +language string specific language code like ``'en_US'``, or ``'all'`` if unspecified +====================== ============ ======================================================================== + +parsed arguments +---------------- + +The function ``def request(query, params):`` always returns the ``params`` +variable. Inside searx, the following paramters can be used to specify a search +request: + +============ =========== ========================================================= +argument type information +============ =========== ========================================================= +url string requested url +method string HTTP request method +headers set HTTP header information +data set HTTP data information (parsed if ``method != 'GET'``) +cookies set HTTP cookies +verify boolean Performing SSL-Validity check +============ =========== ========================================================= + + +example code +------------ + +.. code:: python + + # search-url + base_url = 'https://example.com/' + search_string = 'search?{query}&page={page}' + + # do search-request + def request(query, params): + search_path = search_string.format( + query=urlencode({'q': query}), + page=params['pageno']) + + params['url'] = base_url + search_path + + return params + + +returned results +================ + +Searx is able to return results of different media-types. Currently the +following media-types are supported: + +- default_ +- images_ +- videos_ +- torrent_ +- map_ + +To set another media-type as default, the parameter ``template`` must be set to +the desired type. + +default +------- + +========================= ===================================================== +result-parameter information +========================= ===================================================== +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, time of publish +========================= ===================================================== + +images +------ + +To use this template, the parameter: + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``images.html`` +url string, url to the result site +title string, title of the result *(partly implemented)* +content *(partly implemented)* +publishedDate :py:class:`datetime.datetime`, + time of publish *(partly implemented)* +img\_src string, url to the result image +thumbnail\_src string, url to a small-preview image +========================= ===================================================== + +videos +------ + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``videos.html`` +url string, url of the result +title string, title of the result +content *(not implemented yet)* +publishedDate :py:class:`datetime.datetime`, time of publish +thumbnail string, url to a small-preview image +========================= ===================================================== + +torrent +------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``torrent.html`` +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* +seed int, number of seeder +leech int, number of leecher +filesize int, size of file in bytes +files int, number of files +magnetlink string, magnetlink_ of the result +torrentfile string, torrentfile of the result +========================= ===================================================== + + +map +--- + +========================= ===================================================== +result-parameter information +========================= ===================================================== +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, time of publish +latitude latitude of result (in decimal format) +longitude longitude of result (in decimal format) +boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) +geojson geojson of result (http://geojson.org) +osm.type type of osm-object (if OSM-Result) +osm.id id of osm-object (if OSM-Result) +address.name name of object +address.road street name of object +address.house_number house number of object +address.locality city, place of object +address.postcode postcode of object +address.country country of object +========================= ===================================================== diff --git a/docs/dev/index.rst b/docs/dev/index.rst new file mode 100644 index 000000000..8e18066ca --- /dev/null +++ b/docs/dev/index.rst @@ -0,0 +1,13 @@ +======================= +Developer documentation +======================= + +.. toctree:: + :maxdepth: 1 + + quickstart + contribution_guide + engine_overview + search_api + plugins + translation diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst new file mode 100644 index 000000000..e97bbeb4a --- /dev/null +++ b/docs/dev/plugins.rst @@ -0,0 +1,48 @@ +======= +Plugins +======= + +Plugins can extend or replace functionality of various components of searx. + +Example plugin +============== + +.. code:: python + + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disabled by default + + js_dependencies = tuple() # optional, list of static js files + css_dependencies = tuple() # optional, list of static css files + + + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, ctx): + ctx['search'].suggestions.add('example') + return True + +Plugin entry points +=================== + +Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. + +Pre search hook +--------------- + +Runs BEFORE the search request. Function to implement: ``pre_search`` + +Post search hook +---------------- + +Runs AFTER the search request. Function to implement: ``post_search`` + +Result hook +----------- + +Runs when a new result is added to the result list. Function to implement: +``on_result`` diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst new file mode 100644 index 000000000..a4a37a266 --- /dev/null +++ b/docs/dev/quickstart.rst @@ -0,0 +1,110 @@ +.. _devquickstart: + +====================== +Development Quickstart +====================== + +This quickstart guide gets your environment set up with searx. Furthermore, it +gives a short introduction to the ``manage.sh`` script. + +How to setup your development environment +========================================= + +First, clone the source code of searx to the desired folder. In this case the +source is cloned to ``~/myprojects/searx``. Then create and activate the +searx-ve virtualenv and install the required packages using manage.sh. + +.. code:: sh + + cd ~/myprojects + git clone https://github.com/asciimoo/searx.git + cd searx + virtualenv searx-ve + . ./searx-ve/bin/activate + ./manage.sh update_dev_packages + + +How to run tests +================ + +Tests can be run using the ``manage.sh`` script. Following tests and checks are +available: + +- Unit tests +- Selenium tests +- PEP8 validation +- Unit test coverage check + +For example unit tests are run with the command below: + +.. code:: sh + + ./manage.sh unit_tests + +For further test options, please consult the help of the ``manage.sh`` script. + + +How to compile styles and javascript +==================================== + +.. _less: http://lesscss.org/ +.. _NodeJS: https://nodejs.org + +How to build styles +------------------- + +Less_ is required to build the styles of searx. Less_ can be installed using +either NodeJS_ or Apt. + +.. code:: sh + + sudo -H apt-get install nodejs + sudo -H npm install -g less + +OR + +.. code:: sh + + sudo -H apt-get install node-less + +After satisfying the requirements styles can be build using ``manage.sh`` + +.. code:: sh + + ./manage.sh styles + + +How to build the source of the oscar theme +========================================== + +.. _grunt: https://gruntjs.com/ + +Grunt_ must be installed in order to build the javascript sources. It depends on +NodeJS, so first Node has to be installed. + +.. code:: sh + + sudo -H apt-get install nodejs + sudo -H npm install -g grunt-cli + +After installing grunt, the files can be built using the following command: + +.. code:: sh + + ./manage.sh grunt_build + + +Tips for debugging/development +============================== + +Turn on debug logging + Whether you are working on a new engine or trying to eliminate a bug, it is + always a good idea to turn on debug logging. When debug logging is enabled a + stack trace appears, instead of the cryptic ``Internal Server Error`` + message. It can be turned on by setting ``debug: False`` to ``debug: True`` in + :origin:`settings.yml <searx/settings.yml>`. + +Run ``./manage.sh tests`` before creating a PR. + Failing build on Travis is common because of PEP8 checks. So a new commit + must be created containing these format fixes. This phase can be skipped if + ``./manage.sh tests`` is run locally before creating a PR. diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst new file mode 100644 index 000000000..158cab7c5 --- /dev/null +++ b/docs/dev/search_api.rst @@ -0,0 +1,112 @@ +========== +Search API +========== + +The search supports both ``GET`` and ``POST``. + +Furthermore, two enpoints ``/`` and ``/search`` are available for querying. + + +``GET /`` + +``GET /search`` + +Parameters +========== + +``q`` : required + The search query. This string is passed to external search services. Thus, + searx supports syntax of each search service. For example, ``site:github.com + searx`` is a valid query for Google. However, if simply the query above is + passed to any search engine which does not filter its results based on this + syntax, you might not get the results you wanted. + + See more at :ref:`search-syntax` + +``categories`` : optional + Comma separated list, specifies the active search categories + +``engines``: optional + Comma separated list, specifies the active search engines. + +``lang``: default ``all`` + Code of the language. + +``pageno``: default ``1`` + Search page number. + +``time_range``: optional + [ ``day``, ``month``, ``year`` ] + + Time range of search for engines which support it. See if an engine supports + time range search in the preferences page of an instance. + +``format``: optional + [ ``json``, ``csv``, ``rss`` ] + + Output format of results. + +``results_on_new_tab``: default ``0`` + [ ``0``, ``1`` ] + + Open search results on new tab. + +``image_proxy``: default ``False`` + [ ``True``, ``False`` ] + + Proxy image results through searx. + +``autocomplete``: default *empty* + [ ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` ] + + Service which completes words as you type. + +``safesearch``: default ``None`` + [ ``0``, ``1``, ``None`` ] + + Filter search results of engines which support safe search. See if an engine + supports safe search in the preferences page of an instance. + +``theme``: default ``oscar`` + [ ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` ] + + Theme of instance. + + Please note, available themes depend on an instance. It is possible that an + instance administrator deleted, created or renamed themes on his/her instance. + See the available options in the preferences page of the instance. + +``oscar-style``: default ``logicodev`` + [ ``pointhi``, ``logicodev`` ] + + Style of Oscar theme. It is only parsed if the theme of an instance is + ``oscar``. + + Please note, available styles depend on an instance. It is possible that an + instance administrator deleted, created or renamed styles on his/her + instance. See the available options in the preferences page of the instance. + +``enabled_plugins``: optional + List of enabled plugins. + + :default: ``HTTPS_rewrite``, ``Self_Informations``, + ``Search_on_category_select``, ``Tracker_URL_remover`` + + :values: [ ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, + ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, + ``Search_on_category_select`` ] + +``disabled_plugins``: optional + List of disabled plugins. + + :default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys`` + :values: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, + ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, + ``Search_on_category_select`` + +``enabled_engines``: optional : *all* :origin:`engines <searx/engines>` + List of enabled engines. + +``disabled_engines``: optional : *all* :origin:`engines <searx/engines>` + List of disabled engines. + diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst new file mode 100644 index 000000000..86c4c843b --- /dev/null +++ b/docs/dev/translation.rst @@ -0,0 +1,71 @@ +.. _translation: + +=========== +Translation +=========== + +.. _searx@transifex: https://www.transifex.com/asciimoo/searx/ + +Translation currently takes place on `searx@transifex`_ + +Requirements +============ + +* Transifex account +* Installed CLI tool of Transifex + +Init Transifex project +====================== + +After installing ``transifex`` using pip, run the following command to +initialize the project. + +.. code:: sh + + tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ + + +After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it +into the configuration file. + +Create a configuration file for ``tx`` named ``$HOME/.tx/config``. + +.. code:: ini + + [main] + host = https://www.transifex.com + [searx.messagespo] + file_filter = searx/translations/<lang>/LC_MESSAGES/messages.po + source_file = messages.pot + source_lang = en + type = PO + + +Then run ``tx set``: + +.. code:: shell + + tx set --auto-local -r searx.messagespo 'searx/translations/<lang>/LC_MESSAGES/messages.po' \ + --source-lang en --type PO --source-file messages.pot --execute + + +Update translations +=================== + +To retrieve the latest translations, pull it from Transifex. + +.. code:: sh + + tx pull -a + +Then check the new languages. If strings translated are not enough, delete those +folders, because those should not be compiled. Call the command below to compile +the ``.po`` files. + +.. code:: shell + + pybabel compile -d searx/translations + + +After the compilation is finished commit the ``.po`` and ``.mo`` files and +create a PR. |