Contributing to qutebrowser =========================== The Compiler :icons: :data-uri: :toc: I `<3` footnote:[Of course, that says `<3` in HTML.] contributors! This document contains guidelines for contributing to qutebrowser, as well as useful hints when doing so. If anything mentioned here would prevent you from contributing, please let me know, and contribute anyways! The guidelines are only meant to make life easier for me, but if you don't follow anything in here, I won't be mad at you. I will probably change it for you then, though. If you have any problems, I'm more than happy to help! You can get help in several ways: * Send a mail to the mailing list at mailto:qutebrowser@lists.qutebrowser.org[] (optionally https://lists.schokokeks.org/mailman/listinfo.cgi/qutebrowser[subscribe] first). * Join the IRC channel irc://irc.freenode.org/#qutebrowser[`#qutebrowser`] on http://freenode.net/[Freenode] (https://webchat.freenode.net/?channels=#qutebrowser[webchat]). Finding something to work on ---------------------------- Chances are you already know something to improve or add when you're reading this. It might be a good idea to ask on the mailing list or IRC channel to make sure nobody else started working on the same thing already. If you want to find something useful to do, check the https://github.com/The-Compiler/qutebrowser/issues[issue tracker]. Some pointers: * https://github.com/The-Compiler/qutebrowser/labels/easy[Issues which should be easy to solve] * https://github.com/The-Compiler/qutebrowser/labels/not%20code[Issues which require little/no coding] There are also some things to do if you don't want to write code: * Help the community, e.g. on the mailinglist and the IRC channel. * Improve the documentation. * Help on the website and graphics (logo, etc.). Using git --------- qutebrowser uses http://git-scm.com/[git] for its development. You can clone the repo like this: ---- git clone https://github.com/The-Compiler/qutebrowser.git ---- If you don't know git, a http://git-scm.com/[git cheatsheet] might come in handy. Of course, if using git is the issue which prevents you from contributing, feel free to send normal patches instead, e.g. generated via `diff -Nur`. Getting patches ~~~~~~~~~~~~~~~ The preferred way of submitting changes is to https://help.github.com/articles/fork-a-repo/[fork the repository] and to https://help.github.com/articles/creating-a-pull-request/[submit a pull request]. If you prefer to send a patch to the mailinglist, you can generate a patch based on your changes like this: ---- git format-patch origin/master <1> ---- <1> Replace `master` by the branch your work was based on, e.g. `origin/develop`. Useful utilities ---------------- Checkers ~~~~~~~~ qutebrowser uses http://tox.readthedocs.org/en/latest/[tox] to run its unittests and several linters/checkers. Currently, following tox environments are available: * Tests using https://www.pytest.org[pytest]: - `py34`: Run pytest for python-3.4. - `py35`: Run pytest for python-3.5. - `py34-cov`: Run pytest for python-3.4 with code coverage report. - `py35-cov`: Run pytest for python-3.5 with code coverage report. * `flake8`: Run https://pypi.python.org/pypi/flake8[flake8] checks: https://pypi.python.org/pypi/pyflakes[pyflakes], https://pypi.python.org/pypi/pep8[pep8], https://pypi.python.org/pypi/mccabe[mccabe] * `vulture`: Run https://pypi.python.org/pypi/vulture[vulture] to find unused code portions. * `pylint`: Run http://pylint.org/[pylint] static code analysis. * `pydocstyle`: Check https://www.python.org/dev/peps/pep-0257/[PEP257] compliance with https://github.com/PyCQA/pydocstyle[pydocstyle] * `pyroma`: Check packaging practices with https://pypi.python.org/pypi/pyroma/[pyroma] * `eslint`: Run http://eslint.org/[ESLint] javascript checker. * `check-manifest`: Check MANIFEST.in completeness with https://github.com/mgedmin/check-manifest[check-manifest] * `mkvenv`: Bootstrap a virtualenv for testing. * `misc`: Run `scripts/misc_checks.py` to check for: - untracked git files - VCS conflict markers - common spelling mistakes The default test suite is run with `tox`, the list of default environments is obtained with `tox -l` . Please make sure the checks run without any warnings on your new contributions. There's of course the possibility of false-positives, and the following techniques are useful to handle these: * Use `_foo` for unused parameters, with `foo` being a descriptive name. Using `_` is discouraged. * If you think you have a good reason to suppress a message, add the following comment: + ---- # pylint: disable=message-name ---- + Note you can add this per line, per function/class, or per file. Please use the smallest scope which makes sense. Most of the time, this will be line scope. + * If you really think a check shouldn't be done globally as it yields a lot of false-positives, let me know! I'm still tweaking the parameters. Running Specific Tests ~~~~~~~~~~~~~~~~~~~~~~ While you are developing you often don't want to run the full test suite each time. Specific test environments can be run with `tox -e `. Additional parameters can be passed to the test scripts by separating them from `tox` arguments with `--`. Examples: ---- # run only pytest tests which failed in last run: tox -e py35 -- --lf # run only the integration feature tests: tox -e py35 -- tests/integration/features # run everything with undo in the generated name, based on the scenario text tox -e py35 -- tests/integration/features/test_tabs.py -k undo # run coverage test for specific file (updates htmlcov/index.html) tox -e py35-cov -- tests/unit/browser/test_webelem.py ---- Profiling ~~~~~~~~~ In the _scripts/_ subfolder there's a `run_profile.py` which profiles the code and shows a graphical representation of what takes how much time. It needs https://pypi.python.org/pypi/pyprof2calltree/[pyprof2calltree] and http://kcachegrind.sourceforge.net/html/Home.html[KCacheGrind]. It uses the built-in Python https://docs.python.org/3.4/library/profile.html[cProfile] module. Debugging ~~~~~~~~~ In the `qutebrowser.utils.debug` module there are some useful functions for debugging. When starting qutebrowser with the `--debug` flag you also get useful debug logs. You can add +--logfilter _category[,category,...]_+ to restrict logging to the given categories. With `--debug` there are also some additional +debug-_*_+ commands available, for example `:debug-all-objects` and `:debug-all-widgets` which print a list of all Qt objects/widgets to the debug log -- this is very useful for finding memory leaks. Useful websites ~~~~~~~~~~~~~~~ Some resources which might be handy: * http://doc.qt.io/qt-5/classes.html[The Qt5 reference] * https://docs.python.org/3/library/index.html[The Python reference] * http://httpbin.org/[httpbin, a test service for HTTP requests/responses] * http://requestb.in/[RequestBin, a service to inspect HTTP requests] Documentation of used Python libraries: * http://jinja.pocoo.org/docs/dev/[jinja2] * http://pygments.org/docs/[pygments] * http://fdik.org/pyPEG/index.html[pyPEG2] * http://pythonhosted.org/setuptools/[setuptools] * http://cx-freeze.readthedocs.org/en/latest/overview.html[cx_Freeze] * https://pypi.python.org/pypi/colorama[colorama] * https://pypi.python.org/pypi/colorlog[colorlog] Related RFCs and standards: HTTP ^^^^ * https://tools.ietf.org/html/rfc2616[RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1] (http://www.rfc-editor.org/errata_search.php?rfc=2616[Errata]) * https://tools.ietf.org/html/rfc7230[RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing] (http://www.rfc-editor.org/errata_search.php?rfc=7230[Errata]) * https://tools.ietf.org/html/rfc7231[RFC 7231 - Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content] (http://www.rfc-editor.org/errata_search.php?rfc=7231[Errata]) * https://tools.ietf.org/html/rfc7232[RFC 7232 - Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests] (http://www.rfc-editor.org/errata_search.php?rfc=7232[Errata]) * https://tools.ietf.org/html/rfc7233[RFC 7233 - Hypertext Transfer Protocol (HTTP/1.1): Range Requests] (http://www.rfc-editor.org/errata_search.php?rfc=7233[Errata]) * https://tools.ietf.org/html/rfc7234[RFC 7234 - Hypertext Transfer Protocol (HTTP/1.1): Caching] (http://www.rfc-editor.org/errata_search.php?rfc=7234[Errata]) * https://tools.ietf.org/html/rfc7235[RFC 7235 - Hypertext Transfer Protocol (HTTP/1.1): Authentication] (http://www.rfc-editor.org/errata_search.php?rfc=7235[Errata]) * https://tools.ietf.org/html/rfc5987[RFC 5987 - Character Set and Language Encoding for Hypertext Transfer Protocol (HTTP) Header Field Parameters] (http://www.rfc-editor.org/errata_search.php?rfc=5987[Errata]) * https://tools.ietf.org/html/rfc6266[RFC 6266 - Use of the Content-Disposition Header Field in the Hypertext Transfer Protocol (HTTP)] (http://www.rfc-editor.org/errata_search.php?rfc=6266[Errata]) * http://tools.ietf.org/html/rfc6265[RFC 6265 - HTTP State Management Mechanism (Cookies)] (http://www.rfc-editor.org/errata_search.php?rfc=6265[Errata]) * http://www.cookiecentral.com/faq/#3.5[Netscape Cookie Format] Other ^^^^^ * https://tools.ietf.org/html/rfc5646[RFC 5646 - Tags for Identifying Languages] (http://www.rfc-editor.org/errata_search.php?rfc=5646[Errata]) * http://www.w3.org/TR/CSS2/[Cascading Style Sheets Level 2 Revision 1 (CSS 2.1) Specification] * http://doc.qt.io/qt-5/stylesheet-reference.html[Qt Style Sheets Reference] * http://mimesniff.spec.whatwg.org/[MIME Sniffing Standard] * http://spec.whatwg.org/[WHATWG specifications] * http://www.w3.org/html/wg/drafts/html/master/Overview.html[HTML 5.1 Nightly] * http://www.w3.org/TR/webstorage/[Web Storage] * http://www.brynosaurus.com/cachedir/spec.html[Cache directory tagging standard] * http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html[XDG basedir specification] Hints ----- Python and Qt objects ~~~~~~~~~~~~~~~~~~~~~ For many tasks, there are solutions in both Qt and the Python standard library available. In qutebrowser, the policy is usually using the Python libraries, as they provide exceptions and other benefits. There are some exceptions to that: * `QThread` is used instead of Python threads because it provides signals and slots. * `QProcess` is used instead of Python's `subprocess` * `QUrl` is used instead of storing URLs as string, see the <> section for details. When using Qt objects, two issues must be taken care of: * Methods of Qt objects report their status by using their return values, instead of using exceptions. + If a function gets or returns a Qt object which has an `.isValid()` method such as `QUrl` or `QModelIndex`, there's a helper function `ensure_valid` in `qutebrowser.utils.qtutils` which should get called on all such objects. It will raise `qutebrowser.utils.qtutils.QtValueError` if the value is not valid. + If a function returns something else on error, the return value should carefully be checked. * Methods of Qt objects have certain maximum values, based on their underlying C++ types. + When passing a numeric parameter to a Qt function, all numbers should be range-checked using `qutebrowser.qtutils.check_overflow`, or passing a value which is too large should be avoided by other means (e.g. by setting a maximum value for a config object). [[object-registry]] The object registry ~~~~~~~~~~~~~~~~~~~ The object registry in `qutebrowser.utils.objreg` is a collection of dictionaries which map object names to the actual long-living objects. There are currently these object registries, also called 'scopes': * The `global` scope, with objects which are used globally (`config`, `cookie-jar`, etc.) * The `tab` scope with objects which are per-tab (`hintmanager`, `webview`, etc.). Passing this scope to `objreg.get()` selects the object in the currently focused tab by default. A tab can be explicitly selected by passing +tab=_tab-id_, window=_win-id_+ to it. A new object can be registered by using +objreg.register(_name_, _object_[, scope=_scope_, window=_win-id_, tab=_tab-id_])+. An object should not be registered twice. To update it, `update=True` has to be given. An object can be retrieved by using +objreg.get(_name_[, scope=_scope_, window=_win-id_, tab=_tab-id_])+. The default scope is `global`. All objects can be printed by starting with the `--debug` flag and using the `:debug-all-objects` command. The registry is mainly used for <> but also can be useful in places where using Qt's http://doc.qt.io/qt-5/signalsandslots.html[signals and slots] mechanism would be difficult. Logging ~~~~~~~ Logging is used at various places throughout the qutebrowser code. If you add a new feature, you should also add some strategic debug logging. Unless other Python projects, qutebrowser doesn't use a logger per file, instead it uses custom-named loggers. The existing loggers are defined in `qutebrowser.utils.log`. If your feature doesn't fit in any of the logging categories, simply add a new line like this: [source,python] ---- foo = getLogger('foo') ---- Then in your source files, do this: [source,python] ---- from qutebrowser.utils import log ... log.foo.debug("Hello World") ---- The following logging levels are available for every logger: [width="75%",cols="25%,75%"] |======================================================================= |criticial |Critical issue, qutebrowser can't continue to run. |error |There was an issue and some kind of operation was abandoned. |warning |There was an issue but the operation can continue running. |info |General informational messages. |debug |Verbose debugging informations. |======================================================================= [[commands]] Commands ~~~~~~~~ qutebrowser has the concept of functions which are exposed to the user as commands. Creating a new command is straightforward: [source,python] ---- import qutebrowser.commands.cmdutils ... @cmdutils.register(...) def foo(): ... ---- The commands arguments are automatically deduced by inspecting your function. If the function is a method of a class, the `@cmdutils.register` decorator needs to have an `instance=...` parameter which points to the (single/main) instance of the class. The `instance` parameter is the name of an object in the object registry, which then gets passed as the `self` parameter to the handler. The `scope` argument selects which object registry (global, per-tab, etc.) to use. See the <> section for details. There are also other arguments to customize the way the command is registered, see the class documentation for `register` in `qutebrowser.commands.cmdutils` for details. The types of the function arguments are inferred based on their default values, e.g. an argument `foo=True` will be converted to a flag `-f`/`--foo` in qutebrowser's commandline. This behavior can be overridden using Python's http://legacy.python.org/dev/peps/pep-3107/[function annotations]. The annotation should always be a `dict`, like this: [source,python] ---- @cmdutils.register(...) def foo(bar: {'type': int}, baz=True): ... ---- The following keys are supported in the dict: * `type`: The type this value should have. The value entered by the user is then automatically checked. Possible values: - A callable (`int`, `float`, etc.): Gets called to validate/convert the value. - A string: The value must match exactly (mainly useful with tuples to get a choice of values, see below). - A python enum type: All members of the enum are possible values. - A tuple of multiple types above: Any of these types are valid values, e.g. `('foo', 'bar')` or `(int, 'foo')`. * `flag`: The flag to be used, as 1-char string (default: First char of the long name). * `nargs`: Gets passed to argparse, see https://docs.python.org/dev/library/argparse.html#nargs[its documentation]. The name of an argument will always be the parameter name, with any trailing underscores stripped. [[handling-urls]] Handling URLs ~~~~~~~~~~~~~ qutebrowser handles two different types of URLs: URLs as a string, and URLs as the Qt `QUrl` type. As this can get confusing quickly, please follow the following guidelines: * Convert a string to a QUrl object as early as possible, i.e. directly after the user did enter it. - Use `utils.urlutils.fuzzy_url` if the URL is entered by the user somewhere. - Be sure you handle `utils.urlutils.FuzzyError` and display an error message to the user. * Convert a `QUrl` object to a string as late as possible, e.g. before displaying it to the user. - If you want to display the URL to the user, use `url.toDisplayString()` so password information is removed. - If you want to get the URL as string for some other reason, you most likely want to add the `QUrl.EncodeFully` and `QUrl.RemovePassword` flags. * Name a string URL something like `urlstr`, and a `QUrl` something like `url`. * Mention in the docstring whether your function needs a URL string or a `QUrl`. * Call `ensure_valid` from `utils.qtutils` whenever getting or creating a `QUrl` and take appropriate action if not. Note the URL of the current page always could be an invalid QUrl (if nothing is loaded yet). Running valgrind on QtWebKit ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you want to run qutebrowser (and thus QtWebKit) with http://valgrind.org/[valgrind], you'll need to pass `--smc-check=all` to it or recompile QtWebKit with the Javascript JIT disabled. This is needed so valgrind handles self-modifying code correctly: [quote] ____ This option controls Valgrind's detection of self-modifying code. If no checking is done, if a program executes some code, then overwrites it with new code, and executes the new code, Valgrind will continue to execute the translations it made for the old code. This will likely lead to incorrect behavior and/or crashes. ... Note that the default option will catch the vast majority of cases. The main case it will not catch is programs such as JIT compilers that dynamically generate code and subsequently overwrite part or all of it. Running with all will slow Valgrind down noticeably. ____ Style conventions ----------------- qutebrowser's coding conventions are based on http://legacy.python.org/dev/peps/pep-0008/[PEP8] and the https://google-styleguide.googlecode.com/svn/trunk/pyguide.html[Google Python style guidelines] with some additions: * The _Raise:_ section is not added to the docstring. * Methods overriding Qt methods (obviously!) don't follow the naming schemes. * Everything else does though, even slots. * Docstrings should look like described in http://legacy.python.org/dev/peps/pep-0257/[PEP257] and the google guidelines. * Class docstrings have additional _Attributes:_, _Class attributes:_ and _Signals:_ sections. * In docstrings of command handlers (registered via `@cmdutils.register`), the description should be split into two parts by using `//` - the first part is the description of the command like it will appear in the documentation, the second part is "internal" documentation only relevant to people reading the sourcecode. + Example for a class docstring: + [source,python] ---- """Some object. Attributes: blub: The current thing to handle. Signals: valueChanged: Emitted when a value changed. arg: The new value """ ---- + Example for a method/function docstring: + [source,python] ---- """Do something special. This will do something. // It is based on http://example.com/. Args: foo: ... Return: True if something, False if something else. """ ---- + * The layout of a module should be roughly like this: - Shebang (`#!/usr/bin/python`, if needed) - vim-modeline (`# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et`) - Copyright - GPL boilerplate - Module docstring - Python standard library imports - PyQt imports - qutebrowser imports - functions - classes * The layout of a class should be like this: - docstring - `__magic__` methods - properties - _private methods - public methods - `on_*` methods - overrides of Qt methods Checklists ---------- These are mainly intended for myself, but they also fit in here well. New Qt release ~~~~~~~~~~~~~~ * Run all tests and check nothing is broken. * Check the https://bugreports.qt.io/issues/?jql=reporter%20%3D%20%22The%20Compiler%22%20ORDER%20BY%20fixVersion%20ASC[Qt bugtracker] and make sure all bugs marked as resolved are actually fixed. * Update own PKGBUILDs based on upstream Archlinux updates and rebuild. * Update recommended Qt version in `README` * Grep for `WORKAROUND` in the code and test if fixed stuff works without the workaround. * Check relevant https://github.com/The-Compiler/qutebrowser/issues?q=is%3Aopen+is%3Aissue+label%3Aqt[qutebrowser bugs] and check if they're fixed. New PyQt release ~~~~~~~~~~~~~~~~ * See above * Install new PyQt in Windows VM (32- and 64-bit) * Download new installer and update PyQt installer path in `ci_install.py`. qutebrowser release ~~~~~~~~~~~~~~~~~~~ * Make sure there are no unstaged changes and the tests are green. * Adjust `__version_info__` in `qutebrowser/__init__.py`. * Remove *(unreleased)* from changelog. * Run tests again * Run `asciidoc2html.py`. * Commit * Create annotated git tag (`git tag -s "v0.X.Y" -m "Release v0.X.Y"`) * If it's a new minor, create git branch `v0.X.x` * If committing on minor branch, cherry-pick release commit to master. * `git push origin`; `git push origin v0.X.Y` * Create release on github * Mark the milestone at https://github.com/The-Compiler/qutebrowser/milestones as closed. * Run `scripts/dev/build_release.py` on Linux to build an sdist * Upload to PyPI: `twine upload dist/foo{,.asc}` * Create Windows packages via `scripts/dev/build_release.py` and upload. * Upload to qutebrowser.org with checksum/GPG - On server: `sudo mkdir -p /srv/http/qutebrowser/releases/v0.X.Y/windows` - `rsync -avPh dist/ tonks:` - On server: `sudo mv qutebrowser-0.X.Y.tar.gz* /srv/http/qutebrowser/releases/v0.X.Y` * Update AUR package * Announce to qutebrowser mailinglist