Tips for contributors to run the webkit backendwip/flaky_qt67

My virtualenv I used to run webkit has rotted long ago and I don't remember
how I set it up. There is a PyQtWebKit project on PyPI but I don't know
who that's published by.

So I figured I would write some notes for myself on using the docker container
used for CI instead. I chose to mount the current directory (which is
presumably a qutebrowser checkout!) directly into the container instead of
cloning it so I could have quicker feedback between making code changes and
running tests.

Then there's a couple of things that stem from that. Since the user in the
container is different from the one in the host we have to move some things
that are normally written to the current directory to be written elsewhere.
There are other ways to approach this (eg you can add `-u $(id -u)` to the
docker command line, although that makes things a bit confusing in the
container) but arguably it's good for the container not to be able to write to
the host, hence making that volume read only.

The TOX_WORK_DIR trick is from
[here](https://github.com/tox-dev/tox/issues/20), apart from with
`{toxinidir}` in it too because the pyroma env was failing with just
`.tox`, saying the pyroma binary needed to be in the allowlist, possibly
it was doing full path matching without normalizing.

The hypothesis folks
[here](https://github.com/HypothesisWorks/hypothesis/issues/2367#issuecomment-595524571)
say if you want to override the examples DB location with an env var to
do it yourself. It's actually only a warning from hypothesis, it says it
falls back to an in-memory DB, but I guess the tests run with
warnings-are-errors. You can also pass `database=None` to make
hypothesis skip example storage altogether.

I'm using tox to run commands in a virtualenv with the right stuff in it
because, uh, because I was copying the CI workflow actually. I just found out
about the `exec` subcommand to override the `commands` defined for the env,
neat! One point of awkwardness about that is that since we are using the
PyQt from the OS we need any virtualenv we use to have access to the OS
packages, which isn't the default for virtualenvs created by tox. The
text envs use the link_pyqt script for that but if you are using this
container and the first thing you do is run `tox exec` then that
wouldn't have been run. So I'm setting `VIRTUALENV_SYSTEM_SITE_PACKAGES`
to tell tox to always make the system packages available in the
virtualenvs it manages.

I did try using the mkvenv script instead of tox but it complained when
trying to install the current directory in editable mode because
setup.py tries to write to a git-commit-id file.

3 files changed, 36 insertions, 1 deletions

diff --git a/doc/contributing.asciidoc b/doc/contributing.asciidoc
index 144117677..630a96db7 100644
--- a/doc/contributing.asciidoc
+++ b/doc/contributing.asciidoc
@@ -192,6 +192,28 @@ specific one you can set either of a) the environment variable QUTE_TESTS_BACKEN
 , or b) the command line argument --qute-backend, to the desired backend
 (webkit/webengine).
 
+If you need an environment with webkit installed to do testing while we still
+support it (see #4039) you can re-use the docker container used for the CI
+test runs which has PyQt5Webkit installed from the archlinux package archives.
+Examples:
+
+----
+# Get a bash shell in the docker container with
+# a) the current directory mounted at /work in the container
+# b) the container using the X11 display :27 (for example, a Xephyr instance) from the host
+# c) the tox and hypothesis dirs set to somewhere in the container that it can write to
+# d) the system site packages available in the tox venv so you can use PyQt
+#    from the OS without having to run the link_pyqt script
+docker run -it -v $PWD:/work:ro -w /work -e QUTE_TESTS_BACKEND=webkit -e DISPLAY=:27 -v /tmp/.X11-unix:/tmp/.X11-unix -e TOX_WORK_DIR="/home/user/.tox" -e HYPOTHESIS_EXAMPLES_DIR="/home/user/.hypothesis/examples" -e VIRTUALENV_SYSTEM_SITE_PACKAGES=True qutebrowser/ci:archlinux-webkit bash
+
+# Start a qutebrowser temporary basedir in the appropriate tox environment to
+# play with
+tox exec -e py-qt5 -- python3 -m qutebrowser -T --backend webkit
+
+# Run tests, passing positional args through to pytest.
+tox -e py-qt5 -- tests/unit
+----
+
 Profiling
 ~~~~~~~~~
 
diff --git a/tests/conftest.py b/tests/conftest.py
index ddacc3db1..c834e62a0 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -11,6 +11,7 @@ import ssl
 
 import pytest
 import hypothesis
+import hypothesis.database
 
 pytest.register_assert_rewrite('helpers')
 
@@ -33,10 +34,19 @@ _qute_scheme_handler = None
 
 
 # Set hypothesis settings
+hypotheses_optional_kwargs = {}
+if "HYPOTHESIS_EXAMPLES_DIR" in os.environ:
+    hypotheses_optional_kwargs[
+        "database"
+    ] = hypothesis.database.DirectoryBasedExampleDatabase(
+        os.environ["HYPOTHESIS_EXAMPLES_DIR"]
+    )
+
 hypothesis.settings.register_profile(
     'default', hypothesis.settings(
         deadline=600,
         suppress_health_check=[hypothesis.HealthCheck.function_scoped_fixture],
+        **hypotheses_optional_kwargs,
     )
 )
 hypothesis.settings.register_profile(
@@ -45,7 +55,8 @@ hypothesis.settings.register_profile(
         suppress_health_check=[
             hypothesis.HealthCheck.function_scoped_fixture,
             hypothesis.HealthCheck.too_slow
-        ]
+        ],
+        **hypotheses_optional_kwargs,
     )
 )
 hypothesis.settings.load_profile('ci' if testutils.ON_CI else 'default')
diff --git a/tox.ini b/tox.ini
index b4aada447..87ff70fe2 100644
--- a/tox.ini
+++ b/tox.ini
@@ -8,6 +8,7 @@ envlist = py38-pyqt515-cov,mypy-pyqt5,misc,vulture,flake8,pylint,pyroma,check-ma
 distshare = {toxworkdir}
 skipsdist = true
 minversion = 3.20
+toxworkdir={env:TOX_WORK_DIR:{toxinidir}/.tox}
 
 [testenv]
 setenv =
@@ -31,6 +32,7 @@ passenv =
     FORCE_COLOR
     DBUS_SESSION_BUS_ADDRESS
     RUNNER_TEMP
+    HYPOTHESIS_EXAMPLES_DIR
 basepython =
     py: {env:PYTHON:python3}
     py3: {env:PYTHON:python3}