Replace tox -e mkvenv by a mkvenv.py script

This was mainly needed because PyQt 5.14.1 is a manylinux2014 wheel and needs
an updated pip, see #5013

6 files changed, 266 insertions, 38 deletions

diff --git a/doc/changelog.asciidoc b/doc/changelog.asciidoc
index 4e4ba40dd..a38453826 100644
--- a/doc/changelog.asciidoc
+++ b/doc/changelog.asciidoc
@@ -25,6 +25,13 @@ Added
   `prefer-color-scheme: dark` colors for websites (QtWebEngine with Qt 5.14 or
   newer).
 
+Changed
+~~~~~~~
+
+- The `tox -e mkvenv` (or `mkvenv-pypi`) way of installing qutebrowser is now
+  replaced by a `mkvenv.py` script. See the updated
+  link:install{outfilesuffix}#tox[install instructions] for details.
+
 v1.9.0 (2020-01-08)
 -------------------
 
diff --git a/doc/contributing.asciidoc b/doc/contributing.asciidoc
index ba984f328..96c35e4b3 100644
--- a/doc/contributing.asciidoc
+++ b/doc/contributing.asciidoc
@@ -92,9 +92,9 @@ git format-patch origin/master <1>
 Running qutebrowser
 -------------------
 
-After link:install{outfilesuffix}#tox[installing qutebrowser via tox], you can run
-`.venv/bin/qutebrowser --debug --temp-basedir` to test your changes with debug
-logging enabled and without affecting existing running instances.
+After link:install{outfilesuffix}#tox[installing qutebrowser in a virtualenv],
+you can run `.venv/bin/qutebrowser --debug --temp-basedir` to test your changes
+with debug logging enabled and without affecting existing running instances.
 
 Alternatively, you can install qutebrowser's dependencies system-wide and run
 `python3 -m qutebrowser --debug --temp-basedir`.
diff --git a/doc/install.asciidoc b/doc/install.asciidoc
index 2a8fcd204..0b93b6c32 100644
--- a/doc/install.asciidoc
+++ b/doc/install.asciidoc
@@ -14,9 +14,9 @@ Ubuntu 16.04 LTS / Linux Mint 18
 
 Ubuntu 16.04 doesn't come with an up-to-date engine (a new enough QtWebKit, or
 QtWebEngine). However, it comes with Python 3.5, so you can
-<<tox,install qutebrowser via tox>>.
+<<tox,install qutebrowser in a virtualenv>>.
 
-You'll need some basic libraries to use the tox-installed PyQt:
+You'll need some basic libraries to use the virtualenv-installed PyQt:
 
 ----
 # apt install libglib2.0-0 libgl1 libfontconfig1 libx11-xcb1 libxi6 libxrender1 libdbus-1-3
@@ -62,7 +62,7 @@ can install it with apt:
 Additional hints
 ~~~~~~~~~~~~~~~~
 
-- Alternatively, you can <<tox,install qutebrowser via tox>> to get a newer
+- Alternatively, you can <<tox,install qutebrowser in a virtualenv>> to get a newer
   QtWebEngine version.
 - If running from git, run the following to generate the documentation for the
   `:help` command:
@@ -279,16 +279,10 @@ C:\> scoop install qutebrowser
 Manual install
 ~~~~~~~~~~~~~~
 
-* Use the installer from http://www.python.org/downloads[python.org] to get
+Use the installer from http://www.python.org/downloads[python.org] to get
 Python 3 (be sure to install pip).
-* Install https://testrun.org/tox/latest/index.html[tox] via
-https://pip.pypa.io/en/latest/[pip]:
 
-----
-$ pip install tox
-----
-
-Then <<tox,install qutebrowser via tox>>.
+Then <<tox,install qutebrowser via virtualenv>>.
 
 On macOS
 --------
@@ -346,9 +340,25 @@ standard location for your distro (`/usr/share/applications` and
 The normal `setup.py install` doesn't install these files, so you'll have to do
 it as part of the packaging process.
 
+// The tox anchor is so that old links remain compatible.
+// When switching to Sphinx, that should be changed.
+
 [[tox]]
-Installing qutebrowser with tox
--------------------------------
+Installing qutebrowser with virtualenv
+--------------------------------------
+
+IMPORTANT: Before January 2020, this section used to be about installing
+qutebrowser via `tox` which is a wrapper around `virtualenv`. Now, a dedicated
+script is used instead.
+
+A https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments[virtual environment]
+(virtualenv, venv) allows Python packages to be installed in an isolated
+location for a particular application, rather than being installed globally.
+
+The `scripts/mkvenv.py` script in this repository can be used to create a
+virtualenv for qutebrowser and install it (including all dependencies) there.
+The next couple of sections will explain the most common use-cases - run
+`mkvenv.py` with `--help` to see all available options.
 
 Getting the repository
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -364,11 +374,10 @@ $ cd qutebrowser
 Installing dependencies (including Qt)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Then run tox inside the qutebrowser repository to set up a
-https://docs.python.org/3/library/venv.html[virtual environment]:
+Then run the install script:
 
 ----
-$ tox -e mkvenv-pypi
+$ python3 scripts/mkvenv.py
 ----
 
 This installs all needed Python dependencies in a `.venv` subfolder.
@@ -389,18 +398,21 @@ See the next section for an alternative.
 Installing dependencies (system-wide Qt)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Alternatively, you can use `tox -e mkvenv` (without `-pypi`) to symlink your
-local Qt install instead of installing PyQt in the virtualenv. However, unless
+Alternatively, you can use `mkvenv.py --pyqt-mode link` to symlink your local
+PyQt/Qt install instead of installing PyQt in the virtualenv. However, unless
 you have a new QtWebKit or QtWebEngine available, qutebrowser will not work. It
 also typically means you'll be using an older release of QtWebEngine.
 
 On Windows, run `set PYTHON=C:\path\to\python.exe` (CMD) or ``$Env:PYTHON =
 "..."` (Powershell) first.
 
+There is a third mode, `mkvenv.py --pyqt-mode source` which uses a system-wide
+Qt but builds PyQt from source. In most scenarios, this shouldn't be needed.
+
 Creating a wrapper script
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Running `tox` does not install a system-wide `qutebrowser` script. You can
+Running `mkvenv.py` does not install a system-wide `qutebrowser` script. You can
 launch qutebrowser by doing:
 
 ----
@@ -432,10 +444,5 @@ Updating
 
 When you updated your local copy of the code (e.g. by pulling the git repo, or
 extracting a new version), the virtualenv should automatically use the updated
-code. However, if dependencies got added, this won't be reflected in the
-virtualenv. Thus it's recommended to run the following command to recreate the
-virtualenv:
-
-----
-$ tox -r -e mkvenv-pypi
-----
+code. However, dependencies won't be updated that way. Re-running `mkvenv.py`
+will recreate the virtualenv with updated dependencies.
diff --git a/scripts/mkvenv.py b/scripts/mkvenv.py
new file mode 100644
index 000000000..55f08a618
--- /dev/null
+++ b/scripts/mkvenv.py
@@ -0,0 +1,218 @@
+#!/usr/bin/env python3
+# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
+
+# Copyright 2020 Florian Bruhin (The Compiler) <mail@qutebrowser.org>
+
+# This file is part of qutebrowser.
+#
+# qutebrowser is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# qutebrowser is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with qutebrowser.  If not, see <http://www.gnu.org/licenses/>.
+
+
+"""Create a local virtualenv with a PyQt install."""
+
+import argparse
+import pathlib
+import sys
+import os
+import os.path
+import typing
+import shutil
+import venv
+import subprocess
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), os.pardir))
+from scripts import utils, link_pyqt
+
+
+REPO_ROOT = pathlib.Path(__file__).parent.parent
+
+
+def parse_args() -> argparse.Namespace:
+    """Parse commandline arguments."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--keep',
+                        action='store_true',
+                        help="Reuse an existing virtualenv.")
+    parser.add_argument('--venv-dir',
+                        default='.venv',
+                        help="Where to place the virtualenv.")
+    parser.add_argument('--pyqt-version',
+                        choices=pyqt_versions(),
+                        default='auto',
+                        help="PyQt version to install")
+    parser.add_argument('--pyqt-type',
+                        choices=['binary', 'source', 'link'],
+                        default='binary',
+                        help="How to install PyQt/Qt.")
+    parser.add_argument('--tox-error',
+                        action='store_true',
+                        help=argparse.SUPPRESS)
+    return parser.parse_args()
+
+
+def pyqt_versions() -> typing.List[str]:
+    """Get a list of all available PyQt versions.
+
+    The list is based on the filenames of misc/requirements/ files.
+    """
+    version_set = set()
+
+    requirements_dir = REPO_ROOT / 'misc' / 'requirements'
+    for req in requirements_dir.glob('requirements-pyqt-*.txt'):
+        version_set.add(req.stem.split('-')[-1])
+
+    versions = sorted(version_set,
+                      key=lambda v: [int(c) for c in v.split('.')])
+    return versions + ['auto']
+
+
+def run_venv(venv_dir: pathlib.Path, executable, *args: str) -> None:
+    """Runt he given command inside the virtualenv."""
+    subdir = 'Scripts' if os.name == 'nt' else 'bin'
+
+    try:
+        subprocess.run([str(venv_dir / subdir / executable)] +
+                       [str(arg) for arg in args], check=True)
+    except subprocess.CalledProcessError as e:
+        utils.print_col("Subprocess failed, exiting", 'red')
+        sys.exit(e.returncode)
+
+
+def pip_install(venv_dir: pathlib.Path, *args: str) -> None:
+    """Run a pip install command inside the virtualenv."""
+    arg_str = ' '.join(str(arg) for arg in args)
+    utils.print_col('venv$ pip install {}'.format(arg_str), 'blue')
+    run_venv(venv_dir, 'python3', '-m', 'pip', 'install', *args)
+
+
+def show_tox_error(pyqt_type: str) -> None:
+    """DIsplay an error when invoked from tox."""
+    if pyqt_type == 'link':
+        env = 'mkvenv'
+        args = ' --pyqt-type link'
+    elif pyqt_type == 'binary':
+        env = 'mkvenv-pypi'
+        args = ''
+    else:
+        raise AssertionError
+
+    print()
+    utils.print_col('tox -e {} is deprecated. Please use scripts/mkvenv.py{} '
+                    'instead.'.format(env, args), 'red')
+    print()
+
+
+def delete_old_venv(venv_dir: pathlib.Path) -> None:
+    """Remove an existing virtualenv directory."""
+    if not venv_dir.exists():
+        return
+
+    markers = ['.tox-config1', 'pyvenv.cfg']
+    if not any((venv_dir / m).exists() for m in markers):
+        utils.print_col('{} does not look like a virtualenv, '
+                        'cowardly refusing to remove it.'.format(venv_dir),
+                        'red')
+        sys.exit(1)
+
+    utils.print_col('$ rm -r {}'.format(venv_dir), 'blue')
+    shutil.rmtree(str(venv_dir))
+
+
+def create_venv(venv_dir: pathlib.Path) -> None:
+    """Create a new virtualenv."""
+    utils.print_col('$ python3 -m venv {}'.format(venv_dir), 'blue')
+    venv.create(str(venv_dir), with_pip=True)
+
+
+def upgrade_pip(venv_dir: pathlib.Path) -> None:
+    """Upgrade pip inside a virtualenv."""
+    utils.print_title("Upgrading pip")
+    pip_install(venv_dir, '-U', 'pip')
+
+
+def pyqt_requirements_file(version: str):
+    """Get the filename of the requirements file for the given PyQt version."""
+    suffix = '' if version == 'auto' else '-{}'.format(version)
+    return (REPO_ROOT / 'misc' / 'requirements' /
+            'requirements-pyqt{}.txt'.format(suffix))
+
+
+def install_pyqt_binary(venv_dir: pathlib.Path, version: str) -> None:
+    """Install PyQt from a binary wheel."""
+    utils.print_title("Installing PyQt from binary")
+    utils.print_col("No proprietary codec support will be available in "
+                    "qutebrowser.", 'bold')
+    pip_install(venv_dir, '-r', pyqt_requirements_file(version),
+                '--only-binary', 'PyQt5,PyQtWebEngine')
+
+
+def install_pyqt_source(venv_dir: pathlib.Path, version: str) -> None:
+    """Install PyQt from the source tarball."""
+    utils.print_title("Installing PyQt from sources")
+    pip_install(venv_dir, '-r', pyqt_requirements_file(version),
+                '--verbose', '--no-binary', 'PyQt5,PyQtWebEngine')
+
+
+def install_pyqt_link(venv_dir: pathlib.Path) -> None:
+    """Install PyQt by linking a system-wide install."""
+    utils.print_title("Linking system-wide PyQt")
+    lib_path = link_pyqt.get_venv_lib_path(str(venv_dir))
+    link_pyqt.link_pyqt(sys.executable, lib_path)
+
+
+def install_requirements(venv_dir: pathlib.Path) -> None:
+    """Install qutebrowser's requirement.txt."""
+    utils.print_title("Installing other qutebrowser dependencies")
+    requirements_file = REPO_ROOT / 'requirements.txt'
+    pip_install(venv_dir, '-r', str(requirements_file))
+
+
+def install_qutebrowser(venv_dir: pathlib.Path) -> None:
+    """Install qutebrowser itself as an editable install."""
+    utils.print_title("Installing qutebrowser")
+    pip_install(venv_dir, '-e', str(REPO_ROOT))
+
+
+def main() -> None:
+    """Install qutebrowser in a virtualenv.."""
+    args = parse_args()
+    venv_dir = pathlib.Path(args.venv_dir)
+    utils.change_cwd()
+
+    if args.tox_error:
+        show_tox_error(args.pyqt_type)
+        sys.exit(1)
+
+    if not args.keep:
+        utils.print_title("Creating virtual environment")
+        delete_old_venv(venv_dir)
+        create_venv(venv_dir)
+
+    upgrade_pip(venv_dir)
+
+    if args.pyqt_type == 'binary':
+        install_pyqt_binary(venv_dir, args.pyqt_version)
+    elif args.pyqt_type == 'source':
+        install_pyqt_source(venv_dir, args.pyqt_version)
+    elif args.pyqt_type == 'link':
+        install_pyqt_link(venv_dir)
+    else:
+        raise AssertionError
+
+    install_requirements(venv_dir)
+    install_qutebrowser(venv_dir)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/scripts/utils.py b/scripts/utils.py
index b0314abb8..0d405c8a6 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -37,6 +37,8 @@ use_color = os.name != 'nt' or colorama
 
 
 fg_colors = {
+    'reset': 0,
+    'bold': 1,
     'black': 30,
     'red': 31,
     'green': 32,
@@ -45,7 +47,6 @@ fg_colors = {
     'magenta': 35,
     'cyan': 36,
     'white': 37,
-    'reset': 39,
 }
 
 
@@ -69,6 +70,7 @@ def print_col(text, color):
 
 def print_title(text):
     """Print a title."""
+    print()
     print_col("==================== {} ====================".format(text),
               'yellow')
 
diff --git a/tox.ini b/tox.ini
index c4c9d1a60..f2bd60f0a 100644
--- a/tox.ini
+++ b/tox.ini
@@ -43,29 +43,23 @@ commands =
 
 [testenv:mkvenv]
 basepython = {env:PYTHON:python3}
-commands = {envpython} scripts/link_pyqt.py --tox {envdir}
-envdir = {toxinidir}/.venv
+commands = {envpython} scripts/mkvenv.py --tox-error --pyqt-type link
 usedevelop = true
 deps =
-    -r{toxinidir}/requirements.txt
 
 # This is undocumented, but it's a common typo, so let's make it work
 [testenv:mkenv]
 basepython = {[testenv:mkvenv]basepython}
 commands = {[testenv:mkvenv]commands}
-envdir = {[testenv:mkvenv]envdir}
 usedevelop = {[testenv:mkvenv]usedevelop}
 deps = {[testenv:mkvenv]deps}
 
 # Virtualenv with PyQt5 from PyPI
 [testenv:mkvenv-pypi]
 basepython = {env:PYTHON:python3}
-envdir = {toxinidir}/.venv
-commands = {envpython} -c ""
+commands = {envpython} scripts/mkvenv.py --tox-error
 usedevelop = true
 deps =
-    -r{toxinidir}/requirements.txt
-    -r{toxinidir}/misc/requirements/requirements-pyqt-5.12.txt
 
 [testenv:misc]
 ignore_errors = true