Update pytest summary problem matcher for colored output

We think that at some point pytest added color codes to the summary
output which broke these matcher regex. It looks like there is an issue
about stripping color codes here: https://github.com/actions/runner/issues/2341

I've added wildcard (or no-space) elements instead of trying to match
the color codes exactly (eg `\033\[31m` etc) because 1) that's not very
readable 2) I was having trouble getting that to work with egrep.

The goal is to match the target strings but not to match then later in
the line as part of a test log or whatever. For the '= short test
summary =' that should be pretty unique. For the ERROR|FAILED I reckon
that could be a bit more common. I'm just matching with any mount of
non-space characters because I reckon a space will crop up pretty early
in any line where ERROR isn't the first work.

I think this new matching will only apply to new or rebased PRs after it
is landed.

ref: https://github.com/qutebrowser/qutebrowser/issues/5390#issuecomment-1817503702

1 files changed, 6 insertions, 2 deletions

diff --git a/scripts/dev/ci/problemmatchers.py b/scripts/dev/ci/problemmatchers.py
index fa623dec7..3316c5597 100644
--- a/scripts/dev/ci/problemmatchers.py
+++ b/scripts/dev/ci/problemmatchers.py
@@ -160,13 +160,17 @@ MATCHERS = {
     "tests": [
         {
             # pytest test summary output
+            # Examples (with ANSI color codes around FAILED|ERROR and the
+            # function name):
+            # FAILED tests/end2end/features/test_keyinput_bdd.py::test_fakekey_sending_special_key_to_the_website - end2end.fixtures.testprocess.WaitForTimeout: Timed out after 15000ms waiting for {'category': 'js', 'message': '[*] key press: 27'}.
+            # ERROR tests/end2end/test_insert_mode.py::test_insert_mode[100-textarea.html-qute-textarea-clipboard-qutebrowser] - Failed: Logged unexpected errors:
             "severity": "error",
             "pattern": [
                 {
-                    "regexp": r'^=+ short test summary info =+$',
+                    "regexp": r'^.*=== short test summary info ===.*$',
                 },
                 {
-                    "regexp": r"^((ERROR|FAILED) .*)",
+                    "regexp": r"^[^ ]*((ERROR|FAILED)[^ ]* .*)$",
                     "message": 1,
                     "loop": True,
                 }