tests: wait for flaky iframes to load via JS

TL;DR: I think I stabilized a couple of hinting iframe tests and applied that change to all the iframe usages in the end2end tests in the hopes that it would resolve some other flaky tests. I was facing some tests that hinted elements in iframes failing intermittently recently. They were most consistently failing on the windows runner. This is similar to the trend described in the comment linked from #1525. The tests failing recently where: tests (py39-pyqt515, windows-2019, 3.9) test_using_hintfollow_inside_an_iframe test_using_hintfollow_inside_an_iframe_button tests (py311-pyqt515, ubuntu-20.04, 3.11) test_using_hintfollow_inside_an_iframe test_using_hintfollow_inside_an_iframe_button They are failing because hints don't get generated for the elements in the iframe. I can see hints do get generated for the iframe itself though. Examining the logs in 5.15 it seems that there is a `cur_load_finished(True) (tab *)` log line after we run :hint. I suspect that this is the load finished signal for the iframe. I attempted to change the bdd open_path function to wait for the current message it's looking for and then additionally look for that cur_load_finished signal, but then it starts failing locally when the iframe load signal actually comes before the parent frame one. Just looking for the cur_load_finished signal itself also always found it immediately ("already found"), probably from the parent frame. So instead of trying to deal with that indeterminate ordering or trying to change the signals to say what frame they are coming from, I added javascript to all the pages used in iframes that run on load, and changed all the tests to watch for log messages from the JS. It's not the most maintainable solution, perhaps if we generated our test files with jinja we could have some "subframe loaded" boilerplate, get a message to log from a query param, look at metadata from test files for the open_path checking etc. I then searched for all the iframe usages in the test data and applied that change to all of those files and all the tests that used them. A few of those tests also had `flaky` annotations on them. I removed those annotations for now in the hopes that there were affected by the same problem. And that I actually managed to work around it correctly. ref: #7621
author: toofar <toofar@spalge.com> 2023-03-25 15:57:17 +1300
committer: toofar <toofar@spalge.com> 2023-03-25 18:37:58 +1300
commit: cb0b51c09463f5c3a53fccfb12077f2934db214e (patch)
tree: 0683f6c697620aa05be4055609faad729a54aa4f
parent: e03b81cb35f7433b63031cf08b1fbfb4a8a971e7 (diff)
download: qutebrowser-cb0b51c09463f5c3a53fccfb12077f2934db214e.tar.gz
qutebrowser-cb0b51c09463f5c3a53fccfb12077f2934db214e.zip
9 files changed, 17 insertions, 10 deletions
diff --git a/tests/end2end/data/hints/html/wrapped.html b/tests/end2end/data/hints/html/wrapped.html
index dcc05c8c7..89dceafa4 100644
--- a/tests/end2end/data/hints/html/wrapped.html
+++ b/tests/end2end/data/hints/html/wrapped.html
@@ -7,7 +7,7 @@
         <meta charset="utf-8">
         <title>Link wrapped across multiple lines</title>
     </head>
-    <body>
+    <body onload="console.log('wrapped loaded')">
         <div style="width: 20em;">
             Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis <a href="/data/hello.txt">nostrud exercitation</a> ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
         </div>
diff --git a/tests/end2end/data/hints/html/wrapped_button.html b/tests/end2end/data/hints/html/wrapped_button.html
index 4d5fd81bb..9ce1a28a3 100644
--- a/tests/end2end/data/hints/html/wrapped_button.html
+++ b/tests/end2end/data/hints/html/wrapped_button.html
@@ -7,7 +7,7 @@
         <meta charset="utf-8">
         <title>Button link wrapped across multiple lines</title>
     </head>
-    <body>
+    <body onload="console.log('wrapped_button loaded')">
         <div style="width: 20em;">
             Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis <button id="myButton" class="float-left submit-button" >nostrud exercitation</button> ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
         </div>
diff --git a/tests/end2end/data/hints/input.html b/tests/end2end/data/hints/input.html
index a81361281..c64503ada 100644
--- a/tests/end2end/data/hints/input.html
+++ b/tests/end2end/data/hints/input.html
@@ -6,6 +6,7 @@
         <title>Simple input</title>
         <script type="text/javascript">
           function setup_event_listener() {
+              console.log('input loaded')
               var elem = document.getElementById('qute-input-existing');
               console.log(elem);
               elem.addEventListener('input', function() {
diff --git a/tests/end2end/data/hints/issue3711.html b/tests/end2end/data/hints/issue3711.html
index 6abceccc2..51bf560c1 100644
--- a/tests/end2end/data/hints/issue3711.html
+++ b/tests/end2end/data/hints/issue3711.html
@@ -3,7 +3,7 @@
     <head>
         <title>Issue 3711</title>
     </head>
-    <body>
+    <body onload="console.log('issue3711 loaded')">
         <!--
         Verify no hint error occurs when hinting input range elements in iframes on qt5.9
         Possibly an issue in chrome.
diff --git a/tests/end2end/data/scroll/simple.html b/tests/end2end/data/scroll/simple.html
index 7f1bba4d5..b0a36e6e4 100644
--- a/tests/end2end/data/scroll/simple.html
+++ b/tests/end2end/data/scroll/simple.html
@@ -17,7 +17,7 @@
             }
         </script>
     </head>
-    <body>
+    <body onload="console.log('simple loaded')">
       <a href="/data/hello.txt" id="link">Just a link</a>
       <button>blub</button>
         <p>Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Vestibulum tortor quam, feugiat vitae, ultricies eget, tempor sit amet, ante. Donec eu libero sit amet quam egestas semper. Aenean ultricies mi vitae est. Mauris placerat eleifend leo. Quisque sit amet est et sapien ullamcorper pharetra. Vestibulum erat wisi, condimentum sed, commodo vitae, ornare sit amet, wisi. Aenean fermentum, elit eget tincidunt condimentum, eros ipsum rutrum orci, sagittis tempus lacus enim ac dui. Donec non enim in turpis pulvinar facilisis. Ut felis. Praesent dapibus, neque id cursus faucibus, tortor neque egestas augue, eu vulputate magna eros eu erat. Aliquam erat volutpat. Nam dui mi, tincidunt quis, accumsan porttitor, facilisis luctus, metus</p>
diff --git a/tests/end2end/data/search.html b/tests/end2end/data/search.html
index 1064a7d01..a43cd89a9 100644
--- a/tests/end2end/data/search.html
+++ b/tests/end2end/data/search.html
@@ -4,7 +4,7 @@
         <meta charset="utf-8">
         <title>Searching text on the page</title>
     </head>
-    <body>
+    <body onload="console.load('search loaded')">
         <p>
             foo<br/>
             Foo<br/>
diff --git a/tests/end2end/features/hints.feature b/tests/end2end/features/hints.feature
index 51ffb980a..2d597da13 100644
--- a/tests/end2end/features/hints.feature
+++ b/tests/end2end/features/hints.feature
@@ -251,25 +251,28 @@ Feature: Using hints
     ### iframes
     Scenario: Using :hint-follow inside an iframe
         When I open data/hints/iframe.html
+        And I wait for "* wrapped loaded" in the log
         And I hint with args "links normal" and follow a
         Then "navigation request: url http://localhost:*/data/hello.txt, type Type.link_clicked, *" should be logged
 
     Scenario: Using :hint-follow inside an iframe button
         When I open data/hints/iframe_button.html
+        And I wait for "* wrapped_button loaded" in the log
         And I hint with args "all normal" and follow s
         Then "navigation request: url http://localhost:*/data/hello.txt, *" should be logged
 
     Scenario: Hinting inputs in an iframe without type
         When I open data/hints/iframe_input.html
+        And I wait for "* input loaded" in the log
         And I hint with args "inputs" and follow a
         And I wait for "Entering mode KeyMode.insert (reason: clicking input)" in the log
         And I run :mode-leave
         # The actual check is already done above
         Then no crash should happen
 
-    @flaky  # FIXME https://github.com/qutebrowser/qutebrowser/issues/1525
     Scenario: Using :hint-follow inside a scrolled iframe
         When I open data/hints/iframe_scroll.html
+        And I wait for "* simple loaded" in the log
         And I hint with args "all normal" and follow a
         And I run :scroll bottom
         And I hint with args "links normal" and follow a
@@ -291,11 +294,13 @@ Feature: Using hints
 
     Scenario: Clicking on iframe with :hint all current
         When I open data/hints/iframe.html
+        And I wait for "* wrapped loaded" in the log
         And I hint with args "all current" and follow a
         Then no crash should happen
 
     Scenario: No error when hinting ranged input in frames
         When I open data/hints/issue3711_frame.html
+        And I wait for "* issue3711 loaded" in the log
         And I hint with args "all current" and follow a
         Then no crash should happen
 
diff --git a/tests/end2end/features/javascript.feature b/tests/end2end/features/javascript.feature
index 79b9e7d01..fb077d6f7 100644
--- a/tests/end2end/features/javascript.feature
+++ b/tests/end2end/features/javascript.feature
@@ -124,24 +124,25 @@ Feature: Javascript stuff
     # https://github.com/qutebrowser/qutebrowser/issues/1190
     # https://github.com/qutebrowser/qutebrowser/issues/2495
 
-    @flaky
     Scenario: Have a GreaseMonkey script run at page start
         When I have a GreaseMonkey file saved for document-start with noframes unset
         And I run :greasemonkey-reload
         And I open data/hints/iframe.html
+        And I wait for "* wrapped loaded" in the log
         Then the javascript message "Script is running on /data/hints/iframe.html" should be logged
 
     Scenario: Have a GreaseMonkey script running on frames
         When I have a GreaseMonkey file saved for document-end with noframes unset
         And I run :greasemonkey-reload
         And I open data/hints/iframe.html
+        And I wait for "* wrapped loaded" in the log
         Then the javascript message "Script is running on /data/hints/html/wrapped.html" should be logged
 
-    @flaky
     Scenario: Have a GreaseMonkey script running on noframes
         When I have a GreaseMonkey file saved for document-end with noframes set
         And I run :greasemonkey-reload
         And I open data/hints/iframe.html
+        And I wait for "* wrapped loaded" in the log
         Then the javascript message "Script is running on /data/hints/html/wrapped.html" should not be logged
 
     Scenario: Per-URL localstorage setting
diff --git a/tests/end2end/features/search.feature b/tests/end2end/features/search.feature
index d911a22af..9c477fd4e 100644
--- a/tests/end2end/features/search.feature
+++ b/tests/end2end/features/search.feature
@@ -363,20 +363,20 @@ Feature: Searching on a page
             - data/search.html
             - data/hello.txt (active)
 
-    # Too flaky
     @qtwebkit_skip: Not supported in qtwebkit @skip
     Scenario: Follow a searched link in an iframe
         When I open data/iframe_search.html
+        And I wait for "* search loaded" in the log
         And I run :tab-only
         And I run :search follow
         And I wait for "search found follow" in the log
         And I run :selection-follow
         Then "navigation request: url http://localhost:*/data/hello.txt, type Type.link_clicked, is_main_frame False" should be logged
 
-    # Too flaky
     @qtwebkit_skip: Not supported in qtwebkit @skip
     Scenario: Follow a tabbed searched link in an iframe
         When I open data/iframe_search.html
+        And I wait for "* search loaded" in the log
         And I run :tab-only
         And I run :search follow
         And I wait for "search found follow" in the log
author	toofar <toofar@spalge.com>	2023-03-25 15:57:17 +1300
committer	toofar <toofar@spalge.com>	2023-03-25 18:37:58 +1300
commit	cb0b51c09463f5c3a53fccfb12077f2934db214e (patch)
tree	0683f6c697620aa05be4055609faad729a54aa4f
parent	e03b81cb35f7433b63031cf08b1fbfb4a8a971e7 (diff)
download	qutebrowser-cb0b51c09463f5c3a53fccfb12077f2934db214e.tar.gz qutebrowser-cb0b51c09463f5c3a53fccfb12077f2934db214e.zip