From cb0b51c09463f5c3a53fccfb12077f2934db214e Mon Sep 17 00:00:00 2001 From: toofar Date: Sat, 25 Mar 2023 15:57:17 +1300 Subject: tests: wait for flaky iframes to load via JS TL;DR: I think I stabilized a couple of hinting iframe tests and applied that change to all the iframe usages in the end2end tests in the hopes that it would resolve some other flaky tests. I was facing some tests that hinted elements in iframes failing intermittently recently. They were most consistently failing on the windows runner. This is similar to the trend described in the comment linked from #1525. The tests failing recently where: tests (py39-pyqt515, windows-2019, 3.9) test_using_hintfollow_inside_an_iframe test_using_hintfollow_inside_an_iframe_button tests (py311-pyqt515, ubuntu-20.04, 3.11) test_using_hintfollow_inside_an_iframe test_using_hintfollow_inside_an_iframe_button They are failing because hints don't get generated for the elements in the iframe. I can see hints do get generated for the iframe itself though. Examining the logs in 5.15 it seems that there is a `cur_load_finished(True) (tab *)` log line after we run :hint. I suspect that this is the load finished signal for the iframe. I attempted to change the bdd open_path function to wait for the current message it's looking for and then additionally look for that cur_load_finished signal, but then it starts failing locally when the iframe load signal actually comes before the parent frame one. Just looking for the cur_load_finished signal itself also always found it immediately ("already found"), probably from the parent frame. So instead of trying to deal with that indeterminate ordering or trying to change the signals to say what frame they are coming from, I added javascript to all the pages used in iframes that run on load, and changed all the tests to watch for log messages from the JS. It's not the most maintainable solution, perhaps if we generated our test files with jinja we could have some "subframe loaded" boilerplate, get a message to log from a query param, look at metadata from test files for the open_path checking etc. I then searched for all the iframe usages in the test data and applied that change to all of those files and all the tests that used them. A few of those tests also had `flaky` annotations on them. I removed those annotations for now in the hopes that there were affected by the same problem. And that I actually managed to work around it correctly. ref: #7621 --- tests/end2end/data/hints/html/wrapped.html | 2 +- tests/end2end/data/hints/html/wrapped_button.html | 2 +- tests/end2end/data/hints/input.html | 1 + tests/end2end/data/hints/issue3711.html | 2 +- tests/end2end/data/scroll/simple.html | 2 +- tests/end2end/data/search.html | 2 +- tests/end2end/features/hints.feature | 7 ++++++- tests/end2end/features/javascript.feature | 5 +++-- tests/end2end/features/search.feature | 4 ++-- 9 files changed, 17 insertions(+), 10 deletions(-) diff --git a/tests/end2end/data/hints/html/wrapped.html b/tests/end2end/data/hints/html/wrapped.html index dcc05c8c7..89dceafa4 100644 --- a/tests/end2end/data/hints/html/wrapped.html +++ b/tests/end2end/data/hints/html/wrapped.html @@ -7,7 +7,7 @@ Link wrapped across multiple lines - +
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
diff --git a/tests/end2end/data/hints/html/wrapped_button.html b/tests/end2end/data/hints/html/wrapped_button.html index 4d5fd81bb..9ce1a28a3 100644 --- a/tests/end2end/data/hints/html/wrapped_button.html +++ b/tests/end2end/data/hints/html/wrapped_button.html @@ -7,7 +7,7 @@ Button link wrapped across multiple lines - +
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
diff --git a/tests/end2end/data/hints/input.html b/tests/end2end/data/hints/input.html index a81361281..c64503ada 100644 --- a/tests/end2end/data/hints/input.html +++ b/tests/end2end/data/hints/input.html @@ -6,6 +6,7 @@ Simple input - + Just a link

Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Vestibulum tortor quam, feugiat vitae, ultricies eget, tempor sit amet, ante. Donec eu libero sit amet quam egestas semper. Aenean ultricies mi vitae est. Mauris placerat eleifend leo. Quisque sit amet est et sapien ullamcorper pharetra. Vestibulum erat wisi, condimentum sed, commodo vitae, ornare sit amet, wisi. Aenean fermentum, elit eget tincidunt condimentum, eros ipsum rutrum orci, sagittis tempus lacus enim ac dui. Donec non enim in turpis pulvinar facilisis. Ut felis. Praesent dapibus, neque id cursus faucibus, tortor neque egestas augue, eu vulputate magna eros eu erat. Aliquam erat volutpat. Nam dui mi, tincidunt quis, accumsan porttitor, facilisis luctus, metus

diff --git a/tests/end2end/data/search.html b/tests/end2end/data/search.html index 1064a7d01..a43cd89a9 100644 --- a/tests/end2end/data/search.html +++ b/tests/end2end/data/search.html @@ -4,7 +4,7 @@ Searching text on the page - +

foo
Foo
diff --git a/tests/end2end/features/hints.feature b/tests/end2end/features/hints.feature index 51ffb980a..2d597da13 100644 --- a/tests/end2end/features/hints.feature +++ b/tests/end2end/features/hints.feature @@ -251,25 +251,28 @@ Feature: Using hints ### iframes Scenario: Using :hint-follow inside an iframe When I open data/hints/iframe.html + And I wait for "* wrapped loaded" in the log And I hint with args "links normal" and follow a Then "navigation request: url http://localhost:*/data/hello.txt, type Type.link_clicked, *" should be logged Scenario: Using :hint-follow inside an iframe button When I open data/hints/iframe_button.html + And I wait for "* wrapped_button loaded" in the log And I hint with args "all normal" and follow s Then "navigation request: url http://localhost:*/data/hello.txt, *" should be logged Scenario: Hinting inputs in an iframe without type When I open data/hints/iframe_input.html + And I wait for "* input loaded" in the log And I hint with args "inputs" and follow a And I wait for "Entering mode KeyMode.insert (reason: clicking input)" in the log And I run :mode-leave # The actual check is already done above Then no crash should happen - @flaky # FIXME https://github.com/qutebrowser/qutebrowser/issues/1525 Scenario: Using :hint-follow inside a scrolled iframe When I open data/hints/iframe_scroll.html + And I wait for "* simple loaded" in the log And I hint with args "all normal" and follow a And I run :scroll bottom And I hint with args "links normal" and follow a @@ -291,11 +294,13 @@ Feature: Using hints Scenario: Clicking on iframe with :hint all current When I open data/hints/iframe.html + And I wait for "* wrapped loaded" in the log And I hint with args "all current" and follow a Then no crash should happen Scenario: No error when hinting ranged input in frames When I open data/hints/issue3711_frame.html + And I wait for "* issue3711 loaded" in the log And I hint with args "all current" and follow a Then no crash should happen diff --git a/tests/end2end/features/javascript.feature b/tests/end2end/features/javascript.feature index 79b9e7d01..fb077d6f7 100644 --- a/tests/end2end/features/javascript.feature +++ b/tests/end2end/features/javascript.feature @@ -124,24 +124,25 @@ Feature: Javascript stuff # https://github.com/qutebrowser/qutebrowser/issues/1190 # https://github.com/qutebrowser/qutebrowser/issues/2495 - @flaky Scenario: Have a GreaseMonkey script run at page start When I have a GreaseMonkey file saved for document-start with noframes unset And I run :greasemonkey-reload And I open data/hints/iframe.html + And I wait for "* wrapped loaded" in the log Then the javascript message "Script is running on /data/hints/iframe.html" should be logged Scenario: Have a GreaseMonkey script running on frames When I have a GreaseMonkey file saved for document-end with noframes unset And I run :greasemonkey-reload And I open data/hints/iframe.html + And I wait for "* wrapped loaded" in the log Then the javascript message "Script is running on /data/hints/html/wrapped.html" should be logged - @flaky Scenario: Have a GreaseMonkey script running on noframes When I have a GreaseMonkey file saved for document-end with noframes set And I run :greasemonkey-reload And I open data/hints/iframe.html + And I wait for "* wrapped loaded" in the log Then the javascript message "Script is running on /data/hints/html/wrapped.html" should not be logged Scenario: Per-URL localstorage setting diff --git a/tests/end2end/features/search.feature b/tests/end2end/features/search.feature index d911a22af..9c477fd4e 100644 --- a/tests/end2end/features/search.feature +++ b/tests/end2end/features/search.feature @@ -363,20 +363,20 @@ Feature: Searching on a page - data/search.html - data/hello.txt (active) - # Too flaky @qtwebkit_skip: Not supported in qtwebkit @skip Scenario: Follow a searched link in an iframe When I open data/iframe_search.html + And I wait for "* search loaded" in the log And I run :tab-only And I run :search follow And I wait for "search found follow" in the log And I run :selection-follow Then "navigation request: url http://localhost:*/data/hello.txt, type Type.link_clicked, is_main_frame False" should be logged - # Too flaky @qtwebkit_skip: Not supported in qtwebkit @skip Scenario: Follow a tabbed searched link in an iframe When I open data/iframe_search.html + And I wait for "* search loaded" in the log And I run :tab-only And I run :search follow And I wait for "search found follow" in the log -- cgit v1.2.3-54-g00ecf