202047 2019-09-20 08:02:20 -0700 [GTK] Several webdriver tests fail or timeout with Epiphany 2019-10-03 09:05:41 -0700 1 1 1 Unclassified WebKit WebDriver WebKit Nightly Build Unspecified Unspecified NEW P2 Normal --- 202530 1 clopez webkit-unassigned bburg bugs-noreply cgarcia oldest_to_newest 1572753 0 clopez 2019-09-20 08:02:20 -0700 I have been checking the WebDriver tests from WPT on Epiphany and there are several tests that fail or timeout on Epiphany, but work with Minibrowser. With MiniBrowser: $ ./wpt run --webkit-port gtk --binary=/usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/MiniBrowser --binary-arg=--automation \ --webdriver-binary=/usr/bin/WebKitWebDriver --webdriver-arg=--host=127.0.0.1 \ --log-html=results.html --log-mach=results.txt --log-wptreport=results.json --processes=8 \ webkit /webdriver [...] Ran 151 tests finished in 678.6 seconds. • 90 ran as expected. 0 tests skipped. • 11 tests timed out unexpectedly • 50 tests had unexpected subtest results With Epiphany: $ ./wpt run --webdriver-arg=--host=127.0.0.1 \ --log-html=results.html --log-mach=results.txt --log-wptreport=results.json --processes=8 \ epiphany /webdriver [...] Ran 151 tests finished in 680.2 seconds. • 73 ran as expected. 0 tests skipped. • 15 tests timed out unexpectedly • 63 tests had unexpected subtest results 1572757 1 379241 clopez 2019-09-20 08:11:00 -0700 Created attachment 379241 simplified test case to reproduce on of the issues To test this test case do the following: 1) On one terminal run: WebKitWebDriver --host=127.0.0.1 --port=4444 2) On another terminal execute this script like follows: For MiniBrowser test: ./simplified_test_case_input_text.sh minibrowser /path/to/MiniBrowser For Epihany test: ./simplified_test_case_input_text.sh epiphany /path/to/epiphany (Restart the WebKitWebDriver on terminal 1) between each test). The test passes if you see how the text "foo" is wrote in the input text field of the page that opens (in the browser). 1572758 2 clopez 2019-09-20 08:20:34 -0700 I have tried to run run-webdriver-tests with epiphany by modifying locally webdriver_driver_gtk.py, but it is not working at all. The epiphany window closes as soon as it opens and all the tests give an error. Not sure what is wrong with it. 1575948 3 clopez 2019-10-02 05:04:59 -0700 (In reply to Carlos Alberto Lopez Perez from comment #2) > I have tried to run run-webdriver-tests with epiphany by modifying locally > webdriver_driver_gtk.py, but it is not working at all. > The epiphany window closes as soon as it opens and all the tests give an > error. > Not sure what is wrong with it. This was due to to the test-runner trying to use the wrong browser name in the capabilities negotiation when the webdriver was starting. I have a patch that allows to run the webkit webdriver tests with epiphany: https://people.igalia.com/clopez/wkbug/202047/support-epiphany-webdriver-tests.patch The instructions about how to use it, are in the patch itself (in the commit log). With it I get the following results: MiniBrowser (r250519): https://people.igalia.com/clopez/wkbug/202047/minibrowser_results.txt Unexpected failures (61) Epiphany (3.34 + WebKit r250519): https://people.igalia.com/clopez/wkbug/202047/epiphany_results.txt Unexpected failures (84) Unexpected timeouts (10) 1576365 4 cgarcia 2019-10-03 05:24:05 -0700 The problem, at least one of them if there's more than one, is that epiphany keeps the focus on the location entry. Since the load is not triggered by ephy UI the focus is not transferred to the web view. I'll write a patch for ephy. 1576367 5 cgarcia 2019-10-03 05:33:26 -0700 https://gitlab.gnome.org/GNOME/epiphany/commit/cd77e3944e4ebb6789ae59153e0f66994613719f I pushed it to 3.34 branch too. 1576373 6 cgarcia 2019-10-03 06:06:57 -0700 Timeouts are gone, but I'm still getting more failures with ephy. I'm investigating. Some of them are related to the set/get window rect, there seems to be a race because they are flaky failures. 1576378 7 cgarcia 2019-10-03 06:11:06 -0700 When it fails: webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: POST /session/976322be-cd32-448e-9be5-37b03334cb78/window/rect {"width": 800, "height": 600} webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: {"value":{"x":26,"y":23,"width":1024,"height":768}} webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: GET /session/976322be-cd32-448e-9be5-37b03334cb78/window/rect webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: {"value":{"x":26,"y":23,"width":1024,"height":768}} When it works: webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: POST /session/c6e754a5-7ac7-490c-8b63-ae1f7df438cf/window/rect {"width": 800, "height": 600} webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: {"value":{"x":26,"y":23,"width":800,"height":600}} webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: GET /session/c6e754a5-7ac7-490c-8b63-ae1f7df438cf/window/rect webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: {"value":{"x":26,"y":23,"width":800,"height":600}} I think we assume that setting the window size is sync while it's not. Sometimes it takes a bit more and we get the previous size. We don't notice it in MB because 800x600 is already the default window size of MB. 379241 2019-09-20 08:11:00 -0700 2019-09-20 08:11:00 -0700 simplified test case to reproduce on of the issues simplified_test_case_input_text.sh text/plain 1979 clopez IyEvYmluL2Jhc2gKIwojIFRvIHJ1biB0aGlzIHRlc3QgZmlyc3Qgb3BlbiBXZWJLaXRXZWJEcml2 ZXIgdG8gbGlzdGVuIGluIDEyNy4wLjAuMTo0NDQKIyBFeGFtcGxlOgojICAgV2ViS2l0V2ViRHJp dmVyIC0taG9zdD0xMjcuMC4wLjEgLS1wb3J0PTQ0NDQKIyBUaGVuIHJ1biB0aGlzIHNjcmlwdCwg aWYgaXQgd29ya3MgeW91IHNob3VsZCBzZWUgdGhlIGJyb3dzZXIgb3BlbmluZyBhbmQgdGhlIHdv cmQgZm9vIHdyaXR0ZW4gb24gdGhlIHRleHQgaW5wdXQKIwoKaWYgWyAiJCMiIC1uZSAyIF07IHRo ZW4KICAgIGVjaG8gIkVSUk9SOiBpbmNvcnJlY3QgbnVtYmVyIG9mIHBhcmFtZXRlcnMiIDE+JjIK ICAgIGVjaG8gMT4mMgogICAgZWNobyAiICAqIEZvciBNaW5pQnJvd3NlciB0ZXN0IHVzZToiIDE+ JjIKICAgIGVjaG8gIiAgICAkezB9IG1pbmlicm93c2VyIC9wYXRoL3RvL01pbmlCcm93c2VyIiAx PiYyCiAgICBlY2hvIDE+JjIKICAgIGVjaG8gIiAgKiBGb3IgZXBpcGhhbnkgdGVzdCB1c2U6IiAx PiYyCiAgICBlY2hvICIgICAgJHswfSBlcGlwaGFueSAvcGF0aC90by9lcGlwaGFueSIgMT4mMgog ICAgZXhpdCAxCmZpCgpCUk9XU0VSTkFNRT0iJHsxfSIKQlJPV1NFUlBBVEg9IiR7Mn0iCgppZiAh IHRlc3QgLXggIiR7QlJPV1NFUlBBVEh9IjsgdGhlbgoJZWNobyAiRVJST1I6IENhbid0IGZpbmQg dGhlIGV4ZWN1dGFibGUgZm9yICR7QlJPV1NFUk5BTUV9IGF0ICR7QlJPV1NFUlBBVEh9IiAxPiYy CglleGl0IDEKZmkKCmlmIFtbICIke0JST1dTRVJOQU1FfSIgPT0gImVwaXBoYW55IiBdXTsgdGhl bgoJc2Vzc2lvbmlkPSIkKGN1cmwgLXMgLVggUE9TVCBodHRwOi8vMTI3LjAuMC4xOjQ0NDQvc2Vz c2lvbiAtZCAneyJjYXBhYmlsaXRpZXMiOiB7ImFsd2F5c01hdGNoIjogeyJicm93c2VyTmFtZSI6 ICJFcGlwaGFueSIsICJwbGF0Zm9ybU5hbWUiOiAiQU5ZIiwgIndlYmtpdGd0azpicm93c2VyT3B0 aW9ucyI6IHsiYmluYXJ5IjogIicke0JST1dTRVJQQVRIfSciLCAiYXJncyI6IFsiLS1hdXRvbWF0 aW9uLW1vZGUiXX19fX0nfGF3ayAtRiciJyAne3ByaW50ICQ2fScpIgplbGlmIFtbICIke0JST1dT RVJOQU1FfSIgPT0gIm1pbmlicm93c2VyIiBdXTsgdGhlbgoJc2Vzc2lvbmlkPSIkKGN1cmwgLXMg LVggUE9TVCBodHRwOi8vMTI3LjAuMC4xOjQ0NDQvc2Vzc2lvbiAtZCAneyJjYXBhYmlsaXRpZXMi OiB7ImFsd2F5c01hdGNoIjogeyJicm93c2VyTmFtZSI6ICJNaW5pQnJvd3NlciIsICJwbGF0Zm9y bU5hbWUiOiAiQU5ZIiwgIndlYmtpdGd0azpicm93c2VyT3B0aW9ucyI6IHsiYmluYXJ5IjogIick e0JST1dTRVJQQVRIfSciLCAiYXJncyI6IFsiLS1hdXRvbWF0aW9uIl19fX19J3xhd2sgLUYnIicg J3twcmludCAkNn0nKSIKZWxzZQoJZWNobyAiRVJST1I6IFVua25vd24gYnJvd3NlciAke0JST1dT RVJOQU1FfS4gVXNlIGVwaXBoYW55IG9yIG1pbmlicm93c2VyIiAxPiYyCglleGl0IDEKZmkKY3Vy bCAtcyAtWCBQT1NUICJodHRwOi8vMTI3LjAuMC4xOjQ0NDQvc2Vzc2lvbi8ke3Nlc3Npb25pZH0v dXJsIiAtZCAneyJ1cmwiOiAiaHR0cHM6Ly9wZW9wbGUuaWdhbGlhLmNvbS9jbG9wZXovd2tidWcv MjAyMDQ3L2lucHV0X3RleHQuaHRtbCJ9Jwpub2RlPSIkKGN1cmwgLXMgLVggUE9TVCAiaHR0cDov LzEyNy4wLjAuMTo0NDQ0L3Nlc3Npb24vJHtzZXNzaW9uaWR9L2VsZW1lbnQiIC1kICd7InVzaW5n IjogImNzcyBzZWxlY3RvciIsICJ2YWx1ZSI6ICJpbnB1dCJ9J3xhd2sgLUYnIicgJ3twcmludCAk Nn0nKSIKY3VybCAtcyAtWCBQT1NUICJodHRwOi8vMTI3LjAuMC4xOjQ0NDQvc2Vzc2lvbi8ke3Nl c3Npb25pZH0vZWxlbWVudC8ke25vZGV9L3ZhbHVlIiAtZCAneyJ0ZXh0IjogImZvbyJ9JwplY2hv CmVjaG8gIlRlc3QgZmluaXNoZWQuIFRoZSB0ZXN0IGhhcyBwYXNzZWQgaWYgeW91IHNlZSB0aGUg d29yZCBcImZvb1wiIG9uIHRoZSBpbnB1dCB0eXBlIHRleHQgZm9vIgo=