Bug 202047 – [GTK] Several webdriver tests fail or timeout with Epiphany

• Status: NEW
• Component: WebDriver

[Carlos Alberto Lopez Perez]

I have been checking the WebDriver tests from WPT on Epiphany and there are several tests that fail or timeout on Epiphany, but work with Minibrowser.


With MiniBrowser:

$ ./wpt run --webkit-port gtk --binary=/usr/lib/x86_64-linux-gnu/webkit2gtk-4.0/MiniBrowser --binary-arg=--automation \
 --webdriver-binary=/usr/bin/WebKitWebDriver --webdriver-arg=--host=127.0.0.1 \
 --log-html=results.html --log-mach=results.txt --log-wptreport=results.json --processes=8 \
 webkit /webdriver

[...]

Ran 151 tests finished in 678.6 seconds.
  • 90 ran as expected. 0 tests skipped.
  • 11 tests timed out unexpectedly
  • 50 tests had unexpected subtest results

With Epiphany:

$ ./wpt run --webdriver-arg=--host=127.0.0.1 \
 --log-html=results.html --log-mach=results.txt --log-wptreport=results.json --processes=8 \
  epiphany /webdriver

[...]

Ran 151 tests finished in 680.2 seconds.
  • 73 ran as expected. 0 tests skipped.
  • 15 tests timed out unexpectedly
  • 63 tests had unexpected subtest results

[Carlos Alberto Lopez Perez]

Created attachment 379241 [details]
simplified test case to reproduce on of the issues

To test this test case do the following:

1) On one terminal run: WebKitWebDriver --host=127.0.0.1 --port=4444
2) On another terminal execute this script like follows:

For MiniBrowser test:
./simplified_test_case_input_text.sh minibrowser /path/to/MiniBrowser
For Epihany test:
./simplified_test_case_input_text.sh epiphany /path/to/epiphany

(Restart the WebKitWebDriver on terminal 1) between each test).

The test passes if you see how the text "foo" is wrote in the input text field of the page that opens (in the browser).

[Carlos Alberto Lopez Perez]

I have tried to run run-webdriver-tests with epiphany by modifying locally webdriver_driver_gtk.py, but it is not working at all.
The epiphany window closes as soon as it opens and all the tests give an error.
Not sure what is wrong with it.

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #2)
> I have tried to run run-webdriver-tests with epiphany by modifying locally
> webdriver_driver_gtk.py, but it is not working at all.
> The epiphany window closes as soon as it opens and all the tests give an
> error.
> Not sure what is wrong with it.

This was due to to the test-runner trying to use the wrong browser name in the capabilities negotiation when the webdriver was starting.

I have a patch that allows to run the webkit webdriver tests with epiphany: https://people.igalia.com/clopez/wkbug/202047/support-epiphany-webdriver-tests.patch The instructions about how to use it, are in the patch itself (in the commit log).


With it I get the following results:

MiniBrowser (r250519): https://people.igalia.com/clopez/wkbug/202047/minibrowser_results.txt
Unexpected failures (61)

Epiphany (3.34 + WebKit r250519): https://people.igalia.com/clopez/wkbug/202047/epiphany_results.txt
Unexpected failures (84)
Unexpected timeouts (10)

[Carlos Garcia Campos]

The problem, at least one of them if there's more than one, is that epiphany keeps the focus on the location entry. Since the load is not triggered by ephy UI the focus is not transferred to the web view. I'll write a patch for ephy.

[Carlos Garcia Campos]

https://gitlab.gnome.org/GNOME/epiphany/commit/cd77e3944e4ebb6789ae59153e0f66994613719f

I pushed it to 3.34 branch too.

[Carlos Garcia Campos]

Timeouts are gone, but I'm still getting more failures with ephy. I'm investigating. Some of them are related to the set/get window rect, there seems to be a race because they are flaky failures.

[Carlos Garcia Campos]

When it fails:

webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: POST /session/976322be-cd32-448e-9be5-37b03334cb78/window/rect {"width": 800, "height": 600}
webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: {"value":{"x":26,"y":23,"width":1024,"height":768}}
webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: GET /session/976322be-cd32-448e-9be5-37b03334cb78/window/rect
webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16220): DBG: {"value":{"x":26,"y":23,"width":1024,"height":768}}

When it works:

webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: POST /session/c6e754a5-7ac7-490c-8b63-ae1f7df438cf/window/rect {"width": 800, "height": 600}
webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: {"value":{"x":26,"y":23,"width":800,"height":600}}
webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: GET /session/c6e754a5-7ac7-490c-8b63-ae1f7df438cf/window/rect
webkitpy.webdriver_tests.webdriver_w3c_executor: [DEBUG] (WebKitWebDriver:16139): DBG: {"value":{"x":26,"y":23,"width":800,"height":600}}

I think we assume that setting the window size is sync while it's not. Sometimes it takes a bit more and we get the previous size. We don't notice it in MB because 800x600 is already the default window size of MB.