Bug 281747 – REGRESSION(285310@main): [GTK] Broke WPT tests with WebDriver.

• Status: NEW
• Component: WebKitGTK

[Carlos Alberto Lopez Perez]

Commit 285310@main caused a regresion on the GTK port, making the WPT tests (run with the official WPT runner) to not work anymore with WebDriver.

I have bisected this, so I'm sure this one is the commit introducing the regression.
Regarding the reason, I can speculate that perhaps is related to opening new JavaScript windows, something that WPT tests do. But currently I have no idea.


To reproduce this you can do the following:


    git clone  https://github.com/web-platform-tests/wpt --depth 1
    cd wpt
    ./wpt run --webdriver-binary=/path/to/WebKitWebDriver --binary=/path/to/MiniBrowser  --processes=16 --log-mach=-  webkitgtk_minibrowser /css/css-multicol
    # Note: to speed up testing i'm only running the  /css/css-multicol tests which is enough to reproduce the problem, but this happens with other tests as well


This is the output of that wpt command from above:

 1) with WebKitGTK from 285309@main -> https://paste.rs/epekN
 2) with WebKitGTK from 285310@main -> https://paste.rs/jVbMq


On the output of 285310@main you can see lot of timeouts that don't happen with 285309@main and also you can see this at the end:


285309@main:
	web-platform-test
	~~~~~~~~~~~~~~~~~
	Ran 1662 checks (1060 subtests, 602 tests)
	Expected results: 1523
	Unexpected results: 139
	  test: 118 (118 fail)
	  subtest: 21 (21 fail)

 
285310@main:
	web-platform-test
	~~~~~~~~~~~~~~~~~
	Ran 602 checks (602 tests)
	Expected results: 411
	Unexpected results: 191
	  test: 191 (118 fail, 73 timeout)



Notice how the subtests are not run anymore in 285310@main. Also with 285310@main you get lot of warnings like this one:

** (MiniBrowser:1649843): WARNING **: 14:31:51.402: WebKitWebView returned by WebKitWebView::create signal was not created with the related WebKitWebView


Note: I don't know if this affects WPE also, I didn't tested.

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #0)
> 
> Note: I don't know if this affects WPE also, I didn't tested.


just tested it now: WPE is not affected, only WebKitGTK

[Carlos Alberto Lopez Perez]

On WPT upstream landed recently this change: https://github.com/web-platform-tests/wpt/commit/bef892c48976038b207f3aedf4088a5552cbae67 

Since that change landed the issue is not longer reproducible, however it seems that the above change affected the Safari WebDriver tests causing lots of timeouts.

That issue with the Safari WebDriver was later fixed in https://github.com/web-platform-tests/wpt/commit/70d707faa9a6dca431f89fa0175970c6f0574f31

However, since that last fix for the Safari WebDriver (70d707f) I can observe this difference of behaviour since 285309@main and 285310@main with webkitgtk_minibrowser


# On WPT commit 70d707f with webkitgtk_minibrowser build at 285309@main
Full log: https://paste.rs/OzhRt

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 1201 checks (596 subtests, 605 tests)
Expected results: 1026
Unexpected results: 175
  test: 172 (53 crash, 119 fail)
  subtest: 3 (3 fail)


# On WPT commit 70d707f with webkitgtk_minibrowser build at 285310@main
Full log: https://paste.rs/ZONOH

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 905 checks (300 subtests, 605 tests)
Expected results: 724
Unexpected results: 181
  test: 167 (48 crash, 119 fail)
  subtest: 14 (14 fail)


So as you see there is some difference on the number of subtests run.

Not sure what to conclude about this.


But if you want to reproduce the original issue be sure the checkout WPT commit bef892c^ (the previous one to bef892c that fixes/workarounds the issue)

[Alex Christensen]

Does the affected configuration have PSON enabled?

[Carlos Alberto Lopez Perez]

(In reply to Alex Christensen from comment #3)
> Does the affected configuration have PSON enabled?

yes, we made PSON mandatory when using WebKitGTK with GTK4 (which is since a while the default release nightly build)

However i quickly tried to disable PSON here in a local build and the behavior seems identical than before

[Carlos Alberto Lopez Perez]

Commit https://commits.webkit.org/286275@main removed that warning but the underlying issue still remains.

You not longer see the warning "WebKitWebView returned by WebKitWebView::create signal was not created with the related WebKitWebView" but the timeouts and the failure to run many of the subtests is still there. 


I just tested on WPT 7e47e52d5496ce246ca27fa395cceaa0df8169b0 (master of today) plus this WIP patch to add a WPT wpewebkit_minibrowser driver https://paste.rs/lYSJF (I will be sending a PR there this week with it).


1) With WebKitGTK

$ ./wpt run -y --install-browser  --processes=16 --log-mach=-  webkitgtk_minibrowser /css/css-multicol
[...]
 0:14.61 INFO WebKitGTK MiniBrowser bundle for channel nightly installed: WebKitGTK 2.47.1 (286783@main)
[...]
 6:39.70 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 819 checks (214 subtests, 605 tests)
Expected results: 642
Unexpected results: 177
  test: 170 (51 crash, 119 fail)
  subtest: 7 (7 fail)


Full log: https://paste.rs/9F7pz




2) With WPE

$ ./wpt run -y --install-browser --headless  --processes=16 --log-mach=-  wpewebkit_minibrowser /css/css-multicol
[...]
 0:13.25 INFO WPE WebKit MiniBrowser bundle for channel nightly installed: WPE WebKit 2.47.1 (286783@main)
[...]
 5:11.84 SUITE_END

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 1669 checks (1064 subtests, 605 tests)
Expected results: 1520
Unexpected results: 149
  test: 118 (118 fail)
  subtest: 31 (31 fail)


Full log: https://paste.rs/vyRVF



As you can see the run with WPE took a minute less and ran way more subtests, achieving in the end more than double of expected results (1520 vs 642)

[Michael Catanzaro]

(286275@main definitely fixed bug #282409, though, so there must be multiple problems here.)

[Carlos Alberto Lopez Perez]

I think I know what causes that error of `Exception: the window remained open after sending the window close command` that comes from WPT.


I was checking to run tests now with webkitgtk and WPT and I noticed that sometimes when the minibrowser window opens to start a test then two tabs appear on it
- One that says "MainThread" and is empty, and with the webinspector I can see it has window.location.href==http://web-platform.test:8000/testharness_runner.html
- Another one with the WPT test itself

And the one that is focused is the "MainThread" one

See screenshot: https://people.igalia.com/clopez/wkbug/281747/minibrowser_wpt_tabs.png

So if you do nothing then the test timeouts.

But if you focus or click on the tab with the WPT test then it passes. Alternatively if you close the MainThread tab (which forces a focus on the other one) also passes.


This easily reproducible running several times the acid tests (because those have a higher timeout value and allows you to check things before the wpt timeout triggers)

Run the command below several times until you find this MainThread tab on one of the MiniBrowsers that oppen and then see how it timeouts unless you focus the tab with the tests

./wpt run --log-mach=- --install-browser -y webkitgtk_minibrowser acid

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #5)
> Commit https://commits.webkit.org/286275@main removed that warning but the
> underlying issue still remains.
> 
> You not longer see the warning "WebKitWebView returned by
> WebKitWebView::create signal was not created with the related WebKitWebView"
> but the timeouts and the failure to run many of the subtests is still there. 
> 
> 
> I just tested on WPT 7e47e52d5496ce246ca27fa395cceaa0df8169b0 (master of
> today) plus this WIP patch to add a WPT wpewebkit_minibrowser driver
> https://paste.rs/lYSJF (I will be sending a PR there this week with it).

just sent the WPT PR here: https://github.com/web-platform-tests/wpt/pull/49268

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #7)
> I think I know what causes that error of `Exception: the window remained
> open after sending the window close command` that comes from WPT.
> 
> 
> I was checking to run tests now with webkitgtk and WPT and I noticed that
> sometimes when the minibrowser window opens to start a test then two tabs
> appear on it
> - One that says "MainThread" and is empty, and with the webinspector I can
> see it has
> window.location.href==http://web-platform.test:8000/testharness_runner.html
> - Another one with the WPT test itself
> 
> And the one that is focused is the "MainThread" one
> 
> See screenshot:
> https://people.igalia.com/clopez/wkbug/281747/minibrowser_wpt_tabs.png
> 
> So if you do nothing then the test timeouts.
> 
> But if you focus or click on the tab with the WPT test then it passes.
> Alternatively if you close the MainThread tab (which forces a focus on the
> other one) also passes.
> 
> 
> This easily reproducible running several times the acid tests (because those
> have a higher timeout value and allows you to check things before the wpt
> timeout triggers)
> 
> Run the command below several times until you find this MainThread tab on
> one of the MiniBrowsers that oppen and then see how it timeouts unless you
> focus the tab with the tests
> 
> ./wpt run --log-mach=- --install-browser -y webkitgtk_minibrowser acid

I think WebKitGTK MiniBrowser has an issue with the focus of new tabs that WPT now uses by default.
IIRC previously WPT was opening new windows instead of new tabs when starting a new test
I reported the issue here https://bugs.webkit.org/show_bug.cgi?id=283392 along with a simplified test case