Bug 58536 – [Qt][WK2] Qt should use the mac-wk2 skipped list

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Balazs Kelemen]

To avoid skipped list maintenance burden we should use the mac-wk2 list additionally to our own's as the Apple Windows bots do.
Nearly all of the skipped tests for WebKit2 on Mac are due to cross platform issues so we won't miss many tests that we could pass (if we even miss any).

[Balazs Kelemen]

Current bot results:
16027 test cases (99%) succeeded
6 test cases (<1%) were new
17 test cases (<1%) had stderr output

Results with the mac-wk2 list - runned in an environment that is mostly identical with the bot
15410 test cases (99%) succeeded
1 test case (<1%) had incorrect layout
6 test cases (<1%) were new
17 test cases (<1%) had stderr output

So we have almost 600 passing test that we won't run with the change.

[Balazs Kelemen]

Robert, have you got an idea about which tests are we running that are skipped on Mac? I guess we run those tests with DRT as well.

[Balazs Kelemen]

My bad. Locally I ran the tests with --no-http. 
With this configuration, these are results without the mac skiplist:
15463 test cases (99%) succeeded
6 test cases (<1%) had incorrect layout
22 test cases (<1%) were new
1 test case (<1%) crashed
15 test cases (<1%) had stderr output

... and with it:
15428 test cases (99%) succeeded
5 test cases (<1%) had incorrect layout
21 test cases (<1%) were new
15 test cases (<1%) had stderr output

So we have 36 passing test that we would skip with the change.
These are the following tests:
dom/html/level2/html/HTMLAppletElement01.html
dom/html/level2/html/HTMLAppletElement02.html
dom/html/level2/html/HTMLAppletElement03.html
dom/html/level2/html/HTMLAppletElement04.html
dom/html/level2/html/HTMLAppletElement05.html
dom/html/level2/html/HTMLAppletElement06.html
dom/html/level2/html/HTMLAppletElement07.html
dom/html/level2/html/HTMLAppletElement08.html
dom/html/level2/html/HTMLAppletElement09.html
dom/html/level2/html/HTMLAppletElement10.html
dom/xhtml/level2/html/HTMLAppletElement01.xhtml
dom/xhtml/level2/html/HTMLAppletElement02.xhtml
dom/xhtml/level2/html/HTMLAppletElement03.xhtml
dom/xhtml/level2/html/HTMLAppletElement04.xhtml
dom/xhtml/level2/html/HTMLAppletElement05.xhtml
dom/xhtml/level2/html/HTMLAppletElement06.xhtml
dom/xhtml/level2/html/HTMLAppletElement07.xhtml
dom/xhtml/level2/html/HTMLAppletElement08.xhtml
dom/xhtml/level2/html/HTMLAppletElement09.xhtml
dom/xhtml/level2/html/HTMLAppletElement10.xhtml
editing/selection/click-below-rtl-text.html
fast/block/basic/truncation-rtl.html
fast/dom/global-constructors.html
fast/dom/timer-increase-min-interval-and-reset-part-2.html
fast/dom/StyleSheet/detached-style-2.html
fast/dom/StyleSheet/detached-style-pi-2.xhtml
fast/dom/StyleSheet/detached-style-pi.xhtml
fast/dom/StyleSheet/detached-style.html
fast/dom/StyleSheet/detached-stylesheet-without-wrapper.html
fast/forms/select-live-pseudo-selectors.html
fast/history/history-replace-updates-current-item.html
fast/history/same-document-iframes-changing-fragment.html
fast/history/same-document-iframes-changing-pushstate.html
fast/images/animated-gif-restored-from-bfcache.html
fast/text/zero-font-size.html
svg/css/cursor-replace.svg

We should check these tests on Mac to see how many of them is passing and remove them from the skiplist but I don't think it should block the change of using the mac skiplist.

[Balazs Kelemen]

Created attachment 90679 [details]
Patch

[Csaba Osztrogonác]

Comment on attachment 90679 [details]
Patch

r=me

[Csaba Osztrogonác]

After this patch landed, could you remove unnecessary files from qt-wk2 skipped list?

[WebKit Commit Bot]

Comment on attachment 90679 [details]
Patch

Clearing flags on attachment: 90679

Committed r84615: <http://trac.webkit.org/changeset/84615>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Balazs Kelemen]

(In reply to comment #6)
> After this patch landed, could you remove unnecessary files from qt-wk2 skipped list?

I don't think that we should remove those. In the future when WebKit2 and the Qt WebKit2 port will be more stable we would likely want to have our own skip list again. Creating our skip list at that time could be easier if we leave these items.