Bug 178589 - Rebase WPT tests up to a1c0107
Summary: Rebase WPT tests up to a1c0107
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-10-20 09:12 PDT by youenn fablet
Modified: 2017-11-15 13:15 PST (History)
7 users (show)

See Also:


Attachments
Patch (983.77 KB, patch)
2017-10-20 09:15 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews100 for mac-elcapitan (1.02 MB, application/zip)
2017-10-20 10:23 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews125 for ios-simulator-wk2 (15.97 MB, application/zip)
2017-10-20 10:40 PDT, Build Bot
no flags Details
Archive of layout-test-results from ews115 for mac-elcapitan (1.91 MB, application/zip)
2017-10-20 10:52 PDT, Build Bot
no flags Details
Patch (984.85 KB, patch)
2017-10-20 12:26 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Patch (984.85 KB, patch)
2017-10-20 13:36 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ews126 for ios-simulator-wk2 (10.85 MB, application/zip)
2017-10-20 15:31 PDT, Build Bot
no flags Details
Patch (1.99 KB, patch)
2017-10-29 16:22 PDT, youenn fablet
no flags Details | Formatted Diff | Diff
Patch (2.19 KB, patch)
2017-10-29 16:34 PDT, youenn fablet
cdumez: commit-queue-
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description youenn fablet 2017-10-20 09:12:56 PDT
Rebase WPT tests up to a1c0107
Comment 1 youenn fablet 2017-10-20 09:15:18 PDT
Created attachment 324407 [details]
Patch
Comment 2 Build Bot 2017-10-20 09:23:20 PDT
Attachment 324407 [details] did not pass style-queue:


ERROR: /Volumes/Data/StyleQueue/WebKit/LayoutTests/platform/mac/TestExpectations:1784:  Path does not exist.  [test/expectations] [5]
Total errors found: 1 in 363 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 3 Build Bot 2017-10-20 10:23:39 PDT
Comment on attachment 324407 [details]
Patch

Attachment 324407 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/4934220

New failing tests:
imported/w3c/web-platform-tests/html/browsers/browsing-the-web/history-traversal/window-name-after-cross-origin-main-frame-navigation.sub.html
Comment 4 Build Bot 2017-10-20 10:23:40 PDT
Created attachment 324414 [details]
Archive of layout-test-results from ews100 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-elcapitan  Platform: Mac OS X 10.11.6
Comment 5 Build Bot 2017-10-20 10:40:23 PDT
Comment on attachment 324407 [details]
Patch

Attachment 324407 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/4934096

New failing tests:
imported/w3c/web-platform-tests/html/browsers/browsing-the-web/history-traversal/window-name-after-cross-origin-main-frame-navigation.sub.html
imported/w3c/web-platform-tests/webrtc/RTCPeerConnection-ontrack.https.html
Comment 6 Build Bot 2017-10-20 10:40:24 PDT
Created attachment 324415 [details]
Archive of layout-test-results from ews125 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125  Port: ios-simulator-wk2  Platform: Mac OS X 10.12.6
Comment 7 Build Bot 2017-10-20 10:52:34 PDT
Comment on attachment 324407 [details]
Patch

Attachment 324407 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/4934322

New failing tests:
imported/w3c/web-platform-tests/html/browsers/browsing-the-web/history-traversal/window-name-after-cross-origin-main-frame-navigation.sub.html
Comment 8 Build Bot 2017-10-20 10:52:35 PDT
Created attachment 324416 [details]
Archive of layout-test-results from ews115 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-elcapitan  Platform: Mac OS X 10.11.6
Comment 9 youenn fablet 2017-10-20 12:26:11 PDT
Created attachment 324427 [details]
Patch
Comment 10 Build Bot 2017-10-20 12:35:43 PDT
Attachment 324427 [details] did not pass style-queue:


ERROR: /Volumes/Data/StyleQueue/WebKit/LayoutTests/platform/mac/TestExpectations:1784:  Path does not exist.  [test/expectations] [5]
Total errors found: 1 in 363 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 11 youenn fablet 2017-10-20 13:36:23 PDT
Created attachment 324441 [details]
Patch
Comment 12 Build Bot 2017-10-20 13:45:17 PDT
Attachment 324441 [details] did not pass style-queue:


ERROR: /Volumes/Data/StyleQueue/WebKit/LayoutTests/platform/mac/TestExpectations:1784:  Path does not exist.  [test/expectations] [5]
Total errors found: 1 in 363 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 13 Build Bot 2017-10-20 15:31:23 PDT
Comment on attachment 324441 [details]
Patch

Attachment 324441 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/4937064

New failing tests:
imported/w3c/web-platform-tests/webrtc/RTCPeerConnection-ontrack.https.html
Comment 14 Build Bot 2017-10-20 15:31:25 PDT
Created attachment 324450 [details]
Archive of layout-test-results from ews126 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-wk2  Platform: Mac OS X 10.12.6
Comment 15 WebKit Commit Bot 2017-10-28 20:02:46 PDT
Comment on attachment 324441 [details]
Patch

Clearing flags on attachment: 324441

Committed r224156: <https://trac.webkit.org/changeset/224156>
Comment 16 WebKit Commit Bot 2017-10-28 20:02:47 PDT
All reviewed patches have been landed.  Closing bug.
Comment 17 Chris Dumez 2017-10-29 11:47:35 PDT
(In reply to WebKit Commit Bot from comment #16)
> All reviewed patches have been landed.  Closing bug.

Some service worker tests have wrong expectations or no expected results file:
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r224158%20(5354)/results.html
Comment 18 youenn fablet 2017-10-29 16:17:21 PDT
(In reply to Chris Dumez from comment #17)
> (In reply to WebKit Commit Bot from comment #16)
> > All reviewed patches have been landed.  Closing bug.
> 
> Some service worker tests have wrong expectations or no expected results
> file:
> https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/
> r224158%20(5354)/results.html

Ah, probably need to skip these since the folder is no longer skipped as a whole.
I'll update test expectations.
Comment 19 youenn fablet 2017-10-29 16:22:21 PDT
Reopening to attach new patch.
Comment 20 youenn fablet 2017-10-29 16:22:22 PDT
Created attachment 325296 [details]
Patch
Comment 21 Chris Dumez 2017-10-29 16:24:21 PDT
Comment on attachment 325296 [details]
Patch

Only one of them is timing out. Why skip all of them? I am trying to unskip as many tests as possible. This is counter productive.
Comment 22 youenn fablet 2017-10-29 16:32:30 PDT
(In reply to Chris Dumez from comment #21)
> Comment on attachment 325296 [details]
> Patch
> 
> Only one of them is timing out. Why skip all of them? I am trying to unskip
> as many tests as possible. This is counter productive.

Looking at the results, these tests are not yet able to produce something really meaningful. Until we have more service worker infrastructure, running them or not running them does not change a lot, all of these tests basically cover the same missing features.

Unskipping them once we have more infra is surely our first priority.
I am guessing that we will be able to run some timing out tests at about the same time.
Comment 23 Chris Dumez 2017-10-29 16:33:28 PDT
(In reply to youenn fablet from comment #22)
> (In reply to Chris Dumez from comment #21)
> > Comment on attachment 325296 [details]
> > Patch
> > 
> > Only one of them is timing out. Why skip all of them? I am trying to unskip
> > as many tests as possible. This is counter productive.
> 
> Looking at the results, these tests are not yet able to produce something
> really meaningful. Until we have more service worker infrastructure, running
> them or not running them does not change a lot, all of these tests basically
> cover the same missing features.
> 
> Unskipping them once we have more infra is surely our first priority.
> I am guessing that we will be able to run some timing out tests at about the
> same time.

It is much better if we need to rebaseline tests once we implement something rather than looking for skipped tests to unskip :(((
Comment 24 Chris Dumez 2017-10-29 16:33:42 PDT
(In reply to Chris Dumez from comment #23)
> (In reply to youenn fablet from comment #22)
> > (In reply to Chris Dumez from comment #21)
> > > Comment on attachment 325296 [details]
> > > Patch
> > > 
> > > Only one of them is timing out. Why skip all of them? I am trying to unskip
> > > as many tests as possible. This is counter productive.
> > 
> > Looking at the results, these tests are not yet able to produce something
> > really meaningful. Until we have more service worker infrastructure, running
> > them or not running them does not change a lot, all of these tests basically
> > cover the same missing features.
> > 
> > Unskipping them once we have more infra is surely our first priority.
> > I am guessing that we will be able to run some timing out tests at about the
> > same time.
> 
> It is much better if we need to rebaseline tests once we implement something
> rather than looking for skipped tests to unskip :(((

Also please see progress via https://bugs.webkit.org/show_bug.cgi?id=178985
Comment 25 youenn fablet 2017-10-29 16:34:56 PDT
Created attachment 325298 [details]
Patch
Comment 26 youenn fablet 2017-10-29 16:36:17 PDT
I'll put it again as cq+ so that EWS/other bots are fine.
If you have time to improve the patch by refreshing the expectations, please cq- and update the patch. Otherwise, I'll do it tomorrow.
Comment 27 Chris Dumez 2017-10-29 16:37:58 PDT
Comment on attachment 325298 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=325298&action=review

> LayoutTests/ChangeLog:8
> +        * TestExpectations: Skipping som eservice worker tests.

Typo: som eservice
Comment 28 Chris Dumez 2017-10-29 16:49:07 PDT
(In reply to Chris Dumez from comment #27)
> Comment on attachment 325298 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=325298&action=review
> 
> > LayoutTests/ChangeLog:8
> > +        * TestExpectations: Skipping som eservice worker tests.
> 
> Typo: som eservice

Also, as mentioned earlier, I have a task to get SW WPT tests running so please help me by not unnecessarily skipping tests that do not time out.
It is not trivial to figure out which tests can be unskipped after implementing something. However, rebaselining is.
Comment 29 Chris Dumez 2017-10-29 16:56:10 PDT
(In reply to Chris Dumez from comment #28)
> (In reply to Chris Dumez from comment #27)
> > Comment on attachment 325298 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=325298&action=review
> > 
> > > LayoutTests/ChangeLog:8
> > > +        * TestExpectations: Skipping som eservice worker tests.
> > 
> > Typo: som eservice
> 
> Also, as mentioned earlier, I have a task to get SW WPT tests running so
> please help me by not unnecessarily skipping tests that do not time out.
> It is not trivial to figure out which tests can be unskipped after
> implementing something. However, rebaselining is.

I also would not mind landing baselines if you’d like.
Comment 30 Chris Dumez 2017-10-29 17:05:00 PDT
(In reply to Chris Dumez from comment #29)
> (In reply to Chris Dumez from comment #28)
> > (In reply to Chris Dumez from comment #27)
> > > Comment on attachment 325298 [details]
> > > Patch
> > > 
> > > View in context:
> > > https://bugs.webkit.org/attachment.cgi?id=325298&action=review
> > > 
> > > > LayoutTests/ChangeLog:8
> > > > +        * TestExpectations: Skipping som eservice worker tests.
> > > 
> > > Typo: som eservice
> > 
> > Also, as mentioned earlier, I have a task to get SW WPT tests running so
> > please help me by not unnecessarily skipping tests that do not time out.
> > It is not trivial to figure out which tests can be unskipped after
> > implementing something. However, rebaselining is.
> 
> I also would not mind landing baselines if you’d like.

<https://trac.webkit.org/changeset/224166>
Comment 31 Radar WebKit Bug Importer 2017-11-15 13:15:16 PST
<rdar://problem/35569084>