Bug 161723

Summary: [Media Source] Import updated w3c/web-platform-tests and rebaseline
Product: WebKit Reporter: Jer Noble <jer.noble>
Component: MediaAssignee: Jer Noble <jer.noble>
Status: RESOLVED FIXED    
Severity: Normal CC: buildbot, commit-queue, rniwa, youennf
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 161725    
Attachments:
Description Flags
Patch
none
Archive of layout-test-results from ews126 for ios-simulator-elcapitan-wk2
none
Archive of layout-test-results from ews107 for mac-yosemite-wk2
none
Archive of layout-test-results from ews113 for mac-yosemite
none
Archive of layout-test-results from ews101 for mac-yosemite
none
Patch
none
Archive of layout-test-results from ews102 for mac-yosemite
none
Archive of layout-test-results from ews125 for ios-simulator-elcapitan-wk2
none
Archive of layout-test-results from ews107 for mac-yosemite-wk2
none
Archive of layout-test-results from ews116 for mac-yosemite
none
Patch
youennf: review+, buildbot: commit-queue-
Archive of layout-test-results from ews123 for ios-simulator-elcapitan-wk2
none
Archive of layout-test-results from ews103 for mac-yosemite
none
Archive of layout-test-results from ews105 for mac-yosemite-wk2
none
Archive of layout-test-results from ews115 for mac-yosemite
none
Patch for landing none

Description Jer Noble 2016-09-07 17:45:56 PDT
[Media Source] Import updated w3c/web-platform-tests and rebaseline
Comment 1 Jer Noble 2016-09-08 15:26:16 PDT
Created attachment 288338 [details]
Patch
Comment 2 Build Bot 2016-09-08 16:25:39 PDT
Comment on attachment 288338 [details]
Patch

Attachment 288338 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2037124

Number of test failures exceeded the failure limit.
Comment 3 Build Bot 2016-09-08 16:25:41 PDT
Created attachment 288347 [details]
Archive of layout-test-results from ews126 for ios-simulator-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews126  Port: ios-simulator-elcapitan-wk2  Platform: Mac OS X 10.11.5
Comment 4 Build Bot 2016-09-08 16:29:14 PDT
Comment on attachment 288338 [details]
Patch

Attachment 288338 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2037151

New failing tests:
imported/w3c/web-platform-tests/media-source/mediasource-addsourcebuffer.html
imported/w3c/web-platform-tests/media-source/mediasource-remove.html
imported/w3c/web-platform-tests/media-source/mediasource-sourcebuffer-mode.html
imported/w3c/web-platform-tests/media-source/mediasource-is-type-supported.html
http/tests/media/media-source/mediasource-append-stream.html
imported/w3c/web-platform-tests/media-source/mediasource-append-buffer.html
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-bitrate.html
imported/w3c/web-platform-tests/media-source/mediasource-play.html
Comment 5 Build Bot 2016-09-08 16:29:16 PDT
Created attachment 288350 [details]
Archive of layout-test-results from ews107 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5
Comment 6 Build Bot 2016-09-08 16:35:25 PDT
Comment on attachment 288338 [details]
Patch

Attachment 288338 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2037160

New failing tests:
imported/w3c/web-platform-tests/media-source/mediasource-addsourcebuffer.html
imported/w3c/web-platform-tests/media-source/mediasource-remove.html
imported/w3c/web-platform-tests/media-source/mediasource-sourcebuffer-mode.html
imported/w3c/web-platform-tests/media-source/mediasource-is-type-supported.html
http/tests/media/media-source/mediasource-append-stream.html
imported/w3c/web-platform-tests/media-source/mediasource-append-buffer.html
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-bitrate.html
imported/w3c/web-platform-tests/media-source/mediasource-play.html
Comment 7 Build Bot 2016-09-08 16:35:27 PDT
Created attachment 288353 [details]
Archive of layout-test-results from ews113 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews113  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 8 Build Bot 2016-09-08 16:42:35 PDT
Comment on attachment 288338 [details]
Patch

Attachment 288338 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2037210

New failing tests:
imported/w3c/web-platform-tests/media-source/mediasource-addsourcebuffer.html
imported/w3c/web-platform-tests/media-source/mediasource-remove.html
imported/w3c/web-platform-tests/media-source/mediasource-sourcebuffer-mode.html
imported/w3c/web-platform-tests/media-source/mediasource-is-type-supported.html
http/tests/media/media-source/mediasource-append-stream.html
imported/w3c/web-platform-tests/media-source/mediasource-append-buffer.html
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-bitrate.html
imported/w3c/web-platform-tests/media-source/mediasource-play.html
Comment 9 Build Bot 2016-09-08 16:42:37 PDT
Created attachment 288356 [details]
Archive of layout-test-results from ews101 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 10 Jer Noble 2016-09-09 10:36:48 PDT
Created attachment 288412 [details]
Patch
Comment 11 Build Bot 2016-09-09 11:27:58 PDT
Comment on attachment 288412 [details]
Patch

Attachment 288412 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2042503

New failing tests:
imported/w3c/web-platform-tests/media-source/mediasource-addsourcebuffer.html
imported/w3c/web-platform-tests/media-source/mediasource-remove.html
imported/w3c/web-platform-tests/media-source/mediasource-sourcebuffer-mode.html
imported/w3c/web-platform-tests/media-source/mediasource-is-type-supported.html
imported/w3c/web-platform-tests/media-source/mediasource-append-buffer.html
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-bitrate.html
imported/w3c/web-platform-tests/media-source/mediasource-play.html
Comment 12 Build Bot 2016-09-09 11:28:02 PDT
Created attachment 288418 [details]
Archive of layout-test-results from ews102 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews102  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 13 Build Bot 2016-09-09 11:40:16 PDT
Comment on attachment 288412 [details]
Patch

Attachment 288412 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2042532

Number of test failures exceeded the failure limit.
Comment 14 Build Bot 2016-09-09 11:40:19 PDT
Created attachment 288420 [details]
Archive of layout-test-results from ews125 for ios-simulator-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125  Port: ios-simulator-elcapitan-wk2  Platform: Mac OS X 10.11.5
Comment 15 Build Bot 2016-09-09 11:45:13 PDT
Comment on attachment 288412 [details]
Patch

Attachment 288412 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2042554

New failing tests:
imported/w3c/web-platform-tests/media-source/mediasource-addsourcebuffer.html
imported/w3c/web-platform-tests/media-source/mediasource-remove.html
imported/w3c/web-platform-tests/media-source/mediasource-sourcebuffer-mode.html
imported/w3c/web-platform-tests/media-source/mediasource-is-type-supported.html
imported/w3c/web-platform-tests/media-source/mediasource-append-buffer.html
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-bitrate.html
imported/w3c/web-platform-tests/media-source/mediasource-play.html
Comment 16 Build Bot 2016-09-09 11:45:15 PDT
Created attachment 288421 [details]
Archive of layout-test-results from ews107 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5
Comment 17 Build Bot 2016-09-09 11:46:41 PDT
Comment on attachment 288412 [details]
Patch

Attachment 288412 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2042556

New failing tests:
imported/w3c/web-platform-tests/media-source/mediasource-addsourcebuffer.html
imported/w3c/web-platform-tests/media-source/mediasource-remove.html
imported/w3c/web-platform-tests/media-source/mediasource-sourcebuffer-mode.html
imported/w3c/web-platform-tests/media-source/mediasource-is-type-supported.html
imported/w3c/web-platform-tests/media-source/mediasource-append-buffer.html
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
imported/w3c/web-platform-tests/media-source/mediasource-config-change-mp4-v-bitrate.html
imported/w3c/web-platform-tests/media-source/mediasource-play.html
Comment 18 Build Bot 2016-09-09 11:46:44 PDT
Created attachment 288422 [details]
Archive of layout-test-results from ews116 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews116  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 19 youenn fablet 2016-09-09 12:10:29 PDT
As I understand it, you moved the files from http/tests/media to imported/w3c/web-platform-tests. That seems good to me.

It would be great if a follow-up patch could do the following:
- Delete LayoutTests/imported/w3c/web-platform-tests/media-source
- Update LayoutTests/imported/w3c/resources/ImportExpectations to comment/remove the web-platform-tests/media-source line
- Run 'Tools/Scripts/import-w3c-tests -t web-platform-tests/media-source' to reimport up-to-date media source tests

The first step allows removing potentially obsolete tests.
If some of these tests are still useful, they should be moved back to http/tests, as in a later refresh, I may remove them without knowing that they are actually useful.
Comment 20 Jer Noble 2016-09-09 12:12:20 PDT
(In reply to comment #19)
> As I understand it, you moved the files from http/tests/media to
> imported/w3c/web-platform-tests. That seems good to me.
> 
> It would be great if a follow-up patch could do the following:
> - Delete LayoutTests/imported/w3c/web-platform-tests/media-source

Did you mean, delete http/tests/media/media-source ?  I can't really do that; there's some non-w3c media-source tests in there.

> - Update LayoutTests/imported/w3c/resources/ImportExpectations to
> comment/remove the web-platform-tests/media-source line

That's in this patch.

> - Run 'Tools/Scripts/import-w3c-tests -t web-platform-tests/media-source' to
> reimport up-to-date media source tests

That was done to generate this patch.

> The first step allows removing potentially obsolete tests.
> If some of these tests are still useful, they should be moved back to
> http/tests, as in a later refresh, I may remove them without knowing that
> they are actually useful.
Comment 21 Jer Noble 2016-09-09 12:15:02 PDT
Created attachment 288427 [details]
Patch
Comment 22 youenn fablet 2016-09-09 12:20:33 PDT
Ah, you did it all in one step, that is just great!
I am surprised that there is no w3c-import.log in the patch though.
Or maybe I missed it.
Anyway sounds good to me if the bots are happy.
Comment 23 youenn fablet 2016-09-09 12:21:51 PDT
Comment on attachment 288427 [details]
Patch

r = me as long as the bots are happy
Comment 24 Build Bot 2016-09-09 13:09:33 PDT
Comment on attachment 288427 [details]
Patch

Attachment 288427 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/2042939

Number of test failures exceeded the failure limit.
Comment 25 Build Bot 2016-09-09 13:09:36 PDT
Created attachment 288431 [details]
Archive of layout-test-results from ews123 for ios-simulator-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-elcapitan-wk2  Platform: Mac OS X 10.11.5
Comment 26 Build Bot 2016-09-09 13:16:47 PDT
Comment on attachment 288427 [details]
Patch

Attachment 288427 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/2043011

New failing tests:
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
Comment 27 Build Bot 2016-09-09 13:16:50 PDT
Created attachment 288433 [details]
Archive of layout-test-results from ews103 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews103  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 28 Jer Noble 2016-09-09 13:18:15 PDT
Looks like I need to add a rule to the iOS TestExpectations file to skip these (moved) media-source tests.
Comment 29 Build Bot 2016-09-09 13:20:55 PDT
Comment on attachment 288427 [details]
Patch

Attachment 288427 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/2043016

New failing tests:
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
Comment 30 Build Bot 2016-09-09 13:20:58 PDT
Created attachment 288434 [details]
Archive of layout-test-results from ews105 for mac-yosemite-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews105  Port: mac-yosemite-wk2  Platform: Mac OS X 10.10.5
Comment 31 Jer Noble 2016-09-09 13:24:46 PDT
And it looks like this most recent test failure is because I moved the webm/ and mp4/ directories out of http/tests/media/media-source.
Comment 32 Build Bot 2016-09-09 13:25:56 PDT
Comment on attachment 288427 [details]
Patch

Attachment 288427 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2043018

New failing tests:
http/tests/media/media-source/mediasource-play-then-seek-back-with-remote-control.html
Comment 33 Build Bot 2016-09-09 13:25:59 PDT
Created attachment 288435 [details]
Archive of layout-test-results from ews115 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews115  Port: mac-yosemite  Platform: Mac OS X 10.10.5
Comment 34 Jer Noble 2016-09-09 13:34:47 PDT
Created attachment 288436 [details]
Patch for landing
Comment 35 WebKit Commit Bot 2016-09-09 17:51:32 PDT
Comment on attachment 288436 [details]
Patch for landing

Clearing flags on attachment: 288436

Committed r205777: <http://trac.webkit.org/changeset/205777>
Comment 36 WebKit Commit Bot 2016-09-12 09:33:45 PDT
Comment on attachment 288436 [details]
Patch for landing

Rejecting attachment 288436 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-02', 'apply-attachment', '--no-update', '--non-interactive', 288436, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
json	(revision 0)
|+++ LayoutTests/http/tests/media/media-source/mp4/test-a-128k-44100Hz-1ch-manifest.json	(working copy)
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored
Original content of LayoutTests/http/tests/media/media-source/mp4/test-a-128k-44100Hz-1ch.mp4 mismatches at /Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply line 271.

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Full output: http://webkit-queues.webkit.org/results/2059906