Bug 61541 – Port W3C audio and video tests

• Component: New Bugs
• Status: RESOLVED FIXED

[Mark Pilgrim (Google)]

Original tests: http://dvcs.w3.org/hg/html/file/233364893066/tests/submission/Google , http://dvcs.w3.org/hg/html/file/233364893066/tests/approved/audio , and http://dvcs.w3.org/hg/html/file/233364893066/tests/approved/video

This is a port of the W3C test suite for the [audio] and [video] elements. Rather than port each test individually, I've created a wrapper script (in W3C/resources/w3cwrapper.js) that emulates the W3C test harness within the WebKit layout test harness. As such, the only change I've had to make to the original tests was changing the [script] tags that pointed to the test harness.

WebKit passes all of these tests.

[Mark Pilgrim (Google)]

Created attachment 94993 [details]
Patch

[Eric Seidel (no email)]

Comment on attachment 94993 [details]
Patch

Seems reasonable to me.

Is there a version number associated with these tests that we should be recording somewhere?

[Eric Carlson]

Comment on attachment 94993 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=94993&action=review

Thanks, this is fantastic! 

> LayoutTests/media/W3C/w3cwrapper.js:52
> +function async_test(title, options) {
> +  description(title);
> +  if (window.layoutTestController) {
> +    layoutTestController.dumpAsText();
> +    layoutTestController.waitUntilDone();
> +  }
> +  var t = {
> +    step: function(testFunction) {
> +      try {
> +        testFunction();
> +      }
> +      catch (e) {
> +        testFailed('Aborted with exception: ' + e.message);
> +      }
> +    },
> +
> +    done: function() {
> +      debug('<br /><span class="pass">TEST COMPLETE</span>');
> +      if (window.layoutTestController) {
> +        layoutTestController.notifyDone();
> +      }
> +    }
> +  }
> +
> +  return t;
> +}

I see that the "options" parameter is ignored. It looks like 'timeout' is currently the only parameter, which I assume we can ignore because of DRT's own timeout, but it is probably worth adding a comment to that effect and throwing an exception if any other option is passed in case future tests add other options.

[WebKit Review Bot]

Comment on attachment 94993 [details]
Patch

Clearing flags on attachment: 94993

Committed r103470: <http://trac.webkit.org/changeset/103470>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.

[Joel Parks]

I agree this is fantastic, but it seems to me that some latent errors in the W3C test content were ported along with the good stuff.

A good number of the audio tests[*] are apparently copy-pasted from video tests without changing the video element to audio element.  There are separate tests in the video section for audio media in video elements, which helps convince me that it's a copy-paste error.

Also, as Eric Carlson mentions, it would be good to point out that the options parameter is not being respected by the new wrapper.  However, my experience is that the original harness does not respect attempts to modify the timeout, so perhaps this is a moot point!  I wasn't able to take the time to investigate the cause of that behavior.

(In reply to comment #3)
> (From update of attachment 94993 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=94993&action=review
> 
> Thanks, this is fantastic! 
> 
...snip...
> 
> I see that the "options" parameter is ignored. It looks like 'timeout' is currently the only parameter, which I assume we can ignore because of DRT's own timeout, but it is probably worth adding a comment to that effect and throwing an exception if any other option is passed in case future tests add other options.


[*] This is a list of the audio tests (and expected results files) that contain video elements instead of audio elements:

LayoutTests/media/W3C/audio/events/event_canplay-expected.txt
LayoutTests/media/W3C/audio/events/event_canplay.html
LayoutTests/media/W3C/audio/events/event_canplay_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_canplay_manual.html
LayoutTests/media/W3C/audio/events/event_canplaythrough-expected.txt
LayoutTests/media/W3C/audio/events/event_canplaythrough.html
LayoutTests/media/W3C/audio/events/event_canplaythrough_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_canplaythrough_manual.html
LayoutTests/media/W3C/audio/events/event_loadeddata-expected.txt
LayoutTests/media/W3C/audio/events/event_loadeddata.html
LayoutTests/media/W3C/audio/events/event_loadeddata_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_loadeddata_manual.html
LayoutTests/media/W3C/audio/events/event_loadedmetadata-expected.txt
LayoutTests/media/W3C/audio/events/event_loadedmetadata.html
LayoutTests/media/W3C/audio/events/event_loadedmetadata_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_loadedmetadata_manual.html
LayoutTests/media/W3C/audio/events/event_loadstart-expected.txt
LayoutTests/media/W3C/audio/events/event_loadstart.html
LayoutTests/media/W3C/audio/events/event_loadstart_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_loadstart_manual.html
LayoutTests/media/W3C/audio/events/event_order_canplay_canplaythrough-expected.txt
LayoutTests/media/W3C/audio/events/event_order_canplay_canplaythrough.html
LayoutTests/media/W3C/audio/events/event_order_canplay_playing-expected.txt
LayoutTests/media/W3C/audio/events/event_order_canplay_playing.html
LayoutTests/media/W3C/audio/events/event_order_loadedmetadata_loadeddata-expected.txt
LayoutTests/media/W3C/audio/events/event_order_loadedmetadata_loadeddata.html
LayoutTests/media/W3C/audio/events/event_order_loadstart_progress-expected.txt
LayoutTests/media/W3C/audio/events/event_order_loadstart_progress.html
LayoutTests/media/W3C/audio/events/event_pause_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_pause_manual.html
LayoutTests/media/W3C/audio/events/event_play-expected.txt
LayoutTests/media/W3C/audio/events/event_play.html
LayoutTests/media/W3C/audio/events/event_play_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_play_manual.html
LayoutTests/media/W3C/audio/events/event_playing-expected.txt
LayoutTests/media/W3C/audio/events/event_playing.html
LayoutTests/media/W3C/audio/events/event_playing_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_playing_manual.html
LayoutTests/media/W3C/audio/events/event_progress-expected.txt
LayoutTests/media/W3C/audio/events/event_progress.html
LayoutTests/media/W3C/audio/events/event_progress_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_progress_manual.html
LayoutTests/media/W3C/audio/events/event_timeupdate-expected.txt
LayoutTests/media/W3C/audio/events/event_timeupdate.html
LayoutTests/media/W3C/audio/events/event_timeupdate_manual-expected.txt
LayoutTests/media/W3C/audio/events/event_timeupdate_manual.html

[Simon Fraser (smfr)]

A bunch of these new tests are failing: bug 75175. Please fix.

[Eric Seidel (no email)]

Probably easiest to sheriff-bot rollout if Mark isn't around.  Or even if he is, he could then grab the failing results from the bots and re-land.