Bug 172523 – Some async tests ignore jsTestIsAsync when ported from js-test-pre.js / js-test-post.js to js-test.js

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Chris Dumez]

Some async tests ignore jsTestIsAsync when ported from js-test-pre.js / js-test-post.js to js-test.js.

Impacts LayoutTests/fast/dom/script-charset-update.html.

The onload event is fired too late, the test has already completed.

[Chris Dumez]

Created attachment 311050 [details]
Patch

[Build Bot]

Comment on attachment 311050 [details]
Patch

Attachment 311050 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/3802616

New failing tests:
fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events.html
fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html

[Build Bot]

Created attachment 311059 [details]
Archive of layout-test-results from ews101 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-elcapitan  Platform: Mac OS X 10.11.6

[Chris Dumez]

(In reply to Build Bot from comment #2)
> Comment on attachment 311050 [details]
> Patch
> 
> Attachment 311050 [details] did not pass mac-ews (mac):
> Output: http://webkit-queues.webkit.org/results/3802616
> 
> New failing tests:
> fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events.
> html
> fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html

Well, it shows bugs in 2 existing tests. Those tests do not have "TEST COMPLETE" in their expected result.

[Build Bot]

Comment on attachment 311050 [details]
Patch

Attachment 311050 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/3802679

New failing tests:
fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html

[Build Bot]

Created attachment 311061 [details]
Archive of layout-test-results from ews106 for mac-elcapitan-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews106  Port: mac-elcapitan-wk2  Platform: Mac OS X 10.11.6

[Chris Dumez]

(In reply to Chris Dumez from comment #4)
> (In reply to Build Bot from comment #2)
> > Comment on attachment 311050 [details]
> > Patch
> > 
> > Attachment 311050 [details] did not pass mac-ews (mac):
> > Output: http://webkit-queues.webkit.org/results/3802616
> > 
> > New failing tests:
> > fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events.
> > html
> > fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html
> 
> Well, it shows bugs in 2 existing tests. Those tests do not have "TEST
> COMPLETE" in their expected result.

diff --git a/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events-expected.txt b/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events-expected.txt
index 72acf40705e..3f61c85285c 100644
--- a/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events-expected.txt
+++ b/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events-expected.txt
@@ -1,3 +1,4 @@
+FAIL: Timed out waiting for notifyDone to be called
 Exercises the requestFrame() call on captureStream(0).
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
@@ -7,4 +8,7 @@ PASS Got a stream from canvas.
 PASS stream.getVideoTracks().length is 1
 PASS track.readyState is "live"
 PASS Plugged stream to video tag.
+PASS Video play callback succeeded.
+PASS Drawing to canvas.
+PASS Calling requestFrame().
  

So this once goes further but times out after calling requestFrame() now that jsTestIsAsync is actually taken into consideration. Youenn, any idea?

diff --git a/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events-expected.txt b/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events-expected.txt
index 6d898fe17dd..a1250b415cb 100644
--- a/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events-expected.txt
+++ b/LayoutTests/fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events-expected.txt
@@ -1,3 +1,4 @@
+CONSOLE MESSAGE: line 33: Turning drawing buffer preservation for the WebGL canvas being captured
 Exercises the potential events on CanvasCaptureMediaStream.
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
@@ -7,4 +8,10 @@ PASS Got a stream from canvas.
 PASS stream.getVideoTracks().length is 1
 PASS track.readyState is "live"
 PASS Plugged stream to video tag.
+PASS Video play callback succeeded.
+PASS Drawing to canvas.
+PASS Video canplaythrough callback succeeded.
+PASS successfullyParsed is true
+
+TEST COMPLETE
 
Ok, This test now completes and passes so this is actually a progression.

[Chris Dumez]

Note that these tests are marked as flaky on various platforms already. E.g.
https://bugs.webkit.org/show_bug.cgi?id=169719

[Build Bot]

Comment on attachment 311050 [details]
Patch

Attachment 311050 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/3802639

New failing tests:
fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events.html
fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html

[Build Bot]

Created attachment 311063 [details]
Archive of layout-test-results from ews114 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews114  Port: mac-elcapitan  Platform: Mac OS X 10.11.6

[Build Bot]

Comment on attachment 311050 [details]
Patch

Attachment 311050 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/3802660

New failing tests:
fast/mediacapturefromelement/CanvasCaptureMediaStream-webgl-events.html

[Build Bot]

Created attachment 311065 [details]
Archive of layout-test-results from ews121 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews121  Port: ios-simulator-wk2  Platform: Mac OS X 10.11.6

[Chris Dumez]

Created attachment 311066 [details]
Patch

[Ryosuke Niwa]

Comment on attachment 311066 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=311066&action=review

> LayoutTests/TestExpectations:144
> +# This test times out.
> +webkit.org/b/172527 fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events.html [ Skip ]

Why don't we use Timeout expectation instead?

[Alexey Proskuryakov]

A 100% reproducible timeout takes 30 seconds of time, making tests slower overall.

[Chris Dumez]

(In reply to Ryosuke Niwa from comment #14)
> Comment on attachment 311066 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=311066&action=review
> 
> > LayoutTests/TestExpectations:144
> > +# This test times out.
> > +webkit.org/b/172527 fast/mediacapturefromelement/CanvasCaptureMediaStream-request-frame-events.html [ Skip ]
> 
> Why don't we use Timeout expectation instead?

Doesn't it actually make running the tests slower since we're going to wait for this test to time out?

[Alexey Proskuryakov]

Comment on attachment 311066 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=311066&action=review

> LayoutTests/ChangeLog:19
> +        Unskip test that is not timing out consistently instead of being flaky.

not -> now?

As you are actually skipping the test, it seems better to say "remove expectation for a test that is now skipped in root TestExpectations file".

[Chris Dumez]

Created attachment 311068 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 311068 [details]
Patch

Clearing flags on attachment: 311068

Committed r217312: <http://trac.webkit.org/changeset/217312>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.