Bug 182833 – imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-* LayoutTests are flaky failures

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Ryan Haddad]

The following LayoutTests are flaky failures on macOS and iOS after the tests were updated in https://trac.webkit.org/changeset/228003/webkit

imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-drawImage.html
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-invalid-args.html
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=imported%2Fw3c%2Fweb-platform-tests%2F2dcontext%2Fimagebitmap%2FcreateImageBitmap

The text diffs show a test harness timeout:

--- /Volumes/Data/slave/sierra-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-invalid-args-expected.txt
+++ /Volumes/Data/slave/sierra-debug-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-invalid-args-actual.txt
@@ -1,52 +1,53 @@
-CONSOLE MESSAGE: line 137: Canvas area exceeds the maximum limit (width * height > 268435456).
+
+Harness Error (TIMEOUT), message = null

[Ryan Haddad]

Updated TestExpectations for macOS in https://trac.webkit.org/r228526

[Alicia Boya García]

Same error on GTK, but more frequently.

[Alicia Boya García]

Moved TestExpectations to base (multiplatform) file. https://trac.webkit.org/changeset/230034/webkit

[Ms2ger (he/him; ⌚ UTC+1/+2)]

*** Bug 184562 has been marked as a duplicate of this bug. ***

[Ms2ger (he/him; ⌚ UTC+1/+2)]

The problem here is that the video element sometimes doesn't load. I've been debugging, but haven't found the cause yet.

[Ms2ger (he/him; ⌚ UTC+1/+2)]

Created attachment 338096 [details]
Patch

[Ms2ger (he/him; ⌚ UTC+1/+2)]

The failure that happened at least one every hundred runs before this change, didn't occur in >1000 runs after the change.

[EWS Watchlist]

Comment on attachment 338096 [details]
Patch

Attachment 338096 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/7342418

New failing tests:
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub.html

[EWS Watchlist]

Created attachment 338099 [details]
Archive of layout-test-results from ews100 for mac-sierra

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews100  Port: mac-sierra  Platform: Mac OS X 10.12.6

[EWS Watchlist]

Comment on attachment 338096 [details]
Patch

Attachment 338096 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/7342423

New failing tests:
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub.html

[EWS Watchlist]

Created attachment 338100 [details]
Archive of layout-test-results from ews104 for mac-sierra-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-sierra-wk2  Platform: Mac OS X 10.12.6

[EWS Watchlist]

Comment on attachment 338096 [details]
Patch

Attachment 338096 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/7342444

New failing tests:
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-invalid-args.html

[EWS Watchlist]

Created attachment 338101 [details]
Archive of layout-test-results from ews125 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews125  Port: ios-simulator-wk2  Platform: Mac OS X 10.13.4

[EWS Watchlist]

Comment on attachment 338096 [details]
Patch

Attachment 338096 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/7342451

New failing tests:
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub.html

[EWS Watchlist]

Created attachment 338102 [details]
Archive of layout-test-results from ews117 for mac-sierra

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews117  Port: mac-sierra  Platform: Mac OS X 10.12.6

[Ms2ger (he/him; ⌚ UTC+1/+2)]

Created attachment 338104 [details]
Patch

[EWS Watchlist]

Comment on attachment 338104 [details]
Patch

Attachment 338104 [details] did not pass ios-sim-ews (ios-simulator-wk2):
Output: http://webkit-queues.webkit.org/results/7343576

New failing tests:
imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-invalid-args.html

[EWS Watchlist]

Created attachment 338111 [details]
Archive of layout-test-results from ews123 for ios-simulator-wk2

The attached test failures were seen while running run-webkit-tests on the ios-sim-ews.
Bot: ews123  Port: ios-simulator-wk2  Platform: Mac OS X 10.13.4

[Ms2ger (he/him; ⌚ UTC+1/+2)]

Created attachment 338199 [details]
Patch

[Carlos Alberto Lopez Perez]

Comment on attachment 338199 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=338199&action=review

> LayoutTests/imported/w3c/ChangeLog:9
> +        * web-platform-tests/2dcontext/imagebitmap/common.sub.js: ensure event handlers aren't garbage collected
> +        * web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub-expected.txt: update expectations

Which one is the policy for updating this tests (not the expectation, but the test itself) ??
Shouldn't we send a PR to https://github.com/w3c/web-platform-tests ?

[Ms2ger (he/him; ⌚ UTC+1/+2)]

(In reply to Carlos Alberto Lopez Perez from comment #20)
> Comment on attachment 338199 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=338199&action=review
> 
> > LayoutTests/imported/w3c/ChangeLog:9
> > +        * web-platform-tests/2dcontext/imagebitmap/common.sub.js: ensure event handlers aren't garbage collected
> > +        * web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub-expected.txt: update expectations
> 
> Which one is the policy for updating this tests (not the expectation, but
> the test itself) ??
> Shouldn't we send a PR to https://github.com/w3c/web-platform-tests ?

I will push the change upstream as soon as it has review here. (The wpt review policy just requires a review in some public place, to make this use case possible.)

[Carlos Alberto Lopez Perez]

Comment on attachment 338199 [details]
Patch

r=me
but please submit this changes on the tests to WPT and update this bug with a comment with an URL to the corresponding pull-request or issue on the WPT github so it can be followed later easily in case of need.

[youenn fablet]

> I will push the change upstream as soon as it has review here. (The wpt
> review policy just requires a review in some public place, to make this use
> case possible.)

You can use Tools/Scripts/export-w3c-tests for that purpose.
If this is your last commit locally, something like below should push the changes to your WPT clone:
Tools/Scripts/export-w3c-test-changes -b 182833  -g HEAD~1

Use -n to specific your GitHub username.
Use -t to specify the GitHub token that will allow the script to create the PR on WPT repo and -c to ask the script to do it.

Note that the guidelines are to first commit to WPT, then to WebKit.

At the moment, the WPT bot will probably ask for review for your PR.
ms2ger, let me know if you would like to be added to the list of authors for whose WebKit PR would be review skipped

[Ms2ger (he/him; ⌚ UTC+1/+2)]

(In reply to youenn fablet from comment #23)
> At the moment, the WPT bot will probably ask for review for your PR.
> ms2ger, let me know if you would like to be added to the list of authors for
> whose WebKit PR would be review skipped

That would be great!

[Ms2ger (he/him; ⌚ UTC+1/+2)]

Submitted web-platform-tests pull request: https://github.com/w3c/web-platform-tests/pull/10630

[Ms2ger (he/him; ⌚ UTC+1/+2)]

Comment on attachment 338199 [details]
Patch

Landed in https://github.com/w3c/web-platform-tests/pull/10630

[WebKit Commit Bot]

Comment on attachment 338199 [details]
Patch

Rejecting attachment 338199 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'apply-attachment', '--no-update', '--non-interactive', 338199, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Last 500 characters of output:
ext/imagebitmap/createImageBitmap-origin.sub-expected.txt
patching file LayoutTests/platform/ios/imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-invalid-args-expected.txt
patching file LayoutTests/platform/wpe/imported/w3c/web-platform-tests/2dcontext/imagebitmap/createImageBitmap-origin.sub-expected.txt

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Carlos Alberto Lopez Perez']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: http://webkit-queues.webkit.org/results/7445329

[Ms2ger (he/him; ⌚ UTC+1/+2)]

Created attachment 338718 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 338718 [details]
Patch

Clearing flags on attachment: 338718

Committed r230990: <https://trac.webkit.org/changeset/230990>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Radar WebKit Bug Importer]

<rdar://problem/39718220>

[youenn fablet]

(In reply to Ms2ger from comment #24)
> (In reply to youenn fablet from comment #23)
> > At the moment, the WPT bot will probably ask for review for your PR.
> > ms2ger, let me know if you would like to be added to the list of authors for
> > whose WebKit PR would be review skipped
> 
> That would be great!

Submitted https://github.com/web-platform-tests/wpt-pr-bot/pull/18