import w3c tests for canvas

• Component: Canvas
• Status: RESOLVED FIXED

[Said Abou-Hallawa]

All the tests under https://dvcs.w3.org/hg/html/file/1d30da826739/tests/approved/canvas need to be imported to webkit.  The failed ones will be added to the expected failed tests list.  Individual bugs to be added for each failed test.

[Said Abou-Hallawa]

Created attachment 241462 [details]
Patch

[Said Abou-Hallawa]

Two bugs were logged to track the tests which failed and the tests which need to be modified such that they can be automatically verified.

https://bugs.webkit.org/show_bug.cgi?id=138672
https://bugs.webkit.org/show_bug.cgi?id=138676

[Said Abou-Hallawa]

Created attachment 241497 [details]
Patch

[Dean Jackson]

Comment on attachment 241497 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=241497&action=review

rs=me for the actual tests

> LayoutTests/TestExpectations:219
> +# Imported W3C canvas failed tests 
> +webkit.org/b/138672 imported/w3c/canvas/2d.composite.uncovered.pattern.destination-atop.html [ Skip ]
> +webkit.org/b/138672 imported/w3c/canvas/2d.pattern.crosscanvas.html [ Skip ]

Do these fail everywhere consistently?

If so, they should probably be [ Failure ] rather than [ Skip ]

(this means we'll notice progressions)

> LayoutTests/TestExpectations:331
> +# Imported W3C canvas tests need to be changed such that they can be utomatically verifid

Typo in automatically.

> LayoutTests/imported/w3c/ChangeLog:1665
> +        * common/canvas-frame.css: Added.
> +        (body):
> +        (p):
> +        (p:first-child):
> +        (h1):

I usually remove all the cruft under CSS file changes. It isn't always accurate. Just having the file name is enough in this case.

[Said Abou-Hallawa]

Created attachment 241513 [details]
Patch

[Said Abou-Hallawa]

(In reply to comment #4)
> Comment on attachment 241497 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=241497&action=review
> 
> rs=me for the actual tests
> 
> > LayoutTests/TestExpectations:219
> > +# Imported W3C canvas failed tests 
> > +webkit.org/b/138672 imported/w3c/canvas/2d.composite.uncovered.pattern.destination-atop.html [ Skip ]
> > +webkit.org/b/138672 imported/w3c/canvas/2d.pattern.crosscanvas.html [ Skip ]
> 
> Do these fail everywhere consistently?
> 
> If so, they should probably be [ Failure ] rather than [ Skip ]
> 
> (this means we'll notice progressions)
> 

Done. I changed [ Skip ] with [ Failure ] and I added expected files for the failed tests.

> > LayoutTests/TestExpectations:331
> > +# Imported W3C canvas tests need to be changed such that they can be utomatically verifid
> 
> Typo in automatically.
> 

Fixed.

> > LayoutTests/imported/w3c/ChangeLog:1665
> > +        * common/canvas-frame.css: Added.
> > +        (body):
> > +        (p):
> > +        (p:first-child):
> > +        (h1):
> 
> I usually remove all the cruft under CSS file changes. It isn't always
> accurate. Just having the file name is enough in this case.

Done.

[Dean Jackson]

Comment on attachment 241513 [details]
Patch

rs=me

[WebKit Commit Bot]

Comment on attachment 241513 [details]
Patch

Rejecting attachment 241513 [details] from commit-queue.

Number of test failures exceeded the failure limit.
Full output: http://webkit-queues.appspot.com/results/5383446117482496

[WebKit Commit Bot]

Created attachment 241526 [details]
Archive of layout-test-results from webkit-cq-02 for mac-mountainlion

The attached test failures were seen while running run-webkit-tests on the commit-queue.
Bot: webkit-cq-02  Port: mac-mountainlion  Platform: Mac OS X 10.8.5

[Build Bot]

Comment on attachment 241513 [details]
Patch

Attachment 241513 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/6677766724386816

Number of test failures exceeded the failure limit.

[Build Bot]

Created attachment 241527 [details]
Archive of layout-test-results from webkit-ews-09 for mac-mountainlion-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: webkit-ews-09  Port: mac-mountainlion-wk2  Platform: Mac OS X 10.8.5

[Build Bot]

Comment on attachment 241513 [details]
Patch

Attachment 241513 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/6636305257594880

New failing tests:
imported/w3c/canvas/2d.line.join.parallel.html
imported/w3c/canvas/2d.strokeRect.zero.5.html
imported/w3c/canvas/2d.gradient.interpolate.colouralpha.html

[Build Bot]

Created attachment 241550 [details]
Archive of layout-test-results from webkit-ews-02 for mac-mountainlion

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: webkit-ews-02  Port: mac-mountainlion  Platform: Mac OS X 10.8.5

[Said Abou-Hallawa]

Created attachment 241598 [details]
Patch

[Said Abou-Hallawa]

I added the failed tests to the TestExpectations because they seem to be flaky. They have been passing on my machine but they fail on Bot.  I will investigate them with the other failures.

[Said Abou-Hallawa]

Created attachment 241612 [details]
Patch

[Said Abou-Hallawa]

My source tree was outdated.  I have to sync it and resubmit a new patch.

[Said Abou-Hallawa]

Comment on attachment 241612 [details]
Patch

the EWS is now passing.  This patch should be ready to go.

[Dean Jackson]

Comment on attachment 241612 [details]
Patch

I'll land it manually.

[WebKit Commit Bot]

Comment on attachment 241612 [details]
Patch

Clearing flags on attachment: 241612

Committed r176144: <http://trac.webkit.org/changeset/176144>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alexey Proskuryakov]

Many of these tests use non-existent background.png:

https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/r176694%20(747)/imported/w3c/canvas/2d.imageData.put.null-stderr.txt

Adding it will make tests a lot less noisy, faster, and presumably more of them will pass.

[youenn fablet]

(In reply to comment #22)
> Many of these tests use non-existent background.png:
> 
> https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/
> r176694%20(747)/imported/w3c/canvas/2d.imageData.put.null-stderr.txt
> 
> Adding it will make tests a lot less noisy, faster, and presumably more of
> them will pass.

Moving those tests to imported/w3c/web-platform-tests should take care of those issues.

[Said Abou-Hallawa]

(In reply to comment #23)
> (In reply to comment #22)
> > Many of these tests use non-existent background.png:
> > 
> > https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/
> > r176694%20(747)/imported/w3c/canvas/2d.imageData.put.null-stderr.txt
> > 
> > Adding it will make tests a lot less noisy, faster, and presumably more of
> > them will pass.
> 
> Moving those tests to imported/w3c/web-platform-tests should take care of
> those issues.

This issue has been fixed in https://bugs.webkit.org/show_bug.cgi?id=140712. The problem was in not adopting the same directory structure as w3c did. The css file  imported/w3c/common/canvas-tests.css was referencing an image from imported/w3c/resources/background.png while it was in imported/w3c/canvas/resources/backgorund.png. The fix was to move the common resources from imported/w3c/canvas/resources/ to imported/w3c/resources/.

[youenn fablet]

(In reply to comment #24)
> (In reply to comment #23)
> > (In reply to comment #22)
> > > Many of these tests use non-existent background.png:
> > > 
> > > https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/
> > > r176694%20(747)/imported/w3c/canvas/2d.imageData.put.null-stderr.txt
> > > 
> > > Adding it will make tests a lot less noisy, faster, and presumably more of
> > > them will pass.
> > 
> > Moving those tests to imported/w3c/web-platform-tests should take care of
> > those issues.
> 
> This issue has been fixed in https://bugs.webkit.org/show_bug.cgi?id=140712.
> The problem was in not adopting the same directory structure as w3c did. The
> css file  imported/w3c/common/canvas-tests.css was referencing an image from
> imported/w3c/resources/background.png while it was in
> imported/w3c/canvas/resources/backgorund.png. The fix was to move the common
> resources from imported/w3c/canvas/resources/ to imported/w3c/resources/.

Oh, good.
There may be still some similar issues though, like the fonts folder.
Checking a few canvas tests listed in bug 138676, they seem to pass when imported using import-w3c-test in imported/w3c/web-platform-tests.
That may be worth a try?