It would be excellent if the Khronos WebGL test suite could be added to WebKit's Layout Tests. This would both validate any changes and help fuzzers like ClusterFuzz have greater coverage. I'm not sure what the license issues may be.
<rdar://problem/12146395>
And we probably then don't need to archive it seperately in chromium tree - http://code.google.com/codesearch#OAMlx_jo-ck/src/third_party/webgl_conformance/conformance/attribs/gl-enable-vertex-attrib.html&exact_package=chromium&q=gl-enable-vertex-attrib.html&type=cs&l=2. Right now, the fuzzing is just touching fast/canvas/webgl
<rdar://problem/12146412>
Errg. Why did this make two rdar links? Testing to see which of these will get the updates.
No license issues. The WebGL conformance tests are all MIT licensed. The only issues are technical.
I should probably file a bug on the Khronos github, but I notice that the WebGL conformance suite doesn't have a license file, and many of the tests themselves have difference licenses. e.g. <!-- Copyright (c) 2011 The Chromium Authors. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file. -->
The license issue has already been addressed in the top of tree conformance suite -- e.g. under sdk/tests/conformance/ , not the snapshots under e.g. conformance-suites/1.0.1/ . Note though that there are a lot more tests in the TOT suite, and not all of them are close to passing on all platforms.
I don't know the LayoutTests system that well but my understanding is they either do screenshot comparisons or text output comparisons. Both of those are problematic for the WebGL Conformance Tests. Many of the tests test advertised capabilities. Example: numTextures = gl.getParameter(gl.MAX_TEXTURE_IMAGE_UNITS); for (var i = 0; i < numTextures; ++i) { // test each texture unit } Which means the output of the tests is different on each GPU/Driver/OS combo. I don't know if there is some way in LayoutTests to say "just tell me if the test succeeded, don't look at the output" but I believe that is what will need to be added to the LayoutTest system if it's not already there.
You are right Gregg. I've ever talked with Kenneth about this. He advised to modify these GPU-dependent tests in Khronos' repository. It is needed to remove GPU-dependent prints. After that we will be able to import these tests :) > I don't know the LayoutTests system that well but my understanding is they either do screenshot comparisons or text output comparisons. Both of those are problematic for the WebGL Conformance Tests. Many of the tests test advertised capabilities. Example: > > numTextures = gl.getParameter(gl.MAX_TEXTURE_IMAGE_UNITS); > for (var i = 0; i < numTextures; ++i) { > // test each texture unit > } > > Which means the output of the tests is different on each GPU/Driver/OS combo. > > I don't know if there is some way in LayoutTests to say "just tell me if the test succeeded, don't look at the output" but I believe that is what will need to be added to the LayoutTest system if it's not already there.
I vehemently disagree. The WebGL tests should not be changed for WebKit. WebKit should change to work with the tests. The tests are designed so they show useful information. Hiding that info so it's better for WebKit's test harness is wrong.
Can't we write another harness for WebKit that does effectively the same as the Khronos harness, but removes the iframe before notification that the test is done? We'd only get a pass or fail indication, but that's ok - the test would link to the real location at khronos where you'd get the full (important) debug output.
That would work. Just run each test in an iframe, wait for success or failure, delete the iframe, Insert success or failure on the main page.. end the test. Would you create one <name_of_test>_wrapper.html for each WebGL test? You could certainly generate those with some script that reads the 00_test_list.txt files. Anytime you want the latest tests you'd download the new test, run the script, check in the new tests and the new generated wrappers. Or, you could make some special rule that webgl tests are run by going to a url like http://path/to/layouttests/webkit_webgl_harness.html?url=path/to/real_test_to_run.html That would require changes to the LayoutTest system I'm guessing since AFAIK it just runs every .html file under LayoutTests, there's no way to specify URLs to run (or is there?)
(In reply to comment #12) > That would work. > > Just run each test in an iframe, wait for success or failure, delete the iframe, Insert success or failure on the main page.. end the test. Yeah > > Would you create one <name_of_test>_wrapper.html for each WebGL test? > > You could certainly generate those with some script that reads the 00_test_list.txt files. Anytime you want the latest tests you'd download the new test, run the script, check in the new tests and the new generated wrappers. That sounds good. We'd need to have separate files for each due to the way WK runs tests. The alternative would be one file that runs multiple tests, but then a hang or crash would require more effort to narrow down. > > Or, you could make some special rule that webgl tests are run by going to a url like http://path/to/layouttests/webkit_webgl_harness.html?url=path/to/real_test_to_run.html > > That would require changes to the LayoutTest system I'm guessing since AFAIK it just runs every .html file under LayoutTests, there's no way to specify URLs to run (or is there?) Not that I know of, alas.
I'm happy to add the tests. It shouldn't take long. We'll just have to figure out which once to mark as failing or skip
(In reply to comment #14) > I'm happy to add the tests. It shouldn't take long. We'll just have to figure out which once to mark as failing or skip The easy first step would be to add the Khronos test suite into WebKit, maybe along with a script that would keep it up to date (via a git checkout, or something). Since it is a bunch of HTML files, we'd have to skip the directory completely on all ports. Our test harness (next step) would load them in via iframes. I am really looking forward to this!
Created attachment 185697 [details] Patch
Attachment 185697 [details] did not pass style-queue: Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'LayoutTests/ChangeLog', u'LayoutTests/platform/mac/TestExpectations', u'LayoutTests/webgl/conformance/attribs/gl-disabled-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-disabled-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-enable-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-enable-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-render-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-render.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-zero-issues-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-offsets-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-offsets.html', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer.html', u'LayoutTests/webgl/generate-webgl-tests.py', u'LayoutTests/webgl/resources/js-test-pre.js', u'LayoutTests/webgl/resources/webgl-wrapper-template.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/00_test_list.txt', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-disabled-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-enable-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib-render.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib-zero-issues.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertexattribpointer-offsets.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertexattribpointer.html', u'LayoutTests/webgl/resources/webkit-webgl-test-harness.js']" exit_code: 1 LayoutTests/platform/mac/TestExpectations:1219: Path does not exist. [test/expectations] [5] Total errors found: 1 in 28 files If any of these errors are false positives, please file a bug against check-webkit-style.
Created attachment 185705 [details] Patch
So this patch is just for discussion. Not checking it in There's a script, LayoutTests/webgl/generate-webgl-tests.py which you run like this generate-webgl-tests -w <path_to_webgl_conformance_tests> And it will copy all the original tests to LayoutTests/webgl/resources/webgl_test_files. It will then generate wrappers in LayoutTests/webgl/conformance The patch above I just uploaded the examples from conformance/attribs as the entire list of files gets an error "too many arguments" when I tried webkit-patch upload There's a few issues before I can check it in. 1) It hardlocks my MBP quite often. I added the TestExpections to turn off the one test I know there are issues. The context-creation-and-destruction test. But, I think maybe the fact that DRT runs 8 instances at once is causing issues because it was locking hard in places I wouldn't expect it to lock hard. Either that or maybe WebKit just locks with WebGL often (I'm always running the tests in Chromium) 2) I have no idea how to check this all in. Where are all the places I need to edit TestExpections etc.. 3) I'm not sure where it should go. II choose LayoutTests/webgl because some of the tests are not fast. 'm happy to put it anywhere.
Attachment 185705 [details] did not pass style-queue: Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'LayoutTests/ChangeLog', u'LayoutTests/platform/mac/TestExpectations', u'LayoutTests/webgl/conformance/attribs/gl-disabled-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-disabled-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-enable-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-enable-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-render-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-render.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-zero-issues-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-offsets-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-offsets.html', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer.html', u'LayoutTests/webgl/generate-webgl-tests.py', u'LayoutTests/webgl/resources/js-test-pre.js', u'LayoutTests/webgl/resources/webgl-wrapper-template.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/00_test_list.txt', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-disabled-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-enable-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib-render.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib-zero-issues.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertexattribpointer-offsets.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertexattribpointer.html', u'LayoutTests/webgl/resources/webkit-webgl-test-harness.js']" exit_code: 1 LayoutTests/platform/mac/TestExpectations:1219: Path does not exist. [test/expectations] [5] Total errors found: 1 in 28 files If any of these errors are false positives, please file a bug against check-webkit-style.
Comment on attachment 185705 [details] Patch Attachment 185705 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://queues.webkit.org/results/16278100 New failing tests: webgl/conformance/attribs/gl-vertexattribpointer.html webgl/conformance/attribs/gl-vertex-attrib-render.html webgl/conformance/attribs/gl-enable-vertex-attrib.html webgl/conformance/attribs/gl-disabled-vertex-attrib.html webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html webgl/conformance/attribs/gl-vertexattribpointer-offsets.html webgl/conformance/attribs/gl-vertex-attrib.html
Comment on attachment 185705 [details] Patch Attachment 185705 [details] did not pass chromium-ews (chromium-xvfb): Output: http://queues.webkit.org/results/16266200 New failing tests: webgl/conformance/attribs/gl-vertexattribpointer.html webgl/conformance/attribs/gl-vertex-attrib-render.html webgl/conformance/attribs/gl-enable-vertex-attrib.html webgl/conformance/attribs/gl-disabled-vertex-attrib.html webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html webgl/conformance/attribs/gl-vertexattribpointer-offsets.html webgl/conformance/attribs/gl-vertex-attrib.html
Comment on attachment 185705 [details] Patch Attachment 185705 [details] did not pass mac-ews (mac): Output: http://queues.webkit.org/results/16252399 New failing tests: webgl/conformance/attribs/gl-vertexattribpointer.html webgl/conformance/attribs/gl-vertex-attrib-render.html webgl/conformance/attribs/gl-enable-vertex-attrib.html webgl/conformance/attribs/gl-disabled-vertex-attrib.html webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html webgl/conformance/attribs/gl-vertexattribpointer-offsets.html webgl/conformance/attribs/gl-vertex-attrib.html
Created attachment 185868 [details] Patch
Attachment 185868 [details] did not pass style-queue: Failed to run "['Tools/Scripts/check-webkit-style', '--diff-files', u'LayoutTests/ChangeLog', u'LayoutTests/platform/mac/TestExpectations', u'LayoutTests/webgl/conformance/attribs/gl-disabled-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-disabled-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-enable-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-enable-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-render-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-render.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-zero-issues-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html', u'LayoutTests/webgl/conformance/attribs/gl-vertex-attrib.html', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-offsets-expected.txt', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer-offsets.html', u'LayoutTests/webgl/conformance/attribs/gl-vertexattribpointer.html', u'LayoutTests/webgl/generate-webgl-tests.py', u'LayoutTests/webgl/resources/js-test-pre.js', u'LayoutTests/webgl/resources/webgl-expectation-template.txt', u'LayoutTests/webgl/resources/webgl-wrapper-template.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/00_test_list.txt', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-disabled-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-enable-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib-render.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib-zero-issues.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertex-attrib.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertexattribpointer-offsets.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/attribs/gl-vertexattribpointer.html', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/3x3.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/blue-1x1.jpg', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/boolUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/bug-32888-texture.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/floatUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/fragmentShader.frag', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/glsl-conformance-test.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/glsl-feature-tests.css', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/glsl-generator.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-256-with-128-alpha.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-256.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-default-gamma.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-gamma0.1.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-gamma1.0.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-gamma2.0.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-gamma4.0.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp-gamma9.0.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/gray-ramp.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/green-2x2-16bit.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/intArrayUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/intUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/matUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/noopUniformShader.frag', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/noopUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/npot-video.mp4', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/npot-video.theora.ogv', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/npot-video.webmvp8.webm', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/ogles-tests.css', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/pnglib.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/red-green.mp4', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/red-green.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/red-green.theora.ogv', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/red-green.webmvp8.webm', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/red-indexed.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/samplerUniformShader.frag', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-cie-rgb-profile.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-colormatch-profile.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-colorspin-profile.jpg', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-colorspin-profile.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-e-srgb-profile.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-smpte-c-profile.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/small-square-with-srgb-iec61966-2.1-profile.png', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/structUniformShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/tex-image-and-sub-image-2d-with-canvas.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/tex-image-and-sub-image-2d-with-image-data.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/tex-image-and-sub-image-2d-with-image.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/tex-image-and-sub-image-2d-with-video.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/vertexShader.vert', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/webgl-test-utils.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/webgl-test.js', u'LayoutTests/webgl/resources/webgl_test_files/conformance/resources/zero-alpha.png', u'LayoutTests/webgl/resources/webgl_test_files/resources/desktop-gl-constants.js', u'LayoutTests/webgl/resources/webgl_test_files/resources/js-test-post.js', u'LayoutTests/webgl/resources/webgl_test_files/resources/js-test-pre.js', u'LayoutTests/webgl/resources/webgl_test_files/resources/js-test-style.css', u'LayoutTests/webgl/resources/webgl_test_files/resources/test-eval.js', u'LayoutTests/webgl/resources/webgl_test_files/resources/webgl-logo.png', u'LayoutTests/webgl/resources/webgl_test_files/resources/webgl-test-harness.js', u'LayoutTests/webgl/resources/webkit-webgl-test-harness.js']" exit_code: 1 LayoutTests/platform/mac/TestExpectations:1226: Path does not exist. [test/expectations] [5] Total errors found: 1 in 58 files If any of these errors are false positives, please file a bug against check-webkit-style.
Comment on attachment 185868 [details] Patch Attachment 185868 [details] did not pass chromium-ews (chromium-xvfb): Output: http://queues.webkit.org/results/16306202 New failing tests: webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html
Comment on attachment 185868 [details] Patch Attachment 185868 [details] did not pass chromium-ews (chromium-xvfb): Output: http://queues.webkit.org/results/16306238 New failing tests: webgl/conformance/attribs/gl-vertex-attrib-zero-issues.html
Gregg is actively working on this. Assigning to him.
All the tests are checked in but they are all disabled in all builds AFAIK. When I try to turn them on one machine or another on one build or another has problems. The tests are double marked. There's a global 'webgl' turning off all webgl tests in LayoutTests/platform/mac/TestExpectations and LayoutTests/platform/chromium/TestExpectations There are also some entries for individual tests, all currently marked as Skip. If you remove the 'webgl' entry disabling all the test then the ones that are left are tests that failed on my personal machines but you'll need to do more testing on other machines to find out which other tests need to be marked. There's a certain amount of incompatibility between the WebGL tests and the WebKit LayoutTests. A few of the WebGL tests are not fast. There is no criteria that WebGL tests must be fast. They instead have the criteria that the test must communicate with the harness. No communicate = timeout. Each communication (test report) resets the timeout. It would be nice if LayoutTests has a similar mechanism as several tests seem to be right on the edge of making inside LayoutTests timeout limit and therefore end up being flaky
This work was completed under Bug 123188. *** This bug has been marked as a duplicate of bug 123188 ***