In the interests of making accelerated drawing, the remote layer tree, and other settings more testable, I have a multi-part change: * Add --remote-layer-tree and --accelerated-drawing options to run-webkit-tests, which are passed through to the test runner. These are similar in nature to the existing --complex-text option; I've pulled the three out into their own 'feature switches' options section (anticipating growth here). * Add --treat-ref-tests-as-pixel-tests option to run-webkit-tests, which ... treats ref tests as pixel tests. This is useful for the script below... * Add a script that: * In the default configuration, runs all tests with no regard for TestExpectations, dumping text and pixel results (even for ref tests, but not for -expected.* reference files, of course) into a temporary directory. * In the modified configuration (i.e. passing --remote-layer-tree, or --accelerated-drawing, or even -2 if you want to test WebKit2 vs WebKit1), run all tests again with no expectations, against the results we dumped the first time. * The resultant test result diffs represent changes due to the option you turned on (modulo flakiness and whatnot). Does this sound reasonable? I'm still finishing up the script, but it all seems to work pretty well. <rdar://problem/15199108>
Hmm, would it be better to do the comparison test-by-test rather than running all in one mode first, followed by all in another mode?
(In reply to comment #1) > Hmm, would it be better to do the comparison test-by-test rather than running all in one mode first, followed by all in another mode? I'm not sure what that gains us, and it would make it much more complicated, as that's not possible not possible with two (heavily argumented) instantiations of run-webkit-tests. What were you thinking of as benefits?
Created attachment 218454 [details] take 1 probably need some ifdefs
Comment on attachment 218454 [details] take 1 View in context: https://bugs.webkit.org/attachment.cgi?id=218454&action=review > Tools/DumpRenderTree/mac/DumpRenderTree.mm:658 > - [preferences setCanvasUsesAcceleratedDrawing:YES]; > - [preferences setAcceleratedDrawingEnabled:NO]; > + [preferences setCanvasUsesAcceleratedDrawing:useAcceleratedDrawing]; > + [preferences setAcceleratedDrawingEnabled:useAcceleratedDrawing]; whoops!
Created attachment 218459 [details] take 2 - don't disable canvas accelerated drawing
Comment on attachment 218459 [details] take 2 - don't disable canvas accelerated drawing Attachment 218459 [details] did not pass mac-wk2-ews (mac-wk2): Output: http://webkit-queues.appspot.com/results/43388073 New failing tests: platform/mac-wk2/tiled-drawing/fixed/nested-fixed.html svg/custom/masking-clipping-hidpi.svg platform/mac-wk2/tiled-drawing/fixed/percentage-inside-fixed.html platform/mac-wk2/tiled-drawing/fixed/four-bars.html platform/mac-wk2/tiled-drawing/sticky/negative-scroll-offset.html platform/mac-wk2/tiled-drawing/sticky/sticky-layers.html platform/mac-wk2/tiled-drawing/fixed/negative-scroll-offset.html platform/mac-wk2/tiled-drawing/clamp-out-of-bounds-scrolls.html platform/mac-wk2/tiled-drawing/fixed/fixed-position-out-of-view.html platform/mac-wk2/tiled-drawing/tile-size-slow-zoomed.html platform/mac-wk2/tiled-drawing/fixed-background/fixed-non-propagated-body-background.html platform/mac-wk2/tiled-drawing/slow-scrolling.html platform/mac-wk2/tiled-drawing/fixed/fixed-position-out-of-view-negative-zindex.html platform/mac-wk2/tiled-drawing/fixed/negative-scroll-offset-in-view.html platform/mac-wk2/tiled-drawing/sticky/sticky-horizontal.html platform/mac-wk2/tiled-drawing/fixed/fixed-in-overflow.html platform/mac-wk2/tiled-drawing/fixed/absolute-inside-out-of-view-fixed.html platform/mac-wk2/tiled-drawing/tile-coverage-slow-scrolling.html platform/mac-wk2/tiled-drawing/fixed/four-bars-with-header-and-footer.html platform/mac-wk2/tiled-drawing/scrolling-tree-slow-scrolling.html platform/mac-wk2/tiled-drawing/scrolling-tree-after-scroll.html platform/mac-wk2/tiled-drawing/fixed/absolute-inside-fixed.html
Created attachment 218480 [details] Archive of layout-test-results from webkit-ews-09 for mac-mountainlion-wk2 The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews. Bot: webkit-ews-09 Port: mac-mountainlion-wk2 Platform: Mac OS X 10.8.5
Created attachment 218732 [details] take 3 - unbox the WKBooleanRef :|
Created attachment 218733 [details] take 4 - take 3 was missing the added file...
Comment on attachment 218733 [details] take 4 - take 3 was missing the added file... View in context: https://bugs.webkit.org/attachment.cgi?id=218733&action=review > Tools/WebKitTestRunner/TestController.cpp:409 > + WKRetainPtr<WKStringRef> useRemoteLayerTreeKey = adoptWK(WKStringCreateWithUTF8CString("RemoteLayerTree")); > + WKRetainPtr<WKBooleanRef> useRemoteLayerTreeValue = adoptWK(WKBooleanCreate(m_shouldUseRemoteLayerTree)); > + WKDictionaryAddItem(viewOptions.get(), useRemoteLayerTreeKey.get(), useRemoteLayerTreeValue.get()); Helper functions could make code like this way easier to read. WKDictionaryAddItem(viewOptions.get(), createWKString("removeLayerTree").get(), createWKBoolean(m_shouldUseRemoteLayerTree).get()); I seem to recall making functions like that for WebKitTestRunner, but for some reason I don’t see them.
(In reply to comment #10) > (From update of attachment 218733 [details]) > View in context: https://bugs.webkit.org/attachment.cgi?id=218733&action=review > > > Tools/WebKitTestRunner/TestController.cpp:409 > > + WKRetainPtr<WKStringRef> useRemoteLayerTreeKey = adoptWK(WKStringCreateWithUTF8CString("RemoteLayerTree")); > > + WKRetainPtr<WKBooleanRef> useRemoteLayerTreeValue = adoptWK(WKBooleanCreate(m_shouldUseRemoteLayerTree)); > > + WKDictionaryAddItem(viewOptions.get(), useRemoteLayerTreeKey.get(), useRemoteLayerTreeValue.get()); > > Helper functions could make code like this way easier to read. > > WKDictionaryAddItem(viewOptions.get(), createWKString("removeLayerTree").get(), createWKBoolean(m_shouldUseRemoteLayerTree).get()); > > I seem to recall making functions like that for WebKitTestRunner, but for some reason I don’t see them. Agreed! I didn't do that because I wasn't sure the best place to put them, though. Nor could I find any that you might have created in the past. http://trac.webkit.org/changeset/160459