Bug 125212 – Make it possible to compare layout test results between various configurations of WebKit

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Tim Horton]

In the interests of making accelerated drawing, the remote layer tree, and other settings more testable, I have a multi-part change:

* Add --remote-layer-tree and --accelerated-drawing options to run-webkit-tests, which are passed through to the test runner. These are similar in nature to the existing --complex-text option; I've pulled the three out into their own 'feature switches' options section (anticipating growth here).

* Add --treat-ref-tests-as-pixel-tests option to run-webkit-tests, which ... treats ref tests as pixel tests. This is useful for the script below...

* Add a script that:
    * In the default configuration, runs all tests with no regard for TestExpectations, dumping text and pixel results (even for ref tests, but not for -expected.* reference files, of course) into a temporary directory.
    * In the modified configuration (i.e. passing --remote-layer-tree, or --accelerated-drawing, or even -2 if you want to test WebKit2 vs WebKit1), run all tests again with no expectations, against the results we dumped the first time.
    * The resultant test result diffs represent changes due to the option you turned on (modulo flakiness and whatnot).

Does this sound reasonable? I'm still finishing up the script, but it all seems to work pretty well.

<rdar://problem/15199108>

[Simon Fraser (smfr)]

Hmm, would it be better to do the comparison test-by-test rather than running all in one mode first, followed by all in another mode?

[Tim Horton]

(In reply to comment #1)
> Hmm, would it be better to do the comparison test-by-test rather than running all in one mode first, followed by all in another mode?

I'm not sure what that gains us, and it would make it much more complicated, as that's not possible not possible with two (heavily argumented) instantiations of run-webkit-tests. What were you thinking of as benefits?

[Tim Horton]

Created attachment 218454 [details]
take 1

probably need some ifdefs

[Tim Horton]

Comment on attachment 218454 [details]
take 1

View in context: https://bugs.webkit.org/attachment.cgi?id=218454&action=review

> Tools/DumpRenderTree/mac/DumpRenderTree.mm:658
> -    [preferences setCanvasUsesAcceleratedDrawing:YES];
> -    [preferences setAcceleratedDrawingEnabled:NO];
> +    [preferences setCanvasUsesAcceleratedDrawing:useAcceleratedDrawing];
> +    [preferences setAcceleratedDrawingEnabled:useAcceleratedDrawing];

whoops!

[Tim Horton]

Created attachment 218459 [details]
take 2 - don't disable canvas accelerated drawing

[Build Bot]

Comment on attachment 218459 [details]
take 2 - don't disable canvas accelerated drawing

Attachment 218459 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/43388073

New failing tests:
platform/mac-wk2/tiled-drawing/fixed/nested-fixed.html
svg/custom/masking-clipping-hidpi.svg
platform/mac-wk2/tiled-drawing/fixed/percentage-inside-fixed.html
platform/mac-wk2/tiled-drawing/fixed/four-bars.html
platform/mac-wk2/tiled-drawing/sticky/negative-scroll-offset.html
platform/mac-wk2/tiled-drawing/sticky/sticky-layers.html
platform/mac-wk2/tiled-drawing/fixed/negative-scroll-offset.html
platform/mac-wk2/tiled-drawing/clamp-out-of-bounds-scrolls.html
platform/mac-wk2/tiled-drawing/fixed/fixed-position-out-of-view.html
platform/mac-wk2/tiled-drawing/tile-size-slow-zoomed.html
platform/mac-wk2/tiled-drawing/fixed-background/fixed-non-propagated-body-background.html
platform/mac-wk2/tiled-drawing/slow-scrolling.html
platform/mac-wk2/tiled-drawing/fixed/fixed-position-out-of-view-negative-zindex.html
platform/mac-wk2/tiled-drawing/fixed/negative-scroll-offset-in-view.html
platform/mac-wk2/tiled-drawing/sticky/sticky-horizontal.html
platform/mac-wk2/tiled-drawing/fixed/fixed-in-overflow.html
platform/mac-wk2/tiled-drawing/fixed/absolute-inside-out-of-view-fixed.html
platform/mac-wk2/tiled-drawing/tile-coverage-slow-scrolling.html
platform/mac-wk2/tiled-drawing/fixed/four-bars-with-header-and-footer.html
platform/mac-wk2/tiled-drawing/scrolling-tree-slow-scrolling.html
platform/mac-wk2/tiled-drawing/scrolling-tree-after-scroll.html
platform/mac-wk2/tiled-drawing/fixed/absolute-inside-fixed.html

[Build Bot]

Created attachment 218480 [details]
Archive of layout-test-results from webkit-ews-09 for mac-mountainlion-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: webkit-ews-09  Port: mac-mountainlion-wk2  Platform: Mac OS X 10.8.5

[Tim Horton]

Created attachment 218732 [details]
take 3 - unbox the WKBooleanRef :|

[Tim Horton]

Created attachment 218733 [details]
take 4 - take 3 was missing the added file...

[Darin Adler]

Comment on attachment 218733 [details]
take 4 - take 3 was missing the added file...

View in context: https://bugs.webkit.org/attachment.cgi?id=218733&action=review

> Tools/WebKitTestRunner/TestController.cpp:409
> +        WKRetainPtr<WKStringRef> useRemoteLayerTreeKey = adoptWK(WKStringCreateWithUTF8CString("RemoteLayerTree"));
> +        WKRetainPtr<WKBooleanRef> useRemoteLayerTreeValue = adoptWK(WKBooleanCreate(m_shouldUseRemoteLayerTree));
> +        WKDictionaryAddItem(viewOptions.get(), useRemoteLayerTreeKey.get(), useRemoteLayerTreeValue.get());

Helper functions could make code like this way easier to read.

    WKDictionaryAddItem(viewOptions.get(), createWKString("removeLayerTree").get(), createWKBoolean(m_shouldUseRemoteLayerTree).get());

I seem to recall making functions like that for WebKitTestRunner, but for some reason I don’t see them.

[Tim Horton]

(In reply to comment #10)
> (From update of attachment 218733 [details])
> View in context: https://bugs.webkit.org/attachment.cgi?id=218733&action=review
> 
> > Tools/WebKitTestRunner/TestController.cpp:409
> > +        WKRetainPtr<WKStringRef> useRemoteLayerTreeKey = adoptWK(WKStringCreateWithUTF8CString("RemoteLayerTree"));
> > +        WKRetainPtr<WKBooleanRef> useRemoteLayerTreeValue = adoptWK(WKBooleanCreate(m_shouldUseRemoteLayerTree));
> > +        WKDictionaryAddItem(viewOptions.get(), useRemoteLayerTreeKey.get(), useRemoteLayerTreeValue.get());
> 
> Helper functions could make code like this way easier to read.
> 
>     WKDictionaryAddItem(viewOptions.get(), createWKString("removeLayerTree").get(), createWKBoolean(m_shouldUseRemoteLayerTree).get());
> 
> I seem to recall making functions like that for WebKitTestRunner, but for some reason I don’t see them.

Agreed! I didn't do that because I wasn't sure the best place to put them, though. Nor could I find any that you might have created in the past.

http://trac.webkit.org/changeset/160459