Bug 21821 – Redo all reference images

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Pierre-Olivier Latour]

Now that DumpRenderTree has been updated to really be efficient in dealing with pixel tests (see https://bugs.webkit.org/show_bug.cgi?id=21322), we need to redo all reference images.

[Pierre-Olivier Latour]

There is some useful info here: https://bugs.webkit.org/show_bug.cgi?id=9496

[Pierre-Olivier Latour]

Setup:
- iMac X86 (iMac8,1) + ATI HD 2600 Pro w/ stock 10.5.5 + QT 7.5.5
- WebKit SVN 37896
- Clean build in Release configuration

WebKitTools/Scripts/run-webkit-tests --pixel --tolerance 0.0 LayoutTests/
1017.00s total testing time
5826 test cases (58%) succeeded
4102 test cases (41%) had incorrect layout
2 test cases (<1%) crashed
28 test cases (<1%) had stderr output

WebKitTools/Scripts/run-webkit-tests --pixel --tolerance 1.0 LayoutTests/
954.02s total testing time
9712 test cases (97%) succeeded
216 test cases (2%) had incorrect layout
2 test cases (<1%) crashed
28 test cases (<1%) had stderr output

Non pixel test failures:
dom/html/level2/html/HTMLDocument12.html
dom/xhtml/level2/html/HTMLDocument12.xhtml
fast/cookies/local-file-can-set-cookies.html
fast/profiler/anonymous-event-handler.html
fast/profiler/anonymous-function-called-from-different-contexts.html
fast/profiler/anonymous-function-calls-built-in-functions.html
fast/profiler/anonymous-function-calls-eval.html
fast/profiler/apply.html
fast/profiler/built-in-function-calls-anonymous.html
fast/profiler/built-in-function-calls-user-defined-function.html
fast/profiler/call-nodelist-as-function.html
fast/profiler/call.html
fast/profiler/calling-the-function-that-started-the-profiler-from-another-scope.html
fast/profiler/compare-multiple-profiles.html
fast/profiler/constructor.html
fast/profiler/dead-time.html
fast/profiler/document-dot-write.html
fast/profiler/event-handler.html
fast/profiler/execution-context-and-eval-on-same-line.html
fast/profiler/heavy-view.html
fast/profiler/inline-event-handler.html
fast/profiler/many-calls-in-the-same-scope.html
fast/profiler/multiple-and-different-scoped-anonymous-function-calls.html
fast/profiler/multiple-and-different-scoped-function-calls.html
fast/profiler/multiple-frames.html
fast/profiler/nested-anonymous-functon.html
fast/profiler/nested-start-and-stop-profiler.html
fast/profiler/one-execution-context.html
fast/profiler/profile-calls-in-included-file.html
fast/profiler/profiling-from-a-nested-location-but-stop-profiling-outside-the-nesting.html
fast/profiler/profiling-from-a-nested-location.html
fast/profiler/simple-event-call.html
fast/profiler/simple-no-level-change.html
fast/profiler/start-and-stop-profiler-multiple-times.html
fast/profiler/stop-profiling-after-setTimeout.html
fast/profiler/stop-then-function-call.html
fast/profiler/throw-exception-from-eval.html
fast/profiler/two-execution-contexts.html
fast/profiler/user-defined-function-calls-built-in-functions.html
fast/profiler/window-dot-eval.html

Font rendering differences (acceptable failures):
editing/pasteboard/block-wrappers-necessary.html
fast/backgrounds/repeat/negative-offset-repeat-transformed.html
fast/block/float/editable-text-overlapping-float.html
fast/css/css3-nth-child.html
fast/css/font-face-multiple-faces.html
fast/dynamic/insert-before-table-part-in-continuation.html
fast/frames/frame-scrolling-attribute.html
fast/frames/iframe-scrolling-attribute.html
fast/inline/inline-borders-with-bidi-override.html
fast/inline/inline-text-quirk-bpm.html
fast/repaint/line-flow-with-floats-1.html
fast/repaint/line-flow-with-floats-2.html
fast/repaint/line-flow-with-floats-3.html
fast/repaint/line-flow-with-floats-4.html
fast/repaint/line-flow-with-floats-5.html
fast/repaint/line-flow-with-floats-8.html
fast/repaint/line-flow-with-floats-9.html
fast/repaint/line-flow-with-floats-10.html
fast/table/frame-and-rules.html
fast/text/atsui-spacing-features.html
fast/text/capitalize-boundaries.html
fast/text/trailing-white-space-2.html
fast/text/trailing-white-space.html
fast/text/international/complex-character-based-fallback.html
platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html
scrollbars/overflow-scrollbar-combinations.html
svg/W3C-SVG-1.1/animate-elem-04-t.svg
svg/W3C-SVG-1.1/animate-elem-05-t.svg
svg/W3C-SVG-1.1/animate-elem-06-t.svg
svg/W3C-SVG-1.1/animate-elem-07-t.svg
svg/W3C-SVG-1.1/animate-elem-09-t.svg
svg/W3C-SVG-1.1/animate-elem-10-t.svg
svg/W3C-SVG-1.1/animate-elem-11-t.svg
svg/W3C-SVG-1.1/animate-elem-12-t.svg
svg/W3C-SVG-1.1/animate-elem-14-t.svg
svg/W3C-SVG-1.1/animate-elem-15-t.svg
svg/W3C-SVG-1.1/animate-elem-16-t.svg
svg/W3C-SVG-1.1/animate-elem-17-t.svg
svg/W3C-SVG-1.1/animate-elem-18-t.svg
svg/W3C-SVG-1.1/animate-elem-19-t.svg
svg/W3C-SVG-1.1/animate-elem-31-t.svg
svg/W3C-SVG-1.1/animate-elem-33-t.svg
svg/W3C-SVG-1.1/animate-elem-34-t.svg
svg/W3C-SVG-1.1/animate-elem-40-t.svg
svg/W3C-SVG-1.1/animate-elem-44-t.svg
svg/W3C-SVG-1.1/animate-elem-52-t.svg
svg/W3C-SVG-1.1/animate-elem-80-t.svg
svg/W3C-SVG-1.1/animate-elem-81-t.svg
svg/W3C-SVG-1.1/animate-elem-82-t.svg
svg/W3C-SVG-1.1/animate-elem-83-t.svg
svg/W3C-SVG-1.1/coords-units-01-b.svg
svg/W3C-SVG-1.1/coords-units-02-b.svg
svg/W3C-SVG-1.1/coords-units-03-b.svg
svg/W3C-SVG-1.1/coords-viewattr-03-b.svg
svg/W3C-SVG-1.1/filters-color-01-b.svg
svg/W3C-SVG-1.1/filters-composite-02-b.svg
svg/W3C-SVG-1.1/filters-comptran-01-b.svg
svg/W3C-SVG-1.1/filters-diffuse-01-f.svg
svg/W3C-SVG-1.1/filters-light-01-f.svg
svg/W3C-SVG-1.1/filters-specular-01-f.svg
svg/W3C-SVG-1.1/filters-turb-01-f.svg
svg/W3C-SVG-1.1/fonts-kern-01-t.svg
svg/W3C-SVG-1.1/interact-events-01-b.svg
svg/W3C-SVG-1.1/interact-order-01-b.svg
svg/W3C-SVG-1.1/interact-order-02-b.svg
svg/W3C-SVG-1.1/interact-order-03-b.svg
svg/W3C-SVG-1.1/linking-uri-01-b.svg
svg/W3C-SVG-1.1/linking-uri-02-b.svg
svg/W3C-SVG-1.1/masking-opacity-01-b.svg
svg/W3C-SVG-1.1/masking-path-01-b.svg
svg/W3C-SVG-1.1/masking-path-02-b.svg
svg/W3C-SVG-1.1/masking-path-03-b.svg
svg/W3C-SVG-1.1/painting-fill-01-t.svg
svg/W3C-SVG-1.1/painting-fill-02-t.svg
svg/W3C-SVG-1.1/painting-fill-03-t.svg
svg/W3C-SVG-1.1/painting-stroke-01-t.svg
svg/W3C-SVG-1.1/painting-stroke-02-t.svg
svg/W3C-SVG-1.1/painting-stroke-03-t.svg
svg/W3C-SVG-1.1/painting-stroke-04-t.svg
svg/W3C-SVG-1.1/paths-data-10-t.svg
svg/W3C-SVG-1.1/pservers-grad-04-b.svg
svg/W3C-SVG-1.1/pservers-grad-06-b.svg
svg/W3C-SVG-1.1/pservers-grad-07-b.svg
svg/W3C-SVG-1.1/pservers-grad-08-b.svg
svg/W3C-SVG-1.1/pservers-grad-09-b.svg
svg/W3C-SVG-1.1/pservers-grad-12-b.svg
svg/W3C-SVG-1.1/pservers-grad-14-b.svg
svg/W3C-SVG-1.1/script-handle-02-b.svg
svg/W3C-SVG-1.1/script-handle-03-b.svg
svg/W3C-SVG-1.1/script-handle-04-b.svg
svg/W3C-SVG-1.1/struct-dom-03-b.svg
svg/W3C-SVG-1.1/struct-dom-04-b.svg
svg/W3C-SVG-1.1/struct-dom-05-b.svg
svg/W3C-SVG-1.1/struct-group-03-t.svg
svg/W3C-SVG-1.1/struct-use-01-t.svg
svg/W3C-SVG-1.1/styling-css-02-b.svg
svg/W3C-SVG-1.1/text-align-01-b.svg
svg/W3C-SVG-1.1/text-align-02-b.svg
svg/W3C-SVG-1.1/text-align-03-b.svg
svg/W3C-SVG-1.1/text-align-04-b.svg
svg/W3C-SVG-1.1/text-align-05-b.svg
svg/W3C-SVG-1.1/text-align-06-b.svg
svg/W3C-SVG-1.1/text-altglyph-01-b.svg
svg/W3C-SVG-1.1/text-fonts-01-t.svg
svg/W3C-SVG-1.1/text-fonts-02-t.svg
svg/W3C-SVG-1.1/text-intro-05-t.svg
svg/W3C-SVG-1.1/text-path-01-b.svg
svg/W3C-SVG-1.1/text-spacing-01-b.svg
svg/W3C-SVG-1.1/text-text-01-b.svg
svg/W3C-SVG-1.1/text-text-03-b.svg
svg/W3C-SVG-1.1/text-tselect-01-b.svg
svg/W3C-SVG-1.1/text-tselect-02-f.svg
svg/W3C-SVG-1.1/text-tspan-01-b.svg
svg/W3C-SVG-1.1/text-ws-01-t.svg
svg/W3C-SVG-1.1/text-ws-02-t.svg
svg/W3C-SVG-1.1/types-basicDOM-01-b.svg
svg/batik/text/longTextOnPath.svg
svg/batik/text/textProperties.svg
svg/batik/text/textStyles.svg
svg/batik/text/xmlSpace.svg
svg/custom/altglyph.svg
svg/custom/dynamic-svg-document-creation.svg
svg/custom/pointer-events-path.svg
svg/custom/scrolling-embedded-svg-file-image-repaint-problem.html
svg/custom/text-whitespace-handling.svg
svg/text/kerning.svg
svg/text/multichar-glyph.svg
svg/text/text-align-02-b.svg
svg/text/text-fonts-01-t.svg
svg/text/text-path-01-b.svg
svg/text/text-spacing-01-b.svg
svg/text/text-text-01-b.svg
svg/text/text-text-03-b.svg
svg/text/text-tselect-01-b.svg
svg/text/text-tspan-01-b.svg
svg/text/text-ws-01-t.svg
svg/text/text-ws-02-t.svg
tables/mozilla/core/bloomberg.html
tables/mozilla_expected_failures/bugs/bug2479-5.html

Reference image at wrong location (move to platform/mac):
invalid-percentage-property-expected.png
lang-inheritance-expected.png
lang-inheritance2-expected.png
lang-vs-xml-lang-expected.png
lang-vs-xml-lang-xhtml-expected.png
text-align-expected.png
paste-text-013-expected.png

Reference images needs to be deleted:
5607069-1-expected.png

Reference image not up-to-date:
css2.1/t1202-counter-09-b.html
css2.1/t1202-counters-09-b.html
fast/flexbox/016.html
fast/images/pdf-as-background.html
editing/style/5279521.html
fast/frames/001.html
fast/inline/continuation-outlines-with-layers.html
fast/inline/continuation-outlines.html
fast/parser/broken-comments-vs-parsing-mode.html
fast/text/international/thai-line-breaks.html
fast/text/whitespace/013.html
fast/xsl/document-function.xml
fast/xsl/xslt-relative-path.xml
tables/mozilla/bugs/bug2973.html

SVG not rendering (https://bugs.webkit.org/show_bug.cgi?id=21910):
fast/backgrounds/svg-as-background-1.html
fast/backgrounds/svg-as-background-2.html
fast/backgrounds/svg-as-background-3.html
fast/backgrounds/svg-as-background-5.html
fast/backgrounds/svg-as-background-6.html
fast/backgrounds/svg-as-mask.html
fast/borders/svg-as-border-image-2.html
fast/borders/svg-as-border-image.html
fast/images/svg-as-background.html
fast/images/svg-as-image.html
fast/images/svg-as-relative-image.html
fast/images/svg-as-tiled-background.html

No video images (https://bugs.webkit.org/show_bug.cgi?id=21911):
media/video-aspect-ratio.html
media/video-controls-rendering.html
media/video-display-toggle.html
media/video-layer-crash.html
media/video-transformed.html

Possible Flash failure (https://bugs.webkit.org/show_bug.cgi?id=21912):
fast/dynamic/flash-replacement-test.html

Repaint failure (https://bugs.webkit.org/show_bug.cgi?id=21913):
fast/repaint/body-background-image.html

Excessive repainting (https://bugs.webkit.org/show_bug.cgi?id=21914):
fast/repaint/float-overflow-right.html
fast/repaint/float-overflow.html

Invalid image tiling (https://bugs.webkit.org/show_bug.cgi?id=21915):
fast/backgrounds/size/backgroundSize19.html

Test doesn't repaint entire view so result is corrupted by previous test (https://bugs.webkit.org/show_bug.cgi?id=21916):
tables/mozilla_expected_failures/bugs/bug178855.xml

[Pierre-Olivier Latour]

(In reply to comment #2)
> Reference image at wrong location (move to platform/mac):
> invalid-percentage-property-expected.png
> lang-inheritance-expected.png
> lang-inheritance2-expected.png
> lang-vs-xml-lang-expected.png
> lang-vs-xml-lang-xhtml-expected.png
> text-align-expected.png
> paste-text-013-expected.png

Actually, "paste-text-013-expected.png" is in 2 places, so the one not in platform simply needs to be deleted

[Pierre-Olivier Latour]

Created attachment 24720 [details]
Clean up of reference images

This patch takes care of the first set of issues: ensure all bases are in the right location and delete the ones that aren't needed.

Comment on attachment 24720 [details]
Clean up of reference images

r=me
There's a tab character in the change log but I will fix that when committing the patch.

Comment on attachment 24720 [details]
Clean up of reference images

Clearing the review flag after committing this in r37927.

[Pierre-Olivier Latour]

First set of new image bases was committed here:
http://trac.webkit.org/changeset/38008

[Pierre-Olivier Latour]

(In reply to comment #7)
> First set of new image bases was committed here:
> http://trac.webkit.org/changeset/38008

The re-generated bases come from the same iMac using a release build of WebKit #r38003:
LayoutTests/svg/custom/
LayoutTests/fast/borders/
LayoutTests/fast/css/

[Pierre-Olivier Latour]

Since the bases were generated on 37896, there was one new base checked in independently:
http://trac.webkit.org/changeset/37967/trunk/LayoutTests

[Pierre-Olivier Latour]

Important note on tolerances: to avoid failures due to slight font rendering differences or CG dithering on various HW, pixel tests are expected to be run with a small tolerance:
- Leopard: 0.1%
- Tiger: 1.0%

[Pierre-Olivier Latour]

Created attachment 24880 [details]
Fixed reference images on Tiger

Fixed reference images on Tiger for:
LayoutTests/svg/custom/
LayoutTests/fast/borders/
LayoutTests/fast/css/

Comment on attachment 24880 [details]
Fixed reference images on Tiger

r=me
I am going to tweak the change log a bit when landing this.

Comment on attachment 24880 [details]
Fixed reference images on Tiger

Landed as <http://trac.webkit.org/changeset/38103>. Clearing the review flag.

More updated expected results landed in <http://trac.webkit.org/changeset/38121> and <http://trac.webkit.org/changeset/38122>, including accessibility/, animations/, css1/, css2.1/, css3/, dom/, editing/, fast/ and svg/.

[Pierre-Olivier Latour]

Created attachment 24931 [details]
Other Tiger patch

Fixed more reference images on Tiger

[Pierre-Olivier Latour]

Note that once all new bases are checked in and both Tiger and Leopard are running fine, we need to check in "exception" bases for tests that fail on Tiger with a difference > 1.0% but are still valid.

Comment on attachment 24931 [details]
Other Tiger patch

+        - part of https://bugs.webkit.org/show_bug.cgi?id=21821
+          Removed pixel results in platform/mac-tiger that were only slightly different
+          (within 1% tolerance) from the platform/mac results.
+          Removed results in platform/mac-tiger where the results in platform/mac were
+          also correct for Tiger.
+

The patch also modifies several files (I think just updating images to match the correct render trees already checked in) and you should mention that in the change log.

It also adds files. I don't understand why:
A         LayoutTests/platform/mac-tiger/fast/text/international/complex-character-based-fallback-expected.txt
A         LayoutTests/platform/mac-tiger/fast/text/international/hindi-spacing-expected.txt

[Pierre-Olivier Latour]

(In reply to comment #9)
> Since the bases were generated on 37896, there was one new base checked in
> independently:
> http://trac.webkit.org/changeset/37967/trunk/LayoutTests

http://trac.webkit.org/changeset/38237/trunk/LayoutTests
http://trac.webkit.org/changeset/38273/trunk/LayoutTests

[Pierre-Olivier Latour]

Created attachment 25040 [details]
Revised other Tiger patch

Updated patch following discussion with Dan

Comment on attachment 25040 [details]
Revised other Tiger patch

rs=me

[Pierre-Olivier Latour]

Last round of check in for new bases:
http://trac.webkit.org/changeset/38297/trunk/LayoutTests

Comment on attachment 25040 [details]
Revised other Tiger patch

Landed this in r38305.

[Pierre-Olivier Latour]

Created attachment 25080 [details]
Updated reference images for Leopard

Comment on attachment 25080 [details]
Updated reference images for Leopard

r=me!

Comment on attachment 25080 [details]
Updated reference images for Leopard

Landed in r38335

[Pierre-Olivier Latour]

Created attachment 25110 [details]
Updated reference images for Tiger

Comment on attachment 25110 [details]
Updated reference images for Tiger

r=me

Comment on attachment 25110 [details]
Updated reference images for Tiger

Landed in r38354.

[Pierre-Olivier Latour]

We're finally done here!