Bug 21821 - Redo all reference images
Summary: Redo all reference images
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Macintosh OS X 10.5
: P2 Normal
Assignee: Nobody
URL:
Keywords:
Depends on: 21322
Blocks: 21988
  Show dependency treegraph
 
Reported: 2008-10-22 19:48 PDT by Pierre-Olivier Latour
Modified: 2008-11-12 17:50 PST (History)
2 users (show)

See Also:


Attachments
Clean up of reference images (96.47 KB, patch)
2008-10-28 10:07 PDT, Pierre-Olivier Latour
no flags Details | Formatted Diff | Diff
Fixed reference images on Tiger (268.59 KB, patch)
2008-11-03 19:07 PST, Pierre-Olivier Latour
no flags Details | Formatted Diff | Diff
Other Tiger patch (deleted)
2008-11-05 18:21 PST, Pierre-Olivier Latour
no flags Details | Formatted Diff | Diff
Revised other Tiger patch (1.84 MB, patch)
2008-11-10 19:22 PST, Pierre-Olivier Latour
no flags Details | Formatted Diff | Diff
Updated reference images for Leopard (774.89 KB, patch)
2008-11-11 17:26 PST, Pierre-Olivier Latour
no flags Details | Formatted Diff | Diff
Updated reference images for Tiger (1.63 MB, patch)
2008-11-12 17:38 PST, Pierre-Olivier Latour
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Pierre-Olivier Latour 2008-10-22 19:48:03 PDT
Now that DumpRenderTree has been updated to really be efficient in dealing with pixel tests (see https://bugs.webkit.org/show_bug.cgi?id=21322), we need to redo all reference images.
Comment 1 Pierre-Olivier Latour 2008-10-22 19:49:00 PDT
There is some useful info here: https://bugs.webkit.org/show_bug.cgi?id=9496
Comment 2 Pierre-Olivier Latour 2008-10-27 18:09:45 PDT
Setup:
- iMac X86 (iMac8,1) + ATI HD 2600 Pro w/ stock 10.5.5 + QT 7.5.5
- WebKit SVN 37896
- Clean build in Release configuration

WebKitTools/Scripts/run-webkit-tests --pixel --tolerance 0.0 LayoutTests/
1017.00s total testing time
5826 test cases (58%) succeeded
4102 test cases (41%) had incorrect layout
2 test cases (<1%) crashed
28 test cases (<1%) had stderr output

WebKitTools/Scripts/run-webkit-tests --pixel --tolerance 1.0 LayoutTests/
954.02s total testing time
9712 test cases (97%) succeeded
216 test cases (2%) had incorrect layout
2 test cases (<1%) crashed
28 test cases (<1%) had stderr output

Non pixel test failures:
dom/html/level2/html/HTMLDocument12.html
dom/xhtml/level2/html/HTMLDocument12.xhtml
fast/cookies/local-file-can-set-cookies.html
fast/profiler/anonymous-event-handler.html
fast/profiler/anonymous-function-called-from-different-contexts.html
fast/profiler/anonymous-function-calls-built-in-functions.html
fast/profiler/anonymous-function-calls-eval.html
fast/profiler/apply.html
fast/profiler/built-in-function-calls-anonymous.html
fast/profiler/built-in-function-calls-user-defined-function.html
fast/profiler/call-nodelist-as-function.html
fast/profiler/call.html
fast/profiler/calling-the-function-that-started-the-profiler-from-another-scope.html
fast/profiler/compare-multiple-profiles.html
fast/profiler/constructor.html
fast/profiler/dead-time.html
fast/profiler/document-dot-write.html
fast/profiler/event-handler.html
fast/profiler/execution-context-and-eval-on-same-line.html
fast/profiler/heavy-view.html
fast/profiler/inline-event-handler.html
fast/profiler/many-calls-in-the-same-scope.html
fast/profiler/multiple-and-different-scoped-anonymous-function-calls.html
fast/profiler/multiple-and-different-scoped-function-calls.html
fast/profiler/multiple-frames.html
fast/profiler/nested-anonymous-functon.html
fast/profiler/nested-start-and-stop-profiler.html
fast/profiler/one-execution-context.html
fast/profiler/profile-calls-in-included-file.html
fast/profiler/profiling-from-a-nested-location-but-stop-profiling-outside-the-nesting.html
fast/profiler/profiling-from-a-nested-location.html
fast/profiler/simple-event-call.html
fast/profiler/simple-no-level-change.html
fast/profiler/start-and-stop-profiler-multiple-times.html
fast/profiler/stop-profiling-after-setTimeout.html
fast/profiler/stop-then-function-call.html
fast/profiler/throw-exception-from-eval.html
fast/profiler/two-execution-contexts.html
fast/profiler/user-defined-function-calls-built-in-functions.html
fast/profiler/window-dot-eval.html

Font rendering differences (acceptable failures):
editing/pasteboard/block-wrappers-necessary.html
fast/backgrounds/repeat/negative-offset-repeat-transformed.html
fast/block/float/editable-text-overlapping-float.html
fast/css/css3-nth-child.html
fast/css/font-face-multiple-faces.html
fast/dynamic/insert-before-table-part-in-continuation.html
fast/frames/frame-scrolling-attribute.html
fast/frames/iframe-scrolling-attribute.html
fast/inline/inline-borders-with-bidi-override.html
fast/inline/inline-text-quirk-bpm.html
fast/repaint/line-flow-with-floats-1.html
fast/repaint/line-flow-with-floats-2.html
fast/repaint/line-flow-with-floats-3.html
fast/repaint/line-flow-with-floats-4.html
fast/repaint/line-flow-with-floats-5.html
fast/repaint/line-flow-with-floats-8.html
fast/repaint/line-flow-with-floats-9.html
fast/repaint/line-flow-with-floats-10.html
fast/table/frame-and-rules.html
fast/text/atsui-spacing-features.html
fast/text/capitalize-boundaries.html
fast/text/trailing-white-space-2.html
fast/text/trailing-white-space.html
fast/text/international/complex-character-based-fallback.html
platform/mac/fast/text/international/Geeza-Pro-vertical-metrics-adjustment.html
scrollbars/overflow-scrollbar-combinations.html
svg/W3C-SVG-1.1/animate-elem-04-t.svg
svg/W3C-SVG-1.1/animate-elem-05-t.svg
svg/W3C-SVG-1.1/animate-elem-06-t.svg
svg/W3C-SVG-1.1/animate-elem-07-t.svg
svg/W3C-SVG-1.1/animate-elem-09-t.svg
svg/W3C-SVG-1.1/animate-elem-10-t.svg
svg/W3C-SVG-1.1/animate-elem-11-t.svg
svg/W3C-SVG-1.1/animate-elem-12-t.svg
svg/W3C-SVG-1.1/animate-elem-14-t.svg
svg/W3C-SVG-1.1/animate-elem-15-t.svg
svg/W3C-SVG-1.1/animate-elem-16-t.svg
svg/W3C-SVG-1.1/animate-elem-17-t.svg
svg/W3C-SVG-1.1/animate-elem-18-t.svg
svg/W3C-SVG-1.1/animate-elem-19-t.svg
svg/W3C-SVG-1.1/animate-elem-31-t.svg
svg/W3C-SVG-1.1/animate-elem-33-t.svg
svg/W3C-SVG-1.1/animate-elem-34-t.svg
svg/W3C-SVG-1.1/animate-elem-40-t.svg
svg/W3C-SVG-1.1/animate-elem-44-t.svg
svg/W3C-SVG-1.1/animate-elem-52-t.svg
svg/W3C-SVG-1.1/animate-elem-80-t.svg
svg/W3C-SVG-1.1/animate-elem-81-t.svg
svg/W3C-SVG-1.1/animate-elem-82-t.svg
svg/W3C-SVG-1.1/animate-elem-83-t.svg
svg/W3C-SVG-1.1/coords-units-01-b.svg
svg/W3C-SVG-1.1/coords-units-02-b.svg
svg/W3C-SVG-1.1/coords-units-03-b.svg
svg/W3C-SVG-1.1/coords-viewattr-03-b.svg
svg/W3C-SVG-1.1/filters-color-01-b.svg
svg/W3C-SVG-1.1/filters-composite-02-b.svg
svg/W3C-SVG-1.1/filters-comptran-01-b.svg
svg/W3C-SVG-1.1/filters-diffuse-01-f.svg
svg/W3C-SVG-1.1/filters-light-01-f.svg
svg/W3C-SVG-1.1/filters-specular-01-f.svg
svg/W3C-SVG-1.1/filters-turb-01-f.svg
svg/W3C-SVG-1.1/fonts-kern-01-t.svg
svg/W3C-SVG-1.1/interact-events-01-b.svg
svg/W3C-SVG-1.1/interact-order-01-b.svg
svg/W3C-SVG-1.1/interact-order-02-b.svg
svg/W3C-SVG-1.1/interact-order-03-b.svg
svg/W3C-SVG-1.1/linking-uri-01-b.svg
svg/W3C-SVG-1.1/linking-uri-02-b.svg
svg/W3C-SVG-1.1/masking-opacity-01-b.svg
svg/W3C-SVG-1.1/masking-path-01-b.svg
svg/W3C-SVG-1.1/masking-path-02-b.svg
svg/W3C-SVG-1.1/masking-path-03-b.svg
svg/W3C-SVG-1.1/painting-fill-01-t.svg
svg/W3C-SVG-1.1/painting-fill-02-t.svg
svg/W3C-SVG-1.1/painting-fill-03-t.svg
svg/W3C-SVG-1.1/painting-stroke-01-t.svg
svg/W3C-SVG-1.1/painting-stroke-02-t.svg
svg/W3C-SVG-1.1/painting-stroke-03-t.svg
svg/W3C-SVG-1.1/painting-stroke-04-t.svg
svg/W3C-SVG-1.1/paths-data-10-t.svg
svg/W3C-SVG-1.1/pservers-grad-04-b.svg
svg/W3C-SVG-1.1/pservers-grad-06-b.svg
svg/W3C-SVG-1.1/pservers-grad-07-b.svg
svg/W3C-SVG-1.1/pservers-grad-08-b.svg
svg/W3C-SVG-1.1/pservers-grad-09-b.svg
svg/W3C-SVG-1.1/pservers-grad-12-b.svg
svg/W3C-SVG-1.1/pservers-grad-14-b.svg
svg/W3C-SVG-1.1/script-handle-02-b.svg
svg/W3C-SVG-1.1/script-handle-03-b.svg
svg/W3C-SVG-1.1/script-handle-04-b.svg
svg/W3C-SVG-1.1/struct-dom-03-b.svg
svg/W3C-SVG-1.1/struct-dom-04-b.svg
svg/W3C-SVG-1.1/struct-dom-05-b.svg
svg/W3C-SVG-1.1/struct-group-03-t.svg
svg/W3C-SVG-1.1/struct-use-01-t.svg
svg/W3C-SVG-1.1/styling-css-02-b.svg
svg/W3C-SVG-1.1/text-align-01-b.svg
svg/W3C-SVG-1.1/text-align-02-b.svg
svg/W3C-SVG-1.1/text-align-03-b.svg
svg/W3C-SVG-1.1/text-align-04-b.svg
svg/W3C-SVG-1.1/text-align-05-b.svg
svg/W3C-SVG-1.1/text-align-06-b.svg
svg/W3C-SVG-1.1/text-altglyph-01-b.svg
svg/W3C-SVG-1.1/text-fonts-01-t.svg
svg/W3C-SVG-1.1/text-fonts-02-t.svg
svg/W3C-SVG-1.1/text-intro-05-t.svg
svg/W3C-SVG-1.1/text-path-01-b.svg
svg/W3C-SVG-1.1/text-spacing-01-b.svg
svg/W3C-SVG-1.1/text-text-01-b.svg
svg/W3C-SVG-1.1/text-text-03-b.svg
svg/W3C-SVG-1.1/text-tselect-01-b.svg
svg/W3C-SVG-1.1/text-tselect-02-f.svg
svg/W3C-SVG-1.1/text-tspan-01-b.svg
svg/W3C-SVG-1.1/text-ws-01-t.svg
svg/W3C-SVG-1.1/text-ws-02-t.svg
svg/W3C-SVG-1.1/types-basicDOM-01-b.svg
svg/batik/text/longTextOnPath.svg
svg/batik/text/textProperties.svg
svg/batik/text/textStyles.svg
svg/batik/text/xmlSpace.svg
svg/custom/altglyph.svg
svg/custom/dynamic-svg-document-creation.svg
svg/custom/pointer-events-path.svg
svg/custom/scrolling-embedded-svg-file-image-repaint-problem.html
svg/custom/text-whitespace-handling.svg
svg/text/kerning.svg
svg/text/multichar-glyph.svg
svg/text/text-align-02-b.svg
svg/text/text-fonts-01-t.svg
svg/text/text-path-01-b.svg
svg/text/text-spacing-01-b.svg
svg/text/text-text-01-b.svg
svg/text/text-text-03-b.svg
svg/text/text-tselect-01-b.svg
svg/text/text-tspan-01-b.svg
svg/text/text-ws-01-t.svg
svg/text/text-ws-02-t.svg
tables/mozilla/core/bloomberg.html
tables/mozilla_expected_failures/bugs/bug2479-5.html

Reference image at wrong location (move to platform/mac):
invalid-percentage-property-expected.png
lang-inheritance-expected.png
lang-inheritance2-expected.png
lang-vs-xml-lang-expected.png
lang-vs-xml-lang-xhtml-expected.png
text-align-expected.png
paste-text-013-expected.png

Reference images needs to be deleted:
5607069-1-expected.png

Reference image not up-to-date:
css2.1/t1202-counter-09-b.html
css2.1/t1202-counters-09-b.html
fast/flexbox/016.html
fast/images/pdf-as-background.html
editing/style/5279521.html
fast/frames/001.html
fast/inline/continuation-outlines-with-layers.html
fast/inline/continuation-outlines.html
fast/parser/broken-comments-vs-parsing-mode.html
fast/text/international/thai-line-breaks.html
fast/text/whitespace/013.html
fast/xsl/document-function.xml
fast/xsl/xslt-relative-path.xml
tables/mozilla/bugs/bug2973.html

SVG not rendering (https://bugs.webkit.org/show_bug.cgi?id=21910):
fast/backgrounds/svg-as-background-1.html
fast/backgrounds/svg-as-background-2.html
fast/backgrounds/svg-as-background-3.html
fast/backgrounds/svg-as-background-5.html
fast/backgrounds/svg-as-background-6.html
fast/backgrounds/svg-as-mask.html
fast/borders/svg-as-border-image-2.html
fast/borders/svg-as-border-image.html
fast/images/svg-as-background.html
fast/images/svg-as-image.html
fast/images/svg-as-relative-image.html
fast/images/svg-as-tiled-background.html

No video images (https://bugs.webkit.org/show_bug.cgi?id=21911):
media/video-aspect-ratio.html
media/video-controls-rendering.html
media/video-display-toggle.html
media/video-layer-crash.html
media/video-transformed.html

Possible Flash failure (https://bugs.webkit.org/show_bug.cgi?id=21912):
fast/dynamic/flash-replacement-test.html

Repaint failure (https://bugs.webkit.org/show_bug.cgi?id=21913):
fast/repaint/body-background-image.html

Excessive repainting (https://bugs.webkit.org/show_bug.cgi?id=21914):
fast/repaint/float-overflow-right.html
fast/repaint/float-overflow.html

Invalid image tiling (https://bugs.webkit.org/show_bug.cgi?id=21915):
fast/backgrounds/size/backgroundSize19.html

Test doesn't repaint entire view so result is corrupted by previous test (https://bugs.webkit.org/show_bug.cgi?id=21916):
tables/mozilla_expected_failures/bugs/bug178855.xml

Comment 3 Pierre-Olivier Latour 2008-10-27 18:37:31 PDT
(In reply to comment #2)
> Reference image at wrong location (move to platform/mac):
> invalid-percentage-property-expected.png
> lang-inheritance-expected.png
> lang-inheritance2-expected.png
> lang-vs-xml-lang-expected.png
> lang-vs-xml-lang-xhtml-expected.png
> text-align-expected.png
> paste-text-013-expected.png

Actually, "paste-text-013-expected.png" is in 2 places, so the one not in platform simply needs to be deleted
Comment 4 Pierre-Olivier Latour 2008-10-28 10:07:38 PDT
Created attachment 24720 [details]
Clean up of reference images

This patch takes care of the first set of issues: ensure all bases are in the right location and delete the ones that aren't needed.
Comment 5 mitz 2008-10-28 10:11:19 PDT
Comment on attachment 24720 [details]
Clean up of reference images

r=me
There's a tab character in the change log but I will fix that when committing the patch.
Comment 6 mitz 2008-10-28 10:17:42 PDT
Comment on attachment 24720 [details]
Clean up of reference images

Clearing the review flag after committing this in r37927.
Comment 7 Pierre-Olivier Latour 2008-10-30 14:37:37 PDT
First set of new image bases was committed here:
http://trac.webkit.org/changeset/38008
Comment 8 Pierre-Olivier Latour 2008-10-30 14:43:41 PDT
(In reply to comment #7)
> First set of new image bases was committed here:
> http://trac.webkit.org/changeset/38008

The re-generated bases come from the same iMac using a release build of WebKit #r38003:
LayoutTests/svg/custom/
LayoutTests/fast/borders/
LayoutTests/fast/css/

Comment 9 Pierre-Olivier Latour 2008-10-31 16:17:45 PDT
Since the bases were generated on 37896, there was one new base checked in independently:
http://trac.webkit.org/changeset/37967/trunk/LayoutTests
Comment 10 Pierre-Olivier Latour 2008-11-03 18:55:10 PST
Important note on tolerances: to avoid failures due to slight font rendering differences or CG dithering on various HW, pixel tests are expected to be run with a small tolerance:
- Leopard: 0.1%
- Tiger: 1.0%
Comment 11 Pierre-Olivier Latour 2008-11-03 19:07:16 PST
Created attachment 24880 [details]
Fixed reference images on Tiger

Fixed reference images on Tiger for:
LayoutTests/svg/custom/
LayoutTests/fast/borders/
LayoutTests/fast/css/
Comment 12 mitz 2008-11-04 10:48:02 PST
Comment on attachment 24880 [details]
Fixed reference images on Tiger

r=me
I am going to tweak the change log a bit when landing this.
Comment 13 mitz 2008-11-04 10:58:39 PST
Comment on attachment 24880 [details]
Fixed reference images on Tiger

Landed as <http://trac.webkit.org/changeset/38103>. Clearing the review flag.
Comment 14 mitz 2008-11-05 01:35:29 PST
More updated expected results landed in <http://trac.webkit.org/changeset/38121> and <http://trac.webkit.org/changeset/38122>, including accessibility/, animations/, css1/, css2.1/, css3/, dom/, editing/, fast/ and svg/.
Comment 15 Pierre-Olivier Latour 2008-11-05 18:21:42 PST
Created attachment 24931 [details]
Other Tiger patch

Fixed more reference images on Tiger
Comment 16 Pierre-Olivier Latour 2008-11-05 18:23:32 PST
Note that once all new bases are checked in and both Tiger and Leopard are running fine, we need to check in "exception" bases for tests that fail on Tiger with a difference > 1.0% but are still valid.
Comment 17 mitz 2008-11-09 22:18:42 PST
Comment on attachment 24931 [details]
Other Tiger patch

+        - part of https://bugs.webkit.org/show_bug.cgi?id=21821
+          Removed pixel results in platform/mac-tiger that were only slightly different
+          (within 1% tolerance) from the platform/mac results.
+          Removed results in platform/mac-tiger where the results in platform/mac were
+          also correct for Tiger.
+

The patch also modifies several files (I think just updating images to match the correct render trees already checked in) and you should mention that in the change log.

It also adds files. I don't understand why:
A         LayoutTests/platform/mac-tiger/fast/text/international/complex-character-based-fallback-expected.txt
A         LayoutTests/platform/mac-tiger/fast/text/international/hindi-spacing-expected.txt
Comment 18 Pierre-Olivier Latour 2008-11-10 19:07:04 PST
(In reply to comment #9)
> Since the bases were generated on 37896, there was one new base checked in
> independently:
> http://trac.webkit.org/changeset/37967/trunk/LayoutTests

http://trac.webkit.org/changeset/38237/trunk/LayoutTests
http://trac.webkit.org/changeset/38273/trunk/LayoutTests
Comment 19 Pierre-Olivier Latour 2008-11-10 19:22:58 PST
Created attachment 25040 [details]
Revised other Tiger patch

Updated patch following discussion with Dan
Comment 20 mitz 2008-11-11 09:33:34 PST
Comment on attachment 25040 [details]
Revised other Tiger patch

rs=me
Comment 21 Pierre-Olivier Latour 2008-11-11 10:19:31 PST
Last round of check in for new bases:
http://trac.webkit.org/changeset/38297/trunk/LayoutTests
Comment 22 mitz 2008-11-11 10:27:45 PST
Comment on attachment 25040 [details]
Revised other Tiger patch

Landed this in r38305.
Comment 23 Pierre-Olivier Latour 2008-11-11 17:26:39 PST
Created attachment 25080 [details]
Updated reference images for Leopard
Comment 24 mitz 2008-11-12 10:34:42 PST
Comment on attachment 25080 [details]
Updated reference images for Leopard

r=me!
Comment 25 mitz 2008-11-12 10:35:53 PST
Comment on attachment 25080 [details]
Updated reference images for Leopard

Landed in r38335
Comment 26 Pierre-Olivier Latour 2008-11-12 17:38:01 PST
Created attachment 25110 [details]
Updated reference images for Tiger
Comment 27 mitz 2008-11-12 17:40:41 PST
Comment on attachment 25110 [details]
Updated reference images for Tiger

r=me
Comment 28 mitz 2008-11-12 17:41:41 PST
Comment on attachment 25110 [details]
Updated reference images for Tiger

Landed in r38354.
Comment 29 Pierre-Olivier Latour 2008-11-12 17:50:47 PST
We're finally done here!