Bug 85283 - Reftest mismatches are providing undefined diff percent in results.html
Summary: Reftest mismatches are providing undefined diff percent in results.html
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Zan Dobersek
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-05-01 07:50 PDT by Zan Dobersek
Modified: 2012-05-02 14:06 PDT (History)
6 users (show)

See Also:


Attachments
Patch (1.59 KB, patch)
2012-05-01 07:56 PDT, Zan Dobersek
no flags Details | Formatted Diff | Diff
Archive of layout-test-results from ec2-cr-linux-02 (6.27 MB, application/zip)
2012-05-01 08:25 PDT, WebKit Review Bot
no flags Details
Patch (1.58 KB, patch)
2012-05-01 09:14 PDT, Zan Dobersek
no flags Details | Formatted Diff | Diff
Patch (5.97 KB, patch)
2012-05-01 22:59 PDT, Zan Dobersek
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Zan Dobersek 2012-05-01 07:50:57 PDT
When a reftest mismatch occurs in testing, results.html entry holds a link to the diffs.png that contains a text line 'diff (undefined%)'. It seems reftest mismatch failures are not given a diff percent attribute in JSON data as expected by the results.html file.
Comment 1 Zan Dobersek 2012-05-01 07:56:15 PDT
Created attachment 139627 [details]
Patch
Comment 2 Zan Dobersek 2012-05-01 07:59:09 PDT
Adding authors and reviewers of the last few changes to this file to get this reviewed. Thanks in advance!
Comment 3 WebKit Review Bot 2012-05-01 08:25:32 PDT
Comment on attachment 139627 [details]
Patch

Attachment 139627 [details] did not pass chromium-ews (chromium-xvfb):
Output: http://queues.webkit.org/results/12588578

New failing tests:
fast/harness/results.html
Comment 4 WebKit Review Bot 2012-05-01 08:25:39 PDT
Created attachment 139630 [details]
Archive of layout-test-results from ec2-cr-linux-02

The attached test failures were seen while running run-webkit-tests on the chromium-ews.
Bot: ec2-cr-linux-02  Port: <class 'webkitpy.common.config.ports.ChromiumXVFBPort'>  Platform: Linux-2.6.35-28-virtual-x86_64-with-Ubuntu-10.10-maverick
Comment 5 Zan Dobersek 2012-05-01 09:14:09 PDT
Created attachment 139633 [details]
Patch

Sigh, forgot the percent sign.
Comment 6 Ojan Vafai 2012-05-01 09:30:52 PDT
Comment on attachment 139633 [details]
Patch

This seems like the wrong layer to fix this in. We should fix ImageDiff to include the diff percentage for reftests as well.
Comment 7 Zan Dobersek 2012-05-01 22:59:49 PDT
Created attachment 139752 [details]
Patch
Comment 8 Ojan Vafai 2012-05-02 09:51:49 PDT
Comment on attachment 139752 [details]
Patch

Looks great. Thanks!
Comment 9 Zan Dobersek 2012-05-02 11:00:33 PDT
Comment on attachment 139752 [details]
Patch

Clearing flags on attachment: 139752

Committed r115853: <http://trac.webkit.org/changeset/115853>
Comment 10 Zan Dobersek 2012-05-02 11:00:59 PDT
All reviewed patches have been landed.  Closing bug.
Comment 11 Dirk Pranke 2012-05-02 14:06:31 PDT
Comment on attachment 139752 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=139752&action=review

> Tools/Scripts/webkitpy/layout_tests/controllers/test_result_writer.py:71
> +                diff_result = port.diff_image(driver_output.image, expected_driver_output.image, tolerance=0)

Minor quibble: I would've probably used a destructuring assignment here, like "diff_result, diff_percent = port.diff_image(...)"