Bug 232523

Summary: Expose fuzzy match data in layout test results
Product: WebKit Reporter: Simon Fraser (smfr) <simon.fraser>
Component: Tools / TestsAssignee: Simon Fraser (smfr) <simon.fraser>
Status: RESOLVED FIXED    
Severity: Normal CC: ews-watchlist, glenn, jbedard, simon.fraser, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 149828    
Attachments:
Description Flags
Patch
none
Screenshot of the UI
none
Patch
none
Screenshot of the UI
none
For EWS
none
Patch none

Description Simon Fraser (smfr) 2021-10-29 21:49:57 PDT
Expose fuzzy match data in layout test results
Comment 1 Simon Fraser (smfr) 2021-10-29 21:53:17 PDT Comment hidden (obsolete)
Comment 2 Simon Fraser (smfr) 2021-10-29 21:53:47 PDT Comment hidden (obsolete)
Comment 3 Simon Fraser (smfr) 2021-10-29 22:01:58 PDT
Created attachment 442901 [details]
Patch
Comment 4 Simon Fraser (smfr) 2021-10-29 22:02:24 PDT
Created attachment 442902 [details]
Screenshot of the UI
Comment 5 Jonathan Bedard 2021-11-01 08:25:21 PDT
Comment on attachment 442901 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=442901&action=review

The webkitpy failure (on Python 2) appears to be genuine (webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_pixel_test_directories is the test)

> Tools/ChangeLog:9
> +        in the form "image_difference": { "max_difference": 13, "total_pixels": 167 }, and

Want to call out that this information is not included in results database uploads...I think that's fine, including it there to be displayed in a reasonable way would require some modification of the service.
Comment 6 Simon Fraser (smfr) 2021-11-02 21:28:46 PDT
Created attachment 443170 [details]
For EWS
Comment 7 Simon Fraser (smfr) 2021-11-02 21:55:17 PDT
Created attachment 443173 [details]
Patch
Comment 8 Simon Fraser (smfr) 2021-11-02 21:58:23 PDT
(In reply to Jonathan Bedard from comment #5)
> Comment on attachment 442901 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=442901&action=review
> 
> The webkitpy failure (on Python 2) appears to be genuine
> (webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.
> test_pixel_test_directories is the test)
> 
> > Tools/ChangeLog:9
> > +        in the form "image_difference": { "max_difference": 13, "total_pixels": 167 }, and
> 
> Want to call out that this information is not included in results database
> uploads...I think that's fine, including it there to be displayed in a
> reasonable way would require some modification of the service.

These failures turned out to be about JSON property serialization order (the test was relying on ordering).
Comment 9 Simon Fraser (smfr) 2021-11-02 22:14:01 PDT
(In reply to Jonathan Bedard from comment #5)
> Comment on attachment 442901 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=442901&action=review
> 
> The webkitpy failure (on Python 2) appears to be genuine
> (webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.
> test_pixel_test_directories is the test)
> 
> > Tools/ChangeLog:9
> > +        in the form "image_difference": { "max_difference": 13, "total_pixels": 167 }, and
> 
> Want to call out that this information is not included in results database
> uploads.

This data is in the json, so would that not be included in database uploads? I don't know how that works.
Comment 10 Jonathan Bedard 2021-11-03 09:16:32 PDT
(In reply to Simon Fraser (smfr) from comment #9)
> (In reply to Jonathan Bedard from comment #5)
> > Comment on attachment 442901 [details]
> > Patch
> > 
> > View in context:
> > https://bugs.webkit.org/attachment.cgi?id=442901&action=review
> > 
> > The webkitpy failure (on Python 2) appears to be genuine
> > (webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.
> > test_pixel_test_directories is the test)
> > 
> > > Tools/ChangeLog:9
> > > +        in the form "image_difference": { "max_difference": 13, "total_pixels": 167 }, and
> > 
> > Want to call out that this information is not included in results database
> > uploads.
> 
> This data is in the json, so would that not be included in database uploads?
> I don't know how that works.

Right! Just wanted to call that out so anyone looking at this in the future doesn't have to guess.
Comment 11 EWS 2021-11-03 10:05:57 PDT
Committed r285213 (243838@main): <https://commits.webkit.org/243838@main>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 443173 [details].
Comment 12 Radar WebKit Bug Importer 2021-11-03 10:06:31 PDT
<rdar://problem/84979414>