Bug 265498 - [update-test-expectations-from-bugzilla] analyze layout test results
Summary: [update-test-expectations-from-bugzilla] analyze layout test results
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-11-28 18:23 PST by Sam Sneddon [:gsnedders]
Modified: 2023-12-16 03:05 PST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sam Sneddon [:gsnedders] 2023-11-28 18:23:58 PST
similar to the analyze-layout-tests-results step (c.f., https://github.com/WebKit/WebKit/blob/5965a5a92e802141a8ec48b1a83c9bb88069f1e1/Tools/CISupport/ews-build/steps.py#L4454), we should not *just* take the initial layout-tests run as the definitive set of results

this should vastly reduce the number of failed EWS runs as a result of running the tool, and avoid changing tests because they are ungardened on main.
Comment 1 Radar WebKit Bug Importer 2023-11-28 18:24:07 PST
<rdar://problem/118909142>
Comment 2 Sam Sneddon [:gsnedders] 2023-12-14 16:55:58 PST
https://github.com/WebKit/WebKit/pull/21838 is laying some of the group work there, and the comment I just posted there explains where I expect to build on top of that PR:

> The end goal is to move to having some class representing a group of runs, consisting of one or more layout tests runs plus zero or more runs without changes. We can then have that class tell us "what are the new (stable) failures" and use that.
Comment 3 Sam Sneddon [:gsnedders] 2023-12-16 02:57:55 PST
this also probably needs to either deal directly with the results database (which requires knowing the base revision), or pull the data from the EWS run, as we often ignore (and even avoid re-runs) test failures based on it: https://ews-build.webkit.org/#/builders/36/builds/27897