Bug 178782 - Flakiness dashboard: Make GTK and WPE builds inherit expectations from WK2
Summary: Flakiness dashboard: Make GTK and WPE builds inherit expectations from WK2
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2017-10-25 03:54 PDT by Alicia Boya García
Modified: 2017-12-13 08:57 PST (History)
5 users (show)

See Also:


Attachments
Patch (2.88 KB, patch)
2017-12-12 11:34 PST, Alicia Boya García
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alicia Boya García 2017-10-25 03:54:14 PDT
The following layout test is having an unexpected result on GTK according to the flakiness dashboard:

http/tests/loading/resourceLoadStatistics/grandfathering.html

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#tests=http%2Ftests%2Floading%2FresourceLoadStatistics%2Fgrandfathering.html

The interesting bit is that it should not because it's marked as expected [ Pass Failure Timeout ] in WK2 expectations file.

platform/wk2/TestExpectations
689:webkit.org/b/172452 http/tests/loading/resourceLoadStatistics/grandfathering.html [ Pass Failure Timeout ]

And GTK+ tests are run with those expectations in mind:

> run-webkit-tests --gtk http/tests/loading/resourceLoadStatistics/grandfathering.html 
Using port 'gtk-wk2'
Test configuration: <, x86, release>
Placing test results in /webkit/WebKitBuild/Release/layout-test-results
Baseline search path: platform/gtk -> platform/wk2 -> generic
Using Release build
Pixel tests disabled
Regular timeout: 15000, slow test timeout: 75000
Command line: /webkit/Tools/jhbuild/jhbuild-wrapper --gtk run /webkit/WebKitBuild/Release/bin/WebKitTestRunner -

Found 1 test; running 1, skipping 0.
                        
Running 1 test

Running 1 WebKitTestRunner.     

The test ran as expected.                                          

Stopping HTTP server ...
Comment 1 Alicia Boya García 2017-12-12 10:45:10 PST
Turns out the flakiness dashboard is assuming GTK ports are WK1, therefore WK2 expectations are not taken into account.

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showWrongExpectations=true&builder=GTK%20Linux%2064-bit%20Release%20(Tests)

Look at the top: Expectations for GTK_LINUX_WK1-RELEASE

This is the result of a logic in determineBuilderPlatform() that determines whether a platform is WK2 or WK1 based on its name: If it does not contain WK2, it assumes it's WK1.
Comment 2 Alicia Boya García 2017-12-12 11:34:56 PST
Created attachment 329132 [details]
Patch
Comment 3 WebKit Commit Bot 2017-12-12 16:11:50 PST
Comment on attachment 329132 [details]
Patch

Clearing flags on attachment: 329132

Committed r225819: <https://trac.webkit.org/changeset/225819>
Comment 4 WebKit Commit Bot 2017-12-12 16:11:51 PST
All reviewed patches have been landed.  Closing bug.
Comment 5 Radar WebKit Bug Importer 2017-12-12 16:12:26 PST
<rdar://problem/36008110>
Comment 6 Alicia Boya García 2017-12-13 03:11:37 PST
The hosted version continues to run an old revision:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#builder=GTK%20Linux%2064-bit%20Release%20(Tests)

Note it still says WK1. Who should I ping for fixing that?
Comment 7 Michael Catanzaro 2017-12-13 08:57:16 PST
Try asking admin@webkit.org