Bug 196269 - [ Mac WK2 iOS Sim] Layout Test http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-w* are flaky timeouts
Summary: [ Mac WK2 iOS Sim] Layout Test http/tests/resourceLoadStatistics/capped-lifet...
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: katherine_cheney
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2019-03-26 13:29 PDT by Shawn Roberts
Modified: 2019-10-28 19:25 PDT (History)
9 users (show)

See Also:


Attachments
Patch (3.61 KB, patch)
2019-10-25 16:01 PDT, katherine_cheney
no flags Details | Formatted Diff | Diff
Patch (3.54 KB, patch)
2019-10-28 16:46 PDT, katherine_cheney
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Shawn Roberts 2019-03-26 13:29:07 PDT
The following layout tests are flaky on Mac WK2 & iOS Sim

http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-fragment-from-prevalent-resource.html 
http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-and-fragment-from-prevalent-resource.html 
http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-from-prevalent-resource.html 
http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-without-link-decoration-from-prevalent-resource.html

Probable cause:

It appears 4 of the 5 tests introduced in https://trac.webkit.org/changeset/242288/webkit are flaky failures on the Bots, and most are reproducible locally. Tests were modified in https://trac.webkit.org/changeset/243181/webkit , but flakiness occurred before that change. Though dashboard shows a lot of flakiness on iOS Sim Debug bots, it doesn't time out that often locally for me.

I can reproduce the Mac WK2 Release timeouts locally with : On average only 1 time out in 500 iterations. 

run-webkit-tests http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-without-link-decoration-from-prevalent-resource.html --iterations 500 --child-process 1

I can reproduce the iOS Sim Debug failures using : On average about 3 failures out of 500 iterations.

run-webkit-tests http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-and-fragment-from-prevalent-resource.html --iterations 500 --child-process 1 --debug --ios-simulator

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-decoration-same-site.html%20%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-fragment-from-prevalent-resource.html%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-query-and-fragment-from-prevalent-resource.html%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-query-from-prevalent-resource.html%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-without-link-decoration-from-prevalent-resource.html
Comment 1 Radar WebKit Bug Importer 2019-03-26 13:31:33 PDT
<rdar://problem/49288612>
Comment 2 Shawn Roberts 2019-03-26 13:43:54 PDT
Marked flaky in https://trac.webkit.org/changeset/243517/webkit while waiting for a fix.
Comment 3 katherine_cheney 2019-10-25 16:01:26 PDT
Created attachment 381974 [details]
Patch
Comment 4 Alexey Proskuryakov 2019-10-25 16:06:52 PDT
Comment on attachment 381974 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=381974&action=review

> LayoutTests/platform/ios-simulator-wk2/TestExpectations:-49
> -webkit.org/b/196269 http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-from-prevalent-resource.html [ Pass Timeout ]

FWIW, this test had a timeout yesterday (on Mac, so not affected by this patch, and appears to be very rare).

https://build.webkit.org/results/Apple%20High%20Sierra%20Debug%20WK2%20(Tests)/r251583%20(10109)/results.html
Comment 5 Alexey Proskuryakov 2019-10-25 16:07:40 PDT
However, I see that the bug appears to track both macOS and iOS.
Comment 6 katherine_cheney 2019-10-25 16:13:31 PDT
From the comments in the radar, it seems like cookie-synching was causing flakiness in the past for these tests. Each of these tests has a corresponding test for the resource load statistics database store which theoretically would rely on the exact same cookie synching, and those have been up for a few weeks w/o appearing on the flakiness dashboard. This (plus many failed attempts to reproduce) led me to think the flakiness has been fixed. I will keep an eye on the dashboard.
Comment 7 EWS 2019-10-28 15:44:12 PDT
Comment on attachment 381974 [details]
Patch

Rejecting attachment 381974 [details] from commit-queue.

Katherine_cheney@apple.com does not have committer permissions according to https://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/contributors.json.

- If you do not have committer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/contributors.json by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your committer rights.
Comment 8 WebKit Commit Bot 2019-10-28 16:25:30 PDT
Comment on attachment 381974 [details]
Patch

Rejecting attachment 381974 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'apply-attachment', '--no-update', '--non-interactive', 381974, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Logging in as commit-queue@webkit.org...
Fetching: https://bugs.webkit.org/attachment.cgi?id=381974&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=196269&ctype=xml&excludefield=attachmentdata
Processing 1 patch from 1 bug.
Processing patch 381974 from bug 196269.
Fetching: https://bugs.webkit.org/attachment.cgi?id=381974
Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Alexey Proskuryakov']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Parsed 3 diffs from patch file(s).
patching file LayoutTests/ChangeLog
Hunk #1 succeeded at 1 with fuzz 3.
patching file LayoutTests/platform/ios-simulator-wk2/TestExpectations
Hunk #1 FAILED at 44.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/ios-simulator-wk2/TestExpectations.rej
patching file LayoutTests/platform/ios-wk2/TestExpectations

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Alexey Proskuryakov']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: https://webkit-queues.webkit.org/results/13186556
Comment 9 katherine_cheney 2019-10-28 16:46:36 PDT
Created attachment 382137 [details]
Patch
Comment 10 WebKit Commit Bot 2019-10-28 19:25:39 PDT
Comment on attachment 382137 [details]
Patch

Clearing flags on attachment: 382137

Committed r251692: <https://trac.webkit.org/changeset/251692>
Comment 11 WebKit Commit Bot 2019-10-28 19:25:41 PDT
All reviewed patches have been landed.  Closing bug.