[ Mac WK2 iOS Sim] Layout Test http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-w* are flaky timeouts

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Shawn Roberts]

The following layout tests are flaky on Mac WK2 & iOS Sim

http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-fragment-from-prevalent-resource.html 
http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-and-fragment-from-prevalent-resource.html 
http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-from-prevalent-resource.html 
http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-without-link-decoration-from-prevalent-resource.html

Probable cause:

It appears 4 of the 5 tests introduced in https://trac.webkit.org/changeset/242288/webkit are flaky failures on the Bots, and most are reproducible locally. Tests were modified in https://trac.webkit.org/changeset/243181/webkit , but flakiness occurred before that change. Though dashboard shows a lot of flakiness on iOS Sim Debug bots, it doesn't time out that often locally for me.

I can reproduce the Mac WK2 Release timeouts locally with : On average only 1 time out in 500 iterations. 

run-webkit-tests http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-without-link-decoration-from-prevalent-resource.html --iterations 500 --child-process 1

I can reproduce the iOS Sim Debug failures using : On average about 3 failures out of 500 iterations.

run-webkit-tests http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-and-fragment-from-prevalent-resource.html --iterations 500 --child-process 1 --debug --ios-simulator

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-decoration-same-site.html%20%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-fragment-from-prevalent-resource.html%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-query-and-fragment-from-prevalent-resource.html%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-with-link-query-from-prevalent-resource.html%20http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js-without-link-decoration-from-prevalent-resource.html

[Radar WebKit Bug Importer]

<rdar://problem/49288612>

[Shawn Roberts]

Marked flaky in https://trac.webkit.org/changeset/243517/webkit while waiting for a fix.

[Kate Cheney]

Created attachment 381974 [details]
Patch

[Alexey Proskuryakov]

Comment on attachment 381974 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=381974&action=review

> LayoutTests/platform/ios-simulator-wk2/TestExpectations:-49
> -webkit.org/b/196269 http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js-with-link-query-from-prevalent-resource.html [ Pass Timeout ]

FWIW, this test had a timeout yesterday (on Mac, so not affected by this patch, and appears to be very rare).

https://build.webkit.org/results/Apple%20High%20Sierra%20Debug%20WK2%20(Tests)/r251583%20(10109)/results.html

[Alexey Proskuryakov]

However, I see that the bug appears to track both macOS and iOS.

[Kate Cheney]

From the comments in the radar, it seems like cookie-synching was causing flakiness in the past for these tests. Each of these tests has a corresponding test for the resource load statistics database store which theoretically would rely on the exact same cookie synching, and those have been up for a few weeks w/o appearing on the flakiness dashboard. This (plus many failed attempts to reproduce) led me to think the flakiness has been fixed. I will keep an eye on the dashboard.

[EWS]

Comment on attachment 381974 [details]
Patch

Rejecting attachment 381974 [details] from commit-queue.

[e-mail] does not have committer permissions according to https://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/contributors.json.

- If you do not have committer rights please read http://webkit.org/coding/contributing.html for instructions on how to use bugzilla flags.

- If you have committer rights please correct the error in Tools/Scripts/webkitpy/common/config/contributors.json by adding yourself to the file (no review needed).  The commit-queue restarts itself every 2 hours.  After restart the commit-queue will correctly respect your committer rights.

[WebKit Commit Bot]

Comment on attachment 381974 [details]
Patch

Rejecting attachment 381974 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.webkit.org', '--bot-id=webkit-cq-03', 'apply-attachment', '--no-update', '--non-interactive', 381974, '--port=mac']" exit_code: 2 cwd: /Volumes/Data/EWS/WebKit

Logging in as [e-mail]...
Fetching: https://bugs.webkit.org/attachment.cgi?id=381974&action=edit
Fetching: https://bugs.webkit.org/show_bug.cgi?id=196269&ctype=xml&excludefield=attachmentdata
Processing 1 patch from 1 bug.
Processing patch 381974 from bug 196269.
Fetching: https://bugs.webkit.org/attachment.cgi?id=381974
Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Alexey Proskuryakov']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Parsed 3 diffs from patch file(s).
patching file LayoutTests/ChangeLog
Hunk #1 succeeded at 1 with fuzz 3.
patching file LayoutTests/platform/ios-simulator-wk2/TestExpectations
Hunk #1 FAILED at 44.
1 out of 1 hunk FAILED -- saving rejects to file LayoutTests/platform/ios-simulator-wk2/TestExpectations.rej
patching file LayoutTests/platform/ios-wk2/TestExpectations

Failed to run "[u'/Volumes/Data/EWS/WebKit/Tools/Scripts/svn-apply', '--force', '--reviewer', u'Alexey Proskuryakov']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

Full output: https://webkit-queues.webkit.org/results/13186556

[Kate Cheney]

Created attachment 382137 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 382137 [details]
Patch

Clearing flags on attachment: 382137

Committed r251692: <https://trac.webkit.org/changeset/251692>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.