Bug 193752 – REGRESSION: [ MacOS iOS Release ] Layout Test http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource.html is flaky

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: Tools / Tests

[Truitt Savell]

The following layout test is flaky on MacOS and iOS Release

http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource.html

Probable cause:

Test became flaky very recently around r240360. It suddenly failed across several platforms. Attempted repro by just iterating the test 5000 times but this caused no failures. 

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fcap-cache-max-age-for-prevalent-resource.html

Diff:
--- /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource-expected.txt
+++ /Volumes/Data/slave/highsierra-release-tests-wk2/build/layout-test-results/http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource-actual.txt
@@ -4,11 +4,12 @@
 
 
 PASS loadedUrl is "http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=1234"
-PASS loadedUrl is "http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=4321"
-PASS loadedUrl is "http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value="
-PASS loadedUrl is "http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value="
-PASS loadedUrl is "http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value="
+FAIL loadedUrl should be http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=4321. Was http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=1234.
+FAIL loadedUrl should be http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=. Was http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=1234.
+FAIL loadedUrl should be http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=. Was http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=1234.
+FAIL loadedUrl should be http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=. Was http://localhost:8000/resourceLoadStatistics/resources/echo-query.php?value=1234.
 PASS successfullyParsed is true
+Some tests failed.
 
 TEST COMPLETE

[Truitt Savell]

Created attachment 359981 [details]
test list

this test list run with one child process was able to reproduce the failure local

[Truitt Savell]

used command:
run-webkit-tests --root testbuild-240366 --test-list test1 --child-processes 1

This reproduced the failure on the first attempt. I have not reproduced repeatedly yet. 

This indicates that this failure may have been caused by a test earlier in the run.

[Truitt Savell]

test is flakey when reproducing as well. attempting to bisect the failure

[Truitt Savell]

This tests may also be related, began failing around the same time: 

http/tests/resourceLoadStatistics/capped-lifetime-for-cookie-set-in-js.html

history:
https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fcapped-lifetime-for-cookie-set-in-js.html

[Radar WebKit Bug Importer]

<rdar://problem/47553061>

[Truitt Savell]

I believe this test http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource.html became flakey with r240360. with my repro steps above I bisected down to this revision. I am able to get a failure on r240360 but have yet to get a failure on 240359 after about 20 attempts locally.

[Brent Fulgham]

(In reply to Truitt Savell from comment #6)
> I believe this test
> http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource.
> html became flakey with r240360. with my repro steps above I bisected down
> to this revision. I am able to get a failure on r240360 but have yet to get
> a failure on 240359 after about 20 attempts locally.

You can’t skip the test (referencing this bug) if you want. I have a few more steps in the refactoring aimed at resolving these issues.

[Brent Fulgham]

(In reply to Brent Fulgham from comment #7)
> (In reply to Truitt Savell from comment #6)
> > I believe this test
> > http/tests/resourceLoadStatistics/cap-cache-max-age-for-prevalent-resource.
> > html became flakey with r240360. with my repro steps above I bisected down
> > to this revision. I am able to get a failure on r240360 but have yet to get
> > a failure on 240359 after about 20 attempts locally.
> 
> You can’t skip the test (referencing this bug) if you want. I have a few
> more steps in the refactoring aimed at resolving these issues.

By can't I meant CAN.

Also: One of the causes of flaky timeouts should be resolved in r240509.