Bug 202852 – http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction-database.html is a flaky failure

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Kate Cheney]

The following layout test is a flaky timeout on macOS WK2 and iOS wk2:

http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction-database.html 

Reproducible with:

run-webkit-tests http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction-database.html --iterations 2

Flakiness Dashboard:

https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fswitch-session-on-navigation-to-prevalent-with-interaction-database.html%20

Diff:
--- /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction-expected.txt
+++ /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-with-interaction-actual.txt
@@ -5,8 +5,9 @@
 
 PASS Should have and has the session cookie.
 PASS Should have and has the persistent cookie.
-PASS Origin has isolated session.
+FAIL Origin has no isolated session.
 PASS successfullyParsed is true
+Some tests failed.
 
 TEST COMPLETE
 

Probable cause:

The failures are fixed locally when I clear the NetworkLoad cache and all pending write operations in between the tests. I think this is because clearIsolatedSessions() is called between tests, but a new isolated session is only added with a call to startNetworkLoad() in the NetworkResourceLoader, which doesn't happen if the resource is in the cache. 

Before posting a fix, my questions are:

1. Should the clear() function in the cache also be clearing pending write operations? (I had to do it manually)
2. Should the cache be cleared between tests automatically?
3. The entry in the cache was being stored with a "#step4" on the end of it, is this expected behavior?

[Radar WebKit Bug Importer]

<rdar://problem/56195888>

[Alexey Proskuryakov]

> 2. Should the cache be cleared between tests automatically?

That may have a substantial performance cost (which is worth verifying).

Another consequence is loss of incidental testing. In other words, if we clear all caches, then tests will all be running with 2-3 elements in the cache, which is not very useful. I'm not sure what the right balance between test stability and incidental testing is in this particular case.

[Kate Cheney]

(In reply to Alexey Proskuryakov from comment #2)
> > 2. Should the cache be cleared between tests automatically?
> 
> That may have a substantial performance cost (which is worth verifying).
> 
> Another consequence is loss of incidental testing. In other words, if we
> clear all caches, then tests will all be running with 2-3 elements in the
> cache, which is not very useful. I'm not sure what the right balance between
> test stability and incidental testing is in this particular case.

I don't know if this is the best solution, but I can add in a testRunner function that clears the cache. Then this test, and any others uncovered in the future with the same problem, can call the function before they run to ensure the cache is cleared if the test relies on it.

[Chris Dumez]

(In reply to Katherine_cheney from comment #0)
> The following layout test is a flaky timeout on macOS WK2 and iOS wk2:
> 
> http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> with-interaction-database.html 
> 
> Reproducible with:
> 
> run-webkit-tests
> http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> with-interaction-database.html --iterations 2
> 
> Flakiness Dashboard:
> 
> https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.
> html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fswitch-
> session-on-navigation-to-prevalent-with-interaction-database.html%20
> 
> Diff:
> ---
> /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/
> http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> with-interaction-expected.txt
> +++
> /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/
> http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> with-interaction-actual.txt
> @@ -5,8 +5,9 @@
>  
>  PASS Should have and has the session cookie.
>  PASS Should have and has the persistent cookie.
> -PASS Origin has isolated session.
> +FAIL Origin has no isolated session.
>  PASS successfullyParsed is true
> +Some tests failed.
>  
>  TEST COMPLETE
>  
> 
> Probable cause:
> 
> The failures are fixed locally when I clear the NetworkLoad cache and all
> pending write operations in between the tests. I think this is because
> clearIsolatedSessions() is called between tests, but a new isolated session
> is only added with a call to startNetworkLoad() in the
> NetworkResourceLoader, which doesn't happen if the resource is in the cache. 

So how about you make sure this particular resource does not enter the disk cache by serving the "Cache-Control: no-store" HTTP header. It is pretty common in our tests.

[Chris Dumez]

(In reply to Chris Dumez from comment #4)
> (In reply to Katherine_cheney from comment #0)
> > The following layout test is a flaky timeout on macOS WK2 and iOS wk2:
> > 
> > http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> > with-interaction-database.html 
> > 
> > Reproducible with:
> > 
> > run-webkit-tests
> > http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> > with-interaction-database.html --iterations 2
> > 
> > Flakiness Dashboard:
> > 
> > https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.
> > html#showAllRuns=true&tests=http%2Ftests%2FresourceLoadStatistics%2Fswitch-
> > session-on-navigation-to-prevalent-with-interaction-database.html%20
> > 
> > Diff:
> > ---
> > /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/
> > http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> > with-interaction-expected.txt
> > +++
> > /Volumes/Data/slave/highsierra-debug-tests-wk2/build/layout-test-results/
> > http/tests/resourceLoadStatistics/switch-session-on-navigation-to-prevalent-
> > with-interaction-actual.txt
> > @@ -5,8 +5,9 @@
> >  
> >  PASS Should have and has the session cookie.
> >  PASS Should have and has the persistent cookie.
> > -PASS Origin has isolated session.
> > +FAIL Origin has no isolated session.
> >  PASS successfullyParsed is true
> > +Some tests failed.
> >  
> >  TEST COMPLETE
> >  
> > 
> > Probable cause:
> > 
> > The failures are fixed locally when I clear the NetworkLoad cache and all
> > pending write operations in between the tests. I think this is because
> > clearIsolatedSessions() is called between tests, but a new isolated session
> > is only added with a call to startNetworkLoad() in the
> > NetworkResourceLoader, which doesn't happen if the resource is in the cache. 
> 
> So how about you make sure this particular resource does not enter the disk
> cache by serving the "Cache-Control: no-store" HTTP header. It is pretty
> common in our tests.

See for e.g. LayoutTests/http/tests/misc/resources/random-no-store.php

[Kate Cheney]

Created attachment 381251 [details]
Patch

[WebKit Commit Bot]

Comment on attachment 381251 [details]
Patch

Clearing flags on attachment: 381251

Committed r251402: <https://trac.webkit.org/changeset/251402>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.