Bug 202736 – [GTK][WPE] Several appcache and cache-storage tests are failing or flaky

• Status: NEW
• Component: WPE WebKit

[Miguel Gomez]

All of these tests are now flaky

http/tests/appcache/document-cookie-http-only.php [ Failure Pass ]
http/tests/appcache/history-test.html [ Failure Pass ]
http/tests/appcache/remove-cache.html [ Timeout Pass ]
http/tests/cache-storage/cache-clearing-all.https.html [ Failure Pass ]
http/tests/cache-storage/cache-clearing-origin.https.html [ Failure Pass ]
http/tests/cache-storage/cache-origins.https.html [ Failure Pass ]
http/tests/cache-storage/cache-persistency.https.html [ Failure Pass ]
http/wpt/cache-storage/quota-third-party.https.html [ Failure Timeout Pass ]
http/wpt/cache-storage/cache-quota-add.any.html [ Failure ]

All of them were passing reliably until the changes that came in these 2 revisions
- r249614: Deprecate WKContextGetWebsiteDataStore
- r249619: Remove calls to WKContextGetWebsiteDataStore missed in r249614


Here are the diffs for some of the tests when they fail:

http/tests/appcache/document-cookie-http-only.php [ Failure Pass ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/appcache/document-cookie-http-only-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/appcache/document-cookie-http-only-actual.txt
@@ -1,6 +1,6 @@
 CONSOLE MESSAGE: line 1: ApplicationCache is deprecated. Please use ServiceWorkers instead.
 This tests that HttpOnly cookies set on the main document are used when accessing resources in the manifest.
 This also tests that cookies used by appcache resource loading are scoped properly.
-PASSED: Some cookies should not be visible from JavaScript.
+FAILED: Some cookies should not be visible from JavaScript.
 PASSED: Script Cookie is set to 'script'
 PASSED: Cookie is set to 'bar'


http/tests/appcache/history-test.html [ Failure Pass ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/appcache/history-test-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/appcache/history-test-actual.txt
@@ -1,4 +1,3 @@
-CONSOLE MESSAGE: line 1: ApplicationCache is deprecated. Please use ServiceWorkers instead.
 CONSOLE MESSAGE: line 1: ApplicationCache is deprecated. Please use ServiceWorkers instead.
 Test whether window.history.back() is supported in application cache.
 

http/tests/cache-storage/cache-clearing-all.https.html [ Failure Pass ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-clearing-all.https-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-clearing-all.https-actual.txt
@@ -1,4 +1,4 @@
 
 PASS Cleaning existing caches 
-PASS Clearing all disk cache 
+FAIL Clearing all disk cache assert_equals: keys should be empty expected 0 but got 1
 

http/tests/cache-storage/cache-clearing-origin.https.html [ Failure Pass ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-clearing-origin.https-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-clearing-origin.https-actual.txt
@@ -1,7 +1,7 @@
 
 
 PASS Create a cache storage from localhost and clear it 
-PASS Clearing disk cache of a given origin 
+FAIL Clearing disk cache of a given origin assert_true: hasDOMCache with actual origin expected true got false
 PASS Validating cache representation before clearing 
-PASS Validating cache representation after clearing 
+FAIL Validating cache representation after clearing assert_array_equals: lengths differ, expected 0 got 1
 

http/tests/cache-storage/cache-origins.https.html [ Failure Pass ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-origins.https-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-origins.https-actual.txt
@@ -1,7 +1,7 @@
   
 
 PASS Create a cache storage and look at the representation 
-PASS Verifying that two caches are created on two different client origins 
+FAIL Verifying that two caches are created on two different client origins assert_array_equals: lengths differ, expected 1 got 2
 PASS Verifying that caches are clustered per client origin. 
-PASS Verifying that clearing caches for an origin will clear both top origin and client origin caches. 
+FAIL Verifying that clearing caches for an origin will clear both top origin and client origin caches. assert_array_equals: lengths differ, expected 0 got 2
 

http/tests/cache-storage/cache-persistency.https.html [ Failure Pass ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-persistency.https-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/cache-storage/cache-persistency.https-actual.txt
@@ -1,8 +1,8 @@
 
 
 PASS Cleaning existing caches 
-PASS Clear memory representation but keep disk persistency 
+FAIL Clear memory representation but keep disk persistency assert_true: A cache object should be found expected true got false
 PASS Cleaning added caches from previous test 
-PASS Adding/Removing caches and clearing in memory representation at various times 
+FAIL Adding/Removing caches and clearing in memory representation at various times assert_array_equals: lengths differ, expected 2 got 3
 PASS Cleaning added caches

[Miguel Gomez]

We have more tests failing due to r249614:

http/tests/IndexedDB/storage-limit-2.https.html [ Failure ]
http/tests/IndexedDB/storage-limit.https.html [ Failure ]
storage/indexeddb/storage-limit.html [ Failure ]
storage/indexeddb/modern/handle-user-delete.html [ Timeout ]
storage/indexeddb/modern/new-database-after-user-delete.html [ Timeout ]

and these are the diffs for the non timeout cases:

http/tests/IndexedDB/storage-limit-2.https.html [ Failure ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/IndexedDB/storage-limit-2.https-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/IndexedDB/storage-limit-2.https-actual.txt
@@ -1,4 +1,3 @@
-CONSOLE MESSAGE: Cache API operation failed: Quota exceeded
 This test makes sure that storage of indexedDB and Cache API do not grow unboundedly.
 
 On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
@@ -14,8 +13,9 @@
 store = db.transaction('store', 'readwrite').objectStore('store')
 request = store.add(new Uint8Array(204800), 'key')
 finished idb processing
-Cache API store operation failed: QuotaExceededError: Quota exceeded
+FAIL Cache API store operation succeeded
 PASS successfullyParsed is true
+Some tests failed.
 
 TEST COMPLETE
 


http/tests/IndexedDB/storage-limit.https.html [ Failure ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/IndexedDB/storage-limit.https-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/http/tests/IndexedDB/storage-limit.https-actual.txt
@@ -12,10 +12,9 @@
 db = event.target.result
 store = db.transaction('store', 'readwrite').objectStore('store')
 request = store.add(new Uint8Array(204800), 'key')
-PASS 'error' in request is true
-PASS request.error.code is DOMException.QUOTA_EXCEEDED_ERR
-PASS request.error.name is "QuotaExceededError"
+FAIL Add operation should fail because storage limit is reached, but succeeded.
 PASS successfullyParsed is true
+Some tests failed.
 
 TEST COMPLETE
 


storage/indexeddb/storage-limit.html [ Failure ]
--- /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/storage/indexeddb/storage-limit-expected.txt
+++ /home/buildbot/wpe/wpe-linux-64-release-tests/build/layout-test-results/storage/indexeddb/storage-limit-actual.txt
@@ -27,11 +27,9 @@
 store.add(new Uint8Array(1), 9).onsuccess = ()=> { ++addCount; }
 store.add(new Uint8Array(1), 10).onsuccess = ()=> { ++addCount; }
 request = store.add(new Uint8Array(409601), 0)
-PASS addCount is 10
-PASS 'error' in request is true
-PASS request.error.code is DOMException.QUOTA_EXCEEDED_ERR
-PASS request.error.name is "QuotaExceededError"
+FAIL Add operation should fail because storage limit is reached, but succeeded.
 PASS successfullyParsed is true
+Some tests failed.
 
 TEST COMPLETE

[Alex Christensen]

Are they only flaky on GTK?  I looked on https://webkit-test-results.webkit.org/dashboards/flakiness_dashboard.html and found most of them show no flakiness.

When I've looked into similar issues, it was found that running a test after a certain other test (usually one that calls terminateNetworkProcess) caused the failures to be deterministic.

I think the solution will be to migrate from anything that uses WebProcessPool::websiteDataStore, deprecating API along the way.

[Diego Pino]

There two tests are failing in GTK:

  * http/tests/appcache/remove-cache.html [ Failure Timeout Pass ]
  * http/wpt/cache-storage/quota-third-party.https.html [ Failure Timeout Pass]

The tests started to consistently time out at:

  r249529                       NOERROR
  [r249530-r249619]             UNKNOWN
  r249620                       TIMEOUT TEXT (Expected: PASS)

Likely due to r249614 as well.

[Diego Pino]

*** Bug 206222 has been marked as a duplicate of this bug. ***

[Diego Pino]

A few tests are passing since r259705:

storage/indexeddb/modern/handle-user-delete.html
storage/indexeddb/modern/new-database-after-user-delete.html

Updated test expectations in r260563.

[Diego Pino]

Many of these tests have been consistently passing since r259705. Updated test expectations in r261272. New expectations look like this (both for GTK and WPE):

  webkit.org/b/202736 http/tests/appcache/remove-cache.html [ Failure ]
  webkit.org/b/202736 http/wpt/cache-storage/quota-third-party.https.html [ Slow ]

[Carlos Alberto Lopez Perez]

http/wpt/cache-storage/quota-third-party.https.html is flaky and gives timeouts. On the last 4000 runs on the gtk-release test bot it timed out 1022 times.

Setting slow to it is not enough as it continues to give timeouts.