Bug 209989 – [ macOS debug ] REGRESSION (r259463): http/tests/media/clearkey/collect-webkit-media-session.html is failing

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: Media

[Jacob Uphoff]

http/tests/media/clearkey/collect-webkit-media-session.html

This test is failing on Mojave debug since around commit r259463

History:

https://results.webkit.org/?suite=layout-tests&test=http%2Ftests%2Fmedia%2Fclearkey%2Fcollect-webkit-media-session.html&style=debug

Diff:

--- /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/http/tests/media/clearkey/collect-webkit-media-session-expected.txt
+++ /Volumes/Data/slave/mojave-debug-tests-wk2/build/layout-test-results/http/tests/media/clearkey/collect-webkit-media-session-actual.txt
@@ -1,4 +1,4 @@
 
 
-PASS Ensure that the frame's document get collected after being stopped while doing some webkit media session calls 
+FAIL Ensure that the frame's document get collected after being stopped while doing some webkit media session calls promise_test: Unhandled rejection with value: "Test failed"

[Radar WebKit Bug Importer]

<rdar://problem/61277230>

[Radar WebKit Bug Importer]

<rdar://problem/61277243>

[Jacob Uphoff]

Looks like http/tests/media/media-stream/collect-media-devices.https.html is also failing from the same commit.

[Jason Lawrence]

I can reproduce this issue on r259463, but I cannot reproduce this on r259460.
This is the command that I used:
run-webkit-tests --force -f --iterations  90 --exit-after-n-failures 3 -g --debug http/tests/media/clearkey/collect-webkit-media-session.html

r259460
[4/90] http/tests/media/clearkey/collect-webkit-media-session.html failed unexpectedly (text diff)
[5/90] http/tests/media/clearkey/collect-webkit-media-session.html failed unexpectedly (text diff)
[6/90] http/tests/media/clearkey/collect-webkit-media-session.html failed unexpectedly (text diff)
Exiting early after 3 failures. 3 tests run.


0 tests ran as expected, 3 didn't (87 didn't run):

r259460:
All 90 tests ran as expected.

[Aakash Jain]

This is slowing down mac-wk1-debug ews queue, and also causing false positives.

e.g.:
https://ews-build.webkit.org/#/builders/32/builds/6715
https://ews-build.webkit.org/#/builders/32/builds/6714
https://ews-build.webkit.org/#/builders/32/builds/6708

We should consider updating the test expectations if we can't get the root-caused fixed soon.

[Alexey Proskuryakov]

> I can reproduce this issue on r259463, but I cannot reproduce this on r259460.

Since the other changes in between are for IndexedDB, seems almost certain that it's r259463.

[Yusuke Suzuki]

This is testing "something gets collected by GC". But the nature of conservative GC makes is super flaky.
Any memory layout / register allocation / JIT change can easily break this assumption.

The test is not always failing. This implies that we are observing that document is collected in most cases. But sometimes, it is not.
This sounds super likely that conservative GC finds it and keeps it held, and this behavior change is just introduced because the change in JSC causes memory layout / register allocation / JIT changes, which allows conservative GC to find it sometimes.

The most possible fix would be just adjusting the test to make it more stable.
I'll look it later.

[Jacob Uphoff]

Set expectations here: https://trac.webkit.org/changeset/259570/webkit

[Jason Lawrence]

*** Bug 210073 has been marked as a duplicate of this bug. ***

[Ryan Haddad]

Removed flaky expectations in r261329 since these test are consistently passing now.