RESOLVED INVALID301415
WKTR/WebDriver may exit before whole test is run, allowing obviously failing tests to pass. 
https://bugs.webkit.org/show_bug.cgi?id=301415
Summary WKTR/WebDriver may exit before whole test is run, allowing obviously failing ...
rupin
Reported 2025-10-24 01:07:49 PDT
This test should fail because 4 does not equal 3. Yet, it passes: ``` <!doctype html> <script src="/resources/testharness.js"></script> <script src="/resources/testharnessreport.js"></script> <script> async_test(t => { window.onload = () => t.step_timeout(t.step_func_done(async () => { assert_equals(4, 3); }), 0); }, "this test should fail"); </script> ``` Note: if you put a console log after the assert, it won't get printed when using log stream. So it seems like the bug is that the test is exiting too early and not even getting to running the assert.
Attachments
Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2025-10-24 01:07:55 PDT
<rdar://problem/163323262>
rupin
Comment 2 2025-10-24 01:54:18 PDT
One WPT test that is affected by this bug: https://github.com/web-platform-tests/wpt/pull/55632 (this PR re-writes the test to work around this bug until it gets fixed). I've explained the bug in the commit message of this PR: https://github.com/WebKit/WebKit/pull/52946.
Alexey Proskuryakov
Comment 3 2025-10-24 12:10:39 PDT
This is not what's happening, the assert does run. What's happening is that the exception isn't caught by the harness. The test is essentially this: ``` func = async () => { throw 0; } try { func.apply(this); } catch (e) { alert("why do you expect this to be caught?") } ``` I verified that this is not just about us having a stale copy of the harness, it behaves the same when I substitute with ToT WPT harness.
rupin
Comment 4 2025-10-24 12:24:28 PDT
Hello! Thank you for taking a look at this. I don't know too much about the test infrastructure, just noticed this while debugging the Navigation API test. I filed this because this test (the one in the original top comment) passes with WebKit's driver but does indeed fail when run as a WPT with chrome's driver: Running: ./wpt run --channel dev --binary "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" chrome "navigation-api/navigation-methods/custom-test.html" Leads to: /Volumes/Work/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.3.6'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( Running 1 tests in web-platform-tests /Volumes/Work/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.3.6'. See: https://github.com/urllib3/urllib3/issues/3020 warnings.warn( ▶ ERROR [expected OK] /navigation-api/navigation-methods/custom-test.html │ → Unhandled rejection: assert_equals: expected 3 but got 4 │ │ at Test.<anonymous> (http://web-platform.test:8000/navigation-api/navigation-methods/custom-test.html:7:5) │ at Test.step (http://web-platform.test:8000/resources/testharness.js:2869:25) │ at Test.<anonymous> (http://web-platform.test:8000/resources/testharness.js:2944:32) │ at Test.<anonymous> (http://web-platform.test:8000/resources/testharness.js:2990:25) └ at Test.step (http://web-platform.test:8000/resources/testharness.js:2869:25) Ran 1 tests finished in 2.3 seconds. • 0 ran as expected. 0 tests skipped. • 1 tests had errors unexpectedly
rupin
Comment 5 2025-10-24 12:26:51 PDT
Versus on WebKit: Running: ./wpt run --log-mach-verbose --log-mach-level debug --log-mach - --webdriver-binary="/Volumes/Work/OpenSource/WebKitBuild/Release/safaridriver" --webdriver-arg="--diagnose" safari "navigation-api/navigation-methods/custom-test.html" Leads to: web-platform-test ~~~~~~~~~~~~~~~~~ Ran 2 checks (1 subtests, 1 tests) Expected results: 2 Unexpected results: 0 OK
Alexey Proskuryakov
Comment 6 2025-10-24 13:20:04 PDT
I don't have a wpt checkout to quickly try it out. But stepping through testharness code, it seems clear that the test cannot work as expected. Perhaps Chrome catches the exception through some other mechanism (although only window.onerror comes to mind, and we support that).
rupin
Comment 7 2025-10-24 14:25:50 PDT
Got it that makes sense! I'll update the Navigation API PR so say that we're rewriting the test in a way that does catch exceptions rather than say there was a WebKit test harness bug.
Note You need to log in before you can comment on or make changes to this bug.