Bug 301415
| Summary: | WKTR/WebDriver may exit before whole test is run, allowing obviously failing tests to pass. | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | rupin |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | RESOLVED INVALID | ||
| Severity: | Normal | CC: | ap, gsnedders, webkit-bug-importer |
| Priority: | P2 | Keywords: | InRadar |
| Version: | WebKit Nightly Build | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
rupin
This test should fail because 4 does not equal 3. Yet, it passes:
```
<!doctype html>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
async_test(t => {
window.onload = () => t.step_timeout(t.step_func_done(async () => {
assert_equals(4, 3);
}), 0);
}, "this test should fail");
</script>
```
Note: if you put a console log after the assert, it won't get printed when using log stream. So it seems like the bug is that the test is exiting too early and not even getting to running the assert.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Radar WebKit Bug Importer
<rdar://problem/163323262>
rupin
One WPT test that is affected by this bug: https://github.com/web-platform-tests/wpt/pull/55632 (this PR re-writes the test to work around this bug until it gets fixed). I've explained the bug in the commit message of this PR: https://github.com/WebKit/WebKit/pull/52946.
Alexey Proskuryakov
This is not what's happening, the assert does run. What's happening is that the exception isn't caught by the harness. The test is essentially this:
```
func = async () => { throw 0; }
try {
func.apply(this);
} catch (e) {
alert("why do you expect this to be caught?")
}
```
I verified that this is not just about us having a stale copy of the harness, it behaves the same when I substitute with ToT WPT harness.
rupin
Hello! Thank you for taking a look at this. I don't know too much about the test infrastructure, just noticed this while debugging the Navigation API test.
I filed this because this test (the one in the original top comment) passes with WebKit's driver but does indeed fail when run as a WPT with chrome's driver:
Running:
./wpt run --channel dev --binary "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" chrome "navigation-api/navigation-methods/custom-test.html"
Leads to:
/Volumes/Work/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.3.6'. See: https://github.com/urllib3/urllib3/issues/3020
warnings.warn(
Running 1 tests in web-platform-tests
/Volumes/Work/wpt/_venv3/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 3.3.6'. See: https://github.com/urllib3/urllib3/issues/3020
warnings.warn(
▶ ERROR [expected OK] /navigation-api/navigation-methods/custom-test.html
│ → Unhandled rejection: assert_equals: expected 3 but got 4
│
│ at Test.<anonymous> (http://web-platform.test:8000/navigation-api/navigation-methods/custom-test.html:7:5)
│ at Test.step (http://web-platform.test:8000/resources/testharness.js:2869:25)
│ at Test.<anonymous> (http://web-platform.test:8000/resources/testharness.js:2944:32)
│ at Test.<anonymous> (http://web-platform.test:8000/resources/testharness.js:2990:25)
└ at Test.step (http://web-platform.test:8000/resources/testharness.js:2869:25)
Ran 1 tests finished in 2.3 seconds.
• 0 ran as expected. 0 tests skipped.
• 1 tests had errors unexpectedly
rupin
Versus on WebKit:
Running:
./wpt run --log-mach-verbose --log-mach-level debug --log-mach - --webdriver-binary="/Volumes/Work/OpenSource/WebKitBuild/Release/safaridriver" --webdriver-arg="--diagnose" safari "navigation-api/navigation-methods/custom-test.html"
Leads to:
web-platform-test
~~~~~~~~~~~~~~~~~
Ran 2 checks (1 subtests, 1 tests)
Expected results: 2
Unexpected results: 0
OK
Alexey Proskuryakov
I don't have a wpt checkout to quickly try it out. But stepping through testharness code, it seems clear that the test cannot work as expected. Perhaps Chrome catches the exception through some other mechanism (although only window.onerror comes to mind, and we support that).
rupin
Got it that makes sense! I'll update the Navigation API PR so say that we're rewriting the test in a way that does catch exceptions rather than say there was a WebKit test harness bug.