Bug 265202 - [WebDriver] Pytest errors running run-webdriver-tests on a clean environment
Summary: [WebDriver] Pytest errors running run-webdriver-tests on a clean environment
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Lauro Moura
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-11-21 10:07 PST by Lauro Moura
Modified: 2024-01-24 07:32 PST (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Lauro Moura 2023-11-21 10:07:11 PST
Steps to reproduce:

- Build WPE/WebKitGTK with `./build-webkit --release --wpe`
- Run the tests with `./Tools/Scripts/run-webdriver-tests --verbose --release --gtk --display=xvfb`

Expected result:

- Test files are discovered
- Sequence of pytest runs for each test file in the Selenium and WPT suites

Actual results:

- Test file are are discovered
- But pytest fails to execute the suite with the following message for each individual test case:

```
webkitpy.webdriver_tests.webdriver_test_runner: [INFO] Collected 212 test files                                                                                                                                                                                                           
ERROR: usage: run-webdriver-tests [options] [file_or_dir] [file_or_dir] [...]                                                                                                                                                                                                             
run-webdriver-tests: error: unrecognized arguments: --timeout --driver=WebKitGTK --driver-binary=/app/webkit/WebKitBuild/GTK/Release/bin/WebKitWebDriver --browser-binary=/app/webkit/WebKitBuild/GTK/Release/bin/MiniBrowser --browser-args=--automation --javascript-can-open-windows-au
tomatically=true --enable-xss-auditor=false /app/webkit/WebDriverTests/imported/selenium/py/test/selenium/webdriver/common/alerts_tests.py                                                                                                                                                
  inifile: /app/webkit/WebDriverTests/imported/selenium/py/setup.cfg                                                                                                                                                                                                                      
  rootdir: /app/webkit/WebDriverTests/imported/selenium/py
```

These unrecognized arguments are added by the rootdir `conftest.py`, present both in selenium (`WebDriverTests/imported/selenium/py/conftest.py`) and wpt suites.

But it looks like the `pytest.main` call inside `Tools/Scripts/webkitpy/webdriver_tests/pytest_runner.py` is failing to find these, as strace shows it searching for `conftest.py` in various /home paths but outside the rootdir, opening the right `conftest.py` (inside the root dir) only after parsing the command line arguments.

At first, I thought it was related to the pytest 7.x bump, but the issue is also present if I roll back the pytest update. I could not bisect yet though.

Hacking around, looks like changing the working directory to inside the rootdir make it work. Not sure if this is a reasonable/right approach, though. (This approach would need to update the place where it would save the json file too)

The bots seem to be working, but likely due to some cached state.
Comment 1 Radar WebKit Bug Importer 2023-11-28 10:08:16 PST
<rdar://problem/118887161>
Comment 2 Lauro Moura 2024-01-23 21:00:09 PST
Pull request: https://github.com/WebKit/WebKit/pull/23151
Comment 3 EWS 2024-01-24 07:32:10 PST
Committed 273413@main (abe059142676): <https://commits.webkit.org/273413@main>

Reviewed commits have been landed. Closing PR #23151 and removing active labels.