The test failures seen in the following WK1 run were consistently failing on ToT WK1 until they were rebaselined last night, but the failures were still triggering retries in spite of our new results database checks. https://ews-build.webkit.org/#/builders/73/builds/20801 https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fcss%2Fcss-cascade%2Fall-prop-initial-xml.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fgeolocation-API%2FPositionOptions.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fgeolocation-API%2Fpermission.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Faccumulation-per-property-002.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Faddition-per-property-002.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fweb-animations%2Fanimation-model%2Fanimation-types%2Finterpolation-per-property-002.html The issue might be that when we query the results database to analyze failures, we are only checking for platform (macOS/iOS) and configuration (release/debug), which will return results from both WK1 and WK2 and throw off the pass/fail percentage. Checking Results database for failing tests. Identifier: 257097@main, configuration: {'platform': 'mac', 'style': 'release'} imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml.html: pass_rate: 72, pre-existing-failure=False Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 72, 'text': 27, 'timeout': 0, 'warning': 0} imported/w3c/web-platform-tests/geolocation-API/PositionOptions.https.html: pass_rate: 44, pre-existing-failure=True Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0} imported/w3c/web-platform-tests/geolocation-API/permission.https.html: pass_rate: 44, pre-existing-failure=True Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0} imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/accumulation-per-property-002.html: pass_rate: 44, pre-existing-failure=True Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0} imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/addition-per-property-002.html: pass_rate: 44, pre-existing-failure=True Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0} imported/w3c/web-platform-tests/web-animations/animation-model/animation-types/interpolation-per-property-002.html: pass_rate: 44, pre-existing-failure=True Response from results-db: {'audio': 0, 'crash': 0, 'error': 0, 'fail': 0, 'image': 0, 'pass': 44, 'text': 55, 'timeout': 0, 'warning': 0} If we had been checking for WK1 results specifically, it seems likely that we would have ignored the failures and skipped the retry step.
To add more info, here is pass rate for this test with/without the wk1/wk2 flavor. So passing wk1 flavor would have helped here. python3 results_db.py imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml.html --commit 257097@main --platform mac --style release pass: 71% python3 results_db.py imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml.html --commit 257097@main --platform mac --style release --flavor wk2 pass: 100% python3 results_db.py imported/w3c/web-platform-tests/css/css-cascade/all-prop-initial-xml.html --commit 257097@main --platform mac --style release --flavor wk1 pass: 48%
<rdar://problem/102860227>
Pull request: https://github.com/WebKit/WebKit/pull/7023
Committed 257238@main (e2196aa0ada3): <https://commits.webkit.org/257238@main> Reviewed commits have been landed. Closing PR #7023 and removing active labels.