| Summary: | Mac EWS bots frequently fail several wpt tests | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | youenn fablet <youennf> | ||||
| Component: | Tools / Tests | Assignee: | youenn fablet <youennf> | ||||
| Status: | RESOLVED FIXED | ||||||
| Severity: | Normal | CC: | ap, commit-queue, lforschler, ryanhaddad | ||||
| Priority: | P2 | ||||||
| Version: | WebKit Nightly Build | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| See Also: |
https://bugs.webkit.org/show_bug.cgi?id=152297 https://bugs.webkit.org/show_bug.cgi?id=152298 https://bugs.webkit.org/show_bug.cgi?id=152335 |
||||||
| Bug Depends on: | |||||||
| Bug Blocks: | 152334 | ||||||
| Attachments: |
|
||||||
|
Description
youenn fablet
2015-12-14 09:59:43 PST
(In reply to comment #0) > Some WPT tests have difficulties passing in Mac EWS bots. > But I cannot find flakiness in the dashboard. > imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ] > > imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/ > contenttype_datauri_01.html [ Failure ] > > imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/ > contenttype_datauri_02.html [ Failure ] > imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure > ] > imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ] > > Is there a way to solve this? We noticed this on Friday, but don't have a solution yet. We are going to try to reproduce the failure locally to figure out what is causing the bots to get into this state. My theory is that this happens when running the tests multiple times in the same build directory (slaves on the dashboard always download and unarchive a fresh build directory). There is also this output in the logs, even after deleting the build directory: ////////// WPT tests are not using the same testharness.js file as other WebKit Layout tests. WebKit testharness.js might need to be updated according WPT testharness.js. ////////// (In reply to comment #3) > There is also this output in the logs, even after deleting the build > directory: > > ////////// > WPT tests are not using the same testharness.js file as other WebKit Layout > tests. > WebKit testharness.js might need to be updated according WPT testharness.js. > ////////// It is strange that I do not see it in my Linux machine, while it should. Thanks for the information, I am fixing it in bug 152257. I do not think though that this will help Mac EWS bots. So what should we be looking for? It definitely looks like a harness issue that lines are off by one. Looks like this simply happens on all truly clean builds. Here is what I did, and then I could reproduce this locally: 1. Deleted the build directory. 2. Deleted all unversioned files in Tools and in LayoutTests subdirectories. 3. Deleted all files with svn:ignore in Tools and in LayoutTests subdirectories. Youenn, does this provide enough information to fix this? (In reply to comment #5) > So what should we be looking for? It definitely looks like a harness issue > that lines are off by one. OK, there is a difference of an empty line between WPT and WK testharness.js. Patch for bug 152297 removes that difference, but this will be difficult to deploy it on all bots smoothly. The first step might be to mark these tests as failing temporarily. Then we might need to ensure bots reload the "web-platform-modules/resources" submodule. I will do the first step shortly. Created attachment 267455 [details]
Patch
The commit-queue encountered the following flaky tests while processing attachment 267455 [details]: transitions/default-timing-function.html bug 138901 (author: simon.fraser@apple.com) The commit-queue is continuing to process your patch. The commit-queue encountered the following flaky tests while processing attachment 267455 [details]:
The commit-queue is continuing to process your patch.
Comment on attachment 267455 [details] Patch Clearing flags on attachment: 267455 Committed r194142: <http://trac.webkit.org/changeset/194142> All reviewed patches have been landed. Closing bug. Some additional tests should temporarily be made flaky: LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/interfaces.html LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/responsetype.htm LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/timeout-sync.html LayoutTests/imported/w3c/web-platform-tests/dom/interfaces.htmlLayoutTests/imported/w3c/web-platform-tests/html/dom/interfaces.html |