Bug 152257 - Mac EWS bots frequently fail several wpt tests
Summary: Mac EWS bots frequently fail several wpt tests
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: youenn fablet
URL:
Keywords:
Depends on:
Blocks: 152334
  Show dependency treegraph
 
Reported: 2015-12-14 09:59 PST by youenn fablet
Modified: 2015-12-16 06:48 PST (History)
4 users (show)

See Also:


Attachments
Patch (10.58 KB, patch)
2015-12-16 03:37 PST, youenn fablet
no flags Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description youenn fablet 2015-12-14 09:59:43 PST
Some WPT tests have difficulties passing in Mac EWS bots.
But I cannot find flakiness in the dashboard.
  imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ]
  imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/contenttype_datauri_01.html [ Failure ]
  imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/contenttype_datauri_02.html [ Failure ]
  imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure ]
  imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ]

Is there a way to solve this?
Comment 1 Ryan Haddad 2015-12-14 10:03:32 PST
(In reply to comment #0)
> Some WPT tests have difficulties passing in Mac EWS bots.
> But I cannot find flakiness in the dashboard.
>   imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ]
>  
> imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/
> contenttype_datauri_01.html [ Failure ]
>  
> imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/
> contenttype_datauri_02.html [ Failure ]
>   imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure
> ]
>   imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ]
> 
> Is there a way to solve this?

We noticed this on Friday, but don't have a solution yet. We are going to try to reproduce the failure locally to figure out what is causing the bots to get into this state.
Comment 2 Alexey Proskuryakov 2015-12-14 10:06:53 PST
My theory is that this happens when running the tests multiple times in the same build directory (slaves on the dashboard always download and unarchive a fresh build directory).
Comment 3 Alexey Proskuryakov 2015-12-14 20:25:45 PST
There is also this output in the logs, even after deleting the build directory:

//////////
WPT tests are not using the same testharness.js file as other WebKit Layout tests.
WebKit testharness.js might need to be updated according WPT testharness.js.
//////////
Comment 4 youenn fablet 2015-12-15 02:30:52 PST
(In reply to comment #3)
> There is also this output in the logs, even after deleting the build
> directory:
> 
> //////////
> WPT tests are not using the same testharness.js file as other WebKit Layout
> tests.
> WebKit testharness.js might need to be updated according WPT testharness.js.
> //////////

It is strange that I do not see it in my Linux machine, while it should.

Thanks for the information, I am fixing it in bug 152257.
I do not think though that this will help Mac EWS bots.
Comment 5 Alexey Proskuryakov 2015-12-15 14:46:49 PST
So what should we be looking for? It definitely looks like a harness issue that lines are off by one.
Comment 6 Alexey Proskuryakov 2015-12-15 17:10:27 PST
Looks like this simply happens on all truly clean builds. Here is what I did, and then I could reproduce this locally:

1. Deleted the build directory.
2. Deleted all unversioned files in Tools and in LayoutTests subdirectories.
3. Deleted all files with svn:ignore in Tools and in LayoutTests subdirectories.

Youenn, does this provide enough information to fix this?
Comment 7 youenn fablet 2015-12-16 03:36:39 PST
(In reply to comment #5)
> So what should we be looking for? It definitely looks like a harness issue
> that lines are off by one.

OK, there is a difference of an empty line between WPT and WK testharness.js.
Patch for bug 152297 removes that difference, but this will be difficult to deploy it on all bots smoothly.

The first step might be to mark these tests as failing temporarily.
Then we might need to ensure bots reload the "web-platform-modules/resources" submodule.

I will do the first step shortly.
Comment 8 youenn fablet 2015-12-16 03:37:20 PST
Created attachment 267455 [details]
Patch
Comment 9 WebKit Commit Bot 2015-12-16 04:46:04 PST
The commit-queue encountered the following flaky tests while processing attachment 267455 [details]:

transitions/default-timing-function.html bug 138901 (author: simon.fraser@apple.com)
The commit-queue is continuing to process your patch.
Comment 10 WebKit Commit Bot 2015-12-16 04:46:06 PST
The commit-queue encountered the following flaky tests while processing attachment 267455 [details]:

The commit-queue is continuing to process your patch.
Comment 11 WebKit Commit Bot 2015-12-16 05:30:38 PST
Comment on attachment 267455 [details]
Patch

Clearing flags on attachment: 267455

Committed r194142: <http://trac.webkit.org/changeset/194142>
Comment 12 WebKit Commit Bot 2015-12-16 05:30:42 PST
All reviewed patches have been landed.  Closing bug.
Comment 13 youenn fablet 2015-12-16 06:44:57 PST
Some additional tests should temporarily be made flaky:
LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/interfaces.html
LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/responsetype.htm
LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/timeout-sync.html
LayoutTests/imported/w3c/web-platform-tests/dom/interfaces.htmlLayoutTests/imported/w3c/web-platform-tests/html/dom/interfaces.html