Bug 152257 – Mac EWS bots frequently fail several wpt tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[youenn fablet]

Some WPT tests have difficulties passing in Mac EWS bots.
But I cannot find flakiness in the dashboard.
  imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ]
  imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/contenttype_datauri_01.html [ Failure ]
  imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/contenttype_datauri_02.html [ Failure ]
  imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure ]
  imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ]

Is there a way to solve this?

[Ryan Haddad]

(In reply to comment #0)
> Some WPT tests have difficulties passing in Mac EWS bots.
> But I cannot find flakiness in the dashboard.
>   imported/w3c/web-platform-tests/dom/interfaces.html [ Failure ]
>  
> imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/
> contenttype_datauri_01.html [ Failure ]
>  
> imported/w3c/web-platform-tests/dom/nodes/Document-contentType/contentType/
> contenttype_datauri_02.html [ Failure ]
>   imported/w3c/web-platform-tests/dom/nodes/Node-isEqualNode.xhtml [ Failure
> ]
>   imported/w3c/web-platform-tests/html/dom/interfaces.html [ Failure ]
> 
> Is there a way to solve this?

We noticed this on Friday, but don't have a solution yet. We are going to try to reproduce the failure locally to figure out what is causing the bots to get into this state.

[Alexey Proskuryakov]

My theory is that this happens when running the tests multiple times in the same build directory (slaves on the dashboard always download and unarchive a fresh build directory).

[Alexey Proskuryakov]

There is also this output in the logs, even after deleting the build directory:

//////////
WPT tests are not using the same testharness.js file as other WebKit Layout tests.
WebKit testharness.js might need to be updated according WPT testharness.js.
//////////

[youenn fablet]

(In reply to comment #3)
> There is also this output in the logs, even after deleting the build
> directory:
> 
> //////////
> WPT tests are not using the same testharness.js file as other WebKit Layout
> tests.
> WebKit testharness.js might need to be updated according WPT testharness.js.
> //////////

It is strange that I do not see it in my Linux machine, while it should.

Thanks for the information, I am fixing it in bug 152257.
I do not think though that this will help Mac EWS bots.

[Alexey Proskuryakov]

So what should we be looking for? It definitely looks like a harness issue that lines are off by one.

[Alexey Proskuryakov]

Looks like this simply happens on all truly clean builds. Here is what I did, and then I could reproduce this locally:

1. Deleted the build directory.
2. Deleted all unversioned files in Tools and in LayoutTests subdirectories.
3. Deleted all files with svn:ignore in Tools and in LayoutTests subdirectories.

Youenn, does this provide enough information to fix this?

[youenn fablet]

(In reply to comment #5)
> So what should we be looking for? It definitely looks like a harness issue
> that lines are off by one.

OK, there is a difference of an empty line between WPT and WK testharness.js.
Patch for bug 152297 removes that difference, but this will be difficult to deploy it on all bots smoothly.

The first step might be to mark these tests as failing temporarily.
Then we might need to ensure bots reload the "web-platform-modules/resources" submodule.

I will do the first step shortly.

[youenn fablet]

Created attachment 267455 [details]
Patch

[WebKit Commit Bot]

The commit-queue encountered the following flaky tests while processing attachment 267455 [details]:

transitions/default-timing-function.html bug 138901 (author: [e-mail])
The commit-queue is continuing to process your patch.

[WebKit Commit Bot]

The commit-queue encountered the following flaky tests while processing attachment 267455 [details]:

The commit-queue is continuing to process your patch.

[WebKit Commit Bot]

Comment on attachment 267455 [details]
Patch

Clearing flags on attachment: 267455

Committed r194142: <http://trac.webkit.org/changeset/194142>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[youenn fablet]

Some additional tests should temporarily be made flaky:
LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/interfaces.html
LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/responsetype.htm
LayoutTests/imported/w3c/web-platform-tests/XMLHttpRequest/timeout-sync.html
LayoutTests/imported/w3c/web-platform-tests/dom/interfaces.htmlLayoutTests/imported/w3c/web-platform-tests/html/dom/interfaces.html