Bug 142163 – Import W3C XMLHttpRequest tests

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[youenn fablet]

We should import XHR related web platform tests (https://github.com/w3c/web-platform-tests/tree/master/XMLHttpRequest)

[youenn fablet]

Created attachment 247676 [details]
XHR import based on bug 134766 new tool

[youenn fablet]

Created attachment 254365 [details]
Patch

[Build Bot]

Comment on attachment 254365 [details]
Patch

Attachment 254365 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.appspot.com/results/5811620366254080

New failing tests:
imported/w3c/web-platform-tests/XMLHttpRequest/send-network-error-async-events.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-cors.htm
imported/w3c/web-platform-tests/XMLHttpRequest/open-url-worker-origin.htm
imported/w3c/web-platform-tests/XMLHttpRequest/event-upload-progress-crossorigin.sub.htm

[Build Bot]

Created attachment 254369 [details]
Archive of layout-test-results from ews104 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews104  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 254365 [details]
Patch

Attachment 254365 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.appspot.com/results/4711815606960128

New failing tests:
imported/w3c/web-platform-tests/XMLHttpRequest/send-network-error-async-events.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-cors.htm
imported/w3c/web-platform-tests/XMLHttpRequest/open-url-worker-origin.htm
imported/w3c/web-platform-tests/XMLHttpRequest/event-upload-progress-crossorigin.sub.htm

[Build Bot]

Created attachment 254372 [details]
Archive of layout-test-results from ews101 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[youenn fablet]

Created attachment 263681 [details]
Rebasing tests

[Build Bot]

Comment on attachment 263681 [details]
Rebasing tests

Attachment 263681 [details] did not pass mac-ews (mac):
Output: http://webkit-queues.webkit.org/results/316544

New failing tests:
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-network-error-async-events.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-entity-body-basic.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-repeat-no-args.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-setrequestheader-existing-session.htm
imported/w3c/web-platform-tests/XMLHttpRequest/event-upload-progress-crossorigin.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/preserve-ua-header-on-redirect.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-setrequestheader.htm

[Build Bot]

Created attachment 263690 [details]
Archive of layout-test-results from ews101 for mac-mavericks

The attached test failures were seen while running run-webkit-tests on the mac-ews.
Bot: ews101  Port: mac-mavericks  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 263681 [details]
Rebasing tests

Attachment 263681 [details] did not pass mac-wk2-ews (mac-wk2):
Output: http://webkit-queues.webkit.org/results/316548

New failing tests:
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-network-error-async-events.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-repeat-no-args.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-setrequestheader-existing-session.htm
imported/w3c/web-platform-tests/XMLHttpRequest/preserve-ua-header-on-redirect.htm
imported/w3c/web-platform-tests/XMLHttpRequest/event-upload-progress-crossorigin.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-authentication-basic-setrequestheader.htm

[Build Bot]

Created attachment 263692 [details]
Archive of layout-test-results from ews107 for mac-mavericks-wk2

The attached test failures were seen while running run-webkit-tests on the mac-wk2-ews.
Bot: ews107  Port: mac-mavericks-wk2  Platform: Mac OS X 10.9.5

[Build Bot]

Comment on attachment 263681 [details]
Rebasing tests

Attachment 263681 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/316541

New failing tests:
imported/w3c/web-platform-tests/XMLHttpRequest/send-network-error-async-events.sub.htm
imported/w3c/web-platform-tests/XMLHttpRequest/send-entity-body-basic.htm
imported/w3c/web-platform-tests/XMLHttpRequest/event-upload-progress-crossorigin.sub.htm

[Build Bot]

Created attachment 263693 [details]
Archive of layout-test-results from ews112 for mac-yosemite

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews112  Port: mac-yosemite  Platform: Mac OS X 10.10.5

[youenn fablet]

Created attachment 263694 [details]
Adding additional Failure expectations

[youenn fablet]

Created attachment 263704 [details]
Adding additional Failure expectations

[youenn fablet]

Created attachment 263723 [details]
Skipping import of failing tests

[Darin Adler]

Comment on attachment 263723 [details]
Skipping import of failing tests

View in context: https://bugs.webkit.org/attachment.cgi?id=263723&action=review

> LayoutTests/imported/w3c/ChangeLog:8
> +        * resources/ImportExpectations: skipping import of some XHR tests that are failing or timing out.

Why not skip running them instead of skip importing them?

> LayoutTests/TestExpectations:354
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-aborted.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-abortedonmain.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-overrides.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-overridesexpires.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-simple.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-synconmain.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-twice.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-aborted.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-overrides.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-overridesexpires.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-simple.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-synconworker.html [ Slow ]
> +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-twice.html [ Slow ]

This seems like a problem. Is there any trickier way we can test timeouts without slowing down our testing like this? Long term it’s not good to let these start creeping in.

[youenn fablet]

(In reply to comment #17)
> Comment on attachment 263723 [details]
> Skipping import of failing tests
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=263723&action=review
> 
> > LayoutTests/imported/w3c/ChangeLog:8
> > +        * resources/ImportExpectations: skipping import of some XHR tests that are failing or timing out.
> 
> Why not skip running them instead of skip importing them?

It could be done that way too.

These bugs are failing for various reasons.
For instance, some of them use URLs like www2.localhost which should be ok and served as 127.0.0.1 but are blocked by WTR (and probably unresolvable too).
Some tests are working in some environments but not others.
I thought it would be nicer to import them (as [ Skip ] or not) with proper bugs attached to them.

> 
> > LayoutTests/TestExpectations:354
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-aborted.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-abortedonmain.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-overrides.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-overridesexpires.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-simple.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-synconmain.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-twice.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-aborted.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-overrides.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-overridesexpires.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-simple.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-synconworker.html [ Slow ]
> > +imported/w3c/web-platform-tests/XMLHttpRequest/xmlhttprequest-timeout-worker-twice.html [ Slow ]
> 
> This seems like a problem. Is there any trickier way we can test timeouts
> without slowing down our testing like this? Long term it’s not good to let
> these start creeping in.

These tests are marked as slow (<meta name=timeout content=long>).
In the WPT framework, testharness.js will check this meta tag, and would use a long timeout where necessary.
In current WebKit test framework, we cannot do otherwise than marking these tests as Slow in TestExpectations, at least at first import to reduce the risk of flakiness.

Once they will be run several times, we may be able to remove [ Slow ] for some bots.

Also, these tests are updated versions of LayoutTests/http/tests/xmlhttprequest/timeout/ which are often flaky currently on WebKit bots.
One reason may be that these tests use various XHR timeout values between 200ms and 1s.
WPT tests increased some XHR timeout values to 5s, which is close to some bots regular test timeout.

[WebKit Commit Bot]

Comment on attachment 263723 [details]
Skipping import of failing tests

Clearing flags on attachment: 263723

Committed r191546: <http://trac.webkit.org/changeset/191546>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alexey Proskuryakov]

These tests have uncovered a lot of leaks, jumping from ~600Kb to ~2Mb:

https://build.webkit.org/builders/Apple%20El%20Capitan%20%28Leaks%29/builds/593
https://build.webkit.org/builders/Apple%20El%20Capitan%20%28Leaks%29/builds/594

We really need to find a way to make the leak bot more precise, but this case is pretty clear.

[Alexey Proskuryakov]

Filed bug 150546 about that.

[Alexey Proskuryakov]

Several of these tests fail on Windows:

https://build.webkit.org/results/Apple%20Win%207%20Debug%20(Tests)/r191622%20(68088)/results.html