Bug 33357 – Flaky Test: http/tests/xmlhttprequest/cross-origin-no-authorization.html is failing on Snow Leopard Release bot

• Status: RESOLVED
• Resolution: CONFIGURATION CHANGED
• Component: Tools / Tests

[Nikolas Zimmermann]

http/tests/xmlhttprequest/cross-origin-no-authorization.html is failing on Snow Leopard Release bot

http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r52963%20(4004)/http/tests/xmlhttprequest/cross-origin-no-authorization-pretty-diff.html

Seems to be a new failure (appeared after buildbot restart).

[Eric Seidel (no email)]

Same failure just now:
http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53039%20(4076)/results.html

--- layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-expected.txt	2010-01-09 11:04:24.000000000 -0800
+++ layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-actual.txt	2010-01-09 11:04:24.000000000 -0800
@@ -6,7 +6,7 @@
 SCRIPT SRC='...' Should succeed, since authorization is sent for cross-origin subresource loads.
 PASS: Loaded, user test
 Cross-origin XMLHttpRequest (sync), authorization will not be sent, because withCredentials is false.
-PASS: 401 Authorization required
+PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
 Cross-origin XMLHttpRequest (sync), testing authorization that's not allowed by the server (withCredentials is true, but access control headers are not set).
 PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
 Cross-origin XMLHttpRequest (sync), testing cookies.

[Eric Seidel (no email)]

Again tonight:
http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53056%20(4102)/http/tests/xmlhttprequest/cross-origin-no-authorization-diffs.txt

[Eric Seidel (no email)]

Another failure:
http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53103%20(4149)/http/tests/xmlhttprequest/cross-origin-no-authorization-diffs.txt

[Eric Seidel (no email)]

Different failure this time.  Looks more like a timeout:

http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53113%20(4157)/http/tests/xmlhttprequest/cross-origin-no-authorization-diffs.txt

--- layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-expected.txt	2010-01-11 17:37:19.000000000 -0800
+++ layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-actual.txt	2010-01-11 17:37:19.000000000 -0800
@@ -1,25 +1,4 @@
+CONSOLE MESSAGE: line 5: NETWORK_ERR: XMLHttpRequest Exception 101: A network error occured in synchronous requests.
+FAIL: Timed out waiting for notifyDone to be called
 Start
-Trying different ways to access a password protected resource from another origin. The UA already has login and password for this protection space.
-
-You should see several PASS messages followed by a DONE
-
-SCRIPT SRC='...' Should succeed, since authorization is sent for cross-origin subresource loads.
-PASS: Loaded, user test
-Cross-origin XMLHttpRequest (sync), authorization will not be sent, because withCredentials is false.
-PASS: 401 Authorization required
-Cross-origin XMLHttpRequest (sync), testing authorization that's not allowed by the server (withCredentials is true, but access control headers are not set).
-PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
-Cross-origin XMLHttpRequest (sync), testing cookies.
-PASS
-Cross-origin XMLHttpRequest (async), authorization will not be sent, because withCredentials is false.
-PASS: Received error event.
-Cross-origin XMLHttpRequest (async), testing authorization that's not allowed by the server (withCredentials is true, but access control headers are not set).
-PASS: Received error event.
-Cross-origin XMLHttpRequest (async), testing cookies.
-PASS
-Cross-origin XMLHttpRequest (sync), testing authorization with explicitly provided credentials that should be ignored.
-PASS: 401 Authorization required
-Cross-origin XMLHttpRequest (async), testing authorization with explicitly provided credentials that should be ignored.
-PASS: Received error event.
-DONE

[Eric Seidel (no email)]

Still failing:

http://build.webkit.org/results/SnowLeopard%20Intel%20Leaks/r56894%20(5477)/http/tests/xmlhttprequest/cross-origin-no-authorization-diffs.txt

--- /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-leaks/build/layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-expected.txt	2010-03-31 22:10:31.000000000 -0700
+++ /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-leaks/build/layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-actual.txt	2010-03-31 22:10:31.000000000 -0700
@@ -18,7 +18,7 @@
 Cross-origin XMLHttpRequest (async), testing cookies.
 PASS
 Cross-origin XMLHttpRequest (sync), testing authorization with explicitly provided credentials that should be ignored.
-PASS: 401 Authorization required
+PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
 Cross-origin XMLHttpRequest (async), testing authorization with explicitly provided credentials that should be ignored.
 PASS: Received error event.
 DONE

Seems it's time to skip this if we don't know why it's failing.

[Eric Seidel (no email)]

Created attachment 52262 [details]
Patch

[Alexey Proskuryakov]

Comment on attachment 52262 [details]
Patch

I object against skipping security-related tests. Especially since the other result seems to be another kind of success at a first glance.

[Eric Seidel (no email)]

I'm happy to not skip it. But it's been causing random failures on the bots for 2 months now w/o comment from any of the test authors, so I am somewhat limited in my options. :)

[Alexey Proskuryakov]

Anyone is free to investigate this, don't blame the messenger.

[Eric Seidel (no email)]

What your asking is for a single test to hold the green-ness of multiple bots hostage.  Bots are really only useful if we have systems in place to keep them green.  If we have flaky tests which turn them intermittently red then they're only slightly more useful than non-existant bots. :)

Better solutions for handling flaky tests are coming.  But for now, someone trying to maintain green bots, the only tools are rollouts and skipping tests.

Hopefully I will soon be able to provide you data as to when the first time we ever saw this test fail was.

[Alexey Proskuryakov]

It's critically important to have tests that don't let us regress security.

Possibly related: bug 36843.

[Eric Seidel (no email)]

*** Bug 51732 has been marked as a duplicate of this bug. ***

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake while processing attachment 77807 [details] on bug 51819.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.4

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 79867 [details] on bug 52977.
Bot: eseidel-cq-sl  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 79912 [details]
Failure diff from eseidel-cq-sl

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 79992 [details] on bug 46733.
Bot: eseidel-cq-sl  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 79995 [details]
Failure diff from eseidel-cq-sl

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 80613 [details] on bug 46748.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 80712 [details]
Failure diff from cr-jail-8

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 80971 [details] on bug 48719.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 80986 [details]
Failure diff from cr-jail-4

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 81356 [details] on bug 53849.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.4

[WebKit Commit Bot]

Created attachment 81367 [details]
Failure diff from eseidel-cq-sf

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 81904 [details] on bug 54155.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 81975 [details]
Failure diff from cr-jail-7

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 85600 [details] on bug 56255.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.4

[WebKit Commit Bot]

Created attachment 85797 [details]
Failure diff from eseidel-cq-sf

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 89620 [details] on bug 58259.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.4

[WebKit Commit Bot]

Created attachment 89739 [details]
Failure diff from eseidel-cq-sf

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 89664 [details] on bug 58509.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 89789 [details]
Failure diff from cr-jail-3

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 90082 [details] on bug 58798.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 90137 [details]
Failure diff from cr-jail-3

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 90208 [details] on bug 58656.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 90214 [details]
Failure diff from cr-jail-3

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 90523 [details] on bug 59097.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 90538 [details]
Failure diff from cr-jail-7

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 91835 [details] on bug 59840.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 91852 [details]
Failure diff from cr-jail-8

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 91974 [details] on bug 59933.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92035 [details]
Failure diff from cr-jail-8

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 91712 [details] on bug 58448.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92115 [details]
Failure diff from cr-jail-3

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92074 [details] on bug 60024.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92148 [details]
Failure diff from cr-jail-7

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92095 [details] on bug 59482.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92199 [details]
Failure diff from cr-jail-8

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92378 [details] on bug 60086.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92511 [details]
Failure diff from cr-jail-7

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92483 [details] on bug 60292.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92624 [details]
Failure diff from cr-jail-4

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92751 [details] on bug 60249.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92752 [details]
Failure diff from cr-jail-4

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92797 [details] on bug 60475.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 92828 [details]
Failure diff from cr-jail-4

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92840 [details] on bug 60500.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.6

[WebKit Commit Bot]

Created attachment 92865 [details]
Failure diff from eseidel-cq-sf

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 93237 [details] on bug 58999.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 93258 [details]
Failure diff from cr-jail-8

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 93707 [details] on bug 60412.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 93721 [details]
Failure diff from cr-jail-4

[WebKit Commit Bot]

The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 94841 [details] on bug 61417.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.7

[WebKit Commit Bot]

Created attachment 94910 [details]
Failure diff from cr-jail-7

[Anne van Kesteren]

Reliably fails on Win10 only: https://results.webkit.org/?suite=layout-tests&test=http/tests/xmlhttprequest/cross-origin-no-authorization.html.