Bug 33357 - Flaky Test: http/tests/xmlhttprequest/cross-origin-no-authorization.html is failing on Snow Leopard Release bot
Summary: Flaky Test: http/tests/xmlhttprequest/cross-origin-no-authorization.html is f...
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: PC OS X 10.5
: P2 Normal
Assignee: Nobody
URL:
Keywords:
: 51732 (view as bug list)
Depends on: 37104
Blocks: 33292
  Show dependency treegraph
 
Reported: 2010-01-07 17:24 PST by Nikolas Zimmermann
Modified: 2011-05-25 20:44 PDT (History)
3 users (show)

See Also:


Attachments
Patch (1.22 KB, patch)
2010-03-31 23:17 PDT, Eric Seidel (no email)
ap: review-
ap: commit-queue-
Details | Formatted Diff | Diff
Failure diff from eseidel-cq-sl (1.47 KB, text/plain)
2011-01-24 04:45 PST, WebKit Commit Bot
no flags Details
Failure diff from eseidel-cq-sl (1.66 KB, text/plain)
2011-01-24 17:36 PST, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-8 (866 bytes, text/plain)
2011-01-31 20:33 PST, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-4 (1.22 KB, text/plain)
2011-02-02 15:53 PST, WebKit Commit Bot
no flags Details
Failure diff from eseidel-cq-sf (1.66 KB, text/plain)
2011-02-05 10:45 PST, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-7 (1.22 KB, text/plain)
2011-02-10 07:02 PST, WebKit Commit Bot
no flags Details
Failure diff from eseidel-cq-sf (734 bytes, text/plain)
2011-03-15 05:07 PDT, WebKit Commit Bot
no flags Details
Failure diff from eseidel-cq-sf (866 bytes, text/plain)
2011-04-14 22:52 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-3 (1.22 KB, text/plain)
2011-04-15 07:51 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-3 (734 bytes, text/plain)
2011-04-18 20:50 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-3 (1.66 KB, text/plain)
2011-04-19 10:12 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-7 (1.66 KB, text/plain)
2011-04-21 08:19 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-8 (1.22 KB, text/plain)
2011-05-01 15:43 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-8 (903 bytes, text/plain)
2011-05-02 18:58 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-3 (903 bytes, text/plain)
2011-05-03 13:28 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-7 (734 bytes, text/plain)
2011-05-03 15:20 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-8 (2.30 KB, text/plain)
2011-05-04 01:22 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-7 (1.22 KB, text/plain)
2011-05-05 17:27 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-4 (1.47 KB, text/plain)
2011-05-06 12:32 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-4 (1.22 KB, text/plain)
2011-05-08 19:15 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-4 (2.30 KB, text/plain)
2011-05-09 12:39 PDT, WebKit Commit Bot
no flags Details
Failure diff from eseidel-cq-sf (1.22 KB, text/plain)
2011-05-09 15:32 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-8 (903 bytes, text/plain)
2011-05-12 00:02 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-4 (1.22 KB, text/plain)
2011-05-16 17:48 PDT, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-7 (866 bytes, text/plain)
2011-05-25 20:44 PDT, WebKit Commit Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nikolas Zimmermann 2010-01-07 17:24:21 PST
http/tests/xmlhttprequest/cross-origin-no-authorization.html is failing on Snow Leopard Release bot

http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r52963%20(4004)/http/tests/xmlhttprequest/cross-origin-no-authorization-pretty-diff.html

Seems to be a new failure (appeared after buildbot restart).
Comment 1 Eric Seidel (no email) 2010-01-09 15:23:14 PST
Same failure just now:
http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53039%20(4076)/results.html

--- layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-expected.txt	2010-01-09 11:04:24.000000000 -0800
+++ layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-actual.txt	2010-01-09 11:04:24.000000000 -0800
@@ -6,7 +6,7 @@
 SCRIPT SRC='...' Should succeed, since authorization is sent for cross-origin subresource loads.
 PASS: Loaded, user test
 Cross-origin XMLHttpRequest (sync), authorization will not be sent, because withCredentials is false.
-PASS: 401 Authorization required
+PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
 Cross-origin XMLHttpRequest (sync), testing authorization that's not allowed by the server (withCredentials is true, but access control headers are not set).
 PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
 Cross-origin XMLHttpRequest (sync), testing cookies.
Comment 4 Eric Seidel (no email) 2010-01-11 17:39:17 PST
Different failure this time.  Looks more like a timeout:

http://build.webkit.org/results/SnowLeopard%20Intel%20Release%20(Tests)/r53113%20(4157)/http/tests/xmlhttprequest/cross-origin-no-authorization-diffs.txt

--- layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-expected.txt	2010-01-11 17:37:19.000000000 -0800
+++ layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-actual.txt	2010-01-11 17:37:19.000000000 -0800
@@ -1,25 +1,4 @@
+CONSOLE MESSAGE: line 5: NETWORK_ERR: XMLHttpRequest Exception 101: A network error occured in synchronous requests.
+FAIL: Timed out waiting for notifyDone to be called
 Start
-Trying different ways to access a password protected resource from another origin. The UA already has login and password for this protection space.
-
-You should see several PASS messages followed by a DONE
-
-SCRIPT SRC='...' Should succeed, since authorization is sent for cross-origin subresource loads.
-PASS: Loaded, user test
-Cross-origin XMLHttpRequest (sync), authorization will not be sent, because withCredentials is false.
-PASS: 401 Authorization required
-Cross-origin XMLHttpRequest (sync), testing authorization that's not allowed by the server (withCredentials is true, but access control headers are not set).
-PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
-Cross-origin XMLHttpRequest (sync), testing cookies.
-PASS
-Cross-origin XMLHttpRequest (async), authorization will not be sent, because withCredentials is false.
-PASS: Received error event.
-Cross-origin XMLHttpRequest (async), testing authorization that's not allowed by the server (withCredentials is true, but access control headers are not set).
-PASS: Received error event.
-Cross-origin XMLHttpRequest (async), testing cookies.
-PASS
-Cross-origin XMLHttpRequest (sync), testing authorization with explicitly provided credentials that should be ignored.
-PASS: 401 Authorization required
-Cross-origin XMLHttpRequest (async), testing authorization with explicitly provided credentials that should be ignored.
-PASS: Received error event.
-DONE
Comment 5 Eric Seidel (no email) 2010-03-31 22:57:26 PDT
Still failing:

http://build.webkit.org/results/SnowLeopard%20Intel%20Leaks/r56894%20(5477)/http/tests/xmlhttprequest/cross-origin-no-authorization-diffs.txt

--- /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-leaks/build/layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-expected.txt	2010-03-31 22:10:31.000000000 -0700
+++ /Volumes/Data/WebKit-BuildSlave/snowleopard-intel-leaks/build/layout-test-results/http/tests/xmlhttprequest/cross-origin-no-authorization-actual.txt	2010-03-31 22:10:31.000000000 -0700
@@ -18,7 +18,7 @@
 Cross-origin XMLHttpRequest (async), testing cookies.
 PASS
 Cross-origin XMLHttpRequest (sync), testing authorization with explicitly provided credentials that should be ignored.
-PASS: 401 Authorization required
+PASS: Got an exception. Error: NETWORK_ERR: XMLHttpRequest Exception 101
 Cross-origin XMLHttpRequest (async), testing authorization with explicitly provided credentials that should be ignored.
 PASS: Received error event.
 DONE

Seems it's time to skip this if we don't know why it's failing.
Comment 6 Eric Seidel (no email) 2010-03-31 23:17:25 PDT
Created attachment 52262 [details]
Patch
Comment 7 Alexey Proskuryakov 2010-04-01 00:17:44 PDT
Comment on attachment 52262 [details]
Patch

I object against skipping security-related tests. Especially since the other result seems to be another kind of success at a first glance.
Comment 8 Eric Seidel (no email) 2010-04-01 00:20:19 PDT
I'm happy to not skip it. But it's been causing random failures on the bots for 2 months now w/o comment from any of the test authors, so I am somewhat limited in my options. :)
Comment 9 Alexey Proskuryakov 2010-04-01 00:29:48 PDT
Anyone is free to investigate this, don't blame the messenger.
Comment 10 Eric Seidel (no email) 2010-04-01 00:43:39 PDT
What your asking is for a single test to hold the green-ness of multiple bots hostage.  Bots are really only useful if we have systems in place to keep them green.  If we have flaky tests which turn them intermittently red then they're only slightly more useful than non-existant bots. :)

Better solutions for handling flaky tests are coming.  But for now, someone trying to maintain green bots, the only tools are rollouts and skipping tests.

Hopefully I will soon be able to provide you data as to when the first time we ever saw this test fail was.
Comment 11 Alexey Proskuryakov 2010-04-01 00:56:55 PDT
It's critically important to have tests that don't let us regress security.

Possibly related: bug 36843.
Comment 12 Eric Seidel (no email) 2010-12-29 19:27:32 PST
*** Bug 51732 has been marked as a duplicate of this bug. ***
Comment 13 WebKit Commit Bot 2011-01-03 09:25:11 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake while processing attachment 77807 [details] on bug 51819.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.4
Comment 14 WebKit Commit Bot 2011-01-24 04:45:27 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 79867 [details] on bug 52977.
Bot: eseidel-cq-sl  Port: Mac  Platform: Mac OS X 10.6.6
Comment 15 WebKit Commit Bot 2011-01-24 04:45:30 PST
Created attachment 79912 [details]
Failure diff from eseidel-cq-sl
Comment 16 WebKit Commit Bot 2011-01-24 17:36:07 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 79992 [details] on bug 46733.
Bot: eseidel-cq-sl  Port: Mac  Platform: Mac OS X 10.6.6
Comment 17 WebKit Commit Bot 2011-01-24 17:36:11 PST
Created attachment 79995 [details]
Failure diff from eseidel-cq-sl
Comment 18 WebKit Commit Bot 2011-01-31 20:33:31 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 80613 [details] on bug 46748.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.6
Comment 19 WebKit Commit Bot 2011-01-31 20:33:34 PST
Created attachment 80712 [details]
Failure diff from cr-jail-8
Comment 20 WebKit Commit Bot 2011-02-02 15:53:41 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 80971 [details] on bug 48719.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.6
Comment 21 WebKit Commit Bot 2011-02-02 15:53:44 PST
Created attachment 80986 [details]
Failure diff from cr-jail-4
Comment 22 WebKit Commit Bot 2011-02-05 10:45:21 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 81356 [details] on bug 53849.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.4
Comment 23 WebKit Commit Bot 2011-02-05 10:45:24 PST
Created attachment 81367 [details]
Failure diff from eseidel-cq-sf
Comment 24 WebKit Commit Bot 2011-02-10 07:02:36 PST
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 81904 [details] on bug 54155.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.6
Comment 25 WebKit Commit Bot 2011-02-10 07:02:39 PST
Created attachment 81975 [details]
Failure diff from cr-jail-7
Comment 26 WebKit Commit Bot 2011-03-15 05:07:18 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 85600 [details] on bug 56255.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.4
Comment 27 WebKit Commit Bot 2011-03-15 05:07:20 PDT
Created attachment 85797 [details]
Failure diff from eseidel-cq-sf
Comment 28 WebKit Commit Bot 2011-04-14 22:52:51 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 89620 [details] on bug 58259.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.4
Comment 29 WebKit Commit Bot 2011-04-14 22:52:54 PDT
Created attachment 89739 [details]
Failure diff from eseidel-cq-sf
Comment 30 WebKit Commit Bot 2011-04-15 07:51:29 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 89664 [details] on bug 58509.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.6
Comment 31 WebKit Commit Bot 2011-04-15 07:51:35 PDT
Created attachment 89789 [details]
Failure diff from cr-jail-3
Comment 32 WebKit Commit Bot 2011-04-18 20:50:18 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 90082 [details] on bug 58798.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.6
Comment 33 WebKit Commit Bot 2011-04-18 20:50:22 PDT
Created attachment 90137 [details]
Failure diff from cr-jail-3
Comment 34 WebKit Commit Bot 2011-04-19 10:12:05 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 90208 [details] on bug 58656.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.6
Comment 35 WebKit Commit Bot 2011-04-19 10:12:09 PDT
Created attachment 90214 [details]
Failure diff from cr-jail-3
Comment 36 WebKit Commit Bot 2011-04-21 08:19:19 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 90523 [details] on bug 59097.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.6
Comment 37 WebKit Commit Bot 2011-04-21 08:19:22 PDT
Created attachment 90538 [details]
Failure diff from cr-jail-7
Comment 38 WebKit Commit Bot 2011-05-01 15:43:40 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 91835 [details] on bug 59840.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7
Comment 39 WebKit Commit Bot 2011-05-01 15:43:43 PDT
Created attachment 91852 [details]
Failure diff from cr-jail-8
Comment 40 WebKit Commit Bot 2011-05-02 18:58:46 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 91974 [details] on bug 59933.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7
Comment 41 WebKit Commit Bot 2011-05-02 18:58:50 PDT
Created attachment 92035 [details]
Failure diff from cr-jail-8
Comment 42 WebKit Commit Bot 2011-05-03 13:28:20 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 91712 [details] on bug 58448.
Bot: cr-jail-3  Port: Mac  Platform: Mac OS X 10.6.7
Comment 43 WebKit Commit Bot 2011-05-03 13:28:23 PDT
Created attachment 92115 [details]
Failure diff from cr-jail-3
Comment 44 WebKit Commit Bot 2011-05-03 15:20:54 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92074 [details] on bug 60024.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.7
Comment 45 WebKit Commit Bot 2011-05-03 15:20:57 PDT
Created attachment 92148 [details]
Failure diff from cr-jail-7
Comment 46 WebKit Commit Bot 2011-05-04 01:22:09 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92095 [details] on bug 59482.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7
Comment 47 WebKit Commit Bot 2011-05-04 01:22:12 PDT
Created attachment 92199 [details]
Failure diff from cr-jail-8
Comment 48 WebKit Commit Bot 2011-05-05 17:26:57 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92378 [details] on bug 60086.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.7
Comment 49 WebKit Commit Bot 2011-05-05 17:27:00 PDT
Created attachment 92511 [details]
Failure diff from cr-jail-7
Comment 50 WebKit Commit Bot 2011-05-06 12:32:15 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92483 [details] on bug 60292.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7
Comment 51 WebKit Commit Bot 2011-05-06 12:32:18 PDT
Created attachment 92624 [details]
Failure diff from cr-jail-4
Comment 52 WebKit Commit Bot 2011-05-08 19:15:23 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92751 [details] on bug 60249.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7
Comment 53 WebKit Commit Bot 2011-05-08 19:15:26 PDT
Created attachment 92752 [details]
Failure diff from cr-jail-4
Comment 54 WebKit Commit Bot 2011-05-09 12:39:04 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92797 [details] on bug 60475.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7
Comment 55 WebKit Commit Bot 2011-05-09 12:39:07 PDT
Created attachment 92828 [details]
Failure diff from cr-jail-4
Comment 56 WebKit Commit Bot 2011-05-09 15:32:56 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 92840 [details] on bug 60500.
Bot: eseidel-cq-sf  Port: Mac  Platform: Mac OS X 10.6.6
Comment 57 WebKit Commit Bot 2011-05-09 15:32:58 PDT
Created attachment 92865 [details]
Failure diff from eseidel-cq-sf
Comment 58 WebKit Commit Bot 2011-05-12 00:02:32 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 93237 [details] on bug 58999.
Bot: cr-jail-8  Port: Mac  Platform: Mac OS X 10.6.7
Comment 59 WebKit Commit Bot 2011-05-12 00:02:36 PDT
Created attachment 93258 [details]
Failure diff from cr-jail-8
Comment 60 WebKit Commit Bot 2011-05-16 17:48:44 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 93707 [details] on bug 60412.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.7
Comment 61 WebKit Commit Bot 2011-05-16 17:48:47 PDT
Created attachment 93721 [details]
Failure diff from cr-jail-4
Comment 62 WebKit Commit Bot 2011-05-25 20:44:48 PDT
The commit-queue just saw http/tests/xmlhttprequest/cross-origin-no-authorization.html flake (Text diff mismatch) while processing attachment 94841 [details] on bug 61417.
Bot: cr-jail-7  Port: Mac  Platform: Mac OS X 10.6.7
Comment 63 WebKit Commit Bot 2011-05-25 20:44:51 PDT
Created attachment 94910 [details]
Failure diff from cr-jail-7