Bug 51238 - Flaky Test: http/tests/misc/redirect.php
Summary: Flaky Test: http/tests/misc/redirect.php
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: 528+ (Nightly build)
Hardware: Other OS X 10.5
: P2 Normal
Assignee: WebKit Commit Bot
URL:
Keywords:
Depends on:
Blocks: 50856
  Show dependency treegraph
 
Reported: 2010-12-17 01:24 PST by WebKit Commit Bot
Modified: 2010-12-20 06:49 PST (History)
7 users (show)

See Also:


Attachments
Failure diff from abarth-cq-sl (397 bytes, text/plain)
2010-12-17 01:24 PST, WebKit Commit Bot
no flags Details
Failure diff from cr-jail-4 (397 bytes, text/plain)
2010-12-17 01:26 PST, WebKit Commit Bot
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description WebKit Commit Bot 2010-12-17 01:24:03 PST
This is an automatically generated bug from the commit-queue.
http/tests/misc/redirect.php has been flaky on the commit-queue.

http/tests/misc/redirect.php was authored by mjs@apple.com.
http://trac.webkit.org/browser/trunk/LayoutTests/http/tests/misc/redirect.php

The commit-queue just saw http/tests/misc/redirect.php flake while processing attachment 76763 [details] on bug 51184.
Bot: abarth-cq-sl  Port: Mac  Platform: Mac OS X 10.6.5

The bots will update this with information from each new failure.

If you would like to track this test fix with another bug, please close this bug as a duplicate.
Comment 1 WebKit Commit Bot 2010-12-17 01:24:07 PST
Created attachment 76856 [details]
Failure diff from abarth-cq-sl
Comment 2 WebKit Commit Bot 2010-12-17 01:26:21 PST
The commit-queue just saw http/tests/misc/redirect.php flake while processing attachment 76289 [details] on bug 50419.
Bot: cr-jail-4  Port: Mac  Platform: Mac OS X 10.6.4
Comment 3 WebKit Commit Bot 2010-12-17 01:26:24 PST
Created attachment 76857 [details]
Failure diff from cr-jail-4
Comment 4 Alexey Proskuryakov 2010-12-17 12:12:07 PST
-PASS: User-Agent header contains the string 'WebKit'.
+FAIL: User-Agent header does not contain the string 'WebKit', value is 'Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0)'

I can't begin to imagine how WebKit could send such a user agent. Is there someone accessing the commit bot's httpd from outside? That should also be impossible, as (old-)run-webkit-tests only binds Apache to loopback interface by default!
Comment 5 Alexey Proskuryakov 2010-12-17 12:18:43 PST
Per <http://www.botsvsbrowsers.com/details/108410/index.html>, this UA belongs to a known bot, so the commit queue server is accepting external connections on port 8000, and somehow binds the test Apache instance to all interfaces (how?!)

If accurate, this sounds really bad to me.
Comment 6 Eric Seidel (no email) 2010-12-17 12:46:34 PST
These are two separate failures.  The first is from Adam's machine, which is in our livingroom (and is no longer being used as a regular commit queue).

The second is from the cr-jail.  I don't know much about the network topology there.

I believe that the http server we use for layout tests does open port 8000 on a non-loopback interface.

When you say the user agent is from a "known bot" I'm not sure that's correct.  The page you cite "http://www.botsvsbrowsers.com/details/108410/index.html" talks about a longer user agent, including a <script> tag.
Comment 7 Alexey Proskuryakov 2010-12-17 13:18:16 PST
> I believe that the http server we use for layout tests does open port 8000 on a non-loopback interface.

What server is that? As mentioned before, run-webkit-tests doesn't bind to any other interfaces, and for a very good reason. Code that we have in tests certainly isn't secure, and an attacker can most likely gain access to the machine they run on.

> I don't know much about the network topology there.

Who is the person who controls the commit queue then?

> a longer user agent, including a <script> tag

Indeed. Still, this doesn't look like something sent by DumpRenderTree, and must be coming from outside.
Comment 8 Eric Seidel (no email) 2010-12-17 13:43:24 PST
(In reply to comment #7)
> > I believe that the http server we use for layout tests does open port 8000 on a non-loopback interface.
> 
> What server is that? As mentioned before, run-webkit-tests doesn't bind to any other interfaces, and for a very good reason. Code that we have in tests certainly isn't secure, and an attacker can most likely gain access to the machine they run on.

Interesting.  I just tested locally to confirm.  At least run-webkit-httpd does not bind to anything but loopback, nice.

> > I don't know much about the network topology there.
> 
> Who is the person who controls the commit queue then?

I do.  But I don't control the cr-jail network.

> > a longer user agent, including a <script> tag
> 
> Indeed. Still, this doesn't look like something sent by DumpRenderTree, and must be coming from outside.

Yes, but if my if my testing is to be believed, then it must be coming from the same machine?  I do control the machine. :)
Comment 9 Eric Seidel (no email) 2010-12-17 13:49:25 PST
I think this request is coming from:
http://trac.webkit.org/browser/trunk/LayoutTests/http/tests/inspector/extensions-headers.html
Comment 10 Eric Seidel (no email) 2010-12-17 13:50:41 PST
caseq updated that test 12 hours ago after the commit-queue reported it flaky:
http://trac.webkit.org/changeset/74239

I suspect it's also bleeding into other tests.

In general, the inspector tests make up more than 1/3rd of all flaky tests at the moment. :(
Comment 11 Alexey Proskuryakov 2010-12-17 14:11:37 PST
That's a relief, thanks for finding where the request comes from!
Comment 12 Alexey Proskuryakov 2010-12-17 14:13:53 PST
Ideally, the inspector test should have used a custom header to make it clear where it comes from, to be less confusing in a failure case like this.
Comment 13 Andrey Kosyakov 2010-12-17 15:33:36 PST
(In reply to comment #12)
> Ideally, the inspector test should have used a custom header to make it clear where it comes from, to be less confusing in a failure case like this.

It actually uses a custom header _and_ a user agent -- the latter was added to see if it's not overwritten anywhere in the network stack (this happened to be the case for chromium). In fact, if not for the failure of this test, we wouldn't have noticed that we don't clean up user-supplied extra headers when inspector front-end is closed.

So the bottom line -- this wasn't a flakiness at all, just pure failure as a result of bug in inspector (fixed in r74256) and a lack of cleanup in inspector/extensions-headers.html (fixed in r74237), so I'm closing this. Sorry for the mess -- I realized this failure will cause a lot of confusion when I saw it, but didn't see this bug before Eric added me.

As a note -- this test also failed on the "core" bots, which, I guess, could have been used to tell between flakiness and failure. I've also had one of my tests detected as flaky when it was actually failing for a good reason. Can this bot be fixed to consider tests as flaky only in case these were not failing during build of a base revision on core bots? I guess this would improve s/n ratio a bit...
Comment 14 Eric Seidel (no email) 2010-12-17 15:39:59 PST
OK.  So the assumption is this report was a symptom of bug 51272 then.
Comment 15 Andrey Kosyakov 2010-12-17 15:52:57 PST
(In reply to comment #14)
> OK.  So the assumption is this report was a symptom of bug 51272 then.

Yes, thanks for the pointer!
Comment 16 Andrey Kosyakov 2010-12-20 06:49:42 PST
(In reply to comment #14)
> OK.  So the assumption is this report was a symptom of bug 51272 then.

On a second thought, this one is different: there is no other failing test, the original test that caused failure (extensions-headers.html) was passing ok, just causing side-effects on other tests by leaving custom header overrides (a few revisions later, there was a failure in extensions-headers.html, but it's unrelated to this problem).