Bug 34399 – Http testing is flakey on the Windows Test bots

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Andras Becsi]

Recently there are many sporadic timeouts on the Windows bots after one http test times out all the followon are timing out too. It was suggested that the change in http://trac.webkit.org/changeset/53559 caused a regression (discussed here: https://bugs.webkit.org/show_bug.cgi?id=33153), which does not let run-webkit-tests recover from an http timeout.
I don't beleave this is true since there are several cases where one or two http tests time out and no other after them. Such examples are:

http://build.webkit.org/results/Windows%20Debug%20%28Tests%29/r54107%20%289228%29/results.html
http://build.webkit.org/results/Windows%20Debug%20%28Tests%29/r54106%20%289227%29/results.html
http://build.webkit.org/results/Windows%20Debug%20%28Tests%29/r54100%20%289223%29/results.html
http://build.webkit.org/results/Windows%20Debug%20%28Tests%29/r54056%20%289185%29/results.html
http://build.webkit.org/results/Windows%20Release%20%28Tests%29/r54095%20%288562%29/results.html
http://build.webkit.org/results/Windows%20Release%20%28Tests%29/r54083%20%288555%29/results.html
http://build.webkit.org/results/Windows%20Release%20%28Tests%29/r54069%20%288544%29/results.html
http://build.webkit.org/results/Windows%20Release%20%28Tests%29/r54057%20%288533%29/results.html

All the http tests time out after the following tests' time out (there might be 1 or 2 more, but the bots have a very tiny brain):
http/tests/xmlhttprequest/basic-auth.html
http/tests/security/xss-DENIED-xsl-external-entity-redirect.xml
http/tests/security/aboutBlank/xss-DENIED-navigate-opener-document-write.html
http/tests/workers/text-encoding.html

In addition lately there are numerous failed downloads on the Windows Test bots with stderr like:
"File 'archives/win-i386-release/54067.zip' not available at master"
and
"File 'archives/win-i386-debug/54065.zip' not available at master"

[Brian Weinstein]

Thanks for filing and doing some good legwork on this. I'll investigate further tomorrow.

[Csaba Osztrogonác]

I examined all results.html of Windows Release Test bot between r53559 and r54126. This flakeyness occured in r53891 first time. ( http://build.webkit.org/results/Windows%20Release%20%28Tests%29/r53891%20%288413%29/results.html )

I think, it might caused by http://trac.webkit.org/changeset/53889

Eric, Kent, Is it possible?

[Brian Weinstein]

(In reply to comment #2)
> I examined all results.html of Windows Release Test bot between r53559 and
> r54126. This flakeyness occured in r53891 first time. (
> http://build.webkit.org/results/Windows%20Release%20%28Tests%29/r53891%20%288413%29/results.html
> )
> 
> I think, it might caused by http://trac.webkit.org/changeset/53889
> 
> Eric, Kent, Is it possible?

It's not the first change I would have guessed, but anything that touches DRT is possible.

I'd say the best course of action would be to try rolling it out, and see if it fixes the issue. If not, we can roll it back in.

[Csaba Osztrogonác]

http://trac.webkit.org/changeset/53889 rolled out by http://trac.webkit.org/changeset/54295. If it won't solve the problem, I'll roll back it again.

[Csaba Osztrogonác]

(In reply to comment #4)
> http://trac.webkit.org/changeset/53889 rolled out by
> http://trac.webkit.org/changeset/54295. If it won't solve the problem, I'll
> roll back it again.

Unsuccessful experiment. :( So rolled back again by http://trac.webkit.org/changeset/54307

[Csaba Osztrogonác]

2nd experiment:

http://trac.webkit.org/changeset/53559 and http://trac.webkit.org/changeset/54084 rolled out by http://trac.webkit.org/changeset/54312

If it won't solve the problem, I'll roll back it again.

[Csaba Osztrogonác]

(In reply to comment #6)
> 2nd experiment:
> 
> http://trac.webkit.org/changeset/53559 and
> http://trac.webkit.org/changeset/54084 rolled out by
> http://trac.webkit.org/changeset/54312
> 
> If it won't solve the problem, I'll roll back it again.

Unsuccessful experiment again. :( So rolled back again by
http://trac.webkit.org/changeset/54314

[Andras Becsi]

This is not an issue any more and http testing on the Windows bots seems stable for a long time now so I'll close this bug.

[Eric Seidel (no email)]

I'm surprised if that's actually true.  We don't monitor the Windows test bots because they're not stable enough to be part of core. :)

[Andras Becsi]

(In reply to comment #9)
> I'm surprised if that's actually true.  We don't monitor the Windows test bots because they're not stable enough to be part of core. :)

The problem of the bug (one http test timing out, thereafter all other http tests also time out) didn't appear for a long time now, so the bug seems fixed. This however does not mean that the bots are stable, generally speaking, but this is not an issue of this bug.
My previous comment did not explicitly explain that, so thanks Eric for pointing that out.