Bug 95096 – Random test-webkitpy failures on the buildbot

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Zan Dobersek]

When running webkitpy tests on the GTK 64-bit Release there are failures occurring every few runs:
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/28017/steps/webkitpy-test/logs/stdio
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/28008/steps/webkitpy-test/logs/stdio
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/28007/steps/webkitpy-test/logs/stdio
http://build.webkit.org/builders/GTK%20Linux%2064-bit%20Release/builds/28006/steps/webkitpy-test/logs/stdio

This is really annoying as it's causing unnecessary redness. Looking through other builders, I only found one similar failure occurrence on the Chromium Linux Tests builder:
http://build.webkit.org/builders/Chromium%20Linux%20Release%20%28Tests%29/builds/37362/steps/webkitpy-test/logs/stdio

This probably assures that the failures are not port- or builder-specific. Rather than that I'd blame the python tests being run in parallel. On the GTK 64-bit Release builder this means up to 24 tests run at the same time.

I can't reproduce the failures locally in any way, the system being capable of running only 4 tests in parallel. OTOH, with all the tests being run serially (child_processes=1) the tests pass consistently and still take roughly 10 seconds to run (as compared to the 3-3.5 seconds when run in parallel (locally)).

Because of that I'd propose to either run the python tests serially on all the builders or at least the GTK-specific ones.

Dirk, your comments are most welcome.

[Dirk Pranke]

ideally we should figure out why the tests are failing when run in parallel.

that said, redness is bad so I'm fine if we disable parallel testing on some of the builders, either as a temporary fix while we fix these tests, or indefinitely until you start worrying about cycle time. 

That said, I don't want to disable it on all of them, I'm happy to leave it on for Chromium, for example.

On a side note, we should poke into why test-webkitpy thinks it's logging to a terminal rather than a logfile. Any chance usePty = True in your buildbot slave config?

[Zan Dobersek]

Created attachment 161209 [details]
Run test-webkitpy in 1 child process on GTK builders

[Zan Dobersek]

Comment on attachment 161209 [details]
Run test-webkitpy in 1 child process on GTK builders

Clearing flags on attachment: 161209

Committed r127104: <http://trac.webkit.org/changeset/127104>

[Zan Dobersek]

All reviewed patches have been landed.  Closing bug.