Bug 93847 – [Qt] Very strange incremental build bug

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Csaba Osztrogonác]

Created attachment 157995 [details]
build log from the bot

There are an incremental build bug on the Qt bots intermittently.
(~ once a week) Unfortunately it's impossible to reproduce it.

But now I have a build log for it and I saved the full 
WebKitBuild directory to help to find the root of the bug.

[Csaba Osztrogonác]

Created attachment 157999 [details]
build log from the bot - same revision, second build, success

[Csaba Osztrogonác]

Created attachment 158002 [details]
diff between WebKitBuild directories

 left side (-): broken build
right side (+): second and working build

[Csaba Osztrogonác]

I found many build failures which can be related to this bug:
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/15427/steps/compile-webkit/logs/stdio/text
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/17506/steps/compile-webkit/logs/stdio/text
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/17673/steps/compile-webkit/logs/stdio/text
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/17795/steps/compile-webkit/logs/stdio/text
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/17978/steps/compile-webkit/logs/stdio/text

It's strange that all of these build failures are on our "x86-32 Linux Qt Release NRWT" bot, which builds on 4 threads. But I can't find any build
failure on our "x86-64 Linux Qt Release" bot which builds on 8 threads.

[Csaba Osztrogonác]

More build failures from the "x86-32 Linux Qt Release - Qt5-WebKit1" bot:
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/9610/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/9888/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/10019/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/10131/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/10417/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/10548/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20-%20Qt5-WebKit1/builds/10578/steps/compile-webkit/logs/stdio

This bot builds with -j4 too.

[Csaba Osztrogonác]

Do we need more build logs from other bots?

[Csaba Osztrogonác]

One more occurance:
http://build.webkit.sed.hu/builders/ARMv7%20Linux%20Qt5%20Release%20%28Build%29/builds/10491/steps/compile-webkit/logs/stdio

[Csaba Osztrogonác]

One more different strange build fail: http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/18644/steps/compile-webkit/logs/stdio

[Csaba Osztrogonác]

Two more occurances of this bug:
- http://build.webkit.sed.hu/builders/ARMv7%20Linux%20Qt5%20Release%20%28Build%29/builds/10561/steps/compile-webkit/logs/stdio
- http://build.webkit.sed.hu/builders/ARMv7%20Linux%20Qt5%20Release%20%28Build%29/builds/10564/steps/compile-webkit/logs/stdio

[Csaba Osztrogonác]

One more fail, now on the EWS bot:
- bug: https://bugs.webkit.org/show_bug.cgi?id=95382#c5
- build log: http://queues.webkit.org/results/13689448

It would be great to fix this crazy bug as soon as possible ...
Or else nobody will trust in Qt buildbots and EWS bot. :-(

[Simon Hausmann]

The "make[4]: *** No targets.  Stop." makes me wonder if this is perhaps the result of a corrupted make file due to two qmake instances writing the same file. That would also explain why the likely-hood of this is somewhat related to the number of parallel jobs.

[Csaba Osztrogonác]

Maybe it is related to https://trac.webkit.org/changeset/121966

[Csaba Osztrogonác]

One more fail: http://build.webkit.sed.hu/builders/x86-32%20Linux%20Qt%20Release%20NRWT/builds/18968/steps/compile-webkit/logs/stdio

[Tor Arne Vestbø]

Taking

[Tor Arne Vestbø]

Created attachment 162231 [details]
Patch

[Csaba Osztrogonác]

Comment on attachment 162231 [details]
Patch

r=me, thanks for the fix.

[Tor Arne Vestbø]

Committed r127583: <http://trac.webkit.org/changeset/127583>