Bug 142400 – [buildbot] clean-build script should remove untracked files and revert local changes too

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Csaba Osztrogonác]

inspired by bug141946

Sometimes the repository of the bots has conflicts (who knows how is it possible),
sometimes untracked orphaned files cause issues.

I think Tools/BuildSlaveSupport/clean-build should solve this issues too.

[David Kilzer (:ddkilzer)]

See Bug 141946 Comment #25 and Bug 142083 Comment #19 for how to clean up these two particular issues.

In summary, we probably want to write a script that runs 'svn revert --recursive' on the entire tree, then does an 'svn stat' to find "orphaned" files/directories with status "?", then remove those.

Or we could switch the bots to check out from git.webkit.org, then use git-revert and git-clean without having to write a separate script, and get the added benefit of faster updates from trunk/master.

[Csaba Osztrogonác]

(In reply to comment #1)
> See Bug 141946 Comment #25 and Bug 142083 Comment #19 for how to clean up
> these two particular issues.
> 
> In summary, we probably want to write a script that runs 'svn revert
> --recursive' on the entire tree, then does an 'svn stat' to find "orphaned"
> files/directories with status "?", then remove those.

Sounds good. 

> Or we could switch the bots to check out from git.webkit.org, then use
> git-revert and git-clean without having to write a separate script, and get
> the added benefit of faster updates from trunk/master.

With git we would loose the human readable svn revision number,
which is undesirable. And git would be overkiller for the ARM
bots.

[Csaba Osztrogonác]

Or a third, but longer term option:
- switch to server side SVN step: bug90075
- use it for proper fresh build on demand: bug90075

We can do it once all slaves upgraded to at least 0.8.5, but it seems
there are many slaves still use very old buildslave version:
https://build.webkit.org/buildslaves

- Apple Mavericks bots: 0.8.3
- EFL ARM bots: 0.8.3 (not an issue, I can easiy upgrade them)
- GTK bots: 0.7.12

[David Kilzer (:ddkilzer)]

Found some old (didn't check file dates) files that had been sitting in the svn tree on some Mavericks internal bots as well (that should have been cleaned up):

$ svn stat .
D     C Source/WebKit2/Scripts/webkit2
      >   local unversioned, incoming add upon update
D       Source/WebKit2/Scripts/webkit2/LegacyMessageReceiver-expected.cpp
D       Source/WebKit2/Scripts/webkit2/LegacyMessages-expected.h
D       Source/WebKit2/Scripts/webkit2/MessageReceiver-expected.cpp
D       Source/WebKit2/Scripts/webkit2/MessageReceiverSuperclass-expected.cpp
D       Source/WebKit2/Scripts/webkit2/Messages-expected.h
D       Source/WebKit2/Scripts/webkit2/MessagesSuperclass-expected.h
D       Source/WebKit2/Scripts/webkit2/__init__.py
D       Source/WebKit2/Scripts/webkit2/messages.py
D       Source/WebKit2/Scripts/webkit2/messages_unittest.py
D       Source/WebKit2/Scripts/webkit2/model.py
D       Source/WebKit2/Scripts/webkit2/parser.py
D       Source/WebKit2/Scripts/webkit2/test-legacy-messages.in
D       Source/WebKit2/Scripts/webkit2/test-messages.in
D       Source/WebKit2/Scripts/webkit2/test-superclass-messages.in
?       layout-test-results
?       layout-test-results.zip
Summary of conflicts:
  Tree conflicts: 1

[David Kilzer (:ddkilzer)]

(In reply to comment #2)
> (In reply to comment #1)
> > See Bug 141946 Comment #25 and Bug 142083 Comment #19 for how to clean up
> > these two particular issues.
> > 
> > In summary, we probably want to write a script that runs 'svn revert
> > --recursive' on the entire tree, then does an 'svn stat' to find "orphaned"
> > files/directories with status "?", then remove those.
> 
> Sounds good. 
> 
> > Or we could switch the bots to check out from git.webkit.org, then use
> > git-revert and git-clean without having to write a separate script, and get
> > the added benefit of faster updates from trunk/master.
> 
> With git we would loose the human readable svn revision number,
> which is undesirable. And git would be overkiller for the ARM
> bots.

No, I'm talking about using the existing git-svn repository which still contains all the svn revision number happiness, and none of the svn performance issues when checking out WebKit from scratch.

[Alexey Proskuryakov]

Cleaning up git-svn checkouts seems like a much bigger trouble than cleaning up svn FWIW - just look at EWS configuration, which is super tricky.

That said, we seem to have it working well enough on EWS now.

[Csaba Osztrogonác]

(In reply to comment #5)
> > With git we would loose the human readable svn revision number,
> > which is undesirable. And git would be overkiller for the ARM
> > bots.
> 
> No, I'm talking about using the existing git-svn repository which still
> contains all the svn revision number happiness, and none of the svn
> performance issues when checking out WebKit from scratch.

Not at all, buildbot can't extract svn revision number. If we switched
to git, we would loose svn revision number on build.webkit.org which
would break flakiness dashboard, WebKit perf monitor and bot watcher's
dashboard too.

What kind of SVN performance issue? A new checkout/clone from sratch is 
much more faster with SVN than GIT and consumes much less CPU time.

[Csaba Osztrogonác]

Just to have some statistics, how fast/slow are svn and git

Cloning from official Apple servers:
-------------------------------------
$ time git clone git://git.webkit.org/WebKit.git
real    28m41.117s ---> 5.4x slower than svn
user    27m23.484s ---> 20.5x more consumed CPU time than svn
sys     2m14.190s  ---> 3.2x more consumed kernel time than svn

$ time svn checkout http://svn.webkit.org/repository/webkit/trunk WebKit
real    5m4.963s
user    1m20.516s
sys     0m42.239s

Cloning from our internal mirrors:
-----------------------------------
$ time git clone git://<our-internal-git-mirror>
real    11m52.672s ---> 6.9x slower than svn
user    26m45.524s ---> 25.1x more consumed CPU time than svn
sys     2m10.426s  ---> 3.6x more consumed kernel time than svn

$ time svn checkout svn://<our-internal-svn-mirror>/webkit/trunk WebKit
real    1m43.370s
user    1m4.520s
sys     0m36.122s

[youenn fablet]

(In reply to comment #8)
> Just to have some statistics, how fast/slow are svn and git
> 
> Cloning from official Apple servers:
> -------------------------------------
> $ time git clone git://git.webkit.org/WebKit.git
> real    28m41.117s ---> 5.4x slower than svn
> user    27m23.484s ---> 20.5x more consumed CPU time than svn
> sys     2m14.190s  ---> 3.2x more consumed kernel time than svn
> 
> $ time svn checkout http://svn.webkit.org/repository/webkit/trunk WebKit
> real    5m4.963s
> user    1m20.516s
> sys     0m42.239s
> 
> Cloning from our internal mirrors:
> -----------------------------------
> $ time git clone git://<our-internal-git-mirror>
> real    11m52.672s ---> 6.9x slower than svn
> user    26m45.524s ---> 25.1x more consumed CPU time than svn
> sys     2m10.426s  ---> 3.6x more consumed kernel time than svn
> 
> $ time svn checkout svn://<our-internal-svn-mirror>/webkit/trunk WebKit
> real    1m43.370s
> user    1m4.520s
> sys     0m36.122s

Do EWS bots need the full git history? What about --depth=1?

[youenn fablet]

Created attachment 249118 [details]
WIP

[youenn fablet]

(In reply to comment #10)
> Created attachment 249118 [details]
> WIP

Patch seems to work on my linux env, both SVN and GIT.
Not sure how it is supposed to get tested though.
Any feedback appreciated.

[Ryosuke Niwa]

Comment on attachment 249118 [details]
WIP

The patch looks great!

[Alexey Proskuryakov]

> $ time svn checkout http://svn.webkit.org/repository/webkit/trunk WebKit
> real    5m4.963s

Is this on Linux or Mac? What is your svn client version?

It's way faster than I expected it to be, especially from Europe.

[Csaba Osztrogonác]

(In reply to comment #13)
> > $ time svn checkout http://svn.webkit.org/repository/webkit/trunk WebKit
> > real    5m4.963s
> 
> Is this on Linux or Mac? What is your svn client version?
> 
> It's way faster than I expected it to be, especially from Europe.

It is Linux, Ubuntu 14.04 with the stock SVN 1.8.8.

[youenn fablet]

Created attachment 249226 [details]
Fixing difference between SVN and GIT for folders and adding unit test

[youenn fablet]

(In reply to comment #15)
> Created attachment 249226 [details]
> Fixing difference between SVN and GIT for folders and adding unit test

While adding unit test, it appears that many SCM unit tests are broken on my env. No assertion failures but errors.

[WebKit Commit Bot]

Comment on attachment 249226 [details]
Fixing difference between SVN and GIT for folders and adding unit test

Clearing flags on attachment: 249226

Committed r182119: <http://trac.webkit.org/changeset/182119>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[WebKit Commit Bot]

Re-opened since this is blocked by bug 143196

[youenn fablet]

Created attachment 249692 [details]
Fixing syntax

[youenn fablet]

Created attachment 249693 [details]
Fixing syntax

[WebKit Commit Bot]

Comment on attachment 249693 [details]
Fixing syntax

Rejecting attachment 249693 [details] from commit-queue.

Failed to run "['/Volumes/Data/EWS/WebKit/Tools/Scripts/webkit-patch', '--status-host=webkit-queues.appspot.com', '--bot-id=webkit-cq-02', 'validate-changelog', '--check-oops', '--non-interactive', 249693, '--port=mac']" exit_code: 1 cwd: /Volumes/Data/EWS/WebKit

ChangeLog entry in Tools/ChangeLog is not at the top of the file.

Full output: http://webkit-queues.appspot.com/results/6541139016417280

[youenn fablet]

Created attachment 249714 [details]
Rebasing according 143135 and testing whether WebKitBuild dir exists before deleting it

[WebKit Commit Bot]

Attachment 249714 [details] did not pass style-queue:


ERROR: Tools/Scripts/webkitpy/common/checkout/scm/scm.py:208:  [SCM.discard_untracked_files] Instance of 'SCM' has no 'untracked_files' member  [pylint/E1101] [5]
Total errors found: 1 in 5 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[youenn fablet]

Created attachment 249803 [details]
Landing with additional test

[WebKit Commit Bot]

Comment on attachment 249803 [details]
Landing with additional test

Clearing flags on attachment: 249803

Committed r182177: <http://trac.webkit.org/changeset/182177>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[Alexey Proskuryakov]

Does this script run on builders and/or testers? It doesn't seem like it does.

[youenn fablet]

(In reply to comment #28)
> Does this script run on builders and/or testers? It doesn't seem like it
> does.

AFAIUI, this script is skipped by default as it removes WebKitBuild folder.
I triggered this script when forcing a clean build on two builders (gtk and apple) this morning and it runned fine in both cases.

[Alexey Proskuryakov]

I see. Maybe we need something that really cleans up everything other than WebKitBuild (and maybe WebKitLibraries), and that runs every time.

[Ryosuke Niwa]

(In reply to comment #30)
> I see. Maybe we need something that really cleans up everything other than
> WebKitBuild (and maybe WebKitLibraries), and that runs every time.

Maybe we can cleanup Git/SVN state and remove untracked files?

[youenn fablet]

(In reply to comment #31)
> (In reply to comment #30)
> > I see. Maybe we need something that really cleans up everything other than
> > WebKitBuild (and maybe WebKitLibraries), and that runs every time.
> 
> Maybe we can cleanup Git/SVN state and remove untracked files?

Removing untracked files would mean removing dynamically downloaded stuff like python modules, wpt modules, all pyc... and also WebKitBuild.
Even if we skip WebKitBuild removal, this may be too expensive to do that for all patches.

We could trigger that clean-up whenever applying patches fail.
Is there any other case where that would be useful?

[Alexey Proskuryakov]

We had cases where build failed because of stale files, yes.

[WebKit Commit Bot]

Re-opened since this is blocked by bug 143314

[youenn fablet]

(In reply to comment #34)
> Re-opened since this is blocked by bug 143314

This patch can be landed again.
I plan to do it tomorrow.

[WebKit Commit Bot]

Comment on attachment 249803 [details]
Landing with additional test

Clearing flags on attachment: 249803

Committed r194696: <http://trac.webkit.org/changeset/194696>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.