Bug 216840 – build.webkit.org should run clean build while building revisions having a specified keyword

• Component: Tools / Tests
• Status: ASSIGNED

[Aakash Jain]

build.webkit.org should automatically run clean build while building revisions which have a specified keyword, like [clean-build].

Similar feature being added in EWS in Bug 216610.

[Aakash Jain]

Created attachment 409389 [details]
Patch

[Darin Adler]

The focus on bots exclusively seems wrong here. What about when human beings pull revisions with this keyword?

[Aakash Jain]

I agree that we should solve that problem as well (engineers needing clean build on their machine after a specific revision). 

It's a much larger task though. In case of buildbot (e.g.: build.webkit.org), it's straight-forward, buildbot already know which specific revisions are being built, and can check if any of those revisions needs a clean build (based on keyword like [clean-build]).

For engineer's machines, we would need to figure out which revision was last built, and which revision is currently being build, so that we can check that revision range for [clean-build] keyword. We would probably need to store the last built revision number locally somewhere. Then we might need to support various mechanism in which engineers can build locally on their machine ,e.g.: build-webkit script, make, xcode etc.

Do you think that work should block this patch, or we can land this patch (and patch in 216610), and tackle the clean-build-on-engineers-machine problem afterwards?

[Darin Adler]

(In reply to Aakash Jain from comment #3)
> Do you think that work should block this patch

I do, but others may not agree with me. Adding more differences between bots and desktop way of building worry me.

[Radar WebKit Bug Importer]

<rdar://problem/69758937>

[Carlos Alberto Lopez Perez]

(In reply to Aakash Jain from comment #3)
> For engineer's machines, we would need to figure out which revision was last
> built, and which revision is currently being build, so that we can check
> that revision range for [clean-build] keyword. We would probably need to
> store the last built revision number locally somewhere. Then we might need
> to support various mechanism in which engineers can build locally on their
> machine ,e.g.: build-webkit script, make, xcode etc.

Just sharing a possible idea of how this could be implemented:

1. Instead of relying on the commit log, rely on a new file on the main directory of the source three called CleanBuildsChangeLog or something like that. So when a clean build is needed, the commit should modify that file adding a new changelog entry with (ideally) the reason about why the clean build is needed.

2. The Script build-webkit calculates the hash (md5 or sha256) of the CleanBuildsChangeLog file and checks if there is a file in the build directory named WebKitBuild/Release/.cleanbuild.hashsum (or something similar).

3. If the .cleanbuild.hashsum is not there, then the script simply saves the value of the hash into the file .cleanbuild.hashsum and continues as usual.

4. If the file .cleanbuild.hashsum is there, the script compares both values. If the values are different, then it does a clean build.


This should work for the GTK/WPE bots/developers becase we all use the script build-webkit. But I have no idea if this would work for the developers using Xcode.

WDYT?

[Carlos Alberto Lopez Perez]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> 4. If the file .cleanbuild.hashsum is there, the script compares both
> values. If the values are different, then it does a clean build.

... And then it should save the new value of the hash after wiping the previous build and before starting the new one.

[Carlos Alberto Lopez Perez]

Also I think it would be useful to have a way of ordering clean builds for a specific port. For example, the GTK developers update the flatpak SDK so they want to do clean builds for all the GTK developers. But that should not trigger clean builds for other ports.

So on top of the main CleanBuildsChangeLog file there can be a CleanBuilds${PORTNAME}ChangeLog file and the script can calculate the hash of both files concatenated. That way it would be possible to order clean builds for only one port.

[Darin Adler]

(In reply to Carlos Alberto Lopez Perez from comment #6)
> WDYT?

I think a scheme roughly like this is exactly what I had in mind. It can definitely work with Xcode, but not if the logic is in the build-webkit script.

[Darin Adler]

(In reply to Darin Adler from comment #9)
> I think a scheme roughly like this is exactly what I had in mind. It can
> definitely work with Xcode, but not if the logic is in the build-webkit
> script.

Need a separate script that does this, and is called as an early-enough build step.

One issue with Xcode and any other build technique other than using build-webkit is this isn’t perfectly compatible with building in only one directory. That can be addressed by making the build fail instead of auto-cleaning in cases like.