Make garden-o-matic, rebaseline-server, TestFailures gardening tools QtWebKit friendly

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Ádám Kallai]

I would like to use garden-o-matic and rebaseline-server for gardening our bots.
For example: http://build.webkit.sed.hu/waterfall
I would like garden-o-matic to download data from our bots. It would make gardening much easier. I don't know where to start. Is it possible anyway?

Is any function available in garden-o-matic by chance that makes gardening easier for us on Qt?
Or could you provide any additional information or documentation?
Any advice is appreciated.

Thanks in advance.

Ádám

[Adam Barth]

It's certainly possible, but it's some amount of work.  We'll need to teach garden-o-matic that bots can exist on multiple servers.  It might be easier to have a different builders.js file for the different masters and have the UI display information from one master at a time.

There are probably also some assumptions in the code about the layout of URLs on build.chromium.org that we'd need to generalize.

[Ojan Vafai]

I really support making this happen! Happy to do reviews for this.

(In reply to comment #1)
> It's certainly possible, but it's some amount of work.  We'll need to teach garden-o-matic that bots can exist on multiple servers.  It might be easier to have a different builders.js file for the different masters and have the UI display information from one master at a time.

I was thinking it would make more sense to split it up by platform. Maybe something like this: Apple, GTK, Qt, Chromium. The first three would grab data from build.webkit.org and the Chromium one would do what garden-o-matic currently does (i.e. grab data off build.chromium.org).

It's hard for me to decide whether that's better or whether splitting it up by master is better.

There are two use-cases:
1. I'm a QT engineer and I'm trying to green the QT bots.
2. I just committed a patch and I'm trying to rebaseline my test on all platforms where it fails.

For case 1, splitting it up by platform is clearly better. For case 2, you actually want a unified view across all platforms (i.e. not even split by master).

Now that I've written all this out, I think we should split by platform for now. If we want a grand unified view, we can add UI for that later.

> There are probably also some assumptions in the code about the layout of URLs on build.chromium.org that we'd need to generalize.

If I were doing this, I'd do the following set of patches:
1. Adding a drop-down in the top-right corner of garden-o-matic to choose your platform and then add a builders.js file for QT (we can add Apple and GTK later when all the kinks are worked out).
2. Fix all the garden-o-matic JS code that uses build.chromium.org to use build.webkit.org for non-chromium platforms. Off the top of my head, the only things that need to change are the paths to the expected/actual results and the "master" query parameter passed to test-results.appspot.com (ChromiumWebkit --> webkit.org).
3. Fix webkit-patch rebaseline-test to do the right thing for non-Chromium bots. I think this is just making it grab actual results off build.webkit.org instead of build.chromium.org.

[Ádám Kallai]

Thanks for the detailed information. You helped me a lot. :)

I'm going to take your advice.

[Dirk Pranke]

See also https://bugs.webkit.org/show_bug.cgi?id=84642 .

[Ádám Kallai]

Created attachment 140929 [details]
draft of solution

I tried to explore how garden-o-matic works and hard-coded some parts ( e.q. the returned URLs). I tried this on build.webkit.sed.hu. 

XMLHttpRequest cannot load http://build.webkit.sed.hu/results/x86-64%20Linux%20Qt%20Release/r116095%20(36958)/full_results.json. Origin null is not allowed by Access-Control-Allow-Origin.
builders.js:102["x86-32 Linux Qt Release (Build)"]
builders.js:100http://build.webkit.sed.hu

Unfortunately, I got stuck here. Garden-o-matic cannot access this data source.

net.js: I was tinkering with some things hoping it would work somehow.
There is a function there dealing with XMLHttpRequest, but I'm not sure whether I should set CORS Headers somewhere to get it working (because it cannot load the above).
I wonder if you get the information from your own result server? Because I was trying to download it from our bot.

Could you happen give me some advice?
Thanks in advance.

[Adam Barth]

You need to add CORS support to build.webkit.sed.hu.  Mostly that's a matter of returning * in the Access-Control-Allow-Origin header.

[Adam Barth]

Comment on attachment 140929 [details]
draft of solution

View in context: https://bugs.webkit.org/attachment.cgi?id=140929&action=review

> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/scripts/net.js:28
> +function ADD_RESULTS(x) { return x; }

Please use CORS, not JSONP.

> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/scripts/results.js:181
> +  return "http://build.webkit.sed.hu/results/x86-64%20Linux%20Qt%20Release/r116095%20(36958)/";

The Chromium buildbots have a directory that accumulates results, which is nice because we can query a single URL and get the latest results.  In this URL, you have a specific build number, which we can deal with, it's just more work.

[Zan Dobersek]

I've tried putting something together myself for the Gtk port, but not having the Access-Control-Allow-Origin properly set up on build.webkit.org makes it pretty much impossible to properly test it.

[Ojan Vafai]

(In reply to comment #8)
> I've tried putting something together myself for the Gtk port, but not having the Access-Control-Allow-Origin properly set up on build.webkit.org makes it pretty much impossible to properly test it.

No reason to stall on this. We should setup Access-Control-Allow-Origin. I believe Bill is the right person to set this up. Bill, could we enable access control for everything under http://build.webkit.org/results/?

[Adam Barth]

Maybe we just need to add a .htaccess file to http://trac.webkit.org/browser/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html ?

[Zan Dobersek]

(In reply to comment #9)
> (In reply to comment #8)
> > I've tried putting something together myself for the Gtk port, but not having the Access-Control-Allow-Origin properly set up on build.webkit.org makes it pretty much impossible to properly test it.
> 
> No reason to stall on this. We should setup Access-Control-Allow-Origin. I believe Bill is the right person to set this up. Bill, could we enable access control for everything under http://build.webkit.org/results/?

I believe the access control should be enabled for everything under http://build.webkit.org/json as well.

[Zan Dobersek]

(In reply to comment #10)
> Maybe we just need to add a .htaccess file to http://trac.webkit.org/browser/trunk/Tools/BuildSlaveSupport/build.webkit.org-config/public_html ?

This seems a nice fix, albeit perhaps just momentary.

[Zan Dobersek]

Created attachment 142670 [details]
Patch

[Zan Dobersek]

Comment on attachment 142670 [details]
Patch

Clearing flags and marking as obsolete, this was completed in bug #89164

[Adam Barth]

After http://trac.webkit.org/changeset/120546, getting garden-o-matic working for Qt should be very straight forward.

[Zan Dobersek]

Support for the GTK builders will be added through bug #89360, so I'm un-hijacking the bug.

[Ádám Kallai]

Created attachment 150161 [details]
Added Qt port for garden-o-matic

I added Qt port for config.js. Do I need to add a unittest?

[Adam Barth]

> Do I need to add a unittest?

You can add one if you like, but it's not necessary.  We can also add one the first time we break this.  :)

[WebKit Review Bot]

Comment on attachment 150161 [details]
Added Qt port for garden-o-matic

Clearing flags on attachment: 150161

Committed r121563: <http://trac.webkit.org/changeset/121563>

[WebKit Review Bot]

All reviewed patches have been landed.  Closing bug.