Bug 75473 – Would like a way to generate JUnit-style XML files for run-webkit-tests results

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Tools / Tests

[Adam Roben (:aroben)]

There are some tools out there (e.g., <http://jenkins-ci.org/>) that have built-in support for parsing JUnit-style XML files which describe the results of test runs. It would be nice to be able to generate such files for run-webkit-tests results for use with those tools.

[Adam Roben (:aroben)]

At first I thought a post-processing script could generate these files based on full_results.json and tests_run*.txt, but it seems those files don't contain any information about which tests were skipped (only the total number of skipped tests).

[Adam Roben (:aroben)]

Created attachment 120948 [details]
JUnit-style XML files generated by test-webkitpy --xml

Here's an example set of JUnit-style XML files. They were generated by running "test-webkitpy --xml".

[Dirk Pranke]

I'm a bit puzzled by why we'd want to support integration w/ Jenkins, given that it does roughly the same thing buildbot does? Are there ports that are interested in running Jenkins servers as well? (I'm guessing the answer is yes?)

Also, I'm not sure how you're imagining the layout tests gets mapped onto that style of output ... do we generate one XML file per directory? one file per test? one per worker? one file in total?

Lastly, I suppose it depends on how big the patch ends up being, but I'd rather not have to support two styles of output in NRWT directly; if it makes more sense to add the skipped files to full_results.json I'd probably rather do that.

Ojan, WDYT?

[Ojan Vafai]

(In reply to comment #3)
> Lastly, I suppose it depends on how big the patch ends up being, but I'd rather not have to support two styles of output in NRWT directly; if it makes more sense to add the skipped files to full_results.json I'd probably rather do that.
> 
> Ojan, WDYT?

I'd like to keep full_results.json small. Adding skipped files makes it considerably larger. I don't really see giving a second type of output to be a maintenance problem. Are there other issues you envision with it?

[Adam Roben (:aroben)]

(In reply to comment #3)
> I'm a bit puzzled by why we'd want to support integration w/ Jenkins, given that it does roughly the same thing buildbot does? Are there ports that are interested in running Jenkins servers as well? (I'm guessing the answer is yes?)

I've been experimenting with Jenkins as a replacement for Buildbot. Some of the motivations are listed here: <http://trac.webkit.org/wiki/AdvantagesOfJenkinsOverBuildbot>.

> Also, I'm not sure how you're imagining the layout tests gets mapped onto that style of output ... do we generate one XML file per directory? one file per test? one per worker? one file in total?

I think we can do it however we'd like.

[Dirk Pranke]

(In reply to comment #4)
> (In reply to comment #3)
> > Lastly, I suppose it depends on how big the patch ends up being, but I'd rather not have to support two styles of output in NRWT directly; if it makes more sense to add the skipped files to full_results.json I'd probably rather do that.
> > 
> > Ojan, WDYT?
> 
> I'd like to keep full_results.json small. Adding skipped files makes it considerably larger. I don't really see giving a second type of output to be a maintenance problem. Are there other issues you envision with it?

I was roughly imagining a command line flag to --include-skips-in-results or something like that. However, from looking at the comments in the wiki page below, it seems like if we were to adopt the junit-style file, we might not need the flakiness dashboard at all? (Of course, the junit-style xml file is going to be substantially larger than the json file, regardless of including skipped files or not).

The code is certainly structured well enough to support either mode of result formats easily enough, it's just feature creep that I'm thinking of.

(In reply to comment #5)
> (In reply to comment #3)
> > I'm a bit puzzled by why we'd want to support integration w/ Jenkins, given that it does roughly the same thing buildbot does? Are there ports that are interested in running Jenkins servers as well? (I'm guessing the answer is yes?)
> 
> I've been experimenting with Jenkins as a replacement for Buildbot. Some of the motivations are listed here: <http://trac.webkit.org/wiki/AdvantagesOfJenkinsOverBuildbot>.
> 

Those do seem like good wins. 

> > Also, I'm not sure how you're imagining the layout tests gets mapped onto that style of output ... do we generate one XML file per directory? one file per test? one per worker? one file in total?
> 
> I think we can do it however we'd like.

Sure. I was wondering if you had any leanings.

[Adam Roben (:aroben)]

(In reply to comment #6)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > Also, I'm not sure how you're imagining the layout tests gets mapped onto that style of output ... do we generate one XML file per directory? one file per test? one per worker? one file in total?
> > 
> > I think we can do it however we'd like.
> 
> Sure. I was wondering if you had any leanings.

One file per directory seems like a good way to start. That should keep the total number of files fairly small.

[Dirk Pranke]

I'm closing this as WONTFIX since as far as I know no one is particularly interested in this now.