Bug 13599 – Scripts to generate coverage information for WebKit

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Holger Freyther]

As posted on the mailing-list I will post the scripts peau-a-peau. For ease of reading I will post the scripts and no diff containing the scripts.

I understand apple needs to take care of the licenses. The original author of cov.py and regenerate-coverage-display is Nathaniel Smith of the monotone project. His original work can be found in the org.vorpus.monotone-cov branch pull'able from venge.net. I have mailed him and the question is if this enough or if you need more.

"""
>	-What is the original license of these files

GPLv2+

>	-Would you object relicensing to a BSD license

Not really.  Do you need more from me than those two words to make it
actually happen?

"""

[Holger Freyther]

Created attachment 14363 [details]
README

Describes how to use the scripts to generate coverage data. I assume WebKitTools/Coverage contains these files.

[Holger Freyther]

Created attachment 14364 [details]
cov.py

This script parses tries to match source file with one or more .gcov files and counts the line numbers. It is capable of combining the gcov files and writes the result into a .csv files

[Holger Freyther]

Created attachment 14365 [details]
run-generate-coverage-data

Collect source files from depends (.d) files and try to execute gcov from the found directories with the found source files.

[Holger Freyther]

Created attachment 14366 [details]
regenerate-coverage-display

This grabs the .csv files generate by the run- script and plots the coverage changes (matplotlib is required for that feature) and generates HTML like files.

[Holger Freyther]

I have found someone to generate a css and the three images to avoid the lgcov GPL license issue. With a bit of luck he might finish them by tomorrow.

Oh and btw please don't ask to redo these scripts using perl.

[Björn Schmid]

Created attachment 14387 [details]
image for coverage information - color amber

[Björn Schmid]

Created attachment 14388 [details]
image for coverage information - color emerald

[Björn Schmid]

Created attachment 14389 [details]
image for coverage information - color white

[Björn Schmid]

Created attachment 14390 [details]
image for coverage information - color ruby

[Björn Schmid]

Created attachment 14391 [details]
image for coverage information - color white

[Holger Freyther]

Only the css is missing but Björn did create one already but forgot to attach it. I will ask him to attach it.

[Björn Schmid]

Created attachment 14396 [details]
css for coverage information

[Holger Freyther]

Created attachment 14565 [details]
Script, CSS and images to generate coverage information

Changed the directory to WebKitTools/CodeCoverage, updated the README and created a patch from the previous files.

[Nikolas Zimmermann]

Comment on attachment 14565 [details]
Script, CSS and images to generate coverage information

Nice work Holger!
bdash is going to create a "build slave" (build.webkit.org) to do periodically runs of your stuff.

[Mark Rowe (bdash)]

It'd be great if there was some simple interface exposed to make this easier to use.  Setting up a build slave to run a bunch of commands itself is a bit tricky, especially when some of them depend on output from previous steps as at least one of the coverage-related ones appears to.

[Holger Freyther]

(In reply to comment #15)
> It'd be great if there was some simple interface exposed to make this easier to
> use.  Setting up a build slave to run a bunch of commands itself is a bit
> tricky, especially when some of them depend on output from previous steps as at
> least one of the coverage-related ones appears to.
> 

Simple interface as in one script to compile, run the tests, collect the data and upload?

[David Kilzer (:ddkilzer)]

(In reply to comment #16)
> Simple interface as in one script to compile, run the tests, collect the data
> and upload?

Yes, I'm pretty sure that's what Mark meant.  That way it's easy to make it run periodically with no manual intervention.

[Mark Rowe (bdash)]

Yes, that's what I meant.  The uploading part isn't strictly necessary, but automating the other steps would definitely simplify things.

[Mark Rowe (bdash)]

I'll land things as-is, we can track further enhancements in a future bug report.

[Mark Rowe (bdash)]

Ouch, it seems svn-create-patch doesn't override the LANG setting of 'svn' so your attached patch contains some non-English strings where svn-apply expects the English strings.  When applying the patch, it omits all of the binary files :-(  Can you please regenerate this patch with LANG set to C or en_US?  I guess this is something that svn-create-patch should be doing internally...

[Holger Freyther]

Created attachment 14602 [details]
Generate coverage data

Now generated with LC_ALL WebKitTools/Script/svn-createPatch. But please don't apply yet, I will write this small script to execute the commands from the README.

[Mark Rowe (bdash)]

Comment on attachment 14602 [details]
Generate coverage data

Thanks!

[Nikolas Zimmermann]

Created attachment 14604 [details]
--coverage flags for build-webkit / run-jsc-tests

Hey Holger,

I already started a script, it may be a starting point for you.
I basically added a "--coverage" option to both build-webkit & run-javascriptcore-tests.

All we need is a new script "run-coverage" which executes the commands from your
README (best would be to copy the build-webkit script and modify it.)

Greetings,
Niko

[Holger Freyther]

Created attachment 14613 [details]
Patch to add CoverageTools

Changes:
   -Only use matplotlib.date2num but do not plot images
   -Incorporate niko's --coverage param
   -Provide another script that calls WebKitTools/Scripts/build-webkit and calls the test cases
   -A small hack check-global-initializer to not error out on coverage builds. UserObjectImpl.o of JavaScriptGlue has a global initializer and I didn't feel like adding it to the whitelist.
  -The result will be stored in WebKitBuild/Coverage/html which can be scp'ed somewhere else

Requirements:
   easy_install numpy
   easy_install matplotlib

[Nikolas Zimmermann]

Hey Holger!

Patch looks perfect to me - though I can't give official r+, as parts of it are my stuff.
Will ask Oliver to check it..

Greetings,
Niko

[Mark Rowe (bdash)]

Comment on attachment 14613 [details]
Patch to add CoverageTools

Code like:
die unless (system "...") eq 0;

feels quite unnatural.  It would read better if it were:
system("..") == 0 or die;

Is it really necessary to do find . -name '*.gc*' at the top of the WebKit source tree?  Are the *.gc* files not limited to certain directories?  Even if not, omitting LayoutTests from the search path would make this step a bit quicker.


+my $resultName = $svnVersion."-".join('_', @timeData);
Whitespace between operators would be great.

Other than these stylistic issues I see no major issues.

[Holger Freyther]

Created attachment 14636 [details]
Scripts to generate coverage data

-Add space between the string concatination operator
-Use another way to die
-Only delete .gcda files from WebKitBuild

-Fix a bug in regenerate-coverage-display when a file got removed

Now only a rm -rf WebKitBuild/Coverage can be considered.

[Holger Freyther]

Created attachment 14699 [details]
Scripts to generate coverage data

Run run-webkit-tests with --no-launch-safari.

[Mark Rowe (bdash)]

Comment on attachment 14699 [details]
Scripts to generate coverage data

r=me.  Sorry I didn't get back to this sooner.

[Sam Weinig]

Comment on attachment 14699 [details]
Scripts to generate coverage data

I would love to land this but it really needs a ChangeLog.  If you update it with one I will land ASAP.

[Holger Freyther]

(In reply to comment #30)
> (From update of attachment 14699 [details] [edit])
> I would love to land this but it really needs a ChangeLog.  If you update it
> with one I will land ASAP.
> 
Hmm, http://bugs.webkit.org/attachment.cgi?id=14613 had the ChaneLog I don't know how I removed it, I will readd it.

[Holger Freyther]

Created attachment 14753 [details]
Scripts to generate coverage data

This re-adds the ChangeLog entry.

[Sam Weinig]

Landed in r21848.