Bug 23343

Summary: Html documentation for bisect-builds would be useful
Product: WebKit Reporter: Gavin Sherlock <gsherloc>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: aroben, ddkilzer, mrowe
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Macintosh   
OS: OS X 10.5   
URL: http://webkit.org/quality/bugwriting.html

Description Gavin Sherlock 2009-01-14 22:40:50 PST
I typically work out when something regressed manually, but bisect-builds seems like a better choice.  It didn't work how I thought it would work, and something more illuminating than the command line help would be useful.
Comment 1 David Kilzer (:ddkilzer) 2009-03-17 09:00:41 PDT
Please be more specific about what isn't clear about the command-line help:

$ ./WebKitTools/Scripts/bisect-builds --help
Search WebKit nightly builds for changes in behavior.
Usage: bisect-builds [options] [url]
  [-b|--branch name]             name of the nightly build branch (default: trunk)
  [-d|--download-directory dir]  nightly build download directory (default: ~/Library/Caches/WebKit-Nightlies)
  [-h|--help]                    show this help message
  [-l|--local]                   only use local (already downloaded) nightlies
  [-p|--progression]             searching for a progression, not a regression
  [-r|--revision M[:N]]          specify starting (and optional ending) revisions to search
  [--safari-path path]           path to Safari application bundle (default: /Applications/Safari.app)
  [-s|--sanity-check]            verify both starting and ending revisions before bisecting

-b|--branch was only useful when there were separate nightly builds for SVG and non-SVG.  Could probably be obsoleted.

-d|--download-directory is if you don't want the copies of nightly builds stored in the default location (or if you already have some downloaded).

-l|--local is for when you're not connected to the Internet (or your disk is full), so you only want to search the local copies of nightlies that are already downloaded.

-p|--progression is used if you're searching for a progression (instead of a regression).  It basically makes it much less confusing when answering the question, "Did the bug reproduce in rNNNNN (yes/no/broken)?", because you don't have to reverse your logic.  (A regression assumes that the older revision will NOT reproduce the bug and that the newer revision WILL reproduce the bug.  A progression is the opposite.)  The script assumes a regression by default unless this flag is used.

-r|--revision lets you specify which revisions to test.  You can use various forms:

* No revisions means to test every revision available.  (Note that Safari 2.0.x is able to test the most nightly builds; newer versions of Safari are able to test fewer and fewer nightly builds.)
* Specifying one revision gives a lower bound (the lowest revision to test).  The upper bound is the most recently nightly build.
* Specifying two revisions gives a lower and an upper bound.  Note that two revisions may be specified as "-rNLOWER -rNUPPER" or "-r NLOWER:NUPPER".

--safari-path is useful if you've kept older Safari binaries around and want to use them to test.

-s|--sanity-check tells bisect-builds to check both the lower and upper bound revisions to make sure that the bug does (or does not) reproduce in those revisions (so you don't start bisecting under false pretenses).
Comment 2 Gavin Sherlock 2009-03-19 08:46:43 PDT
The command line documentation is fine.  However, I only discovered that bisect-builds existed when I saw it mentioned in another bug report.  I think it would help if there was also an html page.  If the page at:

http://webkit.org/quality/bugwriting.html

mentioned in step 7 that the range of builds in which a regression occurred can be narrowed down semi-automatically using bisect-builds, with a link to a page about bisect-builds, saying how to get it and run it, maybe more volunteer bug reporters would use it.  I changed the bug title to reflect that html documentation is what I meant
Comment 3 David Kilzer (:ddkilzer) 2009-03-19 10:17:51 PDT
I understand now.  Thanks!