Bug 61636 – TestFailures page should have links to file bugs for failing tests

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Adam Roben (:aroben)]

The TestFailures page should have links that take you to a pre-filled new bug form for each set of test failures.

[Adam Roben (:aroben)]

<rdar://problem/9514886>

[Adam Roben (:aroben)]

Created attachment 95178 [details]
Add links to TestFailures to easily file bugs about failing tests

[Adam Roben (:aroben)]

See bug 61638 for an example bug that was filed using the new link.

[Jessie Berlin]

Comment on attachment 95178 [details]
Add links to TestFailures to easily file bugs about failing tests

View in context: https://bugs.webkit.org/attachment.cgi?id=95178&action=review

Overall, this looks great! Unofficial r=me

> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:192
> +            if (parsedFailingBuildName.revision - parsedPassingBuildName.revision > 1)

Should this be >= ? Or are you explicitly making sure that there are two failing revisions so as to try to discard flakey tests? If so, you should probably add a comment to that effect.

> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:210
> +            short_desc: 'REGRESSION (' + regressionRangeString + '): ' + failingTests.join(', ') + ' failing on ' + tester.name,

I don't we can assume that the failing tests are regressions, as they might be new tests that are failing since their introduction. It would be nice if we could have a way of figuring out whether the failing tests are new (but not necessarily something that needs to be done in this patch).

[Adam Roben (:aroben)]

Comment on attachment 95178 [details]
Add links to TestFailures to easily file bugs about failing tests

View in context: https://bugs.webkit.org/attachment.cgi?id=95178&action=review

Thanks for reviewing (unofficially)!

>> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:192
>> +            if (parsedFailingBuildName.revision - parsedPassingBuildName.revision > 1)
> 
> Should this be >= ? Or are you explicitly making sure that there are two failing revisions so as to try to discard flakey tests? If so, you should probably add a comment to that effect.

This code is building up the string that goes inside the parentheses in "REGRESSION ():". If the test passed in r12344 and failed in r12345, we want to show "REGRESSION (r12345):". If the test passed in r12344 and failed in r12346, and we don't know about r12345, we want to show "REGRESSION (r12344-r12346):". So we only include the passing revision if the failing revision is two or more revisions later.

>> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:210
>> +            short_desc: 'REGRESSION (' + regressionRangeString + '): ' + failingTests.join(', ') + ' failing on ' + tester.name,
> 
> I don't we can assume that the failing tests are regressions, as they might be new tests that are failing since their introduction. It would be nice if we could have a way of figuring out whether the failing tests are new (but not necessarily something that needs to be done in this patch).

I'll add a FIXME and file a bug.

[Adam Roben (:aroben)]

Comment on attachment 95178 [details]
Add links to TestFailures to easily file bugs about failing tests

View in context: https://bugs.webkit.org/attachment.cgi?id=95178&action=review

>>> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:210
>>> +            short_desc: 'REGRESSION (' + regressionRangeString + '): ' + failingTests.join(', ') + ' failing on ' + tester.name,
>> 
>> I don't we can assume that the failing tests are regressions, as they might be new tests that are failing since their introduction. It would be nice if we could have a way of figuring out whether the failing tests are new (but not necessarily something that needs to be done in this patch).
> 
> I'll add a FIXME and file a bug.

Filed bug 61645.

[David Kilzer (:ddkilzer)]

Comment on attachment 95178 [details]
Add links to TestFailures to easily file bugs about failing tests

View in context: https://bugs.webkit.org/attachment.cgi?id=95178&action=review

r=me!

> Tools/BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/ViewController.js:209
> +            keywords: 'Regression, LayoutTestFailure, MakingBotsRed',

Nit: Not that it matters, but if you alphabetized these, they would show up in the same order once the bug was created.

[Adam Roben (:aroben)]

Committed r87528: <http://trac.webkit.org/changeset/87528>

[Adam Roben (:aroben)]

Removed some accidentally left-in debugging code

Committed r87530: <http://trac.webkit.org/changeset/87530>