Bug 170158 – Fix dashboard test results on the dashboard

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Alexey Proskuryakov]

- El Capitan test results are currently yellow, because dashboard tests fail there.

- Link to dashboard test results points to stdio, which is not useful for these.

[Jason Marcell]

The following commits broke the botwatcher's dashboard unit tests.

https://bugs.webkit.org/show_bug.cgi?id=156595
https://bugs.webkit.org/show_bug.cgi?id=157680

I'm working with the authors of these commits to resolve the unit tests. 

You can run the tests in a headless terminal sessions via WebKitTestRunner as follows:

run-dashboard-tests

You can also run the tests in Safari:

open ./Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/tests/index.html

[Jason Marcell]

(In reply to Alexey Proskuryakov from comment #0)
> - Link to dashboard test results points to stdio, which is not useful for
> these.

I followed the same convention as the layout tests. There is a link that shows a diff of expected vs. actual.

[Jason Marcell]

(In reply to Jason Marcell from comment #2)
> (In reply to Alexey Proskuryakov from comment #0)
> > - Link to dashboard test results points to stdio, which is not useful for
> > these.
> 
> I followed the same convention as the layout tests. There is a link that
> shows a diff of expected vs. actual.

e.g. https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20(Tests)/r214433%20(174)/dashboard-layout-test-results/results.html

[Alexey Proskuryakov]

Created attachment 305554 [details]
proposed fix

[Alexey Proskuryakov]

> I'm working with the authors of these commits to resolve the unit tests. 

Do you expect that this will happen soon (tomorrow)? If not, we should hide dashboard test results, so that the dashboard is not constantly red.

[Jason Marcell]

(In reply to Alexey Proskuryakov from comment #5)
> > I'm working with the authors of these commits to resolve the unit tests. 
> 
> Do you expect that this will happen soon (tomorrow)? If not, we should hide
> dashboard test results, so that the dashboard is not constantly red.

I doubt it will be fixed tomorrow. Hiding the results sounds fine. I didn't even know we could do that!

[Alexey Proskuryakov]

I don't think that there is existing support for hiding a step, we would need to add a temporary hack to the dashboard. Or it may be easier to mark the step as "hidden" in buildbot itself - the dashboard already checks for that.

Would you be willing to look into that? I'm unlikely to have another opportunity to look at the code until Wednesday at best.

[Jason Marcell]

(In reply to Alexey Proskuryakov from comment #7)
> I don't think that there is existing support for hiding a step, we would
> need to add a temporary hack to the dashboard. Or it may be easier to mark
> the step as "hidden" in buildbot itself - the dashboard already checks for
> that.
> 
> Would you be willing to look into that? I'm unlikely to have another
> opportunity to look at the code until Wednesday at best.

Yep. No problem.

[Aakash Jain]

(In reply to Alexey Proskuryakov from comment #5)
> Do you expect that this will happen soon (tomorrow)?

I have fixed all the dashboard tests, pending review(170191 and 170193).

[Jason Marcell]

I think we need to update Tools/BuildSlaveSupport/build.webkit.org-config/public_html/dashboard/Scripts/tests/index-expected.txt

[Jason Marcell]

https://bugs.webkit.org/show_bug.cgi?id=170214

[WebKit Commit Bot]

Comment on attachment 305554 [details]
proposed fix

Clearing flags on attachment: 305554

Committed r214512: <http://trac.webkit.org/changeset/214512>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.