Bug 156827 – Add content animation tests to benchmark_runner, and allow the runner to collect device data as part of the results

• Status: RESOLVED
• Resolution: FIXED
• Component: New Bugs

[Simon Fraser (smfr)]

Add content animation tests to benchmark_runner, and allow the runner to collect device data as part of the results

[Simon Fraser (smfr)]

Created attachment 276877 [details]
Patch

[Stephanie Lewis]

Comment on attachment 276877 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=276877&action=review

> Tools/ChangeLog:-640
> - 

logs of changelog changes

> Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner.py:81
> +

We still need to kill the server If something goes wrong serving the driver or launching the test.

[Csaba Osztrogonác]

just to document, patches landed in 
- https://trac.webkit.org/changeset/199805
- https://trac.webkit.org/changeset/199806

But unfortunately the new tests fail on all performance bots:

Running Animation/css-animation.html (2 of 153)
ERROR: layer at (0,0) size 800x600
FAILED
Finished: 0.449525 s

Running Animation/raf-animation.html (3 of 153)
ERROR: layer at (0,0) size 800x600
FAILED
Finished: 0.449265 s

[Simon Fraser (smfr)]

(In reply to comment #3)
> just to document, patches landed in 
> - https://trac.webkit.org/changeset/199805
> - https://trac.webkit.org/changeset/199806
> 
> But unfortunately the new tests fail on all performance bots:
> 
> Running Animation/css-animation.html (2 of 153)
> ERROR: layer at (0,0) size 800x600
> FAILED
> Finished: 0.449525 s
> 
> Running Animation/raf-animation.html (3 of 153)
> ERROR: layer at (0,0) size 800x600
> FAILED
> Finished: 0.449265 s

These tests only really make sense for iOS. We should not run them for other platforms.

[Csaba Osztrogonác]

(In reply to comment #4)
> (In reply to comment #3)
> > just to document, patches landed in 
> > - https://trac.webkit.org/changeset/199805
> > - https://trac.webkit.org/changeset/199806
> > 
> > But unfortunately the new tests fail on all performance bots:
> > 
> > Running Animation/css-animation.html (2 of 153)
> > ERROR: layer at (0,0) size 800x600
> > FAILED
> > Finished: 0.449525 s
> > 
> > Running Animation/raf-animation.html (3 of 153)
> > ERROR: layer at (0,0) size 800x600
> > FAILED
> > Finished: 0.449265 s
> 
> These tests only really make sense for iOS. We should not run them for other
> platforms.

In this case please skip them on other platforms. It's not OK leaving bots red.