Bug 156827

Summary: Add content animation tests to benchmark_runner, and allow the runner to collect device data as part of the results
Product: WebKit Reporter: Simon Fraser (smfr) <simon.fraser>
Component: New BugsAssignee: Simon Fraser (smfr) <simon.fraser>
Status: RESOLVED FIXED    
Severity: Normal CC: cdumez, clopez, commit-queue, dewei_zhu, glenn, ossy, rniwa, simon.fraser, slewis
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on: 160057    
Bug Blocks:    
Attachments:
Description Flags
Patch slewis: review+

Description Simon Fraser (smfr) 2016-04-20 17:30:31 PDT
Add content animation tests to benchmark_runner, and allow the runner to collect device data as part of the results
Comment 1 Simon Fraser (smfr) 2016-04-20 17:46:17 PDT
Created attachment 276877 [details]
Patch
Comment 2 Stephanie Lewis 2016-04-20 20:30:01 PDT
Comment on attachment 276877 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=276877&action=review

> Tools/ChangeLog:-640
> - 

logs of changelog changes

> Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner.py:81
> +

We still need to kill the server If something goes wrong serving the driver or launching the test.
Comment 3 Csaba Osztrogon√°c 2016-04-21 09:34:18 PDT
just to document, patches landed in 
- https://trac.webkit.org/changeset/199805
- https://trac.webkit.org/changeset/199806

But unfortunately the new tests fail on all performance bots:

Running Animation/css-animation.html (2 of 153)
ERROR: layer at (0,0) size 800x600
FAILED
Finished: 0.449525 s

Running Animation/raf-animation.html (3 of 153)
ERROR: layer at (0,0) size 800x600
FAILED
Finished: 0.449265 s
Comment 4 Simon Fraser (smfr) 2016-04-21 11:02:11 PDT
(In reply to comment #3)
> just to document, patches landed in 
> - https://trac.webkit.org/changeset/199805
> - https://trac.webkit.org/changeset/199806
> 
> But unfortunately the new tests fail on all performance bots:
> 
> Running Animation/css-animation.html (2 of 153)
> ERROR: layer at (0,0) size 800x600
> FAILED
> Finished: 0.449525 s
> 
> Running Animation/raf-animation.html (3 of 153)
> ERROR: layer at (0,0) size 800x600
> FAILED
> Finished: 0.449265 s

These tests only really make sense for iOS. We should not run them for other platforms.
Comment 5 Csaba Osztrogon√°c 2016-04-22 01:48:45 PDT
(In reply to comment #4)
> (In reply to comment #3)
> > just to document, patches landed in 
> > - https://trac.webkit.org/changeset/199805
> > - https://trac.webkit.org/changeset/199806
> > 
> > But unfortunately the new tests fail on all performance bots:
> > 
> > Running Animation/css-animation.html (2 of 153)
> > ERROR: layer at (0,0) size 800x600
> > FAILED
> > Finished: 0.449525 s
> > 
> > Running Animation/raf-animation.html (3 of 153)
> > ERROR: layer at (0,0) size 800x600
> > FAILED
> > Finished: 0.449265 s
> 
> These tests only really make sense for iOS. We should not run them for other
> platforms.

In this case please skip them on other platforms. It's not OK leaving bots red.