Bug 164948 – run-benchmark nests MotionMark results by the suite name twice

• Component: Tools / Tests
• Status: RESOLVED FIXED

[Carlos Alberto Lopez Perez]

The json results generated by the Animometer benchmark when ran with the run-benchmark tool have two 


Animometer': {metrics': {Score': [Geometric']},
                 tests': {Animometer': {metrics': {Score': [Geometric']},
                                            tests': {Canvas Arcs': {metrics': {Score': {current': [1425.4671861375848]}}},
                                                       Canvas Lines': {metrics': {Score': {current': [6041.67557789635]}}},
                                                       Design': {metrics': {Score': {current': [120.42884422240773]}}},
                                                       Focus': {metrics': {Score': {current': [106.53534015865043]}}},
                                                       Images': {metrics': {Score': {current': [141.71080576796632]}}},
                                                       Leaves': {metrics': {Score': {current': [986.0850870259096]}}},
                                                       Multiply': {metrics': {Score': {current': [1080.5982243695066]}}},
                                                       Paths': {metrics': {Score': {current': [4659.013255111744]}}},
                                                       Suits': {metrics': {Score': {current': [1137.5734845930879]}}}}}}}


Notice how there is a main aggregated geometric result with name Animometer, which contains inside another result of the same type and name.

This is wrong, see the output of read-results:

$ Tools/Scripts/run-benchmark --read-results-json animometer.result 
Animometer:Score:Geometric: 250pt stdev=0.0%
          Animometer:Score:Geometric: 250pt stdev=0.0%
                    Canvas Arcs:Score: 256pt stdev=0.0%
                    Canvas Lines:Score: 967pt stdev=0.0%
                    Design:Score: 50.0pt stdev=0.0%
                    Focus:Score: 255pt stdev=0.0%
                    Images:Score: 208pt stdev=0.0%
                    Leaves:Score: 103pt stdev=0.0%
                    Multiply:Score: 336pt stdev=0.0%
                    Paths:Score: 915pt stdev=0.0%
                    Suits:Score: 184pt stdev=0.0%


Notice here how two entries with "Animometer:Score:Geometric: 250pt " appear

[Carlos Alberto Lopez Perez]

Created attachment 295180 [details]
Patch

[Carlos Alberto Lopez Perez]

After this patch the contents of the JSON file are (without the debugOutput data):

Animometer': {metrics': {Score': [Geometric']},
                 tests': {Canvas Arcs': {metrics': {Score': {current': [251.94623218197282]}}},
                            Canvas Lines': {metrics': {Score': {current': [871.206693453393]}}},
                            Design': {metrics': {Score': {current': [47.84977960868452]}}},
                            Focus': {metrics': {Score': {current': [211.9234170550923]}}},
                            Images': {metrics': {Score': {current': [199.28169278994972]}}},
                            Leaves': {metrics': {Score': {current': [101.77153542904863]}}},
                            Multiply': {metrics': {Score': {current': [335.41278407113873]}}},
                            Paths': {metrics': {Score': {current': [898.6858128956474]}}},
                            Suits': {metrics': {Score': {current': [181.91975476813016]}}}}}


$ Tools/Scripts/run-benchmark --read-results-json animometer.result 
Animometer:Score:Geometric: 238pt stdev=0.0%
          Canvas Arcs:Score: 252pt stdev=0.0%
          Canvas Lines:Score: 871pt stdev=0.0%
          Design:Score: 47.8pt stdev=0.0%
          Focus:Score: 212pt stdev=0.0%
          Images:Score: 199pt stdev=0.0%
          Leaves:Score: 102pt stdev=0.0%
          Multiply:Score: 335pt stdev=0.0%
          Paths:Score: 899pt stdev=0.0%
          Suits:Score: 182pt stdev=0.0%





And run-benchmark produces:

[Ryosuke Niwa]

Comment on attachment 295180 [details]
Patch

This isn’t quite right. Animometer allows multiple suites to be ran, and the one we’re reporting right now is the MotionMark suite.  I’ll go talk with Jon to figure out what to do here.

[Carlos Alberto Lopez Perez]

(In reply to comment #3)
> Comment on attachment 295180 [details]
> Patch
> 
> This isn’t quite right. Animometer allows multiple suites to be ran, and the
> one we’re reporting right now is the MotionMark suite.  I’ll go talk with
> Jon to figure out what to do here.

Perhaps the MotionMark suite is wrongly named as Animometer?
What about something like this?


--- a/PerformanceTests/Animometer/resources/runner/tests.js
+++ b/PerformanceTests/Animometer/resources/runner/tests.js
@@ -29,7 +29,7 @@ var Suite = function(name, tests) {
 
 var Suites = [];
 
-Suites.push(new Suite("Animometer",
+Suites.push(new Suite("MotionMark",
     [
         {
             url: "master/multiply.html",

[Jon Lee]

We do need to rename the directory to MotionMark.

[Carlos Alberto Lopez Perez]

(In reply to comment #5)
> We do need to rename the directory to MotionMark.

So the benchmark itself is named MotionMark, but the current test-suite that is ran by default is named Animometer?

[Ryosuke Niwa]

(In reply to comment #6)
> (In reply to comment #5)
> > We do need to rename the directory to MotionMark.
> 
> So the benchmark itself is named MotionMark, but the current test-suite that
> is ran by default is named Animometer?

No, it’s more that both needs to be renamed to MotionMark.  MotionMark is both a test suite that runs animation performance tests as well as a specific test suite ran by default so the fact it’s nested is actually correct.

[Said Abou-Hallawa]

The patches 

<http://trac.webkit.org/changeset/210459>
<http://trac.webkit.org/changeset/210463>

rename the directory, the patch and the plan files form Animometer to MotionMark.

[Ryosuke Niwa]

Created attachment 299574 [details]
Fixes this

[Ryosuke Niwa]

Now that the remaining tests have been moved to PerformanceTests/Animation, we can safely make this change.

[Build Bot]

Comment on attachment 299574 [details]
Fixes this

Attachment 299574 [details] did not pass mac-debug-ews (mac):
Output: http://webkit-queues.webkit.org/results/2939035

New failing tests:
media/modern-media-controls/tracks-support/tracks-support-show-panel-fullscreen.html

[Build Bot]

Created attachment 299577 [details]
Archive of layout-test-results from ews114 for mac-elcapitan

The attached test failures were seen while running run-webkit-tests on the mac-debug-ews.
Bot: ews114  Port: mac-elcapitan  Platform: Mac OS X 10.11.6

[Said Abou-Hallawa]

Comment on attachment 299574 [details]
Fixes this

View in context: https://bugs.webkit.org/attachment.cgi?id=299574&action=review

> Tools/ChangeLog:9
> +        since the remaining non-MotionMark tests are moved to PerformanceTests/Animation.

What are the non-MotionMark tests which are moved to PerformanceTests/Animation? Under this directory I see the same set of files unchanged for a long time.

[Ryosuke Niwa]

(In reply to comment #13)
> Comment on attachment 299574 [details]
> Fixes this
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=299574&action=review
> 
> > Tools/ChangeLog:9
> > +        since the remaining non-MotionMark tests are moved to PerformanceTests/Animation.
> 
> What are the non-MotionMark tests which are moved to
> PerformanceTests/Animation? Under this directory I see the same set of files
> unchanged for a long time.

content-animation tests. But I guess they aren't really the ones that used to be in MotionMark's directory that weren't part of the MotionMark suite.

[Ryosuke Niwa]

Committed r211202: <http://trac.webkit.org/changeset/211202>