Bug 90554 – [Gtk] 64-bit Release builder is overpowered for testing purposes

• Status: NEW
• Component: Tools / Tests

[Zan Dobersek]

With its 24 cores, the power of the 64-bit Release builder (working on the gtk-linux-slave-2) comes in handy when compiling, but is actually too much when running the layout tests. Here are per-worker running times from a recent test run:

05:42:43.752 19668 Thread timing:
05:42:43.752 19668      worker/22:  1244 tests,  77.82 secs
05:42:43.752 19668      worker/23:  1233 tests,  72.57 secs
05:42:43.752 19668      worker/20:  1111 tests,  71.24 secs
05:42:43.752 19668      worker/21:   872 tests,  87.23 secs
05:42:43.752 19668       worker/9:  1670 tests,  80.52 secs
05:42:43.752 19668       worker/8:   798 tests, 184.16 secs
05:42:43.752 19668       worker/3:  1480 tests,  91.36 secs
05:42:43.752 19668       worker/2:  1456 tests,  75.10 secs
05:42:43.753 19668       worker/1:   556 tests,  80.89 secs
05:42:43.753 19668       worker/0:  1547 tests, 321.38 secs
05:42:43.753 19668       worker/7:   409 tests, 113.01 secs
05:42:43.753 19668       worker/6:   817 tests, 167.93 secs
05:42:43.753 19668       worker/5:  1714 tests,  75.03 secs
05:42:43.753 19668       worker/4:   241 tests, 106.60 secs
05:42:43.753 19668      worker/19:   914 tests,  92.35 secs
05:42:43.753 19668      worker/18:  1657 tests,  71.64 secs
05:42:43.753 19668      worker/17:  1777 tests,  77.16 secs
05:42:43.753 19668      worker/16:  1196 tests, 159.18 secs
05:42:43.753 19668      worker/15:  1288 tests,  70.74 secs
05:42:43.753 19668      worker/14:  1509 tests,  70.83 secs
05:42:43.753 19668      worker/13:  2112 tests,  82.04 secs
05:42:43.753 19668      worker/12:   569 tests,  77.71 secs
05:42:43.753 19668      worker/11:   930 tests,  74.06 secs
05:42:43.753 19668      worker/10:   599 tests,  73.71 secs
05:42:43.753 19668    2454.27 cumulative, 102.26 optimal

worker/0 takes around 5min20sec to complete the testing while only 5 more workers require more than 100 seconds to complete their test runs. worker/0 is the worker running the locked http tests shard, meaning tests from this shard can't be arranged onto other workers. This means that with sufficient number of workers running in parallel, the worker running the http tests (which are the only tests it will run) will be the one taking the most time, limiting the overall run-webkit-tests execution duration.

Currently the http tests still contain two tests that run for half a minute due to inappropriate test failure handling. They'll be skipped, bringing down the worker testing duration for another minute - that's around 260 seconds for that worker.

With those two tests skipped and now around 2394 seconds taking for the complete test suit to run (when tests are run in succession), this means around 2194 seconds for non-http tests. If the per-worker limit is kept around 260 seconds, it turns out only 9 extra workers are required to be run in parallel for the whole test suit to be tested in around 260 seconds (4min20sec - pretty cool).

Also note that there are a few other tests taking 30 seconds to fail, increasing the testing time unnecessarily. Check out the treemap[1] to spot them.

Then again, the GTK port currently skips around 2500 tests and with some of these being unskipped and general increasing number of tests, the cumulative test time will certainly increase.

In debug builds (based on the last successful testing on the builder) the http tests take around 700 seconds but that will be narrowed down to just around 430 seconds. The cumulative test time will then take around 5580 seconds, or 5150 seconds for non-http tests. With the limit set at around 430 seconds, it would take 12 additional workers to be run in parallel to finish the testing around those 430 seconds (7min10sec).

[1] - http://test-results.appspot.com/dashboards/treemap.html#group=%40ToT%20-%20webkit.org&builder=GTK%20Linux%2064-bit%20Release

[Sergio Villar Senin]

What's exactly the purpose of this bug Zan? Are you suggesting to use it for some other stuff?

We know that most of the cores are in idle state most of the time. We're evaluating the possibility of moving the WK2 testing bot to that machine, but we do not want to assume any kind of risk.

[Zan Dobersek]

(In reply to comment #1)
> What's exactly the purpose of this bug Zan? Are you suggesting to use it for some other stuff?
> 
> We know that most of the cores are in idle state most of the time. We're evaluating the possibility of moving the WK2 testing bot to that machine, but we do not want to assume any kind of risk.

I was thinking of splitting the 24-core build slave into two build slaves (release and build) that would run only the tests. Furthermore, because of such short testing times, I was thinking it might be reasonable to run layout tests on these two builders for both WK1 and WK2.