Bug 142325 – run-webkit-tests and run-perf-tests should use WebKitTestRunner by default

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[Chris Dumez]

run-webkit-tests and run-perf-tests should use WebKitTestRunner by default. As a result, "--webkit-test-runner / -2" parameters would be replaced by "--dump-render-tree / -1" so developers can run DumpRenderTree rather than WebKitTestRunner.

[Chris Dumez]

Created attachment 247929 [details]
Patch

[Alexey Proskuryakov]

Comment on attachment 247929 [details]
Patch

This patch fails EWS, gotta fix that too.

run_webkit_tests.py: error: no such option: -2

[Csaba Osztrogonác]

(In reply to comment #2)
> Comment on attachment 247929 [details]
> Patch
> 
> This patch fails EWS, gotta fix that too.
> 
> run_webkit_tests.py: error: no such option: -2

http://trac.webkit.org/browser/trunk/Tools/Scripts/webkitpy/common/config/ports.py#L132

"-2" argument should be removed from MacWK2Port and GtkWK2Port
and --dump-render-tree should be passed to MacPort to run WK1
tests instead of WK2 tests.

[Csaba Osztrogonác]

Comment on attachment 247929 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=247929&action=review

Please update the perf bot names in mastercfg_unittest.py in expected_build_steps variable.

You can run mastercfg_unittest.py by hand in Tools/BuildSlaveSupport/build.webkit.org-config/
after you applied my fix from bug142219, it showed me all issues I commented here.

> Tools/BuildSlaveSupport/build.webkit.org-config/config.json:83
> +                    { "name": "Apple Mavericks Debug WK2 (Tests)", "type": "TestAllButJSCFactory", "builddir": "mavericks-debug-tests-wk2",

TestAllButJSCFactory -> TestAllButJSC

> Tools/BuildSlaveSupport/build.webkit.org-config/config.json:106
> +                    { "name": "Apple Mavericks Release WK2 (Tests)", "type": "TestAllButJSCFactory", "builddir": "mavericks-release-tests-wk2",

ditto

> Tools/BuildSlaveSupport/build.webkit.org-config/config.json:125
> +                    { "name": "Apple Yosemite Debug WK1 (Tests)", "type": "TestWebKit1AllButJSC", "builddir": "yosemite-debug-tests-wk1",

ditto

> Tools/BuildSlaveSupport/build.webkit.org-config/config.json:130
> +                    { "name": "Apple Yosemite Debug WK2 (Tests)", "type": "TestAllButJSCFactory", "builddir": "yosemite-debug-tests-wk2",

ditto

> Tools/BuildSlaveSupport/build.webkit.org-config/config.json:157
> +                    { "name": "Apple Yosemite Release WK2 (Tests)", "type": "TestAllButJSCFactory", "builddir": "yosemite-release-tests-wk2",

ditto

> Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg:780
> -    LayoutTestClass = RunWebKitTests
> +    LayoutTestClass = RunWebKit2Tests

There is no RunWebKit2Tests anymore. It should be RunWebKitTests.

[Csaba Osztrogonác]

Just out of curiosity, is this change mean that WK1 will be obsolete soon?

[Csaba Osztrogonác]

It seems there will be no WK1 perf tester bot after this 
change, you can remove WK1 perf test related crufts too.

[Csaba Osztrogonác]

(In reply to comment #4)
> Please update the perf bot names in mastercfg_unittest.py in
> expected_build_steps variable.

It should have been done in http://trac.webkit.org/changeset/181050 .

[Chris Dumez]

(In reply to comment #5)
> Just out of curiosity, is this change mean that WK1 will be obsolete soon?

All this means is that WK2 is the focus nowadays and should become the default.

Thanks the reviews, I'll rework the patch.

[Chris Dumez]

Created attachment 247968 [details]
Patch

[Csaba Osztrogonác]

Comment on attachment 247968 [details]
Patch

LGTM, r=me.

[Chris Dumez]

Comment on attachment 247968 [details]
Patch

Clearing flags on attachment: 247968

Committed r181093: <http://trac.webkit.org/changeset/181093>

[Chris Dumez]

All reviewed patches have been landed.  Closing bug.

[Chris Dumez]

The bots on build.webkit.org still try to pass --webkit-test-runner. I don't know why yet :(

[Brent Fulgham]

This broke the Windows test builder:

12:06:50.131 224 WebKitTestRunner was not found at /home/buildbot/slave/win-release-tests/build/WebKitBuild/release/bin32/WebKitTestRunner


Windows (unfortunately) does not build WebKitTestRunner.

Can you fix? Or should I roll this out now?

[Chris Dumez]

(In reply to comment #14)
> This broke the Windows test builder:
> 
> 12:06:50.131 224 WebKitTestRunner was not found at
> /home/buildbot/slave/win-release-tests/build/WebKitBuild/release/bin32/
> WebKitTestRunner
> 
> 
> Windows (unfortunately) does not build WebKitTestRunner.
> 
> Can you fix? Or should I roll this out now?

This is likely the same issue, some things are out of sync. Win is supposed to pass --dump-render-tree now but apparently doesn't on the bots. And the wk2 bots still pass --webkit-test-runner although they should no longer do this. I am still trying to figure this out.

[Chris Dumez]

Ok, the bots should hopefully come back up now.

[Csaba Osztrogonác]

It seems the bots should be renamed in perf.webkit.org too.
Ryosuke could you possibly check it?

Uploaded JSON to https://perf.webkit.org/api/report but got an error:
{
    "status": "BuilderNotFound", 
    "failureStored": false, 
    "name": "Apple Yosemite Release WK2 (Perf)", 
    "processedRuns": 0
}

Uploaded JSON to https://perf.webkit.org/api/report but got an error:
{
    "status": "BuilderNotFound", 
    "failureStored": false, 
    "name": "Apple Mavericks Release WK2 (Perf)", 
    "processedRuns": 0
}