Bug 136848 – run-webkit-tests should count tests submitted as absolute paths once

• Status: RESOLVED
• Resolution: FIXED
• Component: Tools / Tests

[youenn fablet]

Currently, run-webkit-tests is running several times the same tests when the tests are passed as directories using an absolute path.

[youenn fablet]

Created attachment 238168 [details]
Patch

[Alexey Proskuryakov]

It's not obvious to me what use case this fixes. Could you please provide some examples?

[youenn fablet]

As part of bug 134766 and bug 134767, webkit tests are run outside the LayoutTests folder (inside WebKitBuild folder) to keep the source tree clean.

[Alexey Proskuryakov]

That sounds like it has quite a few downsides. I think that we should discuss this on webkit-dev prior ot going forward with bugs.

- If the tests are transient and don't have a common root, how are we going to report these to flakiness dashboard?

- Currently, I can send a trac link to anyone, and a lot of tests will just run in browser. This is very helpful when a test uncovers a bug in a system component, so there is an easy way to reproduce. With transient tests that are not in the tree, that would not work.

[youenn fablet]

(In reply to comment #4)
> That sounds like it has quite a few downsides. I think that we should discuss this on webkit-dev prior ot going forward with bugs.

Sure, I will do that again.

> - If the tests are transient and don't have a common root, how are we going to report these to flakiness dashboard?
> 
> - Currently, I can send a trac link to anyone, and a lot of tests will just run in browser. This is very helpful when a test uncovers a bug in a system component, so there is an easy way to reproduce. With transient tests that are not in the tree, that would not work.

The whole W3C test suite  would actually be imported within the LayoutTests folder and run as usual webkit tests.

The tests with absolute paths would be run offline by people wanting to gather conformance statistics or to bump the imported W3C test suite version and update related test expectations.

[Alexey Proskuryakov]

I see, that makes more sense to me. Not quite sure why the tree needs to remain clean when doing something like this though - it's easy to revert changes once done.

[youenn fablet]

(In reply to comment #6)
> I see, that makes more sense to me. Not quite sure why the tree needs to remain clean when doing something like this though - it's easy to revert changes once done.

Running absolute path tests also ensures that all generated files (expected.txt, crash logs, stderr, diff) are located in the same folder as the tests. That makes it a tad easier to compute the conformance results and generate self-contained reports.
See http://youennf.github.io/w3c-reports/12092014/w3c_conformance_results.html for instance: the report links each test to its relevant .txt file.

We could as well run the tests in a git/svn-ignored LayoutTests sub-folder. The conformance result computation and report generation would probably be somewhat more complex.

[WebKit Commit Bot]

Comment on attachment 238168 [details]
Patch

Clearing flags on attachment: 238168

Committed r173798: <http://trac.webkit.org/changeset/173798>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.