Bug 136848

Summary: run-webkit-tests should count tests submitted as absolute paths once
Product: WebKit Reporter: youenn fablet <youennf>
Component: Tools / TestsAssignee: youenn fablet <youennf>
Status: RESOLVED FIXED    
Severity: Normal CC: ap, commit-queue
Priority: P2    
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 134766    
Attachments:
Description Flags
Patch none

Description youenn fablet 2014-09-16 02:49:19 PDT
Currently, run-webkit-tests is running several times the same tests when the tests are passed as directories using an absolute path.
Comment 1 youenn fablet 2014-09-16 02:55:50 PDT
Created attachment 238168 [details]
Patch
Comment 2 Alexey Proskuryakov 2014-09-16 11:50:09 PDT
It's not obvious to me what use case this fixes. Could you please provide some examples?
Comment 3 youenn fablet 2014-09-16 12:44:23 PDT
As part of bug 134766 and bug 134767, webkit tests are run outside the LayoutTests folder (inside WebKitBuild folder) to keep the source tree clean.
Comment 4 Alexey Proskuryakov 2014-09-16 13:27:36 PDT
That sounds like it has quite a few downsides. I think that we should discuss this on webkit-dev prior ot going forward with bugs.

- If the tests are transient and don't have a common root, how are we going to report these to flakiness dashboard?

- Currently, I can send a trac link to anyone, and a lot of tests will just run in browser. This is very helpful when a test uncovers a bug in a system component, so there is an easy way to reproduce. With transient tests that are not in the tree, that would not work.
Comment 5 youenn fablet 2014-09-16 14:08:49 PDT
(In reply to comment #4)
> That sounds like it has quite a few downsides. I think that we should discuss this on webkit-dev prior ot going forward with bugs.

Sure, I will do that again.

> - If the tests are transient and don't have a common root, how are we going to report these to flakiness dashboard?
> 
> - Currently, I can send a trac link to anyone, and a lot of tests will just run in browser. This is very helpful when a test uncovers a bug in a system component, so there is an easy way to reproduce. With transient tests that are not in the tree, that would not work.

The whole W3C test suite  would actually be imported within the LayoutTests folder and run as usual webkit tests.

The tests with absolute paths would be run offline by people wanting to gather conformance statistics or to bump the imported W3C test suite version and update related test expectations.
Comment 6 Alexey Proskuryakov 2014-09-16 14:47:54 PDT
I see, that makes more sense to me. Not quite sure why the tree needs to remain clean when doing something like this though - it's easy to revert changes once done.
Comment 7 youenn fablet 2014-09-18 02:50:57 PDT
(In reply to comment #6)
> I see, that makes more sense to me. Not quite sure why the tree needs to remain clean when doing something like this though - it's easy to revert changes once done.

Running absolute path tests also ensures that all generated files (expected.txt, crash logs, stderr, diff) are located in the same folder as the tests. That makes it a tad easier to compute the conformance results and generate self-contained reports.
See http://youennf.github.io/w3c-reports/12092014/w3c_conformance_results.html for instance: the report links each test to its relevant .txt file.

We could as well run the tests in a git/svn-ignored LayoutTests sub-folder. The conformance result computation and report generation would probably be somewhat more complex.
Comment 8 WebKit Commit Bot 2014-09-21 13:30:22 PDT
Comment on attachment 238168 [details]
Patch

Clearing flags on attachment: 238168

Committed r173798: <http://trac.webkit.org/changeset/173798>
Comment 9 WebKit Commit Bot 2014-09-21 13:30:24 PDT
All reviewed patches have been landed.  Closing bug.