136848 2014-09-16 02:49:19 -0700 run-webkit-tests should count tests submitted as absolute paths once 2014-09-21 13:30:24 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) Unspecified Unspecified RESOLVED FIXED P2 Normal --- 134766 1 youennf youennf ap commit-queue oldest_to_newest 1035429 0 youennf 2014-09-16 02:49:19 -0700 Currently, run-webkit-tests is running several times the same tests when the tests are passed as directories using an absolute path. 1035431 1 238168 youennf 2014-09-16 02:55:50 -0700 Created attachment 238168 Patch 1035527 2 ap 2014-09-16 11:50:09 -0700 It's not obvious to me what use case this fixes. Could you please provide some examples? 1035544 3 youennf 2014-09-16 12:44:23 -0700 As part of bug 134766 and bug 134767, webkit tests are run outside the LayoutTests folder (inside WebKitBuild folder) to keep the source tree clean. 1035554 4 ap 2014-09-16 13:27:36 -0700 That sounds like it has quite a few downsides. I think that we should discuss this on webkit-dev prior ot going forward with bugs. - If the tests are transient and don't have a common root, how are we going to report these to flakiness dashboard? - Currently, I can send a trac link to anyone, and a lot of tests will just run in browser. This is very helpful when a test uncovers a bug in a system component, so there is an easy way to reproduce. With transient tests that are not in the tree, that would not work. 1035580 5 youennf 2014-09-16 14:08:49 -0700 (In reply to comment #4) > That sounds like it has quite a few downsides. I think that we should discuss this on webkit-dev prior ot going forward with bugs. Sure, I will do that again. > - If the tests are transient and don't have a common root, how are we going to report these to flakiness dashboard? > > - Currently, I can send a trac link to anyone, and a lot of tests will just run in browser. This is very helpful when a test uncovers a bug in a system component, so there is an easy way to reproduce. With transient tests that are not in the tree, that would not work. The whole W3C test suite would actually be imported within the LayoutTests folder and run as usual webkit tests. The tests with absolute paths would be run offline by people wanting to gather conformance statistics or to bump the imported W3C test suite version and update related test expectations. 1035594 6 ap 2014-09-16 14:47:54 -0700 I see, that makes more sense to me. Not quite sure why the tree needs to remain clean when doing something like this though - it's easy to revert changes once done. 1035885 7 youennf 2014-09-18 02:50:57 -0700 (In reply to comment #6) > I see, that makes more sense to me. Not quite sure why the tree needs to remain clean when doing something like this though - it's easy to revert changes once done. Running absolute path tests also ensures that all generated files (expected.txt, crash logs, stderr, diff) are located in the same folder as the tests. That makes it a tad easier to compute the conformance results and generate self-contained reports. See http://youennf.github.io/w3c-reports/12092014/w3c_conformance_results.html for instance: the report links each test to its relevant .txt file. We could as well run the tests in a git/svn-ignored LayoutTests sub-folder. The conformance result computation and report generation would probably be somewhat more complex. 1036369 8 238168 commit-queue 2014-09-21 13:30:22 -0700 Comment on attachment 238168 Patch Clearing flags on attachment: 238168 Committed r173798: <http://trac.webkit.org/changeset/173798> 1036370 9 commit-queue 2014-09-21 13:30:24 -0700 All reviewed patches have been landed. Closing bug. 238168 2014-09-16 02:55:50 -0700 2014-09-21 13:30:21 -0700 Patch bug-136848-20140916115546.patch text/plain 1686 youennf U3VidmVyc2lvbiBSZXZpc2lvbjogMTczNjE0CmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggNjAxYjEwMjM2OGU4YzVmZGE3NWZiZjI1NjQwOWYxYzJj NjVlZmFiNy4uMjEyNDk2ODM3NzMyOGZmZmU0YWRkOGM0NmUzZTUzOWViY2Y0ODRjNCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDEz IEBACisyMDE0LTA5LTE2ICBZb3Vlbm4gRmFibGV0ICA8eW91ZW5uLmZhYmxldEBjcmYuY2Fub24u ZnI+CisKKyAgICAgICAgcnVuLXdlYmtpdC10ZXN0cyBzaG91bGQgY291bnQgdGVzdHMgc3VibWl0 dGVkIGFzIGFic29sdXRlIHBhdGhzIG9uY2UKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5v cmcvc2hvd19idWcuY2dpP2lkPTEzNjg0OAorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAo T09QUyEpLgorCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS9wb3J0L2Jhc2UucHk6CisgICAg ICAgIChQb3J0Ll9leHBhbmRlZF9wYXRocyk6IEluIGNhc2Ugb2YgYWJzb2x1dGUgcGF0aCwgZG8g bm90IHNlYXJjaCBmb3IgcGxhdGZvcm0gc3BlY2lmaWMgdGVzdHMuCisKIDIwMTQtMDktMTIgIEJy aWFuIEouIEJ1cmcgIDxidXJnQGNzLndhc2hpbmd0b24uZWR1PgogCiAgICAgICAgIFdlYiBJbnNw ZWN0b3I6IHJlbWluZCBhYm91dCByZWJhc2VsaW5pbmcgZ2VuZXJhdG9yIHRlc3QgcmVzdWx0cwpk aWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS9wb3J0L2Jhc2UucHkgYi9Ub29scy9T Y3JpcHRzL3dlYmtpdHB5L3BvcnQvYmFzZS5weQppbmRleCBhNGI2YWQxYzA0YjU2NjVmOTM5NzYw NjZlODg1ZDQxOWMxYjExYmNiLi42NjU2MzQxNTk0NDY3NDI5NmRhOTQxNDA5MDFlZTc2MWEwMWQ0 NzU4IDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BvcnQvYmFzZS5weQorKysg Yi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3BvcnQvYmFzZS5weQpAQCAtNTM1LDcgKzUzNSw3IEBA IGNsYXNzIFBvcnQob2JqZWN0KToKICAgICAgICAgYWxsX3BsYXRmb3JtX2RpcnMgPSBbcGF0aCBm b3IgcGF0aCBpbiBmcy5nbG9iKGZzLmpvaW4oc2VsZi5sYXlvdXRfdGVzdHNfZGlyKCksICdwbGF0 Zm9ybScsICcqJykpIGlmIGZzLmlzZGlyKHBhdGgpXQogICAgICAgICBmb3IgcGF0aCBpbiBwYXRo czoKICAgICAgICAgICAgIGV4cGFuZGVkX3BhdGhzLmFwcGVuZChwYXRoKQotICAgICAgICAgICAg aWYgc2VsZi50ZXN0X2lzZGlyKHBhdGgpIGFuZCBub3QgcGF0aC5zdGFydHN3aXRoKCdwbGF0Zm9y bScpOgorICAgICAgICAgICAgaWYgc2VsZi50ZXN0X2lzZGlyKHBhdGgpIGFuZCBub3QgcGF0aC5z dGFydHN3aXRoKCdwbGF0Zm9ybScpIGFuZCBub3QgZnMuaXNhYnMocGF0aCk6CiAgICAgICAgICAg ICAgICAgZm9yIHBsYXRmb3JtX2RpciBpbiBhbGxfcGxhdGZvcm1fZGlyczoKICAgICAgICAgICAg ICAgICAgICAgaWYgZnMuaXNkaXIoZnMuam9pbihwbGF0Zm9ybV9kaXIsIHBhdGgpKSBhbmQgcGxh dGZvcm1fZGlyIGluIHNlbGYuYmFzZWxpbmVfc2VhcmNoX3BhdGgoKToKICAgICAgICAgICAgICAg ICAgICAgICAgIGV4cGFuZGVkX3BhdGhzLmFwcGVuZChzZWxmLnJlbGF0aXZlX3Rlc3RfZmlsZW5h bWUoZnMuam9pbihwbGF0Zm9ybV9kaXIsIHBhdGgpKSkK