38457 2010-05-03 05:56:14 -0700 [Qt] Enable passing Sputnik tests 2010-05-06 08:50:30 -0700 1 1 1 Unclassified WebKit Tools / Tests 528+ (Nightly build) All All RESOLVED FIXED Qt P3 Normal --- 1 ossy webkit-qt-unassigned abecsi ap hausmann kenneth vestbo oldest_to_newest 220069 0 ossy 2010-05-03 05:56:14 -0700 Sputnik test cases landed 4 days before: (~5000 tests) https://bugs.webkit.org/show_bug.cgi?id=38296 I tested them on Qt bot, they are sufficiently stable, most of them pass, only ~50 tests fail. Runtime of LayoutTests would be increased by ~40 sec on the bot. (Now the runtime is ~240 sec.) 220070 1 54919 ossy 2010-05-03 05:56:47 -0700 Created attachment 54919 enable tests 220115 2 54919 ap 2010-05-03 08:15:56 -0700 Comment on attachment 54919 enable tests rs=me +# Failing Sputnik tests You could also consider landing platform specific results. Of course, these aren't all the tests that fail, we have many expected cross-platform failures checked in. 221521 3 ossy 2010-05-05 23:18:24 -0700 (In reply to comment #2) > (From update of attachment 54919 [details]) > rs=me Thx, landed: http://trac.webkit.org/changeset/58863 > You could also consider landing platform specific results. Of course, these > aren't all the tests that fail, we have many expected cross-platform failures > checked in. In general we don't land Qt specific expected results for failures, we prefer putting these tests into the skipped list and file a bug. I know disabling a complex test can be worse than check in couple of failures (eg. FAIL instead of PASS) into the expected file. But I don't see now how can we maintain which expected file is absolutely correct and which consists failures. We will think how can we do it. I cc-ed Simon, Tor Arne, Kenneth and András too. Guys, I think we should discuss check-in or not check-in failures policy sometime somewhere. 221601 4 abecsi 2010-05-06 03:16:58 -0700 If I am correct these tests are dumpAsText() tests, so my opinion is that we could check in results for tests which have any PASS hunks in the output. If someone fixes some issues he or she could easily decide if the resulting test failure is due to wrong expected results or some regression. In general I think this kind of policy would be a step forward, but I am categorically against checking in wrong DRT tree dumps, because to decide afterwards if the dump is correct or not is really hard. (In reply to comment #3) > (In reply to comment #2) > > (From update of attachment 54919 [details] [details]) > > rs=me > Thx, landed: http://trac.webkit.org/changeset/58863 > > > You could also consider landing platform specific results. Of course, these > > aren't all the tests that fail, we have many expected cross-platform failures > > checked in. > > In general we don't land Qt specific expected results for failures, > we prefer putting these tests into the skipped list and file a bug. > > I know disabling a complex test can be worse than check in couple of failures > (eg. FAIL instead of PASS) into the expected file. But I don't see now how can > we maintain which expected file is absolutely correct and which consists > failures. We will think how can we do it. > > I cc-ed Simon, Tor Arne, Kenneth and András too. Guys, I think we > should discuss check-in or not check-in failures policy sometime somewhere. 221604 5 vestbo 2010-05-06 03:29:10 -0700 (In reply to comment #4) > If I am correct these tests are dumpAsText() tests, so my opinion is that we > could check in results for tests which have any PASS hunks in the output. If > someone fixes some issues he or she could easily decide if the resulting test > failure is due to wrong expected results or some regression. > In general I think this kind of policy would be a step forward, but I am > categorically against checking in wrong DRT tree dumps, because to decide > afterwards if the dump is correct or not is really hard. I agree, for dumpAsText it makes sense. For render tree output we prefer to skip tests for now. 221706 6 ap 2010-05-06 08:50:30 -0700 Sputnik tests are weird - they silently run subtests until the first failure, at which time they bail out and print an error message. Only if there were no (unexpected) errors during the run, PASS is printed. 54919 2010-05-03 05:56:47 -0700 2010-05-03 08:15:55 -0700 enable tests sputnik.patch text/plain 3287 ossy ZGlmZiAtLWdpdCBhL0xheW91dFRlc3RzL0NoYW5nZUxvZyBiL0xheW91dFRlc3RzL0NoYW5nZUxv ZwppbmRleCA3MWZjZWVhLi5hMzUwYjRmIDEwMDY0NAotLS0gYS9MYXlvdXRUZXN0cy9DaGFuZ2VM b2cKKysrIGIvTGF5b3V0VGVzdHMvQ2hhbmdlTG9nCkBAIC0xLDMgKzEsMTEgQEAKKzIwMTAtMDUt MDMgIENzYWJhIE9zenRyb2dvbsOhYyAgPG9zc3lAd2Via2l0Lm9yZz4KKworICAgICAgICBSZXZp ZXdlZCBieSBOT0JPRFkgKE9PUFMhKS4KKworICAgICAgICBbUXRdIEVuYWJsZSBTcHV0bmlrIHRl c3RzLgorCisgICAgICAgICogcGxhdGZvcm0vcXQvU2tpcHBlZDogT25seSBza2lwIHRlc3RzIHRo YXQgZmFpbC4KKwogMjAxMC0wNC0zMCAgWW9zaGlraSBIYXlhc2hpICA8eWhheWFzaGlAZ29vZ2xl LmNvbT4KIAogICAgICAgICBSZXZpZXdlZCBieSBTaGluaWNoaXJvIEhhbWFqaS4KZGlmZiAtLWdp dCBhL0xheW91dFRlc3RzL3BsYXRmb3JtL3F0L1NraXBwZWQgYi9MYXlvdXRUZXN0cy9wbGF0Zm9y bS9xdC9Ta2lwcGVkCmluZGV4IDI1MDM5ZmQuLjM5M2JhZTEgMTAwNjQ0Ci0tLSBhL0xheW91dFRl c3RzL3BsYXRmb3JtL3F0L1NraXBwZWQKKysrIGIvTGF5b3V0VGVzdHMvcGxhdGZvcm0vcXQvU2tp cHBlZApAQCAtNTIzMSw4ICs1MjMxLDQ0IEBAIHN2Zy9jdXN0b20vbWFya2VyLW92ZXJmbG93LWNs aXAuc3ZnCiAjIERvZXNuJ3Qgc3VwcG9ydCBXT0ZGIHlldC4KIGZhc3QvY3NzL2ZvbnQtZmFjZS13 b2ZmLmh0bWwKIAotIyBtdWx0aXBsZSB0ZXN0cyBmYWlsIG9uIGRpZmZlcmVudCBwbGF0Zm9ybXMs IG5lZWQgdG8gZmluZCB0aGUgY2F1c2UocykKLWZhc3QvanMvc3B1dG5pawotCiAjIGh0dHBzOi8v YnVncy53ZWJraXQub3JnL3Nob3dfYnVnLmNnaT9pZD0zODM3NgogbWVkaWEvbWVkaWEtZG9jdW1l bnQtYXVkaW8tc2l6ZS5odG1sCisKKyMgRmFpbGluZyBTcHV0bmlrIHRlc3RzCitmYXN0L2pzL3Nw dXRuaWsvQ29uZm9ybWFuY2UvMTBfRXhlY3V0aW9uX0NvbnRleHRzLzEwLjJfRW50ZXJpbmdfQW5f RXhlY3V0aW9uX0NvbnRleHQvMTAuMi4yX0V2YWxfQ29kZQorZmFzdC9qcy9zcHV0bmlrL0NvbmZv cm1hbmNlLzE1X05hdGl2ZV9PYmplY3RzLzE1LjRfQXJyYXkvMTUuNC40LzE1LjQuNC4xMl9BcnJh eV9wcm90b3R5cGVfc3BsaWNlL1MxNS40LjQuMTJfQTIuMV9UMy5odG1sCitmYXN0L2pzL3NwdXRu aWsvQ29uZm9ybWFuY2UvMTVfTmF0aXZlX09iamVjdHMvMTUuOV9EYXRlLzE1LjkuMy9TMTUuOS4z LjFfQTVfVDEuaHRtbAorZmFzdC9qcy9zcHV0bmlrL0NvbmZvcm1hbmNlLzE1X05hdGl2ZV9PYmpl Y3RzLzE1LjlfRGF0ZS8xNS45LjMvUzE1LjkuMy4xX0E1X1QyLmh0bWwKK2Zhc3QvanMvc3B1dG5p ay9Db25mb3JtYW5jZS8xNV9OYXRpdmVfT2JqZWN0cy8xNS45X0RhdGUvMTUuOS4zL1MxNS45LjMu MV9BNV9UMy5odG1sCitmYXN0L2pzL3NwdXRuaWsvQ29uZm9ybWFuY2UvMTVfTmF0aXZlX09iamVj dHMvMTUuOV9EYXRlLzE1LjkuMy9TMTUuOS4zLjFfQTVfVDQuaHRtbAorZmFzdC9qcy9zcHV0bmlr L0NvbmZvcm1hbmNlLzE1X05hdGl2ZV9PYmplY3RzLzE1LjlfRGF0ZS8xNS45LjMvUzE1LjkuMy4x X0E1X1Q1Lmh0bWwKK2Zhc3QvanMvc3B1dG5pay9Db25mb3JtYW5jZS8xNV9OYXRpdmVfT2JqZWN0 cy8xNS45X0RhdGUvMTUuOS4zL1MxNS45LjMuMV9BNV9UNi5odG1sCitmYXN0L2pzL3NwdXRuaWsv VW5pY29kZS9Vbmljb2RlXzUwMC9TNy42X0EzLjEuaHRtbAorZmFzdC9qcy9zcHV0bmlrL1VuaWNv ZGUvVW5pY29kZV81MDAvUzcuNl9BMy4yLmh0bWwKK2Zhc3QvanMvc3B1dG5pay9Vbmljb2RlL1Vu aWNvZGVfNTAwL1M3LjZfQTUuM19UMS5odG1sCitmYXN0L2pzL3NwdXRuaWsvVW5pY29kZS9Vbmlj b2RlXzUwMC9TNy42X0E1LjNfVDIuaHRtbAorZmFzdC9qcy9zcHV0bmlrL1VuaWNvZGUvVW5pY29k ZV81MTAvUzE1LjUuNC4xNl9BMS5odG1sCitmYXN0L2pzL3NwdXRuaWsvVW5pY29kZS9Vbmljb2Rl XzUxMC9TMTUuNS40LjE4X0ExLmh0bWwKK2Zhc3QvanMvc3B1dG5pay9Vbmljb2RlL1VuaWNvZGVf NTEwL1M3LjZfQTEuMV9UMS5odG1sCitmYXN0L2pzL3NwdXRuaWsvVW5pY29kZS9Vbmljb2RlXzUx MC9TNy42X0ExLjFfVDIuaHRtbAorZmFzdC9qcy9zcHV0bmlrL1VuaWNvZGUvVW5pY29kZV81MTAv UzcuNl9BMS4xX1Q0Lmh0bWwKK2Zhc3QvanMvc3B1dG5pay9Vbmljb2RlL1VuaWNvZGVfNTEwL1M3 LjZfQTIuMl9UMS5odG1sCitmYXN0L2pzL3NwdXRuaWsvVW5pY29kZS9Vbmljb2RlXzUxMC9TNy42 X0EyLjJfVDIuaHRtbAorZmFzdC9qcy9zcHV0bmlrL1VuaWNvZGUvVW5pY29kZV81MTAvUzcuNl9B Mi4zLmh0bWwKK2Zhc3QvanMvc3B1dG5pay9Vbmljb2RlL1VuaWNvZGVfNTEwL1M3LjZfQTMuMS5o dG1sCitmYXN0L2pzL3NwdXRuaWsvVW5pY29kZS9Vbmljb2RlXzUxMC9TNy42X0EzLjIuaHRtbAor ZmFzdC9qcy9zcHV0bmlrL1VuaWNvZGUvVW5pY29kZV81MTAvUzcuNl9BNS4yX1QxLmh0bWwKK2Zh c3QvanMvc3B1dG5pay9Vbmljb2RlL1VuaWNvZGVfNTEwL1M3LjZfQTUuMl9UMi5odG1sCitmYXN0 L2pzL3NwdXRuaWsvVW5pY29kZS9Vbmljb2RlXzUxMC9TNy42X0E1LjJfVDQuaHRtbAorZmFzdC9q cy9zcHV0bmlrL1VuaWNvZGUvVW5pY29kZV81MTAvUzcuNl9BNS4yX1Q3Lmh0bWwKK2Zhc3QvanMv c3B1dG5pay9Vbmljb2RlL1VuaWNvZGVfNTEwL1M3LjZfQTUuMl9UOC5odG1sCitmYXN0L2pzL3Nw dXRuaWsvVW5pY29kZS9Vbmljb2RlXzUxMC9TNy42X0E1LjJfVDkuaHRtbAorZmFzdC9qcy9zcHV0 bmlrL1VuaWNvZGUvVW5pY29kZV81MTAvUzcuNl9BNS4zX1QxLmh0bWwKK2Zhc3QvanMvc3B1dG5p ay9Vbmljb2RlL1VuaWNvZGVfNTEwL1M3LjZfQTUuM19UMi5odG1sCisKKyMgRmFpbGluZyBTcHV0 bmlrIHRlc3RzIG9uIDMyIGJpdAorZmFzdC9qcy9zcHV0bmlrL0NvbmZvcm1hbmNlLzE1X05hdGl2 ZV9PYmplY3RzLzE1LjhfTWF0aC8xNS44LjIvMTUuOC4yLjE2X3Npbi9TMTUuOC4yLjE2X0E3Lmh0 bWwKK2Zhc3QvanMvc3B1dG5pay9Db25mb3JtYW5jZS8xNV9OYXRpdmVfT2JqZWN0cy8xNS44X01h dGgvMTUuOC4yLzE1LjguMi4xOF90YW4vUzE1LjguMi4xOF9BNy5odG1sCisKKyMgRmFpbGluZyBT cHV0bmlrIHRlc3Qgb24gNjQgYml0CitmYXN0L2pzL3NwdXRuaWsvQ29uZm9ybWFuY2UvMTFfRXhw cmVzc2lvbnMvMTEuNV9NdWx0aXBsaWNhdGl2ZV9PcGVyYXRvcnMvMTEuNS4zX0FwcGx5aW5nX3Ro ZV9wZXJjZW50X09wZXJhdG9yL1MxMS41LjNfQTRfVDIuaHRtbAo=