186261 2018-06-03 23:24:53 -0700 import-w3c-tests should rely on <meta name="flags"> to detect CSS manual tests 2018-06-05 02:19:27 -0700 1 1 1 Unclassified WebKit Tools / Tests WebKit Nightly Build Unspecified Unspecified RESOLVED FIXED https://github.com/web-platform-tests/wpt/issues/5381 InRadar P2 Normal --- 1 fred.wang fred.wang ews-watchlist glenn lforschler Ms2ger rego webkit-bug-importer youennf oldest_to_newest 1429775 0 fred.wang 2018-06-03 23:24:53 -0700 CSS tests have additional flags, which can be used to indicate that the tests are manual: https://web-platform-tests.org/writing-tests/css-metadata.html#requirement-flags From WPT's tools/manifest/sourcefile.py: def content_is_css_manual(self): """Boolean indicating whether the file content represents a CSS WG-style manual test""" if self.root is None: return None # return True if the intersection between the two sets is non-empty return bool(self.css_flags & {"animated", "font", "history", "interact", "paged", "speech", "userstyle"}) Currently, import-w3c-tests just treat them as "support" files, instead of ignoring them. 1429779 1 341895 fred.wang 2018-06-04 00:12:01 -0700 Created attachment 341895 Patch 1429785 2 Ms2ger 2018-06-04 00:44:51 -0700 LGTM 1429979 3 341895 youennf 2018-06-04 17:56:21 -0700 Comment on attachment 341895 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=341895&action=review > Tools/ChangeLog:14 > + <meta name="flags"> are manual are non-manual according to CSS WG rules. s/are/or/ > Tools/Scripts/webkitpy/w3c/test_parser.py:119 > + elif self.is_wpt_manualtest() or self.is_css_manualtest(): We should probably merge is_wpt_manualtest and is_css_manualtest > Tools/Scripts/webkitpy/w3c/test_parser_unittest.py:183 > + self.assertEqual(test_info, None, 'test_info is not None') Maybe we should add asis which is a valid css flag but not classified as manual. > Tools/Scripts/webkitpy/w3c/test_parser_unittest.py:194 > + self.assertTrue(test_info['manualtest'], 'test with CSS flag %s is not manual' % flag) I never know whether the assertion message tells what is expected or what is unexpected. I would rewrite it to 'test with CSS flag %s should be manual' 1430064 4 341895 fred.wang 2018-06-05 02:13:31 -0700 Comment on attachment 341895 Patch View in context: https://bugs.webkit.org/attachment.cgi?id=341895&action=review >> Tools/ChangeLog:14 >> + <meta name="flags"> are manual are non-manual according to CSS WG rules. > > s/are/or/ Done >> Tools/Scripts/webkitpy/w3c/test_parser.py:119 >> + elif self.is_wpt_manualtest() or self.is_css_manualtest(): > > We should probably merge is_wpt_manualtest and is_css_manualtest Done >> Tools/Scripts/webkitpy/w3c/test_parser_unittest.py:183 >> + self.assertEqual(test_info, None, 'test_info is not None') > > Maybe we should add asis which is a valid css flag but not classified as manual. Done. >> Tools/Scripts/webkitpy/w3c/test_parser_unittest.py:194 >> + self.assertTrue(test_info['manualtest'], 'test with CSS flag %s is not manual' % flag) > > I never know whether the assertion message tells what is expected or what is unexpected. > I would rewrite it to 'test with CSS flag %s should be manual' Yeah, I was also confused by existing tests. Done here and as well above "test_info should be None" 1430065 5 fred.wang 2018-06-05 02:18:16 -0700 Committed r232506: <https://trac.webkit.org/changeset/232506> 1430066 6 webkit-bug-importer 2018-06-05 02:19:27 -0700 <rdar://problem/40800382> 341895 2018-06-04 00:12:01 -0700 2018-06-04 17:56:21 -0700 Patch bug-186261-20180604091200.patch text/plain 4828 fred.wang U3VidmVyc2lvbiBSZXZpc2lvbjogMjMyNDYxCmRpZmYgLS1naXQgYS9Ub29scy9DaGFuZ2VMb2cg Yi9Ub29scy9DaGFuZ2VMb2cKaW5kZXggYTJjZmIwZDY0YjYxMjliOGU5YzI3ZDFlM2RlYjkzMjJk MjgzZDZjNC4uMWQ5M2ViYTI2YmFiNjYxOGJiYjA1N2U3OTFmMzUwZjFmMGRmOTBkOCAxMDA2NDQK LS0tIGEvVG9vbHMvQ2hhbmdlTG9nCisrKyBiL1Rvb2xzL0NoYW5nZUxvZwpAQCAtMSwzICsxLDE4 IEBACisyMDE4LTA2LTA0ICBGcmVkZXJpYyBXYW5nICA8ZndhbmdAaWdhbGlhLmNvbT4KKworICAg ICAgICBpbXBvcnQtdzNjLXRlc3RzIHNob3VsZCByZWx5IG9uIDxtZXRhIG5hbWU9ImZsYWdzIj4g dG8gZGV0ZWN0IENTUyBtYW51YWwgdGVzdHMKKyAgICAgICAgaHR0cHM6Ly9idWdzLndlYmtpdC5v cmcvc2hvd19idWcuY2dpP2lkPTE4NjI2MQorCisgICAgICAgIFJldmlld2VkIGJ5IE5PQk9EWSAo T09QUyEpLgorCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS93M2MvdGVzdF9wYXJzZXIucHk6 CisgICAgICAgIChUZXN0UGFyc2VyLmFuYWx5emVfdGVzdCk6IEFsc28gY29uc2lkZXIgbWFudWFs IHRlc3RzIHBlciBDU1MgV0cgcnVsZXMuCisgICAgICAgIChUZXN0UGFyc2VyLmlzX2Nzc19tYW51 YWx0ZXN0KTogSGVscGVyIGZ1bmN0aW9uIHRvIGRldGVjdCB3aGV0aGVyIGEgZmlsZSBpcyBhIG1h bnVhbCB0ZXN0CisgICAgICAgIGZyb20gaXRzIDxtZXRhIG5hbWU9ImZsYWdzIj4gdGFncy4gVGhp cyBpcyBiYXNlZCBvbiBTb3VyY2VGaWxlOjpjb250ZW50X2lzX2Nzc19tYW51YWwgZnJvbQorICAg ICAgICB0aGUgV1BUIHJlcG9zaXRvcnkuCisgICAgICAgICogU2NyaXB0cy93ZWJraXRweS93M2Mv dGVzdF9wYXJzZXJfdW5pdHRlc3QucHk6IEFkZCBhIHRlc3QgdG8gdmVyaWZ5IHdoZXRoZXIgZmls ZXMgd2l0aAorICAgICAgICA8bWV0YSBuYW1lPSJmbGFncyI+IGFyZSBtYW51YWwgYXJlIG5vbi1t YW51YWwgYWNjb3JkaW5nIHRvIENTUyBXRyBydWxlcy4KKwogMjAxOC0wNi0wMyAgRnVqaWkgSGly b25vcmkgIDxIaXJvbm9yaS5GdWppaUBzb255LmNvbT4KIAogICAgICAgICBbV2luXVtNaW5pQnJv d3Nlcl0gUmVtb3ZlIGdNaW5pQnJvd3NlciBnbG9iYWwgdmFyaWFibGUKZGlmZiAtLWdpdCBhL1Rv b2xzL1NjcmlwdHMvd2Via2l0cHkvdzNjL3Rlc3RfcGFyc2VyLnB5IGIvVG9vbHMvU2NyaXB0cy93 ZWJraXRweS93M2MvdGVzdF9wYXJzZXIucHkKaW5kZXggNzE5Y2I4ZmFiNDJhNGVlYTk0NWExNjhh MDVlZjA3NTUxMDdhNjIyMi4uYzg1YTViMTM3ZDBiZDBlYTA4YjFhMTA5OTE4NWM2Yzg2OTJiYWRm MyAxMDA2NDQKLS0tIGEvVG9vbHMvU2NyaXB0cy93ZWJraXRweS93M2MvdGVzdF9wYXJzZXIucHkK KysrIGIvVG9vbHMvU2NyaXB0cy93ZWJraXRweS93M2MvdGVzdF9wYXJzZXIucHkKQEAgLTExNiw3 ICsxMTYsNyBAQCBjbGFzcyBUZXN0UGFyc2VyKG9iamVjdCk6CiAgICAgICAgICAgICB0ZXN0X2lu Zm8gPSB7J3Rlc3QnOiBzZWxmLmZpbGVuYW1lLCAncmVmZXJlbmNlJzogc2VsZi5wb3RlbnRpYWxf cmVmX2ZpbGVuYW1lKCl9CiAgICAgICAgICAgICB0ZXN0X2luZm9bJ3JlZmVyZW5jZV9zdXBwb3J0 X2luZm8nXSA9IHt9CiAgICAgICAgICMgd2UgY2hlY2sgZm9yIHdwdCBtYW51YWwgdGVzdCBiZWZv cmUgY2hlY2tpbmcgZm9yIGpzdGVzdCwgYXMgc29tZSBXUFQgbWFudWFsIHRlc3RzIGNhbiBiZSBj bGFzc2lmaWVkIGFzIENTUyBKUyB0ZXN0cwotICAgICAgICBlbGlmIHNlbGYuaXNfd3B0X21hbnVh bHRlc3QoKToKKyAgICAgICAgZWxpZiBzZWxmLmlzX3dwdF9tYW51YWx0ZXN0KCkgb3Igc2VsZi5p c19jc3NfbWFudWFsdGVzdCgpOgogICAgICAgICAgICAgdGVzdF9pbmZvID0geyd0ZXN0Jzogc2Vs Zi5maWxlbmFtZSwgJ21hbnVhbHRlc3QnOiBUcnVlfQogICAgICAgICBlbGlmIHNlbGYuaXNfanN0 ZXN0KCk6CiAgICAgICAgICAgICB0ZXN0X2luZm8gPSB7J3Rlc3QnOiBzZWxmLmZpbGVuYW1lLCAn anN0ZXN0JzogVHJ1ZX0KQEAgLTE0MSw2ICsxNDEsMTggQEAgY2xhc3MgVGVzdFBhcnNlcihvYmpl Y3QpOgogICAgICAgICAiIiJSZXR1cm5zIHdoZXRoZXIgdGhlIHRlc3QgaXMgYSBtYW51YWwgdGVz dCBhY2NvcmRpbmcgV1BUIHJ1bGVzIChpLmUuIGZpbGUgZW5kcyB3aXRoIC1tYW51YWwuaHRtIHBh dGgpLiIiIgogICAgICAgICByZXR1cm4gc2VsZi5maWxlbmFtZS5lbmRzd2l0aCgnLW1hbnVhbC5o dG0nKSBvciBzZWxmLmZpbGVuYW1lLmVuZHN3aXRoKCctbWFudWFsLmh0bWwnKQogCisgICAgZGVm IGlzX2Nzc19tYW51YWx0ZXN0KHNlbGYpOgorICAgICAgICAiIiJSZXR1cm5zIHdoZXRoZXIgdGhl IHRlc3QgaXMgYSBtYW51YWwgdGVzdCBhY2NvcmRpbmcgdG8gQ1NTIFdHLXN0eWxlIHJ1bGVzLiIi IgorICAgICAgICAjIEZvciBmdXJ0aGVyIGRldGFpbHMgYW5kIGRpc2N1c3Npb25zLCBzZWUgdGhl IGZvbGxvd2luZyBsaW5rczoKKyAgICAgICAgIyBodHRwczovL3dlYi1wbGF0Zm9ybS10ZXN0cy5v cmcvd3JpdGluZy10ZXN0cy9jc3MtbWV0YWRhdGEuaHRtbCNyZXF1aXJlbWVudC1mbGFncworICAg ICAgICAjIGh0dHBzOi8vZ2l0aHViLmNvbS93ZWItcGxhdGZvcm0tdGVzdHMvd3B0L2lzc3Vlcy81 MzgxCisgICAgICAgICMgaHR0cHM6Ly9naXRodWIuY29tL3dlYi1wbGF0Zm9ybS10ZXN0cy93cHQv aXNzdWVzLzUyOTMKKyAgICAgICAgZm9yIG1hdGNoIGluIHNlbGYudGVzdF9kb2MuZmluZEFsbChu YW1lPSdtZXRhJywgYXR0cnM9eyduYW1lJzogJ2ZsYWdzJywgJ2NvbnRlbnQnOiBUcnVlfSk6Cisg ICAgICAgICAgICBjc3NfZmxhZ3MgPSBzZXQobWF0Y2hbJ2NvbnRlbnQnXS5zcGxpdCgpKQorICAg ICAgICAgICAgaWYgYm9vbChjc3NfZmxhZ3MgJiB7ImFuaW1hdGVkIiwgImZvbnQiLCAiaGlzdG9y eSIsICJpbnRlcmFjdCIsICJwYWdlZCIsICJzcGVlY2giLCAidXNlcnN0eWxlIn0pOgorICAgICAg ICAgICAgICAgIHJldHVybiBUcnVlCisgICAgICAgIHJldHVybiBGYWxzZQorCiAgICAgZGVmIGlz X3Nsb3dfdGVzdChzZWxmKToKICAgICAgICAgcmV0dXJuIGFueShbbWF0Y2gubmFtZSA9PSAnbWV0 YScgYW5kIG1hdGNoWyduYW1lJ10gPT0gJ3RpbWVvdXQnIGZvciBtYXRjaCBpbiBzZWxmLnRlc3Rf ZG9jLmZpbmRBbGwoY29udGVudD0nbG9uZycpXSkKIApkaWZmIC0tZ2l0IGEvVG9vbHMvU2NyaXB0 cy93ZWJraXRweS93M2MvdGVzdF9wYXJzZXJfdW5pdHRlc3QucHkgYi9Ub29scy9TY3JpcHRzL3dl YmtpdHB5L3czYy90ZXN0X3BhcnNlcl91bml0dGVzdC5weQppbmRleCA1ODBhOTkxZDU0OTBkOTU4 MGRhMGQ1NWZmZjkwYjcyMGVjMDg4NmQxLi5hNWFiNmQ0NWUyMjI0ZjEwOThmZTcwNmEzNzAyZDZk MDQ3NmRmM2FhIDEwMDY0NAotLS0gYS9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3czYy90ZXN0X3Bh cnNlcl91bml0dGVzdC5weQorKysgYi9Ub29scy9TY3JpcHRzL3dlYmtpdHB5L3czYy90ZXN0X3Bh cnNlcl91bml0dGVzdC5weQpAQCAtMTY4LDYgKzE2OCwzMSBAQCBjbGFzcyBUZXN0UGFyc2VyVGVz dCh1bml0dGVzdC5UZXN0Q2FzZSk6CiAKICAgICAgICAgc2VsZi5hc3NlcnRUcnVlKHRlc3RfaW5m b1snbWFudWFsdGVzdCddLCAndGVzdF9pbmZvIGlzIE5vbmUnKQogCisgICAgZGVmIHRlc3RfYW5h bHl6ZV9jc3NfbWFudWFsX3Rlc3Qoc2VsZik6CisgICAgICAgICIiIiBUZXN0cyBhbmFseXplX3Rl c3QoKSB1c2luZyBhIGNzcyBtYW51YWwgdGVzdCAiIiIKKworICAgICAgICB0ZXN0X3BhdGggPSBv cy5wYXRoLmpvaW4ob3MucGF0aC5zZXAsICdzb21lJywgJ21hZGV1cCcsICdwYXRoJykKKyAgICAg ICAgcGFyc2VyID0gVGVzdFBhcnNlcihvcHRpb25zLCBvcy5wYXRoLmpvaW4odGVzdF9wYXRoLCAn c29tZWZpbGUuaHRtbCcpKQorCisgICAgICAgIGZvciBjb250ZW50IGluIFsiIiwgImZsYWcxIiwg ImZsYWcxIGZsYWcyIiwgImZsYWcxIGZsYWcyIGZsYWczIl06CisgICAgICAgICAgICB0ZXN0X2h0 bWwgPSAiIiIKKzxoZWFkPgorICA8bWV0YSBuYW1lPSJmbGFncyIgY29udGVudD0iJXMiPgorPC9o ZWFkPiIiIiAlIGNvbnRlbnQKKyAgICAgICAgICAgIHRlc3RfaW5mbyA9IHBhcnNlci5hbmFseXpl X3Rlc3QodGVzdF9jb250ZW50cz10ZXN0X2h0bWwpCisgICAgICAgICAgICBzZWxmLmFzc2VydEVx dWFsKHRlc3RfaW5mbywgTm9uZSwgJ3Rlc3RfaW5mbyBpcyBub3QgTm9uZScpCisKKyAgICAgICAg Zm9yIGZsYWcgaW4gWyJhbmltYXRlZCIsICJmb250IiwgImhpc3RvcnkiLCAiaW50ZXJhY3QiLCAi cGFnZWQiLCAic3BlZWNoIiwgInVzZXJzdHlsZSJdOgorICAgICAgICAgICAgdGVzdF9odG1sID0g IiIiCis8aGVhZD4KKyAgPG1ldGEgbmFtZT0iZmxhZ3MiIGNvbnRlbnQ9ImZsYWcxIGZsYWcyIj4K KyAgPG1ldGEgbmFtZT0iZmxhZ3MiIGNvbnRlbnQ9ImZsYWczICVzIGZsYWc0IGZsYWc1Ij4KKyAg PG1ldGEgbmFtZT0iZmxhZ3MiIGNvbnRlbnQ9ImZsYWc2Ij4KKzwvaGVhZD4KKyIiIiAlIGZsYWcK KyAgICAgICAgICAgIHRlc3RfaW5mbyA9IHBhcnNlci5hbmFseXplX3Rlc3QodGVzdF9jb250ZW50 cz10ZXN0X2h0bWwpCisgICAgICAgICAgICBzZWxmLmFzc2VydFRydWUodGVzdF9pbmZvWydtYW51 YWx0ZXN0J10sICd0ZXN0IHdpdGggQ1NTIGZsYWcgJXMgaXMgbm90IG1hbnVhbCcgJSBmbGFnKQor CiAgICAgZGVmIHRlc3RfYW5hbHl6ZV9waXhlbF90ZXN0X2FsbF90cnVlKHNlbGYpOgogICAgICAg ICAiIiIgVGVzdHMgYW5hbHl6ZV90ZXN0KCkgdXNpbmcgYSB0ZXN0IHRoYXQgaXMgbmVpdGhlciBh IHJlZnRlc3Qgb3IganN0ZXN0IHdpdGggYWxsPUZhbHNlICIiIgogCg==