179307 2017-11-05 16:38:29 -0800 WebKit treats Big5-HKSCS as a distinct encoding from Big5, Encoding standard says it's the same 2022-09-27 06:44:37 -0700 1 1 1 Unclassified WebKit Text Safari Technology Preview Unspecified Unspecified RESOLVED DUPLICATE 216016 P2 Normal --- 179303 1 mjs webkit-unassigned achristensen addison annevk ap darin mmaxfield oldest_to_newest 1368502 0 mjs 2017-11-05 16:38:29 -0800 WebKit treats Big5-HKSCS as a distinct encoding from Big5, but the Encoding standard says it's the same. Chrome and Firefox report Big5 as the canonical name when using the TextDecoder API. It's not clear to me if they actually decode it differently though, I am not sure how to make a test for that. 1368519 1 mjs 2017-11-05 18:29:18 -0800 Here's some past revisions that may explain why we have this behavior (pointed out by Darin): https://trac.webkit.org/changeset/3611/webkit We changed to treat all Big5 as an alias for the Windows version (like the latest Encoding spec does) https://trac.webkit.org/changeset/4054/webkit We changed to treat most Big5 character sets as Big5_HKSCS_1999, unless they were explicitly Microsoft-specific. https://trac.webkit.org/changeset/4689/webkit We changed to treat most Big5 character sets as the DOS/Windows version, but left Big5-HKSCS alone. It's not totally clear why Big5-HKSCS was left alone in that last change. I don't think this is compatible with other browsers do, so we should probably abandon this direction. But I need to make some tests. 1368525 2 ap 2017-11-05 19:32:48 -0800 Big5 is a large family of standards governed by various entities, and we basically never got to check if ICU supported the variant(s) that other browsers used. This is likely moot now, as Chrome also uses ICU. 1368532 3 mjs 2017-11-05 20:36:22 -0800 These are our differences from the standard on Big5-related encodings: MISMATCH: encoding big5-hkscs is Big5 in the standard, but Big5-HKSCS in WebKit EXTRA NAME: WebKit knows extra nonstandard name x-windows-950 for Big5 EXTRA NAME: WebKit knows extra nonstandard name windows-950 for Big5 EXTRA NAME: WebKit knows extra nonstandard name x-big5 for Big5 EXTRA NAME: WebKit knows extra nonstandard name ms950 for Big5 EXTRA NAME: WebKit knows extra nonstandard name windows-950-2000 for Big5 EXTRA ENCODING: WebKit knows nonstandard encoding Big5-HKSCS with names ['big5-hkscs', 'big5hk', 'hkscs-big5', 'ibm-1375', 'ibm-1375_p100-2008'] 1368533 4 326098 mjs 2017-11-05 20:41:52 -0800 Created attachment 326098 Test case for (lack of) WebKit's Big5 quirks, meant to go in LayoutTests/fast/encodings This test case gives exactly the spec-mandated results for Firefox and Chrome. They both have the exact spec behavior. Safari has the differences described above. 1368536 5 mjs 2017-11-05 20:58:37 -0800 Here's the Gecko bug from when they did the merge: https://bugzilla.mozilla.org/show_bug.cgi?id=912470 It seems like their Big5 supports HKSCS character sequences. But I'm not sure if that's the same as our Big5-HKSCS or something that's a larger of that and Windows-flavord Big5. 1368543 6 mjs 2017-11-05 21:56:40 -0800 Based on http://w3c-test.org/encoding/big5-encoder.html , it doesn't look like either Big5 or Big5_HKSCS encodings from ICU quite match what the Encoding standard requires, and their failures are not the same either, so merging down to one of the two is bound to cause bugs. We might need a custom Big5 codec. ICU seems to support several apparent Big5 variants: ibm-1373_P100-2002 windows-950-2000 ibm-950_P110-1999 ibm-1375_P100-2008 ibm-5471_P100-2006 I'm not sure if any of these are the proper web variant. 1649588 7 annevk 2020-05-06 07:11:42 -0700 *** Bug 159890 has been marked as a duplicate of this bug. *** 1901469 8 annevk 2022-09-27 06:27:24 -0700 According to https://wpt.fyi/results/encoding?label=master&label=experimental&aligned&view=subtest&q=big5 we pass all the tests so this was fixed at some point. Probably by Alex? 1901480 9 annevk 2022-09-27 06:44:37 -0700 Confirmed: https://github.com/WebKit/WebKit/commit/70a5c3285eca476faa66c6e6055d615c26c78fc4 *** This bug has been marked as a duplicate of bug 216016 *** 326098 2017-11-05 20:41:52 -0800 2017-11-05 20:41:52 -0800 Test case for (lack of) WebKit's Big5 quirks, meant to go in LayoutTests/fast/encodings big5-encodings.html text/html 853 mjs PCFET0NUWVBFIGh0bWw+CjxodG1sPgo8aGVhZD4KPG1ldGEgY2hhcnNldD0idXRmLTgiPgo8c2Ny aXB0IHNyYz0iLi4vLi4vcmVzb3VyY2VzL2pzLXRlc3QtcHJlLmpzIj48L3NjcmlwdD4KPC9oZWFk Pgo8Ym9keT4KPHNjcmlwdD4KZGVzY3JpcHRpb24oIlRoaXMgdGVzdCBjaGVja3Mgc3VwcG9ydCBm b3IgdmFyaW91cyBCaWc1IHRleHQgZW5jb2RpbmdzLiIpOwoKbGV0IGJpZzVFbmNvZGluZ3MgPSBb ImJpZzUiLCAiYmlnNS1oa3NjcyIsICJjbi1iaWc1IiwgImNzYmlnNSIsICJ4LXgtYmlnNSJdOzsK CmxldCBleHRyYUJpZzVFbmNvZGluZ3MgPSBbIngtd2luZG93cy05NTAiLCAid2luZG93cy05NTAi LCAieC1iaWc1IiwgIm1zOTUwIiwgIndpbmRvd3MtOTUwLTIwMDAiLCAiYmlnNWhrIiwgImhrc2Nz LWJpZzUiLCAiaWJtLTEzNzUiLCAiaWJtLTEzNzVfcDEwMC0yMDA4Il07CgoKZm9yIChsZXQgZW5j b2Rpbmcgb2YgYmlnNUVuY29kaW5ncykgewogICAgbGV0IGNhbm9uaWNhbF9uYW1lX2V4cHIgPSAn bmV3IFRleHREZWNvZGVyKCInICsgZW5jb2RpbmcgKyAnIikuZW5jb2RpbmcnOwogICAgc2hvdWxk QmUoY2Fub25pY2FsX25hbWVfZXhwciwgIidiaWc1JyIpOwp9Cgpmb3IgKGxldCBlbmNvZGluZyBv ZiBleHRyYUJpZzVFbmNvZGluZ3MpIHsKICAgIGxldCBjYW5vbmljYWxfbmFtZV9leHByID0gJ25l dyBUZXh0RGVjb2RlcigiJyArIGVuY29kaW5nICsgJyIpLmVuY29kaW5nJzsKICAgIHNob3VsZFRo cm93KGNhbm9uaWNhbF9uYW1lX2V4cHIpOwp9CgoKPC9zY3JpcHQ+CjxzY3JpcHQgc3JjPSIuLi8u Li9yZXNvdXJjZXMvanMtdGVzdC1wb3N0LmpzIj48L3NjcmlwdD4KPC9ib2R5Pgo8L2h0bWw+Cg==