7602 2006-03-04 13:58:10 -0800 Only use fixupChar for entities 2006-04-13 10:01:17 -0700 1 1 1 Unclassified WebKit DOM 420+ Mac OS X 10.4 RESOLVED FIXED P2 Normal --- 7518 1 ap ap oldest_to_newest 35085 0 ap 2006-03-04 13:58:10 -0800 HTMLTokenizer uses fixupChar() to ensure that the incoming data is treated as windows-1252, not the real Latin-1. This mostly duplicates the logic in StreamingTextDecoder (see effectiveEncoding()). More importantly, shuffling code points in HTMLTokenizer affects cases where the character is already known for certain, such as: document.write("\u0080"); // writes 20AC (Euro sign) 35086 1 6855 ap 2006-03-04 13:58:44 -0800 Created attachment 6855 test case 35088 2 6857 ap 2006-03-04 14:15:08 -0800 Created attachment 6857 proposed patch Besides limiting fixupChar to entities, this removes a semi-broken Latin-1 special case from StreamingTextDecoder. As it stood, this special case was pushing both Latin-1 and windows-1252 into QString's ASCII buffer, which was then converted to Unicode in QString::makeUnicode, as if it were Latin-1. Since the tokenizer called fixupChar on the result, it (usually?) didn't matter, though. If this change negatively affects performance, then we will probably need to postulate that QString's ASCII buffer in fact contains windows-1252 data, and adjust makeUnicode accordingly. 35101 3 6857 darin 2006-03-04 16:44:23 -0800 Comment on attachment 6857 proposed patch Two issues prevent landing this: 1) We need to do performance testing. When I added a fast case for Latin-1 that relied on fixupChar, we saw a significant performance boost. Even if we don't need the super-fast "just put it in the ASCII buffer" version, we'll probably need to consider making a path that does not go through ICU. 2) I believe this may remove the quirk in our existing code that ISO Latin-1 effectively means Windows Latin-1 -- I believe we need that quirk. Maybe Alexey can handle issue (2). Someone at Apple will probably have to tackle (1) since we haven't yet found a good way to measure performance that people outside the Safari team can do easily. 35102 4 mjs 2006-03-04 17:23:21 -0800 You are write that document.write("\u0080"); should not apply the fixup hack. However, characters that are present in the original source text of a supposedly latin1 document *do* get treated as latin1 instead. So I think your patch may be backing off a bit too much. See attached test case. 35103 5 6859 mjs 2006-03-04 17:24:57 -0800 Created attachment 6859 test case to distinguish latin1 / winlatin1 parsing On this test case, Firefox treats the first two examples as WinLatin1 characters and the latter two as Latin1 (i.e. at their true unicode value). 35108 6 6857 ap 2006-03-05 00:36:04 -0800 Comment on attachment 6857 proposed patch Actually, the Latin-1/windows-1252 quirk is still enforced by effectiveEncoding(). It is covered by existing test cases, although indirectly, so adding Maciej's test case will be IMO useful, too. It passes with a single difference from Firefox - the latter maps 0x81 to U+FFFD, while fixupChar leaves it at U+0081. Resetting the r? flag for performance testing... 37498 7 6857 darin 2006-03-24 08:21:50 -0800 Comment on attachment 6857 proposed patch r=me 37760 8 justin.garcia 2006-03-28 03:31:20 -0800 Oops I moved this to fixed accidently. 39179 9 ap 2006-04-13 10:01:17 -0700 Landed, r13863. 6855 2006-03-04 13:58:44 -0800 2006-03-04 13:58:44 -0800 test case fixup.html text/html 525 ap PGh0bWw+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9InRl eHQvaHRtbDsgY2hhcnNldD1sYXRpbjEiPgo8L2hlYWQ+Cjxib2R5Pgo8c3BhbiBpZD0icGFyYSIg c3R5bGU9ImRpc3BsYXk6IG5vbmUiPjwvc3Bhbj4KCjxwPkZpcmVmb3g6ICU4MCU4MSU4MyU4NiU4 NzwvcD4KPHA+V2luSUUgNjogJTgwJTgxJTgzJTg2JTg3PC9wPgo8c3Bhbj5Zb3VyIGJyb3dzZXI6 IDwvc3Bhbj4KPHNjcmlwdD4KICAgIGlmICh3aW5kb3cubGF5b3V0VGVzdENvbnRyb2xsZXIpCiAg ICAgICAgbGF5b3V0VGVzdENvbnRyb2xsZXIuZHVtcEFzVGV4dCgpOwoKICAgIGRvY3VtZW50Lmdl dEVsZW1lbnRCeUlkKCJwYXJhIikuaW5uZXJIVE1MID0gIlx4ODBceDgxXHUwMDgzXHUwMDg2XHUw MDg3IjsKICAgIGRvY3VtZW50LndyaXRlKCI8c3Bhbj4iICsgZXNjYXBlKGRvY3VtZW50LmdldEVs ZW1lbnRCeUlkKCJwYXJhIikuaW5uZXJIVE1MKSArICI8L3NwYW4+Iik7Cjwvc2NyaXB0Pgo8L2Jv ZHk+CjwvaHRtbD4K 6857 2006-03-04 14:15:08 -0800 2006-03-24 08:21:50 -0800 proposed patch RemoveFixupChar.txt text/plain 4176 ap SW5kZXg6IFdlYkNvcmUva2h0bWwvaHRtbC9odG1sdG9rZW5pemVyLmNwcAo9PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0t LSBXZWJDb3JlL2todG1sL2h0bWwvaHRtbHRva2VuaXplci5jcHAJKHJldmlzaW9uIDEzMDgyKQor KysgV2ViQ29yZS9raHRtbC9odG1sL2h0bWx0b2tlbml6ZXIuY3BwCSh3b3JraW5nIGNvcHkpCkBA IC04LDYgKzgsNyBAQAogICAgICAgICAgICAgICAoQykgMTk5OSBBbnR0aSBLb2l2aXN0byAoa29p dmlzdG9Aa2RlLm9yZykKICAgICAgICAgICAgICAgKEMpIDIwMDEgRGlyayBNdWVsbGVyIChtdWVs bGVyQGtkZS5vcmcpCiAgICAgQ29weXJpZ2h0IChDKSAyMDA0LCAyMDA1LCAyMDA2IEFwcGxlIENv bXB1dGVyLCBJbmMuCisgICAgQ29weXJpZ2h0IChDKSAyMDA1LCAyMDA2IEFsZXhleSBQcm9za3Vy eWFrb3YgKGFwQG55cG9wLmNvbSkKIAogICAgIFRoaXMgbGlicmFyeSBpcyBmcmVlIHNvZnR3YXJl OyB5b3UgY2FuIHJlZGlzdHJpYnV0ZSBpdCBhbmQvb3IKICAgICBtb2RpZnkgaXQgdW5kZXIgdGhl IHRlcm1zIG9mIHRoZSBHTlUgTGlicmFyeSBHZW5lcmFsIFB1YmxpYwpAQCAtODMsMTEgKzg0LDcg QEAgc3RhdGljIGNvbnN0IGNoYXIgdGl0bGVFbmQgW10gPSAiPC90aXRsZQogLy8KIC8vIFRoZXJl IG1heSBiZSBiZXR0ZXIgZXF1aXZhbGVudHMKIAotLy8gV2UgbmVlZCB0aGlzIGZvciBlbnRpdGll cyBhdCBsZWFzdC4gRm9yIG5vbi1lbnRpdHkgdGV4dCwgd2UgY291bGQKLS8vIGhhbmRsZSB0aGlz IGluIHRoZSB0ZXh0IGVuY29kaW5nLgotCi0vLyBUbyBjb3ZlciBub24tZW50aXR5IHRleHQsIEkg dGhpbmsgdGhpcyBmdW5jdGlvbiB3b3VsZCBuZWVkIHRvIGJlIGNhbGxlZAotLy8gaW4gbW9yZSBw bGFjZXMuIFRoZXJlIHNlZW0gdG8gYmUgc29tZSBwbGFjZXMgdGhhdCBkb24ndCBjYWxsIGZpeFVw Q2hhci4KKy8vIFdlIG9ubHkgbmVlZCB0aGlzIGZvciBlbnRpdGllcy4gRm9yIG5vbi1lbnRpdHkg dGV4dCwgd2UgaGFuZGxlIHRoaXMgaW4gdGhlIHRleHQgZW5jb2RpbmcuCiAKIHN0YXRpYyBjb25z dCB1bnNpZ25lZCBzaG9ydCB3aW5kb3dzTGF0aW4xRXh0ZW5zaW9uQXJyYXlbMzJdID0gewogICAg IDB4MjBBQywgMHgwMDgxLCAweDIwMUEsIDB4MDE5MiwgMHgyMDFFLCAweDIwMjYsIDB4MjAyMCwg MHgyMDIxLCAvLyA4MC04NwpAQCAtMzI2LDcgKzMyMyw3IEBAIEhUTUxUb2tlbml6ZXI6OlN0YXRl IEhUTUxUb2tlbml6ZXI6OnBhcnMKICAgICAgICAgICAgIHNjcmlwdENvZGVTaXplID0gc2NyaXB0 Q29kZURlc3Qtc2NyaXB0Q29kZTsKICAgICAgICAgfQogICAgICAgICBlbHNlIHsKLSAgICAgICAg ICAgIHNjcmlwdENvZGVbc2NyaXB0Q29kZVNpemUrK10gPSBmaXhVcENoYXIoKnNyYyk7CisgICAg ICAgICAgICBzY3JpcHRDb2RlW3NjcmlwdENvZGVTaXplKytdID0gKnNyYzsKICAgICAgICAgICAg ICsrc3JjOwogICAgICAgICB9CiAgICAgfQpAQCAtNjQ1LDcgKzY0Miw3IEBAIEhUTUxUb2tlbml6 ZXI6OlN0YXRlIEhUTUxUb2tlbml6ZXI6OnBhcnMKICAgICAgICAgICAgIHN0YXRlLnNldFNraXBM Rih0cnVlKTsKICAgICAgICAgICAgICpkZXN0KysgPSAnXG4nOwogICAgICAgICB9IGVsc2UKLSAg ICAgICAgICAgICpkZXN0KysgPSBmaXhVcENoYXIoY2MpOworICAgICAgICAgICAgKmRlc3QrKyA9 IGNjOwogICAgICAgICArK3NyYzsKICAgICB9CiAKQEAgLTEwODYsNyArMTA4Myw3IEBAIEhUTUxU b2tlbml6ZXI6OlN0YXRlIEhUTUxUb2tlbml6ZXI6OnBhcnMKICAgICAgICAgICAgICAgICAgICAg ICAgIGJyZWFrOwogICAgICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgfQotICAg ICAgICAgICAgICAgICpkZXN0KysgPSBmaXhVcENoYXIoKnNyYyk7CisgICAgICAgICAgICAgICAg KmRlc3QrKyA9ICpzcmM7CiAgICAgICAgICAgICAgICAgKytzcmM7CiAgICAgICAgICAgICB9CiAg ICAgICAgICAgICBicmVhazsKQEAgLTExMjAsNyArMTExNyw3IEBAIEhUTUxUb2tlbml6ZXI6OlN0 YXRlIEhUTUxUb2tlbml6ZXI6OnBhcnMKICAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAg ICAgICAgIH0KIAotICAgICAgICAgICAgICAgICpkZXN0KysgPSBmaXhVcENoYXIoKnNyYyk7Cisg ICAgICAgICAgICAgICAgKmRlc3QrKyA9ICpzcmM7CiAgICAgICAgICAgICAgICAgKytzcmM7CiAg ICAgICAgICAgICB9CiAgICAgICAgICAgICBicmVhazsKQEAgLTE0NzEsNyArMTQ2OCw3IEBAIGJv b2wgSFRNTFRva2VuaXplcjo6d3JpdGUoY29uc3QgU2VnbWVudGUKICAgICAgICAgICAgIGlmICgg cm93ID4gMHgwNSAmJiByb3cgPCAweDEwIHx8IHJvdyA+IDB4ZmQgKQogICAgICAgICAgICAgICAg ICAgICBjdXJyVG9rZW4uY29tcGxleFRleHQgPSB0cnVlOwogI2VuZGlmCi0gICAgICAgICAgICAq ZGVzdCsrID0gZml4VXBDaGFyKCpzcmMpOworICAgICAgICAgICAgKmRlc3QrKyA9ICpzcmM7CiAg ICAgICAgICAgICArK3NyYzsKICAgICAgICAgfQogICAgIH0KSW5kZXg6IFdlYkNvcmUvcGxhdGZv cm0vU3RyZWFtaW5nVGV4dERlY29kZXIuY3BwCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFdlYkNvcmUvcGxhdGZv cm0vU3RyZWFtaW5nVGV4dERlY29kZXIuY3BwCShyZXZpc2lvbiAxMzA4MikKKysrIFdlYkNvcmUv cGxhdGZvcm0vU3RyZWFtaW5nVGV4dERlY29kZXIuY3BwCSh3b3JraW5nIGNvcHkpCkBAIC00Mywx MiArNDMsNiBAQCBTdHJlYW1pbmdUZXh0RGVjb2Rlcjo6U3RyZWFtaW5nVGV4dERlY29kCiB7CiB9 CiAKLVFTdHJpbmcgU3RyZWFtaW5nVGV4dERlY29kZXI6OmNvbnZlcnRMYXRpbjEoY29uc3QgdW5z aWduZWQgY2hhciogcywgaW50IGxlbmd0aCkKLXsKLSAgICBBU1NFUlQobV9udW1CdWZmZXJlZEJ5 dGVzID09IDApOwotICAgIHJldHVybiBRU3RyaW5nKHJlaW50ZXJwcmV0X2Nhc3Q8Y29uc3QgY2hh ciAqPihzKSwgbGVuZ3RoKTsKLX0KLQogc3RhdGljIGNvbnN0IFVDaGFyIHJlcGxhY2VtZW50Q2hh cmFjdGVyID0gMHhGRkZEOwogc3RhdGljIGNvbnN0IFVDaGFyIEJPTSA9IDB4RkVGRjsKIHN0YXRp YyBjb25zdCBpbnQgQ29udmVyc2lvbkJ1ZmZlclNpemUgPSAxNjM4NDsKQEAgLTIzMSwxMCArMjI1 LDYgQEAgUVN0cmluZyBTdHJlYW1pbmdUZXh0RGVjb2Rlcjo6Y29udmVydChjbwogICAgIC8vI2Rl ZmluZSBQQVJUSUFMX0NIQVJBQ1RFUl9IQU5ETElOR19URVNUX0NIVU5LX1NJWkUgMTAwMAogCiAg ICAgc3dpdGNoIChtX2VuY29kaW5nLmVuY29kaW5nSUQoKSkgewotICAgIGNhc2UgTGF0aW4xRW5j b2Rpbmc6Ci0gICAgY2FzZSBXaW5MYXRpbjFFbmNvZGluZzoKLSAgICAgICAgcmV0dXJuIGNvbnZl cnRMYXRpbjEoY2hzLCBsZW4pOwotCiAgICAgY2FzZSBVVEYxNkVuY29kaW5nOgogICAgICAgICBy ZXR1cm4gY29udmVydFVURjE2KGNocywgbGVuKTsKIApJbmRleDogV2ViQ29yZS9wbGF0Zm9ybS9T dHJlYW1pbmdUZXh0RGVjb2Rlci5oCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIFdlYkNvcmUvcGxhdGZvcm0vU3Ry ZWFtaW5nVGV4dERlY29kZXIuaAkocmV2aXNpb24gMTMwODIpCisrKyBXZWJDb3JlL3BsYXRmb3Jt L1N0cmVhbWluZ1RleHREZWNvZGVyLmgJKHdvcmtpbmcgY29weSkKQEAgLTQ0LDcgKzQ0LDYgQEAg bmFtZXNwYWNlIFdlYkNvcmUgewogICAgICAgICBRU3RyaW5nIGNvbnZlcnQoY29uc3QgY2hhcipj aHMsIGludCBsZW4sIGJvb2wgZmx1c2gpCiAgICAgICAgICAgICB7IHJldHVybiBjb252ZXJ0KHJl aW50ZXJwcmV0X2Nhc3Q8Y29uc3QgdW5zaWduZWQgY2hhcio+KGNocyksIGxlbiwgZmx1c2gpOyB9 CiAgICAgICAgIFFTdHJpbmcgY29udmVydChjb25zdCB1bnNpZ25lZCBjaGFyKiBjaHMsIGludCBs ZW4sIGJvb2wgZmx1c2gpOwotICAgICAgICBRU3RyaW5nIGNvbnZlcnRMYXRpbjEoY29uc3QgdW5z aWduZWQgY2hhciogY2hzLCBpbnQgbGVuKTsKICAgICAgICAgUVN0cmluZyBjb252ZXJ0VVRGMTYo Y29uc3QgdW5zaWduZWQgY2hhciogY2hzLCBpbnQgbGVuKTsKICAgICAgICAgCiAgICAgICAgIC8v IElDVSBkZWNvZGluZy4K 6859 2006-03-04 17:24:57 -0800 2006-03-04 17:24:57 -0800 test case to distinguish latin1 / winlatin1 parsing latin1.html text/html 982 mjs PGh0bWw+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSdDb250ZW50LVR5cGUnIGNvbnRlbnQ9J3Rl eHQvaHRtbDsgY2hhcnNldD1sYXRpbjEnPgo8L2hlYWQ+Cjxib2R5Pgo8cD5BcyBlbnRpdGllczog PHNwYW4gaWQ9J2VudGl0aWVzJz4mI3g4MDsmI3g4MTsmI3g4MzsmI3g4NjsmI3g4Nzs8L3NwYW4+ PC9wPgo8c2NyaXB0PmRvY3VtZW50LndyaXRlKCc8cD4oZXNjYXBlZCk6ICcgKyBlc2NhcGUoZG9j dW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ2VudGl0aWVzJykuaW5uZXJIVE1MKSArICc8L3A+Jyk7PC9z Y3JpcHQ+PHA+SW5saW5lOiA8c3BhbiBpZD0naW5saW5lJz6AgYOGhzwvc3Bhbj48L3A+CjxzY3Jp cHQ+ZG9jdW1lbnQud3JpdGUoJzxwPihlc2NhcGVkKTogJyArIGVzY2FwZShkb2N1bWVudC5nZXRF bGVtZW50QnlJZCgnaW5saW5lJykuaW5uZXJIVE1MKSArICc8L3A+Jyk7PC9zY3JpcHQ+PHA+ZG9j dW1lbnQud3JpdGUgb2YgdW5pY29kZSBlc2NhcGVzOiA8c2NyaXB0PmRvY3VtZW50LndyaXRlKCc8 c3BhbiBpZD0id3JpdGUiPlx4ODBceDgxXHUwMDgzXHUwMDg2XHUwMDg3PC9zcGFuPicpOzwvc2Ny aXB0PjwvcD48c2NyaXB0PmRvY3VtZW50LndyaXRlKCc8cD4oZXNjYXBlZCk6ICcgKyBlc2NhcGUo ZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ3dyaXRlJykuaW5uZXJIVE1MKSArICc8L3A+Jyk7PC9z Y3JpcHQ+PHA+aW5uZXJIVE1MIGFzc2lnbm1lbnQgb2YgdW5pY29kZSBlc2NhcGVzOiA8c3BhbiBp ZD0naW5uZXInPjwvc3Bhbj48L3A+PHNjcmlwdD5kb2N1bWVudC5nZXRFbGVtZW50QnlJZCgnaW5u ZXInKS5pbm5lckhUTUwgPSAnXHg4MFx4ODFcdTAwODNcdTAwODZcdTAwODcnOzwvc2NyaXB0Pjxz Y3JpcHQ+ZG9jdW1lbnQud3JpdGUoJzxwPihlc2NhcGVkKTogJyArIGVzY2FwZShkb2N1bWVudC5n ZXRFbGVtZW50QnlJZCgnaW5uZXInKS5pbm5lckhUTUwpICsgJzwvcD4nKTs8L3NjcmlwdD4KPC9i b2R5Pgo8L2h0bWw+Cg==