Bug 5027 – Decoder doesn't auto-detect XML encoded as UTF-16 if BOM is not present

• Status: VERIFIED
• Resolution: FIXED
• Component: DOM

[Alexey Proskuryakov]

When viewing XML files encoded as UTF-16 without a byte order mark, user default encoding is used 
instead of UTF-16 (big- or little-endian).

[Alexey Proskuryakov]

Created attachment 3925 [details]
proposed patch

[Alexey Proskuryakov]

Created attachment 3926 [details]
test case

[Alexey Proskuryakov]

Created attachment 3927 [details]
improved test case

The previous test case didn't work if the default encoding in Safari was
Latin-1 (worked with Windows Cyrillic).

[Eric Seidel (no email)]

Comment on attachment 3925 [details]
proposed patch

Well, the code looks fine, as in it does what it says it does.	I'm not sure
what the policy on autodetection is.  The previous if block before this one
finds <?xml and searches for an encoding listed there.	This one finds	< ? x m
l  says, oh those must be 2 byte characters, and assumes unicode 16. I'm not
sure if that's a good thing or not, and thus will leave this one up to Darin to
decide.

[Tobias Lidskog]

This would be the relevant section in the xml spec:
http://www.w3.org/TR/2004/REC-xml-20040204/#sec-guessing-no-ext-info

[Alexey Proskuryakov]

(In reply to comment #5)
So, what is still not auto-detected:
1) UTF-32
2) EBCDIC
3) "other encoding with a 16-bit code unit <...> and ASCII characters encoded as ASCII values (the 
encoding declaration must be read to determine which)"

Of these, only (3) may be relevant in this specific case; but I'm not sure if it is practical at all. Yes, the 
spec suggests that an encoding declaration should be always read because new encodings may be 
invented, but there's nothing we can do about new encodings in advance, so treating them as UTF-16 
should be as good as any other solution.

Even if there are existing encodings such as in (3), this patch doesn't make the situation worse.

[Darin Adler]

Comment on attachment 3925 [details]
proposed patch

What's the argument for looking only at ptr[0] and not also at ptr[2], [4], and
[6]?

Otherwise, looks great!

[Alexey Proskuryakov]

Created attachment 3955 [details]
proposed patch

Well, we're guessing here, and I didn't see how checking for all the zeroes
would be any safer than checking for just the first one - but it makes the code
longer :)

Anyhow, here's a patch with the suggested change.

[Darin Adler]

Comment on attachment 3955 [details]
proposed patch

r=me

[Lucas Forschler]

Mass moving XML DOM bugs to the "DOM" Component.