Bug 5027

Summary: Decoder doesn't auto-detect XML encoded as UTF-16 if BOM is not present
Product: WebKit Reporter: Alexey Proskuryakov <ap>
Component: DOMAssignee: Maciej Stachowiak <mjs>
Severity: Normal CC: cdumez
Priority: P2    
Version: 420+   
Hardware: Mac   
OS: OS X 10.4   
Description Flags
proposed patch
darin: review-
test case
improved test case
proposed patch darin: review+

Description Alexey Proskuryakov 2005-09-17 13:00:36 PDT
When viewing XML files encoded as UTF-16 without a byte order mark, user default encoding is used 
instead of UTF-16 (big- or little-endian).
Comment 1 Alexey Proskuryakov 2005-09-17 13:01:14 PDT
Created attachment 3925 [details]
proposed patch
Comment 2 Alexey Proskuryakov 2005-09-17 13:02:23 PDT
Created attachment 3926 [details]
test case
Comment 3 Alexey Proskuryakov 2005-09-17 13:08:36 PDT
Created attachment 3927 [details]
improved test case

The previous test case didn't work if the default encoding in Safari was
Latin-1 (worked with Windows Cyrillic).
Comment 4 Eric Seidel (no email) 2005-09-17 15:24:32 PDT
Comment on attachment 3925 [details]
proposed patch

Well, the code looks fine, as in it does what it says it does.	I'm not sure
what the policy on autodetection is.  The previous if block before this one
finds <?xml and searches for an encoding listed there.	This one finds	< ? x m
l  says, oh those must be 2 byte characters, and assumes unicode 16. I'm not
sure if that's a good thing or not, and thus will leave this one up to Darin to
Comment 5 Tobias Lidskog 2005-09-18 02:05:49 PDT
This would be the relevant section in the xml spec:
Comment 6 Alexey Proskuryakov 2005-09-18 03:31:53 PDT
(In reply to comment #5)
So, what is still not auto-detected:
1) UTF-32
3) "other encoding with a 16-bit code unit <...> and ASCII characters encoded as ASCII values (the 
encoding declaration must be read to determine which)"

Of these, only (3) may be relevant in this specific case; but I'm not sure if it is practical at all. Yes, the 
spec suggests that an encoding declaration should be always read because new encodings may be 
invented, but there's nothing we can do about new encodings in advance, so treating them as UTF-16 
should be as good as any other solution.

Even if there are existing encodings such as in (3), this patch doesn't make the situation worse.
Comment 7 Darin Adler 2005-09-19 15:56:00 PDT
Comment on attachment 3925 [details]
proposed patch

What's the argument for looking only at ptr[0] and not also at ptr[2], [4], and

Otherwise, looks great!
Comment 8 Alexey Proskuryakov 2005-09-19 21:11:42 PDT
Created attachment 3955 [details]
proposed patch

Well, we're guessing here, and I didn't see how checking for all the zeroes
would be any safer than checking for just the first one - but it makes the code
longer :)

Anyhow, here's a patch with the suggested change.
Comment 9 Darin Adler 2005-09-20 17:02:09 PDT
Comment on attachment 3955 [details]
proposed patch

Comment 10 Lucas Forschler 2019-02-06 09:03:08 PST
Mass moving XML DOM bugs to the "DOM" Component.