Bug 8972 – REGRESSION: invalid UTF-8 sequences are not displayed

• Component: Page Loading
• Status: RESOLVED FIXED

[tim bates]

if you visit the URL in Tiger release Safari, you see a a balck diamond question mark character in the sentence "Store an unlimited amount of data for only 15"

This is also shown in the source
   Store an unlimited amount of data for only 15� per gigabyte

Under 420+, the cents character is simply missing from the view and the source.

Not sure what the bug, if any, is here.

[Alexey Proskuryakov]

This was an intentional change, see bug 3556. However, I cannot confirm the comment that other browsers ignore invalid UTF-8 sequences - WinIE and Mac Firefox do display either question marks or empty boxes at both bug URLs for me. 

[Darin Adler]

I'd like to match other browsers. I don't know how I could have gotten it wrong originally, though. I'm quite sure there were sites with black question marks in Safari only and nothing there in other browsers. Maybe there are different categories of illegal UTF-8 sequences that are handled differently?

Someone should do some research on this and find out what I got wrong originally.

[Darin Adler]

Strange, I tested the http://www.cheap-hotel-rooms.com/Reno/Peppermill-Hotel.htm page mentioned in the original bug. It shows plain old "?" characters in Firefox 1.5.0.3 on Macintosh where we used to use our black diamond question mark. (Around the text "including a 120-screen cube".) But I could have sworn I tested this back when I fixed the bug. Is there a chance Firefox changed its behavior?

I probably never tested Windows Internet Explorer behavior.

[Alexey Proskuryakov]

(In reply to comment #3)
> Is there a chance Firefox changed its behavior?

Firefox 1.0.3 and 1.0.5 also display question marks for me. I don't have other versions archived.

[Alexey Proskuryakov]

Created attachment 8442 [details]
Broken UTF-8

Looks like various kinds of UTF-8 brokenness all give question marks in Firefox 1.5.

Invalid WinLatin bytes get discarded (but it is so in shipping Safari, too); recovery from broken ShiftJIS is very different in Firefox, shipping Safari and ToT WebKit.

[Alexey Proskuryakov]

Created attachment 8443 [details]
windows-1252

[Alexey Proskuryakov]

Created attachment 8444 [details]
ShiftJIS

[Alice Liu]

<rdar://problem/4575223>

[Nicholas Shanks]

I would like to see this fixed. I believe the "fix" to bug 3556 should never have been authorised, as I don't believe it was a bug. It is very important to know that the page is not being displayed in the correct encoding so that I can try alternates manually. Not displaying the black diamonds disguises this and means the user is not aware that data is missing, which could potentially be very bad! Firefox's question marks are not very conspicuous, but at least they are there.

[Alexey Proskuryakov]

Created attachment 8885 [details]
proposed patch

[Darin Adler]

Comment on attachment 8885 [details]
proposed patch

appendOmittingUnwanted should be renamed to appendOmittingBOM -- that was its original name way back in the mists of time before we added null (now gone) and replacement character (now gone) to the list of characters to strip.

[Alexey Proskuryakov]

Created attachment 8895 [details]
proposed patch

Renamed appendOmittingUnwanted().

[Alexey Proskuryakov]

Now that the layout tests work again, I've found that this change uncovers an apparent bug in XML entity handling, looking into it...

[Alexey Proskuryakov]

Created attachment 8897 [details]
proposed patch

Now with a getXHTMLEntity() fix.

[Darin Adler]

Comment on attachment 8897 [details]
proposed patch

r=me

[Alexey Proskuryakov]

Committed revision 14911.