If an HTML entity is not completely contained in one TCP packet, it may be not handled as such, and get rendered as plain text instead. Steps to reproduce: Load the page from bug URL. Won't reproduce if loaded from a local file or from cache, of course. Regression: present in Safari 2.0 (as of Mac OS X 10.4.2 without updates); haven't checked earlier versions.
Confirmed with WebKit 412.7 and ToT.
The issue is likely in htmltokenizer.cpp. The code looks like it's almost right to handle this case, just not quite.
Created attachment 3755 [details] proposed patch Apparently, it was only failing for hexadecimal entities - so, I've just copied a couple of lines from a decimal case. This patch also fixes a bunch of compilation errors with TOKEN_DEBUG defined.
Is XML DOM indeed a correct component for this? Perhaps, I do not understand the differences well enough, but this doesn't sound related to XML at all (maybe HTML DOM?)
Comment on attachment 3755 [details] proposed patch I believe this patch reintroduces a bug we fixed a while back. It was trying to fix that one that we introduced this bug. This was back when we didn't always make tests for the bugs we fixed (bad!) so there's no layout test. The original fix was back on 2003-11-17, you can see it in the ChangeLog. 3485925: Safari does not correctly parse eight-digit hex character entities Also, please don't include the TOKEN_DEBUG changes along with the bug fix. Lets handle those separately. Here's some text from the original bug report: -------- Safari does not correctly parse eight-digit hex character entities. I noticed this at <http://www.alanwood.net/unicode/deseret.html>. "𐐀" in HTML works fine (gives me the glyph for U+10400, "DESERET CAPITAL LETTER LONG I"). But if I use "𐐀", the page renders incorrectly; I get a Last Resort glyph followed by "0;" It looks like the numeric entity parser only looks seven digits into the hex string, so displays U+1040 followed by "0;", instead of realizing that it's all one entity and displaying U+10400. -------- So we should make a new fix that works properly for both and probably make a layout test for the older bug fix.
Created attachment 3760 [details] test case for the older bug
Created attachment 3761 [details] proposed patch Just changed the length from 9 to 10 - this seems to fix the issue... I filed TOKEN_DEBUG patch as bug 4849.
Comment on attachment 3761 [details] proposed patch OK, r=me.
Gecko seems to allow hexadecimal entities with any number of digits; it lets extra leading digits overflow off the left.
I added the test case to layout tests. Perhaps we should change the hexadecimal entity code to not have a limit on number of digits to match Gecko/Firefox? Seems that would be simple. I guess it depends on what the behavior of WinIE is, and probably worth investigating too-long decimal entities too.
Created attachment 3783 [details] Long entities test Here is a test matrix for long and otherwise invalid entities with results from several browsers. Mozilla bug report about bignum entities is here: <https://bugzilla.mozilla.org/show_bug.cgi?id=215622>.
Do the matrix entries for WinIE that say "?" mean an actual question mark character? Do the empty matrix entries for MacIE mean that no characters at all were emitted? It seems like our behavior is not much like other browsers -- we should probably do a little work in this area? What about when non-digit characters are included before the semicolon?
(In reply to comment #12) > Do the matrix entries for WinIE that say "?" mean an actual question mark character? Do the empty > matrix entries for MacIE mean that no characters at all were emitted? Yes in both cases. > It seems like our behavior is not much like other browsers -- we should probably do a little work in > this area? Probably... Would you prefer to have this done while resolving this bug, or in a separate one? > What about when non-digit characters are included before the semicolon? I would expect them to bail out immediately (otherwise, it's too easy to lose a lot of content because of a non-terminated entity)... I'll check on Monday. A note at <http://www.w3.org/TR/REC-html40/charset.html#entities> suggests some more evil cases to possibly deal with: >In SGML, it is possible to eliminate the final ";" after a character >reference in some cases (e.g., at a line break or immediately before >a tag). In other circumstances it may not be eliminated (e.g., in the >middle of a word). We strongly suggest using the ";" in all cases >to avoid problems with user agents that require this character to be present.
I think we should make one or more new bug reports for the issues we turned up here. I'm going to land your patch to fix the "split across two separate tokenizer writes" issue and close this bug report.
(In reply to comment #14) Done; bug 4948.