|Summary:||HTMLDocumentParser should reuse tokens from HTMLPreloadScanner|
|Product:||WebKit||Reporter:||Tony Gentilcore <tonyg>|
|Component:||WebCore Misc.||Assignee:||Nobody <webkit-unassigned>|
|Severity:||Normal||CC:||abarth, ap, eric, kbalazs, koivisto, loki, sam, simonjam, skyul|
|Version:||528+ (Nightly build)|
|OS:||OS X 10.5|
|Bug Depends on:|
Description Tony Gentilcore 2011-03-29 12:07:41 PDT
See FIXME in HTMLPreloadScanner::scan(). Currently we generate tokens during preloading then discard them and retokenize during parsing. We should be able to save them and reuse them if the input steam isn't modified by the script (doc.write). Eric, I'm thinking of picking this up. Do you have any high-level implementation thoughts?
Comment 1 Tony Gentilcore 2011-03-29 12:18:36 PDT
I'm thinking of making a HTMLTokenSegmentedString class which subclasses the SegmentedString used by the parser and maintains a cache of tokens in the stream. The tokenizer can just return the next cache token if it exists and the htmltokensegmentedstring will drop its cache when anything is inserted into the string.
Comment 2 Adam Barth 2011-03-29 13:25:17 PDT
You're going to have a lot better luck saving AtomicHTMLTokens. They're way smaller in most cases. We mostly just need a way to buffer them and to detect when to discard the buffer because something changed.
Comment 3 Adam Barth 2011-03-29 13:36:53 PDT
I'm not sure a subclass is needed. You're probably better off making an object that composites in the SegmentedString, like we do in HTMLInputStream. Actually, maybe you should just make HTMLInputStream smarter. :)
Comment 4 Balazs Kelemen 2011-07-13 07:46:38 PDT
*** Bug 64369 has been marked as a duplicate of this bug. ***
Comment 5 Balazs Kelemen 2011-07-13 08:03:21 PDT
There is a problem with reusing the tokens from the scanner. The parsing algorithm requires the tree builder to participate in the tokenizing process at a few point like: http://trac.webkit.org/browser/trunk/Source/WebCore/html/parser/HTMLTreeBuilder.cpp#L464 http://trac.webkit.org/browser/trunk/Source/WebCore/html/parser/HTMLTreeBuilder.cpp#L809 http://trac.webkit.org/browser/trunk/Source/WebCore/html/parser/HTMLTreeBuilder.cpp#L833 (There is a a few more.) This means we cannot produce the correct token stream without the tree builder. This could be solved by creating a mock tree builder just for guiding the tokenizer but it would make preloading more costly. On the other hand we are running the scanner while we are waiting for the network so maybe it could be worthwile.
Comment 6 Adam Barth 2011-07-13 10:18:51 PDT
If those cases are rare, we could invalidate the token stream.
Comment 7 Eric Seidel (no email) 2013-03-05 02:02:10 PST
This is done as part of the threaded parser. See bug 106127. I'm not sure we want to bother trying to do this on the main-thread parser. Although maybe it would make things nice to use CompactHTMLToken even on the main thread. :)