This change also introduces parse state that is necessary to keep generated lexer code compact.
Created attachment 47361 [details] [PATCH] Proposed change.
Attachment 47361 [details] did not pass style-queue: Failed to run "WebKitTools/Scripts/check-webkit-style" exit_code: 1 Skipping input 'WebCore/inspector/front-end/JavaScriptTokenizer.js': Can't open for reading WebCore/inspector/front-end/TokenizerForHTML.js:78: Line contains tab character. [whitespace/tab] [5] WebCore/inspector/front-end/TokenizerForHTML.js:79: Line contains tab character. [whitespace/tab] [5] Total errors found: 2 If any of these errors are false positives, please file a bug against check-webkit-style.
Created attachment 47364 [details] [PATCH] Proposed change (style fix). Please ignore style failures in the removed file.
Comment on attachment 47361 [details] [PATCH] Proposed change. I prefer HTMLTokenizer over TokenizerForHTML, like you had JavaScriptTokenizer. It is a better class name. (Even though the new name keeps the files together.) Otherwise r+.
Created attachment 47368 [details] [PATCH] Same with better naming and CSS tokenizer bonus.
Comment on attachment 47368 [details] [PATCH] Same with better naming and CSS tokenizer bonus. > - this._tokenizer = new WebInspector.JavaScriptTokenizer(); > + this._tokenizer = new WebInspector.SourceCSSTokenizer(); What tokenizer should this be set to?
(In reply to comment #6) > (From update of attachment 47368 [details]) > > - this._tokenizer = new WebInspector.JavaScriptTokenizer(); > > + this._tokenizer = new WebInspector.SourceCSSTokenizer(); > > What tokenizer should this be set to? This is going to be determined by the mimeType passed to the editor (simple mapping). Currently just hardcoded for testing. I think I am one patch away form the editor as a viewer.
Committing to http://svn.webkit.org/repository/webkit/trunk ... R WebCore/inspector/front-end/JavaScriptTokenizer.js => WebCore/inspector/front-end/SourceJavaScriptTokenizer.js R WebCore/inspector/front-end/JavaScriptTokenizer.re2js => WebCore/inspector/front-end/SourceJavaScriptTokenizer.re2js M WebCore/ChangeLog M WebCore/WebCore.gypi M WebCore/WebCore.vcproj/WebCore.vcproj A WebCore/inspector/front-end/SourceCSSTokenizer.js A WebCore/inspector/front-end/SourceCSSTokenizer.re2js A WebCore/inspector/front-end/SourceHTMLTokenizer.js A WebCore/inspector/front-end/SourceHTMLTokenizer.re2js M WebCore/inspector/front-end/TextEditorHighlighter.js M WebCore/inspector/front-end/WebKit.qrc M WebCore/inspector/front-end/inspector.html Committed r53846