Bug 56294 – Web Inspector: migrate from Inspector.idl to InspectorAPI.json for protocol schema definition meta bug.

• Component: Web Inspector (Deprecated)
• Status: RESOLVED FIXED

[Pavel Feldman]

It is time to describe our protocol in greater detail (i.e. with types and documentation). So I guess we should migrate to JSON schema and use Python for source code generation. I think all the ports support Python by now (probably worth checking). It does mean pretty much complete rewrite for the generator that is currently a 1K line perl script, but I think we can manage it without much pain. Here is the plan:

Step one:
- Add JSON schema generator into CodeGeneratorInspector.pm so that it could convert Inspector.idl into InspectorAPI.json at any time.
- Implement a python script that does backwards conversion (InspectorAPI.json -> Inspector.idl)
- Do a one-time InspectorAPI.json generation and make it primary in the build. I.e. during the build, Inspector.idl is being generated from InspectorAPI.json using script from previous bullet and then original CodeGeneratorInspector.pm steps in place.

Follow up changes:
- Make InspectorAPI.json more detailed, add types and descriptions there
- Deploy this InspectorAPI.json to the front-end and run optional validation of all messages in the debug builds
- Generate documentation off InspectorAPI.json

Longer term tasks:
- Rewrite code generation to eliminate intermediate idl class
- Add schema validation to the backend.

[Patrick Mueller]

I have to admit, though I initially disliked using IDL the way Web Inspector uses it, I've grown to like it.  Very concise, human readable, and human writable.  Compare to JSON, which has none of those qualities.

I'd vote for leaving it has IDL, and adding more to it:

- describe all the data structures with the IDL
- documentation in comments preceding the interfaces, functions, and structures ala JavaDoc

But not a big deal either way.

For weinre, I'll likely continue using IDL for the interfaces that I've added, and change my IDL->JSON converter to generate JSON in whatever format you're generating.

I already have a python IDL to JSON converter, here:

   https://github.com/pmuellr/weinre/blob/master/weinre.build/scripts/simple-idl2json.py
   
I'd be happy to rewrite it into a format you need (my version does stuff that we won't presumably need for the WebKit version), or feel free to borrow concepts from it.  As I'm the only author of the file, I'd be happy to relicense it to just BSD, and put appropriate comments in it suitable for WebKit and post here for folks to work off of.  Let me know.

w/r/t python as the converter.  +1.  Downside is that the "json" module for python only showed up in Python 2.5.  If we still need to support Python 2.4, there are a couple options:

- import the simplejson package into WebKit and use it (the official json package for python IS simplejson)

- use python's "repr" function to generate JSON from python dictionaries - I think it will generate legal JSON for the limited needs here.  Likewise, I believe you can "eval" JSON in Python and get the appropriate thing read in.

Lastly, you should check with Joe Gregorio (Google Chapel Hill NC office), as he's working on the standardization of "Google Data APIs" or whatever, and will surely have some ideas for you.  I believe JSON is the common base they are working on also.

I guess I'd almost prefer we move to Protocol Buffers, as they seem to be a perfect fit for this problem.  Getting Protocol Buffers into WebKit seems like it might be difficult though.

[Patrick Mueller]

One additional note; I found that using my IDL->JSON version of the Web Inspector APIs to be perfectly acceptable in practice, but they were really much larger than what I really needed.

As described here, the InspectorAPI.json file that includes documentation will be bigger than it needs to be - the runtime doesn't need the documentation, for example.  

What I ended up doing was generating two JSON files - one that is used to generate documentation, the other is used at runtime.  In my case, the JSON went from something like 200K to 30K.  The numbers are a bit inflated as I was adding white space to the 200K version as well, which I removed from the 30K version.

[Pavel Feldman]

(In reply to comment #1)
> I'd vote for leaving it has IDL, and adding more to it:
> - describe all the data structures with the IDL
> - documentation in comments preceding the interfaces, functions, and structures ala JavaDoc

The only code that we could reuse for that is perl generator for webkit's variation of idl. I would not want to invest into this technology.

> I already have a python IDL to JSON converter, here:
> 
>    https://github.com/pmuellr/weinre/blob/master/weinre.build/scripts/simple-idl2json.py

It is not clear which JSON we end up using, so I modified our doc gen a bit to generate one. It took me 20 minutes to do that (inserted as an excuse for not using yours).

> - import the simplejson package into WebKit and use it (the official json package for python IS simplejson)
> 
> - use python's "repr" function to generate JSON from python dictionaries - I think it will generate legal JSON for the limited needs here.  Likewise, I believe you can "eval" JSON in Python and get the appropriate thing read in.

I'll need to learn python setup across the bots.

> Lastly, you should check with Joe Gregorio (Google Chapel Hill NC office), as he's working on the standardization of "Google Data APIs" or whatever, and will surely have some ideas for you.  I believe JSON is the common base they are working on also.

> I guess I'd almost prefer we move to Protocol Buffers, as they seem to be a perfect fit for this problem.  Getting Protocol Buffers into WebKit seems like it might be difficult though.

Yeah.

(In reply to comment #2)
> As described here, the InspectorAPI.json file that includes documentation will be bigger than it needs to be - the runtime doesn't need the documentation, for example.

I was not saying we should make this json a part of the runtime. It is just the source for generation.

> What I ended up doing was generating two JSON files - one that is used to generate documentation, the other is used at runtime.  In my case, the JSON went from something like 200K to 30K.  The numbers are a bit inflated as I was adding white space to the 200K version as well, which I removed from the 30K version.

Given above, I think we can keep things co-located.


I'll attach the draft of the JSON I generated off the IDL shortly.

[Pavel Feldman]

Created attachment 86142 [details]
[JSON] Sample protocol definition.

The conversion rate is about x4 (300 lines IDL bloats into 1200 lines JSON). But I can read that.

[Pavel Feldman]

Created attachment 86158 [details]
Patch

[Pavel Feldman]

Created attachment 86159 [details]
Patch

[Early Warning System Bot]

Attachment 86158 [details] did not build on qt:
Build output: http://queues.webkit.org/results/8198303

[WebKit Review Bot]

Attachment 86158 [details] did not build on chromium:
Build output: http://queues.webkit.org/results/8198305

[Early Warning System Bot]

Attachment 86159 [details] did not build on qt:
Build output: http://queues.webkit.org/results/8198307

[WebKit Review Bot]

Attachment 86159 [details] did not build on chromium:
Build output: http://queues.webkit.org/results/8200220

[Pavel Feldman]

Created attachment 86167 [details]
Patch

[WebKit Review Bot]

Attachment 86167 [details] did not build on chromium:
Build output: http://queues.webkit.org/results/8205150

[Patrick Mueller]

(In reply to comment #3)
> (In reply to comment #2)
> > As described here, the InspectorAPI.json file that includes documentation will be bigger than it needs to be - the runtime doesn't need the documentation, for example.
> 
> I was not saying we should make this json a part of the runtime. It is just the source for generation.

You may find you can do away with some of the generation, if you include the JSON instead.  That's what I'm doing in weinre.  I don't generate any JS code at all (or Java - the server-y bits of weinre are currently written in Java) - use the JSON to build proxies dynamically, and do some verification on the interface implementation side.  Also use the same JSON to generate doc dynamically.

[Yury Semikhatsky]

Comment on attachment 86167 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=86167&action=review

> Source/WebCore/inspector/generate-inspector-idl:1
> +#!/usr/bin/env python

This file should have .py extension.

[Pavel Feldman]

Created attachment 86170 [details]
Patch

[Yury Semikhatsky]

Comment on attachment 86170 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=86170&action=review

> Source/WebCore/inspector/generate-inspector-idl:36
> +    # Python 3, change to: print("Usage: %s -o OUTPUT_FILE INPUT_FILE" % program_name, file=sys.stderr)

Remove this confusing comment.

> Source/WebCore/inspector/generate-inspector-idl:106
> +        output_file.write("\n#if defined(ENABLE_%s) && ENABLE_%s" % (macro_traits[domain["domain"]], macro_traits[domain["domain"]]))

Please extract macro_traits[domain["domain"]] int a variable. It's repeated 4 times.

[Pavel Feldman]

Committed r81479: <http://trac.webkit.org/changeset/81479>

[WebKit Review Bot]

http://trac.webkit.org/changeset/81479 might have broken GTK Linux 32-bit Release
The following tests are not passing:
fast/reflections/inline-crash.html

[Patrick R. Gansterer]

(In reply to comment #17)
> Committed r81479: <http://trac.webkit.org/changeset/81479>

Inspector.json causes problems on win32 python with git repository.
By default Inspector.json has CRLF line endings on windows and pythons eval does not accept '\r' and will fail.
Can we replace the eval with a correct json parser which will accept '\r'or do we need to add a line to the .gitattributes file?

[Pavel Feldman]

(In reply to comment #19)
> (In reply to comment #17)
> > Committed r81479: <http://trac.webkit.org/changeset/81479>
> 
> Inspector.json causes problems on win32 python with git repository.
> By default Inspector.json has CRLF line endings on windows and pythons eval does not accept '\r' and will fail.
> Can we replace the eval with a correct json parser which will accept '\r'or do we need to add a line to the .gitattributes file?

json decoder is not supported in Python 2.4 / 2.5 that WebKit should build with Ok. That's why we use eval. You should file a separate bug with failing build logs on the CRLF issue and we will fix it.