Bug 30897 – Web Inspector: Support Debugging via HTTP Headers like FirePHP and FireLogger

• Status: RESOLVED
• Resolution: WONTFIX
• Component: Web Inspector (Deprecated)

[Joseph Pecoraro]

There are existing developer tools that use HTTP Headers to send logging information from the server to the client for display in a console.  The following is a clear description, use case, and reference to existing tools:

> [...] basically it's a way on the server side to log special headers
> back to the client on requests. Firephp/Firepython are FF/Firebug plugins
> that check for those headers and display them in the console. 
> 
> It's very handy with ajax requests where actually outputting any content
> during a json request would break the json formatting and cause errors in
> the application. 
> 
> It's also very handy since it's server side, you can debug/output items
> that you might not see otherwise. 
> 
> One use case I love is that I have a flag in the application config that
> will enable outputting all sql queries during the page load. I can then
> refresh a page and see them all in my browser client to help trace
> potential issues. It's much faster than enabling query logging on the
> server and going through those logs since it's at the application level. 
> 
> http://www.firephp.org/
> http://firelogger.binaryage.com/
> 
> Rick

On the technical side, the FirePHP team has documented their protocol (on a wiki article last edited July 2009):
http://www.firephp.org/Wiki/Reference/Protocol

FireLogger does not seem to have much documentation but its source code is available online:
http://github.com/darwin/firepython

It may be best to contact the teams and discuss integration with the Web Inspector.

[Joseph Pecoraro]

I see there has been some interest and it sounded like some initial work on this feature. Feel free to post questions here. Putting up a patch for discussion is always encouraged as well as it can help clear out small issues and allow for early feedback.  Cheers!

[David Zuelke]

For the record, http://www.wildfirehq.org/ is the initiative for a protocol spec. http://www.fireconsole.org/ will be the next version of the FirePHP Firebug extension; I've read through the mailing list, and found some indications that there might be a new version of the Wildfire protocol spec soon.

In any case, getting a first version of this implemented would be very cool, and might just put some pressure on the Wildfire folks to stabilize things ASAP ;)

[Patrick Mueller]

Since FirePHP and Wildfire both support logging tabular information, it seems like we should support tabular output to the console directly, which FirePHP/etc can then use directly.  I've opened Bug 38664 to track this.

[Christoph Dorn]

Hi, I am the author of FirePHP, FireConsole and Wildfire.

I have been working on a reboot for FirePHP for a while now and am getting to a point where wider testing makes sense.

The main goal of the reboot was to split FirePHP into several projects that will allow for wider use.

The new setup includes:

  - FirePHP: Server library for PHP
  - FireConsole: Language agnostic firefox extension
  - Wildfire: Communication protocol
  - Insight: Multi-language object encoding and rendering library

At the lowest level a debugger such as Web Inspector would integrate wildfire (now a pure CommonJS package) and provide logging receivers that can be targeted from the server. The receiver dictates the format of the message and the simplest use-case would be to allow logging into the Web Inspector console.

The Insight library will sit on top of wildfire and provide a means to encode nested object graphs on the server (via language and class/framework specific encoders) and render these on the client via CommonJS modules. All clients need to do in order to render objects is to provide a CommonJS compliant environment for insight to run in and a DOM to render the result in. This functionality is currently being incubated as part of FireConsole and will be refactored into standalone libraries in the coming months.

The focus of wildfire is simply to transport ASCII text via headers (and other channels in future) from one component to another. The ideal place to start for Web Inspector would be to integrate wildfire and define one or more logging targets along with the expected format for the messages (probably json to begin with).

You can find the wildfire project here: http://github.com/cadorn/wildfire

There is currently a CommonJS implementation for sending and receiving and a PHP implementation for sending messages. If the component protocol is too specific by requiring a sender and receiver header one could easily define a new protocol that allows for message headers only where the target and message format is implied by the protocol.

The goal is to keep the number of protocols and message formats to a minimum to reduce the burden on server library and client implementers and encourage interoperability but there is no real prior art in this area that I am aware of and a certain amount of experimentation is inevitable. The primary purpose of wildfire in this regard is to provide a minimum framework for alternate protocols to evolve without conflict (i.e. allow them to co-exist in the same response)

[David Zuelke]

Hi Christoph,

The current state of the web inspector integration can be found at http://github.com/sthulbourn/WebKit-FirePHP

We too had the idea of implementing basic Wildfire support with handlers for different plugins/structures, so it could decode the headers and dispatch payloads to the responsible handlers (e.g. there would be a handler for http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1 that knows what the FirePHP payloads look like and how to render it).

As far as I understand, a bunch of headers could look like this:

X-Wf-Protocol-1     http://meta.wildfirehq.org/Protocol/JsonStream/0.2
X-Wf-1-Plugin-1     http://meta.firephp.org/Wildfire/Plugin/FirePHP/Library-FirePHPCore/0.3
X-Wf-1-Structure-1  http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1
X-Wf-1-1-1-1        62|[{"Type":"LOG","File":" ... Test.php","Line":3},"Hello World"]|
X-Wf-1-1-1-2        63|[{"Type":"LOG","File":" ... Test.php","Line":3},"Hello World2"]|
X-Wf-1-2-1-1        63|[{"Type":"LOG","File":" ... Test2.php","Line":3},"Hello World"]|
X-Wf-2-Plugin-1     http://lolcats.com/wf/transmitter
X-Wf-2-Structure-1  http://lolcats.com/wf/schema
X-Wf-2-1-1-1        7|[1,2,3]|

Correct? Last digit is the header sequence, second digit is the structure sequence, first digit is the plugin sequence? What is the third digit? And why couldn't the second digit be dropped so it looks like this instead (a bit offtopic; I guess this belongs to the fireconsole mailing list rather than to this discussion):
X-Wf-Protocol    http://meta.wildfirehq.org/Protocol/JsonStream/0.2
X-Wf-1-Plugin    http://meta.firephp.org/Wildfire/Plugin/FirePHP/Library-FirePHPCore/0.3
X-Wf-1-Structure http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1
X-Wf-1-1         62|[{"Type":"LOG","File":" ... Test.php","Line":3},"Hello World"]|
X-Wf-1-2         63|[{"Type":"LOG","File":" ... Test.php","Line":3},"Hello World2"]|
X-Wf-2-Plugin    http://meta.firephp.org/Wildfire/Plugin/FirePHP/Library-FirePHPCore/0.3
X-Wf-2-Structure http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1
X-Wf-2-1         63|[{"Type":"LOG","File":" ... Test2.php","Line":3},"Hello World"]|
X-Wf-3-Plugin    http://lolcats.com/wf/transmitter
X-Wf-3-Structure http://lolcats.com/wf/schema
X-Wf-3-1         7|[1,2,3]|

I'm still a bit unsure about the third digit and what exactly the difference between plugins and structures is (assuming right now that the plugin is simply for informational purposes, and the structure actually influences the message payload formats).

[Christoph Dorn]

(In reply to comment #5)
> The current state of the web inspector integration can be found at
> http://github.com/sthulbourn/WebKit-FirePHP

It should be trivial for you to splice in the wildfire JS library [1] to handle the header parsing. It would provide the greatest ongoing compatibility for the existing JsonStream and the more advanced Component protocol I am working on.

[1] - http://github.com/cadorn/wildfire/tree/master/packages/lib-js/


> We too had the idea of implementing basic Wildfire support with handlers for
> different plugins/structures, so it could decode the headers and dispatch
> payloads to the responsible handlers (e.g. there would be a handler for
> http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1 that
> knows what the FirePHP payloads look like and how to render it).

The JS library mentioned above provides the message routing for that use-case:

var receiver = WILDFIRE.Receiver();
receiver.setId("<Receiver URI which is typically a URL>");
receiver.addListener({
    onMessageReceived: function(context, message) {
        var meta = message.getMeta();
        var data = message.getData();
    }
});

var channel = WILDFIRE.HttpHeaderChannel();
channel.addReceiver(receiver);

// Run received headers through:
channel.parseReceived([
    { "name": "x-wf-...", "value": "..."},
    { "name": "...", "value": "..."}
], context);

A decoder then takes in the "meta" and "data" variables that contain the raw ASCII of the message and it looks like you already have that working for the FirePHP structures.


> As far as I understand, a bunch of headers could look like this:
> 
> X-Wf-Protocol-1     http://meta.wildfirehq.org/Protocol/JsonStream/0.2
> X-Wf-1-Plugin-1    
> http://meta.firephp.org/Wildfire/Plugin/FirePHP/Library-FirePHPCore/0.3
> X-Wf-1-Structure-1 
> http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1
> X-Wf-1-1-1-1        62|[{"Type":"LOG","File":" ... 

The message headers are composed as follows:

X-Wf-<protocol>-<structure>-<plugin>-<messageIndex>

The protocol is included for each message and specifies how the remaining parts are to be interpreted. i.e. anything after "X-Wf-<protocol>-"
This allows more than one protocol to coexist in the same set of headers.


> difference
> between plugins and structures is (assuming right now that the plugin is simply
> for informational purposes, and the structure actually influences the message
> payload formats).

Correct. The plugin is for informational purposes only and the structure determines the format of the message and the target. This aspect was not very well thought out in the original design which the new Component protocol fixes.

The Component protocol is structured as follows:

x-wf-protocol-1: http://registry.pinf.org/cadorn.org/wildfire/@meta/protocol/component/0.1.0
x-wf-1-1-receiver: http://pinf.org/cadorn.org/fireconsole
x-wf-1-1-1-sender: http://pinf.org/cadorn.org/wildfire/packages/lib-js
x-wf-1-1-1-1: 23|{"line":10}|Hello World|
x-wf-1-index: 1

The layout of the header names is easier to follow and a message is targeted at a receiver while disclosing the sender. For now a receiver ID is specific to a client and implies a certain format and target. I am not sure yet how that will play out in terms of server library compatibility between FireConsole, WebInspector and other clients.

To provide interoperability I am considering the following:

1) Use client agnostic receiverIDs such as http://registry.pinf.org/cadorn.org/wildfire/@meta/receiver/console/<version> once a common set of functionality and equivalent targets are implemented in several clients.

2) The client discloses available targets to the server library via request headers:

x-wf-protocol-1: http://registry.pinf.org/cadorn.org/wildfire/@meta/protocol/announce/0.1.0
x-wf-1: {"<receiverID>": {"supports": ["<supportID>"]}}

Where <supportID> is client agnostic and used by server libraries to determine the <receiverID> for the http://registry.pinf.org/cadorn.org/wildfire/@meta/receiver/console/log/<version> functionality for example.


All this protocol detail is taken care of by the wildfire JS library mentioned above which also has the capability to make a secondary request for logging data based on a key sent by the server instead of the logging data. This is a commonly requested feature to bypass limits proxies and some servers set on the size of headers and will be supported by the next version of the server libraries I am in the process of completing. I already use it with Google AppEngine.

[Antonin Hildebrand]

Hi, I'm the author of FireLogger. Hi Christoph! :-)

The protocol description is available here: 
http://wiki.github.com/darwin/firelogger

It is just an UTF-8 JSON encoded with BASE64 and broken into series of HTTP headers.

I don't have big plans with FireLogger, because Christoph is already working on FireConsole which should be probably the next great thing. And it should resolve many things I dislike about FirePHP.

The main reasons why I started FireLogger:
a) Python support for Google App Engine
b) javascript-like interface (you have simply flog(...) call which acts as a console.log)
c) simple JSON protocol, simpler code (PHP server-side library has less than 400LOC [1])
d) reusing firebug's console features to render structured data (server console on the left, DOM inspector aka "watches" on the right)
e) during the time we've got a few special features like  
  * Python Profiling
  * Server Paths Rewriting
  * AppStats support
  * Proxying server-side HTTP requests (you know forwarding logs from CURLed requests)
  * Password protection
  * Open in Text Editor support
  * etc.

Antonin
http://binaryage.com

[1] http://github.com/darwin/firelogger.php/blob/master/firelogger.php