Bug 30567 – add support for exporting HTTP Archive format in Resources panel

• Component: Web Inspector (Deprecated)
• Status: RESOLVED FIXED

[Patrick Mueller]

There is a new feature for browser debuggers called Http Archive format, used to format information such as what is displayed in the Resources panel, but in a standardized format.  Such information can then be used by other tools.

The format is described here:

   http://groups.google.com/group/firebug-working-group/web/http-tracing---export-format

I haven't looked into these other tools that support the data format, but it would be easiest to allow the data to be exported via clipboard and/or DnD.  If the tools only support files, we'll have to add support for saving the data to a file.

[Timothy Hatcher]

Thi would be nice to support. It might be tricky for Chrome to support, since the process the Inspector runs in can't save files to disk.

[Timothy Hatcher]

I think exporting the format should be a seperate task (bug) from reading it in and displaying it. Saving will be easier than displaying.

[Pavel Feldman]

Now that we've serialized the interaction, I actually think this could be a protocol for inspector controller -> inspector frontend interaction. Notice the payload that is being sent into WebInspector.add(update)Resource. This payload could actually conform to this spec and could be stored on the frontend.

In other words: I don't see much of the value in exporting HTTP stuff in a standard way (one can use sniffer for that), but I see the value in interchanging according to the standard. That way we could provide a nice frontend-side API for the 'audits' should they need it. What do you think?

[Timothy Hatcher]

Using this for interchange would be neat, and handy since we want a JSON format.

I also don't see much value in exporting it. I see more value in having a record and playback mode for the network loads, like Chrome has. But maybe this format has enough data for that? What format does Chrome use for it's network load recording?

[Anthony Ricaud]

I don't understand the difference between "export" and "record".

I see value in exporting since you don't have to use another tool.

[Timothy Hatcher]

<rdar://problem/7463672>

[Pavel Feldman]

As we work on the extensions for the web inspector, we feel the need for some solid http-related API we could commit to. Currently, I don't see alternatives to HAR. I am sure extension developers will appreciate exposing it too since they will be able to reuse their code. Here is a recent spec we could target.

http://groups.google.com/group/http-archive-specification/web/har-1-2-spec

There are obvious missing pieces such as detailed timing and cookies. We can look into supporting them in the Chromium's stack as a start.

[Patrick Mueller]

(In reply to comment #7)
> As we work on the extensions for the web inspector, we feel the need for some solid http-related API we could commit to. Currently, I don't see alternatives to HAR. I am sure extension developers will appreciate exposing it too since they will be able to reuse their code. Here is a recent spec we could target.
> 
> http://groups.google.com/group/http-archive-specification/web/har-1-2-spec
> 
> There are obvious missing pieces such as detailed timing and cookies. We can look into supporting them in the Chromium's stack as a start.

It sounds like you'd like to allow extensions to read HAR data directly.  But not do a HAR exporter (or importer).  The obvious suggestion would then be to build a HAR exporter/importer as an extension, if someone needs one.

In comment #3 you mention also as using it internally, between the front-end and controller.  Still thinking of doing that also?  Or maybe that's implied somehow by having it available as an API for extensions to use.

What kind of API are you thinking about?  Could be something like (as JS functions):

   void WIX.HarStartCollecting()
   HarObject WIX.HarStopCollecting()

It seems like a typical thing to do would be to start collecting for a single page or session, from the very beginning.    Maybe a way to start collecting on the next page transition or something (a little wordy):

   void WIX.HarStartCollectingOnNextTransition()

[Andrey Kosyakov]

Created attachment 57474 [details]
patch

- Moved common resource field accessors from ResourceView to Resource
- Added converter to HAR

[WebKit Review Bot]

Attachment 57474 [details] did not pass style-queue:

Failed to run "['WebKitTools/Scripts/check-webkit-style', '--no-squash']" exit_code: 1
WebCore/inspector/front-end/ResourceHARConverter.js:1:  One or more unexpected \r (^M) found; better to use only a \n  [whitespace/carriage_return] [1]
Suppressing further [whitespace/carriage_return] reports for this file.
Total errors found: 138 in 8 files


If any of these errors are false positives, please file a bug against check-webkit-style.

[Pavel Feldman]

Comment on attachment 57474 [details]
patch

1. I would convert it to a prototype-based object with following usage: new WebInspector.HAREntry(resource).build();
2. I'd also like to see a test for it.

WebCore/inspector/front-end/Resource.js:90
 +          delete this._queryParameters;
Consider renaming to this._parsedQueryParameters

WebCore/inspector/front-end/Resource.js:395
 +          return WebInspector.Resource.headerValue(this.requestHeaders, headerName);
consider making private instance method

WebCore/inspector/front-end/Resource.js:406
 +          delete this._formParameters;
this._parsedFormParameters

WebCore/inspector/front-end/Resource.js:452
 +          return this._queryParameters = WebInspector.Resource.parseParameters(queryString);
Also private instance method.

[Andrey Kosyakov]

Created attachment 57535 [details]
patch

- Renamed ResourceHARConverter to HAREntry, moved all methods to prototype.
- Added tests.

[Early Warning System Bot]

Attachment 57535 [details] did not build on qt:
Build output: http://webkit-commit-queue.appspot.com/results/2778055

[Andrey Kosyakov]

Created attachment 57538 [details]
patch (added HAREntry.js itself)

[Pavel Feldman]

Comment on attachment 57538 [details]
patch (added HAREntry.js itself)

WebCore/ChangeLog:5
 +          Added conversion of inspector's resource representation into HAR.
Web Inspector: Added ...


LayoutTests/ChangeLog:18
 +  2010-06-01  Andrey Kosyakov  <[e-mail]>
You don't need two log entries.

LayoutTests/ChangeLog:11
 +          * http/tests/inspector/resource-har-conversion.html: Added.
You should also land expectations.

LayoutTests/http/tests/inspector/inspector-test.js:47
 +          if (prop in (nondeterministicProps || {}))
if (nondeterministicProps && prop in nondeterministicProps)

LayoutTests/http/tests/inspector/inspector-test.js:54
 +              output(prefixWithName  + propValue);
extra space

WebCore/inspector/front-end/HAREntry.js:40
 +  {
Place { on the same line as =

WebCore/inspector/front-end/Resource.js:452
 +          return this._parsedQueryParameters = this._parseParameters(queryString);
Split this into two lines please.

WebCore/inspector/front-end/Resource.js:464
 +          return this._parsedFormParameters = this._parseParameters(this.requestFormData);
ditto

WebCore/inspector/front-end/ResourceView.js:210
 +          this.queryStringTreeElement.hidden = queryParameters == null;
!queryParameters

LayoutTests/http/tests/inspector/resource-har-conversion.html:10
 +      notifyDone();
You should not notifyDone synchronously in case dumpResources is asynchronous.

LayoutTests/http/tests/inspector/resource-tests.js:11
 +  function dumpResources() {
I doubt this one does not deserve to be generic.

[Andrey Kosyakov]

Created attachment 57549 [details]
patch

[Joseph Pecoraro]

Comment on attachment 57549 [details]
patch

A few style nits.

> +        if (this._endTime === -1 || this._responseReceivedTime === -1 )

Extra space at the end should be removed.


> +            if (res.length == 1)

I know this is code that is moving, but we normally us ===.


Otherwise it looks great. I haven't had any experience with HAR, so
I look forward to playing with this.

[WebKit Commit Bot]

Comment on attachment 57549 [details]
patch

Clearing flags on attachment: 57549

Committed r60530: <http://trac.webkit.org/changeset/60530>

[WebKit Commit Bot]

All reviewed patches have been landed.  Closing bug.

[WebKit Review Bot]

http://trac.webkit.org/changeset/60530 might have broken GTK Linux 32-bit Release
The following changes are on the blame list:
http://trac.webkit.org/changeset/60528
http://trac.webkit.org/changeset/60529
http://trac.webkit.org/changeset/60530

[Adam Barth]

Reopening because this broke Gtk:
http://trac.webkit.org/changeset/60535

[Pavel Feldman]

See the diff below. I'd suggest that we add this test to GTK's Skipped list unless it provides valid resource sizes to the clients. See LayoutTests/platform/gtk/Skipped.

I wonder if the resource panel functions well under WebKit GTK. I guess it is not and the test uncovers it. Go tests!

--- /var/lib/buildbot/build/gtk-linux-32-release/build/layout-test-results/http/tests/inspector/resource-har-conversion-expected.txt	2010-06-01 20:33:11.000000000 -0700
+++ /var/lib/buildbot/build/gtk-linux-32-release/build/layout-test-results/http/tests/inspector/resource-har-conversion-actual.txt	2010-06-01 20:33:11.000000000 -0700
@@ -16,12 +16,12 @@
 +-- statusText : "OK"
 +-- headers : object
 +-- content : {
-+--- size : 1010
++--- size : 0
 +--- mimeType : "text/html"
 +-- }
 +-- redirectURL : ""
 +-- headersSize : -1
-+-- bodySize : 1010
++-- bodySize : 0
 +- }
 +- timings : {
 +-- blocked : -1

[Andrey Kosyakov]

Created attachment 57639 [details]
patch (added test to gtk/skipped)

[Pavel Feldman]

Committing to http://svn.webkit.org/repository/webkit/trunk ...
	M	LayoutTests/ChangeLog
	M	LayoutTests/http/tests/inspector/inspector-test.js
	A	LayoutTests/http/tests/inspector/resource-har-conversion-expected.txt
	A	LayoutTests/http/tests/inspector/resource-har-conversion.html
	M	LayoutTests/platform/gtk/Skipped
	M	WebCore/ChangeLog
	M	WebCore/WebCore.gypi
	M	WebCore/WebCore.vcproj/WebCore.vcproj
	A	WebCore/inspector/front-end/HAREntry.js
	M	WebCore/inspector/front-end/Resource.js
	M	WebCore/inspector/front-end/ResourceView.js
	M	WebCore/inspector/front-end/WebKit.qrc
	M	WebCore/inspector/front-end/inspector.html
Committed r60555

[Patrick Mueller]

In looking at the final patch, I don't see anything about "exporting" HAR entries anywhere.  I see "API" ish stuff to get a HAREntry from a resource.

So who can use this stuff?  Presumably extensions, once we support extensions?  Or should we open a new bug to provide a mechanism to export HAREntry's from the existing resources available in a Web Inspector session.

Perhaps we should rename the bug as well, to remove the mention of exporting, since it's not actually exporting anything.

[Andrey Kosyakov]

(In reply to comment #25)
> In looking at the final patch, I don't see anything about "exporting" HAR entries anywhere.  I see "API" ish stuff to get a HAREntry from a resource.
> 
> So who can use this stuff?  Presumably extensions, once we support extensions?  

Exactly -- the proposed extension API implies resources are exposed to extensions in the form of HAREntry (https://bugs.webkit.org/show_bug.cgi?id=40425). It also happens to be quite handy for the tests, as we'd like to set test expectations in a stable format.

> Or should we open a new bug to provide a mechanism to export HAREntry's from the existing resources > available in a Web Inspector session.

I'd rather leave it to extensions, unless we have a sufficiently common use case for that (I don't see one yet).

[Eitan Konigsburg]

(In reply to comment #26)
> 
> > Or should we open a new bug to provide a mechanism to export HAREntry's from the existing resources > available in a Web Inspector session.
> 
> I'd rather leave it to extensions, unless we have a sufficiently common use case for that (I don't see one yet).

+1 for the export option.

HAR is an archive format - the whole point is to be able to archive a particular session.  Each browser has it's own way of loading resources, so having a common format for comparison and analysis is extremely useful.  HAR files can be loaded into other tools for analysis based on some custom ruleset, or entered into a database for tracking.