Bug 30567 - add support for exporting HTTP Archive format in Resources panel
Summary: add support for exporting HTTP Archive format in Resources panel
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Web Inspector (Deprecated) (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Enhancement
Assignee: Nobody
URL:
Keywords: InRadar
Depends on: 40041
Blocks:
  Show dependency treegraph
 
Reported: 2009-10-20 06:23 PDT by Patrick Mueller
Modified: 2010-07-19 23:04 PDT (History)
15 users (show)

See Also:


Attachments
patch (16.62 KB, patch)
2010-05-31 09:02 PDT, Andrey Kosyakov
no flags Details | Formatted Diff | Diff
patch (13.73 KB, patch)
2010-06-01 06:36 PDT, Andrey Kosyakov
no flags Details | Formatted Diff | Diff
patch (added HAREntry.js itself) (20.27 KB, patch)
2010-06-01 06:50 PDT, Andrey Kosyakov
pfeldman: review-
Details | Formatted Diff | Diff
patch (20.58 KB, patch)
2010-06-01 08:52 PDT, Andrey Kosyakov
no flags Details | Formatted Diff | Diff
patch (added test to gtk/skipped) (21.22 KB, patch)
2010-06-02 02:54 PDT, Andrey Kosyakov
pfeldman: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Mueller 2009-10-20 06:23:01 PDT
There is a new feature for browser debuggers called Http Archive format, used to format information such as what is displayed in the Resources panel, but in a standardized format.  Such information can then be used by other tools.

The format is described here:

   http://groups.google.com/group/firebug-working-group/web/http-tracing---export-format

I haven't looked into these other tools that support the data format, but it would be easiest to allow the data to be exported via clipboard and/or DnD.  If the tools only support files, we'll have to add support for saving the data to a file.
Comment 1 Timothy Hatcher 2009-10-20 17:22:03 PDT
Thi would be nice to support. It might be tricky for Chrome to support, since the process the Inspector runs in can't save files to disk.
Comment 2 Timothy Hatcher 2009-10-20 17:23:23 PDT
I think exporting the format should be a seperate task (bug) from reading it in and displaying it. Saving will be easier than displaying.
Comment 3 Pavel Feldman 2009-10-20 18:56:06 PDT
Now that we've serialized the interaction, I actually think this could be a protocol for inspector controller -> inspector frontend interaction. Notice the payload that is being sent into WebInspector.add(update)Resource. This payload could actually conform to this spec and could be stored on the frontend.

In other words: I don't see much of the value in exporting HTTP stuff in a standard way (one can use sniffer for that), but I see the value in interchanging according to the standard. That way we could provide a nice frontend-side API for the 'audits' should they need it. What do you think?
Comment 4 Timothy Hatcher 2009-10-20 20:17:51 PDT
Using this for interchange would be neat, and handy since we want a JSON format.

I also don't see much value in exporting it. I see more value in having a record and playback mode for the network loads, like Chrome has. But maybe this format has enough data for that? What format does Chrome use for it's network load recording?
Comment 5 Anthony Ricaud 2009-10-22 02:18:05 PDT
I don't understand the difference between "export" and "record".

I see value in exporting since you don't have to use another tool.
Comment 6 Timothy Hatcher 2009-12-14 12:53:06 PST
<rdar://problem/7463672>
Comment 7 Pavel Feldman 2010-05-27 13:11:12 PDT
As we work on the extensions for the web inspector, we feel the need for some solid http-related API we could commit to. Currently, I don't see alternatives to HAR. I am sure extension developers will appreciate exposing it too since they will be able to reuse their code. Here is a recent spec we could target.

http://groups.google.com/group/http-archive-specification/web/har-1-2-spec

There are obvious missing pieces such as detailed timing and cookies. We can look into supporting them in the Chromium's stack as a start.
Comment 8 Patrick Mueller 2010-05-27 15:05:19 PDT
(In reply to comment #7)
> As we work on the extensions for the web inspector, we feel the need for some solid http-related API we could commit to. Currently, I don't see alternatives to HAR. I am sure extension developers will appreciate exposing it too since they will be able to reuse their code. Here is a recent spec we could target.
> 
> http://groups.google.com/group/http-archive-specification/web/har-1-2-spec
> 
> There are obvious missing pieces such as detailed timing and cookies. We can look into supporting them in the Chromium's stack as a start.

It sounds like you'd like to allow extensions to read HAR data directly.  But not do a HAR exporter (or importer).  The obvious suggestion would then be to build a HAR exporter/importer as an extension, if someone needs one.

In comment #3 you mention also as using it internally, between the front-end and controller.  Still thinking of doing that also?  Or maybe that's implied somehow by having it available as an API for extensions to use.

What kind of API are you thinking about?  Could be something like (as JS functions):

   void WIX.HarStartCollecting()
   HarObject WIX.HarStopCollecting()

It seems like a typical thing to do would be to start collecting for a single page or session, from the very beginning.    Maybe a way to start collecting on the next page transition or something (a little wordy):

   void WIX.HarStartCollectingOnNextTransition()
Comment 9 Andrey Kosyakov 2010-05-31 09:02:18 PDT
Created attachment 57474 [details]
patch

- Moved common resource field accessors from ResourceView to Resource
- Added converter to HAR
Comment 10 WebKit Review Bot 2010-05-31 09:03:29 PDT
Attachment 57474 [details] did not pass style-queue:

Failed to run "['WebKitTools/Scripts/check-webkit-style', '--no-squash']" exit_code: 1
WebCore/inspector/front-end/ResourceHARConverter.js:1:  One or more unexpected \r (^M) found; better to use only a \n  [whitespace/carriage_return] [1]
Suppressing further [whitespace/carriage_return] reports for this file.
Total errors found: 138 in 8 files


If any of these errors are false positives, please file a bug against check-webkit-style.
Comment 11 Pavel Feldman 2010-05-31 10:15:32 PDT
Comment on attachment 57474 [details]
patch

1. I would convert it to a prototype-based object with following usage: new WebInspector.HAREntry(resource).build();
2. I'd also like to see a test for it.

WebCore/inspector/front-end/Resource.js:90
 +          delete this._queryParameters;
Consider renaming to this._parsedQueryParameters

WebCore/inspector/front-end/Resource.js:395
 +          return WebInspector.Resource.headerValue(this.requestHeaders, headerName);
consider making private instance method

WebCore/inspector/front-end/Resource.js:406
 +          delete this._formParameters;
this._parsedFormParameters

WebCore/inspector/front-end/Resource.js:452
 +          return this._queryParameters = WebInspector.Resource.parseParameters(queryString);
Also private instance method.
Comment 12 Andrey Kosyakov 2010-06-01 06:36:11 PDT
Created attachment 57535 [details]
patch

- Renamed ResourceHARConverter to HAREntry, moved all methods to prototype.
- Added tests.
Comment 13 Early Warning System Bot 2010-06-01 06:42:10 PDT
Attachment 57535 [details] did not build on qt:
Build output: http://webkit-commit-queue.appspot.com/results/2778055
Comment 14 Andrey Kosyakov 2010-06-01 06:50:49 PDT
Created attachment 57538 [details]
patch (added HAREntry.js itself)
Comment 15 Pavel Feldman 2010-06-01 07:42:35 PDT
Comment on attachment 57538 [details]
patch (added HAREntry.js itself)

WebCore/ChangeLog:5
 +          Added conversion of inspector's resource representation into HAR.
Web Inspector: Added ...


LayoutTests/ChangeLog:18
 +  2010-06-01  Andrey Kosyakov  <caseq@chromium.org>
You don't need two log entries.

LayoutTests/ChangeLog:11
 +          * http/tests/inspector/resource-har-conversion.html: Added.
You should also land expectations.

LayoutTests/http/tests/inspector/inspector-test.js:47
 +          if (prop in (nondeterministicProps || {}))
if (nondeterministicProps && prop in nondeterministicProps)

LayoutTests/http/tests/inspector/inspector-test.js:54
 +              output(prefixWithName  + propValue);
extra space

WebCore/inspector/front-end/HAREntry.js:40
 +  {
Place { on the same line as =

WebCore/inspector/front-end/Resource.js:452
 +          return this._parsedQueryParameters = this._parseParameters(queryString);
Split this into two lines please.

WebCore/inspector/front-end/Resource.js:464
 +          return this._parsedFormParameters = this._parseParameters(this.requestFormData);
ditto

WebCore/inspector/front-end/ResourceView.js:210
 +          this.queryStringTreeElement.hidden = queryParameters == null;
!queryParameters

LayoutTests/http/tests/inspector/resource-har-conversion.html:10
 +      notifyDone();
You should not notifyDone synchronously in case dumpResources is asynchronous.

LayoutTests/http/tests/inspector/resource-tests.js:11
 +  function dumpResources() {
I doubt this one does not deserve to be generic.
Comment 16 Andrey Kosyakov 2010-06-01 08:52:35 PDT
Created attachment 57549 [details]
patch
Comment 17 Joseph Pecoraro 2010-06-01 09:19:21 PDT
Comment on attachment 57549 [details]
patch

A few style nits.

> +        if (this._endTime === -1 || this._responseReceivedTime === -1 )

Extra space at the end should be removed.


> +            if (res.length == 1)

I know this is code that is moving, but we normally us ===.


Otherwise it looks great. I haven't had any experience with HAR, so
I look forward to playing with this.
Comment 18 WebKit Commit Bot 2010-06-01 20:15:36 PDT
Comment on attachment 57549 [details]
patch

Clearing flags on attachment: 57549

Committed r60530: <http://trac.webkit.org/changeset/60530>
Comment 19 WebKit Commit Bot 2010-06-01 20:15:43 PDT
All reviewed patches have been landed.  Closing bug.
Comment 20 WebKit Review Bot 2010-06-01 20:44:57 PDT
http://trac.webkit.org/changeset/60530 might have broken GTK Linux 32-bit Release
The following changes are on the blame list:
http://trac.webkit.org/changeset/60528
http://trac.webkit.org/changeset/60529
http://trac.webkit.org/changeset/60530
Comment 21 Adam Barth 2010-06-01 22:06:26 PDT
Reopening because this broke Gtk:
http://trac.webkit.org/changeset/60535
Comment 22 Pavel Feldman 2010-06-02 00:19:45 PDT
See the diff below. I'd suggest that we add this test to GTK's Skipped list unless it provides valid resource sizes to the clients. See LayoutTests/platform/gtk/Skipped.

I wonder if the resource panel functions well under WebKit GTK. I guess it is not and the test uncovers it. Go tests!

--- /var/lib/buildbot/build/gtk-linux-32-release/build/layout-test-results/http/tests/inspector/resource-har-conversion-expected.txt	2010-06-01 20:33:11.000000000 -0700
+++ /var/lib/buildbot/build/gtk-linux-32-release/build/layout-test-results/http/tests/inspector/resource-har-conversion-actual.txt	2010-06-01 20:33:11.000000000 -0700
@@ -16,12 +16,12 @@
 +-- statusText : "OK"
 +-- headers : object
 +-- content : {
-+--- size : 1010
++--- size : 0
 +--- mimeType : "text/html"
 +-- }
 +-- redirectURL : ""
 +-- headersSize : -1
-+-- bodySize : 1010
++-- bodySize : 0
 +- }
 +- timings : {
 +-- blocked : -1
Comment 23 Andrey Kosyakov 2010-06-02 02:54:32 PDT
Created attachment 57639 [details]
patch (added test to gtk/skipped)
Comment 24 Pavel Feldman 2010-06-02 03:17:24 PDT
Committing to http://svn.webkit.org/repository/webkit/trunk ...
	M	LayoutTests/ChangeLog
	M	LayoutTests/http/tests/inspector/inspector-test.js
	A	LayoutTests/http/tests/inspector/resource-har-conversion-expected.txt
	A	LayoutTests/http/tests/inspector/resource-har-conversion.html
	M	LayoutTests/platform/gtk/Skipped
	M	WebCore/ChangeLog
	M	WebCore/WebCore.gypi
	M	WebCore/WebCore.vcproj/WebCore.vcproj
	A	WebCore/inspector/front-end/HAREntry.js
	M	WebCore/inspector/front-end/Resource.js
	M	WebCore/inspector/front-end/ResourceView.js
	M	WebCore/inspector/front-end/WebKit.qrc
	M	WebCore/inspector/front-end/inspector.html
Committed r60555
Comment 25 Patrick Mueller 2010-06-11 09:10:02 PDT
In looking at the final patch, I don't see anything about "exporting" HAR entries anywhere.  I see "API" ish stuff to get a HAREntry from a resource.

So who can use this stuff?  Presumably extensions, once we support extensions?  Or should we open a new bug to provide a mechanism to export HAREntry's from the existing resources available in a Web Inspector session.

Perhaps we should rename the bug as well, to remove the mention of exporting, since it's not actually exporting anything.
Comment 26 Andrey Kosyakov 2010-06-11 14:38:06 PDT
(In reply to comment #25)
> In looking at the final patch, I don't see anything about "exporting" HAR entries anywhere.  I see "API" ish stuff to get a HAREntry from a resource.
> 
> So who can use this stuff?  Presumably extensions, once we support extensions?  

Exactly -- the proposed extension API implies resources are exposed to extensions in the form of HAREntry (https://bugs.webkit.org/show_bug.cgi?id=40425). It also happens to be quite handy for the tests, as we'd like to set test expectations in a stable format.

> Or should we open a new bug to provide a mechanism to export HAREntry's from the existing resources > available in a Web Inspector session.

I'd rather leave it to extensions, unless we have a sufficiently common use case for that (I don't see one yet).
Comment 27 Eitan Konigsburg 2010-07-19 23:04:02 PDT
(In reply to comment #26)
> 
> > Or should we open a new bug to provide a mechanism to export HAREntry's from the existing resources > available in a Web Inspector session.
> 
> I'd rather leave it to extensions, unless we have a sufficiently common use case for that (I don't see one yet).

+1 for the export option.

HAR is an archive format - the whole point is to be able to archive a particular session.  Each browser has it's own way of loading resources, so having a common format for comparison and analysis is extremely useful.  HAR files can be loaded into other tools for analysis based on some custom ruleset, or entered into a database for tracking.