Bug 146692 - Web Inspector: Please support HAR Export for network traffic
Summary: Web Inspector: Please support HAR Export for network traffic
Status: RESOLVED FIXED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Web Inspector (show other bugs)
Version: 528+ (Nightly build)
Hardware: All All
: P2 Normal
Assignee: Joseph Pecoraro
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2015-07-07 13:24 PDT by Eric Lawrence
Modified: 2017-10-24 13:53 PDT (History)
18 users (show)

See Also:


Attachments
[PATCH] Proposed Fix (69.78 KB, patch)
2017-10-22 20:11 PDT, Joseph Pecoraro
bburg: review+
Details | Formatted Diff | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Eric Lawrence 2015-07-07 13:24:21 PDT
HTTP Archive format is the industry standard for traffic export; Chrome, MS Edge, Fiddler, and Firebug all support HAR files. 

It looks like this was supported previously (https://bugs.webkit.org/show_bug.cgi?id=56097) but it isn't any longer per https://twitter.com/xeenon/status/618514694442229760
Comment 1 Radar WebKit Bug Importer 2015-07-07 13:24:58 PDT
<rdar://problem/21710448>
Comment 2 Iraê 2015-08-11 13:33:46 PDT
We often use OS-level support for Link Conditioner, as explained here: http://nshipster.com/network-link-conditioner/

It would be a nice addition to have this feature implemented in WebKit inspector for a number of reasons:


1. It only affects the current tab in the current instance of Webkit, so it does not affect tasks that the developer might be doing in other applications and tabs
2. It makes clear in the UI what is being debugged at the current time
3. It allows side-by side comparison when using multiple windows for debugging network performance issues (different sites, the same site in different conditions, different AB/testing and all permutations)

Chrome already implements it and this could be considered also for developer relation improvements. I use Safari as my main development browser but many co-workers would choose Chrome for this feature amongst others.
Comment 3 Eric Lawrence 2015-08-11 13:49:20 PDT
(In reply to comment #2)
> We often use OS-level support for Link Conditioner, as explained here:
> http://nshipster.com/network-link-conditioner/
> 
> It would be a nice addition to have this feature implemented in WebKit
> inspector for a number of reasons:
> 
> 
> 1. It only affects the current tab in the current instance of Webkit, so it
> does not affect tasks that the developer might be doing in other
> applications and tabs
> 2. It makes clear in the UI what is being debugged at the current time
> 3. It allows side-by side comparison when using multiple windows for
> debugging network performance issues (different sites, the same site in
> different conditions, different AB/testing and all permutations)
> 
> Chrome already implements it and this could be considered also for developer
> relation improvements. I use Safari as my main development browser but many
> co-workers would choose Chrome for this feature amongst others.

Iraê-- This doesn't appear to have anything to do with HTTP Archive Support. If not, you should file a new issue, not bury your comment inside support for HAR.
Comment 4 Yahoo Search Mobile Team 2015-08-11 16:21:54 PDT
I think something when wrong with the POST request or I messed up my tabs. I understand it's not related, I was a mistake. Thanks for posting that out.
Comment 5 Iraê 2015-08-11 18:32:10 PDT
I did mess up with posting the wrong description here, but I would like to say that HAR support is a helpful feature, not only by exporting through the UI, but also if we could get it programmatically. It's not part of the WebDriver bug request (https://bugs.webkit.org/show_bug.cgi?id=135263), but it is also a good tool for automating performance tests.

Having programmatic access, via command-line/launcher of via web APIs (something like Resource Timings (https://bugs.webkit.org/show_bug.cgi?id=61138) might be translatable to HAR, but HAR would be easier to integrate).
Comment 6 Timothy Hatcher 2015-08-25 18:55:19 PDT
<rdar://problem/7463672>
Comment 7 Aral Balkan 2016-04-04 06:34:05 PDT
This would also be helpful for content blockers.
Comment 8 BJ Burg 2017-09-01 10:55:21 PDT
<rdar://problem/7463672>
Comment 9 Joseph Pecoraro 2017-10-04 14:44:31 PDT
HTTP Archive (HAR) format specication:
https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HAR/Overview.html

Older version for reference:
http://www.softwareishard.com/blog/har-12-spec/

Other Resources that maybe useful:
http://www.softwareishard.com/har/viewer/
https://toolbox.googleapps.com/apps/har_analyzer/
Comment 10 Joseph Pecoraro 2017-10-22 20:11:19 PDT
Created attachment 324538 [details]
[PATCH] Proposed Fix
Comment 11 Joseph Pecoraro 2017-10-22 20:12:19 PDT
I'm missing an export button right now.

However this match the main way other browsers include a HAR export (an option in the context menu).

We also make ⌘S in the Network Tab default to this behavior.
Comment 12 BJ Burg 2017-10-23 13:54:46 PDT
Comment on attachment 324538 [details]
[PATCH] Proposed Fix

View in context: https://bugs.webkit.org/attachment.cgi?id=324538&action=review

r=me

> LayoutTests/http/tests/inspector/network/har/har-basic-expected.txt:37
> +        "title": "http://127.0.0.1:8000/inspector/network/har/har-basic.html",

Is loopback hardcoded in other tests like this?

> LayoutTests/http/tests/inspector/network/har/har-basic.html:24
> +            let har = await WI.HARBuilder.buildArchive([]);

Very interesting. I haven't read much async/await code.

> Source/WebInspectorUI/ChangeLog:63
> +        Capture the raw, unmodified, base64 encoded and content. This ends up

Nit: -and

> Source/WebInspectorUI/UserInterface/Controllers/HARBuilder.js:32
> +    static async buildArchive(resources)

I found it a little weird that this is a singleton and all static methods, but I guess it needs to use a ton of other global state anyway.

> Source/WebInspectorUI/UserInterface/Controllers/HARBuilder.js:235
> +        // FIXME: Implement HAR cache data.

Please file a bug for this.

> Source/WebInspectorUI/UserInterface/Controllers/HARBuilder.js:242
> +        // NOTE: Chrome Custom Fields `_blocked_queueing` and `_blocked_proxy`.

Weird grammar.

> Source/WebInspectorUI/UserInterface/Views/NetworkTableContentView.js:1386
> +        if (!resources.length) {

Maybe it would be more obvious when exporting is not possible, by making the button become disabled when this condition is true.
Comment 13 Joseph Pecoraro 2017-10-23 13:59:33 PDT
Comment on attachment 324538 [details]
[PATCH] Proposed Fix

View in context: https://bugs.webkit.org/attachment.cgi?id=324538&action=review

>> LayoutTests/http/tests/inspector/network/har/har-basic-expected.txt:37
>> +        "title": "http://127.0.0.1:8000/inspector/network/har/har-basic.html",
> 
> Is loopback hardcoded in other tests like this?

This is an http/tests/ LayoutTest and those are always 127.0.0.1:8000

>> Source/WebInspectorUI/UserInterface/Views/NetworkTableContentView.js:1386
>> +        if (!resources.length) {
> 
> Maybe it would be more obvious when exporting is not possible, by making the button become disabled when this condition is true.

We currently don't have a button, this path is taken via a ⌘S keyboard shortcut.
Comment 14 Joseph Pecoraro 2017-10-23 14:35:34 PDT
<https://webkit.org/b/146692>
Comment 15 Ryan Haddad 2017-10-23 17:42:21 PDT
One of the LayoutTests for this change is failing:
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r223860%20(5199)/results.html

--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/tests/inspector/network/har/har-page-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/tests/inspector/network/har/har-page-actual.txt
@@ -50,8 +50,7 @@
           },
           "redirectURL": "",
           "headersSize": "<filtered>",
-          "bodySize": "<filtered>",
-          "_transferSize": "<filtered>"
+          "bodySize": "<filtered>"
         },
         "cache": {},
         "timings": {
@@ -63,8 +62,6 @@
           "wait": "<filtered>",
           "receive": "<filtered>"
         },
-        "serverIPAddress": "127.0.0.1",
-        "connection": "<filtered>",
         "_fetchType": "<filtered>"
       },
       {
Comment 16 BJ Burg 2017-10-23 17:54:04 PDT
(In reply to Ryan Haddad from comment #15)
> One of the LayoutTests for this change is failing:
> https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/
> r223860%20(5199)/results.html
> 
> ---
> /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/
> tests/inspector/network/har/har-page-expected.txt
> +++
> /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/
> tests/inspector/network/har/har-page-actual.txt
> @@ -50,8 +50,7 @@
>            },
>            "redirectURL": "",
>            "headersSize": "<filtered>",
> -          "bodySize": "<filtered>",
> -          "_transferSize": "<filtered>"
> +          "bodySize": "<filtered>"
>          },
>          "cache": {},
>          "timings": {
> @@ -63,8 +62,6 @@
>            "wait": "<filtered>",
>            "receive": "<filtered>"
>          },
> -        "serverIPAddress": "127.0.0.1",
> -        "connection": "<filtered>",
>          "_fetchType": "<filtered>"
>        },
>        {

These fields are only available for iOS 11 / High Sierra or later. We just need a new baseline for Sierra.
Comment 17 Joseph Pecoraro 2017-10-23 18:29:43 PDT
> These fields are only available for iOS 11 / High Sierra or later. We just
> need a new baseline for Sierra.

These fields should be available on Sierra.

I guess in the test they may be variable based on whether or not the resource was served from a cache. I may just have to eliminate these fields to avoid flakiness.
Comment 18 Joseph Pecoraro 2017-10-23 18:45:27 PDT
I updated the test to try to reduce flakiness:
<https://trac.webkit.org/changeset/223873/webkit>
Comment 19 Ryan Haddad 2017-10-24 09:58:46 PDT
(In reply to Joseph Pecoraro from comment #18)
> I updated the test to try to reduce flakiness:
> <https://trac.webkit.org/changeset/223873/webkit>
The test still appears to be failing: https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r223885%20(5216)/results.html
Comment 20 Joseph Pecoraro 2017-10-24 10:57:40 PDT
(In reply to Ryan Haddad from comment #19)
> (In reply to Joseph Pecoraro from comment #18)
> > I updated the test to try to reduce flakiness:
> > <https://trac.webkit.org/changeset/223873/webkit>
> The test still appears to be failing:
> https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/
> r223885%20(5216)/results.html

That failure looks like it would be before my changes... unless I didn't reset the results of the test properly
Comment 21 Joseph Pecoraro 2017-10-24 10:58:44 PDT
> That failure looks like it would be before my changes... unless I didn't
> reset the results of the test properly

It appears I may not have reset the results properly since I do see a "connection" property. Let me try again.
Comment 22 Joseph Pecoraro 2017-10-24 11:05:25 PDT
<https://trac.webkit.org/r223899>
Comment 24 Joseph Pecoraro 2017-10-24 13:38:47 PDT
(In reply to Matt Lewis from comment #23)
> The test looks like it is still failing:
> https://build.webkit.org/results/
> Apple%20High%20Sierra%20Release%20WK2%20(Tests)/r223900%20(665)/results.html
> https://build.webkit.org/builders/
> Apple%20High%20Sierra%20Release%20WK2%20(Tests)/builds/665

Oops, I removed whitespace before landing and forgot to update the results with the new size.
Comment 25 Joseph Pecoraro 2017-10-24 13:53:32 PDT
Fingers crossed!

<https://trac.webkit.org/r223917>