Bug 146692 – Web Inspector: Please support HAR Export for network traffic

• Component: Web Inspector
• Status: RESOLVED FIXED

[Eric Lawrence]

HTTP Archive format is the industry standard for traffic export; Chrome, MS Edge, Fiddler, and Firebug all support HAR files. 

It looks like this was supported previously (https://bugs.webkit.org/show_bug.cgi?id=56097) but it isn't any longer per https://twitter.com/xeenon/status/618514694442229760

[Radar WebKit Bug Importer]

<rdar://problem/21710448>

[Iraê]

We often use OS-level support for Link Conditioner, as explained here: http://nshipster.com/network-link-conditioner/

It would be a nice addition to have this feature implemented in WebKit inspector for a number of reasons:


1. It only affects the current tab in the current instance of Webkit, so it does not affect tasks that the developer might be doing in other applications and tabs
2. It makes clear in the UI what is being debugged at the current time
3. It allows side-by side comparison when using multiple windows for debugging network performance issues (different sites, the same site in different conditions, different AB/testing and all permutations)

Chrome already implements it and this could be considered also for developer relation improvements. I use Safari as my main development browser but many co-workers would choose Chrome for this feature amongst others.

[Eric Lawrence]

(In reply to comment #2)
> We often use OS-level support for Link Conditioner, as explained here:
> http://nshipster.com/network-link-conditioner/
> 
> It would be a nice addition to have this feature implemented in WebKit
> inspector for a number of reasons:
> 
> 
> 1. It only affects the current tab in the current instance of Webkit, so it
> does not affect tasks that the developer might be doing in other
> applications and tabs
> 2. It makes clear in the UI what is being debugged at the current time
> 3. It allows side-by side comparison when using multiple windows for
> debugging network performance issues (different sites, the same site in
> different conditions, different AB/testing and all permutations)
> 
> Chrome already implements it and this could be considered also for developer
> relation improvements. I use Safari as my main development browser but many
> co-workers would choose Chrome for this feature amongst others.

Iraê-- This doesn't appear to have anything to do with HTTP Archive Support. If not, you should file a new issue, not bury your comment inside support for HAR.

[Yahoo Search Mobile Team]

I think something when wrong with the POST request or I messed up my tabs. I understand it's not related, I was a mistake. Thanks for posting that out.

[Iraê]

I did mess up with posting the wrong description here, but I would like to say that HAR support is a helpful feature, not only by exporting through the UI, but also if we could get it programmatically. It's not part of the WebDriver bug request (https://bugs.webkit.org/show_bug.cgi?id=135263), but it is also a good tool for automating performance tests.

Having programmatic access, via command-line/launcher of via web APIs (something like Resource Timings (https://bugs.webkit.org/show_bug.cgi?id=61138) might be translatable to HAR, but HAR would be easier to integrate).

[Timothy Hatcher]

<rdar://problem/7463672>

[Aral Balkan]

This would also be helpful for content blockers.

[Blaze Burg]

<rdar://problem/7463672>

[Joseph Pecoraro]

HTTP Archive (HAR) format specication:
https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HAR/Overview.html

Older version for reference:
http://www.softwareishard.com/blog/har-12-spec/

Other Resources that maybe useful:
http://www.softwareishard.com/har/viewer/
https://toolbox.googleapps.com/apps/har_analyzer/

[Joseph Pecoraro]

Created attachment 324538 [details]
[PATCH] Proposed Fix

[Joseph Pecoraro]

I'm missing an export button right now.

However this match the main way other browsers include a HAR export (an option in the context menu).

We also make ⌘S in the Network Tab default to this behavior.

[Blaze Burg]

Comment on attachment 324538 [details]
[PATCH] Proposed Fix

View in context: https://bugs.webkit.org/attachment.cgi?id=324538&action=review

r=me

> LayoutTests/http/tests/inspector/network/har/har-basic-expected.txt:37
> +        "title": "http://127.0.0.1:8000/inspector/network/har/har-basic.html",

Is loopback hardcoded in other tests like this?

> LayoutTests/http/tests/inspector/network/har/har-basic.html:24
> +            let har = await WI.HARBuilder.buildArchive([]);

Very interesting. I haven't read much async/await code.

> Source/WebInspectorUI/ChangeLog:63
> +        Capture the raw, unmodified, base64 encoded and content. This ends up

Nit: -and

> Source/WebInspectorUI/UserInterface/Controllers/HARBuilder.js:32
> +    static async buildArchive(resources)

I found it a little weird that this is a singleton and all static methods, but I guess it needs to use a ton of other global state anyway.

> Source/WebInspectorUI/UserInterface/Controllers/HARBuilder.js:235
> +        // FIXME: Implement HAR cache data.

Please file a bug for this.

> Source/WebInspectorUI/UserInterface/Controllers/HARBuilder.js:242
> +        // NOTE: Chrome Custom Fields `_blocked_queueing` and `_blocked_proxy`.

Weird grammar.

> Source/WebInspectorUI/UserInterface/Views/NetworkTableContentView.js:1386
> +        if (!resources.length) {

Maybe it would be more obvious when exporting is not possible, by making the button become disabled when this condition is true.

[Joseph Pecoraro]

Comment on attachment 324538 [details]
[PATCH] Proposed Fix

View in context: https://bugs.webkit.org/attachment.cgi?id=324538&action=review

>> LayoutTests/http/tests/inspector/network/har/har-basic-expected.txt:37
>> +        "title": "http://127.0.0.1:8000/inspector/network/har/har-basic.html",
> 
> Is loopback hardcoded in other tests like this?

This is an http/tests/ LayoutTest and those are always 127.0.0.1:8000

>> Source/WebInspectorUI/UserInterface/Views/NetworkTableContentView.js:1386
>> +        if (!resources.length) {
> 
> Maybe it would be more obvious when exporting is not possible, by making the button become disabled when this condition is true.

We currently don't have a button, this path is taken via a ⌘S keyboard shortcut.

[Joseph Pecoraro]

<https://webkit.org/b/146692>

[Ryan Haddad]

One of the LayoutTests for this change is failing:
https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r223860%20(5199)/results.html

--- /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/tests/inspector/network/har/har-page-expected.txt
+++ /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/tests/inspector/network/har/har-page-actual.txt
@@ -50,8 +50,7 @@
           },
           "redirectURL": "",
           "headersSize": "<filtered>",
-          "bodySize": "<filtered>",
-          "_transferSize": "<filtered>"
+          "bodySize": "<filtered>"
         },
         "cache": {},
         "timings": {
@@ -63,8 +62,6 @@
           "wait": "<filtered>",
           "receive": "<filtered>"
         },
-        "serverIPAddress": "127.0.0.1",
-        "connection": "<filtered>",
         "_fetchType": "<filtered>"
       },
       {

[Blaze Burg]

(In reply to Ryan Haddad from comment #15)
> One of the LayoutTests for this change is failing:
> https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/
> r223860%20(5199)/results.html
> 
> ---
> /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/
> tests/inspector/network/har/har-page-expected.txt
> +++
> /Volumes/Data/slave/sierra-release-tests-wk2/build/layout-test-results/http/
> tests/inspector/network/har/har-page-actual.txt
> @@ -50,8 +50,7 @@
>            },
>            "redirectURL": "",
>            "headersSize": "<filtered>",
> -          "bodySize": "<filtered>",
> -          "_transferSize": "<filtered>"
> +          "bodySize": "<filtered>"
>          },
>          "cache": {},
>          "timings": {
> @@ -63,8 +62,6 @@
>            "wait": "<filtered>",
>            "receive": "<filtered>"
>          },
> -        "serverIPAddress": "127.0.0.1",
> -        "connection": "<filtered>",
>          "_fetchType": "<filtered>"
>        },
>        {

These fields are only available for iOS 11 / High Sierra or later. We just need a new baseline for Sierra.

[Joseph Pecoraro]

> These fields are only available for iOS 11 / High Sierra or later. We just
> need a new baseline for Sierra.

These fields should be available on Sierra.

I guess in the test they may be variable based on whether or not the resource was served from a cache. I may just have to eliminate these fields to avoid flakiness.

[Joseph Pecoraro]

I updated the test to try to reduce flakiness:
<https://trac.webkit.org/changeset/223873/webkit>

[Ryan Haddad]

(In reply to Joseph Pecoraro from comment #18)
> I updated the test to try to reduce flakiness:
> <https://trac.webkit.org/changeset/223873/webkit>
The test still appears to be failing: https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/r223885%20(5216)/results.html

[Joseph Pecoraro]

(In reply to Ryan Haddad from comment #19)
> (In reply to Joseph Pecoraro from comment #18)
> > I updated the test to try to reduce flakiness:
> > <https://trac.webkit.org/changeset/223873/webkit>
> The test still appears to be failing:
> https://build.webkit.org/results/Apple%20Sierra%20Release%20WK2%20(Tests)/
> r223885%20(5216)/results.html

That failure looks like it would be before my changes... unless I didn't reset the results of the test properly

[Joseph Pecoraro]

> That failure looks like it would be before my changes... unless I didn't
> reset the results of the test properly

It appears I may not have reset the results properly since I do see a "connection" property. Let me try again.

[Joseph Pecoraro]

<https://trac.webkit.org/r223899>

[Matt Lewis]

The test looks like it is still failing:
https://build.webkit.org/results/Apple%20High%20Sierra%20Release%20WK2%20(Tests)/r223900%20(665)/results.html
https://build.webkit.org/builders/Apple%20High%20Sierra%20Release%20WK2%20(Tests)/builds/665

[Joseph Pecoraro]

(In reply to Matt Lewis from comment #23)
> The test looks like it is still failing:
> https://build.webkit.org/results/
> Apple%20High%20Sierra%20Release%20WK2%20(Tests)/r223900%20(665)/results.html
> https://build.webkit.org/builders/
> Apple%20High%20Sierra%20Release%20WK2%20(Tests)/builds/665

Oops, I removed whitespace before landing and forgot to update the results with the new size.

[Joseph Pecoraro]

Fingers crossed!

<https://trac.webkit.org/r223917>