Bug 138169 – WKWebView does not support NSURLProtocol-style interception of http or https

• Status: NEW
• Component: WebKit2

[Carlos Lozano]

Summary:
It was possible to install a custom NSURLProtocol to handle requests from UIWebView. This is no longer possible with WKWebView - the NSURLProtocol subclass *does* receive the +canInitWithRequest: message, but is never instantiated. 


Steps to Reproduce:
1. Create and register a custom NSURLProtocol subclass that handles all requests.
2. Create a WKWebView and load a request.


Expected Results:
The custom NSURLProtocol subclass should receive a +canInitWithRequest: message, and then an instance should be created and initialized with -initWithRequest:cachedResponse:client:.


Actual Results:
The NSURLProtocol subclass *does* receive the +canInitWithRequest: message, but is never instantiated.


Version:
iOS 8.0, iOS 8.0.1, iOS 8.1 (tested)


Notes:
This breaks any behind the scenes encryption, compression, interception, caching, etc. This is breaking many things and blocking the adoption of WKWebView.


Configuration:
All emulators and devices with >= iOS 8 support.

[Trevor Linton]

As a side note, this is also an issue with WKWebView on OSX Yosemite 10.10.3, the circumstances are exactly the same it seems as with iOS.

[Radar WebKit Bug Importer]

<rdar://problem/21349577>

[Justin Caldicott]

I'm surprised this bug has so little activity.

We would love to be able to use NSUrlProtocol with WKWebView (on OS X).  We'd be able to avoid having a local web server running - less resources used and better performance.

[Erwin van Dijk]

Are there any plans on implementing this functionality? I'd really love to use WKWebView, but this limitation forces me to keep using the UIWebView.

[Sam Weinig]

(In reply to comment #4)
> Are there any plans on implementing this functionality? I'd really love to
> use WKWebView, but this limitation forces me to keep using the UIWebView.

We are investigating it, but it would really help us to understand more about why you were using this functionality. What type of loads did you need to intercept?

[Ignacio Calderon]

In my particular case, this interception is needed for 2 main purposes:
1. Implement SSL pinning for web view content (most important)
2. Provide encrypted on-device assets, and be able to decrypt them before pass them to the web view for rendering.

[Erwin van Dijk]

>We are investigating it, but it would really help us to understand more about why you were using this functionality. What type of loads did you need to intercept?

We are loading binary files which are transformed into multiple images. Default browser cache does not provide what we want. We want to be able to use local stored files if available or download them from a remote server. Therefore we need to intercept ajax calls to specific urls.

[Pavel Zdenek]

(In reply to comment #5)
> (In reply to comment #4)
> > Are there any plans on implementing this functionality? I'd really love to
> > use WKWebView, but this limitation forces me to keep using the UIWebView.
> 
> We are investigating it, but it would really help us to understand more
> about why you were using this functionality. What type of loads did you need
> to intercept?

Glad to see that someone at Apple is picking this up. We even spent a TSI on asking this a long time ago. The anwser was basically "not considered a bug" so we shrugged and keep dragging the UIWebView cart.

We have built a web browser which allows for configurable intervention in the complete webview traffic, i.e. not just frames but all resources. Even XHR. The possible interventions are numerous. Cancel the request altogether, redirect, fake a response (e.g. a locally made image), tinker with outgoing or even incoming HTTP headers. It all works quite nicely on UIWebView, but it feels being left behind indeed.

[Stefan Arentz]

(In reply to comment #5)
> (In reply to comment #4)
> > Are there any plans on implementing this functionality? I'd really love to
> > use WKWebView, but this limitation forces me to keep using the UIWebView.
> 
> We are investigating it, but it would really help us to understand more
> about why you were using this functionality. What type of loads did you need
> to intercept?

Some things we would like to do with WKWebView that are currently impossible due to no ability to intercept and modify loading behaviour:

* HTTPS Everywhere - Detect HTTP loads of resources in a whitelist, modify them to become HTTPS loads
* Content blocking - Block resources, both top level and resource loads
* Better control over headers - Ability to look and modify headers, both incoming and outgoing
* Implement custom content caching and content preloading - Need the ability to take over a resource load and provide our own response
* Better control of cookies. Currently there is no way to control cookie policy in WKWebView, this is trivial to imlement if we can control network requests

Giving developers access to the URL loading system opens the door to work around many WKWebView limitations. The WKWebView APIs are extremely shallow, not giving us enough room to customize. But a lot of requests that developers have can be solved if access to networking/request loading is available.

(We are building a browser, Firefox for iOS, but many of our wishes are pretty common and have valid use cases in 'normal' WKWebView usage too)

[Sam Weinig]

(In reply to comment #9)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > Are there any plans on implementing this functionality? I'd really love to
> > > use WKWebView, but this limitation forces me to keep using the UIWebView.
> > 
> > We are investigating it, but it would really help us to understand more
> > about why you were using this functionality. What type of loads did you need
> > to intercept?
> 
> Some things we would like to do with WKWebView that are currently impossible
> due to no ability to intercept and modify loading behaviour:

Thanks! A few additional questions:

> 
> * HTTPS Everywhere - Detect HTTP loads of resources in a whitelist, modify
> them to become HTTPS loads
> * Content blocking - Block resources, both top level and resource loads

Diving into these, are you doing dynamic analysis of of loads, or would a static matching / modification engine suite these needs?  (Similar to the Content Blocker feature in WebKit, though not exposed as API, but with the ability to modify requests in addition to blocking).

> * Better control over headers - Ability to look and modify headers, both
> incoming and outgoing

What are the use cases you have for these? Which headers do you need to you want to modify and why?

> * Implement custom content caching and content preloading - Need the ability
> to take over a resource load and provide our own response

What is the use case for this?

> * Better control of cookies. Currently there is no way to control cookie
> policy in WKWebView, this is trivial to imlement if we can control network
> requests

Which aspects of the cookie policy do you want to control? What are the use cases for it?

[Sam Weinig]

(In reply to comment #6)
> In my particular case, this interception is needed for 2 main purposes:
> 1. Implement SSL pinning for web view content (most important)
> 2. Provide encrypted on-device assets, and be able to decrypt them before
> pass them to the web view for rendering.

Is this for resources with the http/https scheme? Or a different scheme?

[Sam Weinig]

(In reply to comment #8)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > Are there any plans on implementing this functionality? I'd really love to
> > > use WKWebView, but this limitation forces me to keep using the UIWebView.
> > 
> > We are investigating it, but it would really help us to understand more
> > about why you were using this functionality. What type of loads did you need
> > to intercept?
> 
> Glad to see that someone at Apple is picking this up. We even spent a TSI on
> asking this a long time ago. The anwser was basically "not considered a bug"
> so we shrugged and keep dragging the UIWebView cart.
> 
> We have built a web browser which allows for configurable intervention in
> the complete webview traffic, i.e. not just frames but all resources. Even
> XHR. The possible interventions are numerous. Cancel the request altogether,
> redirect, fake a response (e.g. a locally made image), tinker with outgoing
> or even incoming HTTP headers. It all works quite nicely on UIWebView, but
> it feels being left behind indeed.

To be clear, the functionality in UIWebView does not actually allow for intervention in all traffic.  Notably, most traffic for video and audio assets does not get captured as it happens in mediaserverd, not the host process.

[Arthur]

> * Implement custom content caching and content preloading - Need the ability
> to take over a resource load and provide our own response

> What is the use case for this?

Upgrade Progressive Web App to Cordova app, with caching implemented on native level as plugin. i.e.website isn't packaged with app, but served from server and cached on device, much like ServiceWorker does.

[Ignacio Calderon]

(In reply to comment #11)
> (In reply to comment #6)
> > In my particular case, this interception is needed for 2 main purposes:
> > 1. Implement SSL pinning for web view content (most important)
> > 2. Provide encrypted on-device assets, and be able to decrypt them before
> > pass them to the web view for rendering.
> 
> Is this for resources with the http/https scheme? Or a different scheme?

For (1), https resources.
For (2), main case file:// and optionally http*://. The assets could be on the app bundle encrypted to avoid tampering or reverse engineering, and accessed via file:// schemes. Or http/https equivalent to an end-to-end encryption, to dynamically update the content.

[Daniel]

Hello guys,

Following up the discussion:
https://lists.webkit.org/pipermail/webkit-dev/2016-May/thread.html#28241
I'd like to share some use cases I have for this feature.

Opera on iOS implements a custom HTTP(S) protocol to do:
1. Data savings (see http://www.opera.com/turbo ). This greatly improves connectivity under crappy network conditions for millions of users. It's especially important for people in countries which can only dream about 4G.
2. Peer-to-peer inobtrusive security. For that we collect bits of site security information that is only available via low-level network APIs.
3. Presenting sites as icons (and grouping multiple pages into the same icons). For that we hook into the HTML data stream to parse meta data ASAP. In addition we intercept and react on HTTP redirects. This is a part of the http://operacoast.com app identity.
4. Progress loading reporting, automatic retries on bad networks. For that we do traffic QoS monitoring.
5. Fast going back and offline content. That is controlled partially by a custom cache, and partially in NSURLProtocol.
6. Ad-blocking.

Almost all of what Stefan Arentz says and other mentioned applies to us as well, as the need for: untrusted resources blocking, custom cache, content preloading, certificate pinning, HSTS.

With that said, it's so important feature to have for my team that
we are staying with WebKit1 for now.

See also:
https://bugs.webkit.org/show_bug.cgi?id=137299
https://bugs.webkit.org/show_bug.cgi?id=137302
https://bugs.webkit.org/show_bug.cgi?id=138131

[Eugene But]

Early this year Chrome for iOS migrated from UIWebView to WKWebView. WKWebView gave us a huge improvement in stability, but inability to use custom network protocols made impossible to support the following features:

Privacy
 - Block 3rd party cookies
 - Do Not Track
 - Non-ephemeral Incognito browsing (so users are not logged out if they background the app and the app is killed in the background.)

Security
 - HSTS preloads
 - Public key pinning
 - Safe browsing
 - SHA-1 deprecation warning

Other features
 - Data Saver
 - Downloads if authentication is required (no access to cookies)
 - Multiple profiles support
 - QUIC network protocol
 - WebP image format

[Brady Eidson]

(In reply to comment #16)
> Early this year Chrome for iOS migrated from UIWebView to WKWebView.
> WKWebView gave us a huge improvement in stability, but inability to use
> custom network protocols made impossible to support the following features:
> 
> Privacy
>  - Block 3rd party cookies
>  - Do Not Track
>  - Non-ephemeral Incognito browsing (so users are not logged out if they
> background the app and the app is killed in the background.)
> 
> Security
>  - HSTS preloads
>  - Public key pinning
>  - Safe browsing
>  - SHA-1 deprecation warning
> 
> Other features
>  - Data Saver
>  - Downloads if authentication is required (no access to cookies)
>  - Multiple profiles support
>  - QUIC network protocol
>  - WebP image format

This list is full of interesting API requests, but only a few are related to this bug.

I'd urge you to make sure there are independent bug reports for each.

[Daniel]

(In reply to comment #17)
> (In reply to comment #16)
> > Early this year Chrome for iOS migrated from UIWebView to WKWebView.
> > WKWebView gave us a huge improvement in stability, but inability to use
> > custom network protocols made impossible to support the following features:
> > 
> > Privacy
> >  - Block 3rd party cookies
> >  - Do Not Track
> >  - Non-ephemeral Incognito browsing (so users are not logged out if they
> > background the app and the app is killed in the background.)
> > 
> > Security
> >  - HSTS preloads
> >  - Public key pinning
> >  - Safe browsing
> >  - SHA-1 deprecation warning
> > 
> > Other features
> >  - Data Saver
> >  - Downloads if authentication is required (no access to cookies)
> >  - Multiple profiles support
> >  - QUIC network protocol
> >  - WebP image format
> 
> This list is full of interesting API requests, but only a few are related to
> this bug.
> 
> I'd urge you to make sure there are independent bug reports for each.

All of the things here look very familiar to me. It looks related to NSURLProtocol support in a way that it enables all of those features. If NSURLProtocol support was introduced the developers could bring back all that code to implement the listed features.

One could possibly create separate bugs for each of the use cases that were listed, and solve them with separate methods, but I think that the call from Sam was to list all possible use cases. This points out how the lack of NSURLProtocol support affected the apps that migrated (or plan to migrate) to WKWebView, and shows which use cases have to be dropped because they were based on NSURLProtocol-s.

[Brady Eidson]

(In reply to comment #18)
> (In reply to comment #17)
> > (In reply to comment #16)
> > > Early this year Chrome for iOS migrated from UIWebView to WKWebView.
> > > WKWebView gave us a huge improvement in stability, but inability to use
> > > custom network protocols made impossible to support the following features:
> > > 
> > > Privacy
> > >  - Block 3rd party cookies
> > >  - Do Not Track
> > >  - Non-ephemeral Incognito browsing (so users are not logged out if they
> > > background the app and the app is killed in the background.)
> > > 
> > > Security
> > >  - HSTS preloads
> > >  - Public key pinning
> > >  - Safe browsing
> > >  - SHA-1 deprecation warning
> > > 
> > > Other features
> > >  - Data Saver
> > >  - Downloads if authentication is required (no access to cookies)
> > >  - Multiple profiles support
> > >  - QUIC network protocol
> > >  - WebP image format
> > 
> > This list is full of interesting API requests, but only a few are related to
> > this bug.
> > 
> > I'd urge you to make sure there are independent bug reports for each.
> 
> All of the things here look very familiar to me. It looks related to
> NSURLProtocol support in a way that it enables all of those features. If
> NSURLProtocol support was introduced the developers could bring back all
> that code to implement the listed features.
> 
> One could possibly create separate bugs for each of the use cases that were
> listed, and solve them with separate methods, but I think that the call from
> Sam was to list all possible use cases. This points out how the lack of
> NSURLProtocol support affected the apps that migrated (or plan to migrate)
> to WKWebView, and shows which use cases have to be dropped because they were
> based on NSURLProtocol-s.

I'm not saying "stop listing your use cases here and please file a new bug on everything".

I just don't want this bug to start tracking "Here is every single gripe I have with the WKWebView API", which would be unproductive.

There's things in this list for which SPI already exists (e.g. 3rd party cookie blocking), and things in this list for which "Custom NSURLProtocol is the solution!" is not apparent to everyone.

WebP images for example: It's not at all apparent (to me, at least) why implementing a custom image format requires custom network loading. Maybe I'm missing something obvious?

[Eugene But]

We already filed multiple bugs against specific features, which I can list here:

Block 3rd party cookies:
https://bugs.webkit.org/show_bug.cgi?id=140205

Do Not Track:
https://bugs.webkit.org/show_bug.cgi?id=140571

Non-ephemeral Incognito browsing:
https://bugs.webkit.org/show_bug.cgi?id=140201

Downloads if authentication is required:
https://bugs.webkit.org/show_bug.cgi?id=140191

Multiple profiles support:
https://bugs.webkit.org/show_bug.cgi?id=140201


We filed various radars against CFNetwork and Security frameworks to cover missing security features. However the fully working solution would be to allow using custom network protocols.


Concerning WebP support, Chrome did the following things:
1.) Appended WebP to Accept request header, so server could respond with WebP images
2.) Decoded responses with WebP mime types and replaced them with images supported by UIWebView
Neither request part, not response is possible with WKWebView.

[Stefan Arentz]

(In reply to comment #20)

I also want to clarify something with regards to 'please file more detailed bugs' and the following is actually a great example:

> Concerning WebP support, Chrome did the following things:
> 1.) Appended WebP to Accept request header, so server could respond with
> WebP images
> 2.) Decoded responses with WebP mime types and replaced them with images
> supported by UIWebView
> Neither request part, not response is possible with WKWebView.

Since WKWebView has been introduced in iOS 8, almost 2 years ago, its API has moved forward / opened up *very very little*.

This despite dozens of feature requests being filed. Mozilla has filed a good number. So has every other browser vendor and many individual developers. Lots is still missing. From basics (cookie policy configuration or even setting custom headers) to more complicated things like access to networking. Which is what this bug is about.

Almost none of these filed feature requests have been implemented or even given attention on this bugtracker. This is not a surprise. WebKit on iOS is a complicated component and I think we all fully understand how Apple is a) resource constrained and b) cautious of introducing APIs that it needs to support into the very long term future.

The problem however is that after 2 years WKWebView is still very limited. So limited that people are talking about moving back to UIWebView. Mozilla included.

So back to the WebP example from above. I don't think that filing narrow and detailed requests like that is going to make a difference. The past years have proven that.

This is why everyone is asking to open up WKWebView more and specifically the networking part of it. These are *generic* changes we are asking for. Open ended. It is not specific on purpose, so that we, developers, can just do what we need to do to build interesting and competitive features. Without having to ask Apple to spend engineering time on a long list of very specific features.

Networking is the thread that connects most of the things that we want to do. With access to networking we can implement *so many* things that are now missing in WKWebView.

And that is why WebP is actually a great example. A technically complicated feature for Apple to implement. (CoreImage changes, policitical, etc.) But Google can just do it if they could set headers and intercept loads.

Networking is that Generic API that would open up WKWebView and make it 10x more usable.

I'll keep filing specific bugs for thigns that we want. Pretty much everything mentioned in this thread already has bugs. But also please consider opening up the networking for the above reasons.

[Daniel]

> 
> Concerning WebP support, Chrome did the following things:
> 1.) Appended WebP to Accept request header, so server could respond with
> WebP images
> 2.) Decoded responses with WebP mime types and replaced them with images
> supported by UIWebView
> Neither request part, not response is possible with WKWebView.
>

Opera on iOS uses the same strategy to support WebP, which is a part of our data savings feature. In the savings mode all the images are transcoded through this format.

[Erwin van Dijk]

Intercepting web requests for me is no longer necessary. The reason I used this was for caching large binary files. This can be achieved by using IndexedDB as well. (https://developer.mozilla.org/nl/docs/IndexedDB ) The reason that I post this update is that it might help other people who only need the interception for caching files.

Of course this does not solve 'behind the scenes encryption', compression, etc. Therefore the issue can't be closed.

We use NSURLProtocol to actually use a non-HTTP(S) wire protocol.  It's more efficient than the traditional stuff and provides a real UX improvement for various classes of captive native/web applications.

A lot of the usecases in this thread could (in theory) be resolved by new API surface–content blocking, cookies, etc could all be solved by new API.

However, for folks who actually need to intercept the requests themselves, the existing API was very good, and it's a real shame to see it go.

[Garvan Keeley]

NSURLProtocol interception of WKWebView http/https is theoretically possible:

Class cls = [[[WKWebView new] valueForKey:@"browsingContextController"] class];
SEL sel = NSSelectorFromString(@"registerSchemeForCustomProtocol:");
[(id)cls performSelector:sel withObject:@"http"];
[(id)cls performSelector:sel withObject:@"https"];

Followed by an NSURLProtocol registration will intercept all calls.

Credit to this github user who also has a demo project https://github.com/WildDylan/WKWebViewWithURLProtocol

Given that this already works, perhaps Apple can consider exposing it officially.

[Arthur]

to #c25

Or at least not block that thing..

[Garvan Keeley]

There is a major bug in the comment #25 approach, having a registered NSURLProtocol causes post bodies to be stripped.

Returning false always from NSURLProtocol `canInitWithRequest(request: NSURLRequest) -> Bool` results in an empty post body. 

The stripped post body in the NSURLRequest sent to NSURLProtocol is not surprising given the justification in https://bugs.webkit.org/show_bug.cgi?id=145410, that IPC serialization of large post data is a performance hit. What is surprising is that returning 'false' from `canInitWithRequest` does not workaround this.

[Pavel Zdenek]

(In reply to comment #27)
> There is a major bug in the comment #25 approach, having a registered
> NSURLProtocol causes post bodies to be stripped.

I can imagine it being not a bug, but actually an intentional logic along the original behavior of not sending POST body. Where in both cases, the intent correctness can be discussed, but not treated as a bugfix. I have spent some time looking into WK2 sources, hoping to find an approximate place of such "bug". I got a gut feeling instead: when registerSchemeForCustomProtocol is invoked, things appear to get flipped to a "custom protocol" primarily on WK side and the IPC is just a subsystem. So a stripped POST request is made for that custom protocol, and when IPC pipe spits out "not interested", WK side simply goes on with the request already made. Take this with a shovel of salt, i'm a github skimmer, not WK contributor.

[Ian Ragsdale]

Adding a plus one for this bug. My day job is working for VMware/AirWatch, and we have a custom browser that exists to provide access to corporate sites behind a firewall. We use a custom NSURLProtocol to allow tunneling requests to the back end, and we are completely unable to use WKWebView for our purposes.

It looks like some of the code we need is open source. If we invested developer time in extending the WKWebView to allow this, would Apple even be willing to ship it?

[Steve Sauder]

I have an additional use case that might be useful - i'm writing a WKWebKit-based application - not based on Cordoba, but built myself, that is intended to be used in a situation where it may be required to function without network connectivity for a period of time - and it must have a fairly large set of images cached on the device (read: larger than localStorage or other storage methods would permit) in order to maintain it's functionality while temporarily offline.  The only useful way I've found to be able to do that is to use NSURLProtocol to manage caching of the imagery at the app level (using CoreData).  There doesn't appear to be any other comprehensive way to intercept the network requests and provide our own caching mechanism without using NSURLProtocol.

[pyanfield]

I have a use case that when loading a webpage with many images, I want to know which image is loaded failed and then use a placeholder to replace it. But for now, I can't get the information.

[Krystian Szczęsny]

I also have to chip in to this.

In my application we display content, which is stored in an encrypted form on the device. We use NSURLProtocol to decrypt the content before it reaches the user. Now, for us, it is impossible to migrate to WKWebKit.

[Ricci Adams]

Check out the WKURLSchemeHandler protocol and -[WKWebViewConfiguration setURLSchemeHandler:forURLScheme:]. We've been using this in our applications as an NSURLProtocol replacement. Note that it's 10.13+ only.

[Krystian Szczęsny]

@Ricci Adams
WKURLSchemeHandler can only override protocols it does not know about. Unfortunately http or https are not the case. A proof would come from apple staff eskimo's answer here: https://forums.developer.apple.com/thread/87474

[Ian Ragsdale]

Besides the problem of not working with HTTP/HTTPS protocols, it also doesn't handle POST data.

(In reply to Krystian Szczęsny from comment #34)
> @Ricci Adams
> WKURLSchemeHandler can only override protocols it does not know about.
> Unfortunately http or https are not the case. A proof would come from apple
> staff eskimo's answer here: https://forums.developer.apple.com/thread/87474

[Jin]

UIWebView is deprecated in iOS 12. 
However WKWebView still can not work with NSURLProtocol .

With NSURLProtocol, I can custom Image/Resource/ADBlock/DNS/Custom HTTP , which make WebView more powerful than App.

I don't know what WKWebView is complacent and conservative in this 2 years.

Will WKWebView be "free" in the future?

[Mihai]

Is this still considered a new bug? 

I need to render a "local" and dynamic html application and I see that UIWebView has been deprecated. WEWebView does not support custom request loading so we are unable intercept and serve ajax requests. 

Our current workaround is to start a real http server which badly degrades the app performance(especially the start-up time)

[Maciej Stachowiak]

<rdar://problem/58363932>

[Maciej Stachowiak]

Updated title to describe remaining request.

[Maciej Stachowiak]

For folks following this bug, who would consider it sufficient to have a way to define per-app or per-WKWebView proxy setting? If it wouldn't be sufficient, could you describe what it might be missing for your use case?

[Krystian Szczęsny]

I am not entirely sure how would this work?
The proxy could be created within the app?
The custom protocol was created to decrypt files and the files are part of the data of the app, so nothing else apart from the app in question could access the data in the encrypted form.

Our requirements for WkWebView would be met if it supported the remote debugging protocol from within the hosting app. This would allow monitoring of HTTP/HTTPS requests and access to cookies, cached data and performance metrics.

Something similar to the Chrome.Debugger class provided for Chrome extensions would be great.

[Ian Ragsdale]

(In reply to Maciej Stachowiak from comment #40)
> For folks following this bug, who would consider it sufficient to have a way
> to define per-app or per-WKWebView proxy setting? If it wouldn't be
> sufficient, could you describe what it might be missing for your use case?

I believe this would be sufficient for my app's purposes. I think per-WKWebView would be preferable to per-app but per-app would likely suffice.

[Stefan Arentz]

I would highly recommend (request) that this feature is made available on a per-wkwebview basis. That allows for use cases where not all WKWebViews are similar (think private mode vs normal mode vs container, etc.).

It also solves the problem of the proxy not know what WKWebView is talking to it: if it is app wide then there is no way to identify a WKWebView while if they are per-WKWebView then it is trivial to implement something int he proxy (like a unique port number) to make sure that front-end (WKWebView/UIKit) code and backend (Proxy) code can identify each other.

[Eugene But]

Having *per-WKWebView* proxy setting is useful for Chrome to support some Enterprise features that Chrome for iOS lost after moving from UIWebView to WKWebView.


There are some other use cases that proxy setting do not address:

1.) Innovate in networking space, like Google did with Speedy and doing with QUIC (this requires full support of custom network protocols)
2.) Better privacy control over cookies, see https://blog.chromium.org/2019/05/improving-privacy-and-security-on-web.html (can be done without full network protocols support, but may need new API)
3.) Support WebP (can be done without full network protocols support)
4.) Change request headers to send the list of chrome experiments to google servers (can be done without full custom network protocols support, but requires new WKWebView API)

All these use cases were possible with UIWebView, which supported custom network protocols.

[Ian Ragsdale]

Seems like #1 and #4 ought to be doable (if imperfectly) with a proxy, right? Seems like if you have a local proxy accepting requests you can do whatever networking you want over the actual network.

- Ian


(In reply to Eugene But from comment #45)
> Having *per-WKWebView* proxy setting is useful for Chrome to support some
> Enterprise features that Chrome for iOS lost after moving from UIWebView to
> WKWebView.
> 
> 
> There are some other use cases that proxy setting do not address:
> 
> 1.) Innovate in networking space, like Google did with Speedy and doing with
> QUIC (this requires full support of custom network protocols)
> 2.) Better privacy control over cookies, see
> https://blog.chromium.org/2019/05/improving-privacy-and-security-on-web.html
> (can be done without full network protocols support, but may need new API)
> 3.) Support WebP (can be done without full network protocols support)
> 4.) Change request headers to send the list of chrome experiments to google
> servers (can be done without full custom network protocols support, but
> requires new WKWebView API)
> 
> All these use cases were possible with UIWebView, which supported custom
> network protocols.

[Eugene But]

> Seems like #1 and #4 ought to be doable (if imperfectly) with a proxy, right? 

It might be possible, depending how proxy is implemented.

The Psiphon Browser project (https://apps.apple.com/us/app/psiphon-browser/id1193362444), and other projects that depend on our technology, would benefit from being able to define per-WKWebView proxy settings. As things stand, we will have to decommission the project when UIWebView is fully deprecated.

> For folks following this bug, who would consider it sufficient to have a way to define per-app or per-WKWebView proxy setting?

This should be sufficient for our use case.

We currently use NSURLProtocol with UIWebView to proxy webview requests through our local HTTP and SOCKS proxies. We've had to develop workarounds for requests that are not intercepted by NSURLProtocol (e.g. OCSP requests during certificate validation) and media being fetched out of process. If WKWebView eventually provides proxy support, we would desire requests such as these to be proxied as well.

[Simon Brooks]

(In reply to Ian Ragsdale from comment #43)
> (In reply to Maciej Stachowiak from comment #40)
> > For folks following this bug, who would consider it sufficient to have a way
> > to define per-app or per-WKWebView proxy setting? If it wouldn't be
> > sufficient, could you describe what it might be missing for your use case?
> 
> I believe this would be sufficient for my app's purposes. I think
> per-WKWebView would be preferable to per-app but per-app would likely
> suffice.

Few follow up questions to try and understand what the proxy enablement would actually allow (disclosure - I work with Ian):

Would the proxy configuration allow connection to a local proxy? (and if so would that work even with ATS enabled with the local proxy not being able to offer a valid SSL cert)

how to chain a further proxy that may require user authentication?

Would it be better/more consistent to allow the app to participate in a per-app VPN type configuration, possibly even using a packet tunnel extension provided within the app's own package? I'm not sure that any of the other alternatives maintain the background downloading capabilities.

[Leo Natan]

(In reply to Maciej Stachowiak from comment #40)
> For folks following this bug, who would consider it sufficient to have a way
> to define per-app or per-WKWebView proxy setting? If it wouldn't be
> sufficient, could you describe what it might be missing for your use case?

Hello,

One use-case can be found here: https://bugs.webkit.org/show_bug.cgi?id=158812

[Maciej Stachowiak]

Some replies to issues with a per-WKWebView proxy as a possible way to address some of the use cases for this.

(In reply to Eugene But from comment #45)
> Having *per-WKWebView* proxy setting is useful for Chrome to support some
> Enterprise features that Chrome for iOS lost after moving from UIWebView to
> WKWebView.
> 
> 
> There are some other use cases that proxy setting do not address:
> 
> 1.) Innovate in networking space, like Google did with Speedy and doing with
> QUIC (this requires full support of custom network protocols)

We aren't currently looking to support plugging in new protocols under the http:/https: schemes. We do support HTTP2 (the standardized version of SPDY) on Apple platforms, and I think it's no secret that there is a prototype of IETF QUIC + h3.

> 2.) Better privacy control over cookies, see
> https://blog.chromium.org/2019/05/improving-privacy-and-security-on-web.html
> (can be done without full network protocols support, but may need new API)

That post is not very specific, so it's not clear what the ask is.

If it's specifically the behavior to treat cookies as SameSite=Lax by default, we're open to doing that, or perhaps something even more strict:
https://trac.webkit.org/changeset/251467/webkit

> 3.) Support WebP (can be done without full network protocols support)

Current plan is to support WebP directly on Apple platforms:
https://trac.webkit.org/changeset/256501/webkit

We may yet change our minds, but in any case, the network stack seems like the wrong layer to add support for new image formats.

> 4.) Change request headers to send the list of chrome experiments to google
> servers (can be done without full custom network protocols support, but
> requires new WKWebView API)

We may add some kind of support for custom headers, but TBH broadcasting extra bits of entropy to Google servers is not an inspiring use case. Also, replacing the whole network stack to add some custom headers to requests seems like massive overkill.

> 
> All these use cases were possible with UIWebView, which supported custom
> network protocols.

(In reply to Simon Brooks from comment #49)
> 
> Few follow up questions to try and understand what the proxy enablement
> would actually allow (disclosure - I work with Ian):
> 
> Would the proxy configuration allow connection to a local proxy? (and if so
> would that work even with ATS enabled with the local proxy not being able to
> offer a valid SSL cert)
> 
> how to chain a further proxy that may require user authentication?
> 
> Would it be better/more consistent to allow the app to participate in a
> per-app VPN type configuration, possibly even using a packet tunnel
> extension provided within the app's own package? I'm not sure that any of
> the other alternatives maintain the background downloading capabilities.

I don't think we have worked out these details. Off the top of my head, ability to do a proxy that MITMs TLS would be a concerning thing to allow, but I can also understand how some use cases would benefit from it. Chaining to proxies that require auth would require passing on the authentication challenge. I'm not sure I understand the background downloading issue.

[Isaiah]

HI,

Our use case for protocol handlers that can catch and modify http and https requests…

We make a webpage creation tool for novice users (http://yourhead.com/stacks). As the user creates a page it often contains links to resources that have not yet been deployed/published.

We currently use WebView and NSURLProtocol to catch all requests and can redirect the requests for un-published content to the local data within the app.

Without protocol handlers we will be forced to modify the HTML directly before it gets to the WKWebView. This is difficult (regex -- yuck), error prone(we'll miss URLs that are hidden within scripts, etc.), and slow.

Using protocols handlers has been a simple and effect means of handling this situation for about a decade. This has been one of a handful of reasons keeping us on WebView for so long.

[Alex Christensen]

*** Bug 207542 has been marked as a duplicate of this bug. ***

[Pritish Parmar]

Any updates on adding proxy support for WKWebview? With the announcement of Apple that apps with UIWebView cannot be uploaded to store after Dec-2020, it is more important for us to implement our existing functionalities with WKWebview. Please help.

Any news on proxy support for WKWebView? We would really appreciate this feature too.

[Olav Schettler]

Just a +1 from me for proxy support in WKWebView. 

We use WkWebView to display contents and like to offer censorship circumvention to our users. Currently, we look into all kinds of kinky ways to reroute network traffic through the proxy integrated in our app.

Clean support for proxies in WkWebView would be so much simpler and more robust.

[Boyan]

Any update on this one? If this won't make it ever in WKWebView, any advice or workaround how we can intercept the traffic in WKWebView?

[Boyan]

Hi Team,
Any update on this one? It's been a year now since my last comment.

Thanks,
Boyan!

Proxying WKWebView traffic is now supported on iOS 17.0+ https://forums.developer.apple.com/forums/thread/110312?answerId=756922022#756922022.

[Leo Natan]

(In reply to m.kuratczyk from comment #59)
> Proxying WKWebView traffic is now supported on iOS 17.0+
> https://forums.developer.apple.com/forums/thread/
> 110312?answerId=756922022#756922022.

Proxying is not the same as what is requested here. In order for a proxy to intercept SSL traffic, a certificate authority needs be installed in the device and trusted. This complicates things significantly.