Bug 152598 – Feature request: a callback when request is blocked by the Content Blocker

• Status: NEW
• Component: WebKit Misc.

[Andrey Meshkov]

Currently it is very hard to debug content blocker rules as we can't discover which rule has been triggered.

The only notification I see in webkit code is:

if (willBlockLoad) {
    if (currentDocument)
        currentDocument->addConsoleMessage(MessageSource::ContentBlocker,
MessageLevel::Info, makeString("Content blocker prevented frame displaying
", mainDocumentURL.string(), " from loading a resource from ",
request.url().string()));
       return BlockedStatus::Blocked;
    }

But clearly that's not enough.

It would be great if there was some kind of a callback method signaling about triggered content blocker rule.

Essential information:
1. Main document URL
2. Request URL
3. Content blocker rule (if it is possible to pass the whole rule here).

If point 3 is too hard to implement, you might add an ID property to the rule syntax and pass that ID back in the callback method.

[Alex Christensen]

For point 3, we have though about having a mode in which we do not coalesce identical actions so we can tell which rule caused the block.

Why is a console message not enough, and what kind of callback do you have in mind?  We don't have access to much more than the current document here.

[Andrey Meshkov]

The main problem with console log is that we cannot access it from the extension (here I mean both OS X and iOS).

The requested feature will allow us to make quite a few very useful improvements:
1. We can show user a number of blocked items
2. We can show user the list of blocked items (lots of users are interested in what was really blocked on the page)
3. If point 3 is implemented, we can provide a simple rules creation/debugging tool. Just look at how it works in our extension for other browsers:

List of requests:
http://screencast.com/t/5WKw2E8T63

Request details:
http://screencast.com/t/TXrSoUlx


> What kind of callback do you have in mind?  We don't have access to much more than the current document here.

You could provide an additional parameter in setContentBlocker method, allowing us to pass a callback method there. I see that in that place where you write the console message, you have access to all request parameters and main document URL, which is a lot already.

Ideally, I'd like to see the following parameters sent to the callback method:
1. Request URL
2. Referrer URL
3. Requested resource type
3. Main document URL
4. Some kind of a tab/window ID. This would be really handy in OS X as we'll be able to distinguish requests by tabs.
5. Content blocker rule (or it's ID if you'll implement it).

One more important notice on this. It will be better to have this callback method called on every request, not just when it's blocked.

[test]

(In reply to comment #1)
> For point 3, we have though about having a mode in which we do not coalesce
> identical actions so we can tell which rule caused the block.
> 
> Why is a console message not enough, and what kind of callback do you have
> in mind?  We don't have access to much more than the current document here.
Please support this.

Safari OS X and iOS need more Pro Features!! E.g with iPad Pro not possible.

But with other hybrid tablet no issue.

Dear Webkit developers, please implement a statistics callback. It would make blocking addons capable of determining what block-rules are the most popular, which in turn lets blocklist authors optimize their lists (to cut out dead rules that never trigger on domains anymore due to site redesigns, etc).

That's the #1 way of handling two problems: Content Blocker only supports 50,000 rules, and the fact that old rules die with time as sites get redesigned. We need statistics to be able to find and delete all dead rules to keep the blocklists up to date.

I think there should be a single callback function with an array of blocked resources, at the end of the page loading. That sounds like a resource-efficient way of handling blocking-statistics.

I misspoke earlier. I said "I think there should be a single callback function with an array of blocked resources, at the end of the page loading. That sounds like a resource-efficient way of handling blocking-statistics."

What I meant, and think would be even better, is:

I think there should be a single callback function with an array of the IDs of triggered blockrules AND triggered whitelist rules, at the end of the page loading.

Such an array would be easy to build and lightweight to process. It would tell us what rules are active, and what unblock/whitelist rules are active. From that, we can detect which rules never trigger on a domain anymore, as well as which triggers the most. Then we can build optimized blocklists that cut out all the years-old, long-dead rules (about 80% of the blocklist contents are dead rules by now, and we need statistics to find those).

[Andrey Meshkov]

Hello guys, is there any update on this issue?

I'd just like to emphasize, that this feature request will benefit all the community.

Personally, don't mind partial implementation, it'd be great to have at least something:) For instance, the information you print in the console log. Yeah, it is not an ideal solution, but it is much better than nothing.

[Maciej Stachowiak]

One of the design goals of content blockers is that they are not able to observe the user's browsing. A callback that specifically reported what was blocked would defeat that purpose.

Maybe there's higher-level statics we could give without creating that problem.

It's also possible that this could be provided for content blockers that are also app extensions, as app extensions already potentially have the ability to observe browsing.

[Radar WebKit Bug Importer]

<rdar://problem/59584697>

[Krzysztof Jan Modras [:chrmod]]

Cliqz Browser use cases are:
- Show number of blocked Ads / Trackers on current page load https://pasteboard.co/IVoibx2.png
- Show statistical informaton (from some period of time) https://pasteboard.co/

Having a higher level API would do the job if we could attach some metadata to the rule. Currently we want to know the catergory of the blocked request like: Ad, Trackers, CDN, Social, etc. But Ghostery browser for instance chooses to list blocked companies. 

Question that come to my mind:
- would WebKit store statistical information?
- how to handle "back navigation" when website is loaded from in memory cache (no actual requests are fired)?
- how content blockers could access such information from Safari?


It's perhaps worth mentioning again that current solutions used by app developers are far from optimal. For instance Firefox intercepts all url loads with users scripts and send them to UI process. Then those urls are checked agaist the block list which has to be loaded second time in memory (in different representation https://github.com/mozilla-mobile/firefox-ios/blob/7c889f177b6a7c60883238ead6c9113167b196b4/content-blocker-lib-ios/src/TrackingProtectionPageStats.swift#L38) as there is no API to check if block list would block the request (maybe we should report it as separate issues?).

[Krzysztof Jan Modras [:chrmod]]

> - Show statistical informaton (from some period of time)
> https://pasteboard.co/
 
Proper link https://pasteboard.co/IVoiiGE.png

[Andrey Meshkov]

> One of the design goals of content blockers is that they are not able to observe the user's browsing. A callback that specifically reported what was blocked would defeat that purpose.

This makes sense, thank you!

> Maybe there's higher-level statics we could give without creating that problem.

Something like this would be nice to have:
`count(*) GROUP BY action`

Ideally, I'd like to have the rules categorized (ads, tracking, social widgets, etc) so that we could get `GROUP BY category, action`

> It's also possible that this could be provided for content blockers that are also app extensions, as app extensions already potentially have the ability to observe browsing.

That'd be great!

One of the purposes that I have in mind is providing filter lists' maintainers with a convenient way to debug content blocking rules (console log is not enough for this), and they usually use the desktop AdGuard for Safari to do this as it's simply much easier than using an iOS device.

[Maciej Stachowiak]

For the use case of debugging content blocker extensions, maybe we could provide something better via Web Inspector or safaridriver?

[Andrey Meshkov]

> For the use case of debugging content blocker extensions, maybe we could provide something better via Web Inspector or safaridriver?

Yep, this will be even better.

Currently, requests blocked by content blockers simply disappear from the "Network" tab of the Web inspector.

Instead of this, you could mark them as "blocked", and provide additional metadata:
1. What content blocker has blocked (or upgraded, or blocked cookies) this or that request
2. What exact rule was triggered

For the main document and frames, we'd like to see how many `css-display-none` rules were injected.

If there is a rule with `action: ignore-previous-rules` that was triggered on a request, we'd like to see this information as well.

[Aaron Gyes]

Yes the console logging is wildly insufficient. It's very hard to figure out what is going on aside from trial and error. I'd appreciate being able to modify rules for testing and flip them on and off in the inspector as well.

[Maciej Stachowiak]

Cc'd some Web Inspector folks who may be inspired by the tools ideas here. The suggested changes to Network and Elements tabs to expose the action of content blockers seem like a good idea to me.

[Neil Jhaveri]

This would be helpful for my app too, here's my use case...

I'm an email client author. Many email services allow senders to include "tracking pixels" to know when the recipient opened a message. Disabling remote images is the traditional fix, but this is a kindof painful for the typical user to run with. IMO significant value can be provided by blocking tracking pixels from a blocklist, but otherwise allowing remote images.

Objective:
This callback would allow me to present a "A tracking pixel from SERVICE_NAME was blocked" banner in the UI, and a button to load it anyways. To do this, I would need to know that (A) a blocker rule fired, and (B) the URL that was blocked (so I can say "from SERVICE_NAME").

(As of now I think my only option is to iterate every image and check for a blocked pattern match manually, which is not very efficient.)