-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add AdGuard's "$removeparam" as an alias for "$queryprune" #1356
Comments
This is incorrect. Where did you get the idea that uBO removes it only "in the URL bar"? |
I took a guess based on how https://chrome.google.com/webstore/detail/redirector/ocgpenflpmgnfapjedencafcfakcekcd primarily does it, as I presumed that was how browser extensions preferred doing it. I apologise for my mistake. |
You took a guess based of how uBO works on a completely unrelated extension somewhere in the Chrome store, while not coming to the same guess-based conclusion at how AdGuard work? How did you discovered that uBO now supported In any case, the main difference between AdGuard's I could align uBO's behavior to |
AdguardTeam/CoreLibs#1372 (comment)
On a whim I decided to use the uBO logger to look through invalid entries in regional lists. For this specific test session I used Vivaldi, whose uBO version apparently hadn't been updated for some weeks and was still on 1.30.6, thus the logger claimed that |
Fine tuning
|
New filter options ================== Strict partyness: `1P`, `3P` ---------------------------- The current options 1p/3p are meant to "weakly" match partyness, i.e. a network request is considered 1st-party to its context as long as both the context and the request share the same base domain. The new partyness options are meant to check for strict partyness, i.e. a network request will be considered 1st-party if and only if both the context and the request share the same hostname. For examples: - context: `www.example.org` - request: `www.example.org` - `1p`: yes, `1P`: yes - `3p`: no, `3P`: no - context: `www.example.org` - request: `subdomain.example.org` - `1p`: yes, `1P`: no - `3p`: no, `3P`: yes - context: `www.example.org` - request: `www.example.com` - `1p`: no, `1P`: no - `3p`: yes, `3P`: yes The strict partyness options will be visually emphasized in the editor so as to prevent mistakenly using `1P` or `3P` where weak partyness is meant to be used. Filter on response headers: `header=` ------------------------------------- Currently experimental and under evaluation. Disabled by default, enable by toggling `filterOnHeaders` to `true` in advanced settings. Ability to filter network requests according to whether a specific response header is present and whether it matches or does not match a specific value. For example: *$1p,3P,script,header=via:1\.1\s+google The above filter is meant to block network requests which fullfill all the following conditions: - is weakly 1st-party to the context - is not strictly 1st-party to the context - is of type `script` - has a response HTTP header named `via`, which value matches the regular expression `1\.1\s+google`. The matches are always performed in a case-insensitive manner. The header value is assumed to be a literal regular expression, except for the following special characters: - to anchor to start of string, use leading `|`, not `^` - to anchor to end of string, use trailing `|`, not `$` - to invert the test, use a leading `!` To block a network request if it merely contains a specific HTTP header is just a matter of specifying the header name without a header value: *$1p,3P,script,header=via Generic exception filters can be used to disable specific block `header=` filters, i.e. `@@*$1p,3P,script,header` will override the block `header=` filters given as example above. Dynamic filtering's `allow` rules override block `headers=` filters. Important: It is key that filter authors use as many narrowing filter options as possible when using the `header=` option, and the `header=` option should be used ONLY when other filter options are not sufficient. More documentation justifying the purpose of `header=` option will be provided eventually if ever it is decided to move it from experimental to stable status. To be decided: to restrict usage of this filter option to only uBO's own filter lists or "My filters". Changes ======= Fine tuning `queryprune=` ------------------------- The following changes have been implemented: The special value `*` (i.e. `queryprune=*`) means "remove all query parameters". If the `queryprune=` value is made only of alphanumeric characters (including `_`), the value will be internally converted to regex equivalent `^value=`. This ensures a better future compatibility with AdGuard's `removeparam=`. If the `queryprune=` value starts with `!`, the test will be inverted. This can be used to remove all query parameters EXCEPT those who match the specified value. Other ----- The legacy code to test for spurious CSP reports has been removed. This is no longer an issue ever since uBO redirects to local resources through web accessible resources. Notes ===== The following new and recently added filter options are not compatible with Chromium's manifest v3 changes: - `queryprune=` - `1P` - `3P` - `header=`
Related issue: - uBlockOrigin/uBlock-issues#1356 Related commit: - bde3164 It is not possible to achieve perfect compatiblity at this point, but reasonable compatibility should be achieved for a majority of instances of `removeparam=`. Notable differences: -------------------- uBO always matches in a case insensitive manner, there is no need to ask for case-insensitivity, and no need to use uppercase characters in `queryprune=` values. uBO does not escape special regex characters since the `queryprune=` values are always assumed to be literal regex expression (leaving out the documented special characters). This means `removeparam=` with characters which are special regex characters won't be properly translated and are unlikely to work properly in uBO. For example, the `queryprune` value of a filter such as `$removeparam=__xts__[0]` internally become the literal regex `/__xts__[0]/`, and consequently would not match a query parameter such as `...?__xts__[0]=...`. Notes: ------ Additionally, for performance reason, when uBO encounter a pattern-less `queryprune=` (or `removeparam=`) filter, it will try to extract a valid pattern from the `queryprune=` value. For instance, the following filter: $queryprune=utm_campaign Will be translated internally into: utm_campaign$queryprune=utm_campaign The logger will reflect this internal translation.
I am reopening because the requirement to use Since One other ambiguity which can still arise is the use of comma The only thing I want which is not the way Something else, consider the following filter:
There is an ambiguity in there, this can be either:
To resolve that sort of ambiguity, I added the noop filter option
Then it becomes the first case, i.e. the plain pattern cc @ameshkov |
Related discussions: - uBlockOrigin/uBlock-issues#1356 (comment) - AdguardTeam/CoreLibs#1384 Changes: Negation character is `~` (instead of `!`). Drop special anchor character `|` -- leading `|` will be supported until no such filter is present in uBO's own filter lists. For example, instance of `queryprune=|ad` will have to be replaced with `queryprune=/^ad/` (or `queryprune=ad` if the name of the parameter to remove is exactly `ad`). Align semantic with that of AdGuard's `removeparam=`, except that specifying multiple `|`-separated names is not supported.
Would you like to keep
This is an option indeed, but what about simply escaping
Makes perfect sense to me.
Very good idea. This would also help with simple path-based rules like |
@gorhill let me plz answer here to your comment in the CL repo, it'd be more convenient to keep the discussion in one place.
No problem, I agree that it does not provide much value.
So it should be something like this, right?
What do you think about negating $removeparam?
|
Yes. Frankly I don't know why I went with
Yes, agreed, actually excepting Now as I read your comment, it just occurred to me I could also drop the special form to remove all query parameters:
And instead use:
I think this makes it clear that a valueless |
Yes, looks clearer to me. There's also a question about escaping special characters in the prev comment:
Also, please check the suggested syntax: $removeparam / $querypruneRules with
SyntaxBasic syntax
Regular expressions You can also use regular expressions to match query parameters and/or their values:
Remove all query parameters Specify naked
Inversion Use
Negating $removeparam This sort of rules work pretty much the same way it works with Use
Examples
AdGuard-specific stuff below; can be ignored:
|
Looks all good to me. What's the idea to remove params only for |
Yeah, good catch, added As I recall, there are no other body-less HTTP methods. |
This will be done runtime by checking if |
As per agreed upon discussion, `queryprune` now follows exactly the syntax of AdGuard's `removeparam`. Related discussion: - uBlockOrigin/uBlock-issues#1356 Related issue: - uBlockOrigin/uBlock-issues#760 For the short term, `queryprune` will still interpret a leading `|` to mean "anchor to start of name", until no such filters are present in uBO's own filter lists.
That's a good point, currently I went with the CSP approach, i.e. the exception excludes a blocking
Will be excepted by:
But not by:
It's will be more complicated if we want to except at runtime, not sure I like the implications performance-wise. |
Fixed, thank you!
@gorhill which one you prefer? Tbh, the more I think about it, the more I think that people are used to csp-like already. |
Right I prefer Anyway, I will go along with what you decide, probably performance concerns are unwarranted when the code path to remove parameters is already reached anyway. |
Let's do it the easy way then, I don't see much benefit from introducing a new "negation" scheme, the very same result can be achieved using the one everyone is already used to. Please check the updated spec. |
I think there is one thing we should clarify here. What if one adds a rule with the regex which matches the
http://example.com/page?utm**-**source=test => http://example.com/page?utm**_**source=test |
You mean if a filter maintainer just adds something like
Fixed it, thank you! |
Yup, such a rule will match for example |
Well, yeah, there's a minor inconsistency indeed and it bothers me a little. On the other hand, the syntax is simpler this way. Let's just mention that this sort of issues is possible and encourage filters maintainers to use |
The two examples need correction, the trailing |
@gorhill good catch, thank you! |
@ameshkov There is a missing detail regarding parameter values: are the tests made against decoded parameter values, or against raw values (as they appear in URLs)? |
@gorhill huh, that's a good question. I guess I would prefer to do the test against decoded values, what do you think? |
This is what I currently implemented, but then this issue appeared: #1717. I had to fix it (AdGuard is unaffected) -- and this made me wonder how filter list authors should deal with |
In the address bar, they see decoded values. But yeah, they indeed use dev tools mostly and do not just grab parameters from the address bar. I am okay with matching raw values. If you're going to use them, let's update the spec accordingly. |
Prerequisites
Description
I've been working on https://raw.githubusercontent.com/DandelionSprout/adfilt/master/LegitimateURLShortener.txt for some weeks now, which I did while presuming that only AdGuard had a feature to remove $/& parameters from URLs. But during some syntax checking of regional lists, I discovered that
$queryprune
seems to have been added to uBO.However, those two
$
qualifiers seem to me to do the exact same thing, with only 2 current exceptions:$removeparam
does not seem to need|
in its values.I therefore propose that they should be made into aliases of each other. This would save me from having to maintain and sync two different versions of that list, one for uBO and one for AdGuard.
A specific URL where the issue occurs
https://raw.githubusercontent.com/DandelionSprout/adfilt/master/LegitimateURLShortener.txt
Steps to Reproduce
Invalid network filter in https://raw.githubusercontent.com/DandelionSprout/adfilt/master/LegitimateURLShortener.txt:
.Expected behavior:
$removeparam
is treated as being$queryprune
Actual behavior:
$removeparam
does not work in uBO.Your environment
The text was updated successfully, but these errors were encountered: