Add support for 1P, 3P, header= filter options and other changes

New filter options
==================

Strict partyness: `1P`, `3P`
----------------------------

The current options 1p/3p are meant to "weakly" match partyness, i.e. a
network request is considered 1st-party to its context as long as both the
context and the request share the same base domain.

The new partyness options are meant to check for strict partyness, i.e. a
network request will be considered 1st-party if and only if both the context
and the request share the same hostname.

For examples:

- context: `www.example.org`
- request: `www.example.org`
- `1p`: yes, `1P`: yes
- `3p`: no,  `3P`: no

- context: `www.example.org`
- request: `subdomain.example.org`
- `1p`: yes, `1P`: no
- `3p`: no,  `3P`: yes

- context: `www.example.org`
- request: `www.example.com`
- `1p`: no, `1P`: no
- `3p`: yes,  `3P`: yes

The strict partyness options will be visually emphasized in the editor so as
to prevent mistakenly using `1P` or `3P` where weak partyness is meant to be
used.

Filter on response headers: `header=`
-------------------------------------

Currently experimental and under evaluation. Disabled by default, enable by
toggling `filterOnHeaders` to `true` in advanced settings.

Ability to filter network requests according to whether a specific response
header is present and whether it matches or does not match a specific value.

For example:

    *$1p,3P,script,header=via:1\.1\s+google

The above filter is meant to block network requests which fullfill all the
following conditions:

- is weakly 1st-party to the context
- is not strictly 1st-party to the context
- is of type `script`
- has a response HTTP header named `via`, which value matches the regular
  expression `1\.1\s+google`.

The matches are always performed in a case-insensitive manner.

The header value is assumed to be a literal regular expression, except for
the following special characters:

- to anchor to start of string, use leading `|`, not `^`
- to anchor to end of string, use trailing `|`, not `$`
- to invert the test, use a leading `!`

To block a network request if it merely contains a specific HTTP header is
just a matter of specifying the header name without a header value:

    *$1p,3P,script,header=via

Generic exception filters can be used to disable specific block `header=`
filters, i.e. `@@*$1p,3P,script,header` will override the block `header=`
filters given as example above.

Dynamic filtering's `allow` rules override block `headers=` filters.

Important: It is key that filter authors use as many narrowing filter options
as possible when using the `header=` option, and the `header=` option should
be used ONLY when other filter options are not sufficient.

More documentation justifying the purpose of `header=` option will be
provided eventually if ever it is decided to move it from experimental to
stable status.

To be decided: to restrict usage of this filter option to only uBO's own
filter lists or "My filters".

Changes
=======

Fine tuning `queryprune=`
-------------------------

The following changes have been implemented:

The special value `*` (i.e. `queryprune=*`) means "remove all query
parameters".

If the `queryprune=` value is made only of alphanumeric characters
(including `_`), the value will be internally converted to regex  equivalent
`^value=`. This ensures a better future compatibility with AdGuard's
`removeparam=`.

If the `queryprune=` value starts with `!`, the test will be inverted. This
can be used to remove all query parameters EXCEPT those who match the
specified value.

Other
-----

The legacy code to test for spurious CSP reports has been removed. This
is no longer an issue ever since uBO redirects to local resources through
web accessible resources.

Notes
=====

The following new and recently added filter options are not compatible with
Chromium's manifest v3 changes:

- `queryprune=`
- `1P`
- `3P`
- `header=`

src/css/codemirror.css

@@ -74,9 +74,9 @@
 .cm-s-default .cm-keyword {
     color: var(--sf-keyword-ink);
     }
-.cm-s-default .cm-regex {
+.cm-s-default .cm-notice {
     text-underline-position: under;
-    text-decoration-color: var(--sf-regex-ink);
+    text-decoration-color: var(--sf-notice-ink);
     text-decoration-style: solid;
     text-decoration-line: underline;
     }

src/css/themes/default.css

@@ -191,7 +191,7 @@
     --sf-error-ink: #ff0000;
     --sf-error-surface: #ff000016;
     --sf-keyword-ink: var(--purple-60);
-    --sf-regex-ink: var(--light-gray-60);
+    --sf-notice-ink: var(--light-gray-60);
     --sf-tag-ink: #117700;
     --sf-value-ink: var(--orange-80);
     --sf-variable-ink: var(--default-ink);

src/js/background.js

@@ -61,9 +61,10 @@ const µBlock = (( ) => { // jshint ignore:line
         debugScriptletInjector: false,
         disableWebAssembly: false,
         extensionUpdateForceReload: false,
+        filterAuthorMode: false,
+        filterOnHeaders: false,
         ignoreRedirectFilters: false,
         ignoreScriptInjectFilters: false,
-        filterAuthorMode: false,
         loggerPopupType: 'popup',
         manualUpdateAssetFetchPeriod: 500,
         popupFontSize: 'unset',

src/js/codemirror/ubo-static-filtering.js

@@ -212,20 +212,42 @@ CodeMirror.defineMode('ubo-static-filtering', function() {
 
     const colorNetOptionSpan = function(stream) {
         const bits = parser.slices[parserSlot];
-        let style;
         if ( (bits & parser.BITComma) !== 0  ) {
-            style = 'def strong';
             netOptionValueMode = false;
-        } else if ( netOptionValueMode ) {
+            stream.pos += parser.slices[parserSlot+2];
+            parserSlot += 3;
+            return 'def strong';
+        }
+        if ( netOptionValueMode ) {
             return colorNetOptionValueSpan(stream, bits);
-        } else if ( (bits & parser.BITTilde) !== 0 ) {
-            style = 'keyword strong';
-        } else if ( (bits & parser.BITEqual) !== 0 ) {
+        }
+        if ( (bits & parser.BITTilde) !== 0 ) {
+            stream.pos += parser.slices[parserSlot+2];
+            parserSlot += 3;
+            return 'keyword strong';
+        }
+        if ( (bits & parser.BITEqual) !== 0 ) {
             netOptionValueMode = true;
+            stream.pos += parser.slices[parserSlot+2];
+            parserSlot += 3;
+            return 'def';
         }
-        stream.pos += parser.slices[parserSlot+2];
-        parserSlot += 3;
-        return style || 'def';
+        const to = parser.skipUntil(
+            parserSlot,
+            parser.commentSpan.i,
+            parser.BITComma | parser.BITEqual
+        );
+        if (
+            to > parserSlot &&
+            /^[13]P/.test(parser.strFromSlices(parserSlot, to - 3))
+        ) {
+            parserSlot = to;
+            stream.pos = parser.slices[to+1];
+            return 'def notice';
+        }
+        parserSlot = to;
+        stream.pos = parser.slices[to+1];
+        return 'def';
     };
 
     const colorNetSpan = function(stream) {
@@ -259,7 +281,7 @@ CodeMirror.defineMode('ubo-static-filtering', function() {
             if ( parser.patternIsRegex() ) {
                 stream.pos = parser.slices[parser.optionsAnchorSpan.i+1];
                 parserSlot = parser.optionsAnchorSpan.i;
-                return 'variable regex';
+                return 'variable notice';
             }
             if ( (parser.slices[parserSlot] & (parser.BITAsterisk | parser.BITCaret)) !== 0 ) {
                 stream.pos += parser.slices[parserSlot+2];

src/js/pagestore.js

@@ -272,7 +272,6 @@ const PageStore = class {
         this.popupBlockedCount = 0;
         this.largeMediaCount = 0;
         this.largeMediaTimer = null;
-        this.internalRedirectionCount = 0;
         this.allowLargeMediaElementsRegex = undefined;
         this.extraData.clear();
 
@@ -668,11 +667,57 @@ const PageStore = class {
         return result;
     }
 
+    filterOnHeaders(fctxt, headers) {
+        fctxt.filter = undefined;
+
+        if ( this.getNetFilteringSwitch(fctxt) === false ) { return 0; }
+
+        let result = µb.staticNetFilteringEngine.matchHeaders(fctxt, headers);
+        if ( result === 0 ) { return 0; }
+
+        const loggerEnabled = µb.logger.enabled;
+        if ( loggerEnabled ) {
+            fctxt.filter = µb.staticNetFilteringEngine.toLogData();
+        }
+
+        // Dynamic filtering allow rules
+        // URL filtering
+        if (
+            result === 1 &&
+            µb.sessionURLFiltering.evaluateZ(
+                fctxt.getTabHostname(),
+                fctxt.url,
+                fctxt.type
+            ) === 2
+        ) {
+            result = 2;
+            if ( loggerEnabled ) {
+                fctxt.filter = µb.sessionURLFiltering.toLogData();
+            }
+        }
+        // Hostname filtering
+        if (
+            result === 1 &&
+            µb.userSettings.advancedUserEnabled &&
+            µb.sessionFirewall.evaluateCellZY(
+                fctxt.getTabHostname(),
+                fctxt.getHostname(),
+                fctxt.type
+            ) === 2
+        ) {
+            result = 2;
+            if ( loggerEnabled ) {
+                fctxt.filter = µb.sessionFirewall.toLogData();
+            }
+        }
+
+        return result;
+    }
+
     redirectBlockedRequest(fctxt) {
         if ( µb.hiddenSettings.ignoreRedirectFilters === true ) { return; }
         const directive = µb.staticNetFilteringEngine.redirectRequest(fctxt);
         if ( directive === undefined ) { return; }
-        this.internalRedirectionCount += 1;
         if ( µb.logger.enabled !== true ) { return; }
         fctxt.pushFilter(directive.logData());
         if ( fctxt.redirectURL === undefined ) { return; }

src/js/static-filtering-parser.js

@@ -346,6 +346,7 @@ const Parser = class {
     //     patternRightAnchorSpan: first slice to right-hand pattern anchor
     //     optionsAnchorSpan: first slice to options anchor
     //     optionsSpan: first slice to options
+    //     commentSpan: first slice to trailing comment
     analyzeNet() {
         let islice = this.leftSpaceSpan.len;
 
@@ -369,7 +370,7 @@ const Parser = class {
         }
 
         // Assume no options
-        this.optionsAnchorSpan.i = this.optionsSpan.i =  this.commentSpan.i;
+        this.optionsAnchorSpan.i = this.optionsSpan.i = this.commentSpan.i;
 
         // Assume all is part of pattern
         this.patternSpan.i = islice;
@@ -1900,41 +1901,44 @@ const BITFlavorNetAnchor         = BITFlavorNetLeftAnchor | BITFlavorNetRightAnc
 const OPTTokenMask               = 0x000000ff;
 const OPTTokenInvalid            =  0;
 const OPTToken1p                 =  1;
-const OPTToken3p                 =  2;
-const OPTTokenAll                =  3;
-const OPTTokenBadfilter          =  4;
-const OPTTokenCname              =  5;
-const OPTTokenCsp                =  6;
-const OPTTokenCss                =  7;
-const OPTTokenDenyAllow          =  8;
-const OPTTokenDoc                =  9;
-const OPTTokenDomain             = 10;
-const OPTTokenEhide              = 11;
-const OPTTokenEmpty              = 12;
-const OPTTokenFont               = 13;
-const OPTTokenFrame              = 14;
-const OPTTokenGenericblock       = 15;
-const OPTTokenGhide              = 16;
-const OPTTokenImage              = 17;
-const OPTTokenImportant          = 18;
-const OPTTokenInlineFont         = 19;
-const OPTTokenInlineScript       = 20;
-const OPTTokenMedia              = 21;
-const OPTTokenMp4                = 22;
-const OPTTokenObject             = 23;
-const OPTTokenOther              = 24;
-const OPTTokenPing               = 25;
-const OPTTokenPopunder           = 26;
-const OPTTokenPopup              = 27;
-const OPTTokenRedirect           = 28;
-const OPTTokenRedirectRule       = 29;
-const OPTTokenQueryprune         = 30;
-const OPTTokenScript             = 31;
-const OPTTokenShide              = 32;
-const OPTTokenXhr                = 33;
-const OPTTokenWebrtc             = 34;
-const OPTTokenWebsocket          = 35;
-const OPTTokenCount              = 36;
+const OPTToken1pStrict           =  2;
+const OPTToken3p                 =  3;
+const OPTToken3pStrict           =  4;
+const OPTTokenAll                =  5;
+const OPTTokenBadfilter          =  6;
+const OPTTokenCname              =  7;
+const OPTTokenCsp                =  8;
+const OPTTokenCss                =  9;
+const OPTTokenDenyAllow          = 10;
+const OPTTokenDoc                = 11;
+const OPTTokenDomain             = 12;
+const OPTTokenEhide              = 13;
+const OPTTokenEmpty              = 14;
+const OPTTokenFont               = 15;
+const OPTTokenFrame              = 16;
+const OPTTokenGenericblock       = 17;
+const OPTTokenGhide              = 18;
+const OPTTokenHeader             = 19;
+const OPTTokenImage              = 20;
+const OPTTokenImportant          = 21;
+const OPTTokenInlineFont         = 22;
+const OPTTokenInlineScript       = 23;
+const OPTTokenMedia              = 24;
+const OPTTokenMp4                = 25;
+const OPTTokenObject             = 26;
+const OPTTokenOther              = 27;
+const OPTTokenPing               = 28;
+const OPTTokenPopunder           = 29;
+const OPTTokenPopup              = 30;
+const OPTTokenRedirect           = 31;
+const OPTTokenRedirectRule       = 32;
+const OPTTokenQueryprune         = 33;
+const OPTTokenScript             = 34;
+const OPTTokenShide              = 35;
+const OPTTokenXhr                = 36;
+const OPTTokenWebrtc             = 37;
+const OPTTokenWebsocket          = 38;
+const OPTTokenCount              = 39;
 
 //const OPTPerOptionMask           = 0x0000ff00;
 const OPTCanNegate               = 1 <<  8;
@@ -1974,9 +1978,11 @@ Parser.prototype.BITHostname = BITHostname;
 Parser.prototype.BITPeriod = BITPeriod;
 Parser.prototype.BITDash = BITDash;
 Parser.prototype.BITHash = BITHash;
+Parser.prototype.BITNum = BITNum;
 Parser.prototype.BITEqual = BITEqual;
 Parser.prototype.BITQuestion = BITQuestion;
 Parser.prototype.BITPercent = BITPercent;
+Parser.prototype.BITAlpha = BITAlpha;
 Parser.prototype.BITTilde = BITTilde;
 Parser.prototype.BITUnicode = BITUnicode;
 Parser.prototype.BITIgnore = BITIgnore;
@@ -1993,7 +1999,10 @@ Parser.prototype.BITFlavorIgnore = BITFlavorIgnore;
 Parser.prototype.BITFlavorUnsupported = BITFlavorUnsupported;
 Parser.prototype.BITFlavorError = BITFlavorError;
 
-Parser.prototype.OPTTokenInvalid = OPTTokenInvalid;
+Parser.prototype.OPTToken1p = OPTToken1p;
+Parser.prototype.OPTToken1pStrict = OPTToken1pStrict;
+Parser.prototype.OPTToken3p = OPTToken3p;
+Parser.prototype.OPTToken3pStrict = OPTToken3pStrict;
 Parser.prototype.OPTTokenAll = OPTTokenAll;
 Parser.prototype.OPTTokenBadfilter = OPTTokenBadfilter;
 Parser.prototype.OPTTokenCname = OPTTokenCname;
@@ -2003,14 +2012,15 @@ Parser.prototype.OPTTokenDoc = OPTTokenDoc;
 Parser.prototype.OPTTokenDomain = OPTTokenDomain;
 Parser.prototype.OPTTokenEhide = OPTTokenEhide;
 Parser.prototype.OPTTokenEmpty = OPTTokenEmpty;
-Parser.prototype.OPTToken1p = OPTToken1p;
 Parser.prototype.OPTTokenFont = OPTTokenFont;
 Parser.prototype.OPTTokenGenericblock = OPTTokenGenericblock;
 Parser.prototype.OPTTokenGhide = OPTTokenGhide;
+Parser.prototype.OPTTokenHeader = OPTTokenHeader;
 Parser.prototype.OPTTokenImage = OPTTokenImage;
 Parser.prototype.OPTTokenImportant = OPTTokenImportant;
 Parser.prototype.OPTTokenInlineFont = OPTTokenInlineFont;
 Parser.prototype.OPTTokenInlineScript = OPTTokenInlineScript;
+Parser.prototype.OPTTokenInvalid = OPTTokenInvalid;
 Parser.prototype.OPTTokenMedia = OPTTokenMedia;
 Parser.prototype.OPTTokenMp4 = OPTTokenMp4;
 Parser.prototype.OPTTokenObject = OPTTokenObject;
@@ -2025,7 +2035,6 @@ Parser.prototype.OPTTokenScript = OPTTokenScript;
 Parser.prototype.OPTTokenShide = OPTTokenShide;
 Parser.prototype.OPTTokenCss = OPTTokenCss;
 Parser.prototype.OPTTokenFrame = OPTTokenFrame;
-Parser.prototype.OPTToken3p = OPTToken3p;
 Parser.prototype.OPTTokenXhr = OPTTokenXhr;
 Parser.prototype.OPTTokenWebrtc = OPTTokenWebrtc;
 Parser.prototype.OPTTokenWebsocket = OPTTokenWebsocket;
@@ -2045,8 +2054,10 @@ Parser.prototype.OPTNotSupported = OPTNotSupported;
 const netOptionTokenDescriptors = new Map([
     [ '1p', OPTToken1p | OPTCanNegate ],
         [ 'first-party', OPTToken1p | OPTCanNegate ],
+    [ '1P', OPTToken1pStrict ],
     [ '3p', OPTToken3p | OPTCanNegate ],
         [ 'third-party', OPTToken3p | OPTCanNegate ],
+    [ '3P', OPTToken3pStrict ],
     [ 'all', OPTTokenAll | OPTNetworkType | OPTNonCspableType ],
     [ 'badfilter', OPTTokenBadfilter ],
     [ 'cname', OPTTokenCname | OPTAllowOnly | OPTModifierType ],
@@ -2066,6 +2077,7 @@ const netOptionTokenDescriptors = new Map([
     [ 'genericblock', OPTTokenGenericblock | OPTNotSupported ],
     [ 'ghide', OPTTokenGhide | OPTNonNetworkType | OPTNonCspableType | OPTNonRedirectableType ],
         [ 'generichide', OPTTokenGhide | OPTNonNetworkType | OPTNonCspableType | OPTNonRedirectableType ],
+    [ 'header', OPTTokenHeader | OPTMustAssign | OPTAllowMayAssign | OPTNonCspableType | OPTNonRedirectableType ],
     [ 'image', OPTTokenImage | OPTCanNegate | OPTNetworkType | OPTModifiableType | OPTRedirectableType | OPTNonCspableType ],
     [ 'important', OPTTokenImportant | OPTBlockOnly ],
     [ 'inline-font', OPTTokenInlineFont | OPTNonNetworkType | OPTCanNegate | OPTNonCspableType | OPTNonRedirectableType ],
@@ -2097,8 +2109,10 @@ Parser.prototype.netOptionTokenDescriptors =
 Parser.netOptionTokenIds = new Map([
     [ '1p', OPTToken1p ],
         [ 'first-party', OPTToken1p ],
+    [ '1P', OPTToken1pStrict ],
     [ '3p', OPTToken3p ],
         [ 'third-party', OPTToken3p ],
+    [ '3P', OPTToken3pStrict ],
     [ 'all', OPTTokenAll ],
     [ 'badfilter', OPTTokenBadfilter ],
     [ 'cname', OPTTokenCname ],
@@ -2118,6 +2132,7 @@ Parser.netOptionTokenIds = new Map([
     [ 'genericblock', OPTTokenGenericblock ],
     [ 'ghide', OPTTokenGhide ],
         [ 'generichide', OPTTokenGhide ],
+    [ 'header', OPTTokenHeader ],
     [ 'image', OPTTokenImage ],
     [ 'important', OPTTokenImportant ],
     [ 'inline-font', OPTTokenInlineFont ],
@@ -2145,7 +2160,9 @@ Parser.netOptionTokenIds = new Map([
 
 Parser.netOptionTokenNames = new Map([
     [ OPTToken1p, '1p' ],
+    [ OPTToken1pStrict, '1P' ],
     [ OPTToken3p, '3p' ],
+    [ OPTToken3pStrict, '3P' ],
     [ OPTTokenAll, 'all' ],
     [ OPTTokenBadfilter, 'badfilter' ],
     [ OPTTokenCname, 'cname' ],
@@ -2160,6 +2177,7 @@ Parser.netOptionTokenNames = new Map([
     [ OPTTokenFont, 'font' ],
     [ OPTTokenGenericblock, 'genericblock' ],
     [ OPTTokenGhide, 'generichide' ],
+    [ OPTTokenHeader, 'header' ],
     [ OPTTokenImage, 'image' ],
     [ OPTTokenImportant, 'important' ],
     [ OPTTokenInlineFont, 'inline-font' ],
@@ -2300,6 +2318,8 @@ const NetOptionsIterator = class {
             }
             // Keep track of which options are present: any given option can
             // appear only once.
+            // TODO: might need to make an exception for `header=` option so as
+            //       to allow filters which need to match more than one header.
             const tokenId = descriptor & OPTTokenMask;
             if ( tokenId !== OPTTokenInvalid ) {
                 if ( this.tokenPos[tokenId] !== -1 ) {