Skip to content

Commit

Permalink
Add support for removal of response headers
Browse files Browse the repository at this point in the history 
The syntax to remove response header is a special case
of HTML filtering, whereas the response headers are
targeted, rather than the response body:

  example.com##^responseheader(header-name)

Where `header-name` is the name of the header to
remove, and must always be lowercase.

The removal of response headers can only be applied to
document resources, i.e. main- or sub-frames.

Only a limited set of headers can be targeted for
removal:

  location
  refresh
  report-to
  set-cookie

This limitation is to ensure that uBO never lowers the
security profile of web pages, i.e. we wouldn't want to
remove `content-security-policy`.

Given that the header removal occurs at onHeaderReceived
time, this new ability works for all browsers.

The motivation for this new filtering ability is instance
of website using a `refresh` header to redirect a visitor
to an undesirable destination after a few seconds.
  • Loading branch information
@gorhill
gorhill committed Mar 13, 2021
1 parent af980c5 commit f876b68
Show file tree
Hide file tree
Showing 13 changed files with 201 additions and 76 deletions.
1 change: 1 addition & 0 deletions src/background.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
<script src="js/cosmetic-filtering.js"></script>
<script src="js/scriptlet-filtering.js"></script>
<script src="js/html-filtering.js"></script>
<script src="js/httpheader-filtering.js"></script>
<script src="js/hnswitches.js"></script>
<script src="js/ublock.js"></script>
<script src="js/storage.js"></script>
Expand Down
18 changes: 9 additions & 9 deletions src/css/logger-ui.css
View file
Original file line number Diff line number Diff line change
Expand Up @@ -260,11 +260,11 @@ body.colorBlind #netFilteringDialog > .panes > .details > div[data-status="2"] {
#vwRenderer .logEntry > div[data-tabid="-1"] {
text-shadow: 0 0.2em 0.4em #aaa;
}
#vwRenderer .logEntry > div.cosmeticRealm,
#vwRenderer .logEntry > div.extendedRealm,
#vwRenderer .logEntry > div.redirect {
background-color: rgba(255, 255, 0, 0.1);
}
body.colorBlind #vwRenderer .logEntry > div.cosmeticRealm,
body.colorBlind #vwRenderer .logEntry > div.extendedRealm,
body.colorBlind #vwRenderer .logEntry > div.redirect {
background-color: rgba(0, 19, 110, 0.1);
}
Expand Down Expand Up @@ -324,13 +324,13 @@ body[dir="rtl"] #vwRenderer .logEntry > div > span:first-child {
#vwRenderer .logEntry > div.messageRealm[data-type="tabLoad"] > span:nth-of-type(2) {
text-align: center;
}
#vwRenderer .logEntry > div.cosmeticRealm > span:nth-of-type(2) > span:first-of-type {
#vwRenderer .logEntry > div.extendedRealm > span:nth-of-type(2) > span:first-of-type {
display: none;
}
#vwRenderer .logEntry > div.cosmeticRealm > span:nth-of-type(2) > span:last-of-type {
#vwRenderer .logEntry > div.extendedRealm > span:nth-of-type(2) > span:last-of-type {
pointer-events: none;
}
#vwRenderer .logEntry > div.cosmeticRealm.isException > span:nth-of-type(2) > span:last-of-type {
#vwRenderer .logEntry > div.extendedRealm.isException > span:nth-of-type(2) > span:last-of-type {
text-decoration: line-through rgba(0,0,255,0.7);
}
#vwRenderer .logEntry > div > span:nth-of-type(3) {
Expand Down Expand Up @@ -615,12 +615,12 @@ body[dir="rtl"] #netFilteringDialog > .headers > .tools {
#netFilteringDialog > .headers > .tools > span:hover {
background-color: #eee;
}
#netFilteringDialog.cosmeticRealm > .headers > .dynamic,
#netFilteringDialog.cosmeticRealm > .panes > .dynamic {
#netFilteringDialog.extendedRealm > .headers > .dynamic,
#netFilteringDialog.extendedRealm > .panes > .dynamic {
display: none;
}
#netFilteringDialog.cosmeticRealm > .headers > .static,
#netFilteringDialog.cosmeticRealm > .panes > .static {
#netFilteringDialog.extendedRealm > .headers > .static,
#netFilteringDialog.extendedRealm > .panes > .static {
display: none;
}
#netFilteringDialog > div.panes {
Expand Down
1 change: 1 addition & 0 deletions src/js/background.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ const µBlock = (( ) => { // jshint ignore:line
compiledCosmeticSection: 200,
compiledScriptletSection: 300,
compiledHTMLSection: 400,
compiledHTTPHeaderSection: 500,
compiledSentinelSection: 1000,
compiledBadSubsection: 1,

Expand Down
37 changes: 34 additions & 3 deletions src/js/codemirror/ubo-static-filtering.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -549,6 +549,23 @@ const initHints = function() {

const getExtSelectorHints = function(cursor, line) {
const beg = cursor.ch;
// Special selector case: `^responseheader`
{
const match = /#\^([a-z]+)$/.exec(line.slice(0, beg));
if (
match !== null &&
'responseheader'.startsWith(match[1]) &&
line.slice(beg) === ''
) {
return pickBestHints(
cursor,
match[1],
'',
[ 'responseheader()' ]
);
}
}
// Procedural operators
const matchLeft = /#\^?.*:([^:]*)$/.exec(line.slice(0, beg));
const matchRight = /^([a-z-]*)\(?/.exec(line.slice(beg));
if ( matchLeft === null || matchRight === null ) { return; }
Expand All @@ -561,6 +578,18 @@ const initHints = function() {
return pickBestHints(cursor, matchLeft[1], matchRight[1], hints);
};

const getExtHeaderHints = function(cursor, line) {
const beg = cursor.ch;
const matchLeft = /#\^responseheader\((.*)$/.exec(line.slice(0, beg));
const matchRight = /^([^)]*)/.exec(line.slice(beg));
if ( matchLeft === null || matchRight === null ) { return; }
const hints = [];
for ( const hint of parser.removableHTTPHeaders ) {
hints.push(hint);
}
return pickBestHints(cursor, matchLeft[1], matchRight[1], hints);
};

const getExtScriptletHints = function(cursor, line) {
const beg = cursor.ch;
const matchLeft = /#\+\js\(([^,]*)$/.exec(line.slice(0, beg));
Expand Down Expand Up @@ -607,10 +636,12 @@ const initHints = function() {
let hints;
if ( cursor.ch <= parser.slices[parser.optionsAnchorSpan.i+1] ) {
hints = getOriginHints(cursor, line);
} else if ( parser.hasFlavor(parser.BITFlavorExtScriptlet) ) {
hints = getExtScriptletHints(cursor, line);
} else if ( parser.hasFlavor(parser.BITFlavorExtResponseHeader) ) {
hints = getExtHeaderHints(cursor, line);
} else {
hints = parser.hasFlavor(parser.BITFlavorExtScriptlet)
? getExtScriptletHints(cursor, line)
: getExtSelectorHints(cursor, line);
hints = getExtSelectorHints(cursor, line);
}
return hints;
}
Expand Down
9 changes: 9 additions & 0 deletions src/js/cosmetic-filtering.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,15 @@ FilterContainer.prototype.compileSpecificSelector = function(

/******************************************************************************/

FilterContainer.prototype.compileTemporary = function(parser) {
return {
session: this.sessionFilterDB,
selector: parser.result.compiled,
};
};

/******************************************************************************/

FilterContainer.prototype.fromCompiledContent = function(reader, options) {
if ( options.skipCosmetic ) {
this.skipCompiledContent(reader);
Expand Down
29 changes: 18 additions & 11 deletions src/js/html-filtering.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -237,12 +237,12 @@
µBlock.filteringContext
.duplicate()
.fromTabId(details.tabId)
.setRealm('cosmetic')
.setRealm('extended')
.setType('dom')
.setURL(details.url)
.setDocOriginFromURL(details.url)
.setFilter({
source: 'cosmetic',
source: 'extended',
raw: `${exception === 0 ? '##' : '#@#'}^${selector}`
})
.toLogger();
Expand Down Expand Up @@ -322,6 +322,13 @@
}
};

api.compileTemporary = function(parser) {
return {
session: sessionFilterDB,
selector: parser.result.compiled,
};
};

api.fromCompiledContent = function(reader) {
// Don't bother loading filters if stream filtering is not supported.
if ( µb.canFilterResponseData === false ) { return; }
Expand All @@ -348,15 +355,6 @@
api.retrieve = function(details) {
const hostname = details.hostname;

// https://github.com/gorhill/uBlock/issues/2835
// Do not filter if the site is under an `allow` rule.
if (
µb.userSettings.advancedUserEnabled &&
µb.sessionFirewall.evaluateCellZY(hostname, hostname, '*') === 2
) {
return;
}

const plains = new Set();
const procedurals = new Set();
const exceptions = new Set();
Expand All @@ -379,6 +377,15 @@

if ( plains.size === 0 && procedurals.size === 0 ) { return; }

// https://github.com/gorhill/uBlock/issues/2835
// Do not filter if the site is under an `allow` rule.
if (
µb.userSettings.advancedUserEnabled &&
µb.sessionFirewall.evaluateCellZY(hostname, hostname, '*') === 2
) {
return;
}

const out = { plains, procedurals };

if ( exceptions.size === 0 ) {
Expand Down
45 changes: 29 additions & 16 deletions src/js/logger-ui.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@

/******************************************************************************/

// TODO: fix the inconsistencies re. realm vs. filter source which have
// accumulated over time.

const messaging = vAPI.messaging;
const logger = self.logger = { ownerId: Date.now() };
const logDate = new Date();
Expand Down Expand Up @@ -341,6 +344,11 @@ const processLoggerEntries = function(response) {
/******************************************************************************/

const parseLogEntry = function(details) {
// Patch realm until changed all over codebase to make this unecessary
if ( details.realm === 'cosmetic' ) {
details.realm = 'extended';
}

const entry = new LogEntry(details);

// Assemble the text content, i.e. the pre-built string which will be
Expand Down Expand Up @@ -441,6 +449,8 @@ const viewPort = (( ) => {
const vwLogEntryTemplate = document.querySelector('#logEntryTemplate > div');
const vwEntries = [];

const detailableRealms = new Set([ 'network', 'extended' ]);

let vwHeight = 0;
let lineHeight = 0;
let wholeHeight = 0;
Expand Down Expand Up @@ -672,7 +682,7 @@ const viewPort = (( ) => {
return div;
}

if ( details.realm === 'network' || details.realm === 'cosmetic' ) {
if ( detailableRealms.has(details.realm) ) {
divcl.add('canDetails');
}

Expand All @@ -685,13 +695,13 @@ const viewPort = (( ) => {
}
if ( filteringType === 'static' ) {
divcl.add('canLookup');
if ( filter.modifier === true ) {
div.setAttribute('data-modifier', '');
}
} else if ( filteringType === 'cosmetic' ) {
} else if ( details.realm === 'extended' ) {
divcl.add('canLookup');
divcl.toggle('isException', filter.raw.startsWith('#@#'));
}
if ( filter.modifier === true ) {
div.setAttribute('data-modifier', '');
}
}
span = div.children[1];
if ( renderFilterToSpan(span, cells[1]) === false ) {
Expand Down Expand Up @@ -1575,15 +1585,15 @@ const reloadTab = function(ev) {
rawFilter: rawFilter,
});
handleResponse(response);
} else if ( targetRow.classList.contains('cosmeticRealm') ) {
} else if ( targetRow.classList.contains('extendedRealm') ) {
const response = await messaging.send('loggerUI', {
what: 'listsFromCosmeticFilter',
url: targetRow.children[6].textContent,
rawFilter: rawFilter,
});
handleResponse(response);
}
} ;
};

const fillSummaryPane = function() {
const rows = dialog.querySelectorAll('.pane.details > div');
Expand All @@ -1595,7 +1605,7 @@ const reloadTab = function(ev) {
text = filterFromTargetRow();
if (
(text !== '') &&
(trcl.contains('cosmeticRealm') || trcl.contains('networkRealm'))
(trcl.contains('extendedRealm') || trcl.contains('networkRealm'))
) {
toSummaryPaneFilterNode(rows[0], text);
} else {
Expand All @@ -1607,7 +1617,7 @@ const reloadTab = function(ev) {
(
trcl.contains('dynamicHost') ||
trcl.contains('dynamicUrl') ||
trcl.contains('switch')
trcl.contains('switchRealm')
)
) {
rows[2].children[1].textContent = text;
Expand Down Expand Up @@ -1677,7 +1687,9 @@ const reloadTab = function(ev) {

// Fill dynamic URL filtering pane
const fillDynamicPane = function() {
if ( targetRow.classList.contains('cosmeticRealm') ) { return; }
if ( targetRow.classList.contains('extendedRealm') ) {
return;
}

// https://github.com/uBlockOrigin/uBlock-issues/issues/662#issuecomment-509220702
if ( targetType === 'doc' ) { return; }
Expand Down Expand Up @@ -1712,8 +1724,6 @@ const reloadTab = function(ev) {
}

colorize();

uDom('#modalOverlayContainer [data-pane="dynamic"]').removeClass('hide');
};

const fillOriginSelect = function(select, hostname, domain) {
Expand All @@ -1733,7 +1743,9 @@ const reloadTab = function(ev) {

// Fill static filtering pane
const fillStaticPane = function() {
if ( targetRow.classList.contains('cosmeticRealm') ) { return; }
if ( targetRow.classList.contains('extendedRealm') ) {
return;
}

const template = vAPI.i18n('loggerStaticFilteringSentence');
const rePlaceholder = /\{\{[^}]+?\}\}/g;
Expand Down Expand Up @@ -1842,8 +1854,8 @@ const reloadTab = function(ev) {
}
);
dialog.classList.toggle(
'cosmeticRealm',
targetRow.classList.contains('cosmeticRealm')
'extendedRealm',
targetRow.classList.contains('extendedRealm')
);
targetDomain = domains[0];
targetPageDomain = domains[1];
Expand Down Expand Up @@ -2403,10 +2415,10 @@ const popupManager = (( ) => {
// Filter hit stats' MVP ("minimum viable product")
//
const loggerStats = (( ) => {
const enabled = false;
const filterHits = new Map();
let dialog;
let timer;

const makeRow = function() {
const div = document.createElement('div');
div.appendChild(document.createElement('span'));
Expand Down Expand Up @@ -2470,6 +2482,7 @@ const loggerStats = (( ) => {

return {
processFilter: function(filter) {
if ( enabled !== true ) { return; }
if ( filter.source !== 'static' && filter.source !== 'cosmetic' ) {
return;
}
Expand Down
21 changes: 8 additions & 13 deletions src/js/messaging.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1440,21 +1440,14 @@ const getURLFilteringData = function(details) {
const compileTemporaryException = function(filter) {
const parser = new vAPI.StaticFilteringParser();
parser.analyze(filter);
if ( parser.shouldDiscard() ) { return {}; }
let selector = parser.result.compiled;
let session;
if ( (parser.flavorBits & parser.BITFlavorExtScriptlet) !== 0 ) {
session = µb.scriptletFilteringEngine.getSession();
} else if ( (parser.flavorBits & parser.BITFlavorExtHTML) !== 0 ) {
session = µb.htmlFilteringEngine.getSession();
} else {
session = µb.cosmeticFilteringEngine.getSession();
}
return { session, selector };
if ( parser.shouldDiscard() ) { return; }
return µb.staticExtFilteringEngine.compileTemporary(parser);
};

const toggleTemporaryException = function(details) {
const { session, selector } = compileTemporaryException(details.filter);
const result = compileTemporaryException(details.filter);
if ( result === undefined ) { return false; }
const { session, selector } = result;
if ( session.has(1, selector) ) {
session.remove(1, selector);
return false;
Expand All @@ -1464,7 +1457,9 @@ const toggleTemporaryException = function(details) {
};

const hasTemporaryException = function(details) {
const { session, selector } = compileTemporaryException(details.filter);
const result = compileTemporaryException(details.filter);
if ( result === undefined ) { return false; }
const { session, selector } = result;
return session && session.has(1, selector);
};

Expand Down
Loading

2 comments on commit f876b68

@ameshkov
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @gorhill,

We'd like to support this functionality as well, but I've got some questions about the current implementation.

First of all, why limiting it that much and making it a "cosmetic-like" option? I mean I've got it, currently, it's just a measure against "refresh", but there are actually more use-cases and I am pretty sure there will be cases that will require a more fine-grained matching.

What do you think about changing it to a basic rule modifier? Something like ||example.com^$removeresheader=location?

@gorhill
Copy link
Owner Author

@gorhill gorhill commented on f876b68 Mar 15, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

making it a "cosmetic-like" option

It's in the "static extended" category (hostname-based), of which "cosmetic" is a subset. "Static extended" (hostname-based) is for filter syntax beyond "static network" (pattern-based), subsets of which are "cosmetic", "scriptlet injection", "HTML filtering".

Many reasons for this choice:

  • Pattern-based means less performant lookup
    • I considered that hostname-based should be sufficient
    • Given that the filter applies only to document resource, even with pattern-based filtering, I expect filters like *$removeheader=refresh,domain=..., which means no real gain from static extended-based filtering
  • Makes implementation much simpler (on my side at least)
    • With pattern-based, we have to mind all the other options which may or may not mix well

Please sign in to comment.