Test new theme matching and new tokens format

[alexdima]

• @jrieken
• @chrmarti

Cmplexity: 4

Plan item: #18317

The changes are heavily covered through unit tests, but there are some integration things that might not work, so please check:

• create or change a grammar to emit a scope name that contains < or > or &. Please check that these strings do not make it into the editor HTML rendering and also not into the Developer: Inspect TM Scopes widget
• change an editor contribution (in code) to add an inline decoration class name (e.g. link contribution) that contains < or > or &. Please check that these strings do not make it into the editor HTML rendering.
• check that changing a color theme to X produces the same colorization as loading the workbench fresh with the theme X (i.e. caches are correctly cleared)
• open some different files in our most popular languages and in some of the most popular themes. Please compare with the previous stable version, and if you spot differences validate those differences are OK via the Developer: Inspect TM Scopes widget

[smlombardi]

Is there documentation for this new tokens format for theme developers? My theme is no longer coloring items as intended after 1.9.0 insider 2017-01-10T09:15:59.757Z.

[chrmarti]

@smlombardi Could you open a bug report for your case?

[chrmarti]

Verified using, e.g., "name": "comment.line.doubl>e-slash.js" in JavaScript.tmLanguage.json does not corrupt the output:

[chrmarti]

Also verified that changing the class name detected-link in links.ts and stylesContributions.ts to include those characters does not let these show up in the editor. I see them in the DOM properly escaped.

[chrmarti]

Hope that's according to the test description. Also tested the other two points and filed issues.