New tokens format & theme matching

[alexdima]

Fixes #11597
Fixes #16206
Fixes #3008

TODO:

• Common
  • adopt new tokens representation in model & co.
  • remove paranoia check in ViewLineToken
  • remove model.getMode() and model.getModeId()
  • rename LanguageIdentifier.sid and LanguageIdentifier.iid
  • remove token class name from every token
• Standalone editor
  • generate new tokens format in monarch and manual tokenizers
  • align standalone languages tokenizers to generate classic style tokens
  • clean up standalone tokens css rules and move them to JS
  • match tokens using theme rules in JS for monarch and manual tokenizers
  • expose new tokenization API in standalone editor
  • adopt standalone languages tests for tokenization
  • validate new tokenization using tokenization.html standalone test
  • allow for theme switching in standaloneColorService
  • allow for theme customization in standalone editor
  • check editor with bootstrap css (created follow up task)
  • add inspect tokens command
• adopt new vscode-textmate
  • publish vscode-textmate
  • get theme information to vscode-textmate
  • reimplement theme integration tests
  • allow for theme switching
  • handle encountered languages while tokenizing
  • remove dom layer breaker from TMSyntax.ts
  • add inspect tokens command

[joshpeng]

@alexandrudima So all tokens are now hidden behind mtk__ span classes when using Dev Tools? How do theme and grammar developers see the original tokens before it got flattened? We used to be able to use Dev Tools' element inspection.

This merge has caused an issue displaying key-value pairs with different colors, but without being able to see the new tokens I have no way to address it in tmThemes or custom grammars.

[aeschli]

@joshpeng You can use https://marketplace.visualstudio.com/items?itemName=siegebell.scope-info to see a hover with the scope hierarchy at the given position.
We also have #18068 to think about other features that can help theme and grammar authors. Please add comments there is you have ideas & requests.

[joshpeng]

@aeschli Thanks for the extension link, but this extension seems to only show the original VSC scopes. It doesn't seem to be capable of showing scopes from custom grammar extensions.

[joshpeng]

Before we could piggy back off of VS Code's .detected-link css class to stylize URL links with tmThemes. Now it doesn't work anymore after the flattening. Seems like the .detected-link is applied outside of the mtk__. How can we do this now?

[alexdima]

@aeschli @joshpeng Please use F1 > Developer: Inspect TM Scopes

This will show you the token you are on and three sections:

• metadata used for rendering and operations such as comment line, etc.
• the theme rule that matches and gives the foreground color to the token
• the list of scopes the grammar produces for the token

[siegebell]

@joshpeng scope-info should work for custom grammar extensions, but if conflicting grammars are provided by multiple extensions or vscode, then I bet it is picking the wrong one. With "Developer: Inspect TM Scopes", perhaps the issue is moot, but feel free to submit the issue here (with a reproducable example).