Is this project for ANTLR vs IDEA integration dead or alive

[gnom7]

Hey, I see in README it is recommended to use released 0.1 version:

Before 0.1, it was recommended to add this Git repo as a submodule of your own project, or to copy the source files directly.

It is now recommended to use the Maven dependency. The main breaking change is that the base package has been renamed from org.antlr.jetbrains.adaptor to org.antlr.intellij.adaptor

But since there is only one release, produced back in 2019 I'm wondering if it is still the recommended approach or should I rather include source in my own codebase? Project doesn't seem to be abandoned since I see some commits being added over time and 0.1 being released 5 years ago probably lags behind significantly by now.

Is there any resources/guides/documentation concerning ANTLR vs IDEA integration? I'd really prefer to use ANTLR but have hard time figuring out how to integrate it with IDEA/PSI w/o writing and supporting boilerplate code that could have been autogenerated otherwise (org.antlr.jetbrains.sample.psi package in jetbrains-plugin-sample). IDEA GrammarKit seems to generate lots of similar stuff automatically from grammar, but I'd much prefer to stick to ANTLR unless I'm really forced into GrammarKit due to lack of integration support. Is this project maintained? Are there any plans to improve IDEA integration support? How should I get the latest version of this project?

[bjansen]

Hi,

Using the Maven dependency is still the recommended approach. Nothing has been released in the last 5 years because nothing really changed for a long time. There's a few commits that are worth releasing though. I tend to put releases on hold because there's still a couple things that could be added to the next version, then I forget about them because I'm working on other things and I end up in this kind of situation 😅. I'm now trying to change that and release stuff immediately even if there are other things to do (better now than never), so I guess it's a good time to release a new version of this adaptor.

If you're familiar with ANTLR or if you're working on a language that has an existing ANTLR grammar, then this adaptor is the way to go. We don't generate code (yet), I guess it would be a bit overkill to generate one class per grammar rule (I don't remember if that's what GrammarKit is doing). Existing plugins like intellij-plugin-v4 and pebble-intellij use generic PSI classes and sometimes subclasses when needed. This allows for a pretty good IDEA integration in my experience.

[gnom7]

I wonder whether it makes sense to hookup into existing IDEA GrammarKit plugin to provide seamless ANTLR integration given GrammarKit is openly developed and used in JetBrains reference docs and tutorials. Providing ANTLR as and alternative to other 3rd party parser generators (JFlex) there probably would have made a lot of sense and better exposed ANTLR to audiences. Does it make sense? Are there any roadblocks (like I could imagine such integration might require ANTLR grammar extension to support similar PSI-related features GrammarKit grammar files support)? Why different approach have been chosen?

[bjansen]

JFlex only generates lexers, GrammarKit generates parsers that delegate the lexing part to JFlex, and ANTLR generates both a lexer and a parser. Moreover, ANTLR and GrammarKit each have their own grammar syntax and paradigm. ANTLR is a general purpose parser generator that supports multiple target languages, and GrammarKit is specifically tailored to generate PSI parsers for the IntelliJ platform.

[gnom7]

ok, so in such approach it would not be possible to make any use of existing GrammarKit infra/code? makes sense

thank you and looking forward for a new release of the adaptor :)

[gnom7]

btw do you have or plan some documentation on somewhat non-obvious parts of implementation like lexer state support etc? I'm poking with some toy plugin and getting 'Discontinuous sequence of tokens is generated by lexer', I suspect it is because I skip WS symbols in my lexer, am I right? Can I find about this in some documentation/tutorial? I don't mean to turn GitHub issues into an Q&A, sorry

[bjansen]

No, unfortunately there's no real documentation explaining what needs to be done to build a plugin based on ANTLR grammars. There's a plugin sample but it's outdated and the readme lacks info, ideally it should explain why the code does what it does.

You're right, whitespace is significant in PSI parsers, see https://plugins.jetbrains.com/docs/intellij/implementing-parser-and-psi.html#whitespace-and-comments

I don't mean to turn GitHub issues into an Q&A, sorry

Well Discussions is not enabled on this repo, so it's all right :)

[gnom7]

@bjansen hello again :)
To address issue caused by skipped WS (-> skip) I initially tried to handle them explicitly at the lexer and the parser, which is cumbersome, but it worked.
Then I checked plugin sample you referenced and redirected WS to HIDDEN channel, this resulted in all parse tree nodes being direct children of the root file node (flat, invalid tree), any clues of what I might be missing given such a symptom?

EDIT: alright, it looks like I have had to override getWhitespaceTokens in ParserDefinition and then handle new PsiWhiteSpace type everywhere in the code

[gnom7]

Is it possible to access labels of rule alternatives while using adaptor?

PSIElementTypeFactory.getRuleIElementTypes seems to handle rules themselves w/o any additional categorization of alternatives by labels etc.

Is there a recommended workaround? Should I refactor grammar to make alternatives a separate rules?