Add support for dynamically loading grammar libraries at runtime

[jirutka]

Can you please add support for dynamically loading (any) grammar libraries at runtime? The current approach of vendoring every existing grammar and linking them in build-time is not scalable and not compatible with linux distributions.

This issue has been already solved e.g. in diffsitter (afnanenayet/diffsitter#177) or helix (helix-editor/helix#432).

You can either load a grammar library for the specified language from the system library path (this is typically /lib:/usr/lib:/usr/local/lib, but the path may vary between linux distros; the point is that you don’t depend on exact path, this is handled by libc) as any other system library – grammar libraries are expected to be named libtree-sitter-<lang>.so (diffsitter’s approach).

Or you can look up grammar libraries directly in some predefined directory (e.g. /usr/lib/tree-sitter or some app-specific dir) named as <lang>.so (helix’s approach).

The former is better, but both are acceptable for Linux distros – in the latter case, we can just symlink <your-directory-for-grammars> to e.g. /usr/lib/tree-sitter where system-wide grammars are installed by the system’s package manager (and used by all programs that use tree-sitter). These approaches are not mutually exclusive – you can support both.

Also, you can continue to provide some grammars with tree-grepper (as both diffsitter and helix do, they use git submodules) for the convenience of the users that build tree-grepper themselves and their distro don’t provide tree-sitter grammars yet (for example, Alpine Linux and Arch Linux (AUR) already provide many grammars). The aim is to allow package maintainers to package grammars separately and give users the freedom to choose what grammars they will install.

(I opened the same issue also in Wilfred/difftastic#123)

[BrianHicks]

interesting idea! It's nice to know that it's already been implemented elsewhere.

however, here's the deal: this looks like a pretty big project from where I'm sitting, and I don't know that I'll have time to do it anytime soon. I just didn't want to leave you hanging with no issue comment, because I think this is worthwhile. If you want to pursue it, go for it! I'm happy to help out with "yeah or nah" and give reviews, but I can't take care of the primary development at the moment.

[crides]

I've made a quick proto on my fork. Code is basically from helix, still need to figure out what library path to default to (see the hardcoded path for now), but it does work