-
-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch to goyacc-generated parser from participle library #37
Conversation
59c21fb
to
4c33630
Compare
9311958
to
eee30c9
Compare
cf837f2
to
1c470cd
Compare
With this improvement, the performance of I also noticed that jq's implementation for parsing the string interpolation is slightly inefficient. It has a stack for recording the parenthesis state for finding the closing parenthesis of interpolation. But I noticed (at 4e21bd9) that we don't need to distinguish the expression closing parenthesis and string interpolation closing parenthesis. Maybe I will submit a patch to jq soon. |
I'm working on rewriting the query parser. I have been using participle library but I will switch to goyacc-based parser.
Motivation
Using the participle library makes the struct tree too deep. We have to create structs for each precedence priorities. In the current gojq implementation, I embed the builtin functions to skip parsing them every command execution. Deep structure increases the executable size and initial allocation time. By switching to goyacc-based parser, we can reduce the nested depth for struct of the binary operators. Reducing the executable size would be nice for gojq library users too.
gojq -n .
The participle library makes use of reflection. The author seems to take much care about the performance, but it can't be faster than goyacc-based parser. I want to make gojq fast enough as jq.
gojq -n -L . 'include "builtin"; .'
The jq command uses yacc-based parser. Sometimes it is difficult to fix some coner cases of parsing problems. I want to improve the parser compatibility by switching to goyacc.
Also, it is too hard to parse string interpolation using participle library. I solved this problem by generating the regular expression for the string literal of fixed nested depth. But I want to fix is fundamentally by doing the same method of jq.
Breaking changes
This parser switching breaks compatibility for the library users.
Query
stuct is changed a lot.Module
struct is now removed.ModuleLoader
is changed.ParseModule
is removed in favor ofParse
.Drawbacks
The error message of participle is great. It reports the expected token on error. It is hard to implement (not impossible) in yacc-based parser, but maybe it's ok.
The problem is using goyacc is that it does not seem to be developped actively. I'm not sure but the tool may be deprecated in the future. But I believe someone would create yacc style parser then.
Thanks to participle library
I would like to express my gratitude to the author of the participle library. The idea of this library is great. It brought me great productivity so far. It makes it very easy to change the parsing rules or adding some optional tokens. It was always fun to resolving some syntax problem by just moving the struct fields (like #9). I still think the library is great and recommend it to beginners of implementing some interpreters.