Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel lock free testing, remove potential deadlocks, cache static data, go to descriptor via test #3752

Merged
merged 20 commits into from
Jun 26, 2022
Merged

Parallel lock free testing, remove potential deadlocks, cache static data, go to descriptor via test #3752

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
84c9780
Implement fast parallel lock-free in-process testing for Java
KvanTTT Jun 21, 2022
e942e1a
Add final modifier to all static fields where it's possible
KvanTTT Jun 17, 2022
36b885c
Remove doubtful and not used static fields
KvanTTT Jun 17, 2022
4ccbb41
Fix potential deadlocks in Java runtime
KvanTTT Jun 17, 2022
dbce6b2
Unify C#, C++, Dart, Go, Python runtimes
KvanTTT Jun 19, 2022
8861ac4
Use AtomicInteger instead of int for static PredictionContext.globalN…
KvanTTT Jun 17, 2022
3c30268
Remove lock in Generator (now ANTLR tool is thread-safe)
KvanTTT Jun 17, 2022
7c5b3c2
Cache templates for codegen to static field, load only one time
KvanTTT Nov 26, 2021
f19cd67
Cache LeftRecursiveRules.stg to static field
KvanTTT Jun 18, 2022
6681a7a
Cache ANTLR error message templates to static field
KvanTTT Jun 18, 2022
93970f4
Cache runtime test templates to static field
KvanTTT Jun 18, 2022
1b9d77c
Clarify message of failed runtime tests
KvanTTT Jun 19, 2022
99fab5c
Pass testSourceUri to dynamicTest and to dynamicContainer for conveni…
KvanTTT Jun 20, 2022
8e6e1ed
Extract prepareGrammars method
KvanTTT Jun 20, 2022
2e8e3a8
Remove useless and outdated files
KvanTTT Jun 20, 2022
22cbf3c
Update pom.xml in runtime-testsuite
KvanTTT Jun 25, 2022
d287f1c
Use runtimePath instead of targetClassesPath for runtimes' sources
KvanTTT Jun 25, 2022
f235d33
Update doc
KvanTTT Jun 25, 2022
d40067d
Fix potential problems with concurrent access to hash maps
KvanTTT Jun 26, 2022
de255cc
Restore info about `mvn -Dtest=java.** test` in antlr-project-testing.md
KvanTTT Jun 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/antlr-project-testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ And the result of testing the entire subdirectory:

parrt marked this conversation as resolved.
Show resolved Hide resolved
All test are run in parallel both via maven and via IDE.

In IntelliJ, it's very easy to go to source by right-click on any test and pressing `Jump to source` (F4).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"right-clicking"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.


## Running test subsets

From the `runtime-testsuite` dir
Expand Down
39 changes: 31 additions & 8 deletions doc/creating-a-language-target.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,42 @@ This document describes how to make ANTLR generate parsers in a new language, *X

Creating a new target involves the following key elements:

1. For the tool, create class *X*Target as a subclass of class `Target` in package `org.antlr.v4.codegen.target`. This class describes language specific details about escape characters and strings and so on. There is very little to do here typically.
1. Create *X*.stg in directory tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg. This is a [StringTemplate](http://www.stringtemplate.org/) group file (`.stg`) that tells ANTLR how to express all of the parsing elements needed to generate code. You will see templates called `ParserFile`, `Parser`, `Lexer`, `CodeBlockForAlt`, `AltBlock`, etc... Each of these must be described how to build the indicated chunk of code. Your best bet is to find the closest existing target, copy that template file, and tweak to suit.
1. Create a runtime library to support the parsers generated by ANTLR. Under directory runtime/*X*, you are in complete control of the directory structure as dictated by common usage of that target language. For example, Java has: `runtime/Java/lib` and `runtime/Java/src` directories. Under `src`, you will find a directory structure for package `org.antlr.v4.runtime` and below.
1. Create a template file for runtime tests. All you have to do is provide a few templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code using these templates for each target and check the test results. It needs to know how to define various class fields, compare members and so on. You must create a *X*.test.stg file underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/resources/org/antlr/v4/test/runtime). Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
1. Create test files under [/runtime-testsuite/test/org/antlr/v4/test/runtime](https://github.com/antlr/antlr4/tree/master/runtime-testsuite/test/org/antlr/v4/test/runtime). They will load defined test cases in each test descriptor. Also add the `/runtime-testsuite/test/org/antlr/v4/test/runtime/X/BaseXTest.java` which defines how test cases will execute and output.
1. Create/edit shell scripts in [/.travis](https://github.com/antlr/antlr4/blob/master/.travis) and [/appveyor.yml](https://github.com/antlr/antlr4/blob/master/appveyor.yml) to run tests in CI pipelines.
1. For the tool, create class *X*Target as a subclass of class `Target` in package `org.antlr.v4.codegen.target`.
This class describes language specific details about escape characters and strings and so on.
There is very little to do here typically.
2. Create `*X*.stg` in directory `tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg`.
This is a [StringTemplate](http://www.stringtemplate.org/) group file (`.stg`) that tells ANTLR how to express
all the parsing elements needed to generate code.
You will see templates called `ParserFile`, `Parser`, `Lexer`, `CodeBlockForAlt`, `AltBlock`, etc...
Each of these must be described how to build the indicated chunk of code.
Your best bet is to find the closest existing target, copy that template file, and tweak to suit.
3. Create a runtime library to support the parsers generated by ANTLR.
Under directory `runtime/*X*`, you are in complete control of the directory structure as dictated by common usage of that target language.
For example, Java has: `runtime/Java/lib` and `runtime/Java/src` directories.
Under `src`, you will find a directory structure for package `org.antlr.v4.runtime` and below.
4. Create a template file for runtime tests.
All you have to do is provide a few templates that indicate how to print values and declare variables.
Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code using these templates for each target and check the test results.
It needs to know how to define various class fields, compare members and so on.
You must create a `*X*.test.stg` file underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime](../runtime-testsuite/resources/org/antlr/v4/test/runtime)
and `Test.*x*.stg` underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers](../runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers).
Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit.
6. Create test files under [/runtime-testsuite/test/org/antlr/v4/test/runtime](../runtime-testsuite/test/org/antlr/v4/test/runtime).
They will load defined test cases in each test descriptor.
Also add the `/runtime-testsuite/test/org/antlr/v4/test/runtime/X/BaseXTest.java` which defines how test cases will execute and output.
7. Create/edit shell scripts in [/.github](../.github) and [/.circleci](../.circleci) to run tests in CI pipelines.

## Getting started

1. Fork the `antlr/antlr4` repository at github to your own user so that you have repository `username/antlr4`.
2. Clone `username/antlr4`, the forked repository, to your local disk. Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/).
1. Fork the `antlr/antlr4` repository at GitHub to your own user so that you have repository `username/antlr4`.
2. Clone `username/antlr4`, the forked repository, to your local disk.
Your remote `origin` will be the forked repository on GitHub.
Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`).
Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/).
3. Try to build it before doing anything

```bash
$ mvn compile
```

That should proceed with success. See [Building ANTLR](building-antlr.md) for more details.