Simplify test rig related to timeouts

[parrt]

The test rig has a bit of complexity related to making sure that our continuous integration servers don't timeout. I'm trying an experiment where I turn off the quiet option so that we are always getting output. I think as long as all TestX.java finish within 10 minutes we should be okay because he will generates an output. Also, I have turned on -Dparallel=classes -DthreadCount=4 so that 4 TestX will run the same time. This seems a decent level of granularity rather than creating a new thread for possibly 100 tests within a TestX.java file. If this works then I can try taking out the heartbeat mechanism which spits out . every once in a while to let the integration server know that we are still running.

[parrt]

Wow. C++ test on my local box is much faster with 4 test classes running at once. Single threaded is:

[INFO] Total time:  26:52 min

But with four classes running at once we get:

[INFO] Total time:  07:16 min

[KvanTTT]

I'm trying an experiment where I turn off the quiet option so that we are always getting output.

Maybe it makes sense to print the current test name after its completion? It's more clear than just a dot.

[KvanTTT]

This seems a decent level of granularity rather than creating a new thread for possibly 100 tests within a TestX.java file.

Thread is very cheap compare to the process that is created in every test (or several processes). I still don't understand why maven creates a lot of threads that halt your computer (you said before) with the defined option -DthreadCount=4.

[parrt]

Interesting. parallelism works for C++ on my mac and circleci but fails on github. dotnet fails on circleci saying that multiple threads are working on the same artifact; is it rebuilding antlr runtime each test? waiting on results from appveyor and github.

btw, there are 1 billion of these messages for C++:

ld: warning: dylib (./libantlr4-runtime.dylib) was built for newer macOS version (11.2) than being linked (11.0)

error messages on github also indicated can't find the executable which probably explains the failed tests:

java.io.IOException: Cannot run program "/var/folders/md/0_41qtl50zd6zybdswqfjk0r0000gr/T/BaseCppTest-pool-1-thread-5-1640807139015/a.out"

[parrt]

@KvanTTT I can't reproduce the threading issue but it was only with Swift anyway. I also saw that it did something very strange with threading=methods: it didn't actually launch multiple threads where as it does with classes.

getting different behavior on different integration servers is really driving me nuts. I'm happy to throw away this PR / approach, but I would like to increase the speed and simplicity of the testing.

[parrt]

instead of using something from the command line, perhaps it's better to update the poms like @jcking suggests:

<parallel>all</parallel>
<useUnlimitedThreads>true</useUnlimitedThreads>
<forkCount>0</forkCount>

Ah! it also looks like he has discovered that dotnet doesn't deal well with parallelism and has started on a solution as well. See that PR.

Maybe just dotnet and C++ are the issue, in which case, I'm happy to keep them single threaded until we can come up with a solution.

[KvanTTT]

is it rebuilding antlr runtime each test?

No, it's being built a single time during the test session. But C++ is very slow itself and even one test building is slow.

btw, there are 1 billion of these messages for C++:

It's too many, maybe it makes sense to disable all or almost all warnings.

instead of using something from the command line, perhaps it's better to update the poms like @jcking suggests:

It sounds good. I hope it also will reduce duplicated maven commands in config files.

Maybe just dotnet and C++ are the issue, in which case, I'm happy to keep them single threaded until we can come up with a solution.

C# does not work with the following error:

The process cannot access the file '/tmp/ANTLR-runtime-testsuite-cache/CSharp/Antlr4.Runtime.Standard.deps.json' because it is being used by another process. [/home/circleci/project/runtime-testsuite/target/classes/CSharp/src/Antlr4.csproj]

I came up with two solutions to how it could be resolved. The good one is to isolate processes, the bad one is to retry failed test while the file is locked. Interesting, I'll take a look later. But I definitely want to make parallel C#.

I'm happy to throw away this PR / approach, but I would like to increase the speed and simplicity of the testing.

A month ago it was much worse with speed :)

[parrt]

@KvanTTT I'm working on C#...might have a solution for parallel.

[parrt]

Wow. clearly creating 40 processes to run with parallel classes. wth?! I'll try methods next then modifying pom ala @jcking

[parrt]

i really don't understand mave. Check this out

mvn -Dparallel=methods -DuseUnlimitedThreads -Dtest=csharp.** test

So just 4 processes (with many threads?) but just 1 process running? i could have messed up the synch for building runtime but...

Both this:

mvn -DuseUnlimitedThreads -Dtest=csharp.** test

and this:

mvn -Dtest=csharp.** test

gives similar process list of 4 or 5 processes. weird.

This creates 70 processes all using CPU:

mvn -Dparallel=all -DuseUnlimitedThreads -Dtest=csharp.** test

Confirming that this ultimately brings my M1 mac to its knees. Could be virtual memory thrashing?

[parrt]

Wow. Something is really really broken with maven or surefire or whatever. The thread / parallel stuff makes zero sense in its behavior. Wasted 4 hours so far. Might be process monitor isn't fine-grained enough to catch all the threads.

[parrt]

Ah. I got it. The CI uses a matrix (launched in parallel I guess) that means multiple mvn tests which means multiple runtime lib builds at once in race condition:

- test_runtime:
    matrix:
      parameters:
        target: [ cpp, dotnet ]
        test-group: [ LEXER, PARSER1, PARSER2, RECURSION ]

[parrt]

Holy crap! @KvanTTT and @ericvergnaud all tests pass on appveyor, circle, github! this is after I removed the fancy stuff with test groups like parser1, parser2. All are parallel except swift. Test scripts are reduced to this kinda thing:

pushd runtime-testsuite
mvn -Dparallel=classes -DthreadCount=4 -Dtest=cpp.** test
popd

30minutes total on github, 1 hour on appveyor but just 12 minutes on circleci :)

[ericvergnaud]

Good stuff. Appveyor only provides 1 concurrent build for free, we have 4 agents on the Github Mac, whereas CircleCI lets us have multiple agents with 2 threads each, so it all makes sense.

[KvanTTT]

30minutes total on github

It's hard to say for sure if the improvement exists because GitHub tests are very floating. For instance, here we have around 36 minutes, not far from your result (actually almost 32 minutes): https://github.com/antlr/antlr4/actions/runs/1627721550 Do you have local tests performance improvement for Swift and C++?

1 hour on appveyor

No improvement on AppVeyor, it was around 1 hour (before Dart AOT changes): https://ci.appveyor.com/project/parrt/antlr4/builds/41984813

12 minutes on circleci

It looks like a real improvement. It was around 20 minutes before on average. But I don't know what exactly affected performance: merging groups or parallelization.

[parrt]

My main concern here was to clean up the test rig and to get consistency not an improvement in speed.

@KvanTTT are you okay with me squashing these commits and merging the PR?

[KvanTTT]

Generally, I'm okay.

[parrt]

ok, I think it's a good idea to use this as a steppingstone for the next round of cleanups. Wow. appveyor is like 59$/month to get more than one concurrent job.

[KvanTTT]

ok, I think it's a good idea to use this as a steppingstone for the next round of cleanups. Wow. appveyor is like 59$/month to get more than one concurrent job.

Maybe it makes sense to ask them for free support because ANTLR is not a commercial but a valuable project. Also, I can try to ask JetBrains but probably TeamCity is not very suitable. Also, you can try to ask your "new galactic overlords" :)

[parrt]

hahah. if we rent a google cloud linux box, would this solve/help anything?

[KvanTTT]

hahah. if we rent a google cloud linux box, would this solve/help anything?

They don't have anything for Windows? :)

[parrt]

Just to summarize a bit of what I learned. We originally created the different groups of tests like parser1 and parser2 so that continuous integration servers would not timeout I have left in the heartbeat that spits out .s just in case that is still necessary. But it seems that as long as we get some output (not use quiet -q option on maven) at least every 10 minutes then I think we are good to go.

I also learned that this separate groups thing created a problem for parallel testing because group was executing a different maven instance which could not synchronize across VMs to prevent more than one runtime build. This caused C# builds to complain when they see the same file being overwritten by another process.

I have turned on parallelism for all but the Swift target which I couldn't figure out:

mvn -Dparallel=classes -DthreadCount=4 -Dtest=cpp.** test

I also do it at the class level not the method level because that did not seem to actually make things go faster. I also had difficulty figuring out how many threads were running just by looking at the process list in activity monitor. In the end, doing it class by class seemed to make things work.

I ran into a problem where 90 processes were created with -Dparallel=classes -DthreadCount=4 on my M1 mac, or at least I think that was a combination. It really dragged the system down. Something to keep in mind that there might be some issue here.

I added <forkCount>0</forkCount> to the maven pom file in the runtime tests and it somehow made the code not compile. Very strange so I took it back out; note that it works from the command line with -D. ugh.

all of the CI are much simpler now because we don't have to test for the group and the code is much simpler because we don't have the annotations related to groups. parallelism makes at least circleCI complete much faster.

I left the travis stuff alone as we are not currently using it.

[parrt]

They don't have anything for Windows? :)

Azure is very expensive for anything more than a tiny 1 core box. probably wouldn't even run our stuff.

[parrt]

Ok, tests all pass again... here we go!!!

[ericvergnaud]

They don't have anything for Windows? :)

Azure is very expensive for anything more than a tiny 1 core box. probably wouldn't even run our stuff.

We could follow the same strategy we have for MacOS.

[parrt]

That's what I was thinking. I would bet a $600 box even would work. I bought a laptop for $750.

[ericvergnaud]

Yes, grab an Intel NUC, then you only spend on what this CI needs.

[parrt]

So it'd connect to github actions like we do for mac os x now? That would simplify our lives i think.

[ericvergnaud]

Yes

…

Le 31 déc. 2021 à 00:53, Terence Parr ***@***.***> a écrit : So it'd connect to github actions like we do for mac os x now? That would simplify our lives i think. — Reply to this email directly, view it on GitHub <#3445 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNQJD5IYT7UQZKD45OIJLUTTWGXANCNFSM5K6PEBDA>. You are receiving this because you were mentioned.