catkin_make run_tests returns success even when tests fail.

[hersh]

The result code returned by "catkin_make run_tests" and by "cd build; make run_tests" in a catkin workspace is 0 (success) even when tests have failed.

This makes it difficult to script automated testing, since Travis (for one) runs every line in its script until one fails, as determined by the result code.

In ros_comm's .travis.yml, I found this pile of stuff, which I believe is a workaround for the problem I'm describing:

  - cd build
  # Manually invoke make, catkin_make
  - make run_tests > log.run_tests || cat log.*
  # Check test results
  - 'python -c "from catkin.test_results import test_results;import sys;sys.exit(sum([v[1] + v[2] for k, v in test_results(\"test_results\").iteritems()]))" || cat log*'
  - 'python -c "from catkin.test_results import test_results;import sys;sys.exit(sum([v[1] + v[2] for k, v in test_results(\"test_results\").iteritems()]))"'

I notice that there exists a "test" target in the Makefile, and when I run "make test" it does indeed return a failure code when any of the tests fail. Yay! However, it says it ran 12 tests, and when I run "make run_tests" it says it ran 39 tests. So that makes me think that "make test" is not running all of my tests. Also the output formatting and result reporting is totally different between "make test" and "make run_tests", so I'm not sure what is going on there.

[wjwwood]

You have to run catkin_test_results. In order for run_tests to run all tests even if some fail part way through it cannot return a different return code. You can use the CTest target, test, also which will return the code you are looking for but will not generate correct test XML output.

[hersh]

OK, cakin_test_results does seem to do the right thing.

What is all the other junk for in that ros_comm .travis.yml file? Maybe it predates catkin_test_results?

[wjwwood]

That might have been from a point where catkin_test_result did not work recursively, but I'm not sure.

[hersh]

OK, I'm satisfied. I'll close this. It would be helpful if the catkin docs and/or the "USAGE" line for catkin_test_result described the result code behavior.

Thanks for your help!