The goals of this project are:
-
to augment the googletest framework with listener(s) that yield sensible output for testing MPI-based, distributed-memory parallel software
-
to keep this software as low-maintenance as possible
-
ideally, to merge some version of this functionality into Google Test
This software is BSD-3 licensed, and uses some example code from Google Test, which has the following license that must be duplicated in its entirety, per its terms:
Copyright 2005, Google Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- GNU-compatible Make or gmake
- CMake (required for the build directions in this README; this is a soft requirement, and anyone sufficiently motivated and skilled could figure out how to build Google Test and this project with just GNU-compatible Make or gmake -- it would just take time)
- a C++98-standard-compliant C++ compiler
- all of the requirements for Google Test
- a standards-conforming MPI-1.x implementation (e.g., MPICH, Open MPI, MVAPICH, Intel MPI)
- In the root directory of this repository, clone the googletest GitHub repo:
git clone https://github.com/google/googletest.git
- Change to the new
googletest
directory created via cloning:
pushd googletest
- Within this directory, make a
build
directory; this step is standard for CMake-based buildsystems:
mkdir build
- Change to this
build
directory:
cd build
- Run CMake to generate a
Makefile
that will buildgoogletest
:
cmake ..
You should see some CMake output indicating compiler detection, etc.
- Assuming step 5 completed successfully, call
make
to build the googletest library:
make
- Return to the root directory of the repository:
popd
(if you followed this directions)
- Build the example test runner:
make
- Assuming the test runner build proceeds successfully, run the test runner example:
bin/example.exe
Please read the Google Test Primer and Google Test Advanced Guide for background before reading the remainder of this section.
Usage of this listener is illustrated by the example in
src/example.cpp
, which includes some very simple-minded MPI unit
tests that I've used for basic testing. To use this listener, you will
need to write your own int main(int argc, char** argv)
function --
you cannot use the stock gtest_main
supplied with Google Test. Using
a custom main
function enables us to:
- initialize MPI
- add an
Environment
object that will finalize MPI whenmain
terminates - remove the stock Google Test
TestEventListener
, which emits output on tests - add a
TestEventListener
that will emit sensible output (that is, rank-ordered output to stdout) for MPI-based unit tests
all of which is necessary, and not possible using the stock gtest_main
function.
Directions for writing tests can be found in Google Test Primer and Google Test Advanced Guide. My design assumption is that these tests will be executed in distributed fashion using the MPI distributed-memory parallel programming model (that is, a shared-nothing, process-local memory address space). This assumption implies that tests will be run by all MPI processes, or disabled on all processes. Conditionals can be used to execute parts (or all) of the body of a test on a given MPI process or set of processes.
After writing tests, your int main(int argc, char** argv)
function should look like the example test runner:
int main(int argc, char** argv) {
// Filter out Google Test arguments
::testing::InitGoogleTest(&argc, argv);
// Initialize MPI
MPI_Init(&argc, &argv);
// Add object that will finalize MPI on exit; Google Test owns this pointer
::testing::AddGlobalTestEnvironment(new MPIEnvironment);
// Get the event listener list.
::testing::TestEventListeners& listeners =
::testing::UnitTest::GetInstance()->listeners();
// Remove default listener
delete listeners.Release(listeners.default_result_printer());
// Adds MPI listener; Google Test owns this pointer
listeners.Append(new MPIMinimalistPrinter);
// Run tests, then clean up and exit
RUN_ALL_TESTS();
return 0;
}
Comments in this example describe what each line does. The ordering is also important, and should be preserved if you augment this example with additional code.
The most important design consideration was to write something portable, quickly. This meant using MPI-1.x calls (because supercomputers sometimes run older MPI implementations), and relying on the public API calls of a portable unit testing library that is currently being maintained. As a result of these decisions:
-
this software communicates unit test results to MPI process 0 and then writes them to standard out (because MPI I/O is defined in MPI-2)
-
consequently, this software will have an output bottleneck at extreme scale, because the output is, to borrow from Bill Gropp's MPI I/O slides (see slide 3), "worse than sequential"
-
this software doesn't have the awesome, fancy, color-coded output that stock Google Test uses, because all of those functions are implemented as
static
ingoogletest/googletest/src/gtest.cc
, and thus are inaccessible from outside that translation unit because they are not part of the Google Test public API -
this software is header-only, to make vendoring it into software packages no more painful than vendoring in Google Test
The current setup should be usable for small numbers of MPI processes
-- I've tested it on 256 MPI processes and it seems to work fine. If
there is a need to write infrastructure for testing on 100,000 MPI
processes, then an MPI I/O-based TestEventListener
makes more sense,
and would relax the output bottleneck. This new code should be written
in a second header to isolate MPI-2-conforming code from
MPI-1-conforming code. Although I haven't done it yet, I suspect this
modification could be done in days.
Adding fancy, color-coded output is also possible, but it means reimplementing private functions from Google Test that could change at any moment. I'd rather not reimplement that myself, because it will also mean that this software can no longer be header-only due to the way Google Test implements terminal color-coding.
If you want to hack on this code, the most important things to note are:
-
::testing::TestEventListener::OnTestPartResult
is only called after assertion failures or invocation of theSUCCESS()
macro, so it may not be called on some MPI processes, and there should be no communication in this function -
In
TestEventListener::OnTestEnd
, although::testing::TestInfo
has methods to query::testing::TestPartResult
objects within it, when I called those methods, the information I wanted (namely,::testing::TestPartResult
objects) was no longer present, which is why I instead store that information within a class member instead
As stated above, I aim to keep this software low-maintenance, because this package was written in a couple days in order to make the parallel software development for my work easier. I'm happy to accept bug-fixes, and I'm willing to discuss or consider feature requests/additions, but I'd like any feature to be easy to maintain. I do not get paid to write testing frameworks, which is why any responses to bug-fixes, feature requests, documentation improvements, or questions may be delayed; it is also why I want this software to be low-maintenance. In general, even though I intend to respond to correspondence about this software (I would like it to improve), users should assume I may not get back to them in a timely fashion, or at all.