Benchmarking the STL

Running Benchmarks

In order to run benchmarks, you'll need to build the STL with benchmarking support:

cmake -B out\bench -S . -G Ninja -DSTL_BUILD_BENCHMARKING=ON
cmake --build out\bench

once you've done this, you can run a specific benchmark with:

out\bench\benchmarks\benchmark-<benchmark_name>.exe --benchmark_out=<file> --benchmark_out_format=csv

This will give you pretty output to the command prompt, and additionally output a CSV file that you can open in a spreadsheet program like Excel.

Writing a New Benchmark

The STL uses Google Benchmark to benchmark, and the documentation there will be very helpful. Additionally, take a look at the existing benchmarks to see how the STL uses it.

In order to add a new benchmark, see if it would make sense in an existing benchmark file; if not, create a new file. Follow the existing naming convention.

The steps to creating a benchmark are as follows:

1. Create your benchmark function. This should take a benchmark::State&, and return void; it can be a template or not.
   • The basic structure of the function should be something like:

     void f(benchmark::State& state) {
         // setup
         // use `state.range(0)` for the length of arrays, `state.range(1)` for the rows of matrices, etc.
         for ([[maybe_unused]] auto _ : state) {
             benchmark::DoNotOptimize(/* input */);
             // benchmark stuff
             benchmark::DoNotOptimize(/* output */);
         }
     }

   • make certain that you use benchmark::DoNotOptimize(...) on any data that you don't want the compiler to be able to see, so that it doesn't optimize away anything.
2. Call your benchmark function with different parameters using the BENCHMARK and BENCHMARK_TEMPLATE<N> macros:
   • If your function doesn't take a template parameter, just use:

     BENCHMARK(function)->Range(0, 1 << N); // N should be replaced with something like 15 or 18, try to keep it not taking too long

   • For templates, you can use BENCHMARK_TEMPLATE1(ftemp, <type>), or BENCHMARK_TEMPLATE2(ftemp, <type>, <type>), etc.
3. Add your new file (if you created a new file) to the build:
   • At the bottom of benchmarks/CMakeLists.txt, add a new call to add_benchmark:

     add_benchmark(<benchmark-name>
         <source_files>
         CXX_STANDARD <standard> # if you want a non-latest standard version, for some reason
     )

And finally, run your new benchmark by following the instructions above!