Clean up emitted files as the test harness runs

[StephanTLavavej]

Fixes #4082.

Overview

Currently, the test harness leaves behind the OBJ, EXE, etc. files that it builds. These are safely contained within the out subdirectory, and they're easily dealt with via git clean. However, this means that large test runs leave behind a lot of emitted files. After ASAN configurations were added to the matrix, I observed that a full test pass left behind ~200 GB of files on my local machine, causing my 1 TB SSD to run out of space. 🙀

These emitted files are never useful to us. If we have to investigate a particular test, we always want to rebuild it by hand, and extracting the relevant command line from the logs is easy (for all except the header units test, and the emitted artifacts aren't any more helpful there). This is because the vast majority of our plain test configurations don't use /Zi (so if we want debug info we have to add it), and because we typically want to do something like hack a header and rebuild with an overlay /I option while investigating a potential fix.

How I developed this PR

I knew that our test harness created different subdirectories for different test configurations. While trying to find where this happened, I noticed that format.py was organized into comprehensible "steps", and it starts with a "build setup" step that deletes and recreates the test subdirectory:

STL/tests/utils/stl/test/format.py

Lines 161 to 163 in f362f7d

	def getBuildSetupSteps(self, test, litConfig, shared):
	shutil.rmtree(shared.execDir, ignore_errors=True)
	Path(shared.execDir).mkdir(parents=True, exist_ok=True)

Immediately above, getStages() was designed in a really nice way to return an array of stages. This was extensible, instead of having a specific sequence of stages hardcoded everywhere. So all I had to do was introduce a "Clean" stage at the end, performing the same shutil.rmtree() command that's run at the very beginning.

The one thing to pay attention to is the bool at the end of each listed stage. This is isBuildStep, which is used to handle build-only configurations:

STL/tests/utils/stl/test/format.py

Lines 220 to 222 in f362f7d

	for stageName, steps, isBuildStep in stages:
	if not isBuildStep and litConfig.build_only:
	continue

When isBuildStep is True, the step is always run (which is what we want for this new Clean step). False steps are for running tests, which are skipped for build-only configurations.

How I tested this PR

After building the STL, I captured the out\x64 directory contents before and after running a pair of tests (compile-only and runtime). (%STL% is my repo root.)

pushd out\x64

dir /s /b /a-d > C:\temp\beforetests.txt

python tests\utils\stl-lit\stl-lit.py -o testing_x64.log %STL%\tests\std\tests\P0295R0_gcd_lcm %STL%\tests\std\tests\P0768R1_spaceship_operator

dir /s /b /a-d > C:\temp\aftertests.txt

For main, the before-and-after difference displayed a bunch of emitted files. In addition to the testing_x64.log that I explicitly requested, LLVM lit generated one file, and then each subdirectory contained a bunch. Trimmed my repo root for clarity:

out\x64\testing_x64.log

out\x64\tests\std\.lit_test_times.txt

out\x64\tests\std\tests\P0295R0_gcd_lcm\Output\00\test.compile.pass.obj
out\x64\tests\std\tests\P0295R0_gcd_lcm\Output\01\test.compile.pass.obj
out\x64\tests\std\tests\P0295R0_gcd_lcm\Output\01\vc140.pdb
out\x64\tests\std\tests\P0295R0_gcd_lcm\Output\02\test.compile.pass.obj
out\x64\tests\std\tests\P0295R0_gcd_lcm\Output\03\test.compile.pass.obj
out\x64\tests\std\tests\P0295R0_gcd_lcm\Output\03\vc140.pdb
[...]
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\00\_CL_2577c3f4lk
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\00\P0768R1_spaceship_operator.exe
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\00\test.obj
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\01\_CL_7a4d587blk
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\01\P0768R1_spaceship_operator.exe
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\01\P0768R1_spaceship_operator.pdb
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\01\test.obj
out\x64\tests\std\tests\P0768R1_spaceship_operator\Output\01\vc140.pdb
[...]

With this PR, only the explicitly requested log file and the LLVM lit generated file survive:

out\x64\testing_x64.log

out\x64\tests\std\.lit_test_times.txt

[StephanTLavavej]

I'm mirroring this to the MSVC-internal repo - please notify me if any further changes are pushed.