Make it easier to run CI tests locally

[joshlf]

We currently run a large suite of tests in CI, which includes testing with different tools across different toolchains and different targets. It also involves miscellaneous things like confirming that README.md is up to date, that code is properly formatted with rustfmt, etc. In other words, it's not as simple as just running cargo test.

As a result, it's easy to make changes that appear fine locally but break CI, requiring an edit-commit-push cycle just to see if all of the CI tests will pass. We should make it easier to run most or all of these tests locally. For especially fast tests (like README.md and code formatting), we could even provide them as commit hooks.

@djkoloski also reports finding this to be painful (#69 (comment)):

I struggled specifically with formatting and forgetting to make stderr match in tests. Adding a way to format test files and an easier way to update all the stderr files for tests might be helpful. 🙂

The challenge here is going to be making these tests available without either a) moving everything out of our ci.yml and into a script (which means we don't get nice job output and error messages) or, b) duplicating everything (which is brittle).

[AntoniosBarotsis]

You could also look into something like Toast.

[memark]

Some interesting tools in this space are

• act
• earthly.dev

[AntoniosBarotsis]

One thing to keep in mind is that Act and I think Toast as well don't support Windows images (or at least didn't last time I tried IIRC).

[djkoloski]

I struggled specifically with formatting and forgetting to make stderr match in tests. Adding a way to format test files and an easier way to update all the stderr files for tests might be helpful. 🙂

[joshlf]

Awesome, added that to the issue description. This'll definitely be important to solve moving forward.

[joshlf]

@djkoloski I took a stab at pulling the scripts out of ci.yml and into a shell script here. There's still work to be done, but I probably won't get it to a state where I can merge for a few days, so I figured I'd send you that in case you want to just keep a copy of it locally until I get it merged.

[joshlf]

@djkoloski do you feel like #728 and #1171 combined are sufficient to close this issue?

[djkoloski]

Yes, this issue can be closed.