Mark and skip slow tests

[JDBetteridge]

Description

This PR started as an experiment to "cheaply" speed up the test suite by calling mpiexec wrapping pytest, rather than forking a subprocess which calls mpiexec (which is also problematic for other reasons).

This PR now carries around multiple test suite fixes that should be merged back to master and includes fixes including:

• Adding comm arguments to function calls that need them.
• Freeing comms that are created.
• Disabling a test that pollutes the tape.
• "Fixing" ensemble parallel tests by using the simple partitioner (just in tests, Ensemble needs a proper fix!)
• This work had to be rebased on JDBetteridge/update caching #3730 and uses PyOP2 #724 and FInAT #134 due to the deadlocks that they call.

We need to consider what aspects of this experiment we want to incorporate back into master.

Some timings for the actual speed-up (the original intention):

Results

(Real only)

Master

This week's scheduled execution:

Total (inc install): 50m 45s

This branch

With fixed caches, mpispawn, fixed FInAT hashes and pytest-split based on a timed execution.
NB: We tweak vertexonly/test_poisson_inverse_conductivity.py to only do 3 iterations (see diff)

Serial: 17m51s
2: 2m59s
3: 6m43s
4: 45s
6: 19s
7: 48s
8: 12s
Total (inc install): 46m 6s

Important, this branch only runs a maximum of 12 ranks/threads!

[connorjward]

This is cool, but isn't it a bad idea to effectively remove test coverage? If CI doesn't run all the tests no one will.

I can see this being useful in the context of a bigger change where we run the test suite with a number of Firedrake configurations and only one of them would run these slow tests.