New checkpointer features: set! and simulation "pickup"

[glwagner]

This PR adds two new features for interacting with Checkpointers.

The first is the method

set!(model, checkpointer_filepath::AbstractString)

that sets all the checkpointed data in checkpointer_filepath to model, including tendencies, and synchronizes the model clock and iteration with the checkpointed clock and iteration.

The second is a kwarg in run! meant to be used as either

run!(simulation, pickup=true)

or

run!(simulation, pickup=n)

or

run!(simulation, pickup=filepath)

where n is an integer that refers to an iteration number of a checkpointer file to be "picked up" and run from, and filepath is a string that indicates the path to checkpoint data. pickup=true looks for the latest checkpointed file and picks up the simulation from there.

To do:

• actually add run!(simulation, pickup=true)
• tests for set!
• tests for run!
• modify JLD2OutputWriter so it doesn't fail in common use cases

Resolves #1068
Resolves #602

[codecov]

Codecov Report

Merging #1082 into master will increase coverage by 0.36%.
The diff coverage is 90.00%.

@@            Coverage Diff             @@
##           master    #1082      +/-   ##
==========================================
+ Coverage   56.79%   57.16%   +0.36%     
==========================================
  Files         161      161              
  Lines        3796     3838      +42     
==========================================
+ Hits         2156     2194      +38     
- Misses       1640     1644       +4     

Impacted Files	Coverage Δ
src/OutputWriters/jld2_output_writer.jl	83.01% <80.00%> (-1.30%)	⬇️
src/Simulations/run.jl	88.15% <86.36%> (+0.22%)	⬆️
src/OutputWriters/checkpointer.jl	87.77% <95.65%> (+2.48%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1026b0...d018eae. Read the comment docs.

[ali-ramadhan]

Super neat! Should make checkpointing and pickup so much smoother and less prone to expensive mistakes.

[ali-ramadhan]

💯