QoL improvement: Set best epoch in config after train

[sebpuetz]

Maybe for sticker2, if it's too invasive for sticker?

[danieldk]

I think changing people's configuration files is a bit invasive and could lead to race conditions (e.g. if the user has left open an editor). What we could do is adding an option that copies the configuration file on each epoch with an improvement, with the model file for that epoch. So, you'd have epoch-15.conf, epoch-23.conf, etc.

[sebpuetz]

Imo, one for the ultimately best epoch should be enough. But on the other hand, it'd be straight forward to write a script to do predictions for all epochs which showed improvements if there's a config for each of those. So not completely sure what I'd prefer.

It's just a minor thing but I felt like setting the epoch was the only hurdle to automating prediction (& evaluation).

[twuebi]

Tensorflow has this convenience function:

https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/train/latest_checkpoint

picking the epoch with the highest number should correspond to the best epoch when used in conjunction with only saving upon improvements. Maybe there could be a flag use latest?