[WIP]Enh/checkpointing #8
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mschmidt87
suggested changes
Jul 31, 2018
|
|
||
| if load_existing_checkpoint: | ||
| state = load_checkpoint() | ||
| if state: |
There was a problem hiding this comment.
If state is None here, i.e. if there were no checkpoints that can be loaded, I suggest to raise an error. Otherwise checkpoint loading fails silently.
| from . import lib | ||
|
|
||
|
|
||
| def load_checkpoint(): |
There was a problem hiding this comment.
What about enabling the user to specify a specific checkpoint, not only the most recent one? You could implement this by allowing load_existing_checkpoint to be specified as True or -1 (for the most recent one) or as an integer specifying the generation.
| pickle.dump(state, f) | ||
|
|
||
|
|
||
| def extract_keys_from_dict(d, whitelist, blacklist=[]): |
There was a problem hiding this comment.
It appears to me that this function should be located in lib.py.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds basic checkpointing by dumping all local variables that are changed during optimization to a pickle file and automatically loading the most recent checkpoint if any is found.
During the implementation, I found that it may be much easier to implement checkpointing if the optimization routine is a stateful object, so maybe your idea @mschmidt87 to turn all algorithms into classes should be done first and a new checkpointing mechanism added to that. Let me know what you think about this.
Branches of #4, so should not be merged before that.