-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generalize file splitting for output writers #3515
Conversation
…iterion to file size may be used
We should also add a test similar to the one that was added in #3512. |
Co-authored-by: Navid C. Constantinou <[email protected]>
Doesn't that already exist? We just have to adapt the test |
Also open to comments about the new names / kwargs. |
Yes, the test I think only tests the |
There is already a test for the |
I will work on the time splitting schedule. Should we do it in this PR? or should I do another one once it is ready? |
@@ -258,18 +261,17 @@ end | |||
function write_output!(writer::JLD2OutputWriter, model) | |||
|
|||
verbose = writer.verbose | |||
path = writer.filepath |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why this change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because path
was used later on in the code, but it wasn't updated. Thus the code crashed while creating the file and when the writer.filepath
changed but not the path
. The easier fix was to replace all instances of path
by the writer.filepath
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, that's a good reason!
Co-authored-by: Gregory L. Wagner <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm!
Why are some of the tests failing? |
Sometimes (often) they just fail to initialize. Don’t assume you broke them necessarily. Uou need to see the log to understand if actually the tests broke or never started. |
@glwagner would you like to have a look at this or should we merge when tests pass? |
Some of the doctests are failing because of the changes: https://buildkite.com/clima/oceananigans/builds/15164#018e6aa7-67f8-4280-b331-ad7179892dd1/32-18831 |
I think it looks pretty good! Thanks @josuemtzmo for the work on In a future PR, we can add a test to ensure that Also I'm trying to brainstorm a better word to use than "actuated" for talking about scheduling. But I think once the tests pass we should merge this. |
Co-authored-by: Gregory L. Wagner <[email protected]>
Co-authored-by: Gregory L. Wagner <[email protected]>
…ananigans.jl into glw/generalized-file-splitting
This PR generalizes file splitting for
JLD2OutputWriter
, replacingmax_filesize
with afile_splitting
keyword argument. To implement a maximum filesize we useThis feature should also be extended to
NetCDFOutputWriter
in this PR -- hopefully @josuemtzmo can help!This PR will enable
TimeInterval
(or any other schedule ---IterationInterval
,WallTimeInterval
) to be used for splitting output files.Closes #2967