Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable struct use instead of file objects #13

Open
pausz opened this issue Jun 19, 2020 · 0 comments
Open

Enable struct use instead of file objects #13

pausz opened this issue Jun 19, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@pausz
Copy link
Member

pausz commented Jun 19, 2020

neural-flows uses a type of matfile that allows for partial i/o operations. With current SSD w/r speeds it shouldn't be a problem. This choice was made to keep memory usage to minimum. The disadvantage is that this feature may bring an HPC down if the storage directory is on a network drive rather than scratch space.

Describe the solution you'd like
For small datasets, use a struct() with equivalent hierarchy and organisation as the current file_object. Keep everything in memory and write at the end.

Describe alternatives you've considered
Memory mapped files, but they don't work very well with parallelised loops.
Pick any two: minimise bookkeping complexity, enable parallel computations, minimise memory usage.

@pausz pausz added the enhancement New feature or request label Jun 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant