-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
memory error #82
Comments
Hi @aCoalBall , Can you show some more of the code - specifically how you're handling the Also, are you processing many files, or some very large files? |
Hi HalfPhoton, I just create the pod5_reader by
And I am processing a single pod5 file with 141G. |
By the way, it seems not due to memory runs out. I tried to specify different size of memory but it just shut down at the same position (around 300000th reads) |
Interesting. @aCoalBall can you confirm it still crashes if you don't do your downstream processing? I wouldn't expect the code above to retain the signals in memory unless you hold them in your training data. So, this should work:
Can you confirm? Are you able to provide the complete code otherwise and we can investigate further.
|
@jorj1988 I tried what you suggest, the following code runs fine.
However, as long as I use pod5_read.signal, like
|
ok thanks. What is your environment like - what sort of OS, VMem, Physical Memory available etc? Thanks,
|
And can you confirm if you have a virtual memory limit on the system? eg:
|
@jorj1988 Yes, there is a virtual memory limit
|
Hi @aCoalBall , Would you be able to test the following code to ascertain if this is virtual memory issue? pod5_reader = pod5.Reader(pod5_path)
pod5_reader._signal_handle._reader = None
pod5_reader._signal_handle._reader = pod5_reader._signal_handle._open_without_mmap()
for pod5_read in pod5_reads:
# remaining code here... |
Hi @HalfPhoton , |
Hi @aCoalBall, I've been running the below on a system similar to yours and not seen a crash yet... can you confirm this does crash for you? My input file is 1.4TB, 64GB physical memory, with a virtual memory limit set to the same as yours (
I have taken out several bits of your script to put this example together... maybe we need to add some of them back to make it crash? Thanks,
|
Hi @jorj1988 , Actually, even the simplest iteration would cause the error
Im running this task on a HPC but I don't know how they actually designed it. Im trying to change my computation platform now. |
Hi there,
Im using pod5 python API to load signals in to another file. But I meet this error when I iterate through the pod5 reader.
I allocated 256GB memory for this task and I think that I just loaded signals for one read each time and did not save all signals in the pod5 file into the memory manually.
Here is the code:
The text was updated successfully, but these errors were encountered: