-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add test cases for LINDI #47
Comments
Similar to #43 we should also add lindi to the test suite |
We currently have a benchmark to create a LINDI file from a remote NWB file. In my experience, for some large files with many chunks (such as the ophys file which has ~1 million chunks), that is slower than downloading the entire NWB file and creating a LINDI file with the downloaded local NWB file using references to the remote asset. It probably has to do with the many requests to the remote NWB file, one per chunk(?). So I think we should add such a benchmark where we download the file locally and create the LINDI file from that downloaded file. nwb_benchmarks/src/nwb_benchmarks/benchmarks/network_tracking_remote_file_reading.py Lines 218 to 223 in 66b8d92
|
Separately, I think it would be good to have a benchmark that is simply download the entire file, as a point of comparison for some of these streaming methods. For some of the c elegans files with thousands of groups, it is actually faster to download and read than stream with fsspec/ros3, but it also depends on the read pattern. |
No description provided.
The text was updated successfully, but these errors were encountered: