-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
move from ros3 to fsspec #379
base: dev
Are you sure you want to change the base?
Conversation
create a context manager read_nwb, which will probably migrate to PyNWB eventually
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Have you been able to run this branch successfully on your local device? I.e.,
as well as
(which should then run about ~4 times faster) The CI reports certain issues, and when I try this I also encounter a couple of problems when I try it on the Hub (especially the second one - any time I've ever tried parallelizing If this does run for you, then could you also report a speed comparison between this branch and current Also, is there any significant downside to keeping ROS3 as an option for the environments where it is already configured? Given that we haven't run into any more problems with it in the past ~6 months; |
I didn't anticipate these issues with parallelizing. I'll keep working on this and see if I can iron this out, with a speed comparison. In my experience fsspec is faster and more reliable than ros3, but I agree IRS worth a test. I'm not aware of any use case that would require ros3 instead of fsspec, so I'd rather drop support for it unless we can come up with a compelling reason to keep it. |
It might just require a refactor of the parallelization step to use See examples: https://gist.github.com/CodyCBakerPhD/b9d7d10ce1572dafcbd4646251781234 The |
@CodyCBakerPhD What do you think of this? I am adding back ros3 as an available "method" of |
Codecov Report
@@ Coverage Diff @@
## dev #379 +/- ##
==========================================
- Coverage 90.99% 90.37% -0.63%
==========================================
Files 20 20
Lines 1133 1132 -1
==========================================
- Hits 1031 1023 -8
- Misses 102 109 +7
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Sequential stream is working now, and is about 33% faster than ros3 The command
Still stalls out as per my demonstration in #379 (comment) I'd recommend either switching to use |
Update: as mentioned in meeting, Martin mentioned a possible fix for As I play around more with the context, as well as reflections on issue NeurodataWithoutBorders/pynwb#1711 and PR NeurodataWithoutBorders/nwbwidgets#295, I think we might want to make the introduction of the |
also created a context manager
read_nwb
, which will probably migrate to PyNWB eventuallyMotivation
fix #293
I'd like to build zarr support into the new read_nwb context manager, so I'd like to get this merged before looking at supporting the zarr backend