[Feature] Decode with worker pools

[manzt]

It might be interesting to add an option to allow multiple workers to decoded chunks, since chunks are fetched asynchronously now and this could be done in parallel (maybe?).

We have been exploring geotiff to parse tiffs in browser, and it has a (seemingly) simple approach. See pool.js and decoder.worker.js. I like the API as well -- see an example of our use case .

Perhaps we could add this as an optional parameter to ZarrArray.get() or ZarrArray.getRaw(), or even when initializing the ZarrArray:

import { openArray, Pool } from "zarr";

const pool = new Pool();
const z = await openArray({store: "<some-url>", path: "data.zarr");

const arr = await z.get(null, { pool })

[gzuidhof]

Thank you for the suggestion, but I think without a use-case in mind to support and it actually being shown to be a problem this may be feature creep for now.

Sending messages to a worker and back may actually introduce more overhead than it gains, and in general I think it will make the code a bit more complicated (and maybe require another dependency).

The amount of data getting decompressed per chunk is usually not that much, and probably not a bottleneck for the majority of users. For real power users that need to squeeze out more performance, are probably best of using zarr within webworkers to fan out the work.

[manzt]

Thanks for the response. Our use case is to fetch and decode many image tiles stored as zarr chunked arrays. Fetching and decoding zlib tiff tiles with geotiff is quite a bit faster currently. However, I haven't tested the recent updates in zarr.js and I imagine that when more than gzip/zlib are available in the browser this might change. I generally agree that this is beyond the scope of zarr.js