Issue with get_skeletons Function

[michaelfekadu]

Hi!

First of all, thank you for creating this library. It has been extremely useful, and the documentation is excellent.

I am encountering an issue with the get_skeletons function. According to the documentation, this function should work for proofread root IDs 630 and 783. To ensure the neuron IDs are up-to-date, I use the is_latest_root function. Here is the relevant code snippet:

print(fafbseg.__version__)
flywire.is_latest_root([720575940627895056, 720575940507344965], timestamp='mat_783')

This returns:

3.0.9
array([ True,  True])

Based on this, I assumed I could use the get_skeletons function for both IDs. However, while the function works for the first ID, it fails for the second one. Here is the code and error message:

flywire.get_skeletons(720575940507344965, dataset=783)

HTTPError                                 Traceback (most recent call last)
Cell In[67], [line 1](vscode-notebook-cell:?execution_count=67&line=1)
----> [1](vscode-notebook-cell:?execution_count=67&line=1) flywire.get_skeletons(720575940507344965, dataset=783)

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:133](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:133), in parse_neuroncriteria.<locals>.outer.<locals>.inner(*args, **kwargs)
    [131](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:131)         kwargs[key] = nc.get_roots()
    [132](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:132) try:
--> [133](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:133)     return func(*args, **kwargs)
    [134](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:134) except NoMatchesError as e:
    [135](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/annotations.py:135)     if allow_empty:

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:675](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:675), in get_skeletons(root_id, threads, omit_failures, max_threads, dataset, progress)
    [672](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:672) root_id = np.int64(root_id)
    [674](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:674) try:
--> [675](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:675)     tn = navis.read_precomputed(f'{SKELETON_BASE_URL[str(dataset)]}[/](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/){root_id}',
    [676](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:676)                                 datatype='skeleton',
    [677](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:677)                                 info=SKELETON_INFO)
    [678](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:678)     # Force integer (navis.read_precomputed will turn Id into string)
    [679](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/fafbseg/flywire/skeletonize.py:679)     tn.id = root_id

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/precomputed_io.py:344](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/precomputed_io.py:344), in read_precomputed(f, datatype, include_subdirs, fmt, info, limit, parallel, **kwargs)
    [341](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/precomputed_io.py:341) else:
    [342](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/precomputed_io.py:342)     reader = PrecomputedMeshReader(fmt=fmt, attrs=kwargs)
--> [344](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/precomputed_io.py:344) return reader.read_any(f, include_subdirs, parallel, limit=limit)

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:771](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:771), in BaseReader.read_any(self, obj, include_subdirs, parallel, limit, attrs)
    [769](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:769) except TypeError:
    [770](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:770)     pass
--> [771](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:771) return self.read_any_single(obj, attrs)

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:668](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:668), in BaseReader.read_any_single(self, obj, attrs)
    [666](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:666)         return self.read_file_path(p, attrs)
    [667](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:667)     if obj.startswith("http://") or obj.startswith("https://"):
--> [668](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:668)         return self.read_url(obj, attrs)
    [669](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:669)     return self.read_string(obj, attrs)
    [670](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:670) if isinstance(obj, bytes):

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:563](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:563), in BaseReader.read_url(self, url, attrs)
    [553](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:553) # Note: originally, we used stream=True and passed `r.raw` to the
    [554](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:554) # read_buffer function but that caused issue when there was more
    [555](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:555) # than one chunk which would require us to concatenate the chunks
   (...)
    [560](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:560) # may have raised an exception earlier if the file was corrupted or
    [561](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:561) # the wrong format.
    [562](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:562) with requests.get(url, stream=False) as r:
--> [563](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:563)     r.raise_for_status()
    [564](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:564)     props = self.parse_filename(url.split('[/](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/)')[-1])
    [565](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/navis/io/base.py:565)     props['origin'] = url

File [~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1021](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1021), in Response.raise_for_status(self)
   [1016](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1016)     http_error_msg = (
   [1017](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1017)         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   [1018](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1018)     )
   [1020](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1020) if http_error_msg:
-> [1021](https://vscode-remote+ssh-002dremote-002b10-002e243-002e83-002e35.vscode-resource.vscode-cdn.net/home/michaelshewarega/Desktop/test/~/miniconda3/envs/arkouda/lib/python3.11/site-packages/requests/models.py:1021)     raise HTTPError(http_error_msg, response=self)

HTTPError: 404 Client Error: Not Found for url: https://flyem.mrc-lmb.cam.ac.uk/flyconnectome/flywire_skeletons_783/720575940507344965

Btw I retrieved both IDs from the Codex Flywire website.

Does anyone know why this might be happening? Is it possible that the get_skeletons function is accessing an outdated dataset?

Thank you in advance for your help!

[schlegelp]

Hi!

flywire.get_skeletons fetches skeletons that we precomputed for neurons that were already proofread at materialization 783 or 630 neurons, i.e. neurons that are part of the two public releases.

The root Id 720575940507344965 is not proofread and at closer inspection appears to be just a tiny fragment:

>>> flywire.is_proofread([720575940627895056, 720575940507344965], materialization=783)
array([ True, False])

The other ID (720575940627895056) is proofread and should therefore return a skeleton:

>>> flywire.get_skeletons(720575940627895056, dataset=783)
type                navis.TreeNeuron
name                        skeleton
id                720575940627895056
n_nodes                         1006
n_connectors                    None
n_branches                       123
n_leafs                          135
cable_length             540541.1875
soma            [309, 311, 313, 314]
units                    1 nanometer
dtype: object

flywire.get_skeletons does not check if a skeleton should exist for given IDs but you can use the omit_failures parameter to make it so you get the skeletons that exist and ignore those that don't without throwing an exception. On my end, I can try to make the error message more informative.

On a side note: you can always skeletonize neurons on-the-fly (pun intended) yourself using either flywire.get_l2_skeletons (coarser) or flywire.skeletonize_neuron/flywire.skeletonize_neuron_parallel. We used the latter two plus some post processing to generate the precomputed 630/783 skeletons.