Initial implementation of a PoseTraining Group #9

roomrys · 2022-07-27T17:24:58Z

Description

Add PoseTraining group to store annotations for training a pose estimator.

Related Issues

Notes from discussion regarding training input #8

…y as separate array

CBroz1

Minor suggestions. Added specificity regarding instances of skeletons. In all the cases I've seen, subject would work just as well, but instance works as more generic.

spec/ndx-pose.namespace.yaml

src/spec/create_extension_spec.py

spec/ndx-pose.extensions.yaml

src/spec/create_extension_spec.py

spec/ndx-pose.extensions.yaml

src/spec/create_extension_spec.py

rly · 2022-07-28T23:38:20Z

spec/ndx-pose.extensions.yaml

+    - name: path
+      dtype: text
+      doc: Path to original video file.
+      required: false


Should this and frame_index be optional? If a source_video is provided, it seems like a path and frame index should be provided.

The logic of requirement I think was to be applied to the entirety of the source_video attribute. Combined logic being the user either links source_video to an ImageSeries (internal or external storage) plus the frame index used and then they also specify the source_frame as a standalone Image (ideally internal or external, but see other comments regarding that). Hence source_frame is required, but the video it was extracted from is optional (but ideal) provenance.

So should the required logic apply to source_video directly then?

We wanted to provide two ways for supplying the training frame - either as a frame index given the path to the source video (source_video) or as a standalone image (source_frame). Hence why neither are required although at least one must be supplied.

I think we are confusing two different things here. I get that source_video is optional. If source_video is provided, then because the path and frame_index fields are optional, then the user does not have to provide either. From what you have both written, I think if source_video is provided, then path and frame_index should also be required, and we can specify that by removing required: false from both attributes.

rly · 2022-07-28T23:39:58Z

spec/ndx-pose.extensions.yaml

+  - name: source_frame
+    neurodata_type_inc: Image
+    doc: Image frame used for training (stored either internally or externally).
+    quantity: '1'


Note that currently the Image type supports only images stored internally. Also quantity: 1 is fine to include, but it is the default when no quantity is provided, so we tend to omit it.

From the past discussion, internally saving is the preferred approach since there could be thousands of external files otherwise.

However, maximal freedom for the user is also desired - would it be appropriate to extend the Image neurodata type in this extension to allow an external mode or should we alter that in the core instead?

I created an issue NeurodataWithoutBorders/nwb-schema#529 on the topic. This is part of a larger discussion over how we want to support storing external files in NWB.

rly · 2022-07-28T23:44:01Z

These changes look good to me, and I support @CBroz1 's changes. It would be good to get feedback from @bendichter and @AlexEMG on these changes before we proceed.

spec/ndx-pose.extensions.yaml

bendichter

@roomrys , could you look through these suggestions when you have time? This extension looks like it is really quite close to complete

spec/ndx-pose.extensions.yaml

CBroz1 · 2022-09-26T23:50:58Z

Hi @roomrys, Hi @rly - I'd like to get this merged to use with a DataJoint NWB export. Should I open a separate PR to reflect the changes we discuss here?

src/spec/create_extension_spec.py

roomrys · 2022-09-27T17:48:00Z

spec/ndx-pose.extensions.yaml

+  - name: source_frame
+    neurodata_type_inc: Image
+    doc: Image frame used for training (stored either internally or externally).


Hi @CBroz1, @rly,

I believe all the changes have been address (except for Image only allowing externally stored images and our current docs incorrectly say "stored either internally or externally"). With source_video linked to an ImageSeries, I believe we may not even need the source_frame group as we can directly supply a collection of images. However, the images supplied for training are rarely in a sequential format (and may therefore be an incorrect use of ImageSeries which extends TimeSeries).

The problem: We would like to store non-sequential images internally. ImageSeries allows images to be stored internally, but expects them to be sequential. Image allows images to be non-sequential, but requires them to be stored externally.

@CodyCBakerPhD had suggested making a change to the Image datatype to allow this. Is this feasible? Is there another workaround? Should we just hack ImageSeries for our purposes?

In NWB 2.5.0, we recently revamped the Images neurodata type, which represents a collection of images that may or may not be ordered. The IndexSeries neurodata type supports references to an Images type. This type seems like a better fit for the use case.

And to be clear, Image requires images to be stored internally and does not support external storage. External storage of individual images is not currently supported, but we can add that if we need to. External storage of movies is supported through ImageSeries.

Ok, great. I swapped the Image data type for the Images data type and added an images_index attribute to pull out a specific frame from the Images collection.

spec/ndx-pose.extensions.yaml

roomrys · 2022-10-11T18:15:17Z

Hi @rly,

I believe all is ready! Please let me know if there is anything else that needs work.

spec/ndx-pose.extensions.yaml

talmo · 2023-07-17T17:16:30Z

Hey friends, where do we stand with this PR these days @rly @bendichter?

rly · 2023-07-21T07:52:55Z

Hi @talmo . Sorry for the extreme delay on wrapping up this PR... I will finalize this during the developer hackathon next week.

talmo · 2023-07-26T01:19:42Z

Let us know if there's anything we can help with! This is a key piece of functionality for us to be able to convert our hand labeled data for archival :)

rly · 2023-07-29T12:56:10Z

@roomrys Could you please enable edits to this PR / branch from maintainers? There should be a checkbox on the right side. I would like to give you credit for this PR if possible.

roomrys · 2023-08-01T21:03:38Z

Sorry @rly, it seems that is impossible at the moment https://github.com/orgs/community/discussions/5634

talmo · 2023-08-01T21:09:45Z

@rly what if you send a PR with your changes to talmolab:main, we merge it there, and then it updates here?

Or we could follow the above thread's response:

HELP! Please fix this, or we shall move to svn 🥲

😂😂

talmo · 2023-08-01T21:11:59Z

Or maybe add @roomrys as a maintainer to the repo? Or @roomrys add @rly as a maintainer to our fork so the checkbox works?

rly · 2023-08-01T21:12:07Z

Sounds good. Will do.

@rly what if you send a PR with your changes to talmolab:main, we merge it there, and then it updates here?

rly · 2023-08-01T21:12:58Z

Or maybe add @roomrys as a maintainer to the repo? Or @roomrys add @rly as a maintainer to our fork so the checkbox works?

I'll add @roomrys to the repo in case that helps but I think adding me to the fork would work.

rly · 2023-08-01T21:17:26Z

Sounds good. Will do.

@rly what if you send a PR with your changes to talmolab:main, we merge it there, and then it updates here?

It turns out that GitHub does not allow me to fork a fork of my repo.

rly · 2023-08-01T21:17:46Z

I will just merge this and make changes in a new PR. Thanks for the contribution @roomrys , @CBroz1 , and @CodyCBakerPhD !

roomrys added 2 commits July 27, 2022 08:34

Initial implementation of a PoseTraining Group

783c256

List identities as attributes, skeletons as links, and node visibilit…

d63b945

…y as separate array

CBroz1 approved these changes Jul 27, 2022

View reviewed changes

rly reviewed Jul 28, 2022

View reviewed changes

CodyCBakerPhD mentioned this pull request Jul 29, 2022

[Feature]: Add SLEAP catalystneuro/neuroconv#48

Closed

2 tasks

CodyCBakerPhD reviewed Jul 29, 2022

View reviewed changes

spec/ndx-pose.extensions.yaml Outdated Show resolved Hide resolved

CBroz1 mentioned this pull request Aug 1, 2022

Add NWB export function datajoint/element-deeplabcut#46

Merged

CBroz1 mentioned this pull request Aug 15, 2022

OpenCV bug. Ensuring None for edges catalystneuro/DLC2NWB#1

Closed

talmo mentioned this pull request Aug 16, 2022

[Discussion] Adding support for writing to nwb file. talmolab/sleap-io#7

Closed

bendichter reviewed Aug 17, 2022

View reviewed changes

spec/ndx-pose.extensions.yaml Show resolved Hide resolved

bendichter reviewed Aug 17, 2022

View reviewed changes

spec/ndx-pose.extensions.yaml Outdated Show resolved Hide resolved

bendichter reviewed Aug 17, 2022

View reviewed changes

spec/ndx-pose.extensions.yaml Show resolved Hide resolved

Revise wording in docs and some type changes

2f4248a

CBroz1 reviewed Sep 27, 2022

View reviewed changes

src/spec/create_extension_spec.py Outdated Show resolved Hide resolved

roomrys added 2 commits September 27, 2022 10:06

More doc changes and some type/quantity changes

d763fa6

Merge branch 'main' of https://github.com/rly/ndx-pose

da8574b

roomrys commented Sep 27, 2022

View reviewed changes

rly mentioned this pull request Sep 28, 2022

Support storage of external images NeurodataWithoutBorders/nwb-schema#529

Open

rly reviewed Sep 28, 2022

View reviewed changes

spec/ndx-pose.extensions.yaml Outdated Show resolved Hide resolved

Remove path from ImageSeries, swap Image for Images

1f1df74

rly reviewed Nov 16, 2022

View reviewed changes

spec/ndx-pose.extensions.yaml Show resolved Hide resolved

talmo mentioned this pull request Jul 17, 2023

Define data model for representing points and trajectories neuroinformatics-unit/movement#12

Closed

AlexEMG approved these changes Jul 29, 2023

View reviewed changes

rly merged commit b5d91bb into rly:main Aug 1, 2023

rly mentioned this pull request Nov 22, 2023

Update PoseTraining types #21

Merged

7 tasks

talmo mentioned this pull request Aug 16, 2024

NWB export fails talmolab/sleap#1801

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial implementation of a PoseTraining Group #9

Initial implementation of a PoseTraining Group #9

roomrys commented Jul 27, 2022

CBroz1 left a comment

rly Jul 28, 2022

CodyCBakerPhD Jul 29, 2022

roomrys Sep 27, 2022 •

edited

Loading

rly Sep 28, 2022

rly Jul 28, 2022

CodyCBakerPhD Jul 29, 2022

rly Jan 24, 2023

rly commented Jul 28, 2022

bendichter left a comment

CBroz1 commented Sep 26, 2022 •

edited

Loading

roomrys Sep 27, 2022

rly Sep 28, 2022

roomrys Oct 11, 2022

roomrys commented Oct 11, 2022

talmo commented Jul 17, 2023

rly commented Jul 21, 2023

talmo commented Jul 26, 2023

rly commented Jul 29, 2023

roomrys commented Aug 1, 2023

talmo commented Aug 1, 2023

talmo commented Aug 1, 2023

rly commented Aug 1, 2023 •

edited

Loading

rly commented Aug 1, 2023

rly commented Aug 1, 2023

rly commented Aug 1, 2023 •

edited

Loading

Initial implementation of a PoseTraining Group #9

Initial implementation of a PoseTraining Group #9

Conversation

roomrys commented Jul 27, 2022

Description

Related Issues

CBroz1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roomrys Sep 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rly commented Jul 28, 2022

bendichter left a comment

Choose a reason for hiding this comment

CBroz1 commented Sep 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roomrys commented Oct 11, 2022

talmo commented Jul 17, 2023

rly commented Jul 21, 2023

talmo commented Jul 26, 2023

rly commented Jul 29, 2023

roomrys commented Aug 1, 2023

talmo commented Aug 1, 2023

talmo commented Aug 1, 2023

rly commented Aug 1, 2023 • edited Loading

rly commented Aug 1, 2023

rly commented Aug 1, 2023

rly commented Aug 1, 2023 • edited Loading

roomrys Sep 27, 2022 •

edited

Loading

CBroz1 commented Sep 26, 2022 •

edited

Loading

rly commented Aug 1, 2023 •

edited

Loading

rly commented Aug 1, 2023 •

edited

Loading