Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial implementation of a PoseTraining Group #9

Merged
merged 6 commits into from
Aug 1, 2023
Merged

Initial implementation of a PoseTraining Group #9

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
783c256
Initial implementation of a PoseTraining Group
roomrys Jul 27, 2022
d63b945
List identities as attributes, skeletons as links, and node visibilit…
roomrys Jul 27, 2022
2f4248a
Revise wording in docs and some type changes
roomrys Sep 27, 2022
d763fa6
More doc changes and some type/quantity changes
roomrys Sep 27, 2022
da8574b
Merge branch 'main' of https://github.com/rly/ndx-pose
roomrys Sep 27, 2022
1f1df74
Remove path from ImageSeries, swap Image for Images
Oct 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 103 additions & 15 deletions spec/ndx-pose.extensions.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,33 @@
groups:
- neurodata_type_def: Skeleton
neurodata_type_inc: NWBDataInterface
default_name: Skeleton
doc: Group that holds node and edge data for defining parts of a pose and their
connections to one another.
attributes:
- name: id
dtype: uint8
roomrys marked this conversation as resolved.
Show resolved Hide resolved
doc: Unique ID associated with the skeleton.
datasets:
- name: nodes
dtype: text
dims:
- num_body_parts
shape:
- null
doc: Array of body part names corresponding to the names of the PoseEstimationSeries
objects or PoseTraining objects.
- name: edges
dtype: uint8
dims:
- num_edges
- nodes_index, nodes_index
shape:
- null
- 2
doc: Array of pairs of indices corresponding to edges between nodes. Index values
correspond to row indices of the 'nodes' dataset. Index values use 0-indexing.
quantity: '?'
- neurodata_type_def: PoseEstimationSeries
neurodata_type_inc: SpatialSeries
doc: Estimated position (x, y) or (x, y, z) of a body part over time.
Expand Down Expand Up @@ -91,27 +120,86 @@ groups:
dtype: text
doc: Version string of the software tool used.
required: false
- name: nodes
groups:
- neurodata_type_inc: PoseEstimationSeries
doc: Estimated position data for each body part.
quantity: '*'
links:
- target_type: Skeleton
doc: Layout of body part locations and connections.
- neurodata_type_def: TrainingFrame
neurodata_type_inc: NWBDataInterface
default_name: TrainingFrame
doc: Group that holds ground-truth position data for all instances in a single frame.
roomrys marked this conversation as resolved.
Show resolved Hide resolved
attributes:
- name: annotator
dtype: text
dims:
- num_body_parts
shape:
- null
doc: Array of body part names corresponding to the names of the PoseEstimationSeries
objects within this group.
doc: Name of annotator who labeled the TrainingFrame.
required: false
groups:
- neurodata_type_inc: Instance
doc: Position data for a single instance in a single training frame.
roomrys marked this conversation as resolved.
Show resolved Hide resolved
quantity: '*'
- name: source_video
roomrys marked this conversation as resolved.
Show resolved Hide resolved
doc: Path to original video file and frame used.
roomrys marked this conversation as resolved.
Show resolved Hide resolved
quantity: '?'
- name: edges
attributes:
- name: path
roomrys marked this conversation as resolved.
Show resolved Hide resolved
dtype: text
doc: Path to original video file.
required: false
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this and frame_index be optional? If a source_video is provided, it seems like a path and frame index should be provided.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic of requirement I think was to be applied to the entirety of the source_video attribute. Combined logic being the user either links source_video to an ImageSeries (internal or external storage) plus the frame index used and then they also specify the source_frame as a standalone Image (ideally internal or external, but see other comments regarding that). Hence source_frame is required, but the video it was extracted from is optional (but ideal) provenance.

So should the required logic apply to source_video directly then?

Copy link
Collaborator Author

@roomrys roomrys Sep 27, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We wanted to provide two ways for supplying the training frame - either as a frame index given the path to the source video (source_video) or as a standalone image (source_frame). Hence why neither are required although at least one must be supplied.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are confusing two different things here. I get that source_video is optional. If source_video is provided, then because the path and frame_index fields are optional, then the user does not have to provide either. From what you have both written, I think if source_video is provided, then path and frame_index should also be required, and we can specify that by removing required: false from both attributes.

- name: frame_index
dtype: uint8
doc: Frame index of TrainingFrame in original video file.
required: false
- name: source_frame
neurodata_type_inc: Image
doc: Image frame used for training (stored either internally or externally).
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @CBroz1, @rly,

I believe all the changes have been address (except for Image only allowing externally stored images and our current docs incorrectly say "stored either internally or externally"). With source_video linked to an ImageSeries, I believe we may not even need the source_frame group as we can directly supply a collection of images. However, the images supplied for training are rarely in a sequential format (and may therefore be an incorrect use of ImageSeries which extends TimeSeries).

The problem: We would like to store non-sequential images internally. ImageSeries allows images to be stored internally, but expects them to be sequential. Image allows images to be non-sequential, but requires them to be stored externally.

@CodyCBakerPhD had suggested making a change to the Image datatype to allow this. Is this feasible? Is there another workaround? Should we just hack ImageSeries for our purposes?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In NWB 2.5.0, we recently revamped the Images neurodata type, which represents a collection of images that may or may not be ordered. The IndexSeries neurodata type supports references to an Images type. This type seems like a better fit for the use case.

And to be clear, Image requires images to be stored internally and does not support external storage. External storage of individual images is not currently supported, but we can add that if we need to. External storage of movies is supported through ImageSeries.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, great. I swapped the Image data type for the Images data type and added an images_index attribute to pull out a specific frame from the Images collection.

quantity: '1'
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that currently the Image type supports only images stored internally. Also quantity: 1 is fine to include, but it is the default when no quantity is provided, so we tend to omit it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the past discussion, internally saving is the preferred approach since there could be thousands of external files otherwise.

However, maximal freedom for the user is also desired - would it be appropriate to extend the Image neurodata type in this extension to allow an external mode or should we alter that in the core instead?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created an issue NeurodataWithoutBorders/nwb-schema#529 on the topic. This is part of a larger discussion over how we want to support storing external files in NWB.

roomrys marked this conversation as resolved.
Show resolved Hide resolved
- neurodata_type_def: Instance
neurodata_type_inc: NWBDataInterface
default_name: Instance
doc: Group that holds ground-truth pose data for single subject in a single frame.
attributes:
- name: id
dtype: uint8
doc: ID used to differentiate instances.
roomrys marked this conversation as resolved.
Show resolved Hide resolved
required: false
datasets:
- name: node_locations
dtype: float
dims:
- num_edges
- nodes_index, nodes_index
- - num_body_parts
- x, y
- - num_body_parts
- x, y, z
shape:
- - null
- 2
- - null
- 3
doc: Locations (x, y) or (x, y, z) of nodes for single instance in single frame.
roomrys marked this conversation as resolved.
Show resolved Hide resolved
- name: node_visibility
dtype: bool
dims:
- num_body_parts
shape:
- null
- 2
doc: Array of pairs of indices corresponding to edges between nodes. Index values
correspond to row indices of the 'nodes' dataset. Index values use 0-indexing.
doc: Markers for node visibility where true corresponds to a visible node and
false corresponds to an occluded node.
quantity: '?'
links:
- target_type: Skeleton
doc: Layout of body part locations and connections.
- neurodata_type_def: PoseTraining
neurodata_type_inc: NWBDataInterface
default_name: PoseTraining
doc: Group that holds images, ground-truth annotations, and metadata for training
a pose estimator.
groups:
- neurodata_type_inc: PoseEstimationSeries
doc: Estimated position data for each body part.
- neurodata_type_inc: Skeleton
doc: Skeleton used in project where each skeleton corresponds to a unique morphology.
quantity: '*'
- neurodata_type_inc: TrainingFrame
doc: Frames and ground-truth annotations for training a pose estimator.
quantity: '*'
2 changes: 2 additions & 0 deletions spec/ndx-pose.namespace.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@ namespaces:
- Ryan Ly
- Ben Dichter
- Alexander Mathis
- Liezl Maree
roomrys marked this conversation as resolved.
Show resolved Hide resolved
contact:
- rly@lbl.gov
- bdichter@lbl.gov
- alexander.mathis@epfl.ch
- lmaree@salk.edu
roomrys marked this conversation as resolved.
Show resolved Hide resolved
doc: NWB extension to store pose estimation data
name: ndx-pose
schema:
Expand Down
172 changes: 151 additions & 21 deletions src/spec/create_extension_spec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,47 @@ def main():
doc='NWB extension to store pose estimation data',
name='ndx-pose',
version='0.1.1',
author=['Ryan Ly', 'Ben Dichter', 'Alexander Mathis'],
contact=['rly@lbl.gov', 'bdichter@lbl.gov', 'alexander.mathis@epfl.ch'],
author=['Ryan Ly', 'Ben Dichter', 'Alexander Mathis', 'Liezl Maree'],
contact=['rly@lbl.gov', 'bdichter@lbl.gov', 'alexander.mathis@epfl.ch', 'lmaree@salk.edu'],
roomrys marked this conversation as resolved.
Show resolved Hide resolved
)

ns_builder.include_type('SpatialSeries', namespace='core')
ns_builder.include_type('NWBDataInterface', namespace='core')

skeleton = NWBGroupSpec(
neurodata_type_def='Skeleton',
neurodata_type_inc='NWBDataInterface',
doc='Group that holds node and edge data for defining parts of a pose and their connections to one another.',
default_name='Skeleton',
attributes=[
NWBAttributeSpec(
name='id',
doc='Unique ID associated with the skeleton.',
dtype='uint8',
roomrys marked this conversation as resolved.
Show resolved Hide resolved
),
],
datasets=[
NWBDatasetSpec(
name='nodes',
doc=('Array of body part names corresponding to the names of the PoseEstimationSeries objects or '
'PoseTraining objects.'),
dtype='text',
dims=['num_body_parts'],
shape=[None],
quantity=1,
),
NWBDatasetSpec(
name='edges',
doc=("Array of pairs of indices corresponding to edges between nodes. Index values correspond to row "
"indices of the 'nodes' dataset. Index values use 0-indexing."),
dtype='uint8',
dims=['num_edges', 'nodes_index, nodes_index'],
shape=[None, 2],
quantity='?',
),
]
)

pose_estimation_series = NWBGroupSpec(
neurodata_type_def='PoseEstimationSeries',
neurodata_type_inc='SpatialSeries',
Expand Down Expand Up @@ -72,6 +106,13 @@ def main():
quantity='*',
),
],
links=[
NWBLinkSpec(
doc='Layout of body part locations and connections.',
target_type='Skeleton',
quantity=1
),
],
datasets=[
NWBDatasetSpec(
name='description',
Expand Down Expand Up @@ -123,24 +164,6 @@ def main():
),
],
),
NWBDatasetSpec(
name='nodes',
doc=('Array of body part names corresponding to the names of the PoseEstimationSeries objects within '
'this group.'),
dtype='text',
dims=['num_body_parts'],
shape=[None],
quantity='?',
),
NWBDatasetSpec(
name='edges',
doc=("Array of pairs of indices corresponding to edges between nodes. Index values correspond to row "
"indices of the 'nodes' dataset. Index values use 0-indexing."),
dtype='uint8',
dims=['num_edges', 'nodes_index, nodes_index'],
shape=[None, 2],
quantity='?',
),
],
# TODO: collections of multiple links is currently buggy in PyNWB/HDMF
# links=[
Expand All @@ -152,7 +175,114 @@ def main():
# ],
)

new_data_types = [pose_estimation_series, pose_estimation]
training_frame = NWBGroupSpec(
neurodata_type_def='TrainingFrame',
neurodata_type_inc='NWBDataInterface',
doc='Group that holds ground-truth position data for all instances in a single frame.',
roomrys marked this conversation as resolved.
Show resolved Hide resolved
default_name='TrainingFrame',
groups=[
NWBGroupSpec(
neurodata_type_inc='Instance',
doc='Position data for a single instance in a single training frame.',
roomrys marked this conversation as resolved.
Show resolved Hide resolved
quantity='*',
),
NWBGroupSpec(
name='source_video',
doc='Path to original video file and frame used.',
quantity='?',
attributes=[
NWBAttributeSpec(
name='path',
doc='Path to original video file.',
dtype='text',
required=False,
),
NWBAttributeSpec(
name='frame_index',
doc='Frame index of TrainingFrame in original video file.',
dtype='uint8',
required=False,
),
],
),
NWBGroupSpec(
neurodata_type_inc='Image',
name='source_frame',
doc='Image frame used for training (stored either internally or externally).',
quantity='1',
),
],
attributes=[
NWBAttributeSpec(
name='annotator',
doc='Name of annotator who labeled the TrainingFrame.',
dtype='text',
required=False,
),
],
)

instance = NWBGroupSpec(
neurodata_type_def='Instance',
neurodata_type_inc='NWBDataInterface',
doc='Group that holds ground-truth pose data for single subject in a single frame.',
roomrys marked this conversation as resolved.
Show resolved Hide resolved
default_name='Instance',
links=[
NWBLinkSpec(
doc='Layout of body part locations and connections.',
target_type='Skeleton',
quantity=1
),
],
attributes=[
NWBAttributeSpec(
name='id',
doc='ID used to differentiate instances.',
roomrys marked this conversation as resolved.
Show resolved Hide resolved
dtype='uint8',
required=False,
),
],
datasets=[
NWBDatasetSpec(
name='node_locations',
doc=('Locations (x, y) or (x, y, z) of nodes for single instance in single frame.'),
dtype='float',
dims=[['num_body_parts', 'x, y'], ['num_body_parts', 'x, y, z']],
shape=[[None, 2], [None, 3]],
quantity=1,
),
NWBDatasetSpec(
name='node_visibility',
doc=('Markers for node visibility where true corresponds to a visible node and false corresponds to '
'an occluded node.'),
dtype='bool',
dims=['num_body_parts'],
shape=[None],
quantity='?',
),
],
)

pose_training = NWBGroupSpec(
neurodata_type_def='PoseTraining',
neurodata_type_inc='NWBDataInterface',
doc='Group that holds images, ground-truth annotations, and metadata for training a pose estimator.',
default_name='PoseTraining',
groups=[
NWBGroupSpec(
neurodata_type_inc='Skeleton',
doc='Skeleton used in project where each skeleton corresponds to a unique morphology.',
quantity='*',
),
NWBGroupSpec(
neurodata_type_inc='TrainingFrame',
doc='Frames and ground-truth annotations for training a pose estimator.',
quantity='*',
),
],
)

new_data_types = [skeleton, pose_estimation_series, pose_estimation, training_frame, instance, pose_training]

# export the spec to yaml files in the spec folder
output_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', 'spec'))
Expand Down