From 158c8a467041f3022ccaa54eb5e193647c612b44 Mon Sep 17 00:00:00 2001 From: Tal Pal Attia Date: Fri, 29 Sep 2023 08:59:03 -0400 Subject: [PATCH 1/2] [FIX] Hierarchical Event Descriptors (HED) page update --- src/appendices/hed.md | 127 ++++++++++++++++++++++++++++-------------- 1 file changed, 85 insertions(+), 42 deletions(-) diff --git a/src/appendices/hed.md b/src/appendices/hed.md index 866f795cf1..7ae2aefc22 100644 --- a/src/appendices/hed.md +++ b/src/appendices/hed.md @@ -12,19 +12,19 @@ A HED annotation consists of terms selected from a controlled hierarchical vocabulary (the HED schema). Individual terms are comma-separated and may be grouped using parentheses to indicate association. -See [https://www.hedtags.org/display_hed.html](https://www.hedtags.org/display_hed.html) +See the [HED Schema Browser](https://www.hedtags.org/display_hed.html) to view the HED schema and the [HED resources](https://www.hed-resources.org/en/latest/) site for additional information. Starting with HED version 8.0.0, HED allows users to annotate using individual -terms or partial paths in the HED vocabulary (for example `Red` or `Visual-presentation`) +terms or partial paths in the HED vocabulary (for example, `Red` or `Visual-presentation`) rather than the full paths in the HED hierarchy ( `Property/Sensory-property/Sensory-attribute/Visual-attribute/Color/CSS-color/Red-color/Red` or `Property/Sensory-property/Sensory-presentation/Visual-presentation`). HED specific tools MUST treat the short (single term) and long (full path) HED tag forms interchangeably, -converting between the forms when necessary based on the HED schema. +converting between the forms, when necessary, based on the HED schema. Examples of test datasets using the various forms can be found in [hed-examples/datasets](https://github.com/hed-standard/hed-examples/tree/main/datasets) on GitHub. @@ -40,14 +40,14 @@ files in various places in the dataset hierarchy Dataset curators MAY also include additional columns and define their meanings in associated JSON sidecar files (`events.json`). -Example: An excerpt from an `events.tsv` file containing three columns +**Example:** An excerpt from an `events.tsv` file containing three columns (`trial_type`, `response_time`, and `stim_file`) in addition to the required `onset` and `duration` columns. ```Text onset duration trial_type response_time stim_file 1.2 0.6 go 1.435 images/red_square.jpg -5.6 0.6 stop 1.739 images/blue_square.jpg +5.6 0.6 stop n/a images/blue_square.jpg ``` The `trial_type` column in the above example contains a limited number of distinct @@ -66,13 +66,13 @@ The HED annotation for a value column must include a `#` placeholder, which dedicated HED tools MUST replace by the actual column value when the annotations are assembled for analysis. -Example: An accompanying `events.json` sidecar describing both categorical and +**Example:** An accompanying `events.json` sidecar describing both categorical and value columns of the previous example. The `duration` column is also annotated as a value column. ```JSON { - "Duration": { + "duration": { "LongName": "Image duration", "Description": "Duration of the image presentations", "Units": "s", @@ -82,12 +82,12 @@ The `duration` column is also annotated as a value column. "LongName": "Event category", "Description": "Indicator of type of action that is expected", "Levels": { - "go": "A blue square is displayed to indicate starting", - "stop": "A red square is displayed to indicate stopping" + "go": "A red square is displayed to indicate starting", + "stop": "A blue square is displayed to indicate stopping" }, "HED": { - "go": "Sensory-event, Visual-presentation, (Square, Blue)", - "stop": "Sensory-event, Visual-presentation, (Square, Red)" + "go": "Sensory-event, Visual-presentation, (Square, Red)", + "stop": "Sensory-event, Visual-presentation, (Square, Blue)" } }, "response_time": { @@ -106,18 +106,20 @@ The `duration` column is also annotated as a value column. Dedicated HED tools MUST assemble the HED annotation for each event (row) by concatenating the annotations for each column, along with the annotation contained directly in a `HED` column -of that row as described in the next section. +of that row, as described in the next section. -Example: The fully assembled annotation for the first event in the above +**Example:** The fully assembled annotation for the first event in the above `events.tsv` file with onset `1.2` (the first row) is: ```Text -Duration/0.6 s, Sensory-event, Visual-presentation, (Square, Blue), -(Delay/1.435 ms, Agent-action, (Experiment-participant, (Press, Mouse-button))), +(Duration/0.6 s, Sensory-event, Visual-presentation, +((Square, Red), (Computer-screen, Center-of))), +(Delay/1.435 ms, Agent-action, (Experiment-participant, +(Press, Mouse-button))), Pathname/images/red_square.jpg ``` -## Annotation using the `HED` column +### Annotation using the `HED` column Another tagging strategy is to annotate individual events directly by including a `HED` column in the `events.tsv` file. @@ -138,7 +140,7 @@ event annotations. Annotations placed in sidecars are the RECOMMENDED way to annotate data using HED. These annotations are preferred to those placed -directly in the `HED` column, because they are simpler, more compact, +directly in the `HED` column because they are simpler, more compact, more easily edited, and less prone to inconsistencies. ## HED and the BIDS inheritance principle @@ -162,11 +164,11 @@ The HED vocabulary is specified by a HED schema, which delineates the allowed HED path strings. The version of HED used in tagging a dataset should be provided in the `HEDVersion` field of the `dataset_description.json` file located in the dataset root directory. -This allows for a proper validation of the HED annotations -(for example using the `bids-validator`). +This allows for properly validating the HED annotations +(for example, using the `bids-validator`). -Example: The following `dataset_description.json` file specifies that the -[`HED8.1.0.xml`](https://github.com/hed-standard/hed-schemas/blob/main/standard_schema/hedxml/HED8.1.0.xml) +**Example:** The following `dataset_description.json` file specifies that the +[`HED8.2.0.xml`](https://github.com/hed-standard/hed-schemas/blob/main/standard_schema/hedxml/HED8.2.0.xml) file from the `standard_schema/hedxml` directory of the [`hed-schemas`](https://github.com/hed-standard/hed-schemas) repository on GitHub should be used to validate the study event annotations. @@ -174,29 +176,45 @@ repository on GitHub should be used to validate the study event annotations. ```JSON { "Name": "A great experiment", - "BIDSVersion": "1.7.0", - "HEDVersion": "8.1.0" + "BIDSVersion": "1.8.0", + "HEDVersion": "8.2.0" } ``` -If you omit the `HEDVersion` field from the dataset description file, -a warning will be generated and -any present HED information will be validated using the latest version of the HED schema. -This is bound to result in problems, and hence, it is strongly RECOMMENDED that the -`HEDVersion` field be included when using HED in a BIDS dataset. +The BIDS validator will generate an error if your dataset uses HED +and the `HEDVersion` field is missing from the dataset description file. +To avoid this, include a `HEDVersion` field in the `dataset_description.json` +if you are using HED annotations. ### Using HED library schemas -HED also allows you to use one or more specialized vocabularies along with or instead of +HED also allows you to use one or more specialized vocabulary along with or instead of the standard vocabulary. These specialized vocabularies are developed by communities of users and are available in the GitHub [hed-schemas](https://github.com/hed-standard/hed-schemas) repository. -Library schema are specified in the form `library-version>`. +A library schema is specified in the form `library-version>`. + +#### Partnered library schemas + +Partnered library schemas were introduced in HED specification version 3.2.0 +and are supported by HED standard schema versions ≥ 8.2.0. +A partnered library schema is tied to a specific version of +the HED standard schema as specified in its header. A given library schema +version is either partnered or standalone. + +**Note:** Whether a particular library schema version is partnered or +unpartnered is fixed when the library is released and cannot be changed. +For example, +[HED-SCORE version 1.0.0](https://github.com/hed-standard/hed-schemas/blob/main/library_schemas/score/hedwiki/HED_score_1.0.0.mediawiki) +is unpartnered, but [HED-SCORE version 1.1.0](https://github.com/hed-standard/hed-schemas/blob/main/library_schemas/score/hedwiki/HED_score_1.1.0.mediawiki) +is partnered with standard schema version 8.2.0. + +##### Unpartnered library schema example -Example: The following `dataset_description.json` file specifies that the +The following `dataset_description.json` file specifies that the [HED8.1.0.xml](https://github.com/hed-standard/hed-schemas/blob/main/standard_schema/hedxml/HED8.1.0.xml) -standard schema should be used along with the -SCORE library for clinical neurological annotation located at +standard schema should be used along with the HED-SCORE library schema +for clinical neurological annotation located at [HED_score_1.0.0.xml](https://github.com/hed-standard/hed-schemas/blob/main/library_schemas/score/hedxml/HED_score_1.0.0.xml). ```JSON @@ -206,22 +224,24 @@ SCORE library for clinical neurological annotation located at "HEDVersion": ["8.1.0", "sc:score_1.0.0"] } ``` -The `sc:` is a user-chosen prefixes used to distinguish the sources +The `sc:` is a user-chosen prefix used to distinguish the sources of the terms in the HED annotation. The prefixes MUST be alphanumeric. Any number of prefixed schemas may be used in addition to a non-prefixed one. The following HED annotation from this dataset uses the `sc:` prefix with -`Photomyogenic-response` and `Wicket-spikes` because these terms are from the -SCORE library, while `Data-feature` is from the standard HED schema. +`Eye-blink-artifact` and `Seizure-PNES` because these terms are from the +HED-SCORE library schema, while `Data-feature` is from the standard HED schema. ```Text -Data-feature, sc:Photomyogenic-response, sc:Wicket-spikes +Data-feature, sc:Eye-blink-artifact, sc:Seizure-PNES ``` -If only one schema is being used for annotation, the prefix can be omitted entirely. -The following `dataset_description.json` indicates that only the SCORE library version -0.0.1 will be used for HED annotation in this dataset. +##### Single unpartnered library schema example + +If only one schema is used for annotation, the prefix can be omitted entirely. +The following `dataset_description.json` indicates that only the HED-SCORE library schema version +1.0.0 will be used for HED annotation in this dataset. ```JSON { @@ -231,8 +251,31 @@ The following `dataset_description.json` indicates that only the SCORE library v } ``` -The corresponding notations in the dataset do not have a prefix: +The corresponding annotations in the dataset do not have a prefix: + +```Text +Eye-blink-artifact, Seizure-PNES +``` + +##### Partnered library schema example + +The following `dataset_description.json` file specifies that +the HED-SCORE library schema +[version 1.1.0](https://github.com/hed-standard/hed-schemas/blob/main/library_schemas/score/hedwiki/HED_score_1.1.0.mediawiki) is used. +This particular library schema version is partnered with the standard schema version +[8.2.0](https://github.com/hed-standard/hed-schemas/blob/main/standard_schema/hedxml/HED8.2.0.xml). + +```JSON +{ + "Name": "A great experiment", + "BIDSVersion": "1.8.0", + "HEDVersion": "score_1.1.0" +} +``` +The corresponding annotations in the dataset use tags from the +HED-SCORE library schema (`Eye-blink-artifact` and `Seizure-PNES`) and from the standard HED (`Data-feature`) +as follows: ```Text -Photomyogenic-response, Wicket-spikes +Data-feature, Eye-blink-artifact, Seizure-PNES ``` From 518d88df46171a6e8aeaeaedf64bcf29d1bfd8aa Mon Sep 17 00:00:00 2001 From: Chris Markiewicz Date: Sun, 1 Oct 2023 21:21:07 -0400 Subject: [PATCH 2/2] FIX: Implement Kay's suggestions --- src/appendices/hed.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/src/appendices/hed.md b/src/appendices/hed.md index 7ae2aefc22..4582356ee5 100644 --- a/src/appendices/hed.md +++ b/src/appendices/hed.md @@ -112,8 +112,8 @@ of that row, as described in the next section. `events.tsv` file with onset `1.2` (the first row) is: ```Text -(Duration/0.6 s, Sensory-event, Visual-presentation, -((Square, Red), (Computer-screen, Center-of))), +Duration/0.6 s, Sensory-event, Visual-presentation, +((Square, Red), (Computer-screen, Center-of)), (Delay/1.435 ms, Agent-action, (Experiment-participant, (Press, Mouse-button))), Pathname/images/red_square.jpg @@ -188,19 +188,23 @@ if you are using HED annotations. ### Using HED library schemas -HED also allows you to use one or more specialized vocabulary along with or instead of -the standard vocabulary. These specialized vocabularies are developed by -communities of users and are available in the GitHub -[hed-schemas](https://github.com/hed-standard/hed-schemas) repository. +HED also allows you to use one or more specialized vocabularies +along with or instead of the standard vocabulary. +These specialized vocabularies are developed by communities of users +and are available in the +[hed-schemas](https://github.com/hed-standard/hed-schemas) GitHub repository. A library schema is specified in the form `library-version>`. #### Partnered library schemas +A partnered schema is one whose vocabulary trees are merged with +its standard schema partner when the schema is released. +Thus, the two vocabularies appear as one vocabulary to the annotator. Partnered library schemas were introduced in HED specification version 3.2.0 and are supported by HED standard schema versions ≥ 8.2.0. -A partnered library schema is tied to a specific version of -the HED standard schema as specified in its header. A given library schema -version is either partnered or standalone. +Each partnered library schema is tied to a specific version of +the HED standard schema as specified in its header. +A given library schema version is either partnered or standalone. **Note:** Whether a particular library schema version is partnered or unpartnered is fixed when the library is released and cannot be changed. @@ -224,7 +228,7 @@ for clinical neurological annotation located at "HEDVersion": ["8.1.0", "sc:score_1.0.0"] } ``` -The `sc:` is a user-chosen prefix used to distinguish the sources +The `sc:` is a user-chosen prefix used to distinguish the source schemas of the terms in the HED annotation. The prefixes MUST be alphanumeric. Any number of prefixed schemas may be used in addition to a non-prefixed one.