Need permission to update the HED validator component of BIDS for units

[VisLab]

All,
A group of us has been working on improving the HED (https://github.com/hed-standard) event annotation schema and tools. My lab has contributed the HED validation component of the current BIDS validator.

One of the things that we have worked on is units associated with HED tags. BIDS uses SI units (for the most part). The HED schema had a limited list of allowed units. We have created a new branch of the HED schema that allows all of the SI units specified by bids including multiples and sub-multiples. We have also created a branch of the hed-validator that handles this. We would like to merge this branch into our master and incorporate it into the bids-validator.

The issue is that our new validator will no longer support previous versions of the HED schema. We think that this is an important step and should be done as soon as possible for the community. It should not affect datasets that have already been tagged with HED. It mainly requires that they will need to validate against HED schemas 8.0.0 or later.

We want to know if you foresee any issues or there are objections before we do a pull request to make this happen.

[sappelhoff]

Hi @VisLab, thanks for checking in and cool to hear about HED improvements.

We have created a new branch of the HED schema that allows all of the SI units specified by bids

Yes, BIDS RECOMMENDS SI units unless impossible (we are using the word SHOULD in the spec, not MUST), but we the specification is a bit unclear as to how the SI units should be formatted, see bids-standard/bids-specification#411

so my question is: what formatting does the new HED schema accept for units? E.g., micro volts could be theoretically formatted as any of the following: µV, microV, uV, ...

including multiples and sub-multiples

could you please explain what you mean by that? Combined units like V/s or something like that?

The issue is that our new validator will no longer support previous versions of the HED schema.

given that datasets until now have been tagged with previous HED schema versions ... how can these datasets NOT be affected? What happens if the new validator encounters a previous version of HED schemas?

[VisLab]

First, the issue that will be relevant to users tagging their data with HED: *The issue is that our new validator will no longer support previous versions of the HED schema.* *given that datasets until now have been tagged with previous HED schema versions ... how can these datasets * *NOT be affected? What happens if the new validator encounters a previous version of HED schemas?* What we have implemented is a change to the unit classes of the HED schema, which specifies what units are possible for a physical quantity such as time. Users put units into their tags. All of the existing allowed units are still in there so data that has been previously tagged will still validate. What we did was to reorganize the unit classes in three ways: 1) We eliminated all explicit specifications of plurals. Instead, the unit classes specify the singular of a unit type and the validators (both Python and JavaScript) use pluralize to add the plurals to the internal dictionary. Since our original specification was somewhat hit or miss, this adds a lot of allowed singular or plurals that weren't there before but should have been. 2) The second thing that we did was to add all of the SI units (and their abbreviations) to the allowed unit classes. (Some were there before and some weren't). 3) We added several attributes to the schema (and this is why our validators are not backwards compatible without some hassle). The attributes specify that a particular unit type is an SIUnit and whether it is an abbreviation (UnitSymbol). For Units that have the SIUnit attribute, the validator adds all possible combinations of the multiple and submultiple prefixes to the allowed dictionary. Similarly units that have both SIUnit and UnitSymbol attributes will have the corresponding abbreviations. *Example; *Here was the original HED schema specification for the unit class for time (from https://github.com/hed-standard/hed-specification/blob/master/HED-schema.mediawiki ) time {default=s} - s - second - seconds - centiseconds - centisecond - cs - hour:min - day - days - ms - milliseconds - millisecond - minute - minutes - hour - hours Here is the new specification for the unit class for time: ( https://github.com/hed-standard/hed-specification/blob/HED-devunit/HED-schema.mediawiki ) time {defaultUnits=s} - second {SIUnit} - s {SIUnit, unitSymbol} - hour:min {unitSymbol} - day - minute - hour Believe it or not, from a user perspective all of the original allowed unit classes are still in there. There is also a complete list of unit modifiers as part of the new HED schema of which the following is the start of that long list. *Unit modifiers* - deca {SIUnitModifier} [SI unit multiple representing 10^1] - da {SIUnitSymbolModifier} [SI unit multiple representing 10^1] - hecto {SIUnitModifier} [SI unit multiple representing 10^2] - h {SIUnitSymbolModifier} [SI unit multiple representing 10^2] - kilo {SIUnitModifier} [SI unit multiple representing 10^3] - k {SIUnitSymbolModifier} [SI unit multiple representing 10^3] ..... Again the users don't see this. When they annotate data, a particular tag might have a unit class associated with it (such as time or frequency). If it does they have to choose a valid unit from that unit class. The list of valid units is now consistent and expanded.

…

On Sat, Apr 11, 2020 at 12:22 PM Stefan Appelhoff ***@***.***> wrote: Hi @VisLab <https://github.com/VisLab>, thanks for checking in and cool to hear about HED improvements. We have created a new branch of the HED schema that allows all of the SI units specified by bids Yes, BIDS RECOMMENDS SI units unless impossible (we are using the word SHOULD in the spec, not MUST <https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#units>), but we the specification is a bit unclear as to *how* the SI units should be formatted, see bids-standard/bids-specification#411 <bids-standard/bids-specification#411> so my question is: what *formatting* does the new HED schema accept for units? E.g., micro volts could be theoretically formatted as any of the following: µV, microV, uV, ... including multiples and sub-multiples could you please explain what you mean by that? Combined units like V/s or something like that? The issue is that our new validator will no longer support previous versions of the HED schema. given that datasets until now have been tagged with previous HED schema versions ... how can these datasets NOT be affected? What happens if the new validator encounters a previous version of HED schemas? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#942 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCJOVOZFJBAEP5LEMTKW3RMCRMTANCNFSM4MGBHLRA> .

[sappelhoff]

Thanks for the details, unfortunately there is still something that I don't understand (perhaps @effigies or @rwblair can help me).

Specifically, I don't see how these two points you mention go together [emphasis added by me]:

1. All of the existing allowed units are still in there so data that has been previously tagged will still validate
2. We added several attributes to the schema (and this is why our
   validators are not backwards compatible without some hassle).

If old valid HED datasets are still valid with the new HED validator, that would be what I call backward compatible. In a case of backward compatibility, I a see no issue to update the HED validator within bids-validator.

Perhaps you could once more specify what you meant with this sentence from your original post [parts in [] added by me]:

It mainly requires that they [the users] will need to validate against HED schemas 8.0.0 or later.

When users apply bids-validator, they do not specify any HED schema - at least I do not see an option like this in the CLI parameters that are possible (such as --ignoreNiftiHeaders)

[VisLab]

By default, the latest version of the latest version of the HED schema is used ( https://github.com/hed-standard/hed-specification/blob/master/hedxml/HEDLatest.xml). I will need to defer to Alexander Jones as to where that is in the code. Users can specify a schema that is earlier explicitly in the hed-validator. I think that we need to discuss how we might have an optional command line argument for the BIDS validator to specify a HED schema that is not the latest.. (This issue is independent of what we do about support for unit modifiers.) As we go forward, users will not want to change tags in their data because the schema changed. All of the existing versions of the HED schema have unit classes. HED schema 8.0.0 will have unit modifiers but previous versions don't..... We may need to restructure our new version of the validator to handle previous version of the schema.

…

On Sun, Apr 12, 2020 at 3:13 AM Stefan Appelhoff ***@***.***> wrote: Thanks for the details, unfortunately there is still something that I don't understand (perhaps @effigies <https://github.com/effigies> or @rwblair <https://github.com/rwblair> can help me). Specifically, I don't see how these two points you mention go together [emphasis added by me]: 1. All of the existing allowed units are still in there so *data that has been previously tagged will still validate* 2. We added several attributes to the schema (and this is why our *validators are not backwards compatible without some hassle*). If *old valid HED datasets* are still valid with the *new HED validator*, that would be what I call backward compatible. In a case of backward compatibility, I a see no issue to update the HED validator within bids-validator. Perhaps you could once more specify what you meant with this sentence from your original post [parts in [] added by me]: It mainly requires that they [the users] will need to validate against HED schemas 8.0.0 or later. When users apply bids-validator, they do not specify any HED schema - at least I do not see an option like this in the CLI parameters that are possible <https://github.com/bids-standard/bids-validator/blob/master/bids-validator/bin/bids-validator> (such as --ignoreNiftiHeaders) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#942 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJCJOW6O67XO4PUDLDHSBTRMFZ3FANCNFSM4MGBHLRA> .

[sappelhoff]

Users can specify a schema that is earlier explicitly in the hed-validator. I think that we need to discuss how we might have an optional command line argument for the BIDS validator to specify a HED schema that is not the latest..

that is one option. I think that another option would be to add a field to some BIDS .json (e.g., the events.json file) a new HED field called something like HEDSchemaVersion, where users put the version of the schema they used for populating the HED column in their events.tsv file.

This could further be clarified in the HED section in the BIDS specification, because currently, there is nothing about different HED schemas there.

And then the HED-validator within bids-validator could (1) search for the HEDSchemaVersion and apply the appropriate HED-validator, or (2) if no HEDSchemaVersion field is found, apply the latest HED-validator.

What do you think?

[happy5214]

Basically, the new validator branch as written would not permit users to validate against a schema that doesn't include unit modifiers, or everything older than 8.0.0. However, since all explicit units included in earlier versions of the schema (including the current version) are either explicitly in version 8.0.0 or are derivable from those units through plurals and unit prefixes, so no HED string valid now should become invalid because of these changes. But HED strings that are already invalid with the current schema could not be validated at all using the schema version for which they were written, since it wouldn't load the schema at all given the current code.

In bids-validator/validators/events/hed.js, line 102, the buildSchema function exported by the hed-validator package is called with no arguments. That causes the schema code in hed-validator to default to the Latest version, which is a manually maintained symlink in the HED specification repo that points to the latest released version.

There is currently no mechanism through which to pass a version to buildSchema through bids-validator. It could be done by passing as a parameter to buildSchema an object that contains either a path (for a local schema copy) or version (which loads the corresponding version from the same HED spec repo). However, we'd need to get a version override through the code to the call to buildSchema. I don't know how hard it is to pass a command-line option that deep. The code is already loading events.json through the sidecar mechanism to work with the custom column values, so adding the schema version to that would not be difficult.

[effigies]

This sounds reasonable to me.

[VisLab]

Sounds good to me as well. We have also had a discussion about doing some minor restructuring of our code so that:

1. if the schema doesn't have unit modifiers, then only explicitly given unit classes values will be allowed as units.
2. if the schema doesn't have unit classes, the validator doesn't validate units at all.

We believe that these changes will allow the validator to work with previous versions of the schema. We'll get back to you after we look at this in more detail.

[happy5214]

The changes described above have been made to hed-validator.