[FIX] Recommend SI units formatting to adhere to CMIXF-12

[sappelhoff]

replaces #151 as proposed by @satra , please see discussion therein!

Here is a summary:

• In BIDS we accept SI units only (...and in very special circumstances, custom units - but they'd need to be documented in JSON files)
• Currently, we have nowhere in the spec clarified how the FORMATTING of the SI units should be done (although some users and developers have assumed that the units appendix shows the required formatting)
• Thus in practice, users have applied either ASCII formatting (e.g., "u" instead of "µ") or unicode formatting (µ) ... with the problem being that some unicodes look the same, while being different (e.g., µ and μ ... micro vs. mu)

This PR is intended to explicitly state how we want the FORMATTING of SI units to be. Namely, we prefer ASCII formatting according to MIXF (metric interchange format), see --> http://people.csail.mit.edu/jaffer/MIXF/CMIXF-12.

For backward compatibility, we will allow unicode characters, but further clarify the exact unicode we expect (e.g, U+00B5 for the micro sign µ).

Please see the proposed changes in this PR and approve or discuss!

more recent summary: #411 (comment)

To do

• add paragraphs to spec that teach users how we expect the formatting to be
• discuss with community and reach consensus on whether CMIXF is a good idea
• get some kind of a mixf validator (we got https://github.com/sensein/cmixf by Satra)
• stop bids-validator from units validation how it is currently implemented (overhaul units validation in bids-validator bids-validator#969)
• make bids-validator check MIXF units (overhaul units validation in bids-validator bids-validator#969) ... and warn if the units are not compliant (but no error, we only RECOMMEND)

[effigies]

The table looks good. I have some suggested alternative wording for the new text. Hopefully this is more unambiguous about what is permitted and what is required.

I dropped the bit about multiple unicode code points. If you feel it's worth working back in, that's fine.

[sappelhoff]

Thanks @effigies, I think your wording was more unambiguous and I added it.

I think we could still clarify a bit more with the units, before we try to get this PR in.

1. For encoding a unit, can a user pick any combination of:

• prefix and unit, e.g. microV, microvolt
• symbol and unit, e.g. uV, µV, uvolt, µvolt
• factor and unit, e.g. 1e-6V, 1e-6volt (probably not)

... or do we require e.g. SYMBOL and UNIT

2. (implied in 1.) Can a user write out the complete unit? E.g., volt OR V are both permissible?

3. Regarding lower and upper case, I assume that we go with how it's handled in CMIXF-12, yet it may be good to make this more obvious with an additional sentence.

[sappelhoff]

let me just make a proposal that's easier to criticize than my 3 questions above.

Let's add this sentence right before the table:

Units MUST consist of the `unit symbol` with an optionally accompanying
`prefix symbol` (see table below). Appropriate upper- or lower- casing MUST
be applied as declared by CMIXF-12.

Examples:
1. `uV` or `µV` are permissible, but NOT: `microV`, `µvolt`, `1e-6V`, etc.

feel free to add an example. Or adjust this text to be more clear.

[satra]

should we clarify that combinations of units are allowed if required. For example slew rate (T/m/s).

[effigies]

@sappelhoff I agree in principle with what you are proposing, but we should also note that CMIXF-12 does have conventions for derived units, including parentheses, exponents, multiplication and division.

[sappelhoff]

reminder: this PR will also need to be coordinated with #391

[sappelhoff]

This is a biggish change, I'd like to get some more feedback on this @yarikoptic @robertoostenveld @nicholst @oesteban

[sappelhoff]

I think this issue deserves more attention, so let me write a summary for everbody to catch up and offer their opinion.

Selective tag list (please tag more people that I forgot):

@yarikoptic @robertoostenveld @nicholst @oesteban @franklin-feingold @emdupre @Remi-Gau @satra @VisLab @dorahermes @effigies

This issue is about how to format units in BIDS.

Brief history

• Using the CMIXF-12 formatting for units was first proposed by @satra 14 months ago, but we did not act any further on it (RFC: represent units in ascii #151)
• The issue more recently (~3 months ago) came up again, because the validator (cc @rwblair ) started validating units (see: validate units for tsv files bids-validator#885)

The problem

I think it's great that the validator now checks units ... but this also put the finger on the wound in our spec: Namely, that the way units are currently defined in BIDS is unclear with regards to formatting.

We clearly state that we RECOMMEND SI units but we are unclear as to how we want these units to be formatted.

Some users have taken a look at our units appendix and deduced that they can make use of any combination of:

• unit name
• unit symbol
• prefix name
• prefix symbol

For example: µV, µVolt, microVolt, microV as formattings of "micro volts"

and this is also exactly what the validator currently checks.

Yet nowhere in the spec to we REQUIRE this behavior. So other users have (arguably reasonably) formatted their SI units in some other ways. E.g., uV to reflect "micro volts".

Current situation and where to go from here

This brings us to the current situation: The specification is unclear with how to format units, but the validator is strict in what it accepts as a valid unit.

This PR is intended to clarify how we want to have units formatted in BIDS. If you care about this problem, please offer your opinion ... and if you know somebody who might care, please refer them to this place :-) 🚀

[nicholst]

I like the clarity of MIXF but I can understand that doesn't totally specify things.

This seems like an area where others have put some thought, though... I don't have time to digest, but it seems the following will be helpful...

• "Recommended Unit Symbols, SI Prefixes, and Abbreviations", IEEE document
• "SCC 14 Conventions for Metrication of IEEE Standards", IEEE Standards Coordinating Committee document
• IEEE. (2002). IEEE/ASTM Standard for Use of the International System of Units (SI): The Modern Metric System. IEEE/ASTM SI 10-2002 (Revision of IEEE/ASTM SI 10-1997). https://doi.org/10.1109/IEEESTD.2002.94225

The last one, an IEEE/ANSI standard, even has a section titled "Application of SI prefixes".

I'm hoping we can simply direct readers to one of these standards to say "do it that way, but render it as per MIXF".

[satra]

but I can understand that doesn't totally specify things

@nicholst - can you unpack this a bit more? there is an entire yacc parser for cmixf-12 at the end of the document, so technically it does provide a complete representation of formatting as well.

also i know there are some additional issues in BIDS w.r.t units that we have to tackle such as the fact that certain things (age, height, weight) represented as SI are likely to impose significant constraints on certain groups in certain countries. (perhaps this should be a separate issue, but likely to come up in all contexts).

[nicholst]

Hi @satra,

I should have rephrased my comment as "I interpret @sappelhoff's message to mean that MIXF doesn't totally specify things" :)

@sappelhoff: Maybe you can clarify what's missing from MIXF?

[sappelhoff]

Maybe you can clarify what's missing from MIXF?

I don't think that there is something missing from MIXF. And I do think that it may be a nice way forward to require MIXF formatting of SI units.

Yet, I see several issues that need to be discussed with more people than the usual suspects, because we have a high chance of breaking existing tools and datasets:

• Previously users have formatted SI units however they wanted, and nobody (including the validator) complained
• Now, the validator checks BIDS datasets according to rules that are not clearly specified in our specification (see my summary post)
• Thus, users may now start to adopt the "validator" rules, which are not clearly defined BIDS specification rules (because we are unclear in the spec)
• Hence we should clarify BIDS specification rules, and make the validator enforce these rules ... to avoid a second "breakage"

---

I agree with @nicholst that it would be nice to simply refer to another standard: "Please use units according to this" ... and even better if there is an existing validator for that standard that we can simply tie into our bids-validator

---

@satra yes I think that it'd be helpful to split off the discussion about age, and I am interested to hear more about why you think weight and height could be problematic to be represented in SI.

[sappelhoff]

I went through the PR once more and changed the REQUIRED keywords back to RECOMMENDED.

Now the PR achieves the following:

1. RECOMMEND CMIXF-12 units
2. provide examples of CMIXF-12 and say why we want it
3. align present spec examples to our new recommendation
4. more consistently link to the units section instead of specifying how to define units in different places
5. Clarify that some unicodes (5, to be exact) are backward compatible

From what I see, we are not breaking any previous versions of BIDS. Only making some recommendations where previously we were unclear.

This PR is now ready to be merged from my side. Please review and approve ... or tell me that I overlooked something. And thanks for your patience 👍

[nicholst]

LGTM

[effigies]

Most suggestions are to use [<section name>](...) rather than [<section name> section](...), which I think is more consistent with how we link elsewhere in the document. We may be somewhat inconsistent, as well, so feel free to reject if you think this damages readability.

Note that there are a number of links ending parentheticals, so the parentheticals aren't getting closed.

A couple other comments, but otherwise this LGTM.

[effigies]

Sorry, yes. The "Review" button did not show that I had these in queue, I guess because they were outdated. I'll delete them.

[sappelhoff]

I will merge this PR early next week, so everybody has some more time to complain or approve ... a bit like they say at some weddings:

Speak now or forever hold your peace

[sappelhoff]

Thanks a lot to everybody who helped along the way. Especially to @satra who brought this up.