[FIX] Drop _part- reference from example, introduce _split- entity

[sappelhoff]

closes #429

This PR is intended to result in a solution for #429. The proposed changes here should make it easier to discuss how to resolve the issue.

currently proposed changes aim at keeping the _part entity in BIDS and documenting it in a proper way. The _part entity is defined to indicate parts of a split file.

Given the developments in mne-tools/mne-python#7776 I changed the title and proposed changes of this PR -> Drop _part- reference from example, introduce _split- entity

To do

• agree on a term to replace part --> split?
• update existing datasets that used _part
  • there are none on bids-examples, as verified using find . -name '*_part*'
  • need to check OpenNeuro. Mainak did some random checking and it seems fine (BIDS related: potential change of .fif file split naming mne-tools/mne-python#7776 (comment))
• update BIDS validator (https://github.com/bids-standard/bids-validator/pull/960)
• update MNE-Python ([MRG] split_naming='bids' changes from _part-%d to _split-%d mne-tools/mne-python#7794)
• update MNE-BIDS ([MRG] _part- -> _split- to conform with BIDS  mne-tools/mne-bids#417)

---

Reading this fieldtrip mail it seems to me, that FieldTrip does not deal with split FIF files at all --> @robertoostenveld to me this seems like a similar problem as with BrainVision, where the vhdr file contains pointers to the accompanying eeg and vmrk files.

When renaming the files, their counterparts SHOULD (I think even MUST) be renamed --> yet for FIF, how did you deal with this so far?

[robertoostenveld]

Reading this fieldtrip mail it seems to me, that FieldTrip does not deal with split FIF files at all --> @robertoostenveld to me this seems like a similar problem as with BrainVision, where the vhdr file contains pointers to the accompanying eeg and vmrk files.

FieldTrip does deal with split files in general, however not specifically for FIF but using a general mechanism. Please note that not everyone contributing to the mailing list might be aware of this. See https://github.com/fieldtrip/fieldtrip/blob/d8662e204096419b2bfafcc64910e12665368209/fileio/ft_read_header.m#L124, you would use this by specifying

cfg = [];
cfg.dataset = {'file1.ext', 'file2.ext'};
data_concat = ft_preprocessing(cfg);

FieldTrip does (up to now) not autodetect neuromag fif files that should go together, in that sense it behaves similar as most of the vendor software (Neuromag/Elekta/MEGIN) that also does not do that. So it is the responsibility of the user.

Regarding the comparison to BrainVision: yes, I see there is a resemblance of files pointing to each other (in the Neuromag case as a linked-list) where the file names are explicitly coded inside the header content. In most cases (multi-file formats) that I am aware of, there are no explicit pointers between files, but only implicit ones, where links between files are implemented based on the filenames alone. Putting filenames inside the binary content of files IMHO messes up the relation between metadata (the name) and the data (the content) and is prone to problems. E.g renaming requires special tools, symbolic linking does not work, there is the risk of the path (e.g with D:/myproject/data/IdentifyableSubjectName) ending up in the files, etc. I think that MNE-C++ and MNE-Python would be better of by not relying on the explicit linked list represented in the file content, but rather restrict themselves to a pattern in the file names.

[robertoostenveld]

Having said this, I do agree with the proposed pull-request: I do not see it as conflicting with the current interpretation, and it makes it more clear.

[jasmainak]

LGTM so far!

[yarikoptic]

There is no <id>, see further points in the comments

[yarikoptic]

see other comment about <index> vs <label> - let's see if it could be relaxed here to be <label> right away.

[robertoostenveld]

I had not interpreted <index> as index into something explicitly (as in expressing a link), rather as how you would use it in a for-loop, which means a sequence of integers that increment by one. The starting value is not defined AFAIK, and I would expect it either to be 0 or 1.

The run entity is the earliest use of <index>, but it is also used (in the same part of the spec) for the echo entity (although not explicitly defined there, it is in examples and in the entity appendix table).

The use of part- in MEG requires it to be <index> and that is also how it is specified now, so I think the fix in this PR is correct (as long as <id> is changed to <index>). Extending the value to <label> would be another discussion. Right now I don't think we have consensus yet on re-using part (with a different meaning of the entity) for non-MEG data, so I think it would be too early to decide for loosening it up from index to label.

[yarikoptic]

Re <index> as an explicit index, it might be coming as a part of BEP001 (ideally should be a separate BEP or a PR first): https://github.com/bids-standard/bep001/blob/indexable_metadata/src/04-modality-specific-files/01-magnetic-resonance-imaging-data.md#the-indexable_metadata-entity . ATM we use it only for run and echo. Indeed the value of <index> is not yet clearly defined/controlled in BIDS otherwise.

[yarikoptic]

The use of part- in MEG requires it to be ...

That was my question really in above comment -- is it required? For clarity: is the only tool in question (mne-python) relies on that index, or users would just point to the first file in the sequence and then internal pointer to the next file is used? I saw use of _construct_bids_filename (the only place I found _part- mentioned) only within write_data but have not spotted it elsewhere (somewhere in reading).

[robertoostenveld]

Internal pointing can IMHO not be relied on since for most users it will not be possible (i.e. they do not have the tools) to update these internal pointers when they rename the file. AFAIK the only tool that can be used for updating these internal pointers is MNE-Python. I do not recall ever seeing such a tool from the vendor itself.
Putting the responsibility with the user to provide the files in the right sequence (as we e.g. do in FIeldTrip) would always be possible.

[sappelhoff]

<index> is coming up frequently here and this reminded my of @yarikoptic's #152 ... perhaps it may be a good time to finally resolve the issues around that PR as well and get it in.

[teonbrooks]

LGTM

[effigies]

Looks good. Some minor pedantry.

[sappelhoff]

Merging now as we have a sufficient amount of community and maintainer approval. Robert has approved elsewhere (mne-tools/mne-python#7776 (comment))