[SCHEMA] Add patterns for format validation #885

tsalo · 2021-09-24T18:10:21Z

Closes #832.

This currently doesn't include any code that would use the patterns, except for tests.

Changes proposed:

Add formats.yaml file to schema, with regular expressions with which to validate different entity values, metadata types, and metadata string formats. Individual metadata fields should still be allowed to supply a specific "pattern" keyword.

To do:

write tests (post-[SCHEMA] Reorganize schema code into a package #892)
add rrid pattern (for [SCHEMA] Add TSV column files #827 and the existing SoftwareRRID)
review the URI and path patterns

src/schema/formats.yaml

tsalo · 2021-09-24T18:13:10Z

src/schema/formats.yaml

+dataset_relative:
+  description: |
+    A path to a file, relative to the dataset folder.
+
+    The validation for this format is minimal.
+  pattern: ^[0-9a-zA-Z/_-\.]+$
+participant_relative:
+  description: |
+    A path to a file, relative to the participant's folder in the dataset.
+
+    The validation for this format is minimal.
+  pattern: ^[0-9a-zA-Z/_-\.]+$
+uri:
+  description: |
+    A uniform resource indicator.
+  pattern: ^[a-zA-Z]+:[0-9a-zA-Z/_-\.]+$
+bids_uri:
+  description: |
+    A BIDS uniform resource indicator.
+  pattern: ^bids:[0-9a-zA-Z/_-\.]+$


The patterns for all four of the path/URI formats should be revised. I don't know enough about the rules to define solid patterns just yet.

The same goes for stimuli_relative (added after the initial comment).

src/schema/formats.yaml

From https://datatracker.ietf.org/doc/html/rfc3986#appendix-B.

Remi-Gau

This will for sure be superuseful. I don't speak regex fluently so not sure I am the best to review this though.

rwblair

Structure makes sense to me. Eyeballing the regex it seems ok, but the best way to verify it is for it to be used by the validator in tests.

tsalo · 2021-10-05T15:52:09Z

Thanks for taking a look. I'm thinking that I should try writing some unit tests for the schema code, which will necessitate reorganizing it into a proper package. I will open a PR to do that after #883 is merged, so that there aren't too many merge conflicts.

EDIT: I've opened #892.

sappelhoff · 2021-10-06T09:31:27Z

src/schema/formats.yaml

+  description: |
+    The basic string type (not a specific format).
+    This should allow any free-form string *except* "n/a".
+  pattern: ^(?!(n/a)$).*$


This is related to #876

This should allow any free-form string except "n/a".

I am not sure whether disallowing n/a as a string for free-form text fields would break some datasets. For example for the required EEGReference field, users could currently pass n/a according to bids-validator:

https://github.com/bids-standard/bids-validator/blob/75c54e4f7a17bf58c7cfc47c9698eb4e0841315b/bids-validator/validators/json/schemas/eeg.json#L25

Whether or not that's a good idea should be discussed in #876, this is just a note that what you introduce here has not been discussed and has not been formally part of the spec 🤔

My hope is that any cases where "n/a" is acceptable will have "n/a" as an explicit option within the schema. I've tried to add it as an option in all of the cases that I've seen like that, but that doesn't mean we're close to having complete coverage. I can drop any fancy "n/a" handling from this PR if you'd rather wait to tackle it within #876 though.

I've dropped the special "n/a" handling in dfd3057, but I do think we will want to distinguish between "n/a"s and freeform strings in the long run.

I was so proud of it too 😭

yarikoptic · 2021-10-14T16:54:21Z

I was wondering on why formats.yaml and not patterns.yaml if that is what it contains really?

tsalo · 2021-10-14T16:57:43Z

Currently, we have pattern in the schema when there's an idiosyncratic regular expression associated with a single field, e.g.,

bids-specification/src/schema/objects/metadata.yaml

Line 1606 in 638ae5c

pattern: ^(?:2[0-3]|[01][0-9]):[0-5][0-9]:[0-5][0-9]$

Whereas common "patterns" are consolidated as "formats." I guess you could say that this file defines the "patterns" associated with the "formats", but I feel like naming it patterns.yaml might be confusing since the existing "patterns" in the schema are not included in this file.

yarikoptic · 2021-11-22T16:57:42Z

I guess the next step for this pr is adding some tests as @tsalo was planning todo (and #833 is merged) or that would be postponed till later?

Co-authored-by: Chris Markiewicz <effigies@gmail.com>

…ation into format-rules

codecov · 2022-02-01T18:33:45Z

Codecov Report

Merging #885 (4a0ad3f) into master (51e9f1b) will increase coverage by 2.30%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #885      +/-   ##
==========================================
+ Coverage   34.05%   36.35%   +2.30%     
==========================================
  Files           8        8              
  Lines         834      850      +16     
==========================================
+ Hits          284      309      +25     
+ Misses        550      541       -9

Impacted Files	Coverage Δ
tools/schemacode/schemacode/tests/test_schema.py	`100.00% <100.00%> (ø)`
tools/schemacode/schemacode/_version.py	`38.90% <0.00%> (+2.59%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e53a367...4a0ad3f. Read the comment docs.

yarikoptic

I just love testing boundary conditions I guess ;)

src/schema/objects/formats.yaml

tools/schemacode/schemacode/tests/test_schema.py

@yarikoptic

Thanks for the suggestions @yarikoptic. They're great! Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

tools/schemacode/schemacode/tests/test_schema.py

tsalo · 2022-02-02T19:45:57Z

It looks like ftp:// passes validation for the uri format, unfortunately. Not sure how to update the regex appropriately.

EDIT: URIs can't really be validated well with regular expressions, apparently, so I think the best solution would be to do only light testing for the regular expression, but mention in the description that folks should use an appropriate function in practice.

yarikoptic · 2022-02-03T17:22:52Z

It looks like ftp:// passes validation for the uri format, unfortunately. Not sure how to update the regex appropriately.

I can try later to create one following the "Syntax" of https://en.wikipedia.org/wiki/Uniform_Resource_Identifier which would demand to have a non-degenerate authority after //

edit: although not sure if it would come up stringent enough to identify errors. There is also a regex given on https://www.ietf.org/rfc/rfc3986.txt as ^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))? but that one is not stringent enough either:

In [3]: re.match("^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?", "ftp://")
Out[3]: <re.Match object; span=(0, 6), match='ftp://'>

In [4]: re.match("^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?", "ftp://").groups()
Out[4]: ('ftp:', 'ftp', '//', '', '', None, None, None, None)

and making authority non-degenerate shifts // somewhere deeper:

In [5]: re.match("^(([^:/?#]+):)?(//([^/?#]+))?([^?#]*)(\?([^#]*))?(#(.*))?", "ftp://").groups()
Out[5]: ('ftp:', 'ftp', None, None, '//', None, None, None, None)

didn't analyze further. Feel free to just add a note on possible need for improvement or other way to specify extra rules.

tsalo · 2022-03-24T19:45:59Z

Now that I have two approvals, I'm going to merge. I think we can keep improving these patterns (i.e., #999) before the next release.

Draft formats file.

7262fda

tsalo commented Sep 24, 2021

View reviewed changes

src/schema/formats.yaml Outdated Show resolved Hide resolved

Limit time zones to between 3 and 4 characters.

f9a5d15

tsalo commented Sep 24, 2021

View reviewed changes

src/schema/formats.yaml Outdated Show resolved Hide resolved

tsalo added 8 commits September 24, 2021 14:17

Add stimuli_relative format.

0d053d8

Don't allow absolute paths or problematic start folders.

f53d806

Use URI regex.

e6aa4b3

From https://datatracker.ietf.org/doc/html/rfc3986#appendix-B.

Fix(?) BIDS URI regex.

35cf7c2

Update formats.yaml

40fa1d2

Don't allow just zeros in index regex.

00aef74

pattern --> format.

7143ee8

Document the rules for the different formats.

a3fda8e

tsalo marked this pull request as ready for review September 24, 2021 18:59

tsalo requested a review from rwblair September 24, 2021 19:00

Remi-Gau reviewed Oct 5, 2021

View reviewed changes

rwblair approved these changes Oct 5, 2021

View reviewed changes

Merge branch 'bids-standard:master' into format-rules

5b9ccbb

sappelhoff reviewed Oct 6, 2021

View reviewed changes

tsalo mentioned this pull request Oct 7, 2021

[SCHEMA] Reorganize schema code into a package #892

Merged

rwblair mentioned this pull request Oct 8, 2021

overhaul units validation in bids-validator bids-standard/bids-validator#71

Open

tsalo added 3 commits October 14, 2021 12:23

Remove fancy n/a handling.

dfd3057

I was so proud of it too 😭

Move formats file into rules folder.

89b37e6

Actually, the individual formats are really objects.

25944e7

tsalo added 2 commits October 14, 2021 15:06

Add patterns for rrid and time.

3189a88

Be more specific with RRIDs.

a2e56b0

tsalo and others added 8 commits January 5, 2022 13:25

Add more valid pattern tests.

fa6256c

Fix mistake in date/datetime patterns.

b57d891

Make time part of datetimes more restrictive.

a9abc49

Make datetimes more robust.

755285a

Add more bad-pattern checks.

b92aa0f

Apply suggestions from code review

9e2b6cf

Co-authored-by: Chris Markiewicz <effigies@gmail.com>

Merge branch 'format-rules' of https://github.com/tsalo/bids-specific…

245e149

…ation into format-rules

Test the new pattern changes.

e5e3563

tsalo requested a review from yarikoptic January 5, 2022 18:52

sappelhoff modified the milestone: 1.7.0 Feb 1, 2022

Merge branch 'master' into format-rules

9284172

yarikoptic reviewed Feb 2, 2022

View reviewed changes

tsalo and others added 3 commits February 2, 2022 14:30

Apply suggestions from code review

aea7982

Thanks for the suggestions @yarikoptic. They're great! Co-authored-by: Yaroslav Halchenko <debian@onerussian.com>

Reorder formats alphabetically within each section.

84061cc

Run black on test and improve check.

df0dd34

yarikoptic reviewed Feb 2, 2022

View reviewed changes

tools/schemacode/schemacode/tests/test_schema.py Show resolved Hide resolved

Comment out failing pattern, but add a note about it.

4a0ad3f

tsalo mentioned this pull request Feb 3, 2022

Improve/correct unit validation in schema #999

Open

sappelhoff approved these changes Mar 22, 2022

View reviewed changes

tsalo merged commit 9dbab0e into bids-standard:master Mar 24, 2022

tsalo deleted the format-rules branch March 24, 2022 19:46

tsalo added the schema Issues related to the YAML schema representation of the specification. Patch version release. label Apr 11, 2022

effigies added the exclude-from-changelog This item will not feature in the automatically generated changelog label Jul 27, 2022

tsalo mentioned this pull request Aug 18, 2022

[SCHEMA] bep012 functional derivatives #1209

Open

effigies mentioned this pull request May 1, 2023

[FIX] Revert index (entity format) definition to be a non-negative number, permitting zero #1482

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SCHEMA] Add patterns for format validation #885

[SCHEMA] Add patterns for format validation #885

tsalo commented Sep 24, 2021 •

edited

Loading

tsalo Sep 24, 2021

tsalo Sep 24, 2021

Remi-Gau left a comment

rwblair left a comment

tsalo commented Oct 5, 2021 •

edited

Loading

sappelhoff Oct 6, 2021

tsalo Oct 7, 2021

tsalo Oct 14, 2021

yarikoptic commented Oct 14, 2021

tsalo commented Oct 14, 2021

yarikoptic commented Nov 22, 2021

codecov bot commented Feb 1, 2022 •

edited

Loading

yarikoptic left a comment

tsalo commented Feb 2, 2022 •

edited

Loading

yarikoptic commented Feb 3, 2022 •

edited

Loading

tsalo commented Mar 24, 2022

[SCHEMA] Add patterns for format validation #885

[SCHEMA] Add patterns for format validation #885

Conversation

tsalo commented Sep 24, 2021 • edited Loading

tsalo Sep 24, 2021

Choose a reason for hiding this comment

tsalo Sep 24, 2021

Choose a reason for hiding this comment

Remi-Gau left a comment

Choose a reason for hiding this comment

rwblair left a comment

Choose a reason for hiding this comment

tsalo commented Oct 5, 2021 • edited Loading

sappelhoff Oct 6, 2021

Choose a reason for hiding this comment

tsalo Oct 7, 2021

Choose a reason for hiding this comment

tsalo Oct 14, 2021

Choose a reason for hiding this comment

yarikoptic commented Oct 14, 2021

tsalo commented Oct 14, 2021

yarikoptic commented Nov 22, 2021

codecov bot commented Feb 1, 2022 • edited Loading

Codecov Report

yarikoptic left a comment

Choose a reason for hiding this comment

tsalo commented Feb 2, 2022 • edited Loading

yarikoptic commented Feb 3, 2022 • edited Loading

tsalo commented Mar 24, 2022

tsalo commented Sep 24, 2021 •

edited

Loading

tsalo commented Oct 5, 2021 •

edited

Loading

codecov bot commented Feb 1, 2022 •

edited

Loading

tsalo commented Feb 2, 2022 •

edited

Loading

yarikoptic commented Feb 3, 2022 •

edited

Loading