Update "format" and "content*" for new JSON Schema

[handrews]

Closes #2093. Probably also closes #1547, #2191, #2192.

This removes OAS formats and examples that are now superfluous
as they are part of the 2019-09 JSON Schema draft, and notes
that "format" is not validated by default.

Similarly it deprecates the "byte" and "binary" formats in favor
of JSON Schema's "contentEncoding" and "contentMediaType" keywords,
and updates various related examples and other guidance.

Note that I had to get a bit creative interpreting how some of the
"content*" keywords might change things, and may have gone
in the wrong direction, so please review those parts carefully.

It also removes confusingly blank rows in the OAS format table.

[jcarres-mdsol]

In the company I work for we use UUIDs to identify resources so type: string format: uuid is super common in our schemas. And we do leverage that information for a number of things.
It seems that this note is motivated by the current implementations in the wild, which does not seem like a good reason to include it to a standard. If these validators start improving would this note go away? At what point? How many good validators out there would it make it happen?

[handrews]

@jcarres-mdsol uuid and all other formats that are no longer explicitly listed are now part of JSON Schema proper. See Section 7.3.5 "Resource Identifiers" under the format keyword's specification for uuid and also how it relates to URNs of the form urn:uuid:.

[handrews]

@jcarres-mdsol As for validators actually validating format, no, that's not going to happen. I don't mean that in a "we (the JSON Schema project) decided it shouldn't happen", I mean it in "this has been a problem for years, and after talking with numerous maintainers of major implementations, it is 100% clear that format is a confusing mess that will not be validated consistently no matter how much the spec says it should be." At some point you have to bow to reality and figure out a way forward, and saying that by default implementations MUST NOT attempt to validate using the format keyword was the only possible way to produce consistent default behavior. This is explained in detail in the JSON Schema draft's release notes.

The problem with format is that it tries to do a huge number of things, and be open-ended to any extension anyone wants to throw in there. There's not even agreement on what fully validating format would involve. My hope is that with re-usable extension vocabularies being a thing, people will design vocabularies with keywords that validate specific things, instead of shoving them into this all-purpose-but-never-reliable keyword.

So an isUuid keyword would be straightforward to implement b/c UUIDs are well-specified. For something like email, where there's no consensus on what "valid email" means, people could create keywords with specific criteria. isProbablyEmail might just check for an @ in the string (this is what several validators actually do right now). isEmailByRegex might specify validation against a specific regex (instead of what happens now, which is that each validator comes up with its own incredibly complex regex that they probably got from a web site somewhere). isEmailBySomeLibraryName would define what library to use to validate, etc.

These keywords would then be supported or not, which could be detected/enforced using the $vocabulary mechanism, and would reliably provide whatever level of validation they specify. You could require support for isUuid but not require any sort of email validator, for example- that would be likely to be supported. As would keywords for RFC 3339 date and time values, which are likewise well-specified.

Ultimately, I hope to deprecate format because it's always been a compatibility nightmare. But that will only happen if enough replacement vocabularies are developed and adopted widely enough to be a viable migration path.

[jcarres-mdsol]

Thanks for the detailed response. Hopefully the plan will go as expected. I'm worried about the complexity of all this, I wonder if people will shy away from vocabularies and just keep shoveling everything into format anyways. But time will tell.
Thanks for your efforts!

[handrews]

@jcarres-mdsol we're definitely starting to see people understand the utility of the vocabulary concept on our slack. But really, if enough people accept vocabularies as a solution here, we'll deprecate/remove format. If they don't, we won't. [EDIT: although we may look for other ways to do so]

[MikeRalphson]

Should be include a mediaTypeObject in one of these examples? Is that the preferred OAS way to indicate media-type? What if there is a conflict between the mediaTypeObject key and a contentMediaType keyword?

[handrews]

What if there is a conflict between the mediaTypeObject key and a contentMediaType keyword?

So.... technically I would not use the schema in this example in the first place for a purely binary resource. Because what this means is that there is a JSON string (double-quoted) with binary content, which doesn't actually make any sense. You would have to encode the binary content into a legal JSON string (as indicated by contentEncoding).

For pure binary content, I would just us a Media Type Object without a schema.

Does that make sense? I didn't want to start revamping larger things without first trying the minimal changes, as I assume this approach ({"type": "string", "format": "binary"}) has more or less worked for OAS so far. But I wouldn't actually advise it, personally.

[handrews]

@MikeRalphson @webron should I make the larger change here (to not use a Schema Object for binary content, and just use a Media Type Object)?

[darrelmiller]

Should these simply be removed? What would the impact be on other content-types? If someone is using JSON Schema to infer schema on other non-JSON formats, what does format even mean?

[darrelmiller]

Personally, I think we can remove these.

[darrelmiller]

Should this schema be removed?

[darrelmiller]

Schemas should be removed.

[handrews]

I've updated the first part of the file (and rebased the whole change- you can see the update on its own in the last commit).

I have not yet updated the part around the Encoding Object and the examples because I'm having some trouble figuring out how to get it to work consistently in both the Media Type Object and the Encoding Object scenarios, and with the complex defaulting of the Encoding Object content-type based on various schema characteristics.

I've made my brain hurt enough for today, I'll try to sort it out or at least explain the issues tomorrow.

[darrelmiller]

Yay!