Support standard extensions with "$requires"

[handrews]

The most contentious topic in JSON Schema, for many years, has been what sort of re-use functionality should be supported. In particular, how to both use "additionalProperties": false to prevent misspelled property names, and allow adding more property names when re-using such a schema.

This is not actually the intended purpose of "additionalProperties", and as of draft-06 "propertyNames" is preferred, but can be cumbersome. Also, many years of telling people not to use "additionalProperties" in this manner have had no effect whatsoever.

The proposal that attracts both the strongest support and strongest objections is #15 "$merge"/"$patch". It is by far the most powerful, but also breaks the theoretical model of JSON Schema validation as a constraint system in which you can add but never remove constraint. These two keywords allow arbitrary transformations.

We need to balance several things:

• Feature support for people who consider this a barrier to adoption
• Theoretical purity for those who do not need and do not want these features
• Interoperability for both groups, either uniform support or a clear fail-fast outcome

The simplest way to implement this is to allow declaring that your schema requires an optional feature. The implementations can either process your schema, or error out with a message that the feature is not supported.

This is only needed for extensions to core, such as "$merge" and "$patch". Unlike "format", which is still of use to applications even when not used for validation, and does not change the behavior of anything else when it is ignored, there is no possible way to use a schema involving "$merge" or "$patch" without supporting those keywords.

---

So I am proposing adding "$requires" to core as an extensibility mechanism. The value of this keyword MUST be an array. Elements in the array MUST be URIs identifying the extensions required to properly make use of the schema.

Implementations MUST error out if they do not support, or do not recognize, a required extension.
Implementations MUST NOT automatically attempt to dereference the URI, as such URIs need not be dereferencable at all, and the canonical server should not be overloaded by unnecessary requests.

The initial declared extensions would be something like http://json-schema.org/draft-07/extensions.$merge or tag:json-schema.org,2017-10-15:extensions.$patch depending on whether we want to stick with HTTP URIs or use an obviously not-dereferenceable scheme such as tag.

"$merge" and "$patch" would be separately declared extensions, as different users prefer different approaches, needing one or the other or both.

Also paging @erayd

[erayd]

A few thoughts - posting this from my phone, so briefly:

1. Can we call it $features rather than $require? That feels more semantically accurate, and prevents confusion with languages / frameworks that use 'require' as an inclusion keyword.

2. If we're using a URI to refer to a feature, that URI could also point to a meta-schema that validates use of the feature - essentially the same role that $schema serves for the core stuff.

3. If this proposal goes ahead, it gives custom keywords a proper home. Noting this, can we then go ahead and ban undefined keywords completely? I.e. a keyword must be defined by the core or an enabled feature in order to be valid, and unknown keywords result in a validation fail.

[erayd]

To further clarify my point (2) above, I'm not suggesting that implementations dereference this, merely that defining the URI as the canonical home of a meta-schema that validates the referenced feature makes sense, and allows used to easily locate the definition for any given feature.

[Anthropic]

@erayd re: 3, I think failing on undefined keywords would be a huge pain without a prefix based comment exclusion. I always test schema by slashing keywords "//keyword" to reduce what I am testing, it would be an unwelcomed outcome to not be able to easily exclude large parts of a schema when testing due to validation errors I don't want.

[erayd]

@Anthropic
That's a really good point. What if the default was to ban undefined keywords, but there was also a mechanism to turn that constraint off?

[Anthropic]

I'd prefer a definable prefix exclusion, but that's a whole other discussion already going on in the $comment thread I've been meaning to get back to :)

As for this I agree the name doesn't feel right to me, I'd prefer something like $keywords, $uses or $requiredSpecs. Not set on anything clearly, but $requires is too close to required for my comfort.

[erayd]

$uses feels like a good name to me.

I have no objection to a definable prefix exclusion; it's probably not something I'd use, but I can see the value in it.

[handrews]

If this proposal goes ahead, it gives custom keywords a proper home. Noting this, can we then go ahead and ban undefined keywords completely?

Absolutely not. A core IETF principle: be conservative in what you send, and liberal in what you accept. Silently ignoring unknown keywords is a critical aspect of JSON Schema that will not change. It works beautifully for everything except keywords that so thoroughly break JSON Schema's theoretical model that it is not possible to meaningfully ignore them.

There is already an issue (or several) to address less dramatic customizations and additional meta-schema flexibility: #314 being the main one. We should continue to consider those issues for the next draft.

Part of the point of this proposal is that it sandboxes controversial proposals that do not fit the theoretical model but simply will not go away and are barriers to adoption. It also gives us away to officially experiment with things without burdening implementations, and with an easy way to drop them if we do not want to put the organization's stamp on them.

It is very much on purpose that this keyword is only used to flag support for functionality that either MUST be supported, or the schema MUST be considered unusable. Extensions that do not require fail-fast should continue to be implemented as they have been.

Once we have more feedback on this approach, plus more thought on #314, we will consider how to proceed further.

---

This is a huge change and the only way I will support it is if it is fenced off with a high barrier to entry and (nearly) zero impact on those who do not support it (except to error out if it is present and not recognized.

[erayd]

@handrews
Fair enough. I normally like things that are as strict as possible when it comes to validation, but your arguments make a lot of sense, and I concede the point.

What's your opinion on validator implementations possibly providing an optional (defaults to off) strict-mode, outside the spec?

[handrews]

What's your opinion on validator implementations possibly providing an optional (defaults to off) strict-mode, outside the spec?

It's been proposed many times and is mentioned in the "don't do anything" issue in the vote-a-rama. Outside the spec is outside the spec, and I'm in favor of strict debug modes and stuff like that. I may implement a debug mode that checks such things when additionalProperties is false, but can be locally disabled by explicitly setting additionalProperties to true. But there is no need to put that in the spec.

[handrews]

@erayd also, that sort of thing can be implemented with a pre-processor that just goes and adds or removes the keyword, if you want to be able to publish both versions. merge/patch requires lazy evaluation and therefore cannot be pre-processed out.

It's actually quiet hard to construct a keyword that breaks the model enough to need to be a flaggable extension.

[erayd]

After further thought, allowing a schema to self-declare as strict seems like a very valuable feature to have, and would prevent silent validation passes in the case of a validator not implementing all the keywords in the schema.

Would you be open to a new issue that proposes a mechanism for that (e.g. a $strict keyword or declared feature extension), or is there already a consensus that this is unwanted (or an existing issue)?

[handrews]

Would you be open to a new issue that proposes a mechanism for that (e.g. a $strict keyword or declared feature extension), or is there already a consensus that this is unwanted (or an existing issue)?

It has never gathered support, and I would not want to put it in draft-07.

To be clear, it is not a given that this is going in draft-07, as most project members have not weighed in. $strict would not solve "problems" that people demand be solved fully. If we have to go in this direction, I want to do the absolute minimum necessary to move forward.

There are two sharply divided camps: those who think the theoretical model is fine, and work within it, and those who want to monkeypatch anything and everything. The approaches cannot be reconciled, and I'm not sure either camp on its own is big enough to give JSON Schema critical mass. That's the only reason I'm considering this at all.

To be clear, I am not the only person who's opinion matters on this, but no one else is really pushing this forward. One of the other project members may take it on, though, given how strong feelings are on this issue.

[erayd]

It has never gathered support, and I would not want to put it in draft-07.

Then I'll leave that particular can of worms alone for now.

[Relequestual]

@handrews Do you consider this something that would make it to RFC proper, or just till we work out what does and doesn't get used between now and then?

If you're thinking it would be useful to allow implementors to officially optionally support more complex or experimental aspects between draft veresions, and later move those extensions into the main spec, then that sounds OK to me.

If you're thinking that his would be part of a finished RFC, then I'm not sure it's a good idea. fragmentation feels like a bad path.

Does that make sense?

[handrews]

@Relequestual yes, it makes sense, and I don't care whether it makes it to RFC or not right now, just whether it allows us to move forwards.

If you don't like this proposal, than what do you propose? I am dead-set against putting $merge or $patch into the standard proper. It rips the conceptual model of JSON Schema to shreds, and makes it something I would rather not use. And I know I am not alone in that, even if some of the people who objected last time around don't seem to be actively commenting in the past two weeks (which is not a long period of time, particularly as it's still vacation season).

I honestly think it would be a huge mistake to require all implementations to support those keywords, and I would look at alternate technologies or consider submitting a draft outside of the project in order to have something that has a coherent model behind it.

[handrews]

I'm going to point out that this is, as far as I can tell, exactly the argument that cause the whole JSON Schema project to collapse post-draft-04. I don't see it ending any better this time if the options are purely yes or no to merge/patch. If it's yes or no, then I say no for this draft. Someone can try to sell it for the next draft, and I will keep fighting it, and we can do that for another six months on a proposal that's coming up on five years old now.

[handrews]

On the other hand, if we put it out as an opt-in, and it turns out that nearly everyone opts in and it doesn't cause unexpected problems, then I would be at least somewhat amenable to talking about making it mandatory. I'm trying to find a way out of the years-long stalemate here, and we need a solution, not to just stare at #15 for another year.

[handrews]

A bit more about why I am so vehement on this:

My primary goal for draft-07 is to get what I think will be the first truly viable hyper-schema specification out into the wild. In order to do that, I would like for draft-07 validation to be readily adoptable. if/then/else is a compelling reason for implementations to move to draft-07: far more so than the clean-ups and minor additions of draft-06 (as much as I like propertyNames, I think the demand is not as clear as if/then/else).

If $merge/$patch goes into draft-07 as a requirement, then it will suck all the air out of all discussions except that. A lot of implementors will look at the amount of work and decide not to bother. On the other hand, if we've still done nothing in this problem space, there are schema authors who will sit it out as well.

So far we only have three known implementations of draft-06. We really need draft-07 to be picked up more broadly. The best way I can think of to do that is to ensure that compliance is easy, and experimentation can be done in a way that is part of, rather than working around, this group.

Ajv already supports $merge/$patch and has for years. But there's no interoperable way for people to try it out. Either you guarantee that everyone uses Ajv, or you can't use the extension. There's no real point in lobbying other implementations for support because without interoperability it is of limited use. Although @epoberezkin can probably tell us about usage within his user community- my impression is that it is used reasonably often.

So let's take a baby step forward. Provide an interoperability story, but don't force it on anyone so it's easy to move to draft-07 on the implementation side.

If the majority of schema authors insist on support, then they will take care of lobbying implementation developers. If it picks up momentum across most draft-07 implementations, we should look at making it mandatory. If not, we can decide that extensibility is a good compromise, or we can kick it back out.

But we've been stuck for years on this, with no data beyond Ajv.

Please, let's do something that will get us more data without committing us to the single most controversial decision the project has ever had to deal with, which has nearly killed the entire project at least once.

[handrews]

There is also at least some interest in converging with OpenAPI for their next major version. The differences are currently small, and both sides are pretty dug in on those, so that will be a challenge.

Dumping $merge/$patch in as mandatory is something that I feel would make convergence a great deal more difficult. [or maybe it would be easier? I dunno, perhaps worth investigating, but if we do it this way we get the best of both worlds at least for one draft]

[handrews]

@Relequestual, I would be amenable to locking this feature down more so that it is very specific to giving specific controversial features trial runs during the draft process. Reading over reactions, I am probably not comfortable putting a general extensibility feature into draft-07 with just a month and a half of thought. So maybe even for now the syntax is something like "$useMerge": true and/or "$usePatch": true

We can let the extensibility idea sit for another six months to gather comments and work through thoughts, and also see how the limited extensions work in the wild. And then revisit the overall concept for draft-08 (or if there has not been enough adoption of draft-07 yet to get the feedback we need, a later draft than that).

[erayd]

@handrews
More feedback certainly makes sense if you're wanting to get draft-07 out quickly. It sounds like you feel very strongly about this, and there is problematic history with the issue overall, so rushing this probably isn't wise.

Introducing more keywords that are specific to enabling merge / $patch feels like a bad idea though - if going down that track, doesn't it make more sense to simply say that implementations MAY support those keywords, but MUST fail validation if they are present and the implementation does not support them? Seems a lot cleaner than introducing extra toggle keywords when the presence of $merge or $patch in the schema already conveys the same information.

[handrews]

@erayd that's a good point. If it's not a general extension mechanism, why have one at all?

The only thing I can think of is that it makes the nature of the support very clear to have to declare that you require it. I don't think I'm quite articulating what I mean here, I'll have to think on it. But I could see just documenting that the keywords MUST fail-fast with these specific keywords, which is part of giving them a trial run.

While the extensibility feature would be rushed for draft-07, adding $merge/$patch support wouldn't really, seeing as they were first proposed in March of 2014 (at the old repo).

To say that there is "problematic history" is an understatement.

[erayd]

@handrews

If it's not a general extension mechanism, why have one at all?

Indeed. I'd love a general extension mechanism, simply because it feels neater - i.e. it gives a logical home to keywords that aren't part of the official spec - but I don't want to push for it strongly if the concensus is that one isn't desired; IMO reaching an agreement is more important.

The only thing I can think of is that it makes the nature of the support very clear to have to declare that you require it.

What about the following?

• $merge and $patch make it into the spec for draft-07, but are explicitly noted as experimental / may vanish in a later version / may bring about the early heat-death of the universe etc.
• Any keyword that the spec lists as experimental (e.g. $merge / $patch) MAY be supported, but MUST fail validation if unsupported and present in a schema.
• Any schema wishing to use experimental keywords must explicitly opt-in to doing so by declaring "$experimental": true.

Does that sound like an acceptable compromise?

[handrews]

@erayd I like that, let's see what @Relequestual thinks.

There are still several other major project members that have not recently weighed in on seriously moving forward with $merge/$patch (whether it's experimental or not) and I want to wait for them to reach a consensus.

[erayd]

There are still several other major project members that have not recently weighed in on seriously moving forward with $merge/$patch (whether it's experimental or not) and I want to wait for them to reach a consensus.

I agree, waiting for more input before proceeding is a good thing.

[Relequestual]

I don't care whether it makes it to RFC or not right now, just whether it allows us to move forwards.

I dissagree with this sentiment, but I can understand why you might feel this way. However...

On the other hand, if we put it out as an opt-in, and it turns out that nearly everyone opts in and it doesn't cause unexpected problems, then I would be at least somewhat amenable to talking about making it mandatory. I'm trying to find a way out of the years-long stalemate here, and we need a solution, not to just stare at #15 for another year.

... I totaly agree with this, and feel that...

Introducing more keywords that are specific to enabling merge / $patch feels like a bad idea though - if going down that track, doesn't it make more sense to simply say that implementations MAY support those keywords, but MUST fail validation if they are present and the implementation does not support them?

... sounds like the right solution.

To paraphrase to make sure we're on the same page and I'm not missunderstanding here...

Having experimental key words which MAY be supported but MUST cause validation failure for a schema that includes them, if not supported by the validator, sounds like an ideal solution moving forward. Testing the water, and evaluating uptake from implementors and json schema authors feels like a great way to validate the use case, without forcing compliance and without causing unnesessery unrest!

I'm always happier to validate the requirement to make key changes if it's going to really upset thigs. This is a solution I'm happy with.

I think forcing people to use "experimental: true" would make them realise that it may not be supported, and to look into the spec arounf these key words closer. Experimental sounds better than Requires, personally, but that's just a name for the key word.

---

I am dead-set against putting $merge or $patch into the standard proper. It rips the conceptual model of JSON Schema to shreds, and makes it something I would rather not use. And I know I am not alone in that,...

If you're dead set against it, are you still happy to trial them as experimental keywords? If they get uptake, and it seems people want this and it doesn't cause problems, would you then consider alternative technology and leave the project?

Maybe it's because I haven't used JSON Schema enough or written written a validator, but I don't understand what the conceptual model is, why it's important, or why we should care if that changes to make the specification more useable. I'm not suggesting we discuss that specifics here (maybe slack sometime), just pointing out my opinions may be less valid given my lack of understanding. Also, I can't be the only one to think this way.

[dlax]

Experimental sounds better than Requires, personally, but that's just a name for the key word.

To me, $requires clearly indicates that there's extension required. But since having a "generic" extension mechanism is not considered for draft-07, this keyword would be misleading.

By the way, I think that @erayd's proposal #388 (comment) would be a nice and sane step forwards. And I guess that if the feature gets popular enough, it could then be moved from the experimental side to an extension, once an extension mechanism gets defined.

[Relequestual]

I'm really against having an "extensions mechanism" unless it's only to work out which key words are useful while the spec is still in draft. Mainly because I want to totally avoid the chance of fragmentation (although you could argue the draft versions already create this situation).

As for $requires vs something else... I'm happy with $requires if others feel it fits.

For now, I'll say let's +1 this (the issues proposal) forward. Would the URI's be expected to resolve, and if so, what to, and for what purpose? @handrews ?

[handrews]

I don't care whether it makes it to RFC or not right now, just whether it allows us to move forwards.

I dissagree with this sentiment, but I can understand why you might feel this way.

I suppose that came across as more snarky than intended :-P

I just mean that the point of the draft process is to try things and see whether they are worth proceeding to RFC or not. Putting something in a draft doesn't mean it will make RFC, nor should it. We've ripped out several things since draft-04, particularly in hyper-schema.

I am dead-set against putting $merge or $patch into the standard proper. It rips the conceptual model of JSON Schema to shreds, and makes it something I would rather not use. And I know I am not alone in that,...

If you're dead set against it, are you still happy to trial them as experimental keywords?

JSON Schema is not my personal project.

That's all there is to it.

No, I'm not happy to trial them. I don't need a "solution" to this whole thing that I don't consider a "problem" in the first place. But.... JSON Schema is not my personal project. I think we have an obligation to listen to the community. Maybe draft-07 is not the right time. Maybe we should wrap up all of the format, hyper-schema, and application/schema-instance+json work as draft-07, and have a full six months of debate just on how to move this forward.

But either way, my personal distaste for this is irrelevant. I reserve the right to be a grump about it (up to a point, anyway).

If they get uptake, and it seems people want this and it doesn't cause problems, would you then consider alternative technology and leave the project?

Probably not.

It's remotely possible, but if the keywords really don't cause problems, then I won't have much real cause for objection and would have to somehow sell my colleagues on switching technologies. That would be a high bar. FWIW, I have previously left projects that considered me essential in some way and they kept going just fine. No one person is really that important; someone else would step up.

Maybe it's because I haven't used JSON Schema enough or written written a validator, but I don't understand what the conceptual model is, why it's important, or why we should care if that changes to make the specification more useable.

It's the sort of thing that I care a lot about and other people often don't. I like to work in abstractions and applying them. I dislike synthesizing from examples. Most people work the other way around. The way I approach things is neither superior or inferior, it's just a different perspective. It helps with some things and gets in the way of others. You can decide which is going on here :-)

---

It sounds like @erayd's $experimental option is the most broadly supported. I like it as well.

Let's think a bit more on whether this is the right draft to go ahead with here, or if, since we have avoided really resolving this for years, perhaps a few months of rigorous debate focused on producing a solution (I argue we have just debated aimlessly to date) would make us all happier with the path forward.

[handrews]

Note that if we decide to go a few more months on this, we could let it be the sole driver of draft-08 which could go out as little as 2 months after draft-07. We don't have to wait six months.

[Relequestual]

I suppose that came across as more snarky than intended :-P

ha fair enough.

If you're dead set against it, are you still happy to trial them as experimental keywords?

JSON Schema is not my personal project.

That's all there is to it.

No, I'm not happy to trial them. I don't need a "solution" to this whole thing that I don't consider a "problem" in the first place. But.... JSON Schema is not my personal project. I think we have an obligation to listen to the community.

When I say "you" in that context, I meant as an editor of the specification, not as a consumer or implementor of it in your daily work. But I think you clarified that if the community wants it (which I think it is), then we should TRY work it into a draft spec and see if it causes issues or not.

If we do add it (and I think we're in agreement that we should try in maybe draft-8), then you're on under no obligation to use it in your schemas. And it's not like (from what I understand) you're day job involves consuming schemas created by other people.

But either way, my personal distaste for this is irrelevant....

Yeah, I think that's what I've tried to clarify with the above.

I reserve the right to be a grump about it (up to a point, anyway).

Duly noted =D

---

It's the sort of thing that I care a lot about and other people often don't. I like to work in abstractions and applying them. I dislike synthesizing from examples. Most people work the other way around. The way I approach things is neither superior or inferior, it's just a different perspective. It helps with some things and gets in the way of others. You can decide which is going on here :-)

I'm totally open to hearing about it! I was going to ask if you'd write a (short, brefi) wiki page on the conceptual model what it is, why it's important, how it should / can be applied to thinking about making changes to the spec. I'm happy to write my questions as headers as a wiki page to get you started, if you're up for doing it. (I bet it would help me understand and maybe others too). Of course if you want to wait till after draft-7 (or just would rather not), that's fine also.

Note that if we decide to go a few more months on this, we could let it be the sole driver of draft-08 which could go out as little as 2 months after draft-07. We don't have to wait six months.

Yeah, I totally agree on this one! =]

[erayd]

Note that if we decide to go a few more months on this, we could let it be the sole driver of draft-08 which could go out as little as 2 months after draft-07. We don't have to wait six months.

Noting how contentious the whole issue around $merge / $patch et al has been, this feels like a really good idea. Making it the sole focus for draft-08 removes distractions and hopefully means we'll get a better concensus on what the best path to proceed might be. It also means that there is more opportunity for discussion rather than trying to rush something into draft-07.

[handrews]

As explained in the email to the mailing list, I'm going to close this as the generic extension mechanism isn't really motivated by use cases, and we're deferring the merge/patch decision to the next draft. If we still want this general sort of approach then, we will open a new issue focusing on the $experimental form of this proposal which attracted more support. But we don't have any other use cases for it so I'm not going to open it immediately. Only if we need it.