Explain unclear semantics of property $ref in Path Item Object

[cebe]

Currently, as explained in #1038 (comment) the description of $ref in Path Item Object is unclear about the semantics behind it. I took the explaination from issue #1038 to make it more clear.

Hope I got the branching right, if not let me know which branch to send the PR to, I'll fix it then.

[tedepstein]

@cebe, thanks for picking this up where the issue left off. :-)

I have a few suggested changes for precision and alignment with standard terminology:

---

Allows for an external definition of this path item. A Path Item Object containing a $ref property is not a Reference Object, but the semantics of the $ref property value are the same. The $ref value MUST be a URI that resolves to a Path Item Object. The properties of the referenced object are merged into the referring Path Item Object. If the same property exists in both the referenced object and the referring object, this is a conflict. In this case, the behavior is undefined. Applications that do not define their own resolution behavior SHOULD report such conflicts as a warning or error condition.

---

Notes:

• I had some hesitation about saying that the referenced object MUST be a Path Item Object, because it could be a partial object, meaning it doesn't have all of the required properties of a valid Path Item Object. But in fact, Path Item Object doesn't have any required values. So we can say that the $ref URI MUST resolve to a Path Item Object, meaning any object whose properties and property values are allowed in a Path Item Object. This seemed to be the cleanest way to state it.
• I added the last sentence as an incremental step towards better defining validation behavior. This is a core part of conformance testing, discussed in Formalize conformance test description format #1520 and Certification program for OAS consuming implementations Outreach#35. The OpenAPI spec uses the term "undefined" in a few places, but this is the only instance of a property definition with an explicitly undefined value space. I assume that "undefined" allows applications to define their own behavior, whereas a pure error condition would not. So I'm trying to clarify how consuming applications should behave in these cases:
  • Either define your own interpretation of the undefined value space, or report it as an error or warning.
  • The choice of error or warning severity depends on the consuming application. If the consumer is a validator or linter, a warning might be the most appropriate default severity, because you don't want to block a downstream code generator or documentation format that actually defines useful behavior for these cases. If the consuming application is a code generator or documentation format that does not provide its own interpretation of the undefined value space, error severity might be more appropriate.

[cebe]

Thanks for the detailed feedback!

I agree with the change about referencing a Path Item Object, but:

Either define your own interpretation of the undefined value space

I'd like to avoid undefined behavior and would prefer to mark the situation as an error condition explicitly. Allowing different tools to handle the situation differently will definitively cause some trouble. Making it easier to implement the spec without ambiguation will prevent tooling implementors and also users from getting into weird situations.

[tedepstein]

Either define your own interpretation of the undefined value space

I'd like to avoid undefined behavior and would prefer to mark the situation as an error condition explicitly. Allowing different tools to handle the situation differently will definitively cause some trouble. Making it easier to implement the spec without ambiguation will prevent tooling implementors and also users from getting into weird situations.

I agree, but I assume there is some historical reason why the spec says "undefined" instead of making it an error condition. I think we need some clarification from the TSC.

[philsturgeon]

If something changes from "undefined" to "definite nope" then it should probably go in 3.1.0, because its kinda maybe sorta breaking but only if you're relying on undefined behavior which is kinda your own fault? Seems like the wiggle room is there to make it happen in v3.1.

[cebe]

Note that my adjustment does not change the definition, so the adjustment made here can go into 3.0, we could then discuss a change of defined behavior for 3.1.

[tedepstein]

Note that my adjustment does not change the definition, so the adjustment made here can go into 3.0, we could then discuss a change of defined behavior for 3.1.

Right. What we've tried to do here is retain "undefined" so as not to introduce a breaking change, but add a bit of guidance as to how implementations should handle this undefined case. I think that could go into a 3.0.x patch release without any issues.

Disallowing duplicate properties (or conflicting, or overlapping properties) would be a breaking change, so should probably go into 4.0 rather than 3.1. And before we suggest this breaking change, I'm hoping the TSC, or someone, can provide the history of this, explaining why the behavior was left undefined. Maybe there's a good reason for it, and in that case, we might want to define the resolution rules rather than disallowing the duplicate properties.

[tedepstein]

@cebe , there might be some confusion because your PR doesn't incorporate my suggested change here. If you like the suggested revisions, you might want to commit those to your cebe:patch-1 branch so we're all talking about the same thing. :-)

[MikeRalphson]

@tedepstein I would prefer to drop your final proposed sentence:

Applications that do not define their own resolution behavior SHOULD report such conflicts as a warning or error condition.

As declaring the behaviour as undefined and then defining what applications (implementations) SHOULD do appears to be in conflict. From a purist point of view, I would like the specification to declare its own requirements, but never impose behaviour (such as reporting errors or warnings) on implementations.

[tedepstein]

@MikeRalphson , I'm OK with removing that last sentence.

Please tell me if "undefined" means the same thing to you as it does to me:

• We're acknowledging that the OpenAPI spec has consciously declined to specify a meaning or prescribe behavior in this case.
• We could choose to disallow these conflicts, or specify semantics (e.g. properties defined in the referring context take precedence) in a later release. And we could even argue that the newly defined semantics are an addition, not a breaking change, because we've already blocked off this space as undefined.

Common-sense guidance, implied but not stated:

• It's safest to avoid this undefined value space in your OpenAPI documents.
• Applications may define behaviors, and users may rely on those behaviors, at their own risk. We won't stop you from doing your own thing in an undefined value space, as long as you understand that your special semantics might be clobbered by a subsequent spec release.

Is that right?

[MikeRalphson]

Please tell me if "undefined" means the same thing to you as it does to me:

@tedepstein I couldn't swear that your definition covers every case of undefined behaviour, but I don't see anything to disagree with in what you have there.

[handrews]

@tedepstein @earth2marsh this gets into the objections @whitlockjc has around three different $ref behaviors in OAS 3.1, which @webron and I were going to take a look at.

In order to make 3.0.3 compatible with the most likely approach in 3.1, I would recommend slightly different wording. I find "merge" misleading here, because it only works one level down and become undefined if there's s conflict. Which I guess is technically a merge but one that only works in a small set of cases.

As I think @MikeRalphson noted on the call, we should look at this a bit more like JSON schema's annotations in terms of defining behavior. Possible wording for that might replace

The properties of the referenced structure are merged with the local Path Item Object.
If the same property exists in both, the referenced structure and the local one, this is a conflict. In this case the behavior is undefined.

to (changed words in bold italics):

The properties of the referenced structure are examined alongside those in the local Path Item Object.
If the same property exists in both, the referenced structure and the local one, this is a conflict. In this case the behavior is undefined.

If there is a concern over "examined" not being clear enough, we could add a third sentence:

The referenced structure's properties that do not exist in the local structure are used as if they were present in the local structure.

or something like that.

This avoids terminology that sounds like it might combine properties in some deeper way, which is what I think of with the word "merge" (and is what the discarded $merge JSON Schema proposal did, so anyone who is familiar with that will probably think of it similarly).

[webron]

Working on this as part of the 3.0.3 release, so we're a bit more constrained in time.

Some history - in 2.0 we added the ability to $ref a Path Item Object within a Path Item Object to allow reusability (putting aside whether it makes sense or not). Two issues were involved here. The first, we did not allow users to define reusable PIOs within an API definition (should be solved in 3.1.0). The second, is that we used the construct of $ref without fully understanding the restrictions of JSON References.

Since we did not realize back then that JSON References do not allow sibling properties, we had no idea that it would be technically not possible to get to the point where there's a conflict. And so, we decided to allow a 'merge-like' behavior, and since we didn't want to deal with the case of duplication, we just said that it's undefined (instead of saying one takes precedence over the other, for example). Little did we know what this would mean for standard JSON Reference implementations out there.

While we did realize the problem before working on 3.0, actually dealing with the issue slipped through the cracks and the same definition remained. These things happen, and it means the behavior stays the same as in 2.0.

As for the change in 3.03 - I'm not a fan of the rewording. I find that specifying it is not the Reference Object is not needed. Nowhere does it say that it is. Claiming that because the field is called $ref people should assume it's the Reference Object is false, because we also have a field type in Security Scheme Object, and nobody confuses it with the Schema Object's type. The spec, in this case, is actually quite concise in the fact that it is not a JSON Reference Object, because the type of the field is not a JSON Reference Object.

I agree there's room to clarify the behavior of the $ref. I would suggest we change
If there are conflicts between the referenced definition and this Path Item's definition, the behavior is undefined.
to
In case a Path Item Object field appears both in the defined object and the referenced object, the. behavior is undefined.

@handrews - as for rewording/resolving things in 3.1, that should come in 3.1 and not here. As long as we don't use the latest JSON Schema draft in 3.0.3, it's better we steer clear of using similar terminology.

[handrews]

@webron my concern is to make sure we don't make things harder to resolve in 3.1.

Currently, the word "merge" does not appear in 3.0.2. If it did, I would not argue to remove it. But I am going to push very hard to avoid adding it. There is no single word involved in JSON Schema or JSON Reference that has been the subject of more fights and assumptions than "merge."

I will be happy with literally any other word. "examine", "combine", "zipper", "splat", seriously anything but "merge." The way people make assumptions around the word "merge" and connect it to past proposals elsewhere will make the change from 3.0.3 to 3.1.0 in this specific area much, much harder.

[webron]

Sure, but the change I recommend making doesn't have the term "merge" in it either. :)

[handrews]

@webron oh I assumed you meant changing

The properties of the referenced structure are merged with the local Path Item Object.
If the same property exists in both, the referenced structure and the local one, this is a conflict. In this case the behavior is undefined.

to

The properties of the referenced structure are merged with the local Path Item Object. In case a Path Item Object field appears both in the defined object and the referenced object, the. behavior is undefined.

leaving the sentence with "merge" intact.

[handrews]

If you want to cut that first sentence and just have

In case a Path Item Object field appears both in the defined object and the referenced object, the. behavior is undefined.

then I am totally fine with that 😁

[webron]

To clarify, I think what I suggested was the only change we might want to do. I find the rest not needed.

[MikeRalphson]

To clarify, I think what I suggested was the only change we might want to do. I find the rest not needed.

👍 I think this is both a positive change and all we need to do for v3.0.3. Hopefully all the $ref language will be brought as far into line as possible in v3.1