Document $ref "traps" for implementors

[handrews]

This material originally written by @fge and further updated by @sam-at-github at https://github.com/json-schema/json-schema/wiki/$ref-traps

Foreword

If you implement JSON Reference support in your JSON Schema implementation, all points below should be considered carefully.

Some of these points are already mentioned by the JSON Reference draft, so if you have read the draft already, this may not be news to you.

Unable to dereference a URI

This can of course happen. For the most obvious protocol out there, http, you may encounter a 404, for instance. Or the URI scheme may not be supported.

It should be noted that the draft does not dictate a specific implementation behavior in this kind of situation. Some implementations may choose to fail, others may not.

JSON References can be chained

It can happen that you have this reference to resolve:

{
    "$ref": "http://one.site/oneschema.json#"
}

and when you resolve that reference, you have yet another JSON reference:

{
    "$ref": "otherschema.json#"
}

In this case, you MUST also resolve that reference, and so on, until you fetch content which is not a JSON Reference.

JSON References can loop

Consider this example:

{
    "a": { "$ref": "#/b" },
    "b": { "$ref": "#/a" }
}

Clearly, an infinite loop may arise. Implementations must detect loops in a chain of references being resolved and raise an exception.

References through other references

Consider this example:

{
    "a": { "milk": "cow", "eggs": "bird" },
    "b": { "$ref": "#/a" },
    "c": { "$ref": "#/b/milk" }
}

(todo: is this situation resolvable in all cases?? Someone provide an algorithm.)

A JSON Reference can resolve to anything

A JSON reference is only a URI, and if you happen to be able to dereference that URI, the content type is undefined.

Which means, you may get a valid JSON document which is not a schema. For instance:

3

Or you may get something which is not even JSON:

Hello!

[Relequestual]

Good points here. There should be a "notes for library makers" page, or some such.

[awwright]

Notes for implementors should go in the spec itself, I think, that's the document targeted at implementations.

Notes for JSON Schema authors should definitely go on the website, and probably also on the spec (probably without requirements language though).

Many of the observations here have already made it in the spec, I've added a note about infinite loops, and I've made $ref a little less powerful (it must be a schema and can only be found where a schema is expected, due to how existing implementations prefer this behavior and how a $ref anywhere impacts the behavior of "enum" and "properties")

[sgpinkus]

... I've made $ref a little less powerful (it must be a schema and can only be found where a schema is expected, due to how existing implementations prefer this behavior and how a $ref anywhere impacts the behavior of "enum" and "properties")

Hi. Is there a new spec for JSON Pointer / JSON Ref?

[awwright]

@sam-at-github There's only RFC6901, reference of which has been updated in the jsonschema-core document. The other references have been removed as I don't see them advancing to RFC, and it's not really necessary to begin with.

[handrews]

@awwright Why wouldn't JSON Reference advance to RFC? I believe it is used outside of JSON Schema.

[awwright]

I don't think it'll advance because I'm not aware of any interest by any editors and a working group to advance it.

All it could do is publish a new design pattern for new media types anyways. It can't change the behavior of "application/json" documents, and it'd make little sense to make it a new media type all by itself.

And after all that effort, and the actual behavior that JSON Schema needs is still very different, how do you deal with JSON instances that contain {"$ref":...} literally?

[handrews]

And after all that effort, and the actual behavior that JSON Schema needs is still very different, how do you deal with JSON instances that contain {"$ref":...} literally?

Do you mean that the inability to handle literal {"$ref":...} makes JSON Reference on its own of very limited use? I can see that- you would need to define some sort of ref+json media type and... um... yeah. There wouldn't be much point as nearly all uses of $ref are already a specific foo+json media type.

So your position is that "$ref" gets defined as part of JSON Schema, because then its use can be specified in a way that does not conflict with a possible need for a literal $ref? If so, then yeah, that's pretty convincing.

[sgpinkus]

@handrews:

Many of the observations here have already made it in the spec,

Where are these observations made? AFAIK there is only the IETF Note. I agree that note is extremely ambiguous as to how the dereferencing algorithm should work. A reference algorithm or implementation is very much needed.

I've made $ref a little less powerful (it must be a schema and can only be found where a schema is expected, due to how existing implementations prefer this behavior and how a $ref anywhere impacts the behavior of "enum" and "properties")

One of the few things it does actually say is that JSON Refs should yield JSON values not objects.

JSON Reference is independent of JSON Schema and used independently of JSON Schems. Example - https://www.npmjs.com/package/json-schema-ref-parser. The spec should be kept separate. It's good design.

[awwright]

Where are these observations made?

See the current jsonschema-core draft at current master: https://github.com/json-schema-org/json-schema-spec/blob/624602461891e98773251e4230c7a6389f9352ba/jsonschema-core.xml

It includes points about avoiding infinite loops, and that you don't necessarily download the $ref (it's just an identifier).

Also, as of current 'master', it's not true that a "$ref" can resolve to anything, only a schema (or maybe another $ref, I don't see any reason why not).

I'll be publishing this as an I-D within a few days (hopefully). It's only a draft, we can decide what to do with it from there.

The spec should be kept separate. It's good design.

Care to elaborate?

[sgpinkus]

Also, as of current 'master', it's not true that a "$ref" can resolve to anything, only a schema

What's current master? Json Schema currently refers to Json Reference Draft:

Resolution of a JSON Reference object SHOULD yield the referenced
JSON value. Implementations MAY choose to replace the reference with
the referenced value.

I don't understand why you would try to artificially place this restriction on the way JSON Reference works, more over from within a different spec. There is no reason $ref cannot resolve to values. In fact it's useful:

{ 
  "data": { "foo": 99 },
  "type": "object",
  "properties": { "bah": { "type": "number", "maximum": { "$ref": "#/data/foo" }
}

[Relequestual]

I believe current master is meant as https://github.com/json-schema-org/json-schema-spec

Regardin the functionality of $ref, what's the reasoning for making it less powerful? In terms of use in JSOn Schema, it makes sense to me that it may replace a partial schema. I think $ref could be used outside of JSON Schema fine, but the linked JSON Reference draft has a lot deeper functionality. Is there no document which just deinfes the use of $ref?

7.2.4 of draft 4 (http://json-schema.org/latest/json-schema-core.html#anchor31) allows for definitions which can be reused. These definitions, as I understand, are not nessecerailly complete schemas in their own right.

[awwright]

@sam-at-github Work on the JSON Schema draft is tracked at https://github.com/json-schema-org/json-schema-spec on the 'master' branch there.

JSON Schemas generally aren't supposed to change, so the usefulness of a non-schema $ref is limited; I'm not aware of anyone actually doing this in practice, and it's not a tested feature in the official test suite.

It also causes problems with writing schemas describing instances that themselves want to use $ref. What sort of behavior does this have?

{ 
  "type": "object",
  "enum": [ { "$ref": "" } ]
}

[awwright]

@Relequestual "inline referencing" is basically just a method of defining a schema in "definitions" and referencing that definition somewhere else within the same document. Properties in "definitions" must still be complete schemas.

[sgpinkus]

JSON Schemas generally aren't supposed to change, so the usefulness of a non-schema $ref is limited; I'm not aware of anyone actually doing this in practice, and it's not a tested feature in the official test suite.

I use it. In a similar manner to the example above - constants.

It also causes problems with writing schemas describing instances that themselves want to use $ref. What sort of behavior does this have?

{
"type": "object",
"enum": [ { "$ref": "" } ]
}

How does the proposed change to JSON Ref solve this again?

[awwright]

@sam-at-github I'm proposing a behavior that falls in line with a lot of current implementations, where you can only use $ref in places where a schema is expected, meaning you can use "$ref" literally in places that expect a literal value (like "enum" and "properties")

Just because this is supposed to be website-related, I'll close this because I think the website will describe this just fine as a part of the other issues open that are developing content for it...

For questions relating to the normative behavior of $ref, I created an issue at json-schema-org/json-schema-spec#66