Example of $recursiveRoot and $recursiveRef #605

handrews · 2018-06-15T03:13:57Z

THIS WILL NOT BE MERGED.

This PR is being used to illustrate a proposal for issue #558.

If the root schema of a schema document contains

"$recursiveRoot": true

then that root schema is set as the target of all "$recursiveRef" references in that document and any to which it refers, regardless of "$recursiveRef"'s value.

Encountering further "$recursiveRoot" keywords in root schemas of referenced documents does not further change the target. This will be explained in detail with comments added to the diff.

"$recursiveRoot" MUST be ignored in subschemas (this is the same behavior as "$schema").

When encountering a "$recursiveRef", if no "$recursiveRoot": true" has been encountered, then "$recursiveRef" is evaluated exactly as if it were "$ref". Its value is a URI reference, which is resolved according to the usual rules involving `"$id".

The key point is that someone reading the schema will know that a "$recursiveRef" might have its target changed dynamically at runtime, while "$ref" never will.

The following changes were made:

hyper-schema.json, and the additional fake example of hyper-operations.json (further extending hyper-schema.json) all have "$recursiveRoot": true
schema.json does not use "$recursiveRoot", as it does not extend anything else
links.json does not use "$recursiveRoot", as it is not recursive
The reference from each extension meta-schema to its "base" is a normal "$ref". Otherwise it would be an infinite loop.
All other schema references become "$recursiveRef", with the same URI Reference value as before
All of the properties and $defs duplicated from schema.json to hyper-schema.json can now be removed

Note that there were several odd trailing "#" fragments, which should not be present in "$id" in particular, so I dropped those. They are not part of this change I just found them surprising.

Also, "propertyNames" had a bug. How does nobody notice this stuff? How do the meta-schemas have an apparently endless stream of bugs in them? UGH.

THIS WILL NOT BE MERGED. This PR is being used to ***illustrate*** a proposal for issue json-schema-org#558. If the root schema of a schema document contains `"$recursiveRoot": true` then that root schema is set for that schema document and any to which it referes as the target of all `"$recursiveRef"` references, regardless of their values. Encountering further `"$recursiveRoot"` keywords in root schemas of referenced documents does **not** further change the target. This will be explained in detail with comments added to the diff. `"$recursiveRoot"` MUST be ignored in subschemas. If no `"$recursiveRoot": true"` has been encountered, then `"$recursiveRef"` is evaluated exactly as if it were `"$ref"`. Its value is a URI reference, which is resolved according to the usual rules involving `"$id". The key point is that someone reading the schema will know that a `"$recursiveRef"` might have its target changed dynamically at runtime, while `"$ref"` never will. ----------- The following changes were made: * schema.json, hyper-schema.json, and the additional example of hyper-operations.json (further extending hyper-schema.json) all have `"$recursiveRoot": true` * links.json does not use `"$recursiveRoot"` * The reference from each extension meta-schema to its "base" is a _normal_ `"$ref"`. Otherwise it would be an infinite loop. * All other schema references become `"$recursiveRef"`, with the same URI Reference value as before * All of the properties and $defs duplicated from schema.json to hyper-schema.json can now be removed Note that there were several odd trailing "#" fragments, which should not be present in `"$id"` in particular, so I dropped those. They are not part of this change I just found them surprising. Also, "propertyNames" had a bug. How does nobody notice this stuff? How do the meta-schemas have an apparently endless stream of bugs in them? UGH.

handrews

Here are some detailed explanations of how this works.

handrews · 2018-06-15T03:15:29Z

hyper-operations.json

+{
+    "$schema": "http://json-schema.org/draft-08/hyper-operations-FAKE#",
+    "$id": "http://json-schema.org/draft-08/hyper-operations-FAKE",
+    "$recursiveRoot": true,


This schema extends another recursive schema, and in this PR example, nothing extends it. So it is always the entry point schema when it is used at all.

So here "$recursiveRoot": true pins the target of all "$recursiveRef"s that are encountered.

handrews · 2018-06-15T03:16:09Z

hyper-operations.json

+    "$recursiveRoot": true,
+
+    "title": "FAKE DEMO JSON Hypermedia Operations FOR RECURSION EXAMPLE",
+    "$ref": "http://json-schema.org/draft-08/hyper-schema",


This is a regular "$ref", as a "$recursiveRef" here would loop back to this root schema and be an infinite loop. A normal "$ref" is used for the extension reference.

Did we resolve the issue of $ref having sibling keywords? (Otherwise, how does this work?)

Looks like there's an informal decision in #523. I guess you're assuming that?

Yes, there's also PR #585 which will hopefully start moving again soon. I've been acting as if it's done because that's definitely what draft-08 will look like. (I'm not sure what you mean by "informal decision", do we even have formal decisions here? Other than merging PRs?)

handrews · 2018-06-15T03:17:03Z

hyper-operations.json

+                    "items": {
+                        "type": "object",
+                        "properties": {
+                            "someKindOfSchema": { "$recursiveRef": "#" }


This "$recursiveRef" is controlled by the "$recursiveRoot": true above, but since (in our example here in the PR), this is a meta-schema that extends others but is never extended, the default target of "#" and the recursive root are the same.

handrews · 2018-06-15T03:21:48Z

hyper-schema.json

@@ -1,66 +1,19 @@
 {
    "$schema": "http://json-schema.org/draft-08/hyper-schema#",
-    "$id": "http://json-schema.org/draft-08/hyper-schema#",
+    "$id": "http://json-schema.org/draft-08/hyper-schema",
+    "$recursiveRoot": true,


Hyper-schema is a recursive meta-schema. It extends the core/validation recursive meta-schema, and is in turn extended by the hyper-operations meta-schema (which is fake and made up for this example). It also references links.json, which is the stand-alone LDO schema, which refers back to this meta-schema.

If hyper-schema.json is the entry point schema document, then this "$recursiveRot": true sets the "$recursiveRef" target for all processing. However, if hyper-operations.json is the entry point, then this "$recursiveRoot" is ignored because hyper-operations.json already set it, and we only arrive at this meta-schema through hyper-operations.json.

A more complex example is if links.json is the entry point schema. That is not a recursive schema, so it does not set "$recursiveRoot". However, it does "$recursiveRef" this meta-schema. When such a "$recursiveRef" (or "$ref") is followed from a non-recursive schema, then this "$recursiveRoot" is the first one seen, and sets the target for all further "$recursiveRef"s.

handrews · 2018-06-15T03:22:24Z

hyper-schema.json

-        }
-    },
-    "allOf": [ { "$ref": "http://json-schema.org/draft-08/schema#" } ],
+    "$ref": "http://json-schema.org/draft-08/schema",


As with hyper-operations.json referencing this schema, we use "$ref" to extend the regular core/validation schema.

handrews · 2018-06-15T03:24:19Z

hyper-schema.json

        "base": {
            "type": "string",
            "format": "uri-template"
        },
        "links": {
            "type": "array",
            "items": {
-                "$ref": "http://json-schema.org/draft-08/links#"
+                "$ref": "http://json-schema.org/draft-08/links"


The reference to links.json is a regular "$ref" as it is not a recursive reference to start with.

However, since at this point we have already set "$recursiveRoot": true (either here or from hyper-operations.json), when we see "$recursiveRef" inside of links.json from here, those "$recursiveRef"s have their targets changed to point to the appropriate "$recursive Root".

handrews · 2018-06-15T04:33:49Z

links.json

@@ -29,7 +29,7 @@
                    "format": "uri-template"
                },
                "hrefSchema": {
-                    "$ref": "http://json-schema.org/draft-08/hyper-schema#"
+                    "$recursiveRef": "http://json-schema.org/draft-08/hyper-schema"


When using links.json as a standalone schema, it is the entry point schema, and for the initial evaluation it does not matter whether this is a "$ref" or a "$recursiveRef". This schema does not set "$recursiveRoot", so the literal URI reference value provided here is used as-is.

When links.json is referenced from hyper-schema.json (with hyper-schema.json as the entry point), then we have seen as "$recursiveRoot", so the literal value is ignored and the target is the root schema of hyper-schema.json

When links.json is referenced from hyper-schema.json, when the entry point was hyper-operations.json, then the "$recusiveRoot" in hyper-operations.json is the first encountered, so the target of this (and other) "$recursiveRef"s is the root schema of hyper-operations.json

These behaviors show how the same links.json can be used on its own (referring to hyper-schema.json which extends schema.json) and as a component within hyper-schema.json (or an extension of it, such as hyper-operations.json).

handrews · 2018-06-15T04:40:57Z

links.json

@@ -55,21 +55,21 @@
                    "type": "string"
                },
                "targetSchema": {
-                    "$ref": "http://json-schema.org/draft-08/hyper-schema#"
+                    "$recursiveRef": "http://json-schema.org/draft-08/hyper-schema"


Since there is no "$recursiveRoot" in links.json when used as an entry point, each time we reference a document with a "$recursiveRoot", the root is set just for that reference and any further references it makes. Then when we move "out" of this "$recursiveReference" and on to the next one for some other LDO property, the process repeats.

This still works fine when we start from links.json, "$recursiveRef" into hyper-schema, and then reference back to links.json again. The sequence is:

Enter links.json (no "$recursiveRoot")

"$recursiveRef" to hyper-schema (use the URI reference directly as there's no root yet)

Enter hyper-schema.json, see "$recursiveRoot" for the first time, set the recursion target

"$ref" to schema.json (extending the core/validation meta-schema)

Enter schema.json, which does not need "$recursiveRoot" (see 2nd review, I changed this from the initial version)

Any "$recursiveRef"s encountered, such as for the applicator keywords, go to the recursion target, which is hyper-schema.json

Back out in hyper-schema.json, "$ref" to links (this is never recursive)

Re-enter links.json, the recursive root is still set from hyper-schema.json

Encounter a "$recursiveRef" to hyper-schema, but now with the recursion target set use that (in this case, it happens to point to the same place)

This continues as we walk over the instance and bounce back and forth between links.json, hyper-schema.json, and schema.json

because it is not extending anything

handrews · 2018-06-15T04:47:44Z

schema.json

    "title": "Core schema meta-schema",
    "$defs": {
        "schemaArray": {
            "type": "array",
            "minItems": 1,
-            "items": { "$ref": "#" }
+            "items": { "$recursiveRef": "#" }


schema.json does not need "$recursiveRoot": true, as it is not extending anything. So using the literal URI reference of # in "$recursiveRef": "#" produces the correct behavior when using schema.json directly.

When used from hyper-schema.json (with either it, links.json, or hyper-operations.json as the entry schema document), a "$recursiveRoot" will be set, so it is important that these references are "$recursiveRef" rather than "$ref".

awwright

I think this solves the problem.
Note that it limits schemas to overriding one recursive schema.
Suppose I wanted to extend both Hyper-schema, and a Link Definition Object, with my own vocabulary. This might not be possible in exactly the way I expect it to be possible.

But the potential for problems seems minimal.

handrews · 2018-06-21T00:31:21Z

@awwright awesome, thanks! I'm closing this and will work up a full PR.

I did think of the dual extension scenario but decided that the limitation was either neutral or possibly even a feature rather than a bug.

If you want to extend hyper-schema with additional schema keywords while also extending the LDO with additional link keywords, that works fine. You only need the recursion on the hyper-schema side, as the LDO itself is not recursive (its members are not links).

Now, if you want to add LDO keywords that take more LDOs as values, or do something else similar with a nested object where that would make more sense, then there are problems.

Basically, embedding one recursive thing in another is still hard, but embedding a non-recursive extension in a recursive thing works fine.

I think it would be possible to extend this system to make the recursive-in-recursive scenario work, by keeping track of where the target of a $recursiveRef can attach on the instance. In the embedding scenario, the "outer" recursive system can attache to the instance root (as well as other places). The embedded recursive thing would, by definition, not do that.

But that's really complicated and there's not enough of a use case to justify sorting it out, much less asking implementations to support it. And it might also be possible to handle the recursion separately and the glue the results together somehow. Meh.

I say we wait for a justification to worry about it. The meta-schema recursive extension use case is very well established, and now that we are looking at multi-vocabulary, it's clear that it will be a relatively common concern. Previously, when we discussed this sort of thing, people including myself argued that extending meta-schemas would be rare. But now we're making that a fundamental building block, so I think the keyword is justified.

handrews added core vocabulary labels Jun 15, 2018

handrews added this to the draft-08 milestone Jun 15, 2018

handrews requested review from philsturgeon and awwright June 15, 2018 03:13

handrews commented Jun 15, 2018

View reviewed changes

schema.json does not need recursiveRoot

4c0599e

because it is not extending anything

handrews commented Jun 15, 2018

View reviewed changes

handrews mentioned this pull request Jun 15, 2018

Recursive schema composition #558

Closed

awwright approved these changes Jun 20, 2018

View reviewed changes

handrews closed this Jun 21, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Example of $recursiveRoot and $recursiveRef #605

Example of $recursiveRoot and $recursiveRef #605

handrews commented Jun 15, 2018 •

edited

Loading

handrews left a comment

handrews Jun 15, 2018

handrews Jun 15, 2018

gregsdennis Jun 15, 2018 •

edited

Loading

gregsdennis Jun 15, 2018

handrews Jun 15, 2018

handrews Jun 15, 2018

handrews Jun 15, 2018

handrews Jun 15, 2018

handrews Jun 15, 2018

handrews Jun 15, 2018

handrews Jun 15, 2018 •

edited

Loading

handrews Jun 15, 2018

awwright left a comment

handrews commented Jun 21, 2018

Example of $recursiveRoot and $recursiveRef #605

Example of $recursiveRoot and $recursiveRef #605

Conversation

handrews commented Jun 15, 2018 • edited Loading

handrews left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregsdennis Jun 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

handrews Jun 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awwright left a comment

Choose a reason for hiding this comment

handrews commented Jun 21, 2018

handrews commented Jun 15, 2018 •

edited

Loading

gregsdennis Jun 15, 2018 •

edited

Loading

handrews Jun 15, 2018 •

edited

Loading