Rules and their meaning such as implicit-type-union

[jonaslagoni]

I am encountering rule violations such as implicit-type-union but when encountering them (I assume this applies for all rules) they don't say anything about what and where the problem lies.

I assume it might have something to do with the current status and missing features https://jsonbinpack.sourcemeta.com/api/ 🤔

Example stack trace:

assert(received)

    Expected value to be equal to:
      true
    Received:
      false
    
    Message:
      Circular rule: implicit-type-union

      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:77:33)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:79:24)
      at Object.simplifySchema (node_modules/jsonbinpack/dist/lib/simplifier/index.js:163:12)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:102:54)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:79:24)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:79:24)
      at Object.simplifySchema (node_modules/jsonbinpack/dist/lib/simplifier/index.js:163:12)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:135:67)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:79:24)
      at applyRules (node_modules/jsonbinpack/dist/lib/simplifier/index.js:79:24)

So my question is, what do the rules mean (especially implicit-type-union) and is there a way to see where the problem lies?

[jviotti]

Hey @jonaslagoni, thanks for reaching out. To explain what's going on, JSON BinPack will apply a series of transformation rules to schemas to normalize them (we call it "canonicalizing") into a representation that makes them simpler for static analysis. One of the many (https://github.com/sourcemeta/jsonbinpack/tree/main/src/compiler/canonicalizer_rules) rules we run is the implicit-type-union one, which aims to convert a schema without any type definition into an anyOf of each possible type (essentially making these implicit constraints explicit). See https://github.com/sourcemeta/jsonbinpack/blob/main/src/compiler/canonicalizer_rules/implicit_type_union.h.

These canonicalization rules are meant to be internal and not part of the interface, thus they are not documented but you are definitely hitting a bug where the rules don't seem to agree with each other. Can you show me the schema that fails? I can probably find the root cause, but better error messages would be great for sure 😅

In any case, as you pointed out, the project is very much still in progress. I believe you are using the old, archived JavaScript PoC I presented along with my dissertation (https://github.com/jviotti/jsonbinpack-poc), while the new production-ready one is taking place here. That said, I'd still love to check if the issue is present on the new implementation.

This one is C++, but compiles well to WebAssembly (though we don't provide an NPM package yet!) The main puzzle piece to complete this project is a JSON Schema compiler that preprocesses JSON Schema into a simpler representation that can be more efficiently evaluated on embedded devices to support various applicators. All of that work is happening right now on https://github.com/sourcemeta/jsontoolkit. For example: sourcemeta/jsontoolkit#600.

Can you remind me what you used JSON BinPack for right now? I'd love to check what's missing for you out of this repo and hopefully prioritize that + a nice NPM package to use it.

[jonaslagoni]

Can you show me the schema that fails? I can probably find the root cause, but better error messages would be great for sure 😅
...
Can you remind me what you used JSON BinPack for right now? I'd love to check what's missing for you out of this repo and hopefully prioritize that + a nice NPM package to use it.

So the final schema that is being used is:

{
  "$id":"TestObject",
  "description":"This object contains simple types of MetaModel generations. Object, enum, number, boolean array, tuple.",
  "type":"object",
  "properties":{
    "string_type":{
      "type":"string",
      "x-modelgen-inferred-name":"string_type"
    },
    "number_type":{
      "type":"number",
      "x-modelgen-inferred-name":"number_type"
    },
    "boolean_type":{
      "type":"boolean",
      "description":"Status if marriage live in given house",
      "x-modelgen-inferred-name":"boolean_type"
    },
    "array_type":{
      "type":"array",
      "items":{
        "type":"string",
        "x-modelgen-inferred-name":"array_type_item"
      },
      "x-modelgen-inferred-name":"array_type"
    },
    "tuple_type":{
      "type":"array",
      "items":[
        {
          "type":"string",
          "x-modelgen-inferred-name":"tuple_type_item_0"
        },
        {
          "type":"number",
          "x-modelgen-inferred-name":"tuple_type_item_1"
        }
      ],
      "additionalItems":false,
      "x-modelgen-inferred-name":"tuple_type"
    },
    "object_type":{
      "type":"object",
      "properties":{
        "test":{
          "type":"string",
          "x-modelgen-inferred-name":"object_type_test"
        }
      },
      "additionalProperties":false,
      "x-modelgen-inferred-name":"object_type"
    },
    "enum_type":{
      "enum":[
        "test",
        1,
        true,
        {
          "test":2
        }
      ],
      "x-modelgen-inferred-name":"enum_type"
    }
  },
  "additionalProperties":false,
  "required":[
    "string_type", "boolean_type"
  ],
  "x-modelgen-inferred-name":"root",
  "$schema":"https://json-schema.org/draft/2020-12/schema"
}

For context it is through Modelina that I generate payload models where the serialization library is jsonbinpack, the input to Modelina can be JSON Schema draft 4, 6, 7 and is converted into https://json-schema.org/draft/2020-12/schema through your alterschema library. See the implementation here: https://github.com/asyncapi/modelina/blob/696548fa24ecaa0b6de6192dc1272401b43da27d/src/generators/typescript/presets/JsonBinPackPreset.ts#L33

model.originalInput in this context is https://github.com/asyncapi/modelina/pull/1980/files#diff-d5acf0a8610f003df74b211d92197be36072527d2b55d8615576978704e901b8

In any case, as you pointed out, the project is very much still in progress. I believe you are using the old, archived JavaScript PoC I presented along with my dissertation (https://github.com/jviotti/jsonbinpack-poc), while the new production-ready one is taking place here. That said, I'd still love to check if the issue is present on the new implementation.

🤦 You are right, I am definitely using the old one 👍

This one is C++, but compiles well to WebAssembly (though we don't provide an NPM package yet!) The main puzzle piece to complete this project is a JSON Schema compiler that preprocesses JSON Schema into a simpler representation that can be more efficiently evaluated on embedded devices to support various applicators. All of that work is happening right now on https://github.com/sourcemeta/jsontoolkit. For example: sourcemeta/jsontoolkit#600.

Guess I will have to wait a bit then 😄

Thanks for getting back to me 🤙

[jviotti]

For context it is through Modelina that I generate payload models where the serialization library is jsonbinpack

Interesting!

the input to Modelina can be JSON Schema draft 4, 6, 7 and is converted into https://json-schema.org/draft/2020-12/schema through your alterschema library.

BTW, Alterschema will be getting improved as part of GSoC at JSON Schema to polish some remaining edge cases. Good to hear you are using it already, and it will only get better soon!

Guess I will have to wait a bit then

Haha, sorry about that. The internals of JSON BinPack are pretty complex and it took me a long time building LOTS of pre-requisites to be able to pull it off as a production-ready library for IoT. I'm close now though and working full-time on these things as of a month ago :) I'll let you know as soon we the new version if available on NPM for you to integrate.

So the final schema that is being used is:

In any case, I confirm that this new schema works well with the new on-going version of JSON BinPack. The result of the canonicalization process is like this:

{
  "$id": "TestObject",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "additionalProperties": {
    "not": {
      "anyOf": [
        {
          "enum": [
            null
          ]
        },
        {
          "enum": [
            false,
            true
          ]
        },
        {
          "minProperties": 0,
          "properties": {},
          "required": [],
          "type": "object"
        },
        {
          "minItems": 0,
          "type": "array"
        },
        {
          "minLength": 0,
          "type": "string"
        },
        {
          "type": "number"
        },
        {
          "multipleOf": 1,
          "type": "integer"
        }
      ]
    }
  },
  "description": "This object contains simple types of MetaModel generations. Object, enum, number, boolean array, tuple.",
  "minProperties": 2,
  "properties": {
    "array_type": {
      "items": {
        "minLength": 0,
        "type": "string",
        "x-modelgen-inferred-name": "array_type_item"
      },
      "minItems": 0,
      "type": "array",
      "x-modelgen-inferred-name": "array_type"
    },
    "boolean_type": {
      "description": "Status if marriage live in given house",
      "enum": [
        false,
        true
      ],
      "x-modelgen-inferred-name": "boolean_type"
    },
    "enum_type": {
      "enum": [
        "test",
        1,
        true,
        {
          "test": 2
        }
      ],
      "x-modelgen-inferred-name": "enum_type"
    },
    "number_type": {
      "type": "number",
      "x-modelgen-inferred-name": "number_type"
    },
    "object_type": {
      "additionalProperties": {
        "not": {
          "anyOf": [
            {
              "enum": [
                null
              ]
            },
            {
              "enum": [
                false,
                true
              ]
            },
            {
              "minProperties": 0,
              "properties": {},
              "required": [],
              "type": "object"
            },
            {
              "minItems": 0,
              "type": "array"
            },
            {
              "minLength": 0,
              "type": "string"
            },
            {
              "type": "number"
            },
            {
              "multipleOf": 1,
              "type": "integer"
            }
          ]
        }
      },
      "minProperties": 0,
      "properties": {
        "test": {
          "minLength": 0,
          "type": "string",
          "x-modelgen-inferred-name": "object_type_test"
        }
      },
      "required": [],
      "type": "object",
      "x-modelgen-inferred-name": "object_type"
    },
    "string_type": {
      "minLength": 0,
      "type": "string",
      "x-modelgen-inferred-name": "string_type"
    },
    "tuple_type": {
      "additionalItems": false,
      "items": [
        {
          "type": "string",
          "x-modelgen-inferred-name": "tuple_type_item_0"
        },
        {
          "type": "number",
          "x-modelgen-inferred-name": "tuple_type_item_1"
        }
      ],
      "minItems": 0,
      "type": "array",
      "x-modelgen-inferred-name": "tuple_type"
    }
  },
  "required": [
    "string_type",
    "boolean_type"
  ],
  "type": "object",
  "x-modelgen-inferred-name": "root"
}

There is a probably a bug somewhere on the old PoC. That old repo is archived, so I'm no longer releasing versions of it in favor of this new one, but you can probably input it the above de-normalized schema directly, and it should work.

Again, hopefully I have the new version of JSON BinPack ready soon!

[jviotti]

For additional context, the old PoC was just a quick'n dirty prototype I presented along with my dissertation (on a tight deadline!), but as you could tell, is probably buggy and alpha. The one in this repo will be rock solid and battle tested