JSON_SR Schemas Incorrectly Considered Incompatible in CREATE STREAM

[dlfaraujo]

When ksqlDB uses a topic that has a schema already registered in schema registry, it ensures that the schema is compatible with the one that ksqlDB generates given the generated sql code. To test compatibility, it uses the schema registry's testCompatibility method. For JSON_SR (AVRO as well, but there are other factors in play that make it not an issue) case changes cause compatibility issues (see this comment: #5801 (comment)) especially with regards to case sensitivity.

Additionally, there are many properties that are not convertible to SQL schemas:

• additionalProperties
• $id

Changing our schema to include additional properties for the serde might be possible, but it seems that testCompatibility should not fail if missing the above properties.

---

Original Ticket:

When using a JSON Schema with "additionalProperties": true and trying to CREATE STREAM from a topic with that schema it returns the error Could not register schema for topic.. The exact same schema but with "additionalProperties": false works fine.

flight.json.zip

[agavra]

Thanks for reporting this @dlfaraujo - I'm going to change the name of this ticket and clarify the bug a little bit to cover this general class of bugs. I think it will require changes in schema registry to resolve these bugs.

[rayokota]

FYI, it's best to generate schemas with "additionalProperties": false. Otherwise, if set totrue or omitted (default is true), most schema changes will be incompatible.

[agavra]

@rayokota - thanks for the tip. I'm not sure I understand though, wouldn't "additionalProperties": true be strictly more lenient than having it set to false? Or are we concerned about the case where a field that wasn't in the first schema but was in the data is then declared as having a different type?

[rayokota]

Yes, the latter case. See confluentinc/schema-registry#1529