chore: confluent API spec evolution

[amstee]

About this change - What it does

Allow definition of two new fields when schema is registered to keep compatibility with confluent clients

Issue Number 916

Why this way

See: https://docs.confluent.io/platform/current/schema-registry/develop/api.html#post--subjects-(string-%20subject)

[fijolekProjects]

Thank you guys! I'm testing these changes using ghcr.io/aiven-open/karapace:develop docker image, and indeed I can create a subject that contains new ruleSet field without errors, but unfortunately the new fields are not stored in __schemas topic which means that the new fields data is lost and cannot be used.

Example

Create a subject:

curl -X POST --location "http://localhost:8081/subjects/example_order-value/versions?normalize=false" \
    -d '{"ruleSet": {"domainRules": [{"name": "checkPersonAge", "kind": "CONDITION", "mode": "WRITE", "type": "CEL", "params": {"dlq.topic": "bad-data"}, "expr": "message.person.age > 18", "onFailure": "DLQ", "disabled": false}]}, "schema": "{\"type\":\"record\",\"name\":\"OutboxOrder\",\"namespace\":\"com.example.outbox.messages\",\"doc\":\"Order avro message\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"doc\":\"Order Id\",\"default\":null},{\"name\":\"store_id\",\"type\":\"string\"},{\"name\":\"person\",\"type\":{\"type\":\"record\",\"name\":\"OutboxPerson\",\"doc\":\"Person avro message\",\"fields\":[{\"name\":\"first_name\",\"type\":\"string\",\"doc\":\"The person first name - mandatory\"},{\"name\":\"last_name\",\"type\":[\"null\",\"string\"],\"doc\":\"The person last name - optional\",\"default\":null},{\"name\":\"age\",\"type\":[\"null\",\"int\"],\"doc\":\"The person age\",\"default\":null}]},\"doc\":\"The person making the order\"},{\"name\":\"product_id\",\"type\":\"long\"},{\"name\":\"timestamp\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},\"doc\":\"Original timestamp of the event\"}]}"}'

Fetch a subject:

curl -X GET --location "http://localhost:8081/subjects/example_order-value/versions/latest"

Fetched subject from karapace:

{
  "id": 1,
  "schema": "{\"doc\":\"Order avro message\",\"fields\":[{\"default\":null,\"doc\":\"Order Id\",\"name\":\"id\",\"type\":[\"null\",\"string\"]},{\"name\":\"store_id\",\"type\":\"string\"},{\"doc\":\"The person making the order\",\"name\":\"person\",\"type\":{\"doc\":\"Person avro message\",\"fields\":[{\"doc\":\"The person first name - mandatory\",\"name\":\"first_name\",\"type\":\"string\"},{\"default\":null,\"doc\":\"The person last name - optional\",\"name\":\"last_name\",\"type\":[\"null\",\"string\"]},{\"default\":null,\"doc\":\"The person age\",\"name\":\"age\",\"type\":[\"null\",\"int\"]}],\"name\":\"OutboxPerson\",\"type\":\"record\"}},{\"name\":\"product_id\",\"type\":\"long\"},{\"doc\":\"Original timestamp of the event\",\"name\":\"timestamp\",\"type\":{\"logicalType\":\"timestamp-millis\",\"type\":\"long\"}}],\"name\":\"OutboxOrder\",\"namespace\":\"com.example.outbox.messages\",\"type\":\"record\"}",
  "subject": "example_order-value",
  "version": 1
}

Fetched subject when using confluent's schema registry

{
  "subject": "example_order-value",
  "version": 1,
  "id": 1,
  "schema": "{\"type\":\"record\",\"name\":\"OutboxOrder\",\"namespace\":\"com.example.outbox.messages\",\"doc\":\"Order avro message\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"doc\":\"Order Id\",\"default\":null},{\"name\":\"store_id\",\"type\":\"string\"},{\"name\":\"person\",\"type\":{\"type\":\"record\",\"name\":\"OutboxPerson\",\"doc\":\"Person avro message\",\"fields\":[{\"name\":\"first_name\",\"type\":\"string\",\"doc\":\"The person first name - mandatory\"},{\"name\":\"last_name\",\"type\":[\"null\",\"string\"],\"doc\":\"The person last name - optional\",\"default\":null},{\"name\":\"age\",\"type\":[\"null\",\"int\"],\"doc\":\"The person age\",\"default\":null}]},\"doc\":\"The person making the order\"},{\"name\":\"product_id\",\"type\":\"long\"},{\"name\":\"timestamp\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},\"doc\":\"Original timestamp of the event\"}]}",
  "ruleSet": {
    "domainRules": [
      {
        "name": "checkPersonAge",
        "kind": "CONDITION",
        "mode": "WRITE",
        "type": "CEL",
        "params": {
          "dlq.topic": "bad-data"
        },
        "expr": "message.person.age > 18",
        "onFailure": "DLQ",
        "disabled": false
      }
    ]
  }
}

[nosahama]

Hello @fijolekProjects exactly what I had discussed with @amstee , these changes are just to allow the API validation. Going further with adding the new attributes to the stored schema and updating the API response is something that might cause breaking changes, should be well tested and properly versioned, thus, the issue is still left open.

[fijolekProjects]

Thank you @nosahama

[nosahama]

We will look into the full implementation in the coming weeks 👍