Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use different strategies for the key and the value in the same topic #573

Open
vascoferraz opened this issue Sep 18, 2023 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@vascoferraz
Copy link

vascoferraz commented Sep 18, 2023

Hi everyone :)

I read the documentation and I did not find any way to use different strategies, that is, TopicNameStrategy, RecordNameStrategy or TopicRecordNameStrategy, for the key and the value in the same topic.

I also tried the following configurations but them failed while deploying:

First attempt - Add two topics entries (one for the value and another for the key)

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalArgumentException: com.purbon.kafka.topology.exceptions.ValidationException: Missing required value.schema.file on schemas for topic xxx-customer-v1

Second attempt: Duplicate the topic entry and set RecordNameStrategy for the value (first entry) and TopicNameStrategy for the key (second entry).

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalStateException: Duplicate key xxx-customer-v1

@vascoferraz vascoferraz added the enhancement New feature or request label Sep 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant