change data capture breaking with redpanda #64842
Comments
ghost
added
the
C-bug
|
Hello, I am Blathers. I am here to help you get the issue triaged. It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates. I have CC'd a few people who may be able to assist you:
If we have not gotten back to your issue within a few business days, you can try the following:
🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan. |
blathers-crl
bot
added
O-community
|
I am able to produce the same error using CRDB v20.2.8 and redpanda v21.5.1. The same test on v21.1.0-rc.1 works as expected. It looks like the upgrade in facbd60 potentially resolved the underlying issue. Reproduction stepsStacktrace from logs matching reported failureNB: It looks like in redpanda-data/redpanda#898 redpanda itself upgrade to a newer version of Sarama specifically to pull in IBM/sarama#1898. We haven't yet pulled in that fix; although it isn't clear to me that it is relevant here. |
|
Excellent - we want to use 21 anyway. I'll see what it takes to upgrade our environment. Yep, sarami 1.29.0 has no issues talking to redpanda
So that's working... |
|
@stevendanna good notes. change IBM/sarama#1898 from sarama shouldn't be needed anymore starting I'm happy to help debug. but not sure how crdb gets to Are topics relying on autocreate by chance? curious. |
|
looks like a pretty long list of fixes in the sarama, so could be anything trivial to sarama not parsing a setting - https://github.com/Shopify/sarama/compare/v1.22.1..v1.27.1 |
👍 I'll likely still look at whether the sarama upgrade is backportable to 20.2.
Cool. That's good to know, thanks!
Yes, we auto-create topics based on the table name if the topic does not already already exist.
I would not be surprised if there were multiple fixes in there that are important. I believe that the upgrade was motivated by similar incompatibilities with some other kafka-speaking services. |
|
Confirm upgrade to
_That would be very helpful ^ _ Quick-n-dirty upgrayedd script.. Puts all the existing table entries to topic (
|
Just to avoid any bad expectation setting: Since it is a dependency update that will pull in more than just that single library, I'm not sure it'll make sense to backport it. |
|
After looking at this, it seems that Cockroach is using a pretty old producer API. This is what I'm seeing from the logs in Redpanda: |
|
I took a look at the Cockroach CDC implementation, and it looks like they don't set the Kafka version so it defaults to version 1.0 of Kafka. It should just work if they specify the version to 2.4.0 or higher. Here is an example from the Sarama example code: https://github.com/Shopify/sarama/blob/83d633e6e4f71b402df5e9c53ad5c1c334b7065d/examples/consumergroup/main.go#L57 |
|
Ok, my mistake, so with the version of Sarama that Cockroach is currently using, it defaults to the minimum version, which is |
Previously the Kafka sink didn't specify a version to sarama, so it defaulted to the minimum version of 0.8.2. This behavior has been changed in the latest version of sarama to use version 1.0 of the Kafka API. Information on why this change was made by sarama can be found here: IBM/sarama#1787 Resolves cockroachdb#64842 Release note (enterprise change): Set the Kafka CDC sink to use the 1.0 version of the Kafka API.
|
@rkruze Thanks for taking a look! |
|
All - we also patched redpanda to parse v0 of the produce - redpanda-data/redpanda#1389 (prolly out on redpanda v21.5.3) |
|
@senior7515 Nice, thanks! |
|
I'm going to close this for now. In addition to the changes to redpanda, in CRDB,
|
Cluster working, message working when created using
kafka/bin/kafka-console-producer.shorrpk topic produce namestail -f /run/cockroachdb/logs/cockroach.log
The text was updated successfully, but these errors were encountered: