-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CDC without using Kafka #2513
Comments
Thanks for the feedback and interest, @joeblew99 ! That's our ultimate goal - to be able to provide a generic framework that can leveraged across different application stacks. Since Kafka is widely used, we started our Beta with Kafka. Our immediate next step is to provide a sample console app which app developers can then use as a reference to build their own CDC sinks: #2351 You've provided some good suggestions and we'll look into those. |
@ndeodhar I second this request. I'd love to have an agnostic way to plug into other message brokers like NATS. |
Are there any plans to expose or allow the option of exposing a vanilla http2 based grpc endpoint of the CDCService, say on a different port of the yb-master servers? I'm referring to this service definition in: https://github.com/yugabyte/yugabyte-db/blob/master/src/yb/cdc/cdc_service.proto#L44
It would be awesome to be able to invoke a grpc call such as |
Any example on implement your own connectors? I assume for Kafka based CDC, as long as the cluster is functional I will get at least one delivery guarantee? How about for custom connectors case? what if the connectors crash (I assume connectors in this context is an external client subscribe to something like a trigger or gRPC stream), how the connectors know where to begin again? |
Never used it, but may be worth to take a look: https://debezium.io/ |
What's the current status on this? |
Just look at it, it seems also rely on plugin like pgoutput or wal2json then push to Kafka, |
Seems like the preconditions for this are being worked on! NATS Streaming has been deprecated in the meantime and replaced with NATS Jetstream, which would be awesome to have as a connector/sink: https://docs.nats.io/nats-concepts/jetstream |
about CDC.
It would be awesome if the architecture ( and in the future the docs ) allowed a more neutral way to tap into the avro schema types and to integrate with different message queues.
Cockroach dB uses s to and pushes to Kafka also. So it's also not great.
So sure there is not standard for open service brokers ( there is but it's labourious ), so I would have through its possible to provide a neutral CDC feed system that any developer can tap into and write the data into whatever message queue like system they use.
For me it would be NATS streaming server.
https://github.com/nats-io/nats-streaming-server
But even liftbridge is also fine. I note that liftbridge is pure grpc based and so more neutral in that respect.
https://github.com/liftbridge-io/liftbridge
Anyhow I think you understand the intent of this issue
I would be happy to work with the team on this as I like yugabytedb. I also am running CRDB too but don't like its complexity.
The text was updated successfully, but these errors were encountered: