Live subscription using Cosmos change feed

[johnvanloon]

Was planning on building an eventstore for CosmosDb like this one.
One requirement would be to support live subscriptions, which could be implemented using the change feed of CosmosDb.
That would break the idea of having 1 interface for both storage implementations (or you would need to implement some kind of polling for Table Store or use the event grid notifications).

I'm totally fine to implement this myself, but I'm in doubt whether to create a PR as improvement for your project or fork/start over with a Cosmos-only implementation.

What would you prefer?

[Dzoukr]

Hi @johnvanloon,

thank you for asking - I was actually thinking about adding this feature, but decided to wait whether anyone really needs it. I'd like to keep the same API for both, but believe we can have this feature. We can have basically two approaches:

1. Make subscription part of shared EventStore API and implement it for CosmosDB using Change feed, and for Table Storage to implement some kind of non-blocking subscription (to do not let subscriber block the write side).

2. Make subscription part of configuration, which is service specific, so it would be totally fine to have it only for CosmosDB.

First approach is not perfect, but I think it is doable and could be handy to have the same functionality for both - we just need to come up with some reasonable mechanism for firing events (EventRead records) to subscribers without being blocked by them.

Please, let me know what do you think about it.

[johnvanloon]

Agree that option 1 should be possible and also prefer that one over option 2.
Just got my devenv up and running (need to use a real Azure CosmosDb, because there's still no emulator on Linux :( ).
Will try to implement at least the Cosmos version for option 1 later this week.

[Dzoukr]

That would be perfect, thanks for that! I discussed with my friend @jbrestan skilled in pub/sub concepts how to do it for TableStorage so I'll update you with our progress.

[johnvanloon]

Sorry, took a little longer to get started.
But sadly I was blocked pretty soon afterwards. Seems like CosmosDb Mongo-api does not support the changefeed: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/cosmos-db/change-feed.md

Thinking how I want to continue now. You got any great ideas?

[johnvanloon]

Ok. Forget that last comment. You are using SQL API. Somehow I assumed Mongo-API and somehow all tests also pass with a Mongo-API CosmosDb. Anyway, no blocking issues now...

[Dzoukr]

Hi John, I have new draft version in progress but need to write some tests. I think good way would be to expose IObservable (with observeOn thread pool) so consumers will not interfere with each other and can use reactive approach for hooking to newly appended events. The way I have it locally implement, the change feed is not actually needed. What do you think about it?

[johnvanloon]

Then how do you get the changes? Is it some kind of long-polling?
Think IObservable makes sense, although I was aiming for a slightly more basic eventhandler first.
Just like in the first example here: https://fsharpforfunandprofit.com/posts/concurrency-reactive/

[Dzoukr]

No, it is internally normal .net event fired on each success append function. But with System.Reactive we can make it safe for consumers (and Rx approach would be more natural)

[johnvanloon]

But that only works if you're projection runs in the same process as the writer, right? I'm trying to trigger something in another service.

[Dzoukr]

Yeah, you are right. It can be consumed as any other .net event. I think you aim more for some kind of queue (Kafka, RabbitMQ), if you want to observe new events from different process.

[Dzoukr]

I just published v1.4.0 with IObservable<EventRead> as new property of CosmoStore API, which can help observing events within process (or plug it with some queue). Based on our discussion about what is the main purpose of CosmoStore (unified API, simple usage), I would not implement new features based on storage specific behavior (like Cosmos DB Change Feed). However this could be done in separate project.

[johnvanloon]

Rethinking what I try to accomplish I agree that it seems to make sense to put it in a separate project. Basically what I'm trying to accomplish is an architecture as outlined in this blog series .

Using that architecture, you use the event-store both for storage and for communicating with other services. You make the events the single source of truth for all services. For now I'll try to build things out in my own project. Maybe our paths will cross again in future. Thanks for your effort and discussions.

[Dzoukr]

I thank you! Good luck and let me know how it goes with your project.