ADR-038 Implementation

[i-norden]

This is a parent issue to track progress on the ADR-038 state listening epic

• ADR-038 spec: ADR 038: State Listening #8012
• WriteListener, listen.KVStore, MultiStore and KVStore updates: ADR-038 Part 1: WriteListener, listen.KVStore, MultiStore and KVStore updates #8551
• StreamingService interface, file writing implementation, and configuration: feat: ADR-038 Part 2: StreamingService interface, file writing implementation, and configuration #8664
• Plugin architecture spec for streaming service(s): Plugin architecture i-norden/cosmos-sdk#1 docs: ADR-038 plugin proposal #10482
• Plugin architecture implementation: feat: ADR-038 plugin system #10639
• State change files auxiliary gRPC server: feat: ADR-038 Part 3: State change files auxiliary gRPC server #9647
• Kafka streaming service

Original proposals/issues:

• CosmosSDK State Watching Features - KVStore reading and listening capabilities - KVstore listening #7888
• CosmosSDK State Watching Features - KVStore reading and listening capabilities - Subscription/stream service(s) #7889

Related issues:

• Synchronize state to an external database #7099
• Expose support for custom indexing #3469

Related discussions:

• Store Decoding #9158
• Table Store (aka ORM) Package #9156

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[robert-zaremba]

For the part 3, instead of building gRPC service to stream file changes and reinventing a wheel let's use a proper queue service in the first place like Kafka or RabbitMQ. These are market solutions, widely used, with many integrations.

[i-norden]

It's a bit frustrating to have this design flip-flop occur after the file streaming service implementation has been open and undergoing review since February. If this had been brought up during the design phase we could have avoided this unnecessary work (it also voids my work done on the 3rd part of the ADR here).

This was discussed in depth before the ADR was finalized and merged- the file writing service is included as part of the ADR. A lot of the discussion in regards to the "base" implementation can be here and I think Bez's comment here is especially pertinent.

That said, I recognize the appeal and am open to making this change! Of the two listed, I am partial to the push-based approach of RabbitMQ.

[i-norden]

From today's call:

1. Still merge feat: ADR-038 Part 2: StreamingService interface, file writing implementation, and configuration #8664
2. Switch to plugin architecture
3. Don't reopen feat: ADR-038 Part 3: State change files auxiliary gRPC server #9647 (or do we still want to introduce this PR and get a PoC writing to Postgres using it?)
4. Introduce a Kafka or RabitMQ streaming service (consensus seems to be to use Kafka- using a sink connector or a plugin to enable a push consumer model)

[egaxhaj]

Hi, @i-norden. I’m new to the Provenance team and have experience with Kafka. I want to help out on this issue. You can DM me on the Cosmos Developers Discord server as Ergels Gaxhaj | Provenance or we can have a discussion here.

[IbrarMakaveli]

Hey,

I'm a Data Engineer at Osmosis, I had a meeting today on this module that we are interested for the Streaming part, and indeed I am also in favor of migrating to Kafka for several reasons:

• Kafka is resilient, fault tolerant, scalable and one of the leaders in the streaming market (core product is free on Apache licence)
• It avoids you to maintain the code of the Producer & Consumer functions (which is very complicated).
• Kafka has a large number of connectors, which makes integration very easy: Producer can be written in Go and Consumer write in Python, Java or whatever

[clevinson]

@i-norden on today's SDK call we discussed this epic landing in v0.45 (which we're hoping to be feature complete in 2-4 weeks).

Does that timeline sound reasonable to you? I don't think we're wanting to push the release further than necessary, but there was a lot of desire to get this work merged in asap, so we wanted to see if this work can potentially get wrapped up in time for v0.45 under the current timeline.

[i-norden]

@clevinson I think that is a reasonable timeline, the plugin architecture PR is ready to open once the current pending PR gets merged. So the plugin arch with a file streaming service plugin will definitely be ready. @egaxhaj-figure has taken ownership of the Kafka plugin work- does this timeline seem reasonable to you?

[egaxhaj]

@clevinson, @i-norden Yes, 2-4 weeks is a reasonable timeline to get the Kafka plugin in there.

[robert-zaremba]

Thanks @egaxhaj-figure . This is really needed.
Let's focus on merging part-2. It LGTM. Need @alexanderbez to release the block or add more comments. Once this is done, I suggest:

• review and merge plugin interfaces --let's firstly do small PR and quick review rather then going into a big one. @i-norden are you still handling it? I can't see that PR. @egaxhaj-figure - would you collaborate on that?
• review and merge Kafka based on the work above.
• Part-3 still make sense feat: ADR-038 Part 3: State change files auxiliary gRPC server #9647 -- I've reopened it, but it's not in the priority now.

[robert-zaremba]

@IbrarMakaveli -- this was exactly the motivation for moving into the plugin architecture: to reuse existing queuing solutions.

File based approach is still an interesting thing and will be good to merge that functionality.

[i-norden]

• review and merge plugin interfaces --let's firstly do small PR and quick review rather then going into a big one. @i-norden are you still handling it? I can't see that PR. @egaxhaj-figure - would you collaborate on that?

Hey, yeah I'll handle the plugin arch PR. The PR isn't open against the master branch yet, will open it once this 2nd PR is merged (i-norden#1).

• Part-3 still make sense feat: ADR-038 Part 3: State change files auxiliary gRPC server #9647 -- I've reopened it, but it's not in the priority now.

Thanks!

[robert-zaremba]

Discussed with @i-norden . Let's:

• Do an ADR-38 PR update first (the architecture doc) to provide details about plugin architecture
• Live review it next week,
• The do the implementation PR

[tac0turtle]

closing as adr38 in the current form is delivered. Work on a minor change is ongoing