Skip to content
This repository has been archived by the owner on Apr 18, 2024. It is now read-only.

Alpakka Kafka External Sharding Allocation support #192

Merged
merged 12 commits into from
Mar 16, 2020

Conversation

seglo
Copy link
Member

@seglo seglo commented Feb 14, 2020

Enhancements to the sample and a proposal of what to send upstream to Alpakka Kafka/akka.kafka namespace.

  • Introduce Alpakka Kafka API for getting the cluster sharding message extractor and group rebalance listener (akka.kafka namespace) Moved to Alpakka Kafka Kafka cluster sharding support with external sharding allocation alpakka-kafka#1067
  • Get the number of partitions automatically
  • Use the Murmur2 hashing library that the DefaultPartitioner from Apache Kafka uses
  • Use an embedded Kafka to make running the sample easier

@seglo seglo force-pushed the seglo/kafka-sharding branch from 9f54f0c to f53f3b0 Compare February 18, 2020 22:16
Copy link
Member Author

@seglo seglo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ready for an initial review. @chbatey is off this week, but I'm open to others providing feedback @ennru, @raboof, @jrudolph, @ignasi35.

Copy link
Contributor

@ignasi35 ignasi35 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than small suggestions and the 500 Internal Server Error when trying the sample app it all LGTM

@seglo seglo requested a review from chbatey February 19, 2020 20:02
Copy link
Member

@chbatey chbatey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good

@seglo seglo changed the title [WIP] External Sharding with Kafka ideas [WIP] Alpakka Kafka External Sharding Allocation support Mar 6, 2020
@seglo
Copy link
Member Author

seglo commented Mar 6, 2020

I've published a public alpakka kafka snapshot. Should we wait for an alpakka-kafka release before merging?

@seglo seglo marked this pull request as ready for review March 6, 2020 21:27
@seglo seglo changed the title [WIP] Alpakka Kafka External Sharding Allocation support Alpakka Kafka External Sharding Allocation support Mar 6, 2020
@seglo seglo requested a review from chbatey March 9, 2020 14:53
@seglo
Copy link
Member Author

seglo commented Mar 9, 2020

@chbatey I discussed with @ennru that now that this PR is referencing a public snapshot it's probably ok to merge. Do you mind reviewing it again?

Copy link
Member

@chbatey chbatey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@chbatey
Copy link
Member

chbatey commented Mar 16, 2020

I'll fix the conflict

@chbatey chbatey merged commit aafa47c into akka:2.6 Mar 16, 2020
@seglo
Copy link
Member Author

seglo commented Mar 23, 2020

Thanks @chbatey!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants