Skip to content
This repository has been archived by the owner on Sep 2, 2024. It is now read-only.

Backport: Set subscription readiness based on consumer group status (#182) #249

Merged

Conversation

devguyio
Copy link
Contributor

@devguyio devguyio commented Dec 2, 2020

Backport of #182
Fixes #134
Fixes #98
Fixes knative/eventing-contrib#1556

Proposed Changes

  • Set subscription readiness based on consumer group status in the consolidated Kafka
  • Set's KafkaChannel consolidated dispatcher to read-only
  • Enables HA in KafkaChannel consolidated dispatcher

Release Note

- A bug was fixed in the consolidated KafkaChannel where subscriptions would show up in the channel's `status.subscribers` before the dispatcher becomes ready to dispatch messages for those subscribers.
- Consolidated KafkaChannel dispatcher's horizontal scalability works now seamlessly with reconciler leader election.

…tensions#182)

* WIP: Set subscription readiness based on consumer group status

* Refactor KafkaWatcher to an edge-triggered, level driven design

* Refactor the test to prevent a map race

* goimports and gofmt comments

* Add tests for the KafkaWatcher and reconnect for AdminClient

* Improve consumer group watcher code aethetics and robustness

* Make variable names more descriptive for consumer group watcher and adimn client
* Forget watcher when channel is deleted
* Terminate old watcher and free the cache when config is created

* Set the KafkaChannel consolidated dispatcher reconciler to be read-only

* Enable HA for KafkaChannel(consolidated) dispatcher

* Run update-codegen.sh

* Optimize consumer group watcher

* Use sets.String for internal cache instead of string slice
* Allow for a single callback per watcher
* Synchronization on Terminate and Forget
@google-cla google-cla bot added the cla: yes Indicates the PR's author has signed the CLA. label Dec 2, 2020
@knative-prow-robot knative-prow-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 2, 2020
@matzew
Copy link
Contributor

matzew commented Dec 2, 2020

Nice seeing this backport, @devguyio

/lgtm

/assign @pierDipi
/assign @lionelvillard

for final reviews/approvalz

@knative-prow-robot knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 2, 2020
@codecov
Copy link

codecov bot commented Dec 2, 2020

Codecov Report

❗ No coverage uploaded for pull request base (release-0.19@a413410). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff               @@
##             release-0.19     #249   +/-   ##
===============================================
  Coverage                ?   75.52%           
===============================================
  Files                   ?      114           
  Lines                   ?     4408           
  Branches                ?        0           
===============================================
  Hits                    ?     3329           
  Misses                  ?      875           
  Partials                ?      204           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a413410...5ea021b. Read the comment docs.

Copy link
Member

@pierDipi pierDipi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@lionelvillard
Copy link
Contributor

/approve

@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: devguyio, lionelvillard, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow-robot knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 2, 2020
@knative-prow-robot knative-prow-robot merged commit 547ad82 into knative-extensions:release-0.19 Dec 2, 2020
matzew pushed a commit to matzew/eventing-kafka that referenced this pull request Dec 9, 2020
…tensions#182) (knative-extensions#249)

* WIP: Set subscription readiness based on consumer group status

* Refactor KafkaWatcher to an edge-triggered, level driven design

* Refactor the test to prevent a map race

* goimports and gofmt comments

* Add tests for the KafkaWatcher and reconnect for AdminClient

* Improve consumer group watcher code aethetics and robustness

* Make variable names more descriptive for consumer group watcher and adimn client
* Forget watcher when channel is deleted
* Terminate old watcher and free the cache when config is created

* Set the KafkaChannel consolidated dispatcher reconciler to be read-only

* Enable HA for KafkaChannel(consolidated) dispatcher

* Run update-codegen.sh

* Optimize consumer group watcher

* Use sets.String for internal cache instead of string slice
* Allow for a single callback per watcher
* Synchronization on Terminate and Forget
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cla: yes Indicates the PR's author has signed the CLA. lgtm Indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants