Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Kafka EventBus #2502

Merged
merged 41 commits into from
Mar 23, 2023
+3,693 −344
Merged

feat: Kafka EventBus #2502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
c08a009
Initial scaffolding for kafka eventbus
dfarr Jan 11, 2023
81fb278
Add kafka eventbus spec (#12)
premadk Jan 18, 2023
7532cb9
in-cluster kafka (#14)
bilalba Jan 24, 2023
9008646
Kafka Validations (#16)
bilalba Jan 26, 2023
01197a0
Implement kafka eventbus eventsource (#11)
dfarr Jan 27, 2023
0308141
Enable kafka functional tests
dfarr Jan 27, 2023
a1b2575
Kafka eventbus sensor (#18)
dfarr Feb 7, 2023
3722a3e
Enable sasl support for kafka eventbus
dfarr Feb 11, 2023
20cc7bf
Add offset manager
dfarr Feb 15, 2023
0a67273
Enable batching
dfarr Feb 16, 2023
f0d154a
Add argo-events-sa service account to all e2e tests
dfarr Feb 16, 2023
3975699
Consolidate kafka eventbus config
dfarr Feb 17, 2023
785fd2f
Action function need not return an error
dfarr Feb 17, 2023
3820cc3
Enable condition resets
dfarr Feb 22, 2023
3ed44a0
Change event time check logic
dfarr Feb 22, 2023
1543eca
Bypass leader election for kafka eventbus sensor
dfarr Feb 22, 2023
3ac68e0
Guard against batch with no messages
dfarr Feb 22, 2023
b97b80e
Use message timestamp for resets instead of event timestamp
dfarr Feb 27, 2023
71713c4
Add version to kafka config
dfarr Feb 27, 2023
6297ec4
fix version in config (#19)
premadk Feb 28, 2023
9bd2fd7
Remove kafka streamconfig
dfarr Feb 28, 2023
ebc7ca7
Attach access secret for kafka eventbus and sensor
dfarr Feb 28, 2023
5f7690c
Attach secrets to eventsource and sensor when using kafka eventbus
dfarr Mar 1, 2023
54e5e21
Remove accessSecret from kafka config and make group name optional
dfarr Mar 1, 2023
ca80bf6
Log when event is filtered out
dfarr Mar 2, 2023
a676c26
Add batch tests
dfarr Mar 2, 2023
61b5cf5
Provide helpful message if topic(s) do not exist and fail fast
dfarr Mar 2, 2023
d6e1b08
Add timing asserts to batch tests
dfarr Mar 2, 2023
f814685
Nullify batch to reset
dfarr Mar 2, 2023
4a1d1db
Batch every second
dfarr Mar 2, 2023
f945c4e
Revert eventbus controller status copy
dfarr Mar 3, 2023
9d92087
Update comments
dfarr Mar 3, 2023
30e1ad5
Revert ci pull_request branches
dfarr Mar 3, 2023
bd7c23f
Use dashes ('-') instead of dots ('.') in default kafka names
dfarr Mar 17, 2023
334e4d1
Add comments
dfarr Mar 17, 2023
cd241b9
Move kafka specs to test-functional only
dfarr Mar 20, 2023
70d879d
Add batch comment
dfarr Mar 20, 2023
522cfa0
Remove exotic from kafka eventbus spec
dfarr Mar 20, 2023
f1e79a4
Add additional controller tests for eventsource and sensor
dfarr Mar 21, 2023
def91b2
Add Kafka EventBus docs
dfarr Mar 21, 2023
c7ce51d
Remove key from kafka trigger message
dfarr Mar 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ jobs:
e2e-tests:
name: E2E Tests
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 25
env:
KUBECONFIG: /home/runner/.kubeconfig
strategy:
Expand All @@ -123,6 +123,7 @@ jobs:
include:
- driver: stan
- driver: jetstream
- driver: kafka
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This executes all e2e tests against kafka as well

steps:
- name: Checkout code
uses: actions/checkout@v3
Expand Down
9 changes: 8 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,14 @@ test:
go test $(shell go list ./... | grep -v /vendor/ | grep -v /test/e2e/) -race -short -v

test-functional:
go test -v -timeout 15m -count 1 --tags functional -p 1 ./test/e2e
ifeq ($(EventBusDriver),kafka)
kubectl -n argo-events apply -k test/manifests/kafka
kubectl -n argo-events wait -l statefulset.kubernetes.io/pod-name=kafka-0 --for=condition=ready pod --timeout=60s
endif
go test -v -timeout 20m -count 1 --tags functional -p 1 ./test/e2e
ifeq ($(EventBusDriver),kafka)
kubectl -n argo-events delete -k test/manifests/kafka
endif

# to run just one of the functional e2e tests by name (i.e. 'make TestMetricsWithWebhook'):
Test%:
Expand Down
188 changes: 188 additions & 0 deletions api/event-bus.html
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading