Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provision contract with OIDC information #3646

Merged

Conversation

creydr
Copy link
Contributor

@creydr creydr commented Jan 29, 2024

Fixes #3572 (follow up on #3571)

Proposed Changes

  • Update contract with OIDC information. This includes:
    • Audience of the Destination
    • Audience of the Reply
    • Audience of the Dead-Letter-Sink
    • OIDCServiceAccountName
    • Audience of the ingress

Release Note

none

@knative-prow knative-prow bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/control-plane labels Jan 29, 2024
@knative-prow knative-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 29, 2024
Copy link

codecov bot commented Jan 29, 2024

Codecov Report

Attention: 50 lines in your changes are missing coverage. Please review.

Comparison is base (a3b83bc) 62.18% compared to head (d0f825b) 61.96%.
Report is 1 commits behind head on main.

Files Patch % Lines
control-plane/pkg/reconciler/channel/channel.go 0.00% 8 Missing and 4 partials ⚠️
control-plane/pkg/reconciler/consumer/consumer.go 7.69% 9 Missing and 3 partials ⚠️
...lane/pkg/reconciler/consumergroup/consumergroup.go 0.00% 5 Missing and 2 partials ⚠️
control-plane/pkg/core/config/utils.go 0.00% 2 Missing and 1 partial ⚠️
control-plane/pkg/reconciler/broker/broker.go 0.00% 2 Missing and 1 partial ⚠️
...ntrol-plane/pkg/reconciler/channel/v2/channelv2.go 0.00% 2 Missing and 1 partial ⚠️
control-plane/pkg/reconciler/sink/kafka_sink.go 0.00% 2 Missing and 1 partial ⚠️
control-plane/pkg/reconciler/trigger/trigger.go 50.00% 2 Missing and 1 partial ⚠️
...ntrol-plane/pkg/reconciler/trigger/v2/triggerv2.go 0.00% 2 Missing and 1 partial ⚠️
...-plane/pkg/apis/sources/v1beta1/kafka_lifecycle.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3646      +/-   ##
============================================
- Coverage     62.18%   61.96%   -0.22%     
+ Complexity      847      845       -2     
============================================
  Files           188      188              
  Lines         12727    12785      +58     
  Branches        273      273              
============================================
+ Hits           7914     7922       +8     
- Misses         4199     4233      +34     
- Partials        614      630      +16     
Flag Coverage Δ
java-unittests 74.46% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@creydr
Copy link
Contributor Author

creydr commented Jan 29, 2024

/test reconciler-tests

@creydr
Copy link
Contributor Author

creydr commented Jan 29, 2024

/retest-required

@creydr creydr force-pushed the add-oidc-infos-to-contract branch from 6ad3468 to 9bb59d2 Compare February 1, 2024 12:24
@knative-prow knative-prow bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 1, 2024
@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

Can be tested in the following way (currently not for Broker & Triggers, as this is still WIP (#3600)):

  1. Create services (subscriber, reply, DLS):
$ kn service create event-display --image gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
$ kn service create my-reply --image gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
$ kn service create my-dls --image gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
  1. Create a KafkaChannel (having configured to be a KafkaChannel the default):
apiVersion: messaging.knative.dev/v1
kind: Channel            
metadata:  
  name: my-kafka-channel
  1. Create a Subscription:
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
  name: my-sub
spec:
  channel:
    apiVersion: messaging.knative.dev/v1beta1
    kind: KafkaChannel
    name: my-kafka-channel
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: event-display
    audience: ed-audience
  delivery:
    deadLetterSink:
      ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: my-dls
      audience: dls-audience
  reply:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-reply
    audience: reply-audience

Then we see in the kafka-channel-channels-subscriptions the following:

k -n knative-eventing get cm kafka-channel-channels-subscriptions -ojsonpath='{.binaryData.data}' | base64 -d | jq .resources
[
  {
    "uid": "bc395fb8-6812-4429-8127-10c7a77e7bc6",
    "topics": [
      "knative-messaging-kafka.default.my-kafka-channel"
    ],
    "bootstrapServers": "my-cluster-kafka-bootstrap.kafka:9092",
    "ingress": {
      "path": "/default/my-kafka-channel",
      "host": "my-kafka-channel-kn-channel.default.svc.cluster.local",
      "audience": "messaging.knative.dev/kafkachannel/default/my-kafka-channel"
    },
    "egresses": [
      {
        "consumerGroup": "kafka.default.my-kafka-channel.6c26bb94-5208-458d-90b4-a46e58dd8d42",
        "destination": "http://event-display.default.svc.cluster.local",
        "destinationAudience": "ed-audience",
        "replyUrl": "http://my-reply.default.svc.cluster.local",
        "replyUrlAudience": "reply-audience",
        "uid": "6c26bb94-5208-458d-90b4-a46e58dd8d42",
        "egressConfig": {
          "deadLetter": "http://my-dls.default.svc.cluster.local",
          "deadLetterAudience": "dls-audience"
        },
        "deliveryOrder": "ORDERED",
        "reference": {
          "uuid": "6c26bb94-5208-458d-90b4-a46e58dd8d42",
          "namespace": "default",
          "name": "my-sub"
        },
        "oidcServiceAccountName": "my-sub-oidc-messaging.knative.dev-subscription"
      }
    ],
    "reference": {
      "uuid": "bc395fb8-6812-4429-8127-10c7a77e7bc6",
      "namespace": "default",
      "name": "my-kafka-channel",
      "kind": "KafkaChannel",
      "groupVersion": "messaging.knative.dev/v1beta1"
    }
  }
]

note the destinationAudience, replyAudience, deadLetterAudience and the oidcServiceAccount.

Same was tested for a KafkaSource:

  1. Create Tropic and KafkaSource:
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic  
metadata:          
  name: knative-demo-topic
  namespace: kafka                           
  labels:                          
    strimzi.io/cluster: my-cluster           
spec:                 
  partitions: 3           
  replicas: 1
  config:   
    retention.ms: 7200000               
    segment.bytes: 1073741824
---
apiVersion: sources.knative.dev/v1beta1
kind: KafkaSource 
metadata:          
  name: kafka-source      
spec:                                        
  consumerGroup: knative-group     
  bootstrapServers:                          
  - my-cluster-kafka-bootstrap.kafka:9092
  topics:                 
  - knative-demo-topic
  sink:     
    ref:                                
      apiVersion: serving.knative.dev/v1                                                         
      kind: Service      
      name: event-display 
    audience: ed-audience

Resulted in the following contract:

$ k -n knative-eventing get cm kafka-source-dispatcher-0 -ojsonpath='{.binaryData.data}' | base64 -d | jq .resources
[
  {
    "uid": "bd71e8cc-e3ba-4d22-8542-307e2a1d37bd",
    "topics": [
      "knative-demo-topic"
    ],
    "bootstrapServers": "my-cluster-kafka-bootstrap.kafka:9092",
    "egresses": [
      {
        "consumerGroup": "knative-group",
        "destination": "http://event-display.default.svc.cluster.local",
        "destinationAudience": "ed-audience",
        "discardReply": {},
        "uid": "bd71e8cc-e3ba-4d22-8542-307e2a1d37bd",
        "egressConfig": {
          "retry": 10,
          "backoffDelay": "300",
          "timeout": "600000"
        },
        "deliveryOrder": "ORDERED",
        "reference": {
          "uuid": "b08d105d-9f36-4818-b86e-6401ec50ad00",
          "namespace": "default",
          "name": "kafka-source"
        },
        "vReplicas": 1,
        "featureFlags": {
          "enableRateLimiter": true,
          "enableOrderedExecutorMetrics": true
        },
        "oidcServiceAccountName": "my-source-oidc-sa"
      }
    ],
    "multiAuthSecret": {},
    "reference": {
      "uuid": "b08d105d-9f36-4818-b86e-6401ec50ad00",
      "namespace": "default",
      "name": "kafka-source"
    }
  }
]

@creydr creydr changed the title [WIP] Provision contract with OIDC information Provision contract with OIDC information Feb 1, 2024
@knative-prow knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 1, 2024
@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

/retest

Copy link
Member

@pierDipi pierDipi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 1, 2024
@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

/retest

@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

/retest-required

2 similar comments
@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

/retest-required

@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

/retest-required

@creydr
Copy link
Contributor Author

creydr commented Feb 1, 2024

/test reconciler-tests

@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/retest-required

1 similar comment
@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/retest-required

@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/test reconciler-tests

@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/test unit-tests

@creydr creydr force-pushed the add-oidc-infos-to-contract branch from ab1b27d to d0f825b Compare February 2, 2024 10:02
@knative-prow knative-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Feb 2, 2024
@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

@pierDipi I needed to rebase and then fix a unit test. Can you recheck?

@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/test unit-tests

🤔

@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/retest-required

@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/test upgrade-tests

1 similar comment
@creydr
Copy link
Contributor Author

creydr commented Feb 2, 2024

/test upgrade-tests

Copy link
Member

@pierDipi pierDipi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Feb 2, 2024
Copy link

knative-prow bot commented Feb 2, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: creydr, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow bot merged commit 6012e1b into knative-extensions:main Feb 2, 2024
36 of 37 checks passed
creydr added a commit to creydr/knative-eventing-kafka-broker that referenced this pull request Feb 4, 2024
* Provision contract with OIDC information

* Add DLS audience in KafkaChannel CRD

* Update KafkaSource to expose its sinks audience in status

* Update Trigger test to include OIDC SA in contract

* Propagate KafkaSources OIDC serviceAccountName to consumer and consumergroup

* Propagate triggerv2s serviceAccountName to consumergroup

* Fix unit test
creydr added a commit to creydr/knative-eventing-kafka-broker that referenced this pull request Feb 21, 2024
* Provision contract with OIDC information

* Add DLS audience in KafkaChannel CRD

* Update KafkaSource to expose its sinks audience in status

* Update Trigger test to include OIDC SA in contract

* Propagate KafkaSources OIDC serviceAccountName to consumer and consumergroup

* Propagate triggerv2s serviceAccountName to consumergroup

* Fix unit test
creydr added a commit to creydr/knative-eventing-kafka-broker that referenced this pull request Mar 18, 2024
* Provision contract with OIDC information

* Add DLS audience in KafkaChannel CRD

* Update KafkaSource to expose its sinks audience in status

* Update Trigger test to include OIDC SA in contract

* Propagate KafkaSources OIDC serviceAccountName to consumer and consumergroup

* Propagate triggerv2s serviceAccountName to consumergroup

* Fix unit test
creydr added a commit to creydr/knative-eventing-kafka-broker that referenced this pull request Apr 2, 2024
* Provision contract with OIDC information

* Add DLS audience in KafkaChannel CRD

* Update KafkaSource to expose its sinks audience in status

* Update Trigger test to include OIDC SA in contract

* Propagate KafkaSources OIDC serviceAccountName to consumer and consumergroup

* Propagate triggerv2s serviceAccountName to consumergroup

* Fix unit test
openshift-merge-bot bot pushed a commit to openshift-knative/eventing-kafka-broker that referenced this pull request Apr 3, 2024
* Update contract with fields for OIDC information (knative-extensions#3632)

* Update contract to include OIDC information

* Run hack/update-codegen.sh

* Move OIDC SA to egress

* Expose OIDC audience of KafkaChannel in its status (knative-extensions#3622)

* Provision .status.address.audience and .status.addresses[*].audience in KafkaChannel

* Add kafka Channel e2e test to check if audience is provisioned

* Run goimport

* Update deps

* Auto generate Triggers OIDC identity service account and expose in its status (knative-extensions#3604)

* Support auto generation of Triggers identity service account and expose in AuthStatus

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* fix unit test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* make ServiceAccountLister public

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* add oidc unit test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* fix unit test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* integrate oidc unit test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* add some logic to reconcile triggers, if the features config map gets updated

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* fix unit test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* revert vendor/knative.dev/pkg/webhook/resourcesemantics/defaulting/controller.go

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* fix unit test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* fix Verify Deps and Codegen test

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* fix unit test and reconcile triggers, in case of the features configmap changes

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

---------

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>

* Create KafkaSources OIDC service account and expose in its status (knative-extensions#3660)

* Create KafkaSources OIDC service account and expose in its status

* Run goimport

* Provision contract with OIDC information (knative-extensions#3646)

* Provision contract with OIDC information

* Add DLS audience in KafkaChannel CRD

* Update KafkaSource to expose its sinks audience in status

* Update Trigger test to include OIDC SA in contract

* Propagate KafkaSources OIDC serviceAccountName to consumer and consumergroup

* Propagate triggerv2s serviceAccountName to consumergroup

* Fix unit test

* [data-plane] Add the caching for OIDC JWT token to tokenprovider (knative-extensions#3663)

* Add the caching to tokenprovider

* Add the boiler

* Running the codegen

* Running the codegen

* Revert "Running the codegen"

This reverts commit 0ccf69c.

* Use constant, set buffer before token expire, remove unnecessary change

* Codegen changes

* Codegen changes

* Update data-plane/core/src/main/java/dev/knative/eventing/kafka/broker/core/oidc/TokenProvider.java

Co-authored-by: Christoph Stäbler <cstabler@redhat.com>

* Fix the review comments

* Run codegen

* Codegen changes

* Code gen again

---------

Co-authored-by: Christoph Stäbler <cstabler@redhat.com>

* Receiver reject requests for wrong audience (knative-extensions#3675)

* Receiver: reject request for wrong audience

* Switch to AuthenticationHandler

* Fix "Request has already been read" issue

* Change TokenVerifier to an interface

* Initialize TokenVerifier in main

* Add test for AuthenticationHandler

* Only initialize OIDC discovery config in main and create a TokenVerifier per verticle instance.

* Rerun hack/update-codegen.sh

* Move TokenVerifier setup into setup() to prevent null pointer exception when vertx is null

* Update KafkaChannel OIDC e2e tests, to run OIDC conformance tests so the receiver is tested too.

* Run OIDC e2e tests as part of the reconciler suite

* Fix KafkaChannelOIDC e2e test

* Fix lint issue

* Address review comments

* Dispatcher authenticate requests (knative-extensions#3677)

* Change TokenProvider to return future to get a token

* Dispatcher add OIDC to token, when target has an audience set

* Add e2e test

* Support exposing the Audience of a Broker (knative-extensions#3600)

* Support exposing the Audience of a Broker

* fix formatting

* fix formatting

* test fixes

* Populate broker.status.addresses[*].audience field too

* Run goimports and gofmt

* Fix unit test

---------

Co-authored-by: Christoph Stäbler <cstabler@redhat.com>

* Add broker OIDC e2e tests (knative-extensions#3685)

* Add broker OIDC e2e tests

* Fix broker template to allow TLS & OIDC configuration on dead letter sink

* Remove unneeded check when setting broker audience (knative-extensions#3708)

* Check status code of OIDC discovery response (knative-extensions#3707)

* Check status code of OIDC discovery endpoint

* Run update-codegen.sh

* Only allow 200 status code on OIDC discovery endpoint

* Run update-deps.sh

* Add OIDC tests to encryption/auth test suite

* run make generate-release

* TokenVerifier: execute blocking calls in parallel (knative-extensions#3728)

* TokenVerifier: execute blocking calls in parallel

* Revert "TokenVerifier: execute blocking calls in parallel"

This reverts commit f3dbde9.

* Revert: removed changes in contract.pb.go

* Cancel receiver pod start on invalid OIDC config only if authentication.oidc is enabled (knative-extensions#3761)

* Cancel pod start on invalid OIDC config only if authentication.oidc is enabled

* Update namespaced broker to copy features configmap too.

* Add unit test for FeaturesConfig class (knative-extensions#3771)

* Add unit test for FeaturesConfig class

* Update data-plane/core/src/test/java/dev/knative/eventing/kafka/broker/core/features/FeaturesConfigTest.java

Co-authored-by: Calum Murray <cmurray@redhat.com>

---------

Co-authored-by: Calum Murray <cmurray@redhat.com>

* Run make generate-release again

---------

Signed-off-by: pingjiang <xiangpingjiang1998@gmail.com>
Co-authored-by: cola <45722758+xiangpingjiang@users.noreply.github.com>
Co-authored-by: Leo Li <leoli@redhat.com>
Co-authored-by: Gunish Matta <33680363+gunishmatta@users.noreply.github.com>
Co-authored-by: Partha Ghosh <112557191+parth721@users.noreply.github.com>
Co-authored-by: Calum Murray <cmurray@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[control-plane]: Provision Audience and OIDCServiceAccountName fields in contract
2 participants