Trigger HttpClosedException Error

[nkreiger]

Describe the bug

Consistent errors in Knative Eventing Kafka Dispatcher logs

{"@timestamp":"2024-12-16T21:38:21.639Z","@version":"1","message":"failed to send event to subscriber context={topics=[knative-broker-dxcm-default], consumerGroup='knative-trigger-test-github-4', reference=uuid: \"36d85610-bf6f-4e75-a85c-365bfacc5745\"\nnamespace: \"test\"\nname: \"github-4\"\n} target=http://plugin-github.test.svc.cluster.local","logger_name":"dev.knative.eventing.kafka.broker.dispatcher.impl.http.WebClientCloudEventSender","thread_name":"vert.x-eventloop-thread-5","level":"ERROR","level_value":40000,"stack_trace":"io.vertx.core.http.HttpClosedException: Connection was closed\n","context":{},"target":"http://plugin-github.test.svc.cluster.local"}

Expected behavior

Connection would not be closed.

spec:
  broker: default
  delivery:
    retry: 1
    timeout: PT5H
  filter:
    attributes:
      source: webhook.cipipeline.git_diff
      type: origin
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: plugin-github
      namespace: test

To Reproduce

Create a trigger to a long-running service.

Knative release version

v1.15.2

Additional context

I've had it work with long processes (the same thing), 30-45 m just fine. This seems to be happening randomly? Not sure where I should check.

[nkreiger]

It's almost like this:

eventing-kafka-broker/data-plane/dispatcher/src/main/java/dev/knative/eventing/kafka/broker/dispatcher/impl/http/WebClientCloudEventSender.java

Line 202 in 38a5e41

.timeout(

From what I'm seeing in the logs, it is working most of the time, but randomly it is falling back to the Default. Is there any suggested way to debug this? Besides forking, building and changing the default timeout to confirm.

Is it possible if it could be something with the reconciling sometimes resetting it because its a shared resource?