Skip to content
This repository has been archived by the owner on Jun 19, 2022. It is now read-only.

Return 429 when ingress if sending too man messages, instead of 500 #1737

Merged
merged 5 commits into from
Sep 21, 2020

Conversation

tayarani
Copy link
Member

@tayarani tayarani commented Sep 17, 2020

Fixes #1697

Proposed Changes

  • When the ingress is overloaded, propagate a 429 back to the caller instead of masking it with a 500.

Release Note

Return 429 (too many messages) when the ingress broker is sending too many messages, instead of 500 (internal server error).

Docs

@google-cla google-cla bot added the cla: yes (override cla status due to multiple authors bug) label Sep 17, 2020
@@ -28,3 +28,6 @@ var ErrIncomplete = errors.New("incomplete config")

// ErrNotReady is the error when a broker is not ready.
var ErrNotReady = errors.New("not ready")

// ErrOverflow is the error when a broker ingress is sending too many requests to PubSub.
var ErrOverflow = errors.New("bundler reached buffered byte limit")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you import bundler.ErrOverflow rather than redefine it here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@tayarani
Copy link
Member Author

I'm not too familiar with the code yet, but this change doesn't seem like something easily testable in our unit tests. Is it ok for me to push this without tests? @Harwayne

@knative-test-reporter-robot

The following jobs failed:

Test name Triggers Retries
pull-google-knative-gcp-integration-tests 2020-09-18 14:26:08.683 +0000 UTC 1/3

Automatically retrying due to test flakiness...
/test pull-google-knative-gcp-integration-tests

Copy link
Contributor

@Harwayne Harwayne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

I'll leave approval to someone more familiar with the Broker area.

@grantr
Copy link
Contributor

grantr commented Sep 18, 2020

/cc @yolocs

@yolocs
Copy link
Member

yolocs commented Sep 21, 2020

You can actually test this code. You can have a fake DecoupleSink implementation (https://github.com/google/knative-gcp/blob/master/pkg/broker/ingress/handler.go#L76) that always returns bundler.ErrOverflow on Send. Then you can check if the resp code is 429.

@knative-metrics-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-google-knative-gcp-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/broker/ingress/handler.go 89.7% 89.8% 0.2

@yolocs
Copy link
Member

yolocs commented Sep 21, 2020

/lgtm
/approve

@knative-prow-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tayarani, yolocs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow-robot knative-prow-robot merged commit 72e6010 into google:master Sep 21, 2020
@tayarani tayarani deleted the issue-1550_ingress_error branch September 21, 2020 21:33
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
approved cla: yes (override cla status due to multiple authors bug) lgtm size/M
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Return 429 when reaching pubsub publish buffer limit
7 participants