Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Fix PayloadProcessor response payload race condition #120

Merged
merged 1 commit into from
Nov 24, 2023

Conversation

njhill
Copy link
Member

@njhill njhill commented Nov 16, 2023

Motivation

An issue was reported in #111 where response payloads logged by a configured PayloadProcessor were sometimes incorrectly empty.

Modifications

Do not attempt to avoid slicing the response bytebuf in the case that a PayloadProcessor is configured, do not attempt to avoid some additional refcount updates in status != OK case.

Result

Hopefully no more incorrectly empty logged payloads

Resolves #111

@ckadner
Copy link
Member

ckadner commented Nov 17, 2023

FYI @RobGeada

Copy link
Member

@ckadner ckadner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@njhill -- your changes look good. I just rebased your branch to prepare for merging your PR, but I realized this PR should probably wait for a review/approval by @RobGeada or @tteofili ?

Should we create a ticket to track this?

It would also be useful to have a unit test for this, but the tests included here don't exercise the actual bug. Ideally we'd have a test that actually runs a local modelmesh with an asyncprocessor configured and can trigger the problem (other unit tests already run modelmesh to test other stuff and could hopefully be used as a starting point and adapted).

@kserve kserve deleted a comment from kserve-oss-bot Nov 22, 2023
@ckadner ckadner added this to the v0.11.2 milestone Nov 23, 2023
Copy link
Contributor

@tteofili tteofili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

thanks @njhill and @ckadner

@kserve-oss-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ckadner, njhill, tteofili

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Motivation

An issue was reported in #111 where response payloads logged by a configured PayloadProcessor were sometimes incorrectly empty.

Modifications

Do not attempt to avoid slicing the response bytebuf in the case that a PayloadProcessor is configured, do not attempt to avoid some additional refcount updates in status != OK case.

Result

Hopefully no more incorrectly empty logged payloads

Resolves #111

Signed-off-by: Nick Hill <nickhill@us.ibm.com>
@ckadner
Copy link
Member

ckadner commented Nov 24, 2023

/lgtm

@ckadner ckadner merged commit 582021e into main Nov 24, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AsyncPayloadProcessor owned payloads are released too early
5 participants