Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Incorrect unmarshalling self-hosted Kafka event #422

Open
LKay opened this issue Jan 11, 2022 · 1 comment
Open

[BUG] Incorrect unmarshalling self-hosted Kafka event #422

LKay opened this issue Jan 11, 2022 · 1 comment
Assignees
Labels
bug type/events issue or feature request related to the events package

Comments

@LKay
Copy link

LKay commented Jan 11, 2022

We have a lambda function which is hooked up and listens to both MSK and self-hosted Kafka instances to process incoming messages. While in case is MSK the event source mapping works fine, for self-hosted event source mapping SDK panics with error as follow:

{
    "errorMessage": "json: cannot unmarshal number -23 into Go struct field KafkaRecord.records.headers of type uint8",
    "errorType": "UnmarshalTypeError"
}

This seems like some type mismatch (uint8) so it can't properly unmarshal negative numbers coming in the lambda event payload.

We had opened AWS support case and worked with them to some degree but unfortunately they couldn't provide us much context so we did debugging ourselves.

The issue seems to be present in the versions 1.26+ of the Lambda SDK, and downgrading to 1.25 or below doesn't cause this unmarshal error to appear. The main different is the presence of the headers field in the struct introduced through https://github.com/aws/aws-lambda-go/pull/385/files#diff-cf0e5ac9714618f1821f8694da87d7af4579e0eafed210e8dbdcc5977bf5a69eR20. This might also be related to some previous issue #390.

As mentioned above this only applies to self-hosted event mapping, not MSK events.

Environment:

  • Go: 1.17
  • AWS LAmbda SDK Go: 1.26, 1.27, 1.28
  • Kafka version (self-hosted): 2.7.0
@bmoffatt bmoffatt added the bug label Dec 20, 2022
@bmoffatt bmoffatt assigned bmoffatt and unassigned bmoffatt Dec 20, 2022
@bmoffatt bmoffatt added the type/events issue or feature request related to the events package label Dec 21, 2022
@bmoffatt
Copy link
Collaborator

bmoffatt commented Apr 25, 2023

Ah so the test data additions in #385 was correct, but the coverage that would've warned of the type mismatch was removed: https://github.com/aws/aws-lambda-go/pull/385/files#r675441134

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug type/events issue or feature request related to the events package
Projects
None yet
Development

No branches or pull requests

2 participants