Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v3/integrations/nrgrpc: Error message not found and Stack trace is incorrect #221

Closed
Dilip-Nandakumar opened this issue Aug 19, 2020 · 10 comments · Fixed by #317
Closed
Assignees
Labels

Comments

@Dilip-Nandakumar
Copy link

Description

On navigating to errors section under Events we are able to find the error response recored for grpc api calls.

However on clicking on a specific error we cannot find the error message and stack trace is incomplete.

We can only find the grpc error status code response code 2.

Steps to Reproduce

Simulate an error response from grpc handler in the example

Expected Behavior

  1. Error message to be displayed along with grpc error status code
  2. Full stack trace to be displayed

NR Diag results

Your Environment

Package Version
github.com/newrelic/go-agent/v3 v3.8.1
github.com/newrelic/go-agent/v3/integrations/nrgrpc v1.1.0
google.golang.org/grpc v1.31.0

Reproduction case

Additional context

  1. We are only fetching the status code from the error message returned by the hander in the nrgrpc interceptor
@newrelic-eheinlein
Copy link
Contributor

Hi @Dilip-Nandakumar! Sorry for not getting back to you sooner. I know you also created a Support ticket for this issue, and I am currently working with our Support team to investigate. I would like to carry forward with future communication via the Support ticket, although I will come back here and add final details in case someone comes across this same issue in the future and wants to know what the resolution is.

@newrelic-eheinlein
Copy link
Contributor

hi @Dilip-Nandakumar - I have been working on reproducing this issue and will be filing a bug for it. However, I don't think my stack trace seems to be problematic when I repro the issue - could you post what you are seeing and what you expect it to be so I can compare?

@Dilip-Nandakumar
Copy link
Author

Current StackTrace In New Relic

go-agent/v3/integrations/nrgrpc/nrgrpc_server.go (nrgrpc.UnaryServerInterceptor.func2:85)
go-agent/v3/integrations/nrgrpc/example/sampleapp/sampleapp.pb.go (sampleapp._SampleApplication_DoUnaryUnary_Handler:265)
path/pkg/mod/google.golang.org/grpc@v1.15.0/server.go (grpc.(*Server).processUnaryRPC:982)
path/pkg/mod/google.golang.org/grpc@v1.15.0/server.go (grpc.(*Server).handleStream:1208)
path/pkg/mod/google.golang.org/grpc@v1.15.0/server.go (grpc.(*Server).serveStreams.func1.1:686)
/usr/local/go/src/runtime/asm_amd64.s (runtime.goexit:1373)

Expected StackTrace

The stack trace was printed using fmt.Sprintf("%+v", err)

main.(*Server).DoUnaryUnary
        go-agent/v3/integrations/nrgrpc/example/server/server.go:32
github.com/newrelic/go-agent/v3/integrations/nrgrpc/example/sampleapp._SampleApplication_DoUnaryUnary_Handler.func1
        go-agent/v3/integrations/nrgrpc/example/sampleapp/sampleapp.pb.go:263
github.com/newrelic/go-agent/v3/integrations/nrgrpc.UnaryServerInterceptor.func2
        go-agent/v3/integrations/nrgrpc/nrgrpc_server.go:85
github.com/newrelic/go-agent/v3/integrations/nrgrpc/example/sampleapp._SampleApplication_DoUnaryUnary_Handler
        go-agent/v3/integrations/nrgrpc/example/sampleapp/sampleapp.pb.go:265
google.golang.org/grpc.(*Server).processUnaryRPC
        path/mod/google.golang.org/grpc@v1.15.0/server.go:982
google.golang.org/grpc.(*Server).handleStream
        path/mod/google.golang.org/grpc@v1.15.0/server.go:1208
google.golang.org/grpc.(*Server).serveStreams.func1.1
        path/mod/google.golang.org/grpc@v1.15.0/server.go:686
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373(45398)

Missing in Current StackTrace

main.(*Server).DoUnaryUnary
        go-agent/v3/integrations/nrgrpc/example/server/server.go:32

The expectation is to get the full stacktrace along with the handler function from where the error is thrown.

@newrelic-eheinlein
Copy link
Contributor

Thank you for the additional information! That is exactly what I needed. I am adding this bug to our board to be prioritized. We are unfortunately somewhat shorthanded right at the moment so I can't make any promises about how it will be prioritized, but we will update this issue if and when we do a release addressing the bug.

@gunturaf
Copy link
Contributor

gunturaf commented Apr 5, 2021

having this same issue, can NewRelic give us update about this bug? thanks! cc @jodeev @newrelic-eheinlein

@RichVanderwal
Copy link
Contributor

Thanks, @guntraf ! From your reports, it looks like skipping the stop of the stack trace, which generally keeps the New Relic Agent from needlessly appearing in the traces, is working against us in this case.

We'll get a reproduction going and determine the best path forward in the case of the nrgrpc integration.

@odannyc
Copy link
Contributor

odannyc commented May 3, 2021

I am also running into this issue. I'd really like to see the grpc error message in the Errors tab in NewRelic. Currently I am only seeing the grpc code int.

@RichVanderwal Can you provide us with an update on this? It's been close to a month since it was stated that the team was looking into it.

Thanks!

@nr-swilloughby
Copy link
Contributor

There are two separate issues reported here. We're working on the error message issue, but I do have an answer on the stack trace problem. After reviewing the code and running some tests we have determined that the error reporting code's handling of the call stack trace is working as intended, but we see where there may be some confusion about the intended behavior.

The Go agent removes the topmost set of calls that come from modules within the agent itself, so we report the functions in the application itself and don't clutter the report with the calls inside the agent that are merely reporting that error. You're seeing a number of your application's functions omitted from the reported stack trace as well, but that is because your application is the example code supplied along with the Go Agent, which is located in the Go Agent's source tree, and so gets automatically removed.

You should not see this in an actual production application (i.e., one whose source code is not located in the New Relic Go Agent source tree). Can you confirm that this is the case by trying with an application which is not located in the Agent's source tree?

Thanks

@nr-swilloughby nr-swilloughby self-assigned this May 21, 2021
@RichVanderwal RichVanderwal linked a pull request May 26, 2021 that will close this issue
@nr-swilloughby
Copy link
Contributor

With the changes we made, it seems error messages are being reported all the way out to the UI now. In the screenshots below, I had the DoUnaryUnary call return an error with the message string "this is an error". These changes will be in the next release.

Screen Shot 2021-05-27 at 3 21 34 PM
Screen Shot 2021-05-27 at 3 20 38 PM

@Ak-x
Copy link

Ak-x commented Aug 25, 2021

Closing this issue as complete due to @nr-swilloughby 's fix.

@Ak-x Ak-x closed this as completed Aug 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants