feat: debug log gRPC queries #19049

czarcas7ic · 2024-01-14T21:00:30Z

Description

Closes: #XXXX

There is currently an outstanding issue where some unknown gRPC query is causing cache to be polluted, causing gas_used on the node in which this query is called to differ from what is expected and cause a last results hash error. Figuring out what this query is has been very difficult.

This PR introduces a debug log for each gRPC call for easier future debugging.

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

included the correct type prefix in the PR title
confirmed ! in the type prefix if API or client breaking change
targeted the correct branch (see PR Targeting)
provided a link to the relevant issue or specification
reviewed "Files changed" and left comments if necessary
included the necessary unit and integration tests
added a changelog entry to CHANGELOG.md
updated the relevant documentation or specification, including comments for documenting Go code
confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

confirmed the correct type prefix in the PR title
confirmed all author checklist items have been addressed
reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

* config option to log gRPC queries * clarify gRPC log comment * create var

coderabbitai · 2024-01-14T21:00:37Z

Walkthrough

The recent updates introduce a new feature to the gRPC configuration, allowing the logging of gRPC queries. A constant for the default logging behavior and a new field have been added to the configuration struct, along with a corresponding parameter in the TOML configuration file. The gRPC server registration function has been modified to include this new logging option, which is now utilized during server initialization to enhance debugging capabilities.

Changes

File(s)	Change Summary
`server/config/config.go`, `server/config/toml.go`	Introduced `DefaultLogQueries` and the `LogQueries` field for gRPC configuration, along with a new `log-queries` parameter for enabling query logging.
`CHANGELOG.md`	Documented the addition of the gRPC query logging feature.
`baseapp/baseapp.go`, `baseapp/grpcserver.go`, `runtime/builder.go`, `simapp/app.go`	Modified logger initialization, added import statement for "fmt", and updated server initialization to accommodate the new gRPC query logging feature.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Note: Auto-reply has been disabled for this repository by the repository owner. The CodeRabbit bot will not respond to your comments unless it is explicitly tagged.

Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit tests for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository from git and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit tests.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
The JSON schema for the configuration file is available here.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.v2.json

CodeRabbit Discord Community

Join our Discord Community to get help, request features, and share feedback.

julienrbrt · 2024-01-14T21:33:24Z

baseapp/grpcserver.go

@@ -23,7 +24,7 @@ import (
 func (app *BaseApp) GRPCQueryRouter() *GRPCQueryRouter { return app.grpcQueryRouter }

 // RegisterGRPCServer registers gRPC services directly with the gRPC server.
-func (app *BaseApp) RegisterGRPCServer(server gogogrpc.Server) {
+func (app *BaseApp) RegisterGRPCServer(server gogogrpc.Server, logQueries bool) {


Can you add an API breaking changelog?

Added here e82a084

julienrbrt · 2024-01-14T21:34:32Z

server/config/toml.go

@@ -182,6 +182,11 @@ max-recv-msg-size = "{{ .GRPC.MaxRecvMsgSize }}"
 # The default value is math.MaxInt32.
 max-send-msg-size = "{{ .GRPC.MaxSendMsgSize }}"

+# LogQueries if enabled will print an info log containing the query request


Can you add the following here to keep confix up to date: https://github.com/cosmos/cosmos-sdk/blob/main/tools/confix/data/v0.51-app.toml ?

Didn't know this existed! Added here 8e56944

tac0turtle · 2024-01-15T08:48:48Z

baseapp/grpcserver.go

@@ -67,6 +68,10 @@ func (app *BaseApp) RegisterGRPCServer(server gogogrpc.Server) {
 			app.logger.Error("failed to set gRPC header", "err", err)
 		}

+		if logQueries {
+			app.logger.Info("gRPC query received of type: " + fmt.Sprintf("%#v", req))


why not put this as debug and use log filtering for the info you need? another config variable seems overkill here

The thought here is, this will only be enabled in the event we are actively debugging something, and have an integrator to send me their info logs is much easier than asking to send massive debug logs and/or filter it themselves.

Happy to change it in this PR for the sdk, will probably keep as info for the fork for now though.

i think this is what filter logging was created for or one of the reasons. It would be much cleaner to ask an integrator to turn on debugging and paste "....." in the filter logs. This way they only get the logs you want. Currently this approach you will need to ask for a snapshot or they will need to search for the log along side all the other infos.

The filter would be to see blocks are being created and this debug log. So you as the debugger would make your life easier.

This is encroaching on devops territory which there are tools for, we dont want to make the sdk more complicated to run and debug.

Np, will change sdk side

sorry i meant no config changes, just a debug statement

For a node under heavy load like ours (250 rps), this might make the debug logs impossible to grok. I know one can filter but it seems helpful for the logs to be somewhat manageable, and there are rare edge cases where this info is needed. I think a config makes the most sense.

this introduces two ways to do the same thing. feels like this will be confusing to users as if they try to filter for this log it wont show up since its blocked on another config value. We can add it but I feel its over kill for something already suported

I can remove it, I just wanted to explain why I added it as a config in the first place. I am actually not familiar with the debug filtering at all so that's why this wasn't clear to me. Happy to remove it

alexanderbez · 2024-01-15T17:41:58Z

baseapp/grpcserver.go

@@ -23,7 +24,7 @@ import (
 func (app *BaseApp) GRPCQueryRouter() *GRPCQueryRouter { return app.grpcQueryRouter }

 // RegisterGRPCServer registers gRPC services directly with the gRPC server.
-func (app *BaseApp) RegisterGRPCServer(server gogogrpc.Server) {
+func (app *BaseApp) RegisterGRPCServer(server gogogrpc.Server, logQueries bool) {


nit: I would instead directly supply grpcConfig here instead. This future proofs us (even though Baseapp will be sunsetted soon) in case we need other fields or data when calling RegisterGRPCServer.

baseapp/grpcserver.go

czarcas7ic · 2024-01-21T02:19:25Z

@tac0turtle as per your request, I modified this PR to only be a debug log in place of a config option.

CI seems to be failing due to rate limit.

julienrbrt · 2024-01-22T11:35:31Z

baseapp/grpcserver.go

@@ -67,6 +68,8 @@ func (app *BaseApp) RegisterGRPCServer(server gogogrpc.Server) {
 			app.logger.Error("failed to set gRPC header", "err", err)
 		}

+		app.logger.Debug("gRPC query received of type: " + fmt.Sprintf("%#v", req))


Given that the logger is passed directly from the app, maybe we should add a module key here:

cosmos-sdk/baseapp/baseapp.go

Line 196 in 2cde8df

logger: logger,

and maybe name it baseapp (logger.With(log.ModuleKey, "baseapp")). This will make the filtering easier.

IMO, the app.logger should already have the module key set when it's set on BaseApp.

Works too, then it should be added in

cosmos-sdk/runtime/builder.go

Line 32 in 6d10c50

bApp := baseapp.NewBaseApp(a.app.config.AppName, a.app.logger, db, nil, baseAppOptions...)

and

cosmos-sdk/simapp/app.go

Line 251 in 6d10c50

bApp := baseapp.NewBaseApp(appName, logger, db, txConfig.TxDecoder(), baseAppOptions...)

@czarcas7ic could you update this real quick and we'll get it merged 🙏

Hey @alexanderbez, I think this is what you guys were asking for 1c00bb7

alexanderbez

LGTM

coderabbitai · 2024-01-24T03:52:45Z

simapp/app.go

@@ -248,7 +248,7 @@ func NewSimApp(
 	}
 	baseAppOptions = append(baseAppOptions, voteExtOp, baseapp.SetOptimisticExecution())

-	bApp := baseapp.NewBaseApp(appName, logger, db, txConfig.TxDecoder(), baseAppOptions...)
+	bApp := baseapp.NewBaseApp(appName, logger.With(log.ModuleKey, "baseapp"), db, txConfig.TxDecoder(), baseAppOptions...)


Tip
Codebase Verification

The verification process has identified that the logger with the module key "baseapp" is used in simapp/app.go and also in simapp/app_test.go. However, the usage in app_test.go uses a different key ("instance" instead of log.ModuleKey). This indicates that the logger is not used consistently with the module key "baseapp" across the application.

Analysis chain

The NewSimApp function has been modified to include a logger with a module key "baseapp". This change is consistent with the PR objectives and should enhance logging within the baseapp instance. Ensure that the logger is correctly instantiated and that the module key "baseapp" is used consistently across the application.

Scripts Executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash # Verify that the logger with the module key "baseapp" is used consistently across the application. ast-grep --lang go --pattern $'logger.With("ModuleKey", "baseapp")'

Length of output: 68

Script:

#!/bin/bash # Search for any usage of logger.With with any string literal as the module key to check for "baseapp" usage. ast-grep --lang go --pattern $'logger.With($_, "baseapp")'

Length of output: 332

alexanderbez · 2024-01-24T05:38:15Z

runtime/builder.go

@@ -29,7 +30,7 @@ func (a *AppBuilder) Build(db dbm.DB, traceStore io.Writer, baseAppOptions ...fu
 		baseAppOptions = append(baseAppOptions, option)
 	}

-	bApp := baseapp.NewBaseApp(a.app.config.AppName, a.app.logger, db, nil, baseAppOptions...)
+	bApp := baseapp.NewBaseApp(a.app.config.AppName, a.app.logger.With(log.ModuleKey, "baseapp"), db, nil, baseAppOptions...)


I would do it automatically in NewBaseApp itself

feat: config option to log gRPC queries (#503)

8b3db7c

* config option to log gRPC queries * clarify gRPC log comment * create var

czarcas7ic requested a review from a team as a code owner January 14, 2024 21:00

This comment has been minimized.

Sign in to view

changelog

f6a0135

julienrbrt reviewed Jan 14, 2024

View reviewed changes

czarcas7ic added 2 commits January 14, 2024 14:37

add API breaking changelog

e82a084

add log-queries to confix v0.51 app.toml template

8e56944

github-actions bot added the C:Confix Issues and PR related to Confix label Jan 14, 2024

Update toml.go

69590db

tac0turtle reviewed Jan 15, 2024

View reviewed changes

Update grpcserver.go

fde2d27

alexanderbez reviewed Jan 15, 2024

View reviewed changes

czarcas7ic added 2 commits January 20, 2024 19:15

no more config, just debug log

cf94348

Merge branch 'main' into adam/grpc-logs

50120cd

czarcas7ic changed the title ~~feat: config option to log gRPC queries~~ feat: debug log gRPC queries Jan 21, 2024

coderabbitai bot reviewed Jan 21, 2024

View reviewed changes

baseapp/grpcserver.go Show resolved Hide resolved

tac0turtle assigned alexanderbez and julienrbrt Jan 22, 2024

julienrbrt approved these changes Jan 22, 2024

View reviewed changes

alexanderbez approved these changes Jan 22, 2024

View reviewed changes

add baseapp module key to logger

1c00bb7

github-actions bot removed the C:Confix Issues and PR related to Confix label Jan 24, 2024

coderabbitai bot reviewed Jan 24, 2024

View reviewed changes

alexanderbez reviewed Jan 24, 2024

View reviewed changes

lint fix

20e1822

julienrbrt enabled auto-merge January 26, 2024 09:23

julienrbrt added this pull request to the merge queue Jan 26, 2024

Merged via the queue into main with commit 495e185 Jan 26, 2024
57 of 58 checks passed

julienrbrt deleted the adam/grpc-logs branch January 26, 2024 09:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: debug log gRPC queries #19049

feat: debug log gRPC queries #19049

czarcas7ic commented Jan 14, 2024 •

edited

Loading

coderabbitai bot commented Jan 14, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

CodeRabbit Discord Community

This comment has been minimized.

julienrbrt Jan 14, 2024

czarcas7ic Jan 14, 2024

julienrbrt Jan 14, 2024

czarcas7ic Jan 14, 2024

tac0turtle Jan 15, 2024

czarcas7ic Jan 15, 2024

tac0turtle Jan 15, 2024

czarcas7ic Jan 15, 2024

czarcas7ic Jan 15, 2024

tac0turtle Jan 17, 2024

czarcas7ic Jan 17, 2024

tac0turtle Jan 17, 2024

czarcas7ic Jan 17, 2024

alexanderbez Jan 15, 2024

czarcas7ic commented Jan 21, 2024 •

edited

Loading

julienrbrt Jan 22, 2024

alexanderbez Jan 22, 2024

julienrbrt Jan 23, 2024

alexanderbez Jan 23, 2024

czarcas7ic Jan 24, 2024

alexanderbez left a comment

coderabbitai bot Jan 24, 2024

alexanderbez Jan 24, 2024

feat: debug log gRPC queries #19049

feat: debug log gRPC queries #19049

Conversation

czarcas7ic commented Jan 14, 2024 • edited Loading

Description

Author Checklist

Reviewers Checklist

coderabbitai bot commented Jan 14, 2024 • edited Loading

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

CodeRabbit Discord Community

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

czarcas7ic commented Jan 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexanderbez left a comment

Choose a reason for hiding this comment

coderabbitai bot Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

czarcas7ic commented Jan 14, 2024 •

edited

Loading

coderabbitai bot commented Jan 14, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)

czarcas7ic commented Jan 21, 2024 •

edited

Loading