Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Access] Issues with access node and collection node communication on localnet #4728

Closed
sideninja opened this issue Sep 19, 2023 · 0 comments · Fixed by #4731
Closed

[Access] Issues with access node and collection node communication on localnet #4728

sideninja opened this issue Sep 19, 2023 · 0 comments · Fixed by #4731
Assignees
Labels
Bug Something isn't working S-Access

Comments

@sideninja
Copy link
Contributor

sideninja commented Sep 19, 2023

While testing master on localnet I got a weird crash on AN, that seemed like it couldn’t communicate with the collection node, but checking the collection node it seems fine, no crashes there.

A few lines of logs from AN before the crash were:

localnet-access_1-1  | {"level":"info","node_role":"access","node_id":"0c0ddfd34afee7b93bd1bc030d549cc009cb4f9ba003740135166178193eb37b","error":"failed to send transaction to collection node at collection_1:2137: rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2023-09-19T10:49:33.957386221Z","message":"failed to send transactions  to collector nodes"}
localnet-access_1-1  | {"level":"warn","node_role":"access","node_id":"0c0ddfd34afee7b93bd1bc030d549cc009cb4f9ba003740135166178193eb37b","component":"grpc_server","grpc.method":"SendTransaction","grpc.method_type":"unary","grpc.service":"flow.access.AccessAPI","grpc.start_time":"2023-09-19T10:48:06Z","kind":"server","system":"grpc","peer.address":"192.168.65.1:33347","grpc.code":"DeadlineExceeded","error":"rpc error: code = DeadlineExceeded desc = failed to send transaction to a collection node: 3 errors occurred:\n\t* failed to send transaction to collection node at collection_2:2137: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n\t* failed to send transaction to collection node at collection_1:2137: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n\t* failed to send transaction to collection node at collection_3:2137: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n\n","grpc.time_ms":"86445.31","time":"2023-09-19T10:49:34.639178888Z","message":"finished server unary call"}
localnet-access_1-1  | {"level":"debug","node_role":"access","node_id":"0c0ddfd34afee7b93bd1bc030d549cc009cb4f9ba003740135166178193eb37b","grpc_conn_evicted":"collection_3:9000","time":"2023-09-19T10:49:34.647372846Z","message":"closing grpc connection evicted from pool"}
localnet-access_1-1  | panic: runtime error: invalid memory address or nil pointer dereference
localnet-access_1-1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x883940]
localnet-access_1-1  |
localnet-access_1-1  | goroutine 294787 [running]:
localnet-access_1-1  | google.golang.org/grpc.(*ClientConn).Close(0x0)
localnet-access_1-1  | 	/go/pkg/mod/google.golang.org/grpc@v1.55.0/clientconn.go:1019 +0x30
localnet-access_1-1  | github.com/onflow/flow-go/engine/access/rpc/connection.(*CachedClient).Close.func1()
localnet-access_1-1  | 	/app/engine/access/rpc/connection/cache.go:35 +0x38
localnet-access_1-1  | created by github.com/onflow/flow-go/engine/access/rpc/connection.(*CachedClient).Close
localnet-access_1-1  | 	/app/engine/access/rpc/connection/cache.go:31 +0xb0
localnet-access_1-1 exited with code 2

The latest master was used with commit hash 5c927af0993a48e578713e78351a4e68314442d5

I could reproduce this issue if I ran a simulation of transactions:

make load -log-level info -tps 1,10 -tps-durations 30s
@sideninja sideninja changed the title ] [AN] Issues with access node and collection node communication on localnet Sep 19, 2023
@sideninja sideninja added Bug Something isn't working S-Access labels Sep 19, 2023
@peterargue peterargue changed the title [AN] Issues with access node and collection node communication on localnet [Access] Issues with access node and collection node communication on localnet Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working S-Access
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants