Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: properly init session-based node getter to find dag blocks using bitswap #314

Merged
merged 1 commit into from
Jul 9, 2022

Conversation

en0ma
Copy link
Contributor

@en0ma en0ma commented Jul 8, 2022

This PR properly init session-based node getter to find dag blocks using Bitswap when a CID is pinned using the pinning service endpoint.

Currently, the request to the pinning endpoint to pin contents will fail when it tries to walk the dag to find the blocks since it expects to use an exchange (Bitswap) and is not correctly initialized, so we have a lot of these errors;

2022-07-08 18:49:43 | {"level":"error","ts":"2022-07-08T17:49:43.160Z","logger":"pinner","caller":"pinner/pinmgr.go:308","msg":"pinning queue error: context canceled\nfailed to walk DAG\nmain.(*Shuttle).addDatabaseTrackingToContent\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1609\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1495\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nfailed to addDatabaseTrackingToContent - contID(32652720), cid(bafybeibxe2b2zr7agfsmthilepl6mb37dw2g4cfpe7x2si7jcs5qutkvfi)\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1505\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nshuttle RunPinFunc failed\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:202\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263"}
-- | --
  |   | 2022-07-08 18:49:41 | {"level":"error","ts":"2022-07-08T17:49:41.484Z","logger":"pinner","caller":"pinner/pinmgr.go:308","msg":"pinning queue error: context canceled\nfailed to walk DAG\nmain.(*Shuttle).addDatabaseTrackingToContent\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1609\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1495\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nfailed to addDatabaseTrackingToContent - contID(32652719), cid(bafybeibxe2b2zr7agfsmthilepl6mb37dw2g4cfpe7x2si7jcs5qutkvfi)\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1505\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nshuttle RunPinFunc failed\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:202\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263"}
  |   | 2022-07-08 18:49:25 | {"level":"error","ts":"2022-07-08T17:49:24.958Z","logger":"pinner","caller":"pinner/pinmgr.go:308","msg":"pinning queue error: context canceled\nfailed to walk DAG\nmain.(*Shuttle).addDatabaseTrackingToContent\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1609\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1495\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nfailed to addDatabaseTrackingToContent - contID(32652716), cid(bafybeig7lesryo2wsoslqmz6zo3djjlmqvh56s6r5wjnwp6aidlu6jjcp4)\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1505\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nshuttle RunPinFunc failed\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:202\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263"}
  |   | 2022-07-08 18:49:23 | {"level":"error","ts":"2022-07-08T17:49:23.771Z","logger":"pinner","caller":"pinner/pinmgr.go:308","msg":"pinning queue error: context canceled\nfailed to walk DAG\nmain.(*Shuttle).addDatabaseTrackingToContent\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1609\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1495\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nfailed to addDatabaseTrackingToContent - contID(32652715), cid(bafybeig7lesryo2wsoslqmz6zo3djjlmqvh56s6r5wjnwp6aidlu6jjcp4)\nmain.(*Shuttle).doPinning\n\t/home/ubuntu/estuary/cmd/estuary-shuttle/main.go:1505\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:192\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263\nshuttle RunPinFunc failed\ngithub.com/application-research/estuary/pinner.(*PinManager).doPinning\n\t/home/ubuntu/estuary/pinner/pinmgr.go:202\ngithub.com/application-research/estuary/pinner.(*PinManager).pinWorker\n\t/home/ubuntu/estuary/pinner/pinmgr.go:307\nruntime.goexit\n\t/snap/go/9854/src/runtime/asm_arm64.s:1263"}

I have tested this locally, and now works as expected.

@alvin-reyes
Copy link
Contributor

I'd like to include this fix on this weekend's release since this fixes some issues on barge. I'll run some test and approve this asap.

Copy link
Contributor

@alvin-reyes alvin-reyes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@en0ma en0ma merged commit c10ada6 into master Jul 9, 2022
@en0ma en0ma deleted the bug/properly-start-session-based-node-getter branch July 9, 2022 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants