Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry on cid verification #3831

Merged
merged 1 commit into from
Sep 7, 2022
Merged

Retry on cid verification #3831

merged 1 commit into from
Sep 7, 2022

Conversation

vicky-g
Copy link
Contributor

@vicky-g vicky-g commented Sep 6, 2022

Description

Sometimes, writing CIDs get corrupted. We should retry on these errors instead of bailing. This PR is to retry if cid verification fails

Tests

No change in functionality, just adding retries to this flow

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

Monitor the log CID does not match what is expected to be

Copy link
Contributor

@SidSethi SidSethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

@vicky-g vicky-g merged commit cc6f005 into master Sep 7, 2022
@vicky-g vicky-g deleted the vg-retry-on-cid-verification branch September 7, 2022 04:09
audius-infra pushed a commit that referenced this pull request Sep 8, 2022
## Changelog

- 2022-09-08 [1c8ea27] [C-991] Add entity-manager to native-libs (#3842) [Dylan Jeffers]
- 2022-09-08 [be3521b] Split Up `legacyUtil.js` (#3834) [Johannes Naylor]
- 2022-09-08 [108ee77] [CON-332] Refactor ContentNodeInfoManager and consumers for redis caching (#3819) [Theo Ilie]
- 2022-09-08 [4911903] [PAY-487] DN - Handle premium track access for track apis (#3783) [Saliou Diallo]
- 2022-09-07 [9171f7b] [CON-353] Add tracing visualization (#3837) [Johannes Naylor]
- 2022-09-07 [a6e3a54] Fix prune plays test cutoff time (#3839) [Isaac Solo]
- 2022-09-07 [8cdf4f3] Simplify EM social feature deletes to not rely on existing records (#3816) [Isaac Solo]
- 2022-09-07 [f2c4fc1] Add entity manager social features to libs (#3814) [Isaac Solo]
- 2022-09-07 [600fdeb] Add coalesce for release date sort for favorites query (#3836) [Kyle Shanks]
- 2022-09-07 [97bf563] Gate reconfig of primary to node endpoint changing (#3830) [Theo Ilie]
- 2022-09-07 [4ef1d2b] Bump to version 0.3.66 (#3835) [Cheran]
- 2022-09-06 [7ca4fc2] CON-358 - Skip delisted content and do not fail sync jobs anymore (#3765) [vicky :)]
- 2022-09-06 [cc6f005] Retry on cid verification in findCIDInNetwork (#3831) [vicky :)]
- 2022-09-06 [e99e788] Update findCIDInNetwork (#3825) [vicky :)]
- 2022-09-06 [0c3b906] Optimize Entity Manager DB fetch (#3829) [Isaac Solo]
- 2022-09-06 [e90c7ed] Fix duplicate export causing TypeError (#3827) [Theo Ilie]
- 2022-09-06 [aef1b30] Increase recurring sync queue waiting size from 1000 to 10000 (#3828) [Dheeraj Manjunath]
- 2022-09-06 [93cb0b8] Better logging after syncs (#3820) [Dheeraj Manjunath]
- 2022-09-06 [b09a612] Update random sp selection for reconfig (#3824) [vicky :)]
- 2022-09-06 [0a13201] remove utils.js (#3823) [Johannes Naylor]
- 2022-09-02 [4eca681] fix inconsistency with trailing zero (#3758) [Joseph Lee]
- 2022-09-02 [0b85e0c] Provide generateRecoveryLink status (#3817) [Dylan Jeffers]
- 2022-09-02 [12ee57d] [CON-284] Instrumentation of State Machine Queues (#3782) [Johannes Naylor]
- 2022-09-02 [cfe5fb2] [CON-284] Instrumentation of AsyncProcessingQueue (#3781) [Johannes Naylor]
- 2022-09-02 [9944c4b] Let orphaned data recovery run 24/7 (#3815) [Theo Ilie]
- 2022-09-02 [7e5bab1] Add social features to entity manager (#3795) [Isaac Solo]
- 2022-09-02 [c4ca932] Fix registration on dev setup (#3813) [Cheran]
- 2022-09-01 [40de44b] INF-224 Set ${PROTOCOL_DIR} within provision-dev-env.sh (#3811) [Joaquin Casares]
- 2022-09-01 [83394df] Fix get tracks by removing shared cache (#3810) [Isaac Solo]
- 2022-09-01 [3a9227b] [PAY-549] Write out IP with each call to relay (#3798) [Michael Piazza]
- 2022-09-01 [cf99a87] Clear write locks on init (#3809) [vicky :)]
- 2022-09-01 [11783e3] Update replica set and sync concurrency (#3808) [Dheeraj Manjunath]
- 2022-09-01 [de6fc7d] Add a libs function to save transaction metadata (#3804) [Marcus Pasell]
- 2022-09-01 [a475c31] Make queues limit how many jobs they add to each other (#3807) [Theo Ilie]
- 2022-09-01 [aabd49a] [PAY-587] Add transaction metadata to identity for transaction details (#3773) [Marcus Pasell]
- 2022-09-01 [cd0bb38] Fix typo in enum value (#3806) [Johannes Naylor]
- 2022-09-01 [a7ddb23] remove unneeded mount, since we don't load these modules (#3805) [Joaquin Casares]
- 2022-09-01 [f26f4a4] Better get_feed_es error logging.  Fix fetch related saves + reposts (#3797) [Steve Perkins]
- 2022-08-31 [5fc8f62] Add entity manager metrics (#3767) [Isaac Solo]
- 2022-08-31 [e85b8b1] Bump try higher so that lock is always released (#3803) [vicky :)]
- 2022-08-31 [ab26cf1] INF-227 Release Grafana Alerts to Production (#3772) [Joaquin Casares]
- 2022-08-31 [424f2d1] CON-380 CN make primarySyncFromSecondary work for multi-page exports against nodes without export bugfix (#3796) [Sid Sethi]
- 2022-08-31 [5273bf5] Lower orphaned data recovery sync reqs/sec (#3799) [Theo Ilie]
- 2022-08-31 [411b86b] [CON-284] Instrument tracing on syncQueue and immediateSyncQueue (#3780) [Johannes Naylor]
- 2022-08-31 [ff85889] Bump sdk to v1.0.1 [audius-infra]
audius-infra pushed a commit that referenced this pull request Sep 8, 2022
## Changelog

- 2022-09-08 [1c8ea27] [C-991] Add entity-manager to native-libs (#3842) [Dylan Jeffers]
- 2022-09-08 [be3521b] Split Up `legacyUtil.js` (#3834) [Johannes Naylor]
- 2022-09-08 [108ee77] [CON-332] Refactor ContentNodeInfoManager and consumers for redis caching (#3819) [Theo Ilie]
- 2022-09-08 [4911903] [PAY-487] DN - Handle premium track access for track apis (#3783) [Saliou Diallo]
- 2022-09-07 [9171f7b] [CON-353] Add tracing visualization (#3837) [Johannes Naylor]
- 2022-09-07 [a6e3a54] Fix prune plays test cutoff time (#3839) [Isaac Solo]
- 2022-09-07 [8cdf4f3] Simplify EM social feature deletes to not rely on existing records (#3816) [Isaac Solo]
- 2022-09-07 [f2c4fc1] Add entity manager social features to libs (#3814) [Isaac Solo]
- 2022-09-07 [600fdeb] Add coalesce for release date sort for favorites query (#3836) [Kyle Shanks]
- 2022-09-07 [97bf563] Gate reconfig of primary to node endpoint changing (#3830) [Theo Ilie]
- 2022-09-07 [4ef1d2b] Bump to version 0.3.66 (#3835) [Cheran]
- 2022-09-06 [7ca4fc2] CON-358 - Skip delisted content and do not fail sync jobs anymore (#3765) [vicky :)]
- 2022-09-06 [cc6f005] Retry on cid verification in findCIDInNetwork (#3831) [vicky :)]
- 2022-09-06 [e99e788] Update findCIDInNetwork (#3825) [vicky :)]
- 2022-09-06 [0c3b906] Optimize Entity Manager DB fetch (#3829) [Isaac Solo]
- 2022-09-06 [e90c7ed] Fix duplicate export causing TypeError (#3827) [Theo Ilie]
- 2022-09-06 [aef1b30] Increase recurring sync queue waiting size from 1000 to 10000 (#3828) [Dheeraj Manjunath]
- 2022-09-06 [93cb0b8] Better logging after syncs (#3820) [Dheeraj Manjunath]
- 2022-09-06 [b09a612] Update random sp selection for reconfig (#3824) [vicky :)]
- 2022-09-06 [0a13201] remove utils.js (#3823) [Johannes Naylor]
- 2022-09-02 [4eca681] fix inconsistency with trailing zero (#3758) [Joseph Lee]
- 2022-09-02 [0b85e0c] Provide generateRecoveryLink status (#3817) [Dylan Jeffers]
- 2022-09-02 [12ee57d] [CON-284] Instrumentation of State Machine Queues (#3782) [Johannes Naylor]
- 2022-09-02 [cfe5fb2] [CON-284] Instrumentation of AsyncProcessingQueue (#3781) [Johannes Naylor]
- 2022-09-02 [9944c4b] Let orphaned data recovery run 24/7 (#3815) [Theo Ilie]
- 2022-09-02 [7e5bab1] Add social features to entity manager (#3795) [Isaac Solo]
- 2022-09-02 [c4ca932] Fix registration on dev setup (#3813) [Cheran]
- 2022-09-01 [40de44b] INF-224 Set ${PROTOCOL_DIR} within provision-dev-env.sh (#3811) [Joaquin Casares]
- 2022-09-01 [83394df] Fix get tracks by removing shared cache (#3810) [Isaac Solo]
- 2022-09-01 [3a9227b] [PAY-549] Write out IP with each call to relay (#3798) [Michael Piazza]
- 2022-09-01 [cf99a87] Clear write locks on init (#3809) [vicky :)]
- 2022-09-01 [11783e3] Update replica set and sync concurrency (#3808) [Dheeraj Manjunath]
- 2022-09-01 [de6fc7d] Add a libs function to save transaction metadata (#3804) [Marcus Pasell]
- 2022-09-01 [a475c31] Make queues limit how many jobs they add to each other (#3807) [Theo Ilie]
- 2022-09-01 [aabd49a] [PAY-587] Add transaction metadata to identity for transaction details (#3773) [Marcus Pasell]
- 2022-09-01 [cd0bb38] Fix typo in enum value (#3806) [Johannes Naylor]
- 2022-09-01 [a7ddb23] remove unneeded mount, since we don't load these modules (#3805) [Joaquin Casares]
- 2022-09-01 [f26f4a4] Better get_feed_es error logging.  Fix fetch related saves + reposts (#3797) [Steve Perkins]
- 2022-08-31 [5fc8f62] Add entity manager metrics (#3767) [Isaac Solo]
- 2022-08-31 [e85b8b1] Bump try higher so that lock is always released (#3803) [vicky :)]
- 2022-08-31 [ab26cf1] INF-227 Release Grafana Alerts to Production (#3772) [Joaquin Casares]
- 2022-08-31 [424f2d1] CON-380 CN make primarySyncFromSecondary work for multi-page exports against nodes without export bugfix (#3796) [Sid Sethi]
- 2022-08-31 [5273bf5] Lower orphaned data recovery sync reqs/sec (#3799) [Theo Ilie]
- 2022-08-31 [411b86b] [CON-284] Instrument tracing on syncQueue and immediateSyncQueue (#3780) [Johannes Naylor]
- 2022-08-31 [ff85889] Bump sdk to v1.0.1 [audius-infra]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants