fix: skip db consolidation if no new dataset was downloaded #513
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MicaiahReid
had a problem deploying
to
Development-mainnet
GitHub Actions
Error
MicaiahReid
had a problem deploying
to
Development-testnet
GitHub Actions
Error
MicaiahReid
force-pushed
the
skip-db-consolidation
branch
2 times, most recently
from
fa7894f to
d38fdaa
Compare
MicaiahReid
had a problem deploying
to
Development-mainnet
GitHub Actions
Error
MicaiahReid
had a problem deploying
to
Development-testnet
GitHub Actions
Error
csgui
approved these changes
Mar 19, 2024
There was a problem hiding this comment.
Code LGTM.
About the mentioned tradeoff: I think that if the database is broken for any reason, is not Chainhook responsibility to fix it. But it should warn the issue somehow.
Thus, to improve the startup time - a thing that is inherent to Chainhook - my opinion is that the tradeoff of not try to fix the database is worth.
Co-authored-by: Chris Guimarães <cguimaraes.br@gmail.com>
MicaiahReid
force-pushed
the
skip-db-consolidation
branch
from
8d2b81c to
569dec2
Compare
MicaiahReid
temporarily deployed
to
Development-mainnet
GitHub Actions
Inactive
MicaiahReid
temporarily deployed
to
Development-testnet
GitHub Actions
Inactive
MicaiahReid
had a problem deploying
to
Staging-mainnet
GitHub Actions
Failure
MicaiahReid
had a problem deploying
to
Staging-testnet
GitHub Actions
Failure
MicaiahReid
added a commit
that referenced
this pull request
Mar 19, 2024
### Description Previously, when using the `chainhook predicates scan` command, Chainhook would download a very large archive file to use as a data source for the scan **every time the command was run**. The Chainhook service (`chainhook service start`), however, implements rocksdb to allow storing this dataset locally. This PR uses the rocksdb that a user may have created locally to improve performance of the `chainhook predicates scan` command. If a rocksdb doesn't exist or can't be created, Chainhook will fallback to the original method of downloading the archive. Combined with #513, this PR reduces a simple scan of a few blocks from ~20 minutes to .02 seconds. Fixes #485
MicaiahReid
added a commit
that referenced
this pull request
Mar 27, 2024
### Description The database consolidation loop doesn't need to happen if there is no new data. @lgalabru I'm interested in your thoughts on this. Potentially someone could break their stacks.rocksdb, then Chainhook wouldn't automatically fix it unless they deleted their archive to force a redownload, or a new archive becomes available. I think the tradeoff is worth it because of the improvements to startup time if users are repeatedly running `chainhook predicates scan`, or if the service goes down and needs to restart. --------- Co-authored-by: Chris Guimarães <cguimaraes.br@gmail.com>
MicaiahReid
added a commit
that referenced
this pull request
Mar 27, 2024
### Description Previously, when using the `chainhook predicates scan` command, Chainhook would download a very large archive file to use as a data source for the scan **every time the command was run**. The Chainhook service (`chainhook service start`), however, implements rocksdb to allow storing this dataset locally. This PR uses the rocksdb that a user may have created locally to improve performance of the `chainhook predicates scan` command. If a rocksdb doesn't exist or can't be created, Chainhook will fallback to the original method of downloading the archive. Combined with #513, this PR reduces a simple scan of a few blocks from ~20 minutes to .02 seconds. Fixes #485
MicaiahReid
added a commit
that referenced
this pull request
Mar 27, 2024
### Description The database consolidation loop doesn't need to happen if there is no new data. @lgalabru I'm interested in your thoughts on this. Potentially someone could break their stacks.rocksdb, then Chainhook wouldn't automatically fix it unless they deleted their archive to force a redownload, or a new archive becomes available. I think the tradeoff is worth it because of the improvements to startup time if users are repeatedly running `chainhook predicates scan`, or if the service goes down and needs to restart. --------- Co-authored-by: Chris Guimarães <cguimaraes.br@gmail.com>
MicaiahReid
added a commit
that referenced
this pull request
Mar 27, 2024
### Description Previously, when using the `chainhook predicates scan` command, Chainhook would download a very large archive file to use as a data source for the scan **every time the command was run**. The Chainhook service (`chainhook service start`), however, implements rocksdb to allow storing this dataset locally. This PR uses the rocksdb that a user may have created locally to improve performance of the `chainhook predicates scan` command. If a rocksdb doesn't exist or can't be created, Chainhook will fallback to the original method of downloading the archive. Combined with #513, this PR reduces a simple scan of a few blocks from ~20 minutes to .02 seconds. Fixes #485
github-actions bot
pushed a commit
that referenced
this pull request
Mar 27, 2024
## [1.4.0](v1.3.1...v1.4.0) (2024-03-27) ### Features * detect http / rpc errors as early as possible ([ad78669](ad78669)) * use stacks.rocksdb for predicate scan ([#514](#514)) ([a4f1663](a4f1663)), closes [#513](#513) [#485](#485) ### Bug Fixes * enable debug logs in release mode ([#537](#537)) ([fb49e28](fb49e28)) * improve error handling, and more! ([#524](#524)) ([86b5c78](86b5c78)), closes [#498](#498) [#521](#521) [#404](#404) [/github.com//issues/517#issuecomment-1992135101](https://github.com/hirosystems//github.com/hirosystems/chainhook/issues/517/issues/issuecomment-1992135101) [#517](#517) [#506](#506) [#510](#510) [#519](#519) * log errors on block download failure; implement max retries ([#503](#503)) ([0fc38cb](0fc38cb)) * **metrics:** update latest ingested block on reorg ([#515](#515)) ([8f728f7](8f728f7)) * order and filter blocks used to seed forking block pool ([#534](#534)) ([a11bc1c](a11bc1c)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](#505)) ([485394e](485394e)), closes [#487](#487) * skip db consolidation if no new dataset was downloaded ([#513](#513)) ([983a165](983a165)) * update scan status for non-triggering predicates ([#511](#511)) ([9073f42](9073f42)), closes [#498](#498)
github-actions bot
pushed a commit
that referenced
this pull request
Apr 6, 2024
## [1.5.0](v1.4.1...v1.5.0) (2024-04-06) ### Features * add brc-20 schemas for ordhook ([#551](#551)) ([1e25a8f](1e25a8f)) * detect http / rpc errors as early as possible ([e515116](e515116)) * rm blocks ([f35498d](f35498d)) * use stacks.rocksdb for predicate scan ([#514](#514)) ([0baae10](0baae10)), closes [#513](#513) [#485](#485) ### Bug Fixes * build error ([03b3938](03b3938)) * log errors on block download failure; implement max retries ([#503](#503)) ([3bfb0e1](3bfb0e1)) * **metrics:** update latest ingested block on reorg ([#515](#515)) ([823713a](823713a)) * order and filter blocks used to seed forking block pool ([#534](#534)) ([a2865b7](a2865b7)) * revisit 7+ blocks reorg handling ([#553](#553)) ([184fd07](184fd07)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](#505)) ([b77aca2](b77aca2)), closes [#487](#487) * skip db consolidation if no new dataset was downloaded ([#513](#513)) ([b1469a6](b1469a6)) * update scan status for non-triggering predicates ([#511](#511)) ([32cdfee](32cdfee)), closes [#498](#498)
|
🎉 This PR is included in version 1.5.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
vabanaerytk
added a commit
to vabanaerytk/chainhook
that referenced
this pull request
Aug 7, 2024
## [1.4.0](hirosystems/chainhook@v1.3.1...v1.4.0) (2024-03-27) ### Features * detect http / rpc errors as early as possible ([4a3e7e8](hirosystems/chainhook@4a3e7e8)) * use stacks.rocksdb for predicate scan ([#514](hirosystems/chainhook#514)) ([bff1372](hirosystems/chainhook@bff1372)), closes [#513](hirosystems/chainhook#513) [#485](hirosystems/chainhook#485) ### Bug Fixes * enable debug logs in release mode ([#537](hirosystems/chainhook#537)) ([5ff21e0](hirosystems/chainhook@5ff21e0)) * improve error handling, and more! ([#524](hirosystems/chainhook#524)) ([2cfedd7](hirosystems/chainhook@2cfedd7)), closes [#498](hirosystems/chainhook#498) [#521](hirosystems/chainhook#521) [#404](hirosystems/chainhook#404) [/github.com/hirosystems/chainhook/issues/517#issuecomment-1992135101](https://github.com/hirosystems//github.com/hirosystems/chainhook/issues/517/issues/issuecomment-1992135101) [#517](hirosystems/chainhook#517) [#506](hirosystems/chainhook#506) [#510](hirosystems/chainhook#510) [#519](hirosystems/chainhook#519) * log errors on block download failure; implement max retries ([#503](hirosystems/chainhook#503)) ([0454a4f](hirosystems/chainhook@0454a4f)) * **metrics:** update latest ingested block on reorg ([#515](hirosystems/chainhook#515)) ([7653766](hirosystems/chainhook@7653766)) * order and filter blocks used to seed forking block pool ([#534](hirosystems/chainhook#534)) ([ff775de](hirosystems/chainhook@ff775de)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](hirosystems/chainhook#505)) ([063e0e2](hirosystems/chainhook@063e0e2)), closes [#487](hirosystems/chainhook#487) * skip db consolidation if no new dataset was downloaded ([#513](hirosystems/chainhook#513)) ([f34dee3](hirosystems/chainhook@f34dee3)) * update scan status for non-triggering predicates ([#511](hirosystems/chainhook#511)) ([9647e72](hirosystems/chainhook@9647e72)), closes [#498](hirosystems/chainhook#498)
vabanaerytk
added a commit
to vabanaerytk/chainhook
that referenced
this pull request
Aug 7, 2024
## [1.5.0](hirosystems/chainhook@v1.4.1...v1.5.0) (2024-04-06) ### Features * add brc-20 schemas for ordhook ([#551](hirosystems/chainhook#551)) ([4cfa300](hirosystems/chainhook@4cfa300)) * detect http / rpc errors as early as possible ([4ef6650](hirosystems/chainhook@4ef6650)) * rm blocks ([3d57290](hirosystems/chainhook@3d57290)) * use stacks.rocksdb for predicate scan ([#514](hirosystems/chainhook#514)) ([e8bc60b](hirosystems/chainhook@e8bc60b)), closes [#513](hirosystems/chainhook#513) [#485](hirosystems/chainhook#485) ### Bug Fixes * build error ([f0fb623](hirosystems/chainhook@f0fb623)) * log errors on block download failure; implement max retries ([#503](hirosystems/chainhook#503)) ([7bd6efd](hirosystems/chainhook@7bd6efd)) * **metrics:** update latest ingested block on reorg ([#515](hirosystems/chainhook#515)) ([5a7d64e](hirosystems/chainhook@5a7d64e)) * order and filter blocks used to seed forking block pool ([#534](hirosystems/chainhook#534)) ([bb2a545](hirosystems/chainhook@bb2a545)) * revisit 7+ blocks reorg handling ([#553](hirosystems/chainhook#553)) ([3632176](hirosystems/chainhook@3632176)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](hirosystems/chainhook#505)) ([a84fc4c](hirosystems/chainhook@a84fc4c)), closes [#487](hirosystems/chainhook#487) * skip db consolidation if no new dataset was downloaded ([#513](hirosystems/chainhook#513)) ([440a8e3](hirosystems/chainhook@440a8e3)) * update scan status for non-triggering predicates ([#511](hirosystems/chainhook#511)) ([115ac28](hirosystems/chainhook@115ac28)), closes [#498](hirosystems/chainhook#498)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The database consolidation loop doesn't need to happen if there is no new data.
@lgalabru I'm interested in your thoughts on this. Potentially someone could break their stacks.rocksdb, then Chainhook wouldn't automatically fix it unless they deleted their archive to force a redownload, or a new archive becomes available.
I think the tradeoff is worth it because of the improvements to startup time if users are repeatedly running
chainhook predicates scan, or if the service goes down and needs to restart.