feat: use stacks.rocksdb for predicate scan #514
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MicaiahReid
had a problem deploying
to
Development-mainnet
GitHub Actions
Error
MicaiahReid
had a problem deploying
to
Development-testnet
GitHub Actions
Error
Whoa nicely done @MicaiahReid 👏🏽 |
MicaiahReid
requested review from
csgui and
lgalabru
and removed request for
lgalabru
MicaiahReid
force-pushed
the
use-rocksdb-scan
branch
2 times, most recently
from
aa550ec to
d648d5a
Compare
MicaiahReid
had a problem deploying
to
Development-mainnet
GitHub Actions
Error
MicaiahReid
had a problem deploying
to
Development-testnet
GitHub Actions
Error
csgui
reviewed
Mar 19, 2024
|
@MicaiahReid PR looks good. But just added a comment of a thing that was bugging me. Waiting your feedback on that before finishing the review of this PR. Thanks! |
MicaiahReid
had a problem deploying
to
Development-mainnet
GitHub Actions
Error
MicaiahReid
had a problem deploying
to
Development-testnet
GitHub Actions
Error
MicaiahReid
force-pushed
the
use-rocksdb-scan
branch
from
5177479 to
7856c3e
Compare
MicaiahReid
had a problem deploying
to
Development-mainnet
GitHub Actions
Failure
MicaiahReid
had a problem deploying
to
Development-testnet
GitHub Actions
Failure
MicaiahReid
added a commit
that referenced
this pull request
Mar 27, 2024
### Description Previously, when using the `chainhook predicates scan` command, Chainhook would download a very large archive file to use as a data source for the scan **every time the command was run**. The Chainhook service (`chainhook service start`), however, implements rocksdb to allow storing this dataset locally. This PR uses the rocksdb that a user may have created locally to improve performance of the `chainhook predicates scan` command. If a rocksdb doesn't exist or can't be created, Chainhook will fallback to the original method of downloading the archive. Combined with #513, this PR reduces a simple scan of a few blocks from ~20 minutes to .02 seconds. Fixes #485
MicaiahReid
added a commit
that referenced
this pull request
Mar 27, 2024
### Description Previously, when using the `chainhook predicates scan` command, Chainhook would download a very large archive file to use as a data source for the scan **every time the command was run**. The Chainhook service (`chainhook service start`), however, implements rocksdb to allow storing this dataset locally. This PR uses the rocksdb that a user may have created locally to improve performance of the `chainhook predicates scan` command. If a rocksdb doesn't exist or can't be created, Chainhook will fallback to the original method of downloading the archive. Combined with #513, this PR reduces a simple scan of a few blocks from ~20 minutes to .02 seconds. Fixes #485
github-actions bot
pushed a commit
that referenced
this pull request
Mar 27, 2024
## [1.4.0](v1.3.1...v1.4.0) (2024-03-27) ### Features * detect http / rpc errors as early as possible ([ad78669](ad78669)) * use stacks.rocksdb for predicate scan ([#514](#514)) ([a4f1663](a4f1663)), closes [#513](#513) [#485](#485) ### Bug Fixes * enable debug logs in release mode ([#537](#537)) ([fb49e28](fb49e28)) * improve error handling, and more! ([#524](#524)) ([86b5c78](86b5c78)), closes [#498](#498) [#521](#521) [#404](#404) [/github.com//issues/517#issuecomment-1992135101](https://github.com/hirosystems//github.com/hirosystems/chainhook/issues/517/issues/issuecomment-1992135101) [#517](#517) [#506](#506) [#510](#510) [#519](#519) * log errors on block download failure; implement max retries ([#503](#503)) ([0fc38cb](0fc38cb)) * **metrics:** update latest ingested block on reorg ([#515](#515)) ([8f728f7](8f728f7)) * order and filter blocks used to seed forking block pool ([#534](#534)) ([a11bc1c](a11bc1c)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](#505)) ([485394e](485394e)), closes [#487](#487) * skip db consolidation if no new dataset was downloaded ([#513](#513)) ([983a165](983a165)) * update scan status for non-triggering predicates ([#511](#511)) ([9073f42](9073f42)), closes [#498](#498)
github-actions bot
pushed a commit
that referenced
this pull request
Apr 6, 2024
## [1.5.0](v1.4.1...v1.5.0) (2024-04-06) ### Features * add brc-20 schemas for ordhook ([#551](#551)) ([1e25a8f](1e25a8f)) * detect http / rpc errors as early as possible ([e515116](e515116)) * rm blocks ([f35498d](f35498d)) * use stacks.rocksdb for predicate scan ([#514](#514)) ([0baae10](0baae10)), closes [#513](#513) [#485](#485) ### Bug Fixes * build error ([03b3938](03b3938)) * log errors on block download failure; implement max retries ([#503](#503)) ([3bfb0e1](3bfb0e1)) * **metrics:** update latest ingested block on reorg ([#515](#515)) ([823713a](823713a)) * order and filter blocks used to seed forking block pool ([#534](#534)) ([a2865b7](a2865b7)) * revisit 7+ blocks reorg handling ([#553](#553)) ([184fd07](184fd07)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](#505)) ([b77aca2](b77aca2)), closes [#487](#487) * skip db consolidation if no new dataset was downloaded ([#513](#513)) ([b1469a6](b1469a6)) * update scan status for non-triggering predicates ([#511](#511)) ([32cdfee](32cdfee)), closes [#498](#498)
|
🎉 This PR is included in version 1.5.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
vabanaerytk
added a commit
to vabanaerytk/chainhook
that referenced
this pull request
Aug 7, 2024
## [1.4.0](hirosystems/chainhook@v1.3.1...v1.4.0) (2024-03-27) ### Features * detect http / rpc errors as early as possible ([4a3e7e8](hirosystems/chainhook@4a3e7e8)) * use stacks.rocksdb for predicate scan ([#514](hirosystems/chainhook#514)) ([bff1372](hirosystems/chainhook@bff1372)), closes [#513](hirosystems/chainhook#513) [#485](hirosystems/chainhook#485) ### Bug Fixes * enable debug logs in release mode ([#537](hirosystems/chainhook#537)) ([5ff21e0](hirosystems/chainhook@5ff21e0)) * improve error handling, and more! ([#524](hirosystems/chainhook#524)) ([2cfedd7](hirosystems/chainhook@2cfedd7)), closes [#498](hirosystems/chainhook#498) [#521](hirosystems/chainhook#521) [#404](hirosystems/chainhook#404) [/github.com/hirosystems/chainhook/issues/517#issuecomment-1992135101](https://github.com/hirosystems//github.com/hirosystems/chainhook/issues/517/issues/issuecomment-1992135101) [#517](hirosystems/chainhook#517) [#506](hirosystems/chainhook#506) [#510](hirosystems/chainhook#510) [#519](hirosystems/chainhook#519) * log errors on block download failure; implement max retries ([#503](hirosystems/chainhook#503)) ([0454a4f](hirosystems/chainhook@0454a4f)) * **metrics:** update latest ingested block on reorg ([#515](hirosystems/chainhook#515)) ([7653766](hirosystems/chainhook@7653766)) * order and filter blocks used to seed forking block pool ([#534](hirosystems/chainhook#534)) ([ff775de](hirosystems/chainhook@ff775de)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](hirosystems/chainhook#505)) ([063e0e2](hirosystems/chainhook@063e0e2)), closes [#487](hirosystems/chainhook#487) * skip db consolidation if no new dataset was downloaded ([#513](hirosystems/chainhook#513)) ([f34dee3](hirosystems/chainhook@f34dee3)) * update scan status for non-triggering predicates ([#511](hirosystems/chainhook#511)) ([9647e72](hirosystems/chainhook@9647e72)), closes [#498](hirosystems/chainhook#498)
vabanaerytk
added a commit
to vabanaerytk/chainhook
that referenced
this pull request
Aug 7, 2024
## [1.5.0](hirosystems/chainhook@v1.4.1...v1.5.0) (2024-04-06) ### Features * add brc-20 schemas for ordhook ([#551](hirosystems/chainhook#551)) ([4cfa300](hirosystems/chainhook@4cfa300)) * detect http / rpc errors as early as possible ([4ef6650](hirosystems/chainhook@4ef6650)) * rm blocks ([3d57290](hirosystems/chainhook@3d57290)) * use stacks.rocksdb for predicate scan ([#514](hirosystems/chainhook#514)) ([e8bc60b](hirosystems/chainhook@e8bc60b)), closes [#513](hirosystems/chainhook#513) [#485](hirosystems/chainhook#485) ### Bug Fixes * build error ([f0fb623](hirosystems/chainhook@f0fb623)) * log errors on block download failure; implement max retries ([#503](hirosystems/chainhook#503)) ([7bd6efd](hirosystems/chainhook@7bd6efd)) * **metrics:** update latest ingested block on reorg ([#515](hirosystems/chainhook#515)) ([5a7d64e](hirosystems/chainhook@5a7d64e)) * order and filter blocks used to seed forking block pool ([#534](hirosystems/chainhook#534)) ([bb2a545](hirosystems/chainhook@bb2a545)) * revisit 7+ blocks reorg handling ([#553](hirosystems/chainhook#553)) ([3632176](hirosystems/chainhook@3632176)) * seed forking handler with unconfirmed blocks to improve startup stability ([#505](hirosystems/chainhook#505)) ([a84fc4c](hirosystems/chainhook@a84fc4c)), closes [#487](hirosystems/chainhook#487) * skip db consolidation if no new dataset was downloaded ([#513](hirosystems/chainhook#513)) ([440a8e3](hirosystems/chainhook@440a8e3)) * update scan status for non-triggering predicates ([#511](hirosystems/chainhook#511)) ([115ac28](hirosystems/chainhook@115ac28)), closes [#498](hirosystems/chainhook#498)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Previously, when using the
chainhook predicates scancommand, Chainhook would download a very large archive file to use as a data source for the scan every time the command was run.The Chainhook service (
chainhook service start), however, implements rocksdb to allow storing this dataset locally.This PR uses the rocksdb that a user may have created locally to improve performance of the
chainhook predicates scancommand. If a rocksdb doesn't exist or can't be created, Chainhook will fallback to the original method of downloading the archive.Combined with #513, this PR reduces a simple scan of a few blocks from ~20 minutes to .02 seconds.
Fixes #485