Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transaction actions indexer #6582

Merged
merged 71 commits into from
Feb 3, 2023
Merged

Transaction actions indexer #6582

merged 71 commits into from
Feb 3, 2023

Conversation

varasev
Copy link
Contributor

@varasev varasev commented Dec 13, 2022

Motivation

This is a part of #6289. The corresponding changes for the new UI are added in blockscout/frontend#434.

Changelog

This PR contains the following features and changes:

  • an indexer for tx actions
  • an extension of API v2 to handle requests from the new UI

This PR adds support of one protocol yet (Uniswap V3). Other protocols will be added in the next PRs.

Indexing of tx actions runs with realtime and catchup block fetchers (if DISABLE_INDEXER is not true and INDEXER_TX_ACTIONS_ENABLE is true).

There is also a separate transaction_action fetcher for historical blocks which only runs if INDEXER_TX_ACTIONS_ENABLE env variable is true and the block range is defined (see env variables description below). We need it for the case when we need to index tx actions for historical (already indexed) blocks of the chain or reindex tx actions for the defined block range (in case we found a bug in tx action indexer and need to reindex the actions for some blocks and/or protocols).

Also, the PR adds 5 new environment variables:

  • INDEXER_TX_ACTIONS_ENABLE (defaults to false) - enables transaction actions indexer.
  • INDEXER_TX_ACTIONS_MAX_TOKEN_CACHE_SIZE (defaults to 100000) - sets a maximum number of items in an internal cache of tx actions indexing process (to limit memory consumption).
  • INDEXER_TX_ACTIONS_REINDEX_FIRST_BLOCK - defines the first block of a block range for historical indexing or reindexing of tx actions.
  • INDEXER_TX_ACTIONS_REINDEX_LAST_BLOCK - defines the last block of a block range for historical indexing or reindexing of tx actions. Cannot exceed the last block of a chain.
  • INDEXER_TX_ACTIONS_REINDEX_PROTOCOLS (defaults to empty string) - defines comma-separated names of protocols which should be indexed or reindexed on historical blocks defined by the range. Example: uniswap_v3,zkbob - only these protocols will be indexed or reindexed for the defined block range. If the value is empty string (or not defined), all supported protocols will be indexed/reindexed. This option is not applicable to realtime and catchup fetcher (it always indexes all supported protocols).

Checklist for your Pull Request (PR)

@varasev
Copy link
Contributor Author

varasev commented Jan 26, 2023

@nikitosing I finished working on the reviewers' comments. Now you can review as well. Thanks

@nikitosing
Copy link
Member

@varasev Can you please fix failing tests?

@varasev
Copy link
Contributor Author

varasev commented Jan 31, 2023

@varasev Can you please fix failing tests?

Do you mean the following one?

Error:      test/indexer/block/realtime/fetcher_test.exs:45
     ** (FunctionClauseError) no function clause matching in anonymous fn/2 in Indexer.Block.Realtime.FetcherTest."test Indexer.Block.Fetcher.fetch_and_import_range/1 in range with internal transactions"/1

     The following arguments were given to anonymous fn/2 in Indexer.Block.Realtime.FetcherTest."test Indexer.Block.Fetcher.fetch_and_import_range/1 in range with internal transactions"/1:
     
         # 1
         %{id: 0, jsonrpc: "2.0", method: "net_version", params: []}

I can't reproduce it locally for some reason 🤔 Trying to run:

cd apps/indexer
export MIX_ENV=test
mix test --no-start --exclude no_nethermind

Am I missing something?

@vbaranov vbaranov self-requested a review January 31, 2023 10:28
@vbaranov
Copy link
Member

@varasev Can you please fix failing tests?

Do you mean the following one?

Error:      test/indexer/block/realtime/fetcher_test.exs:45
     ** (FunctionClauseError) no function clause matching in anonymous fn/2 in Indexer.Block.Realtime.FetcherTest."test Indexer.Block.Fetcher.fetch_and_import_range/1 in range with internal transactions"/1

     The following arguments were given to anonymous fn/2 in Indexer.Block.Realtime.FetcherTest."test Indexer.Block.Fetcher.fetch_and_import_range/1 in range with internal transactions"/1:
     
         # 1
         %{id: 0, jsonrpc: "2.0", method: "net_version", params: []}

I can't reproduce it locally for some reason 🤔 Trying to run:

cd apps/indexer
export MIX_ENV=test
mix test --no-start --exclude no_nethermind

Am I missing something?

All is good. There were blinking tests. All are green after a manual restart.

@varasev
Copy link
Contributor Author

varasev commented Jan 31, 2023

I added INDEXER_TX_ACTIONS_ENABLE env variable (defaults to false) in be99681.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants