Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

don't run filtered tmp processor when reingestion is enabled #3

Closed

Conversation

sreuland
Copy link

when a reingestion range is being run, the processor runner was still running TransactionFilteredTmpProcessor which is inserting and deleting tx's to history_transactions_filtered_tmp on every flush. This step is un-necessary during reingestion since TransactionFilteredTmpProcessor` only needs to be maintained with transactions from 'forward' ingestion to support tx sub queue processing.

In addtion, during some reingestion runs with parallel workers, observed some long running/hanging DELETE stmts left on the db server which were slowing down overall reingestion workers.

postgres=# SELECT
  pid,
  now() - pg_stat_activity.query_start AS duration,
  query,
  state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';
  pid   |    duration     |                                                query                                                 | state
--------+-----------------+------------------------------------------------------------------------------------------------------+--------
 954110 | 17:10:23.48803  | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954342 | 17:08:26.893771 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954457 | 17:09:19.293392 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954539 | 17:07:52.870689 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954540 | 17:08:40.926155 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954543 | 17:08:24.030046 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954564 | 17:07:58.474278 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active
 954565 | 17:07:59.878985 | DELETE FROM history_transactions_filtered_tmp WHERE now() >= (created_at + interval '1 second' * $1) | active

this behavior may be related to the deadlocks situation that was noted and resolved by stellar#5263.

This pr proposes additional changes related to reingestion sql performance on top of the deadlock fix in stellar#5263:

  • Removed EnableIngestionFiltering flag and global config variable, it is obsolete, as ingestion filter was permanently enabled, the flag provides no value.
  • repoint conditional checks on running TransactionFilteredTmpProcessor to reference ReingestionEnabled instead of EnableIngestionFiltering, this allows the parallel reingestion workers to skip doing that, thus avoiding that additional sql activity

@sreuland
Copy link
Author

closing as will retarget this to master

@sreuland sreuland closed this Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant