Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: indexes to optimize principal-based etag db lookups #2157

Merged
merged 3 commits into from
Nov 5, 2024

Conversation

zone117x
Copy link
Member

@zone117x zone117x commented Nov 4, 2024

Possible fix for #2147

In version v8.1.0, several endpoints (in particular the balance endpoints) were updated to use new etag caching handling. This change was actually an optimization to reduce db load, however, it performs a sql query that is very CPU intensive for postgres deployments without very large memory cache configurations.

This PR adds composite indexes which should be less CPU intensive than the bitmap scans required by the previously used indexes.

I was able to reproduce the slow query perf on a small-ish vm, and verify that this PR significantly improves the query:

// before PR
Planning Time: 4.330 ms
Execution Time: 774.455 ms

// after PR
Planning Time: 0.851 ms
Execution Time: 0.399 ms

Using this sample query:

    WITH activity AS (
      (
        SELECT '0x' || encode(tx_id, 'hex') AS tx_id
        FROM principal_stx_txs
        WHERE principal = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9' AND canonical = true AND microblock_canonical = true
        ORDER BY block_height DESC, microblock_sequence DESC, tx_index DESC
        LIMIT 1
      )
      UNION
      (
        SELECT '0x' || encode(tx_id, 'hex') AS tx_id
        FROM (
          (
            SELECT tx_id, block_height, microblock_sequence, tx_index, event_index
            FROM ft_events
            WHERE sender = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9'
              AND canonical = true
              AND microblock_canonical = true
            ORDER BY block_height DESC, microblock_sequence DESC, tx_index DESC, event_index DESC
            LIMIT 1
          )
          UNION ALL
          (
            SELECT tx_id, block_height, microblock_sequence, tx_index, event_index
            FROM ft_events
            WHERE recipient = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9'
              AND canonical = true
              AND microblock_canonical = true
            ORDER BY block_height DESC, microblock_sequence DESC, tx_index DESC, event_index DESC
            LIMIT 1
          )
        ) AS combined
        ORDER BY block_height DESC, microblock_sequence DESC, tx_index DESC, event_index DESC
        LIMIT 1
      )
      UNION
      (
        SELECT '0x' || encode(tx_id, 'hex') AS tx_id
        FROM nft_events
        WHERE (sender = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9' OR recipient = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9')
          AND canonical = true
          AND microblock_canonical = true
        ORDER BY block_height DESC, microblock_sequence DESC, tx_index DESC, event_index DESC
        LIMIT 1
      )
     UNION
      (
        SELECT 'mempool-' || '0x' || encode(tx_id, 'hex') AS tx_id
        FROM mempool_txs
        WHERE pruned = false AND
          (sender_address = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9'
          OR sponsor_address = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9'
          OR token_transfer_recipient_address = 'SP2ADT4TAV92SPBNP3WE9GPJ12F6CKGWMEJ1H1BB9')
        ORDER BY receipt_time DESC, sender_address DESC, nonce DESC
        LIMIT 1
      )
    )
    SELECT tx_id FROM activity WHERE tx_id IS NOT NULL;

Copy link

github-actions bot commented Nov 4, 2024

Vercel deployment URL: https://stacks-blockchain-fkydrznin-hirosystems.vercel.app 🚀

@zone117x zone117x merged commit 9da4dcd into master Nov 5, 2024
54 checks passed
@zone117x zone117x deleted the fix/principal-etag-pg branch November 5, 2024 08:02
blockstack-devops pushed a commit that referenced this pull request Nov 5, 2024
## [8.2.1](v8.2.0...v8.2.1) (2024-11-05)

### Bug Fixes

* indexes to optimize principal-based etag db lookups ([#2157](#2157)) ([9da4dcd](9da4dcd))
@blockstack-devops
Copy link

🎉 This PR is included in version 8.2.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants