perf(api): optimise count retrieval #590

PJColombo · 2024-10-09T19:19:39Z

Checklist

My change requires a documentation update, and I have done it.
I have added tests to cover my changes.
I have filled out the description and linked the related issues.

Description

It optimizes the retrieval of element counts when filters are applied. By obtaining the total number of elements from the stats tables—specifically when filtering by rollup, category, or timestamp—we can utilize data from the overall or daily stats tables.

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

… to daily date

…ollup, or timestamp filters only Enhances performance by fetching counts directly from the stats table when counting elements with filters applied exclusively to 'category', 'rollup', or 'timestamp'.

vercel · 2024-10-09T19:19:40Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
blobscan-staging	❌ Failed (Inspect)			Oct 31, 2024 4:18pm

5 Skipped Deployments

Name	Status	Preview	Updated (UTC)
blobscan-docs	⬜️ Ignored (Inspect)	Visit Preview	Oct 31, 2024 4:18pm
blobscan-gnosis	⬜️ Ignored (Inspect)	Visit Preview	Oct 31, 2024 4:18pm
blobscan-holesky	⬜️ Ignored (Inspect)	Visit Preview	Oct 31, 2024 4:18pm
blobscan-mainnet	⬜️ Ignored (Inspect)	Visit Preview	Oct 31, 2024 4:18pm
blobscan-sepolia	⬜️ Ignored (Inspect)	Visit Preview	Oct 31, 2024 4:18pm

changeset-bot · 2024-10-09T19:19:43Z

🦋 Changeset detected

Latest commit: 12bb989

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@blobscan/api	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2024-10-09T19:21:52Z

📦 Next.js Bundle Analysis for @blobscan/web

This analysis was generated by the Next.js Bundle Analysis action. 🤖

⚠️ Global Bundle Size Increased

Page	Size (compressed)
`global`	`332.94 KB` (🔴 +87.32 KB)

Details

The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!

New Pages Added

The following pages were added to the bundle from the code in this PR:

Page	Size (compressed)	First Load	% of Budget (`350 KB`)
`/block_neighbor`	`251 B`	333.19 KB	95.20%
`/stats`	`347.54 KB`	680.48 KB	194.42%

Eight Pages Changed Size

The following pages changed size from the code in this PR compared to its base branch:

Page	Size (compressed)	First Load	% of Budget (`350 KB`)
`/`	`351.65 KB`	684.59 KB	195.60% (🟢 -5.06%)
`/address/[address]`	`26.8 KB`	359.74 KB	102.78% (🟢 -4.83%)
`/blob/[hash]`	`27.5 KB`	360.44 KB	102.98% (🟢 -4.26%)
`/blobs`	`76.74 KB`	409.68 KB	117.05% (🟡 +12.55%)
`/block/[id]`	`14.17 KB`	347.11 KB	99.17% (🟢 -4.19%)
`/blocks`	`74.44 KB`	407.38 KB	116.40% (🟡 +12.53%)
`/tx/[hash]`	`16.61 KB`	349.55 KB	99.87% (🟡 +0.96%)
`/txs`	`73.94 KB`	406.88 KB	116.25% (🟡 +10.53%)

Details

Only the gzipped size is provided here based on an expert tip.

First Load is the size of the global bundle plus the bundle for the individual page. If a user were to show up to your website and land on a given page, the first load size represents the amount of javascript that user would need to download. If next/link is used, subsequent page loads would only need to download that page's bundle (the number in the "Size" column), since the global bundle has already been downloaded.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

The "Budget %" column shows what percentage of your performance budget the First Load total takes up. For example, if your budget was 100kb, and a given page's first load size was 10kb, it would be 10% of your budget. You can also see how much this has increased or decreased compared to the base branch of your PR. If this percentage has increased by 20% or more, there will be a red status indicator applied, indicating that special attention should be given to this. If you see "+/- <0.01%" it means that there was a change in bundle size, but it is a trivial enough amount that it can be ignored.

codecov · 2024-10-09T19:22:37Z

Codecov Report

Attention: Patch coverage is 98.66667% with 3 lines in your changes missing coverage. Please review.

Project coverage is 89.09%. Comparing base (7cd4de1) to head (12bb989).
Report is 42 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/db/prisma/extensions/stats.ts	70.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #590      +/-   ##
==========================================
- Coverage   89.31%   89.09%   -0.23%     
==========================================
  Files         153      155       +2     
  Lines       10190    10423     +233     
  Branches     1141     1170      +29     
==========================================
+ Hits         9101     9286     +185     
- Misses       1089     1137      +48

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

packages/api/src/utils/count.ts

xFJA

Nice work!! 💯

packages/api/src/routers/blob/getCount.ts

packages/db/prisma/extensions/stats.ts

packages/api/test/test-suites/filters.ts

packages/api/test/tx.test.ts

packages/api/src/routers/blob/getCount.ts

packages/api/src/routers/tx/getCount.ts

packages/api/src/utils/count.ts

PabloCastellano · 2024-10-17T05:38:02Z

packages/dayjs/src/date.ts

+
+  if (from) {
+    datePeriod.from = toDailyDate(from, "startOf").toDate();
+  }


Suggested change

}

} else {

datePeriod.from = MIN_DATE;

}

PabloCastellano · 2024-10-17T05:38:58Z

packages/dayjs/src/date.ts

+
+  if (to) {
+    datePeriod.to = toDailyDate(to, "endOf").toDate();
+  }


Suggested change

}

} else {

datePeriod.to = new Date();

}

Co-authored-by: Pablo Castellano <pablo@anche.no>

…nt from stats table when applicable

PJColombo added 2 commits October 9, 2024 15:18

fix(dayjs): do not return default value when trying to convert a date…

07f48ed

… to daily date

perf(api): optimize count retrieval using stats table for category, r…

f3ade4b

…ollup, or timestamp filters only Enhances performance by fetching counts directly from the stats table when counting elements with filters applied exclusively to 'category', 'rollup', or 'timestamp'.

vercel bot deployed to Preview – blobscan-staging October 9, 2024 19:22 View deployment

xFJA reviewed Oct 12, 2024

View reviewed changes

packages/api/src/utils/count.ts Outdated Show resolved Hide resolved

xFJA approved these changes Oct 12, 2024

View reviewed changes

PJColombo mentioned this pull request Oct 12, 2024

feat(web): add multiple options to RollupFilter #592

Merged

3 tasks

PabloCastellano reviewed Oct 17, 2024

View reviewed changes

vercel bot deployed to Preview – blobscan-staging October 29, 2024 00:34 View deployment

style(api): add explanatory comments

3efea7f

Co-authored-by: Pablo Castellano <pablo@anche.no>

PJColombo force-pushed the perf/leverage-stats-tables branch from 0fe5e18 to 3efea7f Compare October 29, 2024 10:49

vercel bot deployed to Preview – blobscan-staging October 29, 2024 10:52 View deployment

PJColombo added 3 commits October 29, 2024 14:03

refactor(api): improve readability of requireDirectCount fn

e327193

style(api): rename utility fn

8b030a4

test(api): test(api): verify that getCount procedures retrieves cou…

451b78e

…nt from stats table when applicable

vercel bot had a problem deploying to Preview – blobscan-staging October 30, 2024 15:02 Failure

PJColombo added 2 commits October 30, 2024 16:50

test: delete addressCategoryInfo rows when creating fixtures

16668c4

test(api): re-add null rollup ut to getCount tests

f6d2fae

vercel bot had a problem deploying to Preview – blobscan-staging October 30, 2024 15:52 Failure

chore: add changeset

12bb989

vercel bot had a problem deploying to Preview – blobscan-staging October 31, 2024 16:18 Failure

PJColombo merged commit dd75d56 into main Oct 31, 2024
12 of 14 checks passed

PJColombo deleted the perf/leverage-stats-tables branch October 31, 2024 16:26

github-actions bot mentioned this pull request Oct 31, 2024

chore: version packages #593

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(api): optimise count retrieval #590

perf(api): optimise count retrieval #590

PJColombo commented Oct 9, 2024 •

edited

Loading

vercel bot commented Oct 9, 2024 •

edited

Loading

changeset-bot bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

codecov bot commented Oct 9, 2024 •

edited

Loading

xFJA left a comment

PabloCastellano Oct 17, 2024

PabloCastellano Oct 17, 2024

-  }
+  } else {
+    datePeriod.from = MIN_DATE;
+  }

perf(api): optimise count retrieval #590

perf(api): optimise count retrieval #590

Conversation

PJColombo commented Oct 9, 2024 • edited Loading

Checklist

Description

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

vercel bot commented Oct 9, 2024 • edited Loading

changeset-bot bot commented Oct 9, 2024 • edited Loading

🦋 Changeset detected

github-actions bot commented Oct 9, 2024 • edited Loading

📦 Next.js Bundle Analysis for @blobscan/web

⚠️ Global Bundle Size Increased

New Pages Added

Eight Pages Changed Size

codecov bot commented Oct 9, 2024 • edited Loading

Codecov Report

xFJA left a comment

Choose a reason for hiding this comment

PabloCastellano Oct 17, 2024

Choose a reason for hiding this comment

PabloCastellano Oct 17, 2024

Choose a reason for hiding this comment

PJColombo commented Oct 9, 2024 •

edited

Loading

vercel bot commented Oct 9, 2024 •

edited

Loading

changeset-bot bot commented Oct 9, 2024 •

edited

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

codecov bot commented Oct 9, 2024 •

edited

Loading