Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add stats routes #1400

Merged
merged 23 commits into from
Mar 7, 2022
Merged

feat: add stats routes #1400

merged 23 commits into from
Mar 7, 2022

Conversation

dashcraft
Copy link
Contributor

Working on stats routes, these are currently written while not having a local setup to test against.
I'll be testing over the next few days.

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Feb 25, 2022

Deploying with  Cloudflare Pages  Cloudflare Pages

Latest commit: cabf531
Status: ✅  Deploy successful!
Preview URL: https://733c2d06.nft-storage-1at.pages.dev

View logs

@dashcraft dashcraft marked this pull request as ready for review February 25, 2022 02:16
@JeffLowe JeffLowe requested review from cmunns and alanshaw February 27, 2022 20:24
@dashcraft
Copy link
Contributor Author

This implements the materialized view implementation i discussed with @dchoi27. This should be good enough for 10's of millions of records. Locally, on a postgres instance with 15 million records, it takes roughly 10-13 seconds, sometimes closer to 6. This is much easier to refresh since it is all postgres side, but it is not the fastest solution. This solution is exact though, so numbers are trustworthy.

As noted in my conversation with @dchoi27, the more robust solution should this one be rendered too slow or not robust enough, would be to cash the previous values and programmatically calculate the flow values from the 2 diffed values (previous count and count in past 7 days, added together as current total count).

@dashcraft
Copy link
Contributor Author

Fwiw on a timeseries database i have, this query takes 1.8s on 10 million records, so this is indicative that indexing plays a role in this query speed.

packages/api/db/cargo.testing.sql Outdated Show resolved Hide resolved
packages/api/db/cargo.testing.sql Outdated Show resolved Hide resolved
packages/api/db/cargo.testing.sql Outdated Show resolved Hide resolved
packages/api/db/tables.sql Outdated Show resolved Hide resolved
packages/api/pw-test.config.cjs Outdated Show resolved Hide resolved
packages/api/src/routes/metrics.js Outdated Show resolved Hide resolved
packages/api/src/utils/db-types.d.ts Outdated Show resolved Hide resolved
packages/cron/src/jobs/metrics.js Outdated Show resolved Hide resolved
packages/cron/src/jobs/metrics.js Outdated Show resolved Hide resolved
packages/cron/src/jobs/metrics.js Outdated Show resolved Hide resolved
@dashcraft dashcraft requested a review from alanshaw March 1, 2022 18:40
Copy link
Contributor

@alanshaw alanshaw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, just a minor request to follow the existing naming convention for the metric names.

Also, in main we're now timing the queries so we can debug more easily as the data size grows - can you please rebase/merge with main and add the withTimeLog function call around your queries? 🙏

packages/api/test/stats-show.spec.js Outdated Show resolved Hide resolved
@JeffLowe JeffLowe requested a review from alanshaw March 3, 2022 03:02
@dashcraft dashcraft force-pushed the feat/add-stats-routes branch from fcaca9b to cabf531 Compare March 7, 2022 20:26
@dashcraft dashcraft merged commit 96acd59 into main Mar 7, 2022
@dashcraft dashcraft deleted the feat/add-stats-routes branch March 7, 2022 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants