feat: implement postgres optimization #1305

francois-potato · 2022-05-09T18:36:48Z

🛑 Requires db migration packages/db/postgres/migrations/008-add-backup-urls.sql to add new column
🛑 Requires db migration with the following script

flea89

Great work @francois-potato!

Left some comments

packages/db/postgres/functions.sql

packages/db/index.js

packages/db/postgres/functions.sql

packages/db/postgres/migrations/009-migrate-backup-urls.sql

packages/db/postgres/functions.sql

vasco-santos

Great work @francois-potato

Left two minor comments that should be addressed. Would also love to have @hugomrdias 's input before moving forward as this needs to be changed in nft side and would like to be aligned.

packages/db/index.js

packages/db/test/backup.spec.js

packages/db/index.js

hugomrdias

LGTM

hugomrdias · 2022-05-25T10:01:03Z

packages/db/postgres/migrations/009-migrate-backup-urls.sql

@@ -0,0 +1 @@
+UPDATE "upload" SET "backup_urls" = (SELECT array_agg("url") FROM "backup" WHERE "upload_id" = "upload"."id");


we should plan for this migration, this can lock the upload table a bit.
@vasco-santos any thoughts for timing? should we write the batch script like nft ?

Now that the schema is the same (text[] instead of jsonb), the nft batch script works without modifications. I tested it locally too.

Yeah, I think batch script is best way to go.

@hugomrdias how long did it take for you? We should probably schedule a maintenance windows

humm cant remember, i think under 1 hour

awesome ok thanks, I will setup a migration window for this then.
@francois-potato we should be ready to merge here and do the maintenance next monday right?

vasco-santos · 2022-05-25T14:15:38Z

packages/db/postgres/migrations/009-migrate-backup-urls.sql

@@ -0,0 +1 @@
+UPDATE "upload" SET "backup_urls" = (SELECT array_agg("url") FROM "backup" WHERE "upload_id" = "upload"."id");


awesome ok thanks, I will setup a migration window for this then.
@francois-potato we should be ready to merge here and do the maintenance next monday right?

vasco-santos · 2022-05-25T14:16:28Z

When merging, please comment in the release PR that migration script needs to run and should take up to 1 hour

This reverts commit 75d5697.

@adamalton

* fix: JS errors from docs changes (#1334) * fix: remove React fragment which is causing an error This was causing `Each child in a list should have a unique "key" prop` * fix: indentation from my previous commit * fix: js errors * fix: revert package-lock changes * fix: lint error Co-authored-by: Adam Alton <adamalton@gmail.com> * fix: Removed mistakenly generated link on CID header item in filemanager (#1336) * fix!: psa pinning APIs - rename requestId to requestid * chore(main): release website 2.4.0 (#1299) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * make tooltip accessible (#1340) * chore: rename pinned to psaPinned (#1268) Co-authored-by: Alan Shaw <alan.shaw@protocol.ai> * docs: update peers (#1344) * feat: respond with unique error message when blocked API key is used (#1302) * chore(main): release api 6.0.0 (#1325) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore(main): release website 2.4.1 (#1342) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: Set fetch date before changing isFetching state (#1341) * feat: send email notifications for storage quota usage (#1273) This includes: - a general email component in cron package for sending emails - notifications to web3.storage users when they get to specific thresholds - notifications to web3.storage admins when users go over their quota Co-authored-by: Gary Homewood <gary@potatolondon.com> Co-authored-by: Paolo <paolo@potatolondon.com> Co-authored-by: Oli Evans <oli@tableflip.io> Co-authored-by: francois-potato <103063611+francois-potato@users.noreply.github.com> * fix: inaccurate used_storage migrations (#1360) * feat: add user blocking functionality to web3 (#1322) * chore: do not convert bigint to number (#1366) * chore: use mailchimp provider in crons (#1368) * chore: send list of storage quota violators to support@web3.storage, not admin@ (#1369) * chore: remove unnecessary migration for creating admin user (#1373) Rename the subsequent migration to keep the numbers sequential. Now that we're using support@ rather than admin@ for the admin email address, that user already exists on both staging and prod. * feat: implement postgres optimization (#1305) * Refactor backups * Add db migration scripts * Update db configuration * fix: typo in Logging constructor (#1346) I assume this is a typo? @adamalton can you also look into this logger running during testing in CI? https://github.com/web3-storage/web3.storage/runs/6553500624?check_suite_focus=true * fix: db migrations versioning (#1375) * chore(main): release api 6.1.0 (#1356) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore(main): release cron 1.1.0 (#1357) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * docs: note db schema required in api readme ...and tweak website README to note that you are using a mock API * chore: add test for CORS OPTIONS handler (#1331) * fix: show custom storage quota to user (#1338) * chore: fix tags in api user info (#1379) * chore(main): release api 6.1.1 (#1382) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore(main): release website 2.5.0 (#1348) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: change the storage cron job config to run in prod, and every 6 hours (#1371) * chore: trigger crons workflow on conf change (#1394) * feat: Adding HasDeleteRestriction user_tag (#1390) * Adding the type and failing HTTP DELETE operations if this tag is set. * See nftstorage/admin.storage#66 * fix: clone env so new each request (#1396) * chore(main): release api 6.2.0 (#1397) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * feat: DB schema and API for user_tag_proposal. (#1006) * Users create records in this table and admins manage it. Co-authored-by: trigramdev9 <jsdevel@trigram.co> * feat: Adding admin ability to search by github_id (#1403) * See nftstorage/admin.storage#68 * chore: optimise getUserByStorage query (#1405) * fix: optimise getUserByStorage query to avoid timeouts (#1412) * chore: update SQL migration to drop the old `users_by_storage_used` function before replacing (#1414) * fix: update incorrect dag sizes job (#1059) (#1196) * fix: 404 API http reference links (#1358) Fixes: #1359 * fix(http docs): incorrect endpoint in description (#1429) * fix(http docs): incorrect endpoint in description * test: trigger a rebuild via CI Co-authored-by: orun <orun@agencyundone.com> * chore: add Wrangler worker env for Josh (#1351) * chore: Add Wrangler worker env for Josh. * chore: Add GATEWAY_URL to josh wrangler config. * chore: Add default `GATEWAY_URL` var to wrangler template. * chore(main): release website 2.5.1 (#1417) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: docs toc highlight on click (#1392) * fix: toc highlight on click - docs * fix: Replace document query with inline conditional classes and tweak scroll magic scene settings * test: force rebuild Co-authored-by: svvimming <benmunozj@gmail.com> Co-authored-by: orun <orun@agencyundone.com> * fix: refactor accordion content (#1391) * fix: refactor accordion content * style: reduce mobile padding below faq accordion Co-authored-by: orun <orun@agencyundone.com> * fix: Adding user_tag_proposal schema to reset.sql for local dev (#1445) * Introduced in #1006 * feat: Split file manager table into uploaded & pinned (#1363) * feat: Filemanager header file type tabs and split table by type * fix: Storage manager progress bar double border & height * feat: File manager search results in title + margins and spacing * test: Console log all files * test: Console log files and user storage data * feat: Mock data for files using pinning service + api request for pinned files * test: Fetch pinning data * feat: Add loading state to pins tab table * feat: Add loading state to pins tab table * feat: Add uploaded/pinned tab url param * test: Pinned fetch request * test: Pinned fetch request * revert 'test: Pinned fetch request' * revert 'test: Pinned fetch request' * test: Pinning fetch request with generated token * revert 'test: Pinning fetch request with generated token' * test: Pass generated API token to /pins GET request * test: console log API * feat: Disable pins table if no files are present & revert pins status to pinned in request * chore: fix linting warning * style: Responsive file manager header layout at small breakpoints * feat: Account page UI minor tweaks * feat: reduce font size (#1411) * feat: Message bar incident/maintenance name (#1335) * feat: Docs automatically generated files * refactor: Message banner incident/maintenance message displays name * chore: Remove testing materials * chore: Remove testing materials * Revert "chore: Remove testing materials" This reverts commit ab5f9e9. * Revert "refactor: Message banner incident/maintenance message displays name" This reverts commit 7359dbe. * Revert "feat: Docs automatically generated files" This reverts commit 26ad9ca. * refactor: Message banner incident/maintenance message displays name * chore: Remove testing materials * chore(main): release cron 1.1.1 (#1413) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix: use NODE_TLS_REJECT_UNAUTHORIZED=0 env var for storage cron job (#1418) This allows the direct connection to the Postgres DB to work. * chore: add package-lock change report to PRs (#1453) Bot to add comment on PRs with a human readable report of changes to package-lock.json see: https://github.com/marketplace/actions/npm-lockfile-changes License: (Apache-2.0 AND MIT) Signed-off-by: Oli Evans <oli@tableflip.io> * docs: clarify payload size limit for /car endpoint (#1457) * chore: get back integrity and resolved keys for deps (#1456) * chore: only check package-lock diff on PRs (#1460) only check package-lock diff on PRs see: #1453 which produces nice package-lock reports for our PRs but errors when run from not-a-pr. License: (Apache-2.0 AND MIT) Signed-off-by: Oli Evans <oli@tableflip.io> * fix: Adjust progress bar styles for pinned files minimum cases Co-authored-by: Joanna Ong <ongjfu@gmail.com> Co-authored-by: Adam Alton <adamalton@gmail.com> Co-authored-by: Paolo Chillari <flea89@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Alan Shaw <alan.shaw@protocol.ai> Co-authored-by: e-schneid <99349687+e-schneid@users.noreply.github.com> Co-authored-by: Gary Homewood <gary@potatolondon.com> Co-authored-by: Paolo <paolo@potatolondon.com> Co-authored-by: Oli Evans <oli@tableflip.io> Co-authored-by: francois-potato <103063611+francois-potato@users.noreply.github.com> Co-authored-by: Vasco Santos <santos.vasco10@gmail.com> Co-authored-by: Josh Jarvis <josh.jarvis@potatolondon.com> Co-authored-by: Joe Spencer <js.developer.undefined@gmail.com> Co-authored-by: Joe Spencer <jsdevel@trigram.co> Co-authored-by: Jorropo <jorropo.pgm@gmail.com> Co-authored-by: Yusef Napora <yusef@napora.org> Co-authored-by: orun <orun@agencyundone.com> Co-authored-by: Hugo Dias <hugomrdias@gmail.com>

francois-potato linked an issue May 9, 2022 that may be closed by this pull request

Implement Postgres optimization configuration changes (replicated from nft) #615

Closed

fhoehl force-pushed the feat/615-implement-postgres-optimization branch 2 times, most recently from 28989a8 to 75d5697 Compare May 10, 2022 16:58

francois-potato marked this pull request as ready for review May 11, 2022 14:57

flea89 reviewed May 11, 2022

View reviewed changes

francois-potato self-assigned this May 16, 2022

francois-potato mentioned this pull request May 16, 2022

Implement Postgres optimization configuration changes (replicated from nft) #615

Closed

vasco-santos requested changes May 16, 2022

View reviewed changes

packages/db/postgres/migrations/009-migrate-backup-urls.sql Outdated Show resolved Hide resolved

packages/db/postgres/functions.sql Outdated Show resolved Hide resolved

francois-potato requested a review from hugomrdias May 18, 2022 12:12

vasco-santos requested changes May 18, 2022

View reviewed changes

packages/db/index.js Outdated Show resolved Hide resolved

packages/db/test/backup.spec.js Show resolved Hide resolved

hugomrdias suggested changes May 19, 2022

View reviewed changes

packages/db/index.js Outdated Show resolved Hide resolved

packages/db/index.js Outdated Show resolved Hide resolved

vasco-santos mentioned this pull request May 24, 2022

Backup URLs not properly stored nftstorage/nft.storage#1939

Closed

hugomrdias approved these changes May 25, 2022

View reviewed changes

vasco-santos approved these changes May 25, 2022

View reviewed changes

francois-potato added 14 commits May 27, 2022 16:08

Refactor backup

070268c

Update db configuration

c4a5564

Add db migration scripts

3571c25

Add debug logging

ed558a2

Revert "Add debug logging"

db12af7

This reverts commit 75d5697.

Lint SQL

4146026

Fix typing

f7273c5

Make url unique

d7ba467

Lint

7564606

Add test

3ac383f

Fix type and attribute name

a93030d

Refactor tests

730610a

Move from JSON to TEXT array

c5fa617

Refactor migration

da9ce4f

fhoehl force-pushed the feat/615-implement-postgres-optimization branch from 1a91b41 to da9ce4f Compare May 27, 2022 15:11

francois-potato merged commit 2cd6105 into main May 27, 2022

francois-potato deleted the feat/615-implement-postgres-optimization branch May 27, 2022 15:27

vasco-santos mentioned this pull request May 28, 2022

fix: db migrations versioning #1375

Merged

github-actions bot mentioned this pull request Jun 27, 2022

chore(main): release db 1.0.0 #1578

Closed

vasco-santos mentioned this pull request Jan 23, 2024

feat: list uploads joins to backup table to get backup urls #2352

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement postgres optimization #1305

feat: implement postgres optimization #1305

francois-potato commented May 9, 2022 •

edited

Loading

flea89 left a comment

vasco-santos left a comment

hugomrdias left a comment

hugomrdias May 25, 2022

francois-potato May 25, 2022

vasco-santos May 25, 2022

hugomrdias May 25, 2022

vasco-santos May 25, 2022

vasco-santos May 25, 2022

vasco-santos commented May 25, 2022

		@@ -0,0 +1 @@
		UPDATE "upload" SET "backup_urls" = (SELECT array_agg("url") FROM "backup" WHERE "upload_id" = "upload"."id");

feat: implement postgres optimization #1305

feat: implement postgres optimization #1305

Conversation

francois-potato commented May 9, 2022 • edited Loading

flea89 left a comment

Choose a reason for hiding this comment

vasco-santos left a comment

Choose a reason for hiding this comment

hugomrdias left a comment

Choose a reason for hiding this comment

hugomrdias May 25, 2022

Choose a reason for hiding this comment

francois-potato May 25, 2022

Choose a reason for hiding this comment

vasco-santos May 25, 2022

Choose a reason for hiding this comment

hugomrdias May 25, 2022

Choose a reason for hiding this comment

vasco-santos May 25, 2022

Choose a reason for hiding this comment

vasco-santos May 25, 2022

Choose a reason for hiding this comment

vasco-santos commented May 25, 2022

francois-potato commented May 9, 2022 •

edited

Loading