-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: db optimizations, backups optimizations #1664
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Migration, tested against staging already: ALTER TABLE "upload" ADD COLUMN "backup_urls" TEXT[];
UPDATE "upload" SET "backup_urls" = (SELECT array_agg("url") FROM "backup" WHERE "upload_id" = "upload"."id");
DROP TABLE "backup"; |
Optimized migration for production, batching updates for 64M rows into smaller queries to take less locks and prevent any downtime: https://gist.github.com/yocontra/4ae719b42a54c4582ca28fcac74199d0 |
@mikeal @hugomrdias This is ready to deploy - it has been run against staging and tested against a clone of the production database. Please let me know what next steps are. |
Partial of #1126 (does not include partitioning).
Included in this PR:
backup_urls
for each upload. This removes a redundant large table (12M uploads = > 12M backup rows), and speeds up create_upload by removing more inserts.