Sub-epic: Performance improvement

[PaulClark2]

Tickets

• Research adding API logic to require filter fields for certain endpoints to have an index openFEC#4378
• Research long-running 'Rollback' in pg_stat_activity openFEC#4558
• Research: Bring terminate_backend() task into source control as Celery task openFEC#4500
• (disclosure.fec_fitem_sched_a.sub_id) < (4101920051061418807) not use PK index, need add fec_fitem_sched_a.two_year_transaction_period filter + Union all or "angular " for 60 times faster openFEC#4616
• Test higher page count openFEC#4433
• Internal work to add contributor address to bulk data files openFEC#4212
• Research possible performance improvements for user downloads openFEC#4552
• Use "UNION ALL+limit" for full 45 years SA search by using "two_year_transaction_period" openFEC#4414
• Research: Improve homepage load time (currently ~1.9 seconds) fec-cms#3711
• Research feasibility of creating a weekly schedule A bulk downloads openFEC#4479
• Implement primary key-only count for other endpoints openFEC#4429
• API memory usage too high openFEC#4578
• Remove 'coalesce' from Schedule B sort openFEC#4491
• schedule B slow sql (90 sec to 0.5 sec to 120+ times faster by add a new index: CREATE INDEX idx_sched_b_2019_2020_cmte_id_cmte_tp_disb_dt_sub_id openFEC#4533
• schedule B slow sql (65 to 800+ times faster by add a new index: idx_sched_b_2019_2020_disb_dt_sub_id  openFEC#4531
• Increase CMS DB size to handle more concurrent connections fec-cms#3907
• Test more gevent workers per app instance openFEC#4489
• Improve slowest queries by adding a new composite index: 9mins to 2 sec 270 times faster openFEC#4369
• Investigate speeding up query counts by not specifying columns openFEC#4368
• Consistent 502s hitting /schedule_a openFEC#4404
• Document and hold training on how to identify slowest queries in realtime (and historical) in Aurora openFEC#4330
• Improve slowest queries by adding a new index composited schedue_a for desc sorting  openFEC#4352
• Keep one postgresql database (prod) in sync in parallel then drop all the old postgres databases openFEC#4328
• Improve slowest queries by adding a new index composited real_efile_sb4_comid_idate_con_dx openFEC#4346
• Improve slowest queries by adding a new index composited real_efile_sa7_comid_date_con_idx openFEC#4345
• Improve slowest queries after Aurora migration openFEC#4321
• Migrate postgresql database instance to new Aurora database instances openFEC#4297
• Develop team to use the aurora database as the dev database to fully test API/database openFEC#4298
• Perform data query test (API, data export, PGBadger) openFEC#4289
• Hook up testing data loading with CircleCI openFEC#4288
• Perform loading tests (Java, Python, Flyway migration, GoldenGate) openFEC#4287
• Setting up Aurora database instance version 10.7 for testing (database creation and data loading) openFEC#4286
• Perform locust load testing openFEC#4290
• Initial performance tuning openFEC#4291