Increase CMS DB size to handle more concurrent connections

[JonellaCulmer]

What we're after: Increase CMS DB size to handle more concurrent connections. We're at a medium postgres size currently. In order to handle more concurrent connections, we should upgrade to XL.

Steps used to upgrade to xlarge psql is in a comment below.

Completion criteria:

• CMS database is increased to XL. Backup and restore to this instance.
• Tested on stage first then, then production (Before production work we need to let content team know to pause wagtail work ahead of maintenance window)

[patphongs]

The following steps are modified from procedures that @pkfec documented to upgrade to pg11. These steps should help us create a xlarge CMS db size:

1. create a new PG11 service:
   cf create-service aws-rds xlarge-psql-redundant fec-stage-cms-xlarge

2. unbind CMS app form existing/current DB service
   cf unbind-service cms fec-stage-cms

3. bind CMS app to the xlarge psql service, wait for minimum of 40mins, until the new service is available in cloud.gov.

4. bind cms app with the xlarge psql service
   cf bind-service cms fec-stage-cms-xlarge

5. take a fresh dump of the exising prod db using the pg11 client location installed on your local
   /Library/PostgreSQL/11/bin/pg_dump -F c --no-acl --no-owner -f /[path_to_file]/<name of the dump file>.dump postgres://<username>:<password>@localhost:<port number>/<db name>

6. restore this dump onto the latest db fec-stage-cms-xlarge
   /Library/PostgreSQL/11/bin/pg_restore --dbname postgres://<username>:<password>@localhost:<port number>/<db name> --no-acl --no-owner <name of the dump file>.dump

There were some extension error, which can be ignored:

pg_restore: [archiver (db)] Error from TOC entry 3979; 0 0 COMMENT EXTENSION hstore
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension hstore
    Command was: COMMENT ON EXTENSION hstore IS 'data type for storing sets of (key, value) pairs';



pg_restore: [archiver (db)] Error from TOC entry 3980; 0 0 COMMENT EXTENSION pg_trgm
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension pg_trgm
    Command was: COMMENT ON EXTENSION pg_trgm IS 'text similarity measurement and index searching based on trigrams';



WARNING: errors ignored on restore: 2

7. rename fec-stage-cms service. We should add a date to this name so that we know the copy of the database is current as of the date we switched it.
   cf rename-service fec-stage-cms fec-stage-cms-medium-[date]

8. rename fec-stage-cms-xlarge to fec-stage-cms
cf rename-service fec-stage-cms-xlarge fec-stage-cms

9. rebuild the latest CMS releasse branch build on circleci
   https://app.circleci.com/pipelines/github/fecgov/fec-cms/

10. Test the stage website and wagtail CMS. Make sure that data is loading and saving as expected.

DR steps - This is not required, but good to know if something goes wrong with the xlarge psql service, we can fall back to the medium psql service.

11. unbind cms app from the xlarge psql service
    cf unbind-service cms fec-stage-cms

12. rename the xlarge psql service fec-stage-cms to fec-stage-cms-xlarge
cf rename-service fec-stage-cms fec-stage-cms-xlarge

13. bind the cms app to the OLD medium psql service
    cf bind-service cms fec-stage-cms-medium

14. rename medium psql service to fec-stage-cms
    cf rename-service fec-stage-cms-medium fec-stage-cms

15. rebuild the latest CMS release branch build on circleci
    https://app.circleci.com/pipelines/github/fecgov/fec-cms/

16. test stage website

[patphongs]

Production CMS DB has been updated to xlarge-psql-redundant. This was implemented using cloud.gov's new cf update-service command. Docs can be found here: https://cloud.gov/docs/services/relational-database/#update-an-instance