Django 4.2 upgrade

[davidfischer]

Upgrades Django to the latest LTS.

Upgrading Postgres

Important: Django 4.2 requires Postgres >= 12

Postgres data files aren't compatible between major versions. As a result, if you want to save your local database, you need to follow these steps

# Start the database
make dockershell

# In another terminal window, take a database backup
# The file is saved inside a docker volume
# This should be run on Postgres 11
docker-compose run --rm  postgres backup
# After running this, you can shut down the `make dockershell` from the first window

# Rebuild the docker-compose images with Postgres 15
make dockerbuild

# Stop docker-compose and remove the postgres data directory
docker-compose  down
docker volume rm ethical-ad-server_local_postgres_data

# Back in that first terminal window, start Postgres 15
# This also runs some basic initialization on the new database
make dockerserve
# Wait for Postgres to startup and initialize and create the DB
# Then ^+C `dockerserve`.
# Start the database again (but with no other things like celery connected)
make dockershell

# In the 2nd terminal window,
# restore your local database (no need to start the backup with /backups/)
# The filename comes from the output of `docker-compose run --rm  postgres backup` above
# Or get the filename with `docker-compose run --rm  postgres backups` (backups, plural)
docker-compose run --rm postgres restore backup_2023_08_DDTHH_MM_SS.sql.gz

Verifying in staging

Once this is deployed in staging, the following should be well tested since there were a few changes around those:

• Verify that the Cloudflare GeoIP header is being checked correctly
• Staticfiles are correctly collected and the filenames are hashed

[ericholscher]

This looks similar to our RTD upgrade, mainly admin and test changes. I guess we need to block this on a production DB upgrade? I think it should be simpleish to upgrade things, but will have to do the research.

Looks like we can hopefully do it with 1-click: https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-major-version-upgrade -- but it doesn't support read replicas... so we probably just need to handle it ourselves:

• Spin up a new pg 14/15 server
• Do an initial pg_dump/load to it
• Swap over production to point at the new server
• Backfill any Offer data that came in during the migration

I think if we can get the dump/load to a reasonable time (<5 mins), we can probably also just do a quick scheduled downtime for ad serving and do it without any data migration.

[davidfischer]

I'm seeing some form issues relating to the upgrade that I wasn't seeing in 3.x. Specifically, I'm seeing some duplicated field choice values:

I haven't investigated deeply enough to figure out if this is fixed in a later version of crispyforms or if we'll need to customize some form templates.

[davidfischer]

The form issues are fixed by upgrading crispy and the template pack.

[davidfischer]

The prod database is updated. I'm going to merge this after it is approved.

[ericholscher]

Let's ship it!