Add CI for synapse_port_db

[babolivier]

Paired with matrix-org/pipelines#9

This PR adds:

• a test sqlite database
• a configuration file for the sqlite database
• a configuration file for a postgresql database (using the credentials in .buildkite/docker-compose.pyXX.pgXX.yaml)

as well as a new script named .buildkite/scripts/test_synapse_port_db.sh that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the synapse_port_db script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step 2 is done via a new script located at scripts-dev/update_database.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.

[erikjohnston]

Can you give us a commit range to review or something please? There are too many commits to figure out whats going on without some help :)

[babolivier]

Can you give us a commit range to review or something please? There are too many commits to figure out whats going on without some help :)

Right, sorry, I assumed this review would come after the ones on the PRs this one is built on top of. A list of all commits specific to this PR can be seen here, and for maximum clarity this PR only changes the following files:

• .buildkite/scripts/test_synapse_port_db.sh (creation)
• .buildkite/postgres-config.yaml (creation)
• .buildkite/sqlite-config.yaml (creation)
• .buildkite/test_db.db (creation)
• scripts-dev/create_postgres_db (creation)
• scripts-dev/update_database (creation)
• tox.ini (addition of the synapse_port_db test environment)

None of the other PRs edit these files, so the reviewer can review just the changes to these files, without worrying about whether they've been changed by this PR or another.

[richvdh]

a few comments; it kinda needs #6102 to land before it gets a proper review.

stale

[richvdh]

pending merge of #6102

[babolivier]

#6102 has been merged; I've rebased the branch on develop so it doesn't pull any more commit than it should.

stale

stale

[richvdh]

lgtm!