Squashing a history of migrations.

[jenstroeger]

Looks like discussion #927 is somewhat related.

We have a lengthy history of Alembic migrations on a single Postgres db, and we want to squash them into one single migration step (mainly for performance reasons when testing, or standing up a deployment). So here’s where I am at the moment, with one detail still open (see below):

• Create an empty db in the psql console:

  drop database the_db; create database the_db;
  

• Run all migrations to the head

  alembic upgrade head
  

• Create a dump of everything in the db:

  pg_dump --host localhost --port 5432 --user postgres --inserts the_db > pgdump.sql
  

• Delete all existing Alembic migrations, and create a new one:

  rm alembic/versions/*
  alembic revision -m "Initial db setup"
  

• Paste the dump file into that new migration:

  def upgrade():
      dump = """
      -- Here goes the pgdump.sql
      """
      op.execute(sa.text(dump))
  
  def downgrade():
      pass  # Base is the pg dump.

And that’s where things become a little tricky. First, as mentioned in comment psycopg/psycopg#958 (comment) the psycopg driver doesn’t really support the files written by pg_dump (though it doesn’t error out in our case).

Second, the dump contains the alembic_version table as well, but that’s actually managed by Alembic as per documentation here:

The process which occurred here included that Alembic first checked if the database had a table called alembic_version, and if not, created it. [...] It then invokes the upgrade() method in each file to get to the target revision.

When I remove that table from the dump file, then running that new squashed migration against an empty db raises:

psycopg.errors.UndefinedTable: relation "alembic_version" does not exist
LINE 1: INSERT INTO alembic_version (version_num) VALUES ('48de627e2...

But the docs said the Alembic creates that table before running the migrations, no? So, I used the original dump file (the one with the alembic_version table) and then I get:

sqlalchemy.exc.ProgrammingError: (psycopg.errors.DuplicateTable) relation "alembic_version" already exists

So now that table exists? Not quite sure what’s going on.

But considering the aforementioned psycopg comment that dump files aren’t supported, and considering the previous conversation #927 I’m curious about the recommended approach to squash a history of migrations? Might Alembic not quite be suited here, and should we just restore the dump manually and start Alembic from that?

[zzzeek]

hi -

if you are making a dump file like that where that's to become the initial migration for a clean DB, then yes you have to attend to the alembic_version table in that dump file. that "INSERT INTO alembic_version (version_num) VALUES ('48de627e2.." is in your dump file. you have to remove those INSERT statements also.

if you run your alembic with a SQL dump that has nothing about alembic_version in it, and alembic_version does not exist in the target DB, alembic will see the database as unmanaged and create the table.