Migrating data between versions

[hql287]

The upcoming v1.1 introduces some changes in the way Manta constructs & stores data. So a migration scheme is necessary in order to prevent missing/corrupted data when users upgrade to a newer version.

What I'm thinking at the moment is this:

• Create a backup of invoices, contacts and app's settings data before upgrading the app.
• Upgrade the app as usual
• Adjust the data structure and import content from the backup.

Possible Pros

• No data lost, existing data will be converted to be compatible with the newer version.

Cons:

• Will need to implement export/import feature first.
• The migration script will be used only once but would need to be checked into git (or is it supposed to be?)

Right now this is the only thing that need to be done before releasing v1.1.
Would love to hear your thoughts on this.

[hql287]

@shime @mathieudutour @dawsbot: Any help would be greatly appreciated!

[mathieudutour]

ha migration issues ;) that's why I'm using dexie instead of pouch or similar.

I don't think you need to export/import.

I would change your app/helpers/pouchDB file to

const PouchDB = require('pouchdb-browser');
const contactsDB = new PouchDB('contacts');
const invoicesDB = new PouchDB('invoices');

async function runMigration(db, version, migrations, done) {
  try {
    const migrationsToRun = Object.keys(migrations).filter(k => k > version).sort()
    if (!migrationsToRun.length) {
      return done()
    }

    const result = await db.allDocs({
        include_docs: true,
        attachments: true,
      })
    const resultsDocs = results.rows
      .map(row => row.doc)
      .map(doc => migrationsToRun.reduce((prev, k) => migrations[k](prev), doc));
    await Promise.all(resultsDocs.map(doc => db.put(doc))
    done(null, migrationsToRun[migrationsToRun.length - 1])
  } catch(err) {
    done(err)
  }
}

const invoicesVersion = localStorage.invoicesVersion || 0 // will be 0 the first time
const invoicesMigrations = {
  1: (doc) => {
    // mutable the doc and return it
  }
}
let invoiceQueue = []
let alreadyRunInvoiceMigration = false
runMigration(invoicesDB, invoicesVersion, invoicesMigrations, (err, latestVersion) => {
  alreadyRunInvoiceMigration = true
  if (latestVersion) {
    localStorage.invoicesVersion = latestVersion
  }
  invoiceQueue.forEach(f => f(err))
  invoiceQueue = []
})

const setDB = dbName => {
  if (dbName === 'contacts') return contactsDB;
  if (dbName === 'invoices') {
    if (alreadyRunInvoiceMigration) {
      return invoicesDB
    }
    return new Promise((resolve, reject) => {
      invoiceQueue.push((err) => {
        if (err) return reject(err)
        resolve(invoicesDB)
      })
    })
  }
};

// Get All Document
const getAllDocs = dbName =>
  new Promise((resolve, reject) => {
    setDB(dbName).then(db =>
      db
        .allDocs({
          include_docs: true,
          attachments: true,
        })
     ).then(results => {
        const resultsDocs = results.rows.map(row => row.doc);
        resolve(resultsDocs);
      })
      .catch(err => reject(err));
  });

this is basically queueing the setDB calls. They will only resolve when the migration has finished running.

You can specify different migrations, only the ones that are newer than the version stored in the localStorage will be run.

[hql287]

@mathieudutour: This is great!!! Thank you very much!! 👍 Will try it out now.
I guess for the next migration, just add it to invoicesMigrations, correct?

Also, just curious how would dexie handle migration? And what's your experience with dexie in general compared with pouchDB?

[mathieudutour]

yeah, invoicesMigrations is an object where the keys are the version of the db and the values are a function that takes a doc and returns the migrated doc.
So let's say you are now on version 3 of the db and I haven't open the app since version 1, the migration 2 and 3 will run on each of the docs.

Dexie is basically doing that for you: you define the db version and the list of migrations and it takes care of queuing and such. (the code above is heavily inspired by it)

[hql287]

Dexie is basically doing that for you

That's brilliant! Probably will save a lot of headache in the future.
Do you think it's worth the effort to switch from pouchDB to dexie?

[mathieudutour]

hum, not sure how the migration to dexie would work tho. I'd say no but it's always a good way to learn new tech, so if your goal is to learn, I'd say go for it haha ;)

[hql287]

I guess since they are just wrappers of IndexedDB, if we have a way to backup the data first, then mutate it to a compatible form for dexie then we should be fine.

But as always, it's easier said than done 😅.

I'd say go for it haha ;)

I really don't mind giving it a try if it's worth it. That's why I asked this:

What's your experience with dexie in general compared with pouchDB?

[mathieudutour]

I'd say it's the least bad out there... not a fan of the existing in-browser db APIs, I would probably try to make my own if I had more time haha

[hql287]

I'd say it's the least bad out there

This pretty much sums up the entire in-browser db solutions situation atm 😄