Lock DB Upgrades

[snadrus]

Allow max of 1 db upgrader at a time.

[snadrus]

Well no form of db locking is going to work:
github.com/filecoin-project/curio/harmony/harmonydb.(*DB).upgrade.func1.1
/home/circleci/curio/harmony/harmonydb/harmonydb.go:256
- ERROR: ACCESS EXCLUSIVE lock mode not supported yet (SQLSTATE 0A000)

[snadrus]

Is this a known problem @magik6k or a nice-to-have?
The only window of risk is on upgrade when a new schema arrives if 2 machines get there about the same time (single digit milliseconds). I think today it'll crash the "second" process (which should have a restart), then everything should be fine. I could "sleep on error and retry" to avoid that crash. WDYT?

[magik6k]

Is this a known problem @magik6k or a nice-to-have?

A problem that we didn't have any case that manifested in actual trouble, but I do foresee it being an issue at some point given that schema changes in YB do tend to take a while and are likely to be a large source of edge-cases given what YB does with postgres.

The only window of risk is on upgrade when a new schema arrives if 2 machines get there about the same time (single digit milliseconds).

It's usually much longer, in my deployments most migrations take 2-10s, depending on what they do

[LexLuthr]

https://yugabytedb.tips/suppress-the-error/ advisory locks are not supported. I found out yesterday the hard way after using them for libp2p table.

[snadrus]

I'd say we shelve this. If it becomes a problem, maybe the locks will arrive then and this code will be a drop-in fix.

[snadrus]

One alternative is to have an upgrade 'task' that gets scheduled, deduped, claimed by those only with that version, and ran that way, but we can't start the task engine with any other task. It sounds more difficult.

[magik6k]

I believe I see a good way to do this:

• Add a separate base_upgrades table
• When we don't see an entry in the base table for a given upgrade, write the entry to the base_upgrades table
  • If that write conflicts, wait a bit (~10s) and go to step one. Either someone else has applied the update, or it failed so manual intervention is needed anyways
  • If it didn't conflict, we're now responsible for applying the schema, so do that.