dynamic vats aren't replayed when kernel is restarted

[warner]

Describe the bug

@FUDCo discovered that our dynamic vat support is incomplete. We're saving the transcripts and c-lists of dynamic vats into the kernel state DB, but not their code. At startup, the kernel does not regenerate the dynamic vats, so other vats which interact with them will fail.

This wouldn't be noticable if a contract vat completes its business before the host is shut down (i.e. a validator node reboots), but any contract that is still operating at that time would be non-functional when the node starts back up again.

To fix this, we'll need to save the dynamic vat's code (and/or the bundleName used to create it) in the DB. And we'll need to change the kernel startup code to look for dynamic vats and regenerate them, including replaying their transcripts.

[FUDCo]

From my perspective it's less an issue of capturing the code as capturing the fact that the dynamic vat was created at all (and what parameters it was created with). The code for static vats is taken from the file system (which has its issues but we've managed to live with it) so I don't see why dynamic vats are materially different in that regard (well, if their bundles came in over the network from outside the swingset I guess it's an issue, but in that case the bundle is in the transcript already).

[warner]

Yeah, contract bundles will come in over the network, and the vat which spawned the dynamic vat may have that code in its transcript, but that doesn't help us, because that vat (like all others) are disconnected from their syscalls while being replayed. Since each vat is replayed independently, we really have to save the source (or the bundleName, if any) in the DB. (of course this is a fine candidate for a better DB approach, like add the code to a hash-indexed blob store and then only stash the hash in the DB).

I've got a branch started, with unit tests that fail.

[FUDCo]

The bundle is literally in the callNow syscall that creates the vat. Although we don't currently do anything with the logged syscalls except verify that the replay generates the same ones, in the process of doing that verification we are in a position to observe the act of vat creation. Would it make sense for replay to be able to do something with device calls?

[warner]

I don't think so. We're going to want to be able to reload vats independently and lazily, as messages arrive for them, and we'd have no idea which of the vatAdmin's syscalls might have created the vat we're asked to reload. So needing to replay the entire vatAdmin history first, just to glean the bundles for the dynamic vats, would be a lot of coupling between things that ought to be independent. Most of those vats (hopefully) will have been retired by the time the kernel restarts, so we wouldn't want to recreated every one that the vatAdmin ever knew about: we want to only rebuild the ones that are still active. We may want to migrate a vat from one swingset to another: we'll need to be able to get both its source code and its current state from the DB, without actually loading it right then.

The direction I'm heading is:

//
// vat.names = JSON([names..])
// vat.dynamicIDs = JSON([vatIDs..])
// vat.name.$NAME = $vatID = v$NN
// vat.nextID = $NN
// device.names = JSON([names..])
// device.name.$NAME = $deviceID = d$NN
// device.nextID = $NN

// dynamic vats have these too:
// v$NN.source = JSON({ bundle }) or JSON({ bundleName })
// v$NN.options = JSON

I've got the kernel.js -side part in place. The biggest piece is splitting dynamicVat.js up to separate the "allocate things the first time" part from the "build a dynamic vat once we know the allocations" part, so I can add a third piece to use on the replay pass.