Lotus metadata backup

[jsign]

We at Textile are providing hosted Powergate nodes and running our own Powergates for Hub and internal things.
That means that we're managing more than 30 Lotus client nodes on which we rely on, and also other people rely on for SR2 competition or simply for using the network.

I'm aware that there are high priority issues about solving database corruption problems, which are usually very hard to solve (e.g: this). The main problem with those bugs risks is that they might have is making the Lotus client node irrecoverable, which now that we aren't on testnet isn't a viable option without having a huge problem. I'd guess that for miners is also a big risk, but maybe they already have their own created fallbacks.

It would be useful if there's some Lotus CLI command which can backup all the data that can be fed in a new/fresh Lotus client node. That's to say: all wallet addresses, active deals, in-progress deals, other metadata, etc.

Doing that kind of migration workflow might be:

• Running this new command that can generate some file backup.
• Spinning a new Lotus client node with the usual --import-snapshot for the chain data as usual, and some extra flag --import-metadata <backupfile> to also import this other part of state from the previous Lotus client node.
• Win.

It would be really important if this CLI command can generate these backup files while the client node is running, since stopping the client node to make backups is challenging for other reasons.

This tool might provide a way to schedule regular backups with this CLI command, and having a fallback mechanism to any kind of corruption a Lotus datastore might run into (or actually, any other big problems).

I know this is a weird ask since all this data is the datastores... mostly an idea to circumvent some risks that can cause big problems. Not proposing a final solution, just thinking about how to minimize risks.

[jennijuju]

@jsign is lotus backup - The backup command writes a copy of node metadata under the specified path what you are looking for?

or you want something specific only for client?

[jsign]

Looks like it covers the metadata part, but may be good that also include wallet addresses too since I think that's not covered, and I'm not sure how safe is to copy paste se keystore folder while the node is running.

Also, a AFAIK there's no way to re import that backup... so that's the other half of the process.

I also noticed there are other folders in the datastore folder that I'm not sure are important or not to also cover.

But would be good to define that command should always cover any data to migrate lotus, since other data might be created in other places apart from the metadata datastore.

In short, I might not be the right person to know the answer to your question... But that command looks to cover at least some part.

[pooja]

What does this backup function currently backup? Does it include .lotus/datastore?

[jsign]

@pooja , it makes a backup of .lotus/datastore/metadata.

[jennijuju]

@jsign could you please try #5362 and see if it resolves this?

[jsign]

@jennijuju, I think that PR solves a big part of this.

In the original proposal, I was thinking that might be a good idea that lotus backup not only backups metadata, but also wallet-keys (which isn't the case now). Backuping keys, afaik, it's only copy-pasting the keystore folder, so it's easy anyway to do it separately. Also, maybe it's a good idea to not mix in the same backup file non-sensitive and sensitive data.

I'll close this issue, and if anyone thinks might worth some extra discussion about including wallet-keys in the lotus backup output, then I think it deserves its own discussion in a new issue.