Cannot send an encrypted dataset (key issue) #7117
Comments
This looks like you had an existing pool from before the stability patch and are now trying to send to a new pool. You cannot use a raw send to do this, since the raw send would preserve the old broken format. This is why you are getting the errors you are seeing on those datasets. To un-break those datasets, you need to use a normal send. Unfortunately, the UI for this kind of sucks right now because The kernel panic you got, however is a bit interesting. Do you have steps to reproduce that? |
|
Ok, I've just had a minute to look at this a little closer. Let me see if I understand everything correctly: You performed a (This is mostly for other zfs developers) At this point, zfs hit an ASSERT and panicked. I think I see what happened here, and it seems to be a few bugs combined. I'm making another guess here that your datasets under You attempted to send your data from one pool to another using a non-raw send. This requires the key to be loaded (so the error is correct). You saw this error and attempted to load the encryption key for the dataset itself. Unfortunately, this dataset is inheriting it's key from Your encryption root is Honestly not sure what happened here, but it is almost certainly unrelated and best dealt with in a separate issue (such as #6650 which you opened, so thank you for that). I suspect it doesn't really have anything to do with memory, but that another error is simply printing the wrong message. I should be able to look into fixing the primary issue on Monday. I'll post an update when I have it. |
@tcaputi that definitely looks possible. I don't see a read-only check in |
|
It seems that this issue has been popping up randomly for quite some time, so its unlikely that my patch will fix anything. Nevermind. |
|
Looking through my logs I noticed that I used |
|
Have you tried loading the key for the child datasets? You should need to load the key for |
|
@tcaputi I've gone through Basically can go through all child datasets with each telling me to load the key for its parent and keystatus being unavailable and mounting/sending not working till the original encryption root being |
|
OK. Now I understand. So the problem is that your encryption hierarchy is messed up and I'm not sure how it got that way. |
Could go further back where DataTemp received zfs sends from DataBackup or where I created DataBackup 4 months ago with all datasets unencrypted when I moved each of them into enc encrypted dataset and then all of those into my main pool (which is now corrupted)... Extensive Log |
|
I think this might be fixed with #7200. We were unable to reproduce the issue with the commands you gave us, but that patch fixes some of the error handling in the receive code path. Specifically, there was an issue where a failed receive would still update the encryption keys, even though the rest of the state was rolled back. When that patch is merged, can you give it a shot and let us know if the problem happens again. |
|
Sure, I'll get this on both sender and receiver. Do I need to recreate the dataset that already exists on the receive side (since I'm doing incrimentals)? |
|
I would just to be safe. I'm still not sure the exact steps to get into your state |
|
@prometheanfire any update here? |
|
I ran into the issue that you fixed Monday. I can't test the fix for a week though due to travel (stuck in dublin for now). But I think we are fixed here (from what I've seen here and elsewhere). |
|
Currently I can't load the key anymore as zfs reports that the given passphrase (I copied from my password manager) is wrong. Not sure what's that about. |
|
@prometheanfire If you are ok with it then, would you mind closing this ticket? Unless you have other concerns. @Celmor This sounds similar to a bug that should be fixed in master. Would you mind repeating your sends with that code and, if it's still a problem, opening a new ticket? I'm going on vacation for a week soon, so this might have to wait until I get back. |
|
ok, I'm good with closing this, sent raw and also incremental raws (I don't have perms to do so, and I was not the initial reporter) for SNAPSHOT in $(zfs list -t snap | cut -d@ -f 1 | uniq | tail -n+2); do
SIZE=$(zfs send -LwecpnvP -I ${SNAPSHOT}@${PREVIOUS_BACKUP_DATE} ${SNAPSHOT}@backup-${DATE} | tail -n 1 | awk '{ print $2 }')
if [[ "$(zfs get -H -o value type ${SNAPSHOT})" == 'volume' ]]; then
echo "zfs send -Lwecp -I ${SNAPSHOT}@${PREVIOUS_BACKUP_DATE} ${SNAPSHOT}@backup-${DATE} | pv -s "${SIZE}" | ssh 10.0.1.14 zfs recv -uvs storage-pool/remote-backups/${SNAPSHOT}"
else
echo "zfs send -Lwecp -I ${SNAPSHOT}@${PREVIOUS_BACKUP_DATE} ${SNAPSHOT}@backup-${DATE} | pv -s "${SIZE}" | ssh 10.0.1.14 zfs recv -uvs -o canmount=off storage-pool/remote-backups/${SNAPSHOT}"
fi
done |
System information
**added patches to ZFS (expression):SPL expression/patches
Also attempted on arch (4.14.15-1-ARCH) with 0.7.0-255_g6bc4a237 zfs version resulting in same error.
Describe the problem you're observing
Not able to send an encrypted dataset
Describe how to reproduce the problem
Sending from "Backup"-Pool to a new pool, "enc" dataset is encryption root
Another attempt:
Include any warning/errors/backtraces from the system logs
(full dmesg with highlighting)
The text was updated successfully, but these errors were encountered: