Moving files/folders from/to shared folders causes Encryption Errors

[arcanatigris]

Steps to reproduce

1. Enable Encryption
2. Let user1 share 2 folders, place some dummy content in folder1 with multiple layers of directories. (folder1/test/test2/file.txt)
3. Let user2 move the folder inside folder1 to folder2 (Using web GUI)
4. Process get stuck and Encryption errors in Logging

Expected behaviour

Folder should move without errors

Actual behaviour

Encryption errors

Server configuration

Operating system: Debian 9

Web server: NGINX

Database: MariaDB, Redis

PHP version: 7.3

Nextcloud version: 16.0.3

Updated from an older Nextcloud/ownCloud or fresh install: fresh

Where did you install Nextcloud from: latest archive

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.2.0
  - activity: 2.9.1
  - bruteforcesettings: 1.4.0
  - cloud_federation_api: 0.2.0
  - comments: 1.6.0
  - dav: 1.9.2
  - encryption: 2.4.0
  - federatedfilesharing: 1.6.0
  - federation: 1.6.0
  - files: 1.11.0
  - files_pdfviewer: 1.5.0
  - files_rightclick: 0.13.0
  - files_sharing: 1.8.0
  - files_texteditor: 2.8.0
  - files_trashbin: 1.6.0
  - files_versions: 1.9.0
  - files_videoplayer: 1.5.0
  - firstrunwizard: 2.5.0
  - gallery: 18.3.0
  - logreader: 2.1.0
  - lookup_server_connector: 1.4.0
  - nextcloud_announcements: 1.5.0
  - notifications: 2.4.1
  - oauth2: 1.4.2
  - password_policy: 1.6.0
  - privacy: 1.0.0
  - provisioning_api: 1.6.0
  - recommendations: 0.4.0
  - serverinfo: 1.6.0
  - sharebymail: 1.6.0
  - support: 1.0.0
  - survey_client: 1.4.0
  - systemtags: 1.6.0
  - theming: 1.7.0
  - twofactor_backupcodes: 1.5.0
  - twofactor_u2f: 3.0.0
  - updatenotification: 1.6.0
  - viewer: 1.0.0
  - workflowengine: 1.6.0
Disabled:
  - admin_audit
  - files_external
  - user_ldap

Nextcloud configuration:

Config report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "16.0.3.0",
        "overwrite.cli.url": "https:\/\/nc.yisp.nl",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        }
    }
}

Are you using external storage, if yes which one: no

Are you using encryption: yes

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Chrome

Operating system: Ubuntu 19.04

Logs

Nextcloud log GUI

Nextcloud log

OCP\Encryption\Exceptions\GenericEncryptionException: Bad Signature
/var/www/nextcloud/apps/encryption/lib/Crypto/Crypt.php - line 467:

OCA\Encryption\Crypto\Crypt->checkSignature("d+nOgAED6pO ... E", null, "65ac461c517 ... 5")

/var/www/nextcloud/apps/encryption/lib/Crypto/Encryption.php - line 379:

OCA\Encryption\Crypto\Crypt->symmetricDecryptFileContent("*** sensiti ... *", null, "AES-256-CTR", "*** sensiti ... *", "*** sensiti ... *")

/var/www/nextcloud/lib/private/Files/Stream/Encryption.php - line 479:

OCA\Encryption\Crypto\Encryption->decrypt("*** sensiti ... *")

/var/www/nextcloud/lib/private/Files/Stream/Encryption.php - line 299:

OC\Files\Stream\Encryption->readCache()

<<closure>>

OC\Files\Stream\Encryption->stream_read(8192)

/var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php - line 91:

fread(null, 8192)

/var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php - line 98:

Icewind\Streams\Wrapper->stream_read(8192)

<<closure>>

Icewind\Streams\CallbackWrapper->stream_read(8192)

/var/www/nextcloud/3rdparty/sabre/http/lib/Sapi.php - line 80:

stream_copy_to_stream(null, null, "12624")

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 498:

Sabre\HTTP\Sapi::sendResponse(Sabre\HTTP\Response {})

/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 254:

Sabre\DAV\Server->invokeMethod(Sabre\HTTP\R ... "}, Sabre\HTTP\Response {})

/var/www/nextcloud/apps/dav/lib/Server.php - line 316:

Sabre\DAV\Server->exec()

/var/www/nextcloud/apps/dav/appinfo/v2/remote.php - line 35:

OCA\DAV\Server->exec()

/var/www/nextcloud/remote.php - line 163:

require_once("/var/www/ne ... p")

[yahesh]

It seems that we have the same issue (see nextcloud/desktop#1168). We're currently looking into this issue as the corresponding restores take a long time and we'd like to have this issue resolved.

We're trying to reproduce this problem by creating two shared folders and moving huge subfolders between those two shared folders.

One thing we've seen is that at some point the web UI issues as warning message that moving the subfolder failed. However, when watching the target folder we see that files are still moved after this message has been issued. Even after all files seem to have been moved on disk, the corresponding PHP process is still hard at work and the database is active as well. (As long as this task isn't finished, the moved files don't show up in the web UI.)

As we've learned, when moving files around the oc_filecache table has to be rewritten for every single file, which is important for signature checks as the database contains the encrypted value (which is actually a version number) that is required to derive the passphase that is used to calculate the HMAC "signatures" in the moved files.

Furthermore, it seems as if files get completely re-encrypted when copying or moving them between shared folder. In contrast, they "only" get re-encrypted when being copied within the same shared folder. Unnecessary re-encryptions increase the risk of data loss during these actions. (We found out that Nextcloud doesn't move the file but actually creates a copy and with that re-encrypts the file.)

[yahesh]

We now found a reliable way that breaks the signature in 100% of our test cases.

Howto:

1. Create a new text file in a shared folder.
2. Enter some text and wait until it is saved.
3. Enter some more text and wait until it is saved.
4. Try to move the file into another shared folder. You'll get an Could not move "filename.txt" error. At this stage the file is already broken. The working copy of the file will have been moved to the trashbin of the owner of the source folder. A broken copy will be stored on disk in the target folder but the oc_filecache table will not yet have been updated - so the broken copy of the file will not be visible through the web UI.
5. If you have the web UI still open in which you tried to move the file then you have the possibility to "move" the file again. You'll get an Could not move "filename.txt", target exists error because the file already exists on disk but this second try updates the oc_filecache table to make the file visible. The other alternative is to do a ./occ files:scan.

There are some more things we found out in the meantime: We were wondering why the files got re-encrypted when moving them from one shared folder to another shared folder. The reason for the re-encryption is that the file is not really moved but it is rather copied over and then deleted. This leads to several things happening:

• In some cases this re-encryption breaks, making the file unreadable for all users.
• The original copy of the file is moved to the trashbin of the owner of the source folder. This unaltered copy still works.
• Whenever a file is removed from a shared folder, a re-encrypted copy of the file is stored in the trashbin of every user that is allowed to read that shared folder. Depending on the number of users and on the number and size of the files this can lead to a massive storage consumption. This also seems to be the reason why there is so much processing going on in the background. Each moved file has to be re-encrypted for the trashbin of each user with access rights. And all those newly created copies get a separate entry in the oc_filecache table as well.

[yahesh]

Final revelation: Encrypted files must have an encrypted value higher or equal to 1 in oc_filecache or otherwise Nextcloud will assume a bad signature in an encrypted file even if the signature itself is correct.

We found out that Nextcloud stores the value 0 as the encrypted value in oc_filecache for the example file shown above, even though the file is signed with the value 1. Updating the oc_filecache table accordingly fixes the Bad Signature error for this single issue.

The problem persists that moved files may be signed with other encrypted values than 1, we're not sure yet whether rewriting this value in oc_filecache fixes all Bad Signature errors.

[yahesh]

To be able to debug all this we have written two helpful tools by reimplementing the signature checks (and updates) of Nextcloud and by reimplementing the decryption process of Nextcloud. These are standalone scripts that need some configuration information of Nextcloud, but not the Nextcloud codebase itself:

• check-signature.php is able to check the correctness of master-key-encrypted master private keys, pub share private keys, files, version files, trashed files and trashed version files. It is also able to rewrite the signatures within the mentioned file types (by setting the experimental FIXSIGNATURES constant). As we didn't want to implement a database abstraction layer we rely on partial CSV dumps of the two tables oc_filecache and oc_storages. The comment in the script gives an example of how to create these dumps from PostgreSQL.
• decrypt-file.php is able to decrypt a single master-key-encrypted file, version file, trashed file or trashed version file. It ignores the signatures contained in the file and simply writes the decrypted content to STDOUT.

[jospoortvliet]

Thanks for the extensive debugging, that will sure help finding and fixing the issue!

[nickvergessen]

cc @icewind1991

[yahesh]

I just wanted to document some more thoughts on the design of the encryption and signature scheme:

• We were wondering why the key used for the signature has the form $filekey.$version.$position instead of just $filekey. Adding the $position info prevents a properly signed block from being moved within a file (because there would be a $position mismatch). As different file versions reuse the same $filekey, adding the $version info prevents a properly signed block from being moved from one version of the file into another version of the file.
• We were also wondering why the last block uses a slightly different signature key in the form of $filekey.$version.$position."end". By having a different key for the last block you prevent the file from being shortened by one or more blocks as then the signature of the last block wouldn't match anymore.
• Finally: We were wondering why the key for the signature didn't use some identifier of the actual file being signed. But some kind of identifier is in fact included indirectly through $filekey as each new file has its individual file encryption key (as long as it's not a versioned file which is distinguished through $version instead).
• This final revelation in our opinion also is the reason why files have to be re-encrypted when copies of them are created. Otherwise you would have separate files with the same file encryption key and thus would open up a possibility for blocks to be swapped between independent files (as long as their $version and $position within the file also match).

[yahesh]

To get a complete understanding of the inner workings of the default encryption module we now also implemented the support for public sharing keys, recovery keys and user keys in our nextcloud-tools. Our goal is to create a document containing our newly-gained knowledge about how the encryption works.

[yahesh]

We're currently in the process of testing our Nextcloud dataset with the written tools. We stumbled upon lots of files that seemingly have an encrypted value of 0 assigned to them. When we looked into the database, we saw that the files actually have two entries in the oc_filecache table:

• One of them with path in the form of <username>/files/<filename> and storage equal to 1 (which denotes the "data directory" according to the oc_storages table). These entries have the encrypted field set to 1 for regular files.
• The other one with path in the form of files/<filename> and storage with the ID of the home directory of <username> according to the oc_storages table. These entries have the encrypted field set to 0 for regular files.

We're not sure yet where these duplicate entries come from. Even in our test installation some files have duplicate entries in the mentioned formats.

[yahesh]

We also found that you should not use ./occ files:scan when operating an encrypted Nextcloud instance (for example to correct the database after moving files on the file level). Files are added to the oc_filecache table with encrypted equal to 0 even if the files are encrypted.

[nickvergessen]

Uff, yeah files:scan should be really used with care. I would argue to even disable it when encryption is on.

cc @MorrisJobke @rullzer

[MorrisJobke]

We also found that you should not use ./occ files:scan when operating an encrypted Nextcloud instance (for example to correct the database after moving files on the file level). Files are added to the oc_filecache table with encrypted equal to 0 even if the files are encrypted.

It's hard to know when a file is encrypted ... it could also be just a normal file with similar headers. Thus the file:scan can't do much here. Also moving files on the hard disk is just not supported. Doing so causes always (also non-encrypted) issues, because it is from our side just guessing and files are simply just removed and newly added and thus shares are also lost. Long story short: don't move files on filesystem level if you do not want to loose metadata (like shares, encryption state, tags, activities, versions, ...).

[yahesh]

@MorrisJobke Don't misunderstand. This is just one thing we noticed while testing how the encryption module reacts. The problem with bad signature checks also occurs when moving files on the application level (see the mentioned text file example above).

As there doesn't seem to be an extensive description of how the encryption works we have to find this out ourselves.

[yahesh]

@MorrisJobke @nextcloud/encryption @nickvergessen @rullzer Due to this issue why dug pretty deep into the default encryption module and gathered at lot of knowledge about the inner workings of the encryption and signature processes. We created a document called server-side-encryption.md that contains the general knowledge and tiny details that we learned.

Is there the possibility to add this to the official documentation of the encryption module so that this knowledge is easier to find for others?

[nickvergessen]

I guess it could either go into https://github.com/nextcloud/documentation/blob/master/admin_manual/configuration_files/encryption_configuration.rst or in a new page next to it.

[yahesh]

Now that we know how to calculate the MACs and decrypt files we started looking into the actual problems of the encryption module more closely:

We started by creating a reproducible failure again. To do this we created three files called bild.png. We upload the first version of the picture (A) into the shared folder shared. Then we upload the second version of the picture (B) into the same shared folder and finally we upload the third version of the picture (C) into the same shared folder again. The resulting database entries for select fileid, storage, path, encrypted from oc_filecache where path like '%bild%' order by fileid; look like this:

 fileid | storage |                                      path                                      | encrypted 
--------+---------+--------------------------------------------------------------------------------+-----------
    957 |       1 | files_encryption/keys/files/shared/bild.png                                    |         0
    958 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE                  |         0
    959 |       1 | files/shared/bild.png                                                          |         3
    962 |       1 | files_versions/shared/bild.png.v1564491200                                     |         1
    963 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE/fileKey          |         0
    964 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE/ncadmin.shareKey |         0
    965 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE/kenny.shareKey   |         0
    968 |       1 | files_versions/shared/bild.png.v1564491211                                     |         2

Fileid 959 is the third version of the picture (C), fileid 968 is the second version of the file (B) and fileid 962 is the first version of the picture (A). We see that C has the field encrypted set to 3, B has it set to 2 while A has it set to 1.

Now we try to move bild.png to the shared folder shared2. This will break expectedly with an Could not move "bild.png" error. The database now looks like this:

 fileid | storage |                                                path                                                | encrypted 
--------+---------+----------------------------------------------------------------------------------------------------+-----------
    957 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024                                    |         0
    958 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE                  |         0
    959 |       1 | files_trashbin/files/bild.png.d1564492024                                                          |         1
    963 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/fileKey          |         0
    964 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/ncadmin.shareKey |         0
    965 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/kenny.shareKey   |         0
    968 |       3 | files_trashbin/versions/bild.png.v1564491211.d1564492024                                           |         2
    971 |       1 | files_encryption/keys/files/shared2/bild.png                                                       |         0
    972 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE                                     |         0
    973 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE/fileKey                             |         0
    974 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE/ncadmin.shareKey                    |         0
    975 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE/kenny.shareKey                      |         0
    982 |       1 | files_trashbin/versions/bild.png.v1564491211.d1564492024                                           |         2
    985 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024                                    |         0
    986 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE                  |         0
    987 |       3 | files_trashbin/files/bild.png.d1564492024                                                          |         0
    988 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/fileKey          |         0
    989 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/kenny.shareKey   |         0

Fileid 959 (which is C) has been rewritten to be located in the trashbin now but its encrypted field has been changed to 1 even though its signed with the version 3. This breaks the signature of the trashed file and it will also stay broken after a restore.

Fileid 987 is a copy of C which has been put into the trashbin of the user the file has been shared with but its encrypted field has been changed to 0. This would mean that the file is not encrypted but it actually is encrypted, also breaking the signature of this trashed file.

Fileids 968 and 982 are the trashed version files containing B and the version information has stayed intact. The version file containing A has been lost during the moving process.

The actual file containing C that was meant to be moved isn't written to the database at all but it exists on disk. If you still have the website opened where you initially tried to move the file you have the possibility to try and move the file again. This will result in an Could not move "bild.png", target exists error but this will at least fix the database a bit:

 fileid | storage |                                                path                                                | encrypted 
--------+---------+----------------------------------------------------------------------------------------------------+-----------
    957 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024                                    |         0
    958 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE                  |         0
    959 |       1 | files_trashbin/files/bild.png.d1564492024                                                          |         1
    963 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/fileKey          |         0
    964 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/ncadmin.shareKey |         0
    965 |       1 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/kenny.shareKey   |         0
    968 |       3 | files_trashbin/versions/bild.png.v1564491211.d1564492024                                           |         2
    971 |       1 | files_encryption/keys/files/shared2/bild.png                                                       |         0
    972 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE                                     |         0
    973 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE/fileKey                             |         0
    974 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE/ncadmin.shareKey                    |         0
    975 |       1 | files_encryption/keys/files/shared2/bild.png/OC_DEFAULT_MODULE/kenny.shareKey                      |         0
    982 |       1 | files_trashbin/versions/bild.png.v1564491211.d1564492024                                           |         2
    985 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024                                    |         0
    986 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE                  |         0
    987 |       3 | files_trashbin/files/bild.png.d1564492024                                                          |         0
    988 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/fileKey          |         0
    989 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/kenny.shareKey   |         0
    990 |       3 | files_encryption/keys/files_trashbin/files/bild.png.d1564492024/OC_DEFAULT_MODULE/ncadmin.shareKey |         0
    991 |       1 | files/shared2/bild.png                                                                             |         0

As you can see the actual file containing C has now been added as fileid 991 but the encrypted field has been set to 0. This would mean that the file is not encrypted but it actually is encrypted, also breaking the signature of this trashed file. Additionally, a previously missing shareKey file has been added to the database.

[yahesh]

Some more things we learned about copying and moving files around:

• We learned that files with multiple versions can be moved from one shared folder to another shared folder without a problem if both folders are owned by the user executing the file move. No copies of the moved file are put in the trashbin of the recipients.
• We learned that files with multiple versions can be moved from one shared folder to another shared folder if only the source folder is owned by the user executing the file move but a copy of that file will be put in the trashbin of that user and the oc_filecache table will have the wrong encrypted value for that trashed file.
• We learned that files with multiple versions cannot be moved from one shared folder to another shared folder if only the target folder is owned by the user executing the file move.
• We learned that files with multiple versions cannot be moved from one shared folder to another shared folder if no folder is owned by the user executing the file move.
• When the owner of the shared folder deletes a file then it is just moved to their trashbin.
• When a recipient of the shared folder moves a file then it is copied over and each recipient (including the owner) gets another copy of the file added to the trashbin.
• When a recipient of the shared folder deletes a file then each recipient (including the owner) gets a copy of the file added to the trashbin.
• Moving files with multiple versions between shared folders breaks. Copying files with multiple versions between shared folders doesn't break.
• Copying a file somewhere resets the encrypted value of the created file copy back to 1. Copies of files also lose the versions of the source file.

This leads us to the conclusion that files seem to be handled differently depending on whether they are handled by the owner of the shared folder or by the recipient of a shared folder. Putting broken files in the trashbin and failing to properly move a file to a shared folder seem to be two different problems. One seems to be related with the ownership of the source folder while the other seems to be related to the ownership of the target folder.

There are also some database inconsistencies that we saw during our tests:

• When a file is first uploaded then only the following entries are written to the database:

 fileid | storage |                             path                              | encrypted 
--------+---------+---------------------------------------------------------------+-----------
   1095 |       1 | files_encryption/keys/files/shared/bild.png                   |         0
   1096 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE |         0
   1097 |       1 | files/shared/bild.png                                         |         1

• The contents of the OC_DEFAULT_MODULE folder is only added to the database when a new version of that file is added or if it is moved. However, these entries don't seem to be necessary for Nextcloud to properly decrypt the file:

 fileid | storage |                                      path                                      | encrypted 
--------+---------+--------------------------------------------------------------------------------+-----------
   1095 |       1 | files_encryption/keys/files/shared/bild.png                                    |         0
   1096 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE                  |         0
   1097 |       1 | files/shared/bild.png                                                          |         2
   1100 |       1 | files_versions/shared/bild.png.v1564503210                                     |         1
   1101 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE/fileKey          |         0
   1102 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE/ncadmin.shareKey |         0
   1103 |       1 | files_encryption/keys/files/shared/bild.png/OC_DEFAULT_MODULE/kenny.shareKey   |         0

• If the contents of the OC_DEFAULT_MODULE folder has been added to the database then it is properly rewritten when moving the file.
• The contents of the OC_DEFAULT_MODULE folder will not be added to the database if the file is copied somewhere (even if the contents of the OC_DEFAULT_MODULE folder had beed added to the database for the source file).

[hostingnuggets]

@yahesh could you please see and mention the persons did these changes in the codebase which breaks your code? We have to ping them so that they fix what they broke asap else your hard work will just be useless :(

[dcom42]

Same problem here. Dataloss with moving encrypted files/folders occurred first with owncloud, years ago, and still occurs with nextcloud (we're stuck with NC 15, but as far as I can see there's no progress in newer versions).

[raceface2nd]

We started on 15 with the server side encryption and are now on 20. On our side the issue still exists. Some weeks ago a user crashed roughly 25% of all data in one moment. Luckily I could restore full file system and db backups which took me about 48 hrs restless work.

But especially with this experience I fear to deactivate the server side encryption or to fresh install.

Our system is hosting millions of files of like 10 years.

The good thing of this event was that I now know that my backup strategy works and where to optimize it.

[PVince81]

I've looked into Roeland's fix in #25249 and discovered that the problem is not reproducible any more in recent Nextcloud versions, so it is likely that the fix is not needed any more as it was fixed through a different way.

If you do have files with "bad signature", please run the command mentioned in https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#problems-when-downloading-or-decrypting-files for the affected files/folders to repair their signature. After that, I expect that move operations won't cause any troubles.

After that, please let me know if anyone is still experiencing "bad signature" errors on files after moving.

[PVince81]

moving a folder structure worked fine for me in NC 20.0.10 and 21.0.4, but it used to be broken.

I'll close this for now.
Feel free to reopen if you still get "bad signature" after a move, but only if you repaired your files first with #16419 (comment)

[redtux]

Hi, I don't fully understand why this has been closed, as the issue is still there. Are you saying it has been fixed for the latest release only? If so the issue should be kept open until the patch has been backported for older (but still supported) releases. Thank you!

[PVince81]

NC 20.0.12 is the latest supported version and in my tests I couldn't reproduce the issue there (but could on older NC 20.0 versions), so the fix must be there already.

@redtux what version are you on ?

[redtux]

Hi @PVince81, thank you for your quick reply! I am using NC 21.0.3 on a Hetzner instance - apart from Nextcloud GmbH probably the biggest provider of NC instances atm. When trying to open an affected PDF inside the browser (I have kept them untouched for the last years just in case this gets fixed some day, so I could test the patch with the affected files), I get the following error:

PDF.js v2.5.207 (build: 0974d6052).
Message: Unexpected server response (500) while retrieving PDF.

Or are those files just unrecoverable crypto junk now that can be deleted?

[PVince81]

see https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#problems-when-downloading-or-decrypting-files and run it on the file in question

this is only if the error in log is "bad signature", for anything else it could be a different issue

[redtux]

Hi, I already tried - but it is not working…

• Let's say my user is redtux and my files are stored in /var/www/html/data/redtux.
• First I run
  • occ encryption:scan:legacy-format (see here)
• Then I run
  • occ encryption:fix-encrypted-version redtux --path="/my-docs/My Broken Doc v0123.pdf"
• This results in an error

  • Path "/redtux/files/my-docs/My Broken Doc v0123.pdf" does not exist. Please provide a valid path.

Verifying the content of whole directory shows for every single file status is: OK, but the respective file is missing. (Edit: The file path is shown but the status line below is missing.)

The file is listed when connecting via WebDAV, but accessing (opening/downloading) it results in an error 500…

[PVince81]

@redtux can you check:

1. select * from oc_filecache where path like '%My Broken Doc v0123%' and see what value there is on the "encrypted" column
2. then see in the data directory whether "My Broken Doc.pdf" is encrypted or not, for example try opening it with a PDF viewer
3. if encrypted, run "head My Broken Doc.pdf" to see the first bytes and see if it starts with "HBEGIN" or not

Depending what you find, a manual repair might be necessary as the "fix-encrypted-version" command didn't seem to work.

[harcesz]

Seemingly just hit the same issue on 22.2.7, seems mostly documents (and not graphic files) were affected. Will try to recover based on the answers here and update.

[harcesz]

I seem unable to retrieve the files, so nextcloud basically shredded a month of my teams work while moving a bunch of files from a normal (encrypted) directory to a shared one. I'd say this is rather critical, as it undermines any trust in this solution and fails at it's core functionality,

[redtux]

I'd say this is rather critical, as it undermines any trust in this solution and fails at its core functionality.

I agree.

Concerning the lost work: In case you have no external backup (you could use a service user with access to the respective folders), I hope that at least one device has not synced the data chunk (or that at least one team member has a backup) so that you can recover the files.

[yahesh]

@harcesz You could try to recover your files using our rescue tool.

[redtux]

@harcesz This looks nice, did not know about that.

The problem with this PHP script or with the SQL statement provided by @PVince81 is that those solutions only work with on-premise installations, not with hosted NextCloud instances (like Hetzner's Storage Share).

In those (not so rare) cases the hoster normally just tells you not to use encryption, which IMHO really is a shame - and negatively impacts the reputation of NextCloud.

[jknockaert]

In those (not so rare) cases the hoster normally just tells you not to use encryption, which IMHO really is a shame - and negatively impacts the reputation of NextCloud.

Unfortunately this reputation is well deserved. There's a long history of new features that were announced with great fanfare upon (pre-)release (such as end-to-end encryption, virtual drive) and which receive little or no attention afterwards. When they break upon an upgrade the devs often don't react at all (even when you take the effort of identifying the author of the PR that broke the feature). I guess it has to do with the business model where paying customers do not use these features. It makes perfect sense not to dedicate paid dev time to such issues. But it would be better if Nextcloud made a clear statement about which features are deprecated so that users know what to reasonably expect. As one of the authors of the encryption stream wrapper I already indicated that I am no longer supporting any code I contributed to the encryption feature of Nextcloud. In current circumstances it's simply not possible to fix any issues that arise (I tried to but hit the wall when I needed feedback from the devs).

[shibco]

Why is this issue closed when people are still reporting it as a reproduce-able bug?

[yahesh]

@shibacomputer Who reported this as a reproducible bug? The last comments in this issue are one year old now. Files that broke due to this bug aren't fixed automagically but need to be fixed through the introduced occ encryption:fix-encrypted-version command. Alternatively, the files can be recovered by using the encryption-recovery-tools.

[shibco]

Under Nextcloud official policy, is advising users to use system scripts to mitigate data loss considered a successful bugfix?

The last comments in this issue are one year old now.

Yes, and the issue was reproducible a year ago. Outside of a bunch of php scripts, has the actual underlying issue been fixed? Because the comments from a year ago suggest that Nextcloud encryption should not be enabled.

[yahesh]

Yes, and the issue was reproducible a year ago. Outside of a bunch of php scripts, has the actual underlying issue been fixed? Because the comments from a year ago suggest that Nextcloud encryption should not be enabled.

From personal experience the issue discussed here is not reproducible anymore. The mentioned solutions were developed to help those people who encountered the issue before it was fixed.

[redtux]

From personal experience the issue discussed here is not reproducible anymore.

Hi @yahesh, could you elaborate a bit on this? It might help others stumbling through this issue.

Are you referring to new installations? As has been explained by the issue closer before, the issue is not automatically fixed for old files, and there is no UI element for doing so (nor is such a tool or app planned upstream from what I understand).

In case your files are being stored on an external cloud hosted by a third party like Hetzner, you might ask them to run an occ script or to make manual changes to the database, but I am afraid they will just say no (for understandable reasons).

So while the bug might have been fixed for newly created files, I cannot confirm that the issue is not reproducible anymore - nor did the issue closer claim so. Therefore I'd be interested in your experience, as this might help others - and as such could increase the user experience with Nextcloud for a lot of people and enterprises that ran into this issue in the past. Thank you a lot in advance!

[yahesh]

@redtux Reproducibility refers to that moving files between shared folders doesn't break them anymore. If it still does, then please say so. Never in my life as a software developer did non-reproducibility mean that already-broken data are magically fixed. How those data can be fixed has been mentioned several times now. If your supplier is unwilling to do it, then you probably want to choose a different one or operate your critical applications yourself.

As I'm not paid to provide Nextcloud support, I don't feel obliged to elaborate much more on this topic. There are enough of my comments in this issue that go into great detail how this bug was tracked down and fixed as well as how broken files can be recovered.

[redtux]

Okay, this might have been a misunderstanding on my side then. I can confirm that encrypted files are not "broken" anymore when moved as explained.

NB: The remaining "issue" is (or might be) that old files can only be repaired if you have an on-premise installation or can convince the admin to do a risky operation on their system. Maybe sharing the files via federation and running the script on the remote Nextcloud instance might work. I shall try this out.

Btw I fully understand the situation and I am not complaining that the issue has been closed. I just misunderstood your claim I guess, so thank you for the clarification.