File upload fails with encryption on s3 storage "ObjectUpload not found" #10455

TecJon · 2018-07-30T19:40:57Z

Steps to reproduce

Use s3 as external storage
Use default encryption module
Use NC web interface to upload a small image

Expected behaviour

It should upload successfully

Actual behaviour

Upload bar stops at 100% and does not respond anymore.

I should add, this is a test install on a shared web space. However, so far I couldn't make an external storage with encryption work on any installation or release.

Could be related to #10430, #8299, #7420

Server configuration

Web server: shared web space

Database: mysql 5.7.21

PHP version: 7.0.25

Nextcloud version: 14.0.0.13

Updated from an older Nextcloud/ownCloud or fresh install: fresh install

Where did you install Nextcloud from:

Signing status: No errors have been found.

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Nextcloud configuration:

Config report

<?php
$CONFIG = array (
'instanceid' => '',
'passwordsalt' => '',
'secret' => '',
'trusted_domains' =>
array (
0 => '',
),
'datadirectory' => '/www/htdocs/cloud/example/data',
'overwrite.cli.url' => 'http://cloud.example.com/',
'dbtype' => 'mysql',
'version' => '14.0.0.13',
'dbname' => 'nc',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'nc',
'mysql.utf8mb4' => true,
'dbuser' => 'd02b4219',
'dbpassword' => 'DdSAa7fW6DcEtzrdPw8h',
'installed' => true,
'tempdirectory' => '/www/htdocs/cloud/example/tmp/',
'mail_from_address' => 'aa',
'mail_smtpmode' => 'php',
'mail_smtpauthtype' => 'LOGIN',
'mail_domain' => 'aa',
'updater.release.channel' => 'beta',
'maintenance' => false,
'theme' => '',
'loglevel' => 2,
);

Are you using external storage, if yes which one: yes, s3

Are you using encryption: yes

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Firefox 61 64 Bit

Operating system: shared webspace

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

Error: Call to undefined method OC\Files\Stream\Encryption::isDirectoryHandle()

Uncaught Error: Class 'Aws\S3\ObjectUploader' not found in /www/htdocs/cloud.example.com/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php:46 Stack trace: #0 /www/htdocs/cloud.example.com/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php(32): Aws\S3\S3Client->uploadAsync('name', 'New folder/drou...', false, 'private', Array) #1 /www/htdocs/cloud.example.com/lib/private/Files/ObjectStore/S3ObjectTrait.php(77): Aws\S3\S3Client->upload('name', 'New folder/drou...', false, 'private', Array) #2 /www/htdocs/cloud.example.com/apps/files_external/lib/Lib/Storage/AmazonS3.php(589): OCA\Files_External\Lib\Storage\AmazonS3->writeObject('New folder/drou...', false) #3 /www/htdocs/cloud.example.com/apps/files_external/lib/Lib/Storage/AmazonS3.php(419): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/www/htdocs/w01...', 'New folder/drou...') #4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure} at /www/htdocs/cloud.example.com/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php#46

Browser log

Content Security Policy: Directive ‘child-src’ has been deprecated. Please use directive ‘worker-src’ to control workers, or directive ‘frame-src’ to control frames respectively.
JQMIGRATE: Migrate is installed, version 1.4.0 core.js:7:542
Content Security Policy: The page’s settings blocked the loading of a resource at https://cloud.example.com/index.php/apps/files/ (“script-src”).
<script> source URI is not allowed in this document: “https://cloud.example.com/apps/notifications/js/notifications.js?v=4554f71a-0”. files:55
window.controllers/Controllers is deprecated. Do not use it for UA detection. merged.js:2171
New search handler registered search.js:66:4
querying Whats New data was successful: nocontent merged-template-prepend.js:4455:4
undefined merged-template-prepend.js:4456:4
[Show/hide message details.] TypeError: xml is undefined[Learn More] merged-index.js:664:1
unreachable code after return statement[Learn More]

The text was updated successfully, but these errors were encountered:

nextcloud-bot · 2018-07-30T19:47:04Z

GitMate.io thinks possibly related issues are #4730 (Storage), #3748 (Primary S3 Storage Backend + Encryption problem with file size, mimetype and first download failed), #8954 (Uploading large files fails), #8299 (Decryption failed with s3), and #3294 (No Previews/Thumbnails with Primary Storage S3 + Encryption).

schiessle · 2018-07-30T20:58:24Z

@TecJon can you test if this fix also solves the issue for you?

TecJon · 2018-07-30T21:50:39Z

@schiessle I am not sure what you mean. Do you speak of the branch “encryption-s3-fix” commited 6 days ago?

schiessle · 2018-07-31T08:20:00Z

@TecJon yes, sorry... Forgot to add the link: #10397 😉

TecJon · 2018-07-31T16:52:08Z

@schiessle I'm afraid the error stays the same.

To avoid any misunderstanding, I had to manually upload the changed files. Only 3 relevant files changed compared to NC 14 beta 1?
apps/encryption/lib/Crypto/Encryption.php
lib/private/Files/Stream/Encryption.php and
tests/lib/Files/Stream/EncryptionTest.php

schiessle · 2018-08-01T15:29:31Z

@TecJon that's correct. This will not fix the problem for existing files, but should solve the issue for newly uploaded files after the fix was deployed. Did you tested it with old or new files?

TecJon · 2018-08-01T16:06:12Z

@schiessle I still can't upload (new) files. The progress bar will go to 100% and then nothing happens. Refreshing shows an empty folder. Not sure if related, I'm getting many 504 gateway timeouts suddenly on my test server.

#10430 (comment) is the error log of the one click web host test install.

schiessle · 2018-08-02T09:55:10Z

@TecJon can you check the storage directly? Was the file uploaded. Also check the oc_file_cache table if there is a entry for the file you tried to upload.

TecJon · 2018-08-02T14:15:49Z

@schiessle here's what I found. Using NC 14 beta 2.

Without adding the function isDirectoryHandle in lib/private/Files/Stream/Encryption.php

I can add a folder in the encrypted s3 container
it does appear in the storage also
I cannot upload a file in this folder
it does not seem to be in oc_file_cache either
first error Error: Call to undefined method OC\Files\Stream\Encryption::isDirectoryHandle() appears
then this Uncaught Error: Class 'Aws\S3\ObjectUploader' not found in /var/www/nextcloud_test/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php:46 Stack trace: #0 /var/www/nextcloud_test/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php(32): Aws\S3\S3Client->uploadAsync('nc15testbucket', 'test1/New Text ...', false, 'private', Array) #1 /var/www/nextcloud_test/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php(77): Aws\S3\S3Client->upload('nc15testbucket', 'test1/New Text ...', false, 'private', Array) #2 /var/www/nextcloud_test/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php(589): OCA\Files_External\Lib\Storage\AmazonS3->writeObject('test1/New Text ...', false) #3 /var/www/nextcloud_test/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php(419): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_cs7...', 'test1/New Text ...') #4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}() #5 /var/www/nextcloud_test/next at /var/www/nextcloud_test/nextcloud/3rdparty/aws/aws-sdk-php/src/S3/S3ClientTrait.php#46

After adding the function isDirectoryHandle in lib/private/Files/Stream/Encryption.php

the file can be uploaded via the web interface, no errors in nextcloud log
the file appears in the s3 container, too (encrypted).
I can open the txt file on the web interface, no errors this time
download is also possible
uploading a png file works without visible errors. No preview is available. browser log GET https://example.com/core/preview?fileId=356&c=5b63105fceb5e&x=32&y=32&forceIcon=0 503 ()
Trying to download the image produces a 503 error.
nextcloud.log: Missing Signature

error log

edit

Trying to download the image again resulted in different errors:

error 1

{"reqId":"0toespJjVlNaIqyv17tR","level":4,"time":"2018-08-02T14:52:12+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"webdav","method":"GET","url":"\/remote.php\/webdav\/s3-test\/test1\/contac.PNG?downloadStartSecret=bcg76jk50xk","message":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Encryption library: Decryption (symmetric) of content failed: ","Code":0,"Trace":[{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","line":469,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/encryption\/lib\/Crypto\/Encryption.php","line":380,"function":"symmetricDecryptFileContent","class":"OCA\\Encryption\\Crypto\\Crypt","type":"->","args":["*** sensitive parameter replaced ***","*** sensitive parameter replaced ***","*** sensitive parameter replaced ***",1,"*** sensitive parameter replaced ***"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Stream\/Encryption.php","line":469,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Stream\/Encryption.php","line":300,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[]},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/Wrapper.php","line":83,"function":"fread","args":[null,8192]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php","line":91,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/3rdparty\/sabre\/http\/lib\/Sapi.php","line":80,"function":"stream_copy_to_stream","args":[null,null,"45344"]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":498,"function":"sendResponse","class":"Sabre\\HTTP\\Sapi","type":"::","args":[{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https:\/\example.com\/remote.php\/webdav\/s3-test\/test1\/contac.PNG?downloadStartSecret=bcg76jk50xk","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php","line":80,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"\/var\/www\/nextcloud_test\/nextcloud\/remote.php","line":163,"args":["\/var\/www\/nextcloud_test\/nextcloud\/apps\/dav\/appinfo\/v1\/webdav.php"],"function":"require_once"}],"File":"\/var\/www\/nextcloud_test\/nextcloud\/apps\/encryption\/lib\/Crypto\/Crypt.php","Line":586,"Hint":"Encryption library: Decryption (symmetric) of content failed: ","CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"14.0.0.14"}

and

error 2

{"reqId":"vFaR3ZjtyxBsDIxA8hBY","level":3,"time":"2018-08-02T14:52:22+00:00","remoteAddr":"xx.xxx.x.xx","user":"User","app":"PHP","method":"GET","url":"\/avatar\/User\/32?v=0","message":"Uncaught Error: Access to undeclared static property: OC\\Files\\Filesystem::$normalizedPathCache in \/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Filesystem.php:797\nStack trace:\n#0 \/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/View.php(2056): OC\\Files\\Filesystem::normalizePath('\/User\/files\/s3...')\n#1 \/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/View.php(1180): OC\\Files\\View->unlockFile('\/s3-test\/test1\/...', 1)\n#2 [internal function]: OC\\Files\\View->OC\\Files\\{closure}()\n#3 \/var\/www\/nextcloud_test\/nextcloud\/apps\/files_external\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(109): call_user_func(Object(Closure))\n#4 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_close()\n#5 {main}\n thrown at \/var\/www\/nextcloud_test\/nextcloud\/lib\/private\/Files\/Filesystem.php#797","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/61.0.3163.91 Safari\/537.36","version":"14.0.0.14"}

Hope this helps 👍

schiessle · 2018-08-03T10:35:50Z

I think the missing signature error should be fixed here: #10397
Did you tested 14.0.0 beta2 with or without this patch. As said in the other issue, I really don't understand this isDirectoryHandle() issue and can't reproduce it.

@icewind1991 any idea?

TecJon · 2018-08-03T12:00:09Z

Did you tested 14.0.0 beta2 with or without this patch.

Without it. Tested it again with the patch and the result looks like this.
Note, the error does not appear on a very small text file. This can be edited and downloaded. The image files (44KB) or a larger text file (3MB) will still produce missing signature or a 503 error.

nextcloud log

As said in the other issue, I really don't understand this isDirectoryHandle() issue and can't reproduce it.

That is strange. It happened to me on both test installations on completely different servers.

rullzer · 2018-08-08T17:57:48Z

Should be fixed with #10397

schiessle · 2018-08-10T12:20:40Z

This should be fixed now with Nextcloud 1 beta 3. Please re-open the ticket and/or reply here if the problem still persist for you.

TecJon added feature: encryption (server-side) feature: external storage 14-beta-feedback labels Jul 30, 2018

rullzer mentioned this issue Aug 8, 2018

Nextcloud 14 Beta 3 #10502

Closed

58 tasks

TecJon mentioned this issue Aug 8, 2018

Decryption failed with s3 #8299

Closed

rullzer closed this as completed Aug 8, 2018

nextcloud-bot mentioned this issue Aug 20, 2018

Decryption (symmetric) of content fails using s3 #10767

Closed

nextcloud-bot mentioned this issue Oct 14, 2018

Encryption not working with S3 object storage as primary storage #11826

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

File upload fails with encryption on s3 storage "ObjectUpload not found" #10455

File upload fails with encryption on s3 storage "ObjectUpload not found" #10455

TecJon commented Jul 30, 2018 •

edited

Loading

nextcloud-bot commented Jul 30, 2018

schiessle commented Jul 30, 2018

TecJon commented Jul 30, 2018

schiessle commented Jul 31, 2018

TecJon commented Jul 31, 2018

schiessle commented Aug 1, 2018

TecJon commented Aug 1, 2018

schiessle commented Aug 2, 2018

TecJon commented Aug 2, 2018 •

edited

Loading

schiessle commented Aug 3, 2018

TecJon commented Aug 3, 2018 •

edited

Loading

rullzer commented Aug 8, 2018

schiessle commented Aug 10, 2018

File upload fails with encryption on s3 storage "ObjectUpload not found" #10455

File upload fails with encryption on s3 storage "ObjectUpload not found" #10455

Comments

TecJon commented Jul 30, 2018 • edited Loading

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Nextcloud log (data/nextcloud.log)

Browser log

nextcloud-bot commented Jul 30, 2018

schiessle commented Jul 30, 2018

TecJon commented Jul 30, 2018

schiessle commented Jul 31, 2018

TecJon commented Jul 31, 2018

schiessle commented Aug 1, 2018

TecJon commented Aug 1, 2018

schiessle commented Aug 2, 2018

TecJon commented Aug 2, 2018 • edited Loading

edit

schiessle commented Aug 3, 2018

TecJon commented Aug 3, 2018 • edited Loading

rullzer commented Aug 8, 2018

schiessle commented Aug 10, 2018

TecJon commented Jul 30, 2018 •

edited

Loading

TecJon commented Aug 2, 2018 •

edited

Loading

TecJon commented Aug 3, 2018 •

edited

Loading