iOS app: HTTP 500 after uploading ~ 44'000 photos

[nielsson4711]

Steps to reproduce

1. Provide an iPhone with approx. 44,000 photos in the iCloud/camera memory.

2. Connect the NC app via user login to a NC instance (in the LAN) with sufficient storage space.

3. Configure the iOS device to not lock the screen at all.

4. Configure this in the settings:

• automatic upload
• separate folder (not default "Photos")
• ... of pictures (only via WLAN)
• ... of videos (only via WLAN)
• !! upload complete camera memory !!!
• Use subfolders
• Subfolder granularity: monthly
• Filename mask: YYYY-MM-DDThh.mm.ss_

Expected behaviour

After a reasonable transfer and processing time all photos are uploaded to the NC instance AND all uploaded photos are removed from the iOS device.

Actual behaviour

1. After a reasonable transfer and processing time (~ 36h) all photos are uploaded to the NC instance.
2. The Count indicator at the apps ui left bottom shows a small number of files to upload (once 3, once 1, once 8).
3. A red banner at the apps ui top saying "file upload: 500: internal server error" shows up cyclically. It seems to be coupled with the "upload file counter". The app tries to upload the files and repeatedly ends up in HTTP 500.
4. NO photo got removed from the ios device
5. The server reports HTTP 500 for files which are obviously uploaded

AND all uploaded photos are removed from the iOS device.

fetching for existing / missing files along HTTP 500
for f in $(grep 'HTTP/1.1" 500' syslog daemon.log |
sed -e 's# HTTP/1.1.*$##' |
sed -s 's#^.* - testuser ##' |
sed -e 's#^.*PUT /remote.php/dav/files/testuser/##' | sort -u);
do [ -e "/srv/nextcloud/data/testuser/files/$f" ] &&
(echo -n "file exists: "; ls -lih "/srv/nextcloud/data/nisse/files/$f" ) || "file missing: $f";
done

Screenshots

If applicable, add a screenshot showing the issue.

Logs

iPad logs of the whole upload are attached.

If applicable, you can post the iOS app or server logs (removing any sensitive information).
```



### Reasoning or why should it be changed/implemented?
Moving the whole camera storage seems to fail on larger amounts of media files.
Maybe i'm not the only person with a larger set of photos.

A staggered upload, e.g. in stacks of e.g. 5000 files (adjustable?)  could be of help.


### Environment data
apache docker container with PHP Memory Limit of 640MByte. I'd extend the memory limit from 512 to 640Mbyte as the first upload experiments ended up with some of these messages: "PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 4005144 bytes) in /var/www/html/custom_apps/maps/lib/Helper/ExifGeoData.php on line 120"


**iOS version:** e.g. iOS 14.4.1
iPhone SE (2) iOS 16.5.1
iPad Pro 10.5" iOS 16.5.1

**Nextcloud iOS app version:** see More > Settings
Nextcloud Liquid for iOS 4.8.5.1
[ipad-communication2-1.log](https://github.com/nextcloud/ios/files/12011605/ipad-communication2-1.log)
[ipad-communication2-2.log](https://github.com/nextcloud/ios/files/12011606/ipad-communication2-2.log)
[ipad-communication2-3.log](https://github.com/nextcloud/ios/files/12011607/ipad-communication2-3.log)

**Server operating system:**
Linux nextbox 5.10.103-v8+ #1529 SMP PREEMPT Tue Mar 8 12:26:46 GMT 2022 aarch64 GNU/Linux

**Web server:** Apache, nginx
Server version: Apache/2.4.56 (Debian)
Server built:   2023-04-02T03:06:01

**Database:**
MariaDB
Server version: 10.5.9-MariaDB-1:10.5.9+maria~focal mariadb.org binary distribution

**PHP version:**
HP 8.2.7 (cli) (built: Jun 13 2023 23:22:26) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.7, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.7, Copyright (c), by Zend Technologies

**Nextcloud version:** see Nextcloud admin page
26.0.3

[marinofaggiana]

Hi, thanks for you report, but it's impossible for me understand the origin of your HTTP 500, probable some photo/video is remain in loop without the possibility of upload ..

[nielsson4711]

Hi, thanks for you report, but it's impossible for me understand the origin of your HTTP 500, probable some photo/video is remain in loop without the possibility of upload ..

What does

More information

• data provided (01):
  • X-OC-MTime header failure - see below
• Storage: 6.2TByte available, for details see below.
• Memory: see "free" output below.

data provided (01)

data provided to pin down "root" causes.
One example:
{
"reqId": "some_crazy-stringified_id",
"level": 3,
"time": "2023-07-11T20:01:44+00:00",
"remoteAddr": "10.20.10.20",
"user": "testuser",
"app": "webdav",
"method": "PUT",
"url": "/remote.php/dav/files/testuser/Photos/2010/11/2010-11-01T08.21.52_9999.jpg",
"message":X-OC-MTime header must be a valid positive integer",
"userAgent": "Mozilla/5.0 (iOS) Nextcloud-iOS/4.8.5",
"version": "26.0.3.2",
"exception": {
"Exception": "InvalidArgumentException",
"Message": "X-OC-MTime header must be a valid positive integer",
"Code": 0,
"Trace": [
{
"file": "/var/www/html/apps/dav/lib/Connector/Sabre/Node.php",
"line": 425,
"function": "sanitizeMtime",
"class": "OCA\\DAV\\Connector\\Sabre\\MtimeSanitizer",
"type": "::",
"args": [
"0.0"
]
},
{
"file": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
"line": 378,
"function": "sanitizeMtime",
"class": "OCA\\DAV\\Connector\\Sabre\\Node",
"type": "->",
"args": [
"0.0"
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 1137,
"function": "put",
"class": "OCA\\DAV\\Connector\\Sabre\\File",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
"line": 492,
"function": "updateFile",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": [
"*** sensitive parameters replaced ***"
]
},
{
"file": "/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
"line": 89,
"function": "httpPut",
"class": "Sabre\\DAV\\CorePlugin",
"type": "->",
"args": [
[
"Sabre\\HTTP\\Request"
],
[
"Sabre\\HTTP\\Response"
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 472,
"function": "emit",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": [
"method:PUT",
[
[
"Sabre\\HTTP\\Request"
],
[
"Sabre\\HTTP\\Response"
]
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 253,
"function": "invokeMethod",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": [
[
"Sabre\\HTTP\\Request"
],
[
"Sabre\\HTTP\\Response"
]
]
},
{
"file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
"line": 321,
"function": "start",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": []
},
{
"file": "/var/www/html/apps/dav/lib/Server.php",
"line": 366,
"function": "exec",
"class": "Sabre\\DAV\\Server",
"type": "->",
"args": []
},
{
"file": "/var/www/html/apps/dav/appinfo/v2/remote.php",
"line": 35,
"function": "exec",
"class": "OCA\\DAV\\Server",
"type": "->",
"args": []
},
{
"file": "/var/www/html/remote.php",
"line": 172,
"args": [
"/var/www/html/apps/dav/appinfo/v2/remote.php"
],
"function": "require_once"
}
],
"File": "/var/www/html/apps/dav/lib/Connector/Sabre/MtimeSanitizer.php",
"Line": 37,
"message": "X-OC-MTime header must be a valid positive integer",
"exception": {},
"CustomMessage": "X-OC-MTime header must be a valid positive integer"
}
}

Storage

The NC servers data volume still provides 6.2TByte storage for files, dbms, logs.

Filesystem Size Used Avail Use% Mounted on
overlay 7.3T 716G 6.2T 11% /
tmpfs 64M 0 64M 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdc1 7.3T 716G 6.2T 11% /etc/apache2
shm 64M 0 64M 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /sys/firmware

Memory

There seems to be plenty of memory available:
# free -m
total used free shared buff/cache available
Mem: 7813 1086 411 355 6315 6249
Swap: 2047 158 1889

[marinofaggiana]

mmm have you used the chunk mode ? (> 0)

[juergen852]

Did you check user storage quota?

[judicandus]

I am having exactly the same problem. Fresh new installation of Nextcloud. Related problem: Issue 31464

[mike12806]

I’m having the same issue with fresh install of latest Nextcloud and iOS app.

[nielsson4711]

mmm have you used the chunk mode ? (> 0)

no chunk mode (as it targets large file and not large amounts fo files)

[nielsson4711]

Did you check user storage quota?

no storage quota.
and no stroage shortages, rather the opposite: enhanced a "nitroykey nextbox" with am 8TB SSD.

[NlightN22]

Have the same problem on server 27.1.3 and android client 3.26.0 with 23 files on client side. Tried to update server to the latest version, but it didn't help.
On server syslogs see same error "X-OC-MTime header must be a valid positive integer"