Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: FilesMetadataException - json cannot exceed 100000 characters long #42308

Open
4 of 8 tasks
Tracked by #3888
SystemKeeper opened this issue Dec 15, 2023 · 22 comments
Open
4 of 8 tasks
Tracked by #3888
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback bug feature: previews and thumbnails

Comments

@SystemKeeper
Copy link
Contributor

⚠️ This issue respects the following points: ⚠️

Bug description

Saw this exception in the log:

{
    "reqId": "LKZ6NZIAZIqyK75Gy13j",
    "level": 3,
    "time": "2023-12-14T23:50:46+01:00",
    "remoteAddr": "",
    "user": "--",
    "app": "core",
    "method": "",
    "url": "--",
    "message": "{\"Exception\":\"OCP\\\\FilesMetadata\\\\Exceptions\\\\FilesMetadataException\",\"Message\":\"json cannot exceed 100000 characters long\",\"Code\":0,\"Trace\":[{\"file\":\"/var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php\",\"line\":116,\"function\":\"saveMetadata\",\"class\":\"OC\\\\FilesMetadata\\\\FilesMetadataManager\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":105,\"function\":\"refreshMetadata\",\"class\":\"OC\\\\FilesMetadata\\\\FilesMetadataManager\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":101,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":101,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":90,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":81,\"function\":\"scanFilesForUser\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/public/BackgroundJob/Job.php\",\"line\":81,\"function\":\"run\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php\",\"line\":102,\"function\":\"start\",\"class\":\"OCP\\\\BackgroundJob\\\\Job\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/lib/public/BackgroundJob/TimedJob.php\",\"line\":92,\"function\":\"start\",\"class\":\"OCP\\\\BackgroundJob\\\\TimedJob\",\"type\":\"->\"},{\"file\":\"/var/www/nextcloud/cron.php\",\"line\":152,\"function\":\"execute\",\"class\":\"OCP\\\\BackgroundJob\\\\TimedJob\",\"type\":\"->\"}],\"File\":\"/var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php\",\"Line\":179,\"message\":\"Error while running background job (class: OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob, arguments: )\",\"exception\":{},\"CustomMessage\":\"Error while running background job (class: OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob, arguments: )\"}",
    "userAgent": "--",
    "version": "28.0.0.11"
}

Steps to reproduce

No idea, happened in the background job.

Expected behavior

Should not fail.

Installation method

Community Manual installation with Archive

Nextcloud Server version

28

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

No response

List of activated Apps

No response

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

@SystemKeeper SystemKeeper added bug 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Dec 15, 2023
@witchent
Copy link

I get the same bug using the docker image and postgres as database

@o-live-r
Copy link

I get the same bug using ubuntu 22.04.03 LTS with mariadb and nginx

@Langelus
Copy link

Seing the same running Linuxservers Dockerimage with MariaDB and Redis

@M3taKn1ght
Copy link

Using a updated Debian 12 bookworm an updated nextcloud 28.0.1 with external mariadb.
Quiet confusing about this Message.

@szaimen
Copy link
Contributor

szaimen commented Jan 3, 2024

cc @artonge @ArtificialOwl

@artonge
Copy link
Contributor

artonge commented Jan 3, 2024

Could one of you try to pinpoint what's filling the metadata like that?
I suspect the EXIF provider, but it might be something else.

@SystemKeeper
Copy link
Contributor Author

Could one of you try to pinpoint what's filling the metadata like that?

Any idea on how we could figure that out? 😕

@ArtificialOwl
Copy link
Member

we will provide more logs and wait for more feedback. this exception is supposed to be here to avoid useless information (more than 100k chars) within metadata and should be high enough for 'normal' data :-)

@ArtificialOwl
Copy link
Member

Can you please patch your Nextcloud using this Pull Request: #42560 ?

the easiest way would be to run this command from the root of your Nextcloud:

wget https://github.com/nextcloud/server/pull/42560.diff
patch -p1 < 42560.diff

Switch your loglevel in config/config.php to 0:

'loglevel' => 0,

Then run this occ command:

./occ files:scan --generate-metadata --all

once this is done, verify your logs for any entry that starts with 'huge metadata content detected'

@SystemKeeper
Copy link
Contributor Author

Tried the patch and the steps, but the error did not occur in this case for me. Will keep an eye on the log.

@nicrame

This comment was marked as duplicate.

@artonge
Copy link
Contributor

artonge commented Jan 24, 2024

Same problem here on RHEL 9.3 with PHP 8.2 (Apache web server). Using Nextcloud 28.0.2 rc2.

Please apply the patch and post the detailed log, without that, we cannot make any progress.

@nicrame
Copy link

nicrame commented Jan 24, 2024

Same problem here on RHEL 9.3 with PHP 8.2 (Apache web server). Using Nextcloud 28.0.2 rc2.

Please apply the patch and post the detailed log, without that, we cannot make any progress.

I installed NC 28.0.2 RC3, applied patch, changed config.php, and started the "'./occ files:scan --generate-metadata --all". After 3 hours it's still running but i've got many photos there (more than 850GB of data, about 150.000 photos) so it must take time (even when it's fast SSD disk (KC3000), and there is still 45GB of free RAM, CPU is also fine, it's Ryzen 5 5600G).

Most of the results are thing like that:

Error during scan: exif_read_data(): Illegal IFD size: 2 + 0x534F*12 = 0x3E7B6 > 0x36B0
Error during scan: exif_read_data(): File not supported
Error during scan: exif_read_data(): Potentially invalid endianess, trying again with different endianness before imminent failure.
Error during scan: exif_read_data(): Illegal IFD size: x046C + 2 + x0500*12 = x406C > x04C3
Error during scan: exif_read_data(): Incorrect APP1 Exif Identifier Code
Error during scan: exif_read_data(): Illegal IFD Pointer

One had this message:

Error during scan: exif_read_data(): Process tag(x0000=UndefinedTag): Illegal format code 0x0000, suppose BYTE
Exception during scan: json cannot exceed 100000 characters long; fileId: 1505759; size: 533754
#0 /var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php(116): OC\FilesMetadata\FilesMetadataManager->saveMetadata()
#1 /var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php(122): OC\FilesMetadata\FilesMetadataManager->refreshMetadata()
#2 /var/www/nextcloud/apps/files/lib/Command/Scan.php(142): OC\FilesMetadata\FilesMetadataManager->refreshMetadata()
#3 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}()
#4 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array()
#5 /var/www/nextcloud/lib/private/Hooks/PublicEmitter.php(40): OC\Hooks\BasicEmitter->emit()
#6 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(133): OC\Hooks\PublicEmitter->emit()
#7 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}()
#8 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array()
#9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(173): OC\Hooks\BasicEmitter->emit()
#10 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(460): OC\Files\Cache\Scanner->scanFile()
#11 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(397): OC\Files\Cache\Scanner->handleChildren()
#12 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren()
#13 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren()
#14 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren()
#15 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren()
#16 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(347): OC\Files\Cache\Scanner->scanChildren()
#17 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(273): OC\Files\Cache\Scanner->scan()
#18 /var/www/nextcloud/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan()
#19 /var/www/nextcloud/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles()
#20 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()
#21 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#22 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#23 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#24 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#25 /var/www/nextcloud/lib/private/Console/Application.php(206): Symfony\Component\Console\Application->run()
#26 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#27 /var/www/nextcloud/occ(11): require_once('...')
#28 {main}

But it was all on terminal. I'll check nextcloud.log when it will finish working (and maybe even more of such things will appear).

Update: i've checked that nextcloud.log is already more than 14MB, it will be hard to find there anything because almost everything is logged inside :)

@artonge
Copy link
Contributor

artonge commented Jan 24, 2024

We are only interested in logs containing "huge metadata content detected".

@nicrame
Copy link

nicrame commented Jan 24, 2024

nextcloud.log
Here is the extracted the line with this error.

@artonge
Copy link
Contributor

artonge commented Jan 24, 2024

Thanks. Can you confirm that the file has been edited by Photoshop ?
There is a huge entry in the EXIF data. The solution might simply be to filter out such entries.

@artonge
Copy link
Contributor

artonge commented Jan 24, 2024

Can you test the following PR: nextcloud/photos#2280 ?

@nicrame
Copy link

nicrame commented Jan 24, 2024

Thanks. Can you confirm that the file has been edited by Photoshop ? There is a huge entry in the EXIF data. The solution might simply be to filter out such entries.

I got no idea. I can't find filename in the log, so i can't verify. This is some old file so it is possible Adobe was used, but i'm not sure. Currently i use Affinity software, and haven't been using Adobe for years.
Do You know how can i find what is the file that make problems?

@artonge
Copy link
Contributor

artonge commented Jan 24, 2024

Do You know how can i find what is the file that make problems?

Looks like I went too quickly over your comment. Here is the fileid of the problematic file: 1505759. You can find out more information with the following command: occ info:file 1505759

@nicrame
Copy link

nicrame commented Jan 24, 2024

Can you test the following PR: nextcloud/photos#2280 ?

I'll check that out, but i will prepare the file so it will be tested first.
Also the script crashed so it is impossible to know if more files are problematic. Here is the crash error log from terminal:
An unhandled exception has been thrown: TypeError: OCA\Photos\Listener\OriginalDateTimeMetadataProvider::dateToTimestamp(): Argument #2 ($date) must be of type string, null given, called in /var/www/nextcloud/apps/photos/lib/Listener/OriginalDateTimeMetadataProvider.php on line 98 and defined in /var/www/nextcloud/apps/photos/lib/Listener/OriginalDateTimeMetadataProvider.php:50 Stack trace: #0 /var/www/nextcloud/apps/photos/lib/Listener/OriginalDateTimeMetadataProvider.php(98): OCA\Photos\Listener\OriginalDateTimeMetadataProvider->dateToTimestamp() #1 /var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php(86): OCA\Photos\Listener\OriginalDateTimeMetadataProvider->handle() #2 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(230): OC\EventDispatcher\ServiceEventListener->__invoke() #3 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(59): Symfony\Component\EventDispatcher\EventDispatcher->callListeners() #4 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(94): Symfony\Component\EventDispatcher\EventDispatcher->dispatch() #5 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(106): OC\EventDispatcher\EventDispatcher->dispatch() #6 /var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php(115): OC\EventDispatcher\EventDispatcher->dispatchTyped() #7 /var/www/nextcloud/apps/files/lib/Command/Scan.php(142): OC\FilesMetadata\FilesMetadataManager->refreshMetadata() #8 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}() #9 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array() #10 /var/www/nextcloud/lib/private/Hooks/PublicEmitter.php(40): OC\Hooks\BasicEmitter->emit() #11 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(133): OC\Hooks\PublicEmitter->emit() #12 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}() #13 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array() #14 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(173): OC\Hooks\BasicEmitter->emit() #15 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(460): OC\Files\Cache\Scanner->scanFile() #16 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(397): OC\Files\Cache\Scanner->handleChildren() #17 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren() #18 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren() #19 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren() #20 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren() #21 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(347): OC\Files\Cache\Scanner->scanChildren() #22 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(273): OC\Files\Cache\Scanner->scan() #23 /var/www/nextcloud/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan() #24 /var/www/nextcloud/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles() #25 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute() #26 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run() #27 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run() #28 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand() #29 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun() #30 /var/www/nextcloud/lib/private/Console/Application.php(206): Symfony\Component\Console\Application->run() #31 /var/www/nextcloud/console.php(100): OC\Console\Application->run() #32 /var/www/nextcloud/occ(11): require_once('...')

I will try this PR now and let You know. Thank You for help.

@nicrame
Copy link

nicrame commented Jan 24, 2024

Can you test the following PR: nextcloud/photos#2280 ?

After changing PHP file i've got this error while scanning same file:
An unhandled exception has been thrown: TypeError: strlen(): Argument #1 ($string) must be of type string, int given in /var/www/nextcloud/apps/photos/lib/Listener/ExifMetadataProvider.php:173 Stack trace: #0 /var/www/nextcloud/apps/photos/lib/Listener/ExifMetadataProvider.php(173): strlen() #1 /var/www/nextcloud/apps/photos/lib/Listener/ExifMetadataProvider.php(91): OCA\Photos\Listener\ExifMetadataProvider->sanitizeEntries() #2 /var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php(86): OCA\Photos\Listener\ExifMetadataProvider->handle() #3 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(230): OC\EventDispatcher\ServiceEventListener->__invoke() #4 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(59): Symfony\Component\EventDispatcher\EventDispatcher->callListeners() #5 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(94): Symfony\Component\EventDispatcher\EventDispatcher->dispatch() #6 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(106): OC\EventDispatcher\EventDispatcher->dispatch() #7 /var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php(115): OC\EventDispatcher\EventDispatcher->dispatchTyped() #8 /var/www/nextcloud/apps/files/lib/Command/Scan.php(142): OC\FilesMetadata\FilesMetadataManager->refreshMetadata() #9 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}() #10 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array() #11 /var/www/nextcloud/lib/private/Hooks/PublicEmitter.php(40): OC\Hooks\BasicEmitter->emit() #12 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(133): OC\Hooks\PublicEmitter->emit() #13 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}() #14 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array() #15 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(173): OC\Hooks\BasicEmitter->emit() #16 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(460): OC\Files\Cache\Scanner->scanFile() #17 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(397): OC\Files\Cache\Scanner->handleChildren() #18 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren() #19 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(347): OC\Files\Cache\Scanner->scanChildren() #20 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(273): OC\Files\Cache\Scanner->scan() #21 /var/www/nextcloud/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan() #22 /var/www/nextcloud/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles() #23 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute() #24 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run() #25 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run() #26 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand() #27 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun() #28 /var/www/nextcloud/lib/private/Console/Application.php(206): Symfony\Component\Console\Application->run() #29 /var/www/nextcloud/console.php(100): OC\Console\Application->run() #30 /var/www/nextcloud/occ(11): require_once('...')

Maybe i edited something wrong, please check screenshot:
Zrzut ekranu (1187)

@artonge
Copy link
Contributor

artonge commented Jan 30, 2024

@nicrame indeed, it was an error in the code. If you revert the patch and apply its new version, it should work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 28-feedback bug feature: previews and thumbnails
Projects
None yet
Development

No branches or pull requests