Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

files:scan" the process stops scanning Files after a Thumbs.db file is hit #33215

Closed
4 of 8 tasks
IgnaceSannen opened this issue Jul 12, 2022 · 8 comments
Closed
4 of 8 tasks
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug needs info

Comments

@IgnaceSannen
Copy link

IgnaceSannen commented Jul 12, 2022

⚠️ This issue respects the following points: ⚠️

  • This is a bug, not a question or a configuration/webserver/proxy issue.
  • This issue is not already reported on Github (I've searched it).
  • Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
  • I agree to follow Nextcloud's Code of Conduct.

Bug description

Nextcloud version: 24.0.2
OS: Ubuntu 22.04 running in LXC container
From the system info:
Operating System: Linux 5.13.0-51-generic x86_64
Fresh install.
This issue is happening on external storage, mounted with SMB.
When running following command "sudo -u www-data /usr/bin/php /srv/nextcloud/occ -vvv files:scan" the process stops
scanning files when a Thumbs.db file is hit.
So the output of the above command gives something like this:
Folder /user/files/mountpoint/FOLDER1
File /user/files/mountpoint/FOLDER/FILE1
File /user/files/mountpoint/FOLDER/FILE2
File /user/files/mountpoint/FOLDER/Thumbs.db
Folder /user/files/mountpoint/FOLDER2
Folder /user/files/mountpoint/FOLDER3
Folder /user/files/mountpoint/FOLDER4

The outcome of this behavior is that in nextcloud FOLDER2, 3 and 4 are empty.
I deleted Thumbs.db with rm and the issue was solved. Since I just wiped Thumbs.db, I did not know what was in the file.
But when I create a file with the same name and just 'test' as content the same issue appears.

Steps to reproduce

  1. create file Thumbs.db in a folder.

Expected behavior

Scan should run without issue, even with thumbs.db file in it.

Installation method

All components installed (no snap install)

Operating system

22.04 in LXC

PHP engine version

8.1.2

Web server

No response

Database engine version

Type: mysql
Version: 10.6.7

Is this bug present after an update or on a fresh install?

No, it is a fresh install

Are you using the Nextcloud Server Encryption module?

No

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "myprivateip",
            "myFQDN"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "24.0.2.1",
        "overwrite.cli.url": "http:\/\/myprivateip\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "overwritehost": "myFQDN",
        "overwritewebroot": "\/nextcloud",
        "overwritecondaddr": "myprivate_ip",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - bruteforcesettings: 2.4.0
  - circles: 24.0.0
  - cloud_federation_api: 1.7.0
  - comments: 1.14.0
  - contactsinteraction: 1.5.0
  - dashboard: 7.4.0
  - dav: 1.22.0
  - external: 4.0.0
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_external: 1.16.1
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - firstrunwizard: 2.13.0
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - nextcloud_announcements: 1.13.0
  - notifications: 2.12.0
  - oauth2: 1.12.0
  - password_policy: 1.14.0
  - passwords: 2022.6.20
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - recommendations: 1.3.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - support: 1.7.0
  - survey_client: 1.12.0
  - systemtags: 1.14.0
  - tasks: 0.14.4
  - text: 3.5.1
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - unsplash: 1.2.5
  - updatenotification: 1.14.0
  - user_status: 1.4.0
  - viewer: 1.8.0
  - weather_status: 1.4.0
  - workflowengine: 2.6.0
Disabled:
  - admin_audit
  - encryption
  - photos: 1.6.0
  - user_ldap

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

@IgnaceSannen IgnaceSannen added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Jul 12, 2022
@ruppo68
Copy link

ruppo68 commented Jul 15, 2022

Same problem here.
We mount a smb share. In the Extern storage Config is Show hidden files not selected.
There are Thumbs.db in the Filesystem, with the Explorer they are not shown, because they have the Attribute HS.
But the file scan in the Background inventories the file !!!!!!
These causes a download of a folder with a file Thumbs.db in it as a zip file generates a corrupt zip file.
Nextcloud Release is the same.

Regards,
Peter Rupp

@IgnaceSannen
Copy link
Author

Another odd behavior, after deletion of the Thumbs.db file, the scan process runs without issues.
But since we are dealing with external storage, which is also accessible through SMB, I scheduled a scan job every night.
I see now that exact the same folders are empty again and the Thumbs.db file has reappeared.

@mobilandi
Copy link

Hi all,

I'm not sure if my problem is related to this one but I'll add the info just in case:

  • I'm currently on nextcloud 24.0.4 but the problem exists for longer.
  • When trying to access a directory on external storage (mounted via SFTP) I often get an error pop-up "Operation not allowed"
  • When this happens I open the directory on the storage and delete the "thumbs.db" in it.
  • After removing the thumbs.db I can access the directory via nextcloud without any issues.
  • A new thumbs.db gets created after time, but accessing the directory most of the time is still possible. Sometimes it happens that I have to repeat the process for the same directory after some time.

@sierra2256
Copy link

sierra2256 commented Sep 20, 2022

Same issue here. files:scan crashes when it reaches Thumbs.db

Exception during scan: Invalid path: files/files/xxxxxxxxxx/xxxxxxxx/xxxxxxx/xxxxxx/Thumbs.db
#0 /var/www/nextcloud/lib/private/Files/Storage/Local.php(253): OC\Files\Storage\Local->getSourcePath()
#1 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(227): OC\Files\Storage\Local->file_exists()
#2 /var/www/nextcloud/lib/private/Files/View.php(1345): OC\Files\Storage\Wrapper\Wrapper->file_exists()
#3 /var/www/nextcloud/lib/private/Files/View.php(1390): OC\Files\View->getCacheEntry()
#4 /var/www/nextcloud/lib/private/Metadata/FileEventListener.php(83): OC\Files\View->getFileInfo()
#5 /var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php(87): OC\Metadata\FileEventListener->handle()
#6 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(251): OC\EventDispatcher\ServiceEventListener->__invoke()
#7 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(73): Symfony\Component\EventDispatcher\EventDispatcher->callListeners()
#8 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(88): Symfony\Component\EventDispatcher\EventDispatcher->dispatch()
#9 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(100): OC\EventDispatcher\EventDispatcher->dispatch()
#10 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(235): OC\EventDispatcher\EventDispatcher->dispatchTyped()
#11 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils{closure}()
#12 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array()
...

Deleting Thumbs.db from file system does not help and issue persists.

Nextcloud Version: 24.0.5.1

@sierra2256
Copy link

I found the issue - it relates to my local installation.

I had added "Thumbs.db" to blacklisted_files property in config.php. Now I removed that from blacklist and it works fine again.

@luxifr
Copy link

luxifr commented Dec 31, 2022

Soemthing similar here... it seems the scanning generally stops when it hits a file that causes any kind of exception

Exception during scan: getimagesizefromstring(): Corrupt JPEG data: 14 extraneous bytes before marker
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler(2, 'getimagesizefro...', '/var/www/html/l...', 71)
#1 /var/www/html/lib/private/Metadata/Provider/ExifProvider.php(71): getimagesizefromstring('\xFF\xD8\xFF\xE1\x02\x12Exif\x00\x00II*...')
#2 /var/www/html/lib/private/Metadata/MetadataManager.php(68): OC\Metadata\Provider\ExifProvider->execute(Object(OC\Files\Node\File))
#3 /var/www/html/apps/files/lib/Command/Scan.php(138): OC\Metadata\MetadataManager->generateMetadata(Object(OC\Files\Node\File), false)
#4 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}('/username/files/Do...')
#5 /var/www/html/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array(Object(Closure), Array)
#6 /var/www/html/lib/private/Hooks/PublicEmitter.php(40): OC\Hooks\BasicEmitter->emit('\\OC\\Files\\Utils...', 'scanFile', Array)
#7 /var/www/html/lib/private/Files/Utils/Scanner.php(131): OC\Hooks\PublicEmitter->emit('\\OC\\Files\\Utils...', 'scanFile', Array)
#8 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}('files/Dokumente...', 'home::username')
#9 /var/www/html/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array(Object(Closure), Array)
#10 /var/www/html/lib/private/Files/Cache/Scanner.php(170): OC\Hooks\BasicEmitter->emit('\\OC\\Files\\Cache...', 'scanFile', Array)
#11 /var/www/html/lib/private/Files/Cache/Scanner.php(445): OC\Files\Cache\Scanner->scanFile('files/Dokumente...', 3, 323042, Object(OC\Files\Cache\CacheEntry), true, Array)
#12 /var/www/html/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->handleChildren('files/Dokumente...', true, 3, 323042, true, 0)
#13 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 323042, true)
#14 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 323037, true)
#15 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 323034, true)
#16 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente...', true, 3, 322890, true)
#17 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files/Dokumente', true, 3, 320870, true)
#18 /var/www/html/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('files', true, 3, 92738, true)
#19 /var/www/html/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren('', true, 3, 92734, true, Array)
#20 /var/www/html/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan('', true, 3)
#21 /var/www/html/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan('/username', true, NULL)
#22 /var/www/html/apps/files/lib/Command/Scan.php(217): OCA\Files\Command\Scan->scanFiles('username', '/username', true, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#23 /var/www/html/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /var/www/html/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /var/www/html/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /var/www/html/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /var/www/html/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#28 /var/www/html/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#29 /var/www/html/console.php(100): OC\Console\Application->run()
#30 /var/www/html/occ(11): require_once('/var/www/html/c...')
#31 {main}

This is a bit of a pain when trying to mass-load data into nextcloud externally and then trying to scan them into the index

@szaimen
Copy link
Contributor

szaimen commented May 22, 2023

Hi, please update to 25.0.7 or better 26.0.2 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 26-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

@joshtrichards
Copy link
Member

Hi @IgnaceSannen Thanks for the report.

What is your SMB server? Is it Samba by chance? Check if you have any veto files lines in your smb.conf. Sometimes Thumbs.db gets added there.

@mobilandi While your report involves Thumbs.db I don't think it's being caused by the same situation. Please open a separate Issue for your situation and provide your details including logs.

@ruppo68 Hidden files have different definitions across operating systems. The System/Hidden attributes under Windows are not the same as what Unix environments consider hidden (which are files that start with dots - e.g. .mysecretfile). There are different ways of dealing with your situation. I suggest posting/searching at the Nextcloud Help Forum. Not clear about the zip issue, but that sounds interesting enough I'd suggest opening an Issue about it since it's outside the scope of this Issue.

@sierra2256 That makes sense. The telltale sign of hidden a blacklisted file is that Invalid path: line. Glad you got it worked out.

@luxifr Unrelated to this particular Issue.

@szaimen szaimen closed this as completed Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug needs info
Projects
None yet
Development

No branches or pull requests

7 participants