Writing in a directory shared with an AD group is very slow

[vatanparvar]

Steps to reproduce

1. Setup Nextcloud with LDAP authentication.
2. Make a folder with admin account.
3. Share it with other groups (fetched from AD)
4. Try to upload or make a folder in the shared directory or delete from it.

Expected behaviour

It should upload with a reasonable (depends on bandwidth) or create a folder immediately.

Actual behaviour

Takes at least 10 - 15 minutes to only create a folder.

Server configuration

Operating system:
CentOS 7 (3.10.0-327.36.3.el7.x86_64)

Web server:
Apache/2.4.6 (CentOS)

Database:
Type: mysql
Version: 5.6.34
Size: 1.1 GB

PHP version:
PHP 5.5.21

Nextcloud version:
Nextcloud 10.0.1 (stable)

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install

Where did you install Nextcloud from:
https://download.nextcloud.com/server/releases/nextcloud-10.0.1.zip

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled: - activity: 2.3.2 - admin_audit: 1.0.0 - comments: 1.0.0 - dav: 1.0.1 - documents: 0.13.1 - federatedfilesharing: 1.0.1 - federation: 1.0.1 - files: 1.5.2 - files_external: 1.0.2 - files_pdfviewer: 0.8.1 - files_sharing: 1.0.0 - files_texteditor: 2.1 - files_trashbin: 1.0.0 - files_versions: 1.3.0 - files_videoplayer: 0.9.8 - firstrunwizard: 1.1 - gallery: 15.0.0 - notifications: 0.3.0 - password_policy: 1.0.0 - provisioning_api: 1.0.0 - serverinfo: 1.1.1 - survey_client: 0.1.5 - systemtags: 1.0.2 - theming: 1.0.1 - updatenotification: 1.0.1 - user_ldap: 1.0.1 - workflowengine: 1.0.1 Disabled: - encryption - external - files_accesscontrol - files_automatedtagging - files_retention - templateeditor - user_external - user_saml

The content of config/config.php:

Config report

{ "system": { "instanceid": "ocunawr2qdmw", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "192.168.230.51", "192.168.230.48" ], "datadirectory": "\/mountpoints\/filesrv\/clouddata\/cloud\/data\/", "overwrite.cli.url": "http:\/\/192.168.230.48", "dbtype": "mysql", "version": "9.1.1.5", "dbname": "cloud_db", "dbhost": "localhost", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "logtimezone": "UTC", "installed": true, "preview_libreoffice_path": "\/bin\/libreoffice", "preview_office_cl_parameters": " --headless --nologo --nofirststartwizard --invisible --norestore -convert-to pdf -outdir ", "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "localhost", "port": 6379 }, "knowledgebaseenabled": false, "allow_user_to_change_display_name": false, "activity_expire_days": 365, "has_internet_connection": false, "log_type": "owncloud", "logfile": "nextcloud.log", "loglevel": "2", "logdateformat": "F d, Y H:i:s", "ldapIgnoreNamingRules": false, "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory" } }

Are you using external storage, if yes which one: No

Are you using encryption: No

Are you using an external user-backend, if yes which one: ActiveDirectory

LDAP configuration (delete this part if not used)

LDAP config

| Configuration | | hasMemberOfFilterSupport | 1 | hasPagedResultSupport | | homeFolderNamingRule | | lastJpegPhotoLookup | 0 | ldapAgentName | CN=XXXXXXX,CN=XXXXXXX,DC=XXXXXXX,DC=com | ldapAgentPassword | *** | ldapAttributesForGroupSearch | | ldapAttributesForUserSearch | displayName;mail;sAMAccountName | ldapBackupHost | | ldapBackupPort | | ldapBase | DC=XXXXXXX,DC=com | ldapBaseGroups | OU=File Server,DC=XXXXXXX,DC=com | ldapBaseUsers | DC=XXXXXXX,DC=com | ldapCacheTTL | 1800 | ldapConfigurationActive | 1 | ldapDynamicGroupMemberURL | | ldapEmailAttribute | mail | ldapExperiencedAdmin | 0 | ldapExpertUUIDGroupAttr | | ldapExpertUUIDUserAttr | | ldapExpertUsernameAttr | sAMAccountName | ldapGroupDisplayName | cn | ldapGroupFilter | (&(|(objectclass=group))(|(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)(cn=XXXXXXX)))| | ldapGroupFilterGroups | XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX | ldapGroupFilterMode | 0 | ldapGroupFilterObjectclass | group | ldapGroupMemberAssocAttr | member | ldapHost | 192.168.231.5 | ldapIgnoreNamingRules | | ldapLoginFilter | (&(&(|(objectclass=person))(|(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4053))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4020))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4021))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4022))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4023))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4024))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4025))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4026))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4027))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4028))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4029))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4030))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4031))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4032))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4033))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4034))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4035))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4036))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4038))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4039))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4040))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4041))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4016))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4052))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4050))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4049))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4047))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4048))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4015))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4042))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4051))))(|(samaccountname=%uid)(|(mailPrimaryAddress=%uid)(mail=%uid))(|(sAMAccountName=%uid)))) | | ldapLoginFilterAttributes | sAMAccountName | ldapLoginFilterEmail | 1 | ldapLoginFilterMode | 0 | ldapLoginFilterUsername | 1 | ldapNestedGroups | 1 | ldapOverrideMainServer| | ldapPagingSize | 1000 | ldapPort | 389 | ldapQuotaAttribute | | ldapQuotaDefault | 20 GB | ldapTLS | 0 | ldapUserDisplayName | displayname | ldapUserDisplayName2 | | ldapUserFilter | (&(|(objectclass=person))(|(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4053))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4020))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4021))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4022))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4023))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4024))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4025))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4026))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4027))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4028))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4029))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4030))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4031))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4032))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4033))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4034))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4035))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4036))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4038))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4039))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4040))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4041))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4016))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4052))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4050))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4049))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4047))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4048))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4015))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4042))(|(memberof=CN=XXXXXXX,OU=File Server,DC=XXXXXXX,DC=com)(primaryGroupID=4051))))| | ldapUserFilterGroups | XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX;XXXXXXX | ldapUserFilterMode | 0 | ldapUserFilterObjectclass | person | ldapUuidGroupAttribute | auto | ldapUuidUserAttribute | auto | turnOffCertCheck | 0 | useMemberOfToDetectMembership | 1

Client configuration

Browser:
Firefox,Chrome,IE
Operating system:
Tested on Linux Arch, Ubuntu 16.04, Windows 7 and 10

Logs

Web server error log

Web server error log

NONE

Nextcloud log (data/nextcloud.log)

Nextcloud log

{"reqId":"WD01uAb7wxOASVkbgauqMwAAAA8","remoteAddr":"192.168.230.248","app":"PHP","message":"ldap_search(): Partial search results returned: Sizelimit exceeded at \/var\/www\/html\/cloud\/apps\/user_ldap\/lib\/LDAP.php#256","level":3,"time":"November 29, 2016 08:00:56","method":"POST","url":"\/index.php\/apps\/user_ldap\/ajax\/wizard.php","user":"cloudadmin"} {"reqId":"WD1OVAGmz3ah2cyIGp9v0QAAAAI","remoteAddr":"192.168.230.125","app":"PHP","message":"rename(): The first argument to copy() function cannot be a directory at \/var\/www\/html\/cloud\/lib\/private\/Files\/Storage\/Local.php#262","level":3,"time":"November 29, 2016 09:46:41","method":"DELETE","url":"\/remote.php\/webdav\/Projects\/R454%20-%20Fewa\/Corresspondence%20to%20Fewa\/E-19-12%20(3)-16-6143%20PTOC%20Transmittal","user":"Mike"} {"reqId":"WD1OVAGmz3ah2cyIGp9v0QAAAAI","remoteAddr":"192.168.230.125","app":"PHP","message":"rename(\/mountpoints\/filesrv\/clouddata\/cloud\/data\/\/cloudadmin\/files\/Shared Directories\/Dubai\/Projects\/R454 - Fewa\/Corresspondence to Fewa\/E-19-12 (3)-16-6143 PTOC Transmittal,\/mountpoints\/filesrv\/clouddata\/cloud\/data\/\/cloudadmin\/files_trashbin\/files\/E-19-12 (3)-16-6143 PTOC Transmittal.d1480412801): Invalid cross-device link at \/var\/www\/html\/cloud\/lib\/private\/Files\/Storage\/Local.php#262","level":3,"time":"November 29, 2016 09:46:41","method":"DELETE","url":"\/remote.php\/webdav\/Projects\/R454%20-%20Fewa\/Corresspondence%20to%20Fewa\/E-19-12%20(3)-16-6143%20PTOC%20Transmittal","user":"Mike"}

[vatanparvar]

UPDATE: This problem disappeared by disabling Activity app (Version 2.3.2). As soon as this app is disabled, everything worked smoothly.

[putt1ck]

For interest (& reason I subscribed) do you find the users list more responsive after this workaround?

[vatanparvar]

@putt1ck Yes it's much more responsive.
I initially thought this is a LDAP related problem and for that I written a set of scripts to fetch all the users and groups from Active Directory and create them locally in Nextcloud. However, after loading all 480 users, as soon as I shared a folder with a group with more than 100 users in it, the problem (unresponsiveness) started and I was back to square one.
To find the problem I put
slow-query-log = 'ON'
slow-query-log-file = /var/log/mysqld/mysql-slow.log
long_query_time=1
in MySQL config and started monitoring slow queries. Apart from SELECTs on 'oc_filecache' which indicates user is searching for a file, there were also huge amount of queries on 'oc_activity' tables. This made me think of disabling Activity app and to my surprise it worked.
Now I went back to my original setup with LDAP authentication and still everything is OK.

[putt1ck]

I tried disabling Activity but (Samba) LDAP user list still very slow, and search only works if you've forcibly loaded the userlist (wait for first screen, page down until loading symbol appears, wait, page down etc. until no loading symbol)

[vatanparvar]

@putt1ck I have experienced the same behaviour with users list. Even when all the users were created locally in Nextcloud, users list load them in chunks, meaning you had to still go through the same 'wait, page down' procedure. And to be honest, with 480 users, it wasn't that much faster than what it is now when using LDAP.
However the main objective for me was write speed in shared directories which is resolved after disabling Activity app.

[putt1ck]

K, thx for info - suggests I should post bug report on user management

[MorrisJobke]

Could you try with a more recent version - 12.0.3 for example. We improved there the fetching of LDAP user lists. And also make sure to have a distributed cache (Redis - see config.sample.php for details how to configure) in place.