Migrate existing local Nextcloud folder to NC3.2.0 VirtualClient is not working properly

[wagnbeu0]

How to use GitHub

Expected behaviour

After switching from selected folder to virtual file system I get a list of all files and can decide which folder should be available locally like in OneDrive

Actual behaviour

After the Installation of the NC3.2.0 Client I turned on the feature “Support Virtual files”. It took some time, then NC told me that all is synced.
But:
1.) All my Folder s are shown the “sync” Symbol instead if the clout symbol or the green OK symbol. Even if I click on “Keep files always local”, it does not dissapear after a successful sync.

2.) Now in the NC Client I cannot select specific folders for syncronization anymore. It should therefore show all Folder in the OS with the posibilty to keep them local or only in the cloud, but my Explorer only shows me the already existing local files. Not all folders in my NC account on the server are shown locally, I therefore cannot even say “Keep them locally available” because they are not listed.

So it seems that I need to recreate the local NC folder. It this behaviour known for migrated local NC folders?

Steps to reproduce

1. Enable in existing nextcloud installation the virtual file system

Client configuration

Client version:
3.2.0

Operating system: Windows 10 Pro x64

OS language: German

Qt version used by client package (Linux only, see also Settings dialog):

Client package (From Nextcloud or distro) (Linux only):

Installation path of client:
C:\Program Files (x86)\Nextcloud

Server configuration

Nextcloud version: 21.0.1

Storage backend (external storage):
NFS Mount on QNAP

1. Client logfile:
   n/a

2. Web server error log:
   n/a

3. Server logfile: nextcloud log (data/nextcloud.log):

4. n/a

[johkoenig]

Similar issue: #3082

[wagnbeu0]

I have now checked this issue with lots of installations. It seems to be related to huge folders.
When I setup a new user, then vfs or the traditional file sync is working fine.
When sharing to this new user an already existing folder with 280 GB and 60.000 files the client will never finish the sync, independent with or without VFS - it will aleways show this error:

The error is always on a different folder.
Attached the log files I created.
error.zip

[johkoenig]

Hello everybody,
its working a little bit better with version 3.2.1, but still not perfect.

In my case, I had 3.1.3 and earlier versions running with one folder in sync. Then I updated to 3.2.0, converted to vfs and converted back to normal (due to problems). Then I updated to 3.2.1, converted again to vfs, and thats where I am right now.
In the nextcloud settings, this is what I see (permanently):

(Text: Syncing VirtualFiles with local folder)

In my explorer, this is what I see:

These lots of markings with "in sync" are there for quite some time now (several days). Folders that existed before conversion mostly show up as "in sync". One folder, that existed before the conversion is empty. This one shows up as "cloud". Folders that I created after conversion are working perfectly fine. So, the problem is limited to those which were converted.

Best regards,
jk

[wagnbeu0]

Not working for me with NC3.2.1
The process of syncing is starting, and then at my FOTOS folder ( with 280 GB and 60.000 files) or any other huge folder still starting again from scratch.
I also setup a VM and installed a new 3.2.1 installations - same behaviour. For me it seems that the new VFS Feature can only work with small folders. Very disapointing

[johkoenig]

Problem still exists on 3.2.2

[wagnbeu0]

Had the same problem yesterday with a 21.0.1 installation and a new 3.2.2 client - in my homefolder is a photo folder with total 60.000 files in 20 subfolders (photos sorted by year) which cannot be initially syncronized.

The Workaround was to select only maybe 3 or 4 subfolders to sync, do the local sync, when done add the next folders. When selecting the mainfolder with all files and subfolders the client will abort. So it could be a problem with an internal buffer overrun or whatever on the client - the amount of files seems to be too huge - no matter if I use VFS or not.
But especially with VFS you cannot sync this photo folder because it will never be ready syncing the client with server and with VFS you cannot choose what to sync.

I also tried to raise the variable innodb_buffer_pool_size from
innodb_buffer_pool_size=1G
to
innodb_buffer_pool_size=5G

but it did also not fix it, so it could only be a client related problem.

[wagnbeu0]

Because it seems that no one knows the problem or knows how to test it - here is a windows cmd script which you can use to create huge folders easily. Just create a folder D:\TEMP, and place a small file named bild.jpg in it. Then you specify how many folder below D:\TEMP should be created, and how many subfoldes each should have. Last you specify how many copyied of the pictures every subfolder should have. Now you can create within a few minutes a folder with 100.000 items and lots of folders which can be used for testing.

@echo off
set Mainfolder=20
set subfolder=50
set count_files=100
set template_file=bild.jpg
set root_folder=D:\TEMP

REM Create Mainfolder
FOR /L %%i IN (1,1,%MAINFOLDER%) DO (
  cd /d %root_folder%
  mkdir Mainfolder_%%i
  cd Mainfolder_%%i

REM Create Subfolder in Current Mainfolder
	FOR /L %%k IN (1,1,%subfolder%) DO (
	mkdir Subfolder_%%k
	cd /d Subfolder_%%k
	
		REM Create Files in Current Subfolder
		FOR /L %%l IN (1,1,%count_files%) DO (
		copy /Y %root_folder%\%template_file% .\%%l_%template_file% >NUL
		)
	echo Creating Files in Mainfolder_%%i\Subfolder_%%k done		
	cd /d %root_folder%\Mainfolder_%%i	
	)  
cd /d %ROOT_folder%
)

It should then be possible to upload the files to nextcloud, and use a fresh installed VM with a 3.2.2 client to see that the client will not be able to sync huge folders then.
With that script I am able to reproduce several errors during upload and download.

[mgallien]

@wagnbeu0 this has nothing to do with the size of the sync folder but all with switching between classic sync and virtual files. I have a work in progress branch to fix that. So let's keep working on it.

[joshtrichards]

Sounds like #3082 / #3642 fixed in #3612

Closing this has been quiet for >3 years. If you're still experiencing this with newer client versions let me know and I'll re-open.