OneDrive Client Version v2.5.0-rc1 is now available for testing

[abraunegg]

Summary of Changes

Important

OneDrive Client for Linux v2.5.0 contains a database schema change to support the additional functionality and bug fixes in the v2.5.x branch. As such, on your first use of this client version, the database schema will be updated and the client will perform an entire re-validation of your data - both locally and online which will take an extended period of time.

Caution

OneDrive Client for Linux v2.5.0 is not backwards compatible. Do not use v2.5.0 and v2.4.x applications at the same time against the same Microsoft OneDrive account. If using v2.5.0, you must upgrade all your clients to v2.5.0

Feature Requests

• Implement Feature Request: Multi-threaded uploading/downloading of files FR-232
• Implement Feature Request: Renaming/Relocation of OneDrive Business shared folders FR-1251
• Implement Feature Request: Support the syncing of individual business shared files FR-1300
• Implement Feature Request: Log when manual Authorization is required when using --auth-files FR-2475
• Implement Feature Request: Add cmdline parameter to display (human readable) quota status FR-2359
• Implement Feature Request: Ability to set --disable-download-validation from Docker environment variable FR-2559
• Implement Feature Request: Ability to set --sync-shared-files from Docker environment variable FR-2680

Bug Fixes

• Fix Bug: Business shared folders with same basename get merged BUG-2434
• Fix Bug: --dry-run prevents authorization BUG-2451
• Fix Bug: Log timestamps lacking trailing zeros, leading to poor log file output alignment BUG-2479
• Fix Bug: Subscription ID already exists when using webhooks BUG-2501
• Fix Bug: Not all files being downloaded when API data includes HTML ASCII Control Sequences BUG-2562
• Fix Bug: vim backups when editing files cause edited file to be deleted rather than the edited file being uploaded BUG-2586
• Fix Bug: skip_dir does not always work as intended for all directory entries BUG-2590
• Fix Bug: Online date being changed in download-only mode BUG-2591
• Fix Bug: Resolve that upload session are not canceled with resync option BUG-2624
• Fix Bug: Local files should be safely backed up when the item is not in sync locally to prevent data loss when they are deleted online BUG-2625

Other Changes | Fixes

• Implement buffered logging so that all logging from all upload & download activities are handled correctly
• Replace polling monitor loop with blocking wait
• Update how the application utilises curl to fix socket reuse
• Various performance enhancements with many thanks to @JC-comp
• Enforcement of operational conflicts
• Enforcement of application configuration defaults and minimums
• Implement application output to detail upload|download failures at the end of a sync process
• Utilise threadsafe sqlite DB access methods
• Various code cleanup and optimisations

Documentation

• Most documentation has been totally overhauled

Documentation List

• Advanced Usage
• Application Config Options
• Application Security
• Build RPM How-To
• Business Shared Items
• Client Architecture
• Docker Installation
• Install
• Known Issues
• National Cloud Deployments
• Podman Installation
• Privacy Policy
• SharePoint Libraries
• Terms of Service
• Ubuntu Package Install
• Usage

How can you get v2.5.0-rc1 ?

You need to build this client from source. Follow the high level directions here to get the development requirements installed first: https://github.com/abraunegg/onedrive/blob/master/docs/INSTALL.md#building-from-source---high-level-requirements

Next clone and checkout the PR:

git clone https://github.com/abraunegg/onedrive.git
cd onedrive
git fetch origin pull/2661/head:pr2661
git checkout pr2661
./configure --enable-debug; make clean; make;

To run the PR, you need to run the client from the PR build directory:

./onedrive <any options needed>

When running the PR, your version should be as per below or greater:

onedrive v2.5.0-rc1-2-g1535084

The important point is onedrive v2.5.0-rc1-2-g1535084 as this will change as new commits are put into the Release Candidate PR branch, whilst things are added/fixed/updated.

[bpozdena]

./onedrive --version                                                                               
onedrive v2.5.0-rc1-11-g235fbf7

./onedrive --list-shared-items
...
ERROR: onedrive application is already running - check system process list for active application instances


./onedrive --get-sharepoint-drive-id 'test'
...
ERROR: onedrive application is already running - check system process list for active application instances

@abraunegg , would it be possible to again allow running non-sync operations while the sync is running under a different process ID?

This is essentially the same request as #2106

[abraunegg]

Let me see how best to re-introduce this ...

[bpozdena]

./onedrive --version                                                                               
onedrive v2.5.0-rc1-11-g235fbf7

./onedrive  --get-sharepoint-drive-id                       

Missing value for argument --get-sharepoint-drive-id.
Try 'onedrive -h' for more information
Aborting from core/sync/mutex.d(149) Error: pthread_mutex_destroy failed.[1]    44576 IOT instruction (core dumped)  ./onedrive  --get-sharepoint-drive-id

Perhaps the Aborting from error message is a little bit of an overshare?

[abraunegg]

@bpozdena

Perhaps the Aborting from error message is a little bit of an overshare?

Thats essentially a unhandled exception bug ... and despite trying I have not been able to replicate this in a consistent manner (once - first try, not again in 50 attempts). If you can reproduce this consistently - can you provide debug log output (add -v -v) so that this can be traced a little further?

[bpozdena]

Just one note regarding documentation. The command the argument --get-O365-drive-id should probably be replaced for the new one --get-sharepoint-drive-id in https://github.com/abraunegg/onedrive/blob/onedrive-v2.5.0-release-candidate-1/docs/sharepoint-libraries.md .

[abraunegg]

@bpozdena
Thanks for catching that - this has been fixed

[bpozdena]

onedrive v2.5.0-rc1-11-g235fbf7 is still prone to crashing in --monitor mode when reconnecting to network.

Steps to reproduce:
1)Start onedrive --monitor
2)Disconnect from wifi during the sync process
3)Wait until the below messages start appearing in the console:

Retry attempt:          2
ERROR: There was a timeout in accessing the Microsoft OneDrive service - Internet connectivity issue?
This attempt timestamp: 2024-Mar-19 10:05:28

4)Re-connect to WiFi and the client will crash with the below error messages for each thread.

Waiting for all internal threads to complete before exiting application
std.json.JSONException@std/json.d(317): JSONValue is not an object
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x5d3a9de8be75]
??:? inout pure @property @trusted inout(std.json.JSONValue[immutable(char)[]]) std.json.JSONValue.objectNoRef() [0x766c806703ec]
/usr/include/dlang/ldc/std/json.d:796 [0x5d3a9dd80061]
src/sync.d:5033 [0x5d3a9dd90dce]
/usr/include/dlang/ldc/std/parallelism.d-mixin-4114:4156 [0x5d3a9de8ebe9]
??:? void std.parallelism.TaskPool.executeWorkLoop() [0x766c8064e3ed]
??:? thread_entryPoint [0x766c809c4c57]
??:? [0x766c801bd559]
??:? [0x766c8023aa3b]
std.json.JSONException@std/json.d(317): JSONValue is not an object
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x5d3a9de8be75]
??:? inout pure @property @trusted inout(std.json.JSONValue[immutable(char)[]]) std.json.JSONValue.objectNoRef() [0x766c806703ec]
/usr/include/dlang/ldc/std/json.d:796 [0x5d3a9dd80061]
src/sync.d:5033 [0x5d3a9dd90dce]
/usr/include/dlang/ldc/std/parallelism.d-mixin-4114:4156 [0x5d3a9de8ebe9]
??:? void std.parallelism.TaskPool.executeWorkLoop() [0x766c8064e3ed]
??:? thread_entryPoint [0x766c809c4c57]
??:? [0x766c801bd559]
??:? [0x766c8023aa3b]
std.json.JSONException@std/json.d(317): JSONValue is not an object
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x5d3a9de8be75]
??:? inout pure @property @trusted inout(std.json.JSONValue[immutable(char)[]]) std.json.JSONValue.objectNoRef() [0x766c806703ec]
/usr/include/dlang/ldc/std/json.d:796 [0x5d3a9dd80061]
src/sync.d:5033 [0x5d3a9dd90dce]
/usr/include/dlang/ldc/std/parallelism.d-mixin-4114:4156 [0x5d3a9de8ebe9]
??:? void std.parallelism.TaskPool.executeWorkLoop() [0x766c8064e3ed]
??:? thread_entryPoint [0x766c809c4c57]
??:? [0x766c801bd559]
??:? [0x766c8023aa3b]
std.json.JSONException@std/json.d(317): JSONValue is not an object
...

[abraunegg]

@bpozdena
Please could you add --enable-debug to your build process so that all the above thread resolution is possible?

Once done, could you post the updates crash outpur here please.

I will have to also see if this is wifi specific, as I do test network connection loss and I have not seen this thus far.

[bpozdena]

I did have --enable-debug option during the build process.
I have not been able to reproduce the same crash yet. I will let you know if if manage to find a reliable trigger.

[bpozdena]

I just came across another unhanded exception when I press CTRL+C after a sync cycle completes in --monitor mode.

./onedrive --version                                                                           
onedrive v2.5.0-rc1-12-ga6e2f77

 ./onedrive --confdir='/home/bob/.config/onedrive/accounts/bpozdena@company.com' --monitor
Reading configuration file: /home/bob/.config/onedrive/accounts/bpozdena@company.com/config
Configuration file successfully loaded
....
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
^C
Got termination signal, performing clean up
Waiting for any existing upload|download process to complete
Shutting down DB connection and merging temporary data
Aborting from core/sync/mutex.d(149) Error: pthread_mutex_destroy failed.[1]    56273 IOT instruction (core dumped)  ./onedrive  --monitor

[abraunegg]

So this will be the same item as your prior one ... something in the logging changes that @JC-comp did will be the cause of this - it will need digging into with gdb to work out why the threads are not syncing before exit.

[JC-comp]

Aborting from core/sync/mutex.d(149) Error: pthread_mutex_destroy failed.[1] 56273 IOT instruction (core dumped) ./onedrive --monitor

Unable to reproduce this in my environment, detailed operating system and library version information are needed to investigate the cause. While it is likely expected, as the outcome in the interrupt handler could be unpredictable.

4)Re-connect to WiFi and the client will crash with the below error messages for each thread.

This could be related to a known bug in #2601, and the fix for it has still not been implemented.

[abraunegg]

@bpozdena
Regarding this error:

Aborting from core/sync/mutex.d(149) Error: pthread_mutex_destroy failed.[1]    44576 IOT instruction (core dumped)

Potentially resolved with 46dfb05 - please rebuild your client so that you are running onedrive v2.5.0-rc1-13-g46dfb05 or greater.

Besides getting this to occur only once, I have not yet at all been able to get another instance. Given that you have reported this error now twice, I suspect in your environment this is more reproducible, so any update would be greatly appreciated.

[abraunegg]

@bpozdena

./onedrive --version                                                                               
onedrive v2.5.0-rc1-11-g235fbf7

./onedrive --list-shared-items
...
ERROR: onedrive application is already running - check system process list for active application instances


./onedrive --get-sharepoint-drive-id 'test'
...
ERROR: onedrive application is already running - check system process list for active application instances

@abraunegg , would it be possible to again allow running non-sync operations while the sync is running under a different process ID?

This is essentially the same request as #2106

This functionality regression should be now resolved via 68d6921

Please rebuild your client so that you are running onedrive v2.5.0-rc1-14-g68d6921 or greater.

[abraunegg]

@bpozdena

onedrive v2.5.0-rc1-11-g235fbf7 is still prone to crashing in --monitor mode when reconnecting to network.

Steps to reproduce: 1)Start onedrive --monitor 2)Disconnect from wifi during the sync process 3)Wait until the below messages start appearing in the console:

Retry attempt:          2
ERROR: There was a timeout in accessing the Microsoft OneDrive service - Internet connectivity issue?
This attempt timestamp: 2024-Mar-19 10:05:28

4)Re-connect to WiFi and the client will crash with the below error messages for each thread.

Waiting for all internal threads to complete before exiting application
std.json.JSONException@std/json.d(317): JSONValue is not an object
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x5d3a9de8be75]
??:? inout pure @property @trusted inout(std.json.JSONValue[immutable(char)[]]) std.json.JSONValue.objectNoRef() [0x766c806703ec]
/usr/include/dlang/ldc/std/json.d:796 [0x5d3a9dd80061]
src/sync.d:5033 [0x5d3a9dd90dce]
/usr/include/dlang/ldc/std/parallelism.d-mixin-4114:4156 [0x5d3a9de8ebe9]
??:? void std.parallelism.TaskPool.executeWorkLoop() [0x766c8064e3ed]
??:? thread_entryPoint [0x766c809c4c57]
??:? [0x766c801bd559]
??:? [0x766c8023aa3b]
std.json.JSONException@std/json.d(317): JSONValue is not an object
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x5d3a9de8be75]
??:? inout pure @property @trusted inout(std.json.JSONValue[immutable(char)[]]) std.json.JSONValue.objectNoRef() [0x766c806703ec]
/usr/include/dlang/ldc/std/json.d:796 [0x5d3a9dd80061]
src/sync.d:5033 [0x5d3a9dd90dce]
/usr/include/dlang/ldc/std/parallelism.d-mixin-4114:4156 [0x5d3a9de8ebe9]
??:? void std.parallelism.TaskPool.executeWorkLoop() [0x766c8064e3ed]
??:? thread_entryPoint [0x766c809c4c57]
??:? [0x766c801bd559]
??:? [0x766c8023aa3b]
std.json.JSONException@std/json.d(317): JSONValue is not an object
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(std.json.JSONException).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x5d3a9de8be75]
??:? inout pure @property @trusted inout(std.json.JSONValue[immutable(char)[]]) std.json.JSONValue.objectNoRef() [0x766c806703ec]
/usr/include/dlang/ldc/std/json.d:796 [0x5d3a9dd80061]
src/sync.d:5033 [0x5d3a9dd90dce]
/usr/include/dlang/ldc/std/parallelism.d-mixin-4114:4156 [0x5d3a9de8ebe9]
??:? void std.parallelism.TaskPool.executeWorkLoop() [0x766c8064e3ed]
??:? thread_entryPoint [0x766c809c4c57]
??:? [0x766c801bd559]
??:? [0x766c8023aa3b]
std.json.JSONException@std/json.d(317): JSONValue is not an object
...

Could you provide some host operating system details here ?

I cant get the same scenario to occur using a spare laptop using those repoduction steps.

I have tested so far Ubuntu 23.x and Fedora 39 ... both do something different (and different to each other even) but no crash when wifi is re-connected .. the different handling also needs to be looked at.

I am also wondering if this crash is related to your environment (I recall you said you are doing something different with DNS) so any environment configuration you can advise would be greatly appreciated.

[bpozdena]

I'm away from PC for several days, and I was not able to reproduce it since my last report. But if you really need to reproduce it, I use stock Arch Linux with KDE Plasma. Arch has DNS caching disabled by default.

[abraunegg]

No problem. Once I fix the issues I found with Ubuntu | Fedora , will test again on Arch