onedrive client version v2.5.0-alpha-3

[abraunegg]

• Development tree for onedrive client version v2.5.0-alpha-3

[abraunegg]

@Lyncredible

Please can you test this 'alpha-3' version?

This is an uplift of your original code (as per v2.4.25) and does not include any of your changes as per #2516

If you can confirm that this current uplift works, could you then please uplift this PR with your changes from #2516 ?

This also includes your changes from #2516 - the code compiles, but I am unable to validate and check.

If you could so some sanity checking that would be greatly appreciated.

[Lyncredible]

@abraunegg I had to resync but it seems to be working. I will report back after the resync is done.

See in-progress logs

Base Args: --monitor
# Launching onedrive
Reading configuration file: /onedrive/conf/config
Configuration file successfully loaded

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

The item database is incompatible, re-creating database table structures
Deleting the saved application sync status ...
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 86400
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Initializing subscription for updates ...
Webhook: handled validation request
Created new subscription <REDACTED> with expiration: <REDACTED>
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: <REDACTED> ..................

[Lyncredible]

@abraunegg Resync is done. Webhook is working, but it seems something is wrong with the resync detection logic. It asks me to do a resync again when I restarted it without --resync. I can debug further tomorrow.

Log for successful resync

Base Args: --monitor
# Launching onedrive
Reading configuration file: /onedrive/conf/config
Configuration file successfully loaded

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

The item database is incompatible, re-creating database table structures
Deleting the saved application sync status ...
Configuring Global Azure AD Endpoints
Sync Engine Initialised with new Onedrive API instance
OneDrive synchronisation interval (seconds): 86400
Initialising filesystem inotify monitoring ...
Performing initial syncronisation to ensure consistent local state ...
Initializing subscription for updates ...
Webhook: handled validation request
Created new subscription <REDACTED> with expiration: <REDACTED>
Starting a sync with Microsoft OneDrive
Fetching items from the OneDrive API for Drive ID: <REDACTED> ..........................................................................................................................................................................................................................................................
Processing <REDACTED> applicable changes and items received from Microsoft OneDrive ...........................................................................................................
Performing a database consistency and integrity check on locally stored data ...
Scanning the local file system '/onedrive/data' for new data to upload ...
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: <REDACTED> ..
No additional changes or items that can be applied were discovered while processing the data received from Microsoft OneDrive
Sync with Microsoft OneDrive is complete
Initializing subscription for updates ...
Webhook: handled validation request
Created new subscription <REDACTED> with expiration: <REDACTED>
^CGot termination signal, performing clean up
Shutting down DB connection and merging temporary data

Log for a subsequent run without `--resync`

usermod: no changes
Base Args: --monitor
# Launching onedrive
Reading configuration file: /onedrive/conf/config
Configuration file successfully loaded

An application configuration change has been detected where a --resync is required

[abraunegg]

@Lyncredible

Log for a subsequent run without --resync

Can you run the application in debug --verbose --verbose mode ??

Need to understand what is triggering the --resync here post reading in the configuration file .. something is changing ... need to understand what it is and why the application thinks it has changed.

Also all this logic is 100% different to v2.4.x series - so if you ran v2.4.x inbetween ... could be an indicator as to where the issue is occurring from.

[Lyncredible]

@abraunegg
Resync was deemed necessary because it detected sync_dir override from CLI:

# The log statement was changed to also log the two different values
[DEBUG] sync_dir: CLI override of config file option, --resync needed, config = ~/OneDrive, CLI = /onedrive/data

The configured value is hard-coded since I never specified sync_dir in my own config file:

onedrive/src/config.d

Line 29 in 77c294b

immutable string defaultSyncDir = "~/OneDrive";

The CLI override comes from the entrypoint.sh in the default docker image:

onedrive/contrib/docker/entrypoint.sh

Line 27 in 77c294b

ARGS=(--monitor --confdir /onedrive/conf --syncdir /onedrive/data)

I now have specified sync_dir in my own config file to match the CLI override as a workaround, but this requires another resync since the config file has changed...

[Lyncredible]

@abraunegg
I hit a new issue where access token renewal fails:

AADSTS9000410: Malformed JSON.
ERROR: You will need to issue a --reauth and re-authorise this client to obtain a fresh auth token.

This is because the request to renew token is sent with the Content-Type: application/json header, but the token endpoint requires Content-Type: application/x-www-form-urlencoded. The wrong header is sent because http.clearRequestHeaders() is no longer called in scope(exit) {} in OneDriveApi.post() (or other methods like OneDriveApi.patch().

For example, I see http.clearRequestHeaders() being called in master branch:

onedrive/src/onedrive.d

Lines 1392 to 1401 in 4a60654

	private auto post(T)(const(char)[] url, const(T)[] postData)
	{
	scope(exit) http.clearRequestHeaders();
	http.method = HTTP.Method.post;
	http.url = url;
	addAccessTokenHeader();
	auto response = perform(postData);
	checkHttpCode(response);
	return response;
	}

But it is not found in onedrive-v2.5.0-alpha-3:

onedrive/src/onedrive.d

Lines 1427 to 1434 in 77c294b

	private auto post(T)(string url, const(T)[] postData) {
	curlEngine.setMethodPost();
	curlEngine.http.url = url;
	addAccessTokenHeader();
	auto response = perform(postData);
	checkHttpResponseCode(response);
	return response;
	}

Why was the line removed?

[abraunegg]

@Lyncredible

For example, I see http.clearRequestHeaders() being called in master branch

OK - thats a gap & bug in 'alpha-3' that needs to be corrected. Will look at that shortly.

Also - just to keep comments / feedback in the same place, could you provide feedback here: #2521 rather than in this PR that would be great.

[abraunegg]

@Lyncredible

Resync was deemed necessary because it detected sync_dir override from CLI:

OK .. thanks for this feedback .. essentially as 'sync_dir' is now being correctly being picked up as a change, this is what is happening. I am going to have to look at this a little closer on how to work around this for containerised environments.

[abraunegg]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.