Explore user data sync API

[sandy081]

This is to explore an API proposal for synchronising user data.

An API that extensions can register the user data and login providers for synchronising user data in VS Code

[sandy081]

From @shanalikhan

Okay, its great to see UX provided by VSCode itself and allow extensions to wrap their own system.

I would suggest you to provide some of the API alongwith the UX when we talk about "Settings Synch" in general.

I have few questions in general for some of which there are already seperate issues opened. I'm merging all of them here to allow us to see bigger picture.

1. Settings Sync needs to manually find the User path for each of the code variant. Initially Settings Sync had to hard code the paths but now although its very much improved. Settings Sync still uses some techniques to find the User folder and Extension folder as written here I would suggest to allow authors to read the User folder directly by just an API.

2. Extension Synch is not completely supported until extension.all is updated after code restarts. I would suggest to either improve that or provide API to catch them. Reference : vscode.extensions Doesn't Update On Installing / Removing / Disabling Extensions without restarting #14444

3. Currently Settings Sync cannot disable / enable the extensions due to the limitations from the API. I would request to provide API for that. Reference : How to enable / disable extension from API or commands #15466

These are the some improvement i see from the API side to allow Settings Sync extension to fully utilize its power and increase productivity.

Now talking about adding UX. I would like to see the following things in the UX

1. There should be configurations to allow extension to read the following:
   • Settings
   • Extensions
   • Keybindings
   • Alot more

Settings Sync has these configuration of its own, i would like to see this a part of Code so extension authors wont have to manage this.

2. As suggested by @borekb (Explore settings sync UX #78869 (comment)) "Sync Pragmas" are essential part of Settings Sync, i would suggest to provide UX to allow me to see which Settings needs to be ignored by Settings Sync.

3. Provide UX to allow users to include workspace settings in a part of Sync. Settings Sync supports it. Reference : here

At the end, how you guys will enforce the newly made API. I would suggest you to send PR to allow Settings Sync to integrate complely with the new UX. It would take time from my side to integrate the new UX.

I have recenly added new UX in Settings Sync that integrates the Github. This UI needs to be integrated with your UX to let Settings Sync only manage the external storage part and keep the permissions and configurations to Code UI.

These are the initial things that comes to my mind.

[sandy081]

@shanalikhan Sorry for not providing the complete details. Following is our approach to support user data sync in VS Code.

Core

VS Code will completely control what data to sync. Currently following data is being considered

• Settings
• Keybindings
• Snippets
• Extensions
• Global State (Activity bar icons, Open Editors etc)

VS Code will provide the UI for the user to configure what to sync and also provide options to auto sync or not. VS Code also provide a special view in extensions to show already user installed extension in other machines and allow user to install them locally.

VS Code will not have a database or cloud to store user data, but will provide an API so that extensions can do this. VS Code will use this API to sync (read/write) user data.

API

VS Code will provide an API for extensions to provide a backend data service to store user data thereby enabling user data sync. For example

export function registerUserDataProvider(account: string, userDataProvider: FileSystemProvider): Disposable;

account is the user account to which this userDataProvider is associated to. For eg., gitbub. (We also have plans to provide a log in mechanism so that log in is consistent across and extensions can share log in details. Will not go in deeper as it will side track our main discussion.)

Extensions can implement this API and provide VS Code a way to read and write user data which can be used to sync across machines.

Summary

Extensions will provide just a back end service to read and write user data and VS Code will drive the synchronisation from UI to what to sync.

So, it means extensions do not need to worry about how to read user data which can be at different locations and also an implementation detail of VS Code. Also all the requirements you have mentioned above will be handled by VS Code by using just above API.

Hope it is clear. Please let me know if you have any questions.

[sandy081]

@shanalikhan

I would suggest to provide UX to allow me to see which Settings needs to be ignored by Settings Sync.

Yes this is being considered.

Provide UX to allow users to include workspace settings in a part of Sync. Settings Sync supports it.

Assuming you are not mentioning about workspace settings which is already shareable. If it is about custom files we can discuss about it.

At the end, how you guys will enforce the newly made API. I would suggest you to send PR to allow Settings Sync to integrate complely with the new UX. It would take time from my side to integrate the new UX.

Regarding adopting, It is needed to provide a User Data Provider by the extension to provide access to the user data it is storing. In case of Settings Sync extension which is storing user data in github gist, a User Data Provider that reads and writes from and into github gists is needed. This can be done by mapping the requests to User Data Provider to the gists, for eg., settings.json request can be mapped to settings gist. Also we can discuss about how the format of extensions could be.

I do not think there is any adoption needed for UX.

I have recenly added new UX in Settings Sync that integrates the Github. This UI needs to be integrated with your UX to let Settings Sync only manage the external storage part and keep the permissions and configurations to Code UI.

Log in and permissions is not part of user data sync and VS Code core does not drive it as of now. So extension can still provide the UI like you have now for log in and permissions.

[sandy081]

Here is the api proposal for supporting user data synchronisation.

export namespace window {

		/**
		 * Register an [UserDataSyncProvider](#UserDataSyncProvider) to read and write user data.
		 * @param name Name of the user data sync provider
		 * @param userDataSyncProvider [UserDataSyncProvider](#UserDataSyncProvider) to read and write user data
		 */
		export function registerUserDataSyncProvider(name: string, userDataSyncProvider: UserDataSyncProvider): Disposable;

	}

	export class UserDataError extends Error {

		/**
		 * Create an error to signal that writing user data with given ref is rejected, becase of new ref.
		 */
		static Rejected(): FileSystemError;

		/**
		 * Creates a new userData error.
		 */
		constructor();
	}

	/**
	 * User data sync provider to read and write user data.
	 */
	export interface UserDataSyncProvider {

		/**
		 * Reads the content and its ref for the given key.
		 * Return <code>null</code> if key does not exists.
		 *
		 * @param key key of the content to read
		 * @returns the content and its ref for the given key. Return <code>null</code> if key does not exists.
		 */
		read(key: string): Promise<{ content: string, ref: string } | null>;

		/**
		 * Writes the new content based on the given ref for the given key.
		 *
		 * @param key key of the content to write
		 * @param content new content to write
		 * @param ref ref of the content on which the content to write is based on
		 * @throws [Rejected](#UserDataError.Rejected) if the ref is not the latest.
		 * @returns the latest ref of the content.
		 */
		write(key: string, content: string, ref: string | null): Promise<string>;

	}

This will allow extension to provide an user data sync store from which VS Code can read and write user data to sync.

[borekb]

Two questions about the current API suggestion:

1. How is selective sync going to be supported? For example, zooming VSCode immediately puts window.zoomLevel to settings.json but I need to exclude that from the sync, which I'm currently achieving via a @sync-ignore pragma in Settings Sync. A much cleaner solution, IMO, would be to implement Add support for additional global config files (in addition to settings.json) #17634 / [Feature] Local Workspace settings #40233 – are you possibly considering it?
2. The API doesn't seem to contain any functions to deal with diffs and conflicts. From my experience with Settings Sync, not being sure if I'm overwriting my remote settings or not is bad for confidence. This would be even worse here as some backends like S3 are not versioned (by default).

[sandy081]

@borekb

How is selective sync going to be supported?

Selective Sync is supported by VS Code. There will be a way for the user to select/unselect settings from sync. I am still thinking how to be provide this feature from VS Code. One idea is to expose a setting to ignore settings from sync.

The API doesn't seem to contain any functions to deal with diffs and conflicts

Handling conflicts and merging settings is supported. It is nothing to do with the API. API just provides the ref of the remote contents and VS Code handles the merge and conflicts (if there are any) while syncing local and remote. An extension providing the back end (store) should take care of handling the refs correctly.

Currently I am planning to show a textual merge editor when there are conflicts in settings while syncing.

[borekb]

Thanks, that sounds great. Especially seeing diffs / merges inside VSCode is awesome!

BTW, is there another issue for UX around data sync? This one seems to be mainly about the API, the UX one was #78869 but there's a note that the discussion will happen here: #78869 (comment).

If there are early UI/UX prototypes, I'd like to provide early feedback, so if I should watch this issue or some other one. Thanks!

[sandy081]

Update:

To begin with, there are no plans to expose an API like above and hence I am closing this.

Settings Sync feature work #2743 is still in progress and we are currently investigating and working on a different approach to read and write settings data. It can be a rest service end point or anything.