Cross-file Typescript support in vscode-web (#169311)

* recreate logging from other machine

* comment out openSystemBrowser

Because I dont have the default browser set up correctnly on any of my
machiens

* Add vscode-wasm-typescript dep

And some logging as I figure out how to use it

* remove unused reference to module

* use require reference that linter allows

* Add vscode-wasm-typescript to tsserver.web.js

Use webpack's CopyPlugin transform pattern to do this manually. This is
probably a bad idea! It's just for prototyping purposes.

* Update vscode-wasm-typescript dependency

* Fix minor syntax in webpack hack

Nonetheless required for it to work!

* Fix another typo in webpack hack!

* Fix provided typescript path

Another typo. Guess my local test wasn't running the contents really

* Try to improve module.exports handling in webpack hac

* tsserver.web.js comes from local builds

Also:
- tsserver.js is no longer minified
- log crossOriginIsolated

* First attempt to set up server-side support

* Remove auto-imported identifier

* Move sync-api setup code to serverProcess.browser.ts

Because it's browser-specific

* Reorder webpack hack and clean up unused logging

* Update vscode-wasm/vscode-wasm-typescript dependencies

* Add file watching

* Extract webpack hack

Build only the ts parts of tsserver.web.js, don't rebuild the vscode
extension. This is a lot faster.

* Remove manual verbose logging

Sheetal showed me the correct way to create a verbose logger instead.

* Add vscode-test-web to semantic-supported schemes

And make isWeb support semantic mode.

* Also update the webpack-hack-only build

* Switch to tsserverlibrary

Also paste in some example code for cancellation, which is not finished
at all.

* Remove bogus auto-import and unneeded (?) dep

* remove webpack-like hack

* move code from vscode-wasm-typescript

* Initial prototype of cancellation

It compiles and looks kind of plausible. But I haven't tested it yet.

* Switch tsserver to separate MessageChannel

* Move watches to a separate MessagePort

Further simplifies the message dispatch code by shifting complexity to
setup. And the setup is straight-line code.

* switch vscode-web from in-memory to real filesystem

goto-def is currently broken because some part of main vscode still
needs treat the files as in-memory, though.

* Make toResource translate / -> vscode-test-web

* Encode scheme and authority in TS filenames

Like the previous host did, but without the leading ^ that TS hard-codes
as "in-memory".

The tsserver host needs to know about the encoding, but the translation
is in a single function. This also means that displayed file paths are
prefixed with /scheme/authority (/vscode-test-web/mount in my testing), but I think that's fine.

* Lift parseUri outside createServerHost

I'm not using it to set the schema/authority for getCurrentDirectory
right now, so there's no shared state to mutate.

* Special-case URI of lib*d.ts in webServer.toResource

Similar to the special-casing in typescriptServiceClient.toResource.
Also requires passing in the extensionUri. This feels like it's breaking
through at least one abstraction layer, which might be a problem.

* Improve cancellation

1. Simplify cancellation checking in web tsserver host to match the
checking in typescript's node host.
2. Move cancellation of request in the extension to tryCancelRequest
from sendNextRequests.
3. Only send cancellation via node or web cancellation, not both.

* Pass in current request instead of waiting for a fresh one.

* Address initial PR comments

Also add some TODO comments for the revision to watches.

* Add cancellation bit to each (cancellable) request, locally fix an issue with retrieving the cancellation bit

* Switch to per-file/directory watches

Watching the entire filesystem recursively is supposed to be
inefficient.

Not done yet: there is an error when watching directories, but it works.
And I can't tell whether watching files works yet.

* Parse --serverMode partialSemantic in webServer

Now the syntax server actually runs as a syntax server.

* Simplify logging code

* Cleanup in webServer

1. Remove a little logging.
2. Correct failure return value for getFileSize
3. Reorder some methods and parameters.

* Switch to markdown extension's FileWatcherManager

I'm not sure if it's OK to depend on a module from another extension;
it's probably better to include the files from a central place instead.

* Clean up host methods

1. Copy and adapt implementations from node host where possible.
2. Note questions for the PR elsewhere.
3. Remove logging except for caught exceptions.

* More logging/TODO cleanup

* Remove duplicate dependency

* Add setting to enable/disable semantic mode on web

Also gate it behind a check to `crossOriginIsolated`

* Re-order and re-arrange code to minimise PR diff

It won't minimise it *much*, but I also consolidated some
unnecessarily-spread-out code that will be easier to read in the long
term, and possibly easier to read in diff form as well.

* Copy fileWatchingManager to typescript extension

Copy from markdown extension to typescript extension. I used the
existing dependencies in the typescript extension, but verified that
they would work the same.

* Fix linting of webServer

* Align formatting of catch / else

* Extract isProjectWideIntellisenseOnWebEnabled and keep using in-memory prefix when project wide intellisense is disabled

* Make sure we still work if SharedArrayBuffers aren't supported

* Remove symlink support and fix typo

Symlinks are implicitly supported by the filesystem right now.

* Fix compile errors

Co-authored-by: Johannes <johannes.rieken@gmail.com>
Co-authored-by: Matt Bierner <matb@microsoft.com>

.eslintignore

@@ -15,7 +15,6 @@
 **/extensions/typescript-language-features/test-workspace/**
 **/extensions/typescript-language-features/extension.webpack.config.js
 **/extensions/typescript-language-features/extension-browser.webpack.config.js
-**/extensions/typescript-language-features/web/**
 **/extensions/vscode-api-tests/testWorkspace/**
 **/extensions/vscode-api-tests/testWorkspace2/**
 **/fixtures/**

extensions/typescript-language-features/package.json

@@ -38,6 +38,9 @@
     "jsonc-parser": "^3.2.0",
     "semver": "5.5.1",
     "vscode-tas-client": "^0.1.63",
+    "@vscode/sync-api-client": "^0.7.2",
+    "@vscode/sync-api-common": "^0.7.2",
+    "@vscode/sync-api-service": "^0.7.3",
     "vscode-uri": "^3.0.3"
   },
   "devDependencies": {
@@ -1217,6 +1220,15 @@
           "default": true,
           "description": "%configuration.suggest.objectLiteralMethodSnippets.enabled%",
           "scope": "resource"
+        },
+        "typescript.experimental.tsserver.web.enableProjectWideIntellisense": {
+          "type": "boolean",
+          "default": false,
+          "description": "%typescript.experimental.tsserver.web.enableProjectWideIntellisense%",
+          "scope": "window",
+          "tags": [
+            "experimental"
+          ]
         }
       }
     },

extensions/typescript-language-features/package.nls.json

@@ -194,6 +194,8 @@
 	"configuration.suggest.classMemberSnippets.enabled": "Enable/disable snippet completions for class members.",
 	"configuration.suggest.objectLiteralMethodSnippets.enabled": "Enable/disable snippet completions for methods in object literals. Requires using TypeScript 4.7+ in the workspace.",
 
+	"typescript.experimental.tsserver.web.enableProjectWideIntellisense": "Enable/disable project-wide IntelliSense on web. Requires that VS Code is running in a trusted context.",
+
 	"walkthroughs.nodejsWelcome.title": "Get started with JavaScript and Node.js",
 	"walkthroughs.nodejsWelcome.description": "Make the most of Visual Studio Code's first-class JavaScript experience.",
 

extensions/typescript-language-features/src/tsServer/fileWatchingManager.ts

@@ -0,0 +1,98 @@
+/*---------------------------------------------------------------------------------------------
+ *  Copyright (c) Microsoft Corporation. All rights reserved.
+ *  Licensed under the MIT License. See License.txt in the project root for license information.
+ *--------------------------------------------------------------------------------------------*/
+
+import * as vscode from 'vscode';
+import { Utils } from 'vscode-uri';
+import { disposeAll, IDisposable } from '../utils/dispose';
+import { ResourceMap } from '../utils/resourceMap';
+import { Schemes } from '../utils/schemes';
+
+type DirWatcherEntry = {
+	readonly uri: vscode.Uri;
+	readonly listeners: IDisposable[];
+};
+
+
+export class FileWatcherManager {
+
+	private readonly _fileWatchers = new Map<number, {
+		readonly watcher: vscode.FileSystemWatcher;
+		readonly dirWatchers: DirWatcherEntry[];
+	}>();
+
+	private readonly _dirWatchers = new ResourceMap<{
+		readonly watcher: vscode.FileSystemWatcher;
+		refCount: number;
+	}>(uri => uri.toString(), { onCaseInsensitiveFileSystem: false });
+
+	create(id: number, uri: vscode.Uri, watchParentDirs: boolean, isRecursive: boolean, listeners: { create?: (uri: vscode.Uri) => void; change?: (uri: vscode.Uri) => void; delete?: (uri: vscode.Uri) => void }): void {
+		const watcher = vscode.workspace.createFileSystemWatcher(new vscode.RelativePattern(uri, isRecursive ? '**' : '*'), !listeners.create, !listeners.change, !listeners.delete);
+		const parentDirWatchers: DirWatcherEntry[] = [];
+		this._fileWatchers.set(id, { watcher, dirWatchers: parentDirWatchers });
+
+		if (listeners.create) { watcher.onDidCreate(listeners.create); }
+		if (listeners.change) { watcher.onDidChange(listeners.change); }
+		if (listeners.delete) { watcher.onDidDelete(listeners.delete); }
+
+		if (watchParentDirs && uri.scheme !== Schemes.untitled) {
+			// We need to watch the parent directories too for when these are deleted / created
+			for (let dirUri = Utils.dirname(uri); dirUri.path.length > 1; dirUri = Utils.dirname(dirUri)) {
+				const dirWatcher: DirWatcherEntry = { uri: dirUri, listeners: [] };
+
+				let parentDirWatcher = this._dirWatchers.get(dirUri);
+				if (!parentDirWatcher) {
+					const glob = new vscode.RelativePattern(Utils.dirname(dirUri), Utils.basename(dirUri));
+					const parentWatcher = vscode.workspace.createFileSystemWatcher(glob, !listeners.create, true, !listeners.delete);
+					parentDirWatcher = { refCount: 0, watcher: parentWatcher };
+					this._dirWatchers.set(dirUri, parentDirWatcher);
+				}
+				parentDirWatcher.refCount++;
+
+				if (listeners.create) {
+					dirWatcher.listeners.push(parentDirWatcher.watcher.onDidCreate(async () => {
+						// Just because the parent dir was created doesn't mean our file was created
+						try {
+							const stat = await vscode.workspace.fs.stat(uri);
+							if (stat.type === vscode.FileType.File) {
+								listeners.create!(uri);
+							}
+						} catch {
+							// Noop
+						}
+					}));
+				}
+
+				if (listeners.delete) {
+					// When the parent dir is deleted, consider our file deleted too
+					// TODO: this fires if the file previously did not exist and then the parent is deleted
+					dirWatcher.listeners.push(parentDirWatcher.watcher.onDidDelete(listeners.delete));
+				}
+
+				parentDirWatchers.push(dirWatcher);
+			}
+		}
+	}
+
+	delete(id: number): void {
+		const entry = this._fileWatchers.get(id);
+		if (entry) {
+			for (const dirWatcher of entry.dirWatchers) {
+				disposeAll(dirWatcher.listeners);
+
+				const dirWatcherEntry = this._dirWatchers.get(dirWatcher.uri);
+				if (dirWatcherEntry) {
+					if (--dirWatcherEntry.refCount <= 0) {
+						dirWatcherEntry.watcher.dispose();
+						this._dirWatchers.delete(dirWatcher.uri);
+					}
+				}
+			}
+
+			entry.watcher.dispose();
+		}
+
+		this._fileWatchers.delete(id);
+	}
+}

extensions/typescript-language-features/src/tsServer/server.ts

@@ -4,6 +4,7 @@
  *--------------------------------------------------------------------------------------------*/
 
 import * as vscode from 'vscode';
+import { Cancellation } from '@vscode/sync-api-common/lib/common/messageCancellation';
 import type * as Proto from '../protocol';
 import { EventName } from '../protocol.const';
 import { CallbackMap } from '../tsServer/callbackMap';
@@ -17,6 +18,7 @@ import Tracer from '../utils/tracer';
 import { OngoingRequestCanceller } from './cancellation';
 import { TypeScriptVersionManager } from './versionManager';
 import { TypeScriptVersion } from './versionProvider';
+import { isWebAndHasSharedArrayBuffers } from '../utils/platform';
 
 export enum ExecutionTarget {
 	Semantic,
@@ -64,6 +66,7 @@ export interface TsServerProcessFactory {
 		kind: TsServerProcessKind,
 		configuration: TypeScriptServiceConfiguration,
 		versionManager: TypeScriptVersionManager,
+		extensionUri: vscode.Uri,
 	): TsServerProcess;
 }
 
@@ -171,17 +174,16 @@ export class ProcessBasedTsServer extends Disposable implements ITypeScriptServe
 		}
 	}
 
-	private tryCancelRequest(seq: number, command: string): boolean {
+	private tryCancelRequest(request: Proto.Request, command: string): boolean {
+		const seq = request.seq;
 		try {
 			if (this._requestQueue.tryDeletePendingRequest(seq)) {
 				this.logTrace(`Canceled request with sequence number ${seq}`);
 				return true;
 			}
-
 			if (this._requestCanceller.tryCancelOngoingRequest(seq)) {
 				return true;
 			}
-
 			this.logTrace(`Tried to cancel request with sequence number ${seq}. But request got already delivered.`);
 			return false;
 		} finally {
@@ -221,8 +223,14 @@ export class ProcessBasedTsServer extends Disposable implements ITypeScriptServe
 				this._callbacks.add(request.seq, { onSuccess: resolve as () => ServerResponse.Response<Proto.Response> | undefined, onError: reject, queuingStartTime: Date.now(), isAsync: executeInfo.isAsync }, executeInfo.isAsync);
 
 				if (executeInfo.token) {
+
+					const cancelViaSAB = isWebAndHasSharedArrayBuffers()
+						? Cancellation.addData(request)
+						: undefined;
+
 					executeInfo.token.onCancellationRequested(() => {
-						this.tryCancelRequest(request.seq, command);
+						cancelViaSAB?.();
+						this.tryCancelRequest(request, command);
 					});
 				}
 			}).catch((err: Error) => {

extensions/typescript-language-features/src/tsServer/serverProcess.browser.ts

@@ -2,30 +2,43 @@
  *  Copyright (c) Microsoft Corporation. All rights reserved.
  *  Licensed under the MIT License. See License.txt in the project root for license information.
  *--------------------------------------------------------------------------------------------*/
-
+/// <reference lib='webworker' />
 import * as vscode from 'vscode';
 import type * as Proto from '../protocol';
 import { TypeScriptServiceConfiguration } from '../utils/configuration';
 import { memoize } from '../utils/memoize';
 import { TsServerProcess, TsServerProcessKind } from './server';
 import { TypeScriptVersion } from './versionProvider';
-
-
-
-declare const Worker: any;
-declare type Worker = any;
+import { ServiceConnection } from '@vscode/sync-api-common/browser';
+import { Requests, ApiService } from '@vscode/sync-api-service';
+import { TypeScriptVersionManager } from './versionManager';
+import { FileWatcherManager } from './fileWatchingManager';
+type BrowserWatchEvent = {
+	type: 'watchDirectory' | 'watchFile';
+	recursive?: boolean;
+	uri: {
+		scheme: string;
+		authority: string;
+		path: string;
+	};
+	id: number;
+} | {
+	type: 'dispose';
+	id: number;
+};
 
 export class WorkerServerProcess implements TsServerProcess {
-
 	public static fork(
 		version: TypeScriptVersion,
 		args: readonly string[],
 		_kind: TsServerProcessKind,
 		_configuration: TypeScriptServiceConfiguration,
+		_versionManager: TypeScriptVersionManager,
+		extensionUri: vscode.Uri,
 	) {
 		const tsServerPath = version.tsServerPath;
 		const worker = new Worker(tsServerPath);
-		return new WorkerServerProcess(worker, [
+		return new WorkerServerProcess(worker, extensionUri, [
 			...args,
 
 			// Explicitly give TS Server its path so it can
@@ -37,27 +50,78 @@ export class WorkerServerProcess implements TsServerProcess {
 	private readonly _onDataHandlers = new Set<(data: Proto.Response) => void>();
 	private readonly _onErrorHandlers = new Set<(err: Error) => void>();
 	private readonly _onExitHandlers = new Set<(code: number | null, signal: string | null) => void>();
+	private readonly watches = new FileWatcherManager();
+	/** For communicating with TS server synchronously */
+	private readonly tsserver: MessagePort;
+	/** For communicating watches asynchronously */
+	private readonly watcher: MessagePort;
+	/** For communicating with filesystem synchronously */
+	private readonly syncFs: MessagePort;
 
 	public constructor(
-		private readonly worker: Worker,
+		/** For logging and initial setup */
+		private readonly mainChannel: Worker,
+		extensionUri: vscode.Uri,
 		args: readonly string[],
 	) {
-		worker.addEventListener('message', (msg: any) => {
-			if (msg.data.type === 'log') {
-				this.output.append(msg.data.body);
+		const tsserverChannel = new MessageChannel();
+		const watcherChannel = new MessageChannel();
+		const syncChannel = new MessageChannel();
+		this.tsserver = tsserverChannel.port2;
+		this.watcher = watcherChannel.port2;
+		this.syncFs = syncChannel.port2;
+		this.tsserver.onmessage = (event) => {
+			if (event.data.type === 'log') {
+				console.error(`unexpected log message on tsserver channel: ${JSON.stringify(event)}`);
 				return;
 			}
-
 			for (const handler of this._onDataHandlers) {
-				handler(msg.data);
+				handler(event.data);
 			}
-		});
-		worker.onerror = (err: Error) => {
+		};
+		this.watcher.onmessage = (event: MessageEvent<BrowserWatchEvent>) => {
+			switch (event.data.type) {
+				case 'dispose': {
+					this.watches.delete(event.data.id);
+					break;
+				}
+				case 'watchDirectory':
+				case 'watchFile': {
+					this.watches.create(event.data.id, vscode.Uri.from(event.data.uri), /*watchParentDirs*/ true, !!event.data.recursive, {
+						change: uri => this.watcher.postMessage({ type: 'watch', event: 'change', uri }),
+						create: uri => this.watcher.postMessage({ type: 'watch', event: 'create', uri }),
+						delete: uri => this.watcher.postMessage({ type: 'watch', event: 'delete', uri }),
+					});
+					break;
+				}
+				default:
+					console.error(`unexpected message on watcher channel: ${JSON.stringify(event)}`);
+			}
+		};
+		mainChannel.onmessage = (msg: any) => {
+			// for logging only
+			if (msg.data.type === 'log') {
+				this.output.append(msg.data.body);
+				return;
+			}
+			console.error(`unexpected message on main channel: ${JSON.stringify(msg)}`);
+		};
+		mainChannel.onerror = (err: ErrorEvent) => {
+			console.error('error! ' + JSON.stringify(err));
 			for (const handler of this._onErrorHandlers) {
-				handler(err);
+				// TODO: The ErrorEvent type might be wrong; previously this was typed as Error and didn't have the property access.
+				handler(err.error);
 			}
 		};
-		worker.postMessage(args);
+		this.output.append(`creating new MessageChannel and posting its port2 + args: ${args.join(' ')}\n`);
+		mainChannel.postMessage(
+			{ args, extensionUri },
+			[syncChannel.port1, tsserverChannel.port1, watcherChannel.port1]
+		);
+		const connection = new ServiceConnection<Requests>(syncChannel.port2);
+		new ApiService('vscode-wasm-typescript', connection);
+		connection.signalReady();
+		this.output.append('done constructing WorkerServerProcess\n');
 	}
 
 	@memoize
@@ -66,7 +130,7 @@ export class WorkerServerProcess implements TsServerProcess {
 	}
 
 	write(serverRequest: Proto.Request): void {
-		this.worker.postMessage(serverRequest);
+		this.tsserver.postMessage(serverRequest);
 	}
 
 	onData(handler: (response: Proto.Response) => void): void {
@@ -83,6 +147,10 @@ export class WorkerServerProcess implements TsServerProcess {
 	}
 
 	kill(): void {
-		this.worker.terminate();
+		this.mainChannel.terminate();
+		this.tsserver.close();
+		this.watcher.close();
+		this.syncFs.close();
 	}
 }
+

extensions/typescript-language-features/src/tsServer/spawner.ts

@@ -44,6 +44,7 @@ export class TypeScriptServerSpawner {
 		private readonly _telemetryReporter: TelemetryReporter,
 		private readonly _tracer: Tracer,
 		private readonly _factory: TsServerProcessFactory,
+		private readonly _extensionUri: vscode.Uri,
 	) { }
 
 	public spawn(
@@ -152,7 +153,7 @@ export class TypeScriptServerSpawner {
 		}
 
 		this._logger.info(`<${kind}> Forking...`);
-		const process = this._factory.fork(version, args, kind, configuration, this._versionManager);
+		const process = this._factory.fork(version, args, kind, configuration, this._versionManager, this._extensionUri);
 		this._logger.info(`<${kind}> Starting...`);
 
 		return new ProcessBasedTsServer(