Intellisense for notebooks

Table of Contents

Overview Notebooks Architecture

Intellisense Overview

Intellisense in VS code works by sending LSP requests to a separate process (well in most cases, see this for more info)

Something like so:

Intellisense for notebooks

Intellisense for notebooks works pretty much the same way but with each cell of a notebook being a text document:

Concatentation of the notebook cells

This poses a problem for the language server (Pylance) because code from one cell can be referenced in another.

Example:

In that example, the pandas import crosses the cell boundary.

This means pylance cannot just analyze each cell individually.

The solution was to concatenate the cells in order.

This changes the original architecture to something like so:

How does concatenation actually work

Concatenation is mostly just a raw concat of all of the contents of each cell on top of each other. Then the concat document has functions to map back and forth between the original cells and the concatenated contents.

Code for this can be found here

Here's an example of using it:

public async provideReferences(
        document: vscode.TextDocument,
        position: vscode.Position,
        options: {
            includeDeclaration: boolean;
        },
        token: vscode.CancellationToken,
        _next: protocol.ProvideReferencesSignature
    ) {
        const client = this.getClient();
        if (this.shouldProvideIntellisense(document.uri) && client) {
            const documentId = this.asTextDocumentIdentifier(document);
            const newDoc = this.converter.toConcatDocument(documentId);
            const newPos = this.converter.toConcatPosition(documentId, position);
            const params: protocol.ReferenceParams = {
                textDocument: newDoc,
                position: newPos,
                context: {
                    includeDeclaration: options.includeDeclaration
                }
            };
            const result = await client.sendRequest(protocolNode.ReferencesRequest.type, params, token);
            const notebookResults = this.converter.toNotebookLocations(result);
            return client.protocol2CodeConverter.asReferences(notebookResults);
        }
    }

That is the handler for the references LSP request.

It is

• translating the incoming cell uri into a concat document
• translating the incoming cell position into a concat document
• sending the request using the concat data
• translating the results back into a cell uri

Kernel vs Interpreter

When pylance starts up, it is passed an interpreter that defines what modules are installed. For example, in this example, pylance is running with my Python 3.10 environment which is missing scikit-learn:

For python files, this interpreter is determined in the bottom left of VS code:

That interpreter is used by pylance to determine where it will find all of the modules it checks. So in this example, the window's 3.10 64 bit environment does not have the module 'scikit-learn'

Notebooks interpreter

Notebooks don't have a 'global' interpreter, but rather a 'kernel' that is used to run the code. This kernel is almost always associated with a python interpreter.

This interpreter is what we need to pass to pylance so it can find the correct modules.

This complicates how pylance is started.

For a normal python file, this is how things are started: