Intellisense for notebooks
Concatentation of the notebook cells
Intellisense in VS code works by sending LSP requests to a separate process (well in most cases, see this for more info)
Something like so:
Intellisense for notebooks works pretty much the same way but with each cell of a notebook being a text document:
This poses a problem for the language server (Pylance) because code from one cell can be referenced in another.
Example:
In that example, the pandas import crosses the cell boundary.
This means pylance cannot just analyze each cell individually.
The solution was changes to LSP:
| Message | When it is sent | What it's for |
|---|---|---|
| DidOpenNotebookDocument | On notebook open | Let's the server know what documents are part of a notebook so that they can be concatenated together |
| DidChangeNotebookDocument | On moving, deleting or adding cells | Let's the server update its representation of the notebook cell order |
| DidSaveNotebookDocument | On notebook save | When the notebook is saved its relative path is established. This is necessary to determine location with respect to imports. |
| DidCloseNotebookDocument | On notebook close | Server can clean up in memory state about the notebook. |
These 4 new messages means that pylance knows the relative order of all of the cells in a notebook. Here's generally the sequence of events:
[sequence diagram of lsp messages]
From this pylance knows that:
- There are 3 python documents
- They belong to the notebook
foo.ipynb - Their order is 3, 1, 2
So if the notebook looked like so:
[picture of notebook with 3 cells]
Internally pylance could have a concatenated document like so:
# Cell 1
import pandas as pd
# Cell 2
df = pd.read_csv('./data/test.csv')
# Cell 3
df.head()There's still one missing part in this design. How does pylance know what interpreter to use?
When Pylance is started, the Python extension tells it what the pythonPath is:
[picture of pylance and python extension]
But a notebook uses a kernel. The kernel isn't tied to the global pythonPath.
To workaround this problem, a custom message is sent from Pylance : foobar
The Python extension handles this message and then [asks](jupyter extension location) if it knows the pythonPath for a URI.
[picture of jupyter with pylance and python]
If Jupyter has a kernel active for a notebook, it returns the known pythonPath for that kernel.
Sometimes things go wrong with intellisense. And one of the most difficult parts of diagnosing these failures is reproing the problem.
Pylance logs a lot of data about