Intellisense for notebooks
This page describes how intellisense works for notebooks (and the interactive window) when the following experiment is enabled:
"python.pylanceLspNotebooksEnabled": trueIf that setting is false, the old way is used instead.
Concatentation of the notebook cells
Understanding notebooks in the server
Pylance can now understand notebooks
Intellisense in VS code works by sending LSP requests to a separate process (well in most cases, see this for more info)
Something like so:
Intellisense for notebooks works pretty much the same way but with each cell of a notebook being a text document:
This poses a problem for the language server (Pylance) because code from one cell can be referenced in another.
Example:
In that example, the pandas import crosses the cell boundary.
This means pylance cannot just analyze each cell individually.
The solution was to have messages in the LSP to teach Pylance about notebooks:
| Message | When it is sent | What it's for |
|---|---|---|
| DidOpenNotebookDocument | On notebook open | Sends the cells and their contents on open |
| DidChangeNotebookDocument | On moving, deleting or adding cells | For cell changes, moves, deletes, and adds |
| DidSaveNotebookDocument | On notebook save | When the notebook is saved its relative path is established. This is necessary to determine location with respect to imports. |
| DidCloseNotebookDocument | On notebook close | Server can clean up in memory state about the notebook. |
These 4 new messages means that pylance knows the relative order and contents of all of the cells in a notebook. Here's generally the sequence of events:
From this pylance knows that:
- There are 3 cells
- They belong to the notebook
foo.ipynb
So if the notebook looked like so:
Internally pylance could have a concatenated document like so (pylance may or may not concat the cells):
# Cell 1
import pandas as pd
# Cell 2
df = pd.read_csv('./data/test.csv')
# Cell 3
df.head()There's still one missing part in this design. How does pylance know what interpreter to use?
When Pylance is started, the Python extension tells it what the pythonPath is through the workspace/configuration message:
But a notebook uses a kernel. The kernel isn't tied to the global pythonPath.
To workaround this problem, pylance sends the workspace/configuration for every notebook opened.
The Python extension handles this message and then asks if Jupyter knows the pythonPath for a URI.
If Jupyter has a kernel active for a notebook, it returns the known pythonPath for that kernel.
There's another provider for intellisense for notebooks, the jupyter kernel itself. Jupyter provides a completions message that can provide completions at a code location.
We register a separate completion provider that queries the jupyter kernel for completions.
VS code calls into this completion provider when the user types (the same as all other completion providers, like the one the Python extension registered to talk to Pylance).
There's a problem though. The Jupyter completions will end up being dupes of the pylance data. With the old way of intellisense, the Jupyter extension had a direct connection to the pylance server and could eliminate dupes from the Kernel completions.
With the new way, dupes are currently not eliminated.
Sometimes things go wrong with intellisense. And one of the most difficult parts of diagnosing these failures is reproing the problem.
Pylance logs a lot of data about each document and their changes in this output window here:
This window is almost like a history window. It contains the state of a notebook and every change that was typed into it.
If you're in old mode, there's a separate language server output for each kernel path. Example:
You can ask the user to turn on full logging for this window with these settings:
"notebook-intellisense.logLevel": "Trace",
"notebook-intellisense.trace.server.verbosity": "Verbose",
The output in this window constitutes the log of everything sent to the language server. We can use this to replay what happened to a notebook to try and recreate the state that caused a bug.
To replay the log:
- Save output of Pylance to a .log file
- Build Jupyter extension locally
- Create an empty notebook
- Name it the same as the notebook in the original issue (you can generally find the name in the cell URIs in the log)
- Run the
Jupyter (Dev): Replay Pylance Logcommand - Open the log you saved
- This should fill in all the cells that were in the original file. Path matters so you might have to replicate the user's path.
- Run the
Jupyter (Dev): Step Pylance Logcommand over and over
Theoretically it will make all the edits to the notebook that the user made. (Edit : it doesn't work with newer VS code because fragment URIs have changed, it uses this to compute indices)
The replay tool understands textDocument/didOpen and textDocument/didChange events. If it was modified to understand the notebook messages, it could theoretically replay logs from customers in the 'new' mode for notebook intellisense without the fragment problems because the notebook messages should be enough to fully recreate the notebook.