Add support for non-Python ipynb notebooks to DABs

[shreyas-goenka]

Changes

Background

The workspace import APIs recently added support for importing Jupyter notebooks written in R, Scala, or SQL, that is non-Python notebooks. This now works for the /import-file API which we leverage in the CLI.

Note: We do not need any changes in databricks sync. It works out of the box because any state mapping of local names to remote names that we store is only scoped to the notebook extension (i.e., .ipynb in this case) and is agnostic of the notebook's specific language.

Problem this PR addresses

The extension-aware filer previously did not function because it checks that a .ipynb notebook is written in Python. This PR relaxes that constraint and adds integration tests for both the normal workspace filer and extensions aware filer writing and reading non-Python .ipynb notebooks.

This implies that after this PR DABs in the workspace / CLI from DBR will work for non-Python notebooks as well. non-Python notebooks for DABs deployment from local machines already works after the platform side changes to the API landed, this PR just adds integration tests for that bit of functionality.

Note: Any platform side changes we needed for the import API have already been rolled out to production.

Before

DABs deploy would work fine for non-Python notebooks. But DABs deployments from DBR would not.

After

DABs deploys both from local machines and DBR will work fine.

Testing

For creating the .ipynb notebook fixtures used in the integration tests I created them directly from the VSCode UI. This ensures high fidelity with how users will create their non-Python notebooks locally. For Python notebooks this is supported out of the box by VSCode but for R and Scala notebooks this requires installing the Jupyter kernel for R and Scala on my local machine and using that from VSCode.

For SQL, I ended up directly modifying the language_info field in the Jupyter metadata to create the test fixture.

Discussion: Issues with configuring language at the cell level

The language metadata for a Jupyter notebook is standardized at the notebook level (in the language_info field). Unfortunately, it's not standardized at the cell level. Thus, for example, if a user changes the language for their cell in VSCode (which is supported by the standard Jupyter VSCode integration), it'll cause a runtime error when the user actually attempts to run the notebook. This is because the cell-level metadata is encoded in a format specific to VSCode:

cells: []{
    "vscode": {
     "languageId": "sql"
    }
}

Supporting cell level languages is thus out of scope for this PR and can be revisited along with the workspace files team if there's strong customer interest.

[shreyas-goenka]

triggered nightlies on this PR...

[shreyas-goenka]

nightlies are green

[pietern]

Thanks!

Please update the PR description to say this only affects the read path of the extension-aware filer, i.e. when the CLI runs on Databricks compute and reads from WSFS.

The comments about the cell-aware language settings don't apply to the PR and are more general commentary. You can keep it but delineate it with a header or so to indicate it's an appendix/commentary and doesn't apply to the change itself.

[pietern]

Comments on the description (nuances):

The workspace import APIs recently added support for importing notebooks written in R, Scala, or SQL, which are non-Python notebooks. This now work for the /import-file API which we leverage in the CLI.

The newly added support was for ipynb notebooks specifically.

Notebooks in R, Scala, or SQL were always importable if they were encoded in the source format.

[pietern]

Thanks, two minor things.

[pietern]

Integration tests are red.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 1827
• Commit SHA: be53fa5ff48adf94996bc5333ca2605e9d45760c

Checks will be approved automatically on success.

[eng-dev-ecosystem-bot]

Test Details: go/deco-tests/11819726261