Source-linked deployments for bundles in the workspace #1884

ilyakuz-db · 2024-11-05T17:22:06Z

Changes

This change adds a preset for source-linked deployments. It is enabled by default for targets in development mode if the Databricks CLI is running from the /Workspace directory on DBR. It does not have an effect when running the CLI anywhere else.

Key highlights:

Files in this mode won't be uploaded to workspace
Created resources will use references to source files instead of their workspace copies

Tests

Apply preset unit test covering conditional logic
High-level process target mode unit test for testing integration between mutators

…ototype

bundle/config/mutator/translate_paths.go

…ototype

shreyas-goenka

Thanks! Left a few comments for a few things to note when you do a proper pass!

bundle/config/mutator/apply_presets.go

…ototype

…y test

pietern · 2024-11-20T09:34:42Z

bundle/config/mutator/apply_presets.go

+		if !isDatabricksWorkspace {
+			disabled := false
+			b.Config.Presets.SourceLinkedDeployment = &disabled
+			diags = diags.Extend(diag.Warningf("source-linked deployment is available only in the Databricks Workspace"))


Warnings should include position/file/line information where possible.

This condition triggers only if the user explicitly configured the setting, so we should have this information. You can call b.Config.GetLocations() with the path to the setting to get these.

Not blocking for this PR.

We have this setting in targets.[target].presets.source_linked_deployment and this should be in the config indeed in that case, but we have nil in b.Config.Targets which makes location lookup not possible

It is assigned here

cli/bundle/config/mutator/select_target.go

Line 53 in ed19466

b.Config.Targets = nil

I can try to remove this nil assingment from there, not sure why it is needed. If I remove this line locations are available

Also we can show something like this without locations

You should just be able to use the locations from top-level presets.source_linked_deployment. That should include the target override locations as well IIRC.

That works indeed, thanks!

Will make another PR later

bundle/config/presets.go

pietern · 2024-11-20T10:01:42Z

bundle/deploy/files/upload.go

@@ -23,6 +24,11 @@ func (m *upload) Name() string {
 }

 func (m *upload) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
+	if config.IsExplicitlyEnabled(b.Config.Presets.SourceLinkedDeployment) {
+		cmdio.LogString(ctx, "Source-linked deployment is enabled. Resources will point to the source files directly without making copies")


All users will see this, let's go through copy-editing.

pietern · 2024-11-20T10:02:25Z

bundle/trampoline/python_dbr_warning.go

@@ -22,6 +24,9 @@ func WrapperWarning() bundle.Mutator {

 func (m *wrapperWarning) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
 	if isPythonWheelWrapperOn(b) {
+		if config.IsExplicitlyEnabled(b.Config.Presets.SourceLinkedDeployment) {
+			cmdio.LogString(ctx, "Python wheel notebook wrapper is not available when using source-linked deployment mode. You can disable this mode by setting 'presets.source_linked_deployment: false'")


Why is this a log vs a warning?

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

…to tmp/in-place-prototype

github-actions · 2024-11-20T12:00:43Z

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

PR number: 1884
Commit SHA: 66bf5f35c17f7e8652778b4d1975ea496239c933

Checks will be approved automatically on success.

eng-dev-ecosystem-bot · 2024-11-20T12:01:17Z

Test Details: go/deco-tests/11932942515

**Note:** the `bundle generate` command now uses the `.<resource-type>.yml` sub-extension for the configuration files it writes. Existing configuration files that do not use this sub-extension are renamed to include it. Bundles: * Make `TableName` field part of quality monitor schema ([#1903](#1903)). * Do not prepend paths starting with ~ or variable reference ([#1905](#1905)). * Fix workspace extensions filer accidentally reading notebooks ([#1891](#1891)). * Fix template initialization when running on Databricks ([#1912](#1912)). * Source-linked deployments for bundles in the workspace ([#1884](#1884)). * Added integration test to deploy bundle to /Shared root path ([#1914](#1914)). * Update filenames used by bundle generate to use `.<resource-type>.yml` ([#1901](#1901)). Internal: * Extract functionality to detect if the CLI is running on DBR ([#1889](#1889)). * Consolidate test helpers for `io/fs` ([#1906](#1906)). * Use `fs.FS` interface to read template ([#1910](#1910)). * Use `filer.Filer` to write template instantiation ([#1911](#1911)).

ilyakuz-db added 2 commits November 5, 2024 18:12

Prototype for in-place deployments in DEV mode

6b22fde

Merge branch 'main' of github.com:databricks/cli into tmp/in-place-pr…

ae26bea

…ototype

ilyakuz-db temporarily deployed to test-trigger-is November 5, 2024 17:22 — with GitHub Actions Inactive

ilyakuz-db commented Nov 5, 2024

View reviewed changes

bundle/config/mutator/translate_paths.go Outdated Show resolved Hide resolved

ilyakuz-db requested a review from pietern November 5, 2024 17:24

fix: Add SyncRoot mock to tests

fbb9be9

ilyakuz-db temporarily deployed to test-trigger-is November 5, 2024 18:03 — with GitHub Actions Inactive

ilyakuz-db temporarily deployed to test-trigger-is November 5, 2024 18:04 — with GitHub Actions Inactive

ilyakuz-db added 4 commits November 11, 2024 11:20

Merge branch 'main' of github.com:databricks/cli into tmp/in-place-pr…

086dfbc

…ototype

feat: Use global FilePath instead of locally overriden paths

a43f7dd

feat: Move FilePath update to apply preset step

5a22151

Merge branch 'main' of github.com:databricks/cli into tmp/in-place-pr…

6c12308

…ototype

ilyakuz-db temporarily deployed to test-trigger-is November 12, 2024 12:25 — with GitHub Actions Inactive

shreyas-goenka reviewed Nov 12, 2024

View reviewed changes

fix: Use SyncRootPath to allow using parent directories

b2164c0

ilyakuz-db temporarily deployed to test-trigger-is November 13, 2024 16:51 — with GitHub Actions Inactive

ilyakuz-db added 2 commits November 14, 2024 18:12

Merge branch 'main' of github.com:databricks/cli into tmp/in-place-pr…

0004ed2

…ototype

feat: Use dbr package for runtime check

26f2453

ilyakuz-db changed the title ~~WIP: Prototype for in-place deployments in DEV mode~~ Prototype for in-place deployments in DEV mode Nov 14, 2024

ilyakuz-db temporarily deployed to test-trigger-is November 14, 2024 17:29 — with GitHub Actions Inactive

fix: Applies autoformat

51c8ef5

ilyakuz-db temporarily deployed to test-trigger-is November 14, 2024 17:38 — with GitHub Actions Inactive

ilyakuz-db temporarily deployed to test-trigger-is November 14, 2024 17:39 — with GitHub Actions Inactive

fix: Missing root variable

4cf6929

ilyakuz-db temporarily deployed to test-trigger-is November 14, 2024 17:48 — with GitHub Actions Inactive

ilyakuz-db requested review from pietern and andrewnester November 19, 2024 11:52

fix: Windows tests

5d04569

ilyakuz-db temporarily deployed to test-trigger-is November 19, 2024 12:16 — with GitHub Actions Inactive

andrewnester approved these changes Nov 19, 2024

View reviewed changes

fix: Skipping path translation tests for windows

530a2a9

pietern changed the title ~~Source-linked deployments for bundles in the Workspace (In-Place)~~ Source-linked deployments for bundles in the workspace Nov 19, 2024

ilyakuz-db temporarily deployed to test-trigger-is November 19, 2024 12:33 — with GitHub Actions Inactive

fix: Replaced diag warning message with logger and removed unnecessar…

aba35ae

…y test

ilyakuz-db temporarily deployed to test-trigger-is November 19, 2024 21:09 — with GitHub Actions Inactive

pietern reviewed Nov 20, 2024

View reviewed changes

ilyakuz-db and others added 2 commits November 20, 2024 11:33

fix: Update skip uploading log entry with more clear message

ddb68a7

Update bundle/config/presets.go

eede522

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

ilyakuz-db temporarily deployed to test-trigger-is November 20, 2024 10:34 — with GitHub Actions Inactive

ilyakuz-db added 2 commits November 20, 2024 11:39

Merge branch 'tmp/in-place-prototype' of github.com:databricks/cli in…

6a036d1

…to tmp/in-place-prototype

fix: Use warning instead of log in python wheel wrapper message

66bf5f3

ilyakuz-db temporarily deployed to test-trigger-is November 20, 2024 12:00 — with GitHub Actions Inactive

pietern approved these changes Nov 20, 2024

View reviewed changes

pietern merged commit 756e55f into main Nov 20, 2024
9 checks passed

pietern deleted the tmp/in-place-prototype branch November 20, 2024 12:22

pietern mentioned this pull request Nov 20, 2024

[Release] Release v0.235.0 #1918

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source-linked deployments for bundles in the workspace #1884

Source-linked deployments for bundles in the workspace #1884

ilyakuz-db commented Nov 5, 2024 •

edited by pietern

Loading

shreyas-goenka left a comment

pietern Nov 20, 2024

ilyakuz-db Nov 20, 2024

shreyas-goenka Nov 20, 2024

ilyakuz-db Nov 20, 2024

pietern Nov 20, 2024

ilyakuz-db Nov 20, 2024

pietern Nov 20, 2024

ilyakuz-db Nov 20, 2024

github-actions bot commented Nov 20, 2024

eng-dev-ecosystem-bot commented Nov 20, 2024

Source-linked deployments for bundles in the workspace #1884

Source-linked deployments for bundles in the workspace #1884

Conversation

ilyakuz-db commented Nov 5, 2024 • edited by pietern Loading

Changes

Tests

shreyas-goenka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 20, 2024

eng-dev-ecosystem-bot commented Nov 20, 2024

ilyakuz-db commented Nov 5, 2024 •

edited by pietern

Loading