Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make a notebook wrapper for Python wheel tasks optional #797

Merged
merged 5 commits into from
Sep 26, 2023

Conversation

andrewnester
Copy link
Contributor

@andrewnester andrewnester commented Sep 25, 2023

Changes

Instead of always using notebook wrapper for Python wheel tasks, let's make this an opt-in option.

Now by default Python wheel tasks will be deployed as is to Databricks platform.
If notebook wrapper required (DBR < 13.1 or other configuration differences), users can provide a following experimental setting

experimental:
  python_wheel_wrapper: true

Fixes #783, databricks/databricks-asset-bundles-dais2023#8

Tests

Added unit tests.

Integration tests passed for both cases

    helpers.go:163: [databricks stdout]: Hello from my func
    helpers.go:163: [databricks stdout]: Got arguments:
    helpers.go:163: [databricks stdout]: ['my_test_code', 'one', 'two']
    ...
Bundle remote directory is ***/.bundle/ac05d5e8-ed4b-4e34-b3f2-afa73f62b021
Deleted snapshot file at /var/folders/nt/xjv68qzs45319w4k36dhpylc0000gp/T/TestAccPythonWheelTaskDeployAndRunWithWrapper3733431114/001/.databricks/bundle/default/sync-snapshots/cac1e02f3941a97b.json
Successfully deleted files!
--- PASS: TestAccPythonWheelTaskDeployAndRunWithWrapper (214.18s)
PASS
coverage: 93.5% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       214.495s        coverage: 93.5% of statements in ./...

    helpers.go:163: [databricks stdout]: Hello from my func
    helpers.go:163: [databricks stdout]: Got arguments:
    helpers.go:163: [databricks stdout]: ['my_test_code', 'one', 'two']
    ...
Bundle remote directory is ***/.bundle/0ef67aaf-5960-4049-bf1d-dc9e29157421
Deleted snapshot file at /var/folders/nt/xjv68qzs45319w4k36dhpylc0000gp/T/TestAccPythonWheelTaskDeployAndRunWithoutWrapper2340216760/001/.databricks/bundle/default/sync-snapshots/edf0b322cee93b13.json
Successfully deleted files!
--- PASS: TestAccPythonWheelTaskDeployAndRunWithoutWrapper (192.36s)
PASS
coverage: 93.5% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       195.130s        coverage: 93.5% of statements in ./...

@andrewnester andrewnester added this pull request to the merge queue Sep 26, 2023
Merged via the queue into main with commit 0daa002 Sep 26, 2023
4 checks passed
@andrewnester andrewnester deleted the optional-wrapper branch September 26, 2023 14:41
andrewnester added a commit that referenced this pull request Sep 27, 2023
Bundles:
 * Enable target overrides for pipeline clusters ([#792](#792)).
 * Add support for regex patterns in template schema ([#768](#768)).
 * Make the default `databricks bundle init` template more self-explanatory ([#796](#796)).
 * Make a notebook wrapper for Python wheel tasks optional ([#797](#797)).
 * Added a warning when Python wheel wrapper needs to be used ([#807](#807)).

Internal:
 * Added `process.Background()` and `process.Forwarded()` ([#804](#804)).

Dependency updates:
 * Bump golang.org/x/term from 0.11.0 to 0.12.0 ([#798](#798)).
 * Bump github.com/hashicorp/terraform-exec from 0.18.1 to 0.19.0 ([#801](#801)).
 * Bump golang.org/x/oauth2 from 0.11.0 to 0.12.0 ([#802](#802)).
@andrewnester andrewnester mentioned this pull request Sep 27, 2023
github-merge-queue bot pushed a commit that referenced this pull request Sep 27, 2023
Bundles:
* Enable target overrides for pipeline clusters
([#792](#792)).
* Add support for regex patterns in template schema
([#768](#768)).
* Make the default `databricks bundle init` template more
self-explanatory ([#796](#796)).
* Make a notebook wrapper for Python wheel tasks optional
([#797](#797)).
* Added a warning when Python wheel wrapper needs to be used
([#807](#807)).

Internal:
* Added `process.Background()` and `process.Forwarded()`
([#804](#804)).

Dependency updates:
* Bump golang.org/x/term from 0.11.0 to 0.12.0
([#798](#798)).
* Bump github.com/hashicorp/terraform-exec from 0.18.1 to 0.19.0
([#801](#801)).
* Bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
([#802](#802)).
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
## Changes
Instead of always using notebook wrapper for Python wheel tasks, let's
make this an opt-in option.

Now by default Python wheel tasks will be deployed as is to Databricks
platform.
If notebook wrapper required (DBR < 13.1 or other configuration
differences), users can provide a following experimental setting

```
experimental:
  python_wheel_wrapper: true
```

Fixes #783,
databricks/databricks-asset-bundles-dais2023#8

## Tests
Added unit tests.

Integration tests passed for both cases

```
    helpers.go:163: [databricks stdout]: Hello from my func
    helpers.go:163: [databricks stdout]: Got arguments:
    helpers.go:163: [databricks stdout]: ['my_test_code', 'one', 'two']
    ...
Bundle remote directory is ***/.bundle/ac05d5e8-ed4b-4e34-b3f2-afa73f62b021
Deleted snapshot file at /var/folders/nt/xjv68qzs45319w4k36dhpylc0000gp/T/TestAccPythonWheelTaskDeployAndRunWithWrapper3733431114/001/.databricks/bundle/default/sync-snapshots/cac1e02f3941a97b.json
Successfully deleted files!
--- PASS: TestAccPythonWheelTaskDeployAndRunWithWrapper (214.18s)
PASS
coverage: 93.5% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       214.495s        coverage: 93.5% of statements in ./...

```

```
    helpers.go:163: [databricks stdout]: Hello from my func
    helpers.go:163: [databricks stdout]: Got arguments:
    helpers.go:163: [databricks stdout]: ['my_test_code', 'one', 'two']
    ...
Bundle remote directory is ***/.bundle/0ef67aaf-5960-4049-bf1d-dc9e29157421
Deleted snapshot file at /var/folders/nt/xjv68qzs45319w4k36dhpylc0000gp/T/TestAccPythonWheelTaskDeployAndRunWithoutWrapper2340216760/001/.databricks/bundle/default/sync-snapshots/edf0b322cee93b13.json
Successfully deleted files!
--- PASS: TestAccPythonWheelTaskDeployAndRunWithoutWrapper (192.36s)
PASS
coverage: 93.5% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       195.130s        coverage: 93.5% of statements in ./...

```
hectorcast-db pushed a commit that referenced this pull request Oct 13, 2023
Bundles:
* Enable target overrides for pipeline clusters
([#792](#792)).
* Add support for regex patterns in template schema
([#768](#768)).
* Make the default `databricks bundle init` template more
self-explanatory ([#796](#796)).
* Make a notebook wrapper for Python wheel tasks optional
([#797](#797)).
* Added a warning when Python wheel wrapper needs to be used
([#807](#807)).

Internal:
* Added `process.Background()` and `process.Forwarded()`
([#804](#804)).

Dependency updates:
* Bump golang.org/x/term from 0.11.0 to 0.12.0
([#798](#798)).
* Bump github.com/hashicorp/terraform-exec from 0.18.1 to 0.19.0
([#801](#801)).
* Bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
([#802](#802)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python wheel based tasks no longer deployed correctly
2 participants