-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add validation for single node clusters #1909
Conversation
} | ||
} | ||
|
||
func TestValidateSingleNodePipelineClustersFail(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We duplicate the test cases for pipeline clusters because they have a separate proto / Go SDK type for pipeline clusters that mirrors the spec from the cluster APIs.
Duplicating the validation code and test seemed easier and more robust than generalizing the code by converting between the two cluster structs or having a common representation.
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Test Details: go/deco-tests/11974490258 |
**New features for Databricks Asset Bundles:** This release adds support for managing Unity Catalog volumes as part of your bundle configuration. Bundles: * Add DABs support for Unity Catalog volumes ([#1762](#1762)). * Support lookup by name of notification destinations ([#1922](#1922)). * Extend "notebook not found" error to warn about missing extension ([#1920](#1920)). * Skip sync warning if no sync paths are defined ([#1926](#1926)). * Add validation for single node clusters ([#1909](#1909)). * Fix segfault in bundle summary command ([#1937](#1937)). * Add the `bundle_uuid` helper function for templates ([#1947](#1947)). * Add default value for `volume_type` for DABs ([#1952](#1952)). * Properly read Git metadata when running inside workspace ([#1945](#1945)). * Upgrade TF provider to 1.59.0 ([#1960](#1960)). Internal: * Breakout variable lookup into separate files and tests ([#1921](#1921)). * Add golangci-lint v1.62.2 ([#1953](#1953)). Dependency updates: * Bump golang.org/x/term from 0.25.0 to 0.26.0 ([#1907](#1907)). * Bump github.com/Masterminds/semver/v3 from 3.3.0 to 3.3.1 ([#1930](#1930)). * Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 ([#1932](#1932)). * Bump github.com/databricks/databricks-sdk-go from 0.51.0 to 0.52.0 ([#1931](#1931)).
**New features for Databricks Asset Bundles:** This release adds support for managing Unity Catalog volumes as part of your bundle configuration. Bundles: * Add DABs support for Unity Catalog volumes ([#1762](#1762)). * Support lookup by name of notification destinations ([#1922](#1922)). * Extend "notebook not found" error to warn about missing extension ([#1920](#1920)). * Skip sync warning if no sync paths are defined ([#1926](#1926)). * Add validation for single node clusters ([#1909](#1909)). * Fix segfault in bundle summary command ([#1937](#1937)). * Add the `bundle_uuid` helper function for templates ([#1947](#1947)). * Add default value for `volume_type` for DABs ([#1952](#1952)). * Properly read Git metadata when running inside workspace ([#1945](#1945)). * Upgrade TF provider to 1.59.0 ([#1960](#1960)). Internal: * Breakout variable lookup into separate files and tests ([#1921](#1921)). * Add golangci-lint v1.62.2 ([#1953](#1953)). Dependency updates: * Bump golang.org/x/term from 0.25.0 to 0.26.0 ([#1907](#1907)). * Bump github.com/Masterminds/semver/v3 from 3.3.0 to 3.3.1 ([#1930](#1930)). * Bump github.com/stretchr/testify from 1.9.0 to 1.10.0 ([#1932](#1932)). * Bump github.com/databricks/databricks-sdk-go from 0.51.0 to 0.52.0 ([#1931](#1931)).
Changes
This PR adds a warning validating that the configuration for a single node cluster is valid for interactive, job, job-task, and pipeline clusters.
Note: We skip the validation if a cluster policy is configured because the policy is likely to configure
spark_conf
/custom_tags
itself.Note: Terrform originally only had validation for interactive, job, and job-task clusters. This PR adding the validation for pipeline clusters as well is new.
This PR follows the same logic as we used to have in Terraform. The validation was removed from Terraform because we had no way to demote the error to a warning:
databricks/terraform-provider-databricks#4222
Background
Single-node clusters require
spark_conf
andcustom_tags
to be correctly set in the cluster definition for them to function optimally. The cluster will be created even if incorrectly configured, but its performance will not be great.For example, if both
spark_conf
andcustom_tags
are not set andnum_workers
is 0, then only the driver process will be launched on the cluster compute instance thus leading to sub-optimal utilization of available compute resources and no parallelization across worker processes when processing a spark query.Issue
This PR addresses some issues reported in #1546
Tests
Unit tests and manually.
Example output of the warning: