Failed to create single node cluster using databricks asset bundle on databricks CLI v0.221.0

[iswariyam]

Describe the issue

I am using Databricks asset bundles to deploy my Databricks jobs which consists of dbt tasks. When I try to deploy the jobs using Databricks asset bundles, the deploy step fails with the error Error: cannot create job: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size. My jobs use a single node cluster with num_workers set to 0 similar to the configuration over here. This issue only occurs in the recently released version v0.221.0 of the Databricks CLI.

Configuration

Error occurs in databricks-cli version 0.221.0 but not in 0.219.0

Steps to reproduce the behavior

1. Deploy the dbt-sql asset bundle using the example here. Note, a single node cluster is used with num_workers 0.
2. Run databricks bundle deploy
3. Terraform apply fails with error Error: cannot create job: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size

Expected Behavior

Databricks asset bundle deployment should work for single node clusters with num_workers 0.

Actual Behavior

Creation of Databricks jobs with single node cluster (num_workers 0) fails when using v0.221.0 of the databricks CLI.

OS and CLI version

• databricks cli version v0.221.0
• OS - Mac OS Sonoma 14.5 as well as Ubuntu 22.04

Is this a regression?

Yes, I did not experience this issue on databricks-cli version v0.219.0

Debug Logs

Error: cannot update job: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size

  with databricks_job.historical_customer_data_ingestion,
  on bundle.tf.json line 150, in resource.databricks_job.historical_customer_data_ingestion:
 150:       }

 pid=50843 mutator=seq
Error: terraform apply: exit status 1

[pietern]

Thanks for reporting! We identified the same issue earlier today and are working on a fix. We're targeting a patch release of the CLI with a patched TF version tomorrow. Are you able to use v0.220.0 in the mean time?

[iswariyam]

Thanks for reporting! We identified the same issue earlier today and are working on a fix. We're targeting a patch release of the CLI with a patched TF version tomorrow. Are you able to use v0.220.0 in the mean time?

Yes thanks. v0.220.0 works for the time being.

[DaveRuijter]

I'm running into the same issue. I don't want to hijack this thread, but I need a little help on how to do install a specific version.
Currently I'm doing this:
brew tap databricks/tap brew install databricks
How do I specify to pick version 0.220.0?

[iswariyam]

I'm running into the same issue. I don't want to hijack this thread, but I need a little help on how to do install a specific version. Currently I'm doing this: brew tap databricks/tap brew install databricks How do I specify to pick version 0.220.0?

@DaveRuijter I installed the previous version v0.220.0 using curl as follows

curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/<version-tag>/install.sh | sh

<version-tag> can be substituted with one of the tags from the databricks/setup-cli repository. Alternatively you can use main for the <version-tag> as given in the Databricks documentation to always install the latest version of the CLI.

I am not expert, but I guess you can also do something similar with Homebrew as follows

brew tap-new myuser/mytap
brew extract --version=0.220.0 databricks myuser/mytap
brew install myuser/mytap/databricks@0.220.0

[pietern]

@iswariyam Many thanks for the Homebrew instructions. I wasn't aware it was possible to extract a formula at a specific version. I have updated the README at https://github.com/databricks/homebrew-tap/blob/main/README.md to include this.