Add docs databricks asset bundles #3744

erwinpaillacan · 2024-03-26T19:02:07Z

Description

This pull request was initiated to assist in establishing a project utilizing asset bundles on Databricks, as the use of DBX is deprecated and no longer recommended.

https://www.databricks.com/blog/announcing-general-availability-databricks-asset-bundles

Development notes

make build-docs

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

erwinpaillacan · 2024-03-26T19:06:05Z

@cilopezs also is helping!

astrojuanlu · 2024-03-26T20:42:44Z

Thank you @erwinpaillacan and @cilopezs for this PR!

This addresses #3360 in part. In your opinion, do you think it still makes sense to keep the DBX docs around? I was thinking that we should remove them, and replace them by what you did here.

erwinpaillacan · 2024-03-26T21:02:55Z

Thank you @erwinpaillacan and @cilopezs for this PR!

This addresses #3360 in part. In your opinion, do you think it still makes sense to keep the DBX docs around? I was thinking that we should remove them, and replace them by what you did here.

Yes, in our opinion we could update the page https://docs.kedro.org/en/stable/deployment/databricks/databricks_ide_development_workflow.html with databricks connect, which for some time was deprecated but now it is live again and being recommended for development https://docs.databricks.com/en/dev-tools/databricks-connect/python/index.html

and for deployment we can rely on asset bundle which is the main purpose.

I think we need to update the decision plot, right?

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

erwinpaillacan · 2024-03-27T01:28:19Z

Heads up! @astrojuanlu @cilopezs

Development workflow updated to Databricks Connect: https://docs.databricks.com/en/dev-tools/databricks-connect/python/index.html
Deployment workflow asset bundles: https://docs.databricks.com/en/dev-tools/bundles/index.html

We are removing dbx

docs/source/deployment/databricks/databricks_deployment_workflow.md

datajoely

This is amazing @erwinpaillacan thank you for the hard work, only challenge is if we should make the hook part of the addon/starter and not require a manual change

docs/source/deployment/databricks/databricks_ide_development_workflow.md

astrojuanlu · 2024-03-27T10:54:40Z

The docs errors are legitimate, please address them so RTD can render the new docs 👍🏽

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

astrojuanlu · 2024-03-27T17:37:06Z

Rendered docs for reviewers:

astrojuanlu

Hi folks! @erwinpaillacan I understand that there aren't really any major outstanding comments here, and that #3744 (comment) can be addressed as a separate PR. Is that right?

Could you have a look at the vale check, which flagged some spelling and style errors? And hopefully we can get this merged.

docs/source/deployment/databricks/databricks_deployment_workflow.md

…ow.md Co-authored-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Signed-off-by: erwinpaillacan <43705290+erwinpaillacan@users.noreply.github.com>

astrojuanlu

I think this is a clear improvement over what we have now 💯 and just needs some style fixes reported by Vale for all the checks to pass. Thanks @erwinpaillacan for your patience!

astrojuanlu · 2024-06-04T15:30:02Z

Erwin confirmed internally that this PR will need an update to work on Kedro 0.19 👍🏼

Stale

ankatiyar · 2024-06-11T11:00:47Z

Hey @erwinpaillacan, can I help you update this PR to work with Kedro 0.19.x?

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

…-asset-bundles-deployment Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

erwinpaillacan · 2024-06-20T15:26:01Z

@astrojuanlu
Just tested today with kedro 0.19.6 all working good, except that I needed to change manually the kedro dataset dependencies

slack

JenspederM · 2024-06-21T11:18:26Z

Hey!

I'm a bit late to the party, but I just wanted to let you know that I have previously made a databricks bundle template to illustrate how one could get started with Kedro on databricks.

I'm in the process of converting the logic introduced in the template into a kedro plugin - see more here.

I think the plugin would be very helpful as it makes it easier to deploy existing projects to Databricks, whereas both the template made by me or the databricks-iris starter are only relevant for new projects.

Please note that the plugin is still in early development, so if you have any suggestions to align with your vision please let me know!

erwinpaillacan · 2024-06-21T12:47:55Z

Hey!

I'm a bit late to the party, but I just wanted to let you know that I have previously made a databricks bundle template to illustrate how one could get started with Kedro on databricks.

I'm in the process of converting the logic introduced in the template into a kedro plugin - see more here.

I think the plugin would be very helpful as it makes it easier to deploy existing projects to Databricks, whereas both the template made by me or the databricks-iris starter are only relevant for new projects.

Please note that the plugin is still in early development, so if you have any suggestions to align with your vision please let me know!

Looks great!!
Just one question: is this plugin trying to maps n nodes to n tasks? or n pipelines to n task

JenspederM · 2024-06-21T13:12:21Z

@erwinpaillacan it maps pipelines to workflows with nodes as tasks.

That is to say, I did my best to mimic the view of Kedro-viz in the workflow tab of the Databricks UI

JenspederM · 2024-06-25T09:28:55Z

I just updated the readme to shed some light on the functionality that I'm intending to implement. I say intend as the 'deploy' command isn't ready yet.

I also published it as 'kedro-databricks-dev' as the other name is already taken by an empty project. I will reach out to the author of the other project so that we can hopefully get a sensible name for the package 😊

astrojuanlu · 2024-06-25T09:34:18Z

I also published it as 'kedro-databricks-dev' as the other name is already taken by an empty project. I will reach out to the author of the other project so that we can hopefully get a sensible name for the package 😊

cc @em-pe :)

felipemonroy · 2024-07-02T15:14:02Z

docs/source/deployment/databricks/databricks_deployment_workflow.md

+            package_name: iris #<package_name>
+            entry_point: databricks_run
+            named_parameters:
+              --conf-source: /Workspace${workspace.file_path}/conf


What about passing the project-path instead and the change the working directory in the databricks_run.py script? I do not know if this a good option but sometimes I have more files that I need in the project path, like sql files that are not reachable in databricks unless I change the working directory or I specify the full path.

felipemonroy · 2024-07-02T15:30:03Z

docs/source/deployment/databricks/databricks_deployment_workflow.md

+            named_parameters:
+              --conf-source: /Workspace${workspace.file_path}/conf
+              --package-name: iris #<package_name>
+              --env: ${var.environment}


I do not know why it only worked for me without the two -- at the start

felipemonroy · 2024-07-06T03:52:36Z

docs/source/deployment/databricks/databricks_deployment_workflow.md

+          new_cluster:
+              #Azure nodes
+              node_type_id: Standard_DS3_v2
+              spark_version: 14.3.x-scala2.12


There are some issues when specifying a cluster with no workers, which was fixed in cli version v.0.221.1. I was able to create a cluster and run spark with no issues following the configuration from here. See details of the problem here and here.

astrojuanlu · 2024-07-11T08:33:36Z

I have to say I did some experiments with @JenspederM using the databricks-iris starter and it mostly worked! Maybe we could instruct users to use that instead of writing the bundle configs by hand.

Beyond this point, I leave it on the hands of @ankatiyar, who will be looking at this soon 😄

em-pe · 2024-07-11T09:07:25Z

I also published it as 'kedro-databricks-dev' as the other name is already taken by an empty project. I will reach out to the author of the other project so that we can hopefully get a sensible name for the package

@JenspederM I'm happy to pass kedro-databricks to you, just you let me know your pypi username.

JenspederM · 2024-07-12T09:02:00Z

I also published it as 'kedro-databricks-dev' as the other name is already taken by an empty project. I will reach out to the author of the other project so that we can hopefully get a sensible name for the package

@JenspederM I'm happy to pass kedro-databricks to you, just you let me know your pypi username.

I see you found it without my help. But thank you for transferring the project, @em-pe!

I have now published the first release to kedro-databricks. This release solved the most obvious issues found by @astrojuanlu. I'm just finishing up the databricks_run script after which all immediate issues should have been addressed.

I will make an announcement on Slack as soon as I have a working example with the databricks-iris starter :)

Add docs asset bundles

3b7b814

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

erwinpaillacan requested review from yetudada and astrojuanlu as code owners March 26, 2024 19:02

Fix cluster variable

b927353

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

erwinpaillacan changed the title ~~Add docs asset bundles~~ Add docs databricks asset bundles Mar 26, 2024

erwinpaillacan added 2 commits March 26, 2024 18:16

Fix word for vale CI

a86029f

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

Add databricks connect and remove dbx

ec5b0e6

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

astrojuanlu reviewed Mar 27, 2024

View reviewed changes

docs/source/deployment/databricks/databricks_deployment_workflow.md Outdated Show resolved Hide resolved

datajoely reviewed Mar 27, 2024

View reviewed changes

docs/source/deployment/databricks/databricks_ide_development_workflow.md Show resolved Hide resolved

astrojuanlu reviewed Mar 27, 2024

View reviewed changes

docs/source/deployment/databricks/databricks_ide_development_workflow.md Outdated Show resolved Hide resolved

astrojuanlu reviewed Mar 27, 2024

View reviewed changes

docs/source/deployment/databricks/databricks_ide_development_workflow.md Outdated Show resolved Hide resolved

astrojuanlu reviewed Mar 27, 2024

View reviewed changes

docs/source/deployment/databricks/databricks_ide_development_workflow.md Outdated Show resolved Hide resolved

erwinpaillacan added 4 commits March 27, 2024 11:02

Solve pr comments

5986131

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

Fix job name in bundle run

b7f39e0

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

Add jobs clusters to asset bundles

11c8372

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

Update databricks flow chart

a174550

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

Merge branch 'main' into docs/add-asset-bundles-deployment

3e8d6c3

astrojuanlu reviewed May 6, 2024

View reviewed changes

docs/source/deployment/databricks/databricks_deployment_workflow.md Outdated Show resolved Hide resolved

Update docs/source/deployment/databricks/databricks_deployment_workfl…

af8b37f

…ow.md Co-authored-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Signed-off-by: erwinpaillacan <43705290+erwinpaillacan@users.noreply.github.com>

astrojuanlu previously approved these changes May 17, 2024

View reviewed changes

Update docs with better commands

95e604a

Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

Merge commit 'af8b37fbaaec770fb1fdee406f8007df5c4f978b' into docs/add…

ae623fb

…-asset-bundles-deployment Signed-off-by: erwinpaillacan <erwin_paillacan@mckinsey.com>

felipemonroy reviewed Jul 2, 2024

View reviewed changes

felipemonroy reviewed Jul 6, 2024

View reviewed changes

merelcht mentioned this pull request Jul 8, 2024

Close all (or as many as possible) open PRs! #3996

Closed

9 tasks

erwinpaillacan closed this Jul 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docs databricks asset bundles #3744

Add docs databricks asset bundles #3744

erwinpaillacan commented Mar 26, 2024 •

edited

Loading

erwinpaillacan commented Mar 26, 2024 •

edited

Loading

astrojuanlu commented Mar 26, 2024

erwinpaillacan commented Mar 26, 2024

erwinpaillacan commented Mar 27, 2024

datajoely left a comment

astrojuanlu commented Mar 27, 2024

astrojuanlu commented Mar 27, 2024

astrojuanlu left a comment

astrojuanlu left a comment

astrojuanlu commented Jun 4, 2024

ankatiyar commented Jun 11, 2024

erwinpaillacan commented Jun 20, 2024

JenspederM commented Jun 21, 2024

erwinpaillacan commented Jun 21, 2024

JenspederM commented Jun 21, 2024 •

edited

Loading

JenspederM commented Jun 25, 2024

astrojuanlu commented Jun 25, 2024

felipemonroy Jul 2, 2024

felipemonroy Jul 2, 2024

felipemonroy Jul 6, 2024

astrojuanlu commented Jul 11, 2024

em-pe commented Jul 11, 2024

JenspederM commented Jul 12, 2024

Add docs databricks asset bundles #3744

Add docs databricks asset bundles #3744

Conversation

erwinpaillacan commented Mar 26, 2024 • edited Loading

Description

Development notes

Developer Certificate of Origin

Checklist

erwinpaillacan commented Mar 26, 2024 • edited Loading

astrojuanlu commented Mar 26, 2024

erwinpaillacan commented Mar 26, 2024

erwinpaillacan commented Mar 27, 2024

datajoely left a comment

Choose a reason for hiding this comment

astrojuanlu commented Mar 27, 2024

astrojuanlu commented Mar 27, 2024

astrojuanlu left a comment

Choose a reason for hiding this comment

astrojuanlu left a comment

Choose a reason for hiding this comment

astrojuanlu commented Jun 4, 2024

ankatiyar commented Jun 11, 2024

erwinpaillacan commented Jun 20, 2024

JenspederM commented Jun 21, 2024

erwinpaillacan commented Jun 21, 2024

JenspederM commented Jun 21, 2024 • edited Loading

JenspederM commented Jun 25, 2024

astrojuanlu commented Jun 25, 2024

felipemonroy Jul 2, 2024

Choose a reason for hiding this comment

felipemonroy Jul 2, 2024

Choose a reason for hiding this comment

felipemonroy Jul 6, 2024

Choose a reason for hiding this comment

astrojuanlu commented Jul 11, 2024

em-pe commented Jul 11, 2024

JenspederM commented Jul 12, 2024

erwinpaillacan commented Mar 26, 2024 •

edited

Loading

erwinpaillacan commented Mar 26, 2024 •

edited

Loading

JenspederM commented Jun 21, 2024 •

edited

Loading