✨ restructure deps command with package-lock.yml #6735

justbldwn · 2023-01-25T21:02:40Z

resolves #6643

Description

this PR restructures the dbt deps command, aiming to add a few new key features:

create a new package-lock.yml file that will keep record of all packages installed during the last dbt deps run
use newly created package-lock.yml as the default file to install packages from, which helps reduce install time
adds 2 new dbt deps subcommands: dbt deps lock and dbt deps add
refresh dbt_packages/ directory each time dbt deps is run (no more persisting old packages left in the directory)

the below will provide an overview of the changes, what they accomplish, and ~~some~~ a lot of detail on implementation and examples. sorry in advance for the length! 😅

`dbt deps lock`

the new lock command will take the packages in packages.yml and list them and their transitive dependencies in a new package-lock.yml file.
package versions should be resolved for each dependency, else, error thrown.
package-lock.yml will get written to the project_root alongside packages.yml.

example command:

dbt deps lock

`dbt deps add`

allows a user to add a package to their packages.yml file with configurable version and location of a given package.
added a --dry-run flag to the dbt deps add sub-command that, when False, will re-create the lock file after a new package is added to the packages.yml file. (this is setup as False by default in argparse, a user can use the flag and set to True to not persist changes down to the package-lock.yml
overwrite package info in packages.yml if a package name already exists (either explicit or fuzzy name matching)
added a quality check to ensure a version is passed in whenever the source != "local"

example command:

# add package from hub (--source arg defaults to "hub")
dbt deps add --package dbt-labs/dbt_utils --version 1.0.0

# add package from hub with semantic version
dbt deps add --package dbt-labs/snowplow --version ">=0.7.0,<0.8.0"

# add package from git
dbt deps add --package https://github.com/fivetran/dbt_amplitude --version v0.3.0 --source git 

# add package from local (--version not required for local)
dbt deps add --package /opt/dbt/redshift --source local

# add package to packages.yml WITHOUT updating package-lock.yml
dbt deps add --package dbt-labs/dbt_utils --version 1.0.0 --dry-run True

`dbt deps` 🚩(this is now changed as of 2/10/2023, changes noted in strikethrough) 🚩

~~1. largely remains unchanged in order to stay backward compatible~~

given the recent changes to migrate away from argparse -> click, i had to make changes to the dbt deps command. when working with click, i couldn't get dbt deps to be a @cli.group(), while also allowing it to run as an isolated command. i tried using invoke_without_command=True in the @cli.group() decorator, but still didn't get it to work. so for now, i changed dbt deps to be an isolated @cli.group() and added a new @deps.command("install") to handle the old dbt deps functionality (this mimics how dbt docs and dbt source are setup.

key takeaway: dbt deps has been changed to dbt deps install and should be treated as a breaking change. happy to discuss this further!

when a user runs ~~dbt deps~~ dbt deps install, it will check immediately to see if a package-lock.yml file exists in the project root directory. if not, it will create one based on the packages.yml in the project root directory.
package-lock.yml now becomes the default for installing packages.

example command:

# install packages
dbt deps install

# shows help page for dbt deps sub-commands
dbt deps

Examples:

example packages.yml:

packages:
  - package: dbt-labs/dbt_utils
    version: 0.9.2
  - git: "https://github.com/google/fhir-dbt-analytics"
    revision: 386e8430a4b9735466c90c3313ae67c31dbef58f
  - git: "https://github.com/elementary-data/dbt-data-reliability"
    revision: 0.6.11

example package-lock.yml

packages:
- package: dbt-labs/dbt_utils
  version: 0.9.2
- git: https://github.com/google/fhir-dbt-analytics
  revision: 386e8430a4b9735466c90c3313ae67c31dbef58f
- git: https://github.com/elementary-data/dbt-data-reliability
  revision: 0.6.11
- package: calogica/dbt_expectations
  version: 0.7.0
- package: yu-iskw/dbt_unittest
  version: 0.3.1
- package: calogica/dbt_date
  version: 0.6.3

⚠️ Some advice/clarification needed:

first, i believe the breaking change from dbt deps to dbt deps install should warrant discussion on this PR. notes are in the above section titled dbt deps

additionally, there were 2 notable strange yaml formatting issues i ran into with this when outputting data to yaml files:

number 1:
using yaml.dump() or yaml.safe_dump() when writing package/lock results formatted slightly differently than what's in a packages.yml and advertised on the dbt website. i tried indent formatting but nothing I tried seemed to be able to replicate like what's above in the packages.yml vs. package-lock.yml examples.

number 2:
another strange one that i couldn't figure out with yaml formatting was how to get the semantic versioning to display like it's advertised on the dbt website. for example:

online:

packages:
  - package: dbt-labs/dbt_utils
    version: 0.9.2
  - package: dbt-labs/snowplow
    version: [">=0.7.0", "<0.8.0"]

dbt deps add

packages:
- package: dbt-labs/dbt_utils
  version: 0.9.2
- package: dbt-labs/snowplow
  version:
  - '>=0.7.0'
  - <0.8.0

if i use the default_flow_style=None argument in yaml.safe_dump(), we can fix how lists are displayed in the yaml file, but it then messes up how dicts are displayed....

packages:
- {package: dbt-labs/dbt_utils, version: 0.9.2}
- package: dbt-labs/snowplow
  version: ['>=0.7.0', <0.8.0]

functionally, neither of these should impact programmatic reading of the yaml data, but it does make things a touch different than what's in the examples online/the expectations you hope for with file formatting.

Conclusion:

I think that covers nearly everything! please don't hesitate to ask me any questions on this. i'm really looking forward to your review and working on any changes that may be needed. thank you!

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

also leave dbt deps as just a new click group

change deps command to deps install

ChenyuLInx · 2023-03-27T22:51:26Z

core/dbt/cli/main.py

@@ -289,7 +289,13 @@ def debug(ctx, **kwargs):


 # dbt deps
-@cli.command("deps")
+@cli.group()


I figured out why invoke_without_command=True doesn't work for this case. Everything in click works, but in Flags we did some not ideal lookup here that made the assumption that a function name should always match the last part of input command. I will create a follow up ticket for that but you will find that if you change function name deps_install and deps_lock below to just install and lock, you should be able to run without any issue. One thing that I didn't check is whether the params at group level is correctly represented in Flags.

ChenyuLInx

Thanks @justbldwn on this amazing PR! And sorry for getting back to you late! This makes a lot of sense to me, left some comments questions to see what you think of them!

ChenyuLInx · 2023-03-27T22:54:56Z

core/dbt/config/project.py

@@ -94,13 +94,14 @@ def _load_yaml(path):
    return load_yaml_text(contents)


-def package_data_from_root(project_root):
-    package_filepath = resolve_path_from_base("packages.yml", project_root)
+def package_data_from_root(project_root, package_file_name="packages.yml"):


Looking at the update and how this is being used it seems like just more of a function to load a yaml file vs anything related to package data. Maybe consider not have the default value for package_file_name and just rename the function completely?

ChenyuLInx · 2023-03-27T22:59:57Z

core/dbt/task/deps.py

 class DepsTask(BaseTask):
    def __init__(self, args: Any, project: Project):
        move_to_nearest_project_dir(project.project_root)
        super().__init__(args=args, config=None, project=project)
        self.cli_vars = args.vars

+        if not system.path_exists(f"{self.project.project_root}/package-lock.yml"):


If ja lock file already exists, I updated my packages.yml, and run dbt deps, do we update the lock yaml somehow?

ChenyuLInx · 2023-03-27T23:01:29Z

core/dbt/task/deps.py

@@ -55,39 +96,149 @@ def track_package_install(
        )

    def run(self) -> None:
+        if system.path_exists(self.project.packages_install_path):


Do you think this could cause any surprise behavior that's different from how deps works today?

ChenyuLInx · 2023-03-27T23:06:44Z

core/dbt/task/deps.py

+            return
+
+        with downloads_directory():
+            resolved_deps = resolve_packages(packages, self.project, self.cli_vars)


Depending on where those packages lives, this might actually pull down a git repo right? Just to understand what's happening, we will pull it down, and remove it in the function above that has comment on it, and pull it down again when install? Did we always pull down git repos twice before?

ChenyuLInx · 2023-03-27T23:13:12Z

core/dbt/task/deps.py

+            fire_event(
+                DepsLockUpdating(lock_filepath=f"{self.project.project_root}/package-lock.yml")
+            )
+            LockTask(self.args, self.project).run()


Make sure I understand correctly, both deps install(when no lock file added before) and deps add will run LockTask and update lock yaml. Have you considered if it is not dry run deps add would just also install the package? I am wondering do you think it would be more natural to have deps add just install the newly added package. CC @dbeatty10

ChenyuLInx · 2023-03-27T23:31:19Z

In terms of the yaml write I found this answer works.

ChenyuLInx · 2023-06-06T23:43:59Z

@justbldwn are you still interested in working on this feature? Otherwise we can help bring it across the finish line.

justbldwn · 2023-08-01T16:48:44Z

@justbldwn are you still interested in working on this feature? Otherwise we can help bring it across the finish line.

@ChenyuLInx so sorry for the delay in responding! i may be able to help in the coming weeks, if you have someone that takes over and you need my assistance, too, i'm happy to help where possible.

ChenyuLInx · 2023-08-14T09:13:43Z

@justbldwn no worries at all! Thank you so much for the contribution! Deps has been something that didn't got much improvement over the years and we are excited to see changes to it.

I am just back to pick this up and working on getting it into dbt-core. I just merged the main branch into your branch and pushed it to branch deps_lock_by_justbldwn. I will likely working on getting everything ready to go on that branch over the next few days. Depending on how you would like to collaborate and your schedule I am sure we can figure something out. Either merge that branch to your's or just open another PR with that branch. Let me know if there's a way you would prefer to do this.

dbeatty10 · 2023-12-05T19:20:58Z

This PR was continued in #8408 and ultimately merged.

✨ adding installed_packages.json functionality

5261bfe

cla-bot bot added the cla:yes label Jan 25, 2023

✅ update test_simple_dependency_deps test

5507a0e

justbldwn mentioned this pull request Jan 25, 2023

[CT-1845] [Feature] Write a file of installed versions during dbt deps #6643

Open

3 tasks

📝 adding changelog for deps feature via changie

f0cf216

dbeatty10 mentioned this pull request Feb 1, 2023

[CT-1986] [Feature] add packages to manifest.json #6818

Closed

3 tasks

✨ restructure deps command, include lock/add

8ae24e4

justbldwn changed the title ~~✨ adding installed_packages.json functionality~~ ✨ restructure deps command with package-lock.yml Feb 10, 2023

justbldwn added 9 commits February 9, 2023 19:43

✅ add new deps event types to sample_values

94d208f

✅ fix test_simple_dependency_deps test

948d400

🔀 merging changes with latest on main

93f72f4

🐛 attempting to fix cli commands

a3df280

🐛 convert dbt deps to dbt deps install

0e01c2e

also leave dbt deps as just a new click group

✅ update test_command_mutually_exclusive_option

9995193

change deps command to deps install

✅ update functional tests from deps > deps install

e182821

✅ change missing deps to deps install

5615910

✅ convert adapter tests to deps install from deps

5eb279f

justbldwn marked this pull request as ready for review February 10, 2023 17:14

justbldwn requested a review from a team as a code owner February 10, 2023 17:14

justbldwn requested a review from a team February 10, 2023 17:14

justbldwn requested review from a team as code owners February 10, 2023 17:14

justbldwn requested review from nathaniel-may, ChenyuLInx and peterallenwebb February 10, 2023 17:14

ChenyuLInx reviewed Mar 27, 2023

View reviewed changes

ChenyuLInx mentioned this pull request Aug 15, 2023

Deps lock by justbldwn #8408

Merged

14 tasks

dbeatty10 mentioned this pull request Nov 14, 2023

[CT-3368] [Regression] warn-unpinned is ignored (dbt deps, packages.yml) #9050

Closed

2 tasks

dbeatty10 closed this Dec 5, 2023

dbeatty10 mentioned this pull request Mar 22, 2024

[Core] Default behavior of dbt deps changed in v1.7 dbt-labs/docs.getdbt.com#5140

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ restructure deps command with package-lock.yml #6735

✨ restructure deps command with package-lock.yml #6735

justbldwn commented Jan 25, 2023 •

edited

Loading

ChenyuLInx Mar 27, 2023

ChenyuLInx left a comment

ChenyuLInx Mar 27, 2023

ChenyuLInx Mar 27, 2023

ChenyuLInx Mar 27, 2023

ChenyuLInx Mar 27, 2023

ChenyuLInx Mar 27, 2023

ChenyuLInx commented Mar 27, 2023

ChenyuLInx commented Jun 6, 2023

justbldwn commented Aug 1, 2023 •

edited

Loading

ChenyuLInx commented Aug 14, 2023

dbeatty10 commented Dec 5, 2023

✨ restructure deps command with package-lock.yml #6735

✨ restructure deps command with package-lock.yml #6735

Conversation

justbldwn commented Jan 25, 2023 • edited Loading

Description

dbt deps lock

dbt deps add

dbt deps 🚩(this is now changed as of 2/10/2023, changes noted in strikethrough) 🚩

Examples:

⚠️ Some advice/clarification needed:

Conclusion:

Checklist

ChenyuLInx Mar 27, 2023

Choose a reason for hiding this comment

ChenyuLInx left a comment

Choose a reason for hiding this comment

ChenyuLInx Mar 27, 2023

Choose a reason for hiding this comment

ChenyuLInx Mar 27, 2023

Choose a reason for hiding this comment

ChenyuLInx Mar 27, 2023

Choose a reason for hiding this comment

ChenyuLInx Mar 27, 2023

Choose a reason for hiding this comment

ChenyuLInx Mar 27, 2023

Choose a reason for hiding this comment

ChenyuLInx commented Mar 27, 2023

ChenyuLInx commented Jun 6, 2023

justbldwn commented Aug 1, 2023 • edited Loading

ChenyuLInx commented Aug 14, 2023

dbeatty10 commented Dec 5, 2023

justbldwn commented Jan 25, 2023 •

edited

Loading

`dbt deps lock`

`dbt deps add`

`dbt deps` 🚩(this is now changed as of 2/10/2023, changes noted in strikethrough) 🚩

justbldwn commented Aug 1, 2023 •

edited

Loading