[Bug] package-lock.yml changes sha1-hash value after running dbt deps post package installation

[simadma]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When adding a new package to dbt using:

dbt deps --add-package <package-name>@<version>

the package-lock.yml gets updated with a new sha1. However, if one runs:

dbt deps

afterwards, the sha1 gets a new value.

Expected Behavior

When running:

dbt deps

afterwards, the sha1-hash should remain unchanged. The sha1-hash produced from running:

dbt deps --add-package <package-name>@<version>

should be consistent and sufficient.

Steps To Reproduce

Step 1: Set up dbt-core

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip && pip install dbt-core==1.8.8
dbt init dbt_project; cd dbt_project  # ignoring "No adapters available" error

Step 2: Install dbt package

dbt deps --add-package dbt-labs/dbt_utils@1.3.0
cat package-lock.yml
# packages:
#   - package: dbt-labs/dbt_utils
#     version: 1.3.0
# sha1_hash: 9c30241ae29a43cdbf99a3398f49bf74568f2d54

Step 3: Run dbt deps

dbt deps
cat package-lock.yml
# packages:
#   - package: dbt-labs/dbt_utils
#     version: 1.3.0
# sha1_hash: 226ae69cdfbc9367e2aa2c472b01f99dbce11de0

Notice the new sha1_hash.

Relevant log output

No response

Environment

- OS: Ubuntu 22.04
- Python: 3.11.1
- dbt: 1.8.8

Which database adapter are you using with dbt?

No response

Additional Context

No response

[dbeatty10]

Thanks for reporting this @simadma 👍

I was able to reproduce this issue following the instructions you provided.

When I looked at the value of package_strs here, it shows where the different hash is coming from:

9c30241ae29a43cdbf99a3398f49bf74568f2d54:

['{"install_prerelease": false, "package": "dbt-labs/dbt_utils", "unrendered": {}, "version": "1.3.0"}']

226ae69cdfbc9367e2aa2c472b01f99dbce11de0:

['{"install_prerelease": false, "package": "dbt-labs/dbt_utils", "unrendered": {"package": "dbt-labs/dbt_utils", "version": "1.3.0"}, "version": "1.3.0"}']

I didn't investigate further of why unrendered is an empty dictionary in the former and has content in the latter.

Could you share more about the effect the changing hash is having for your workflows?

[simadma]

Sure! Thank you for looking into this @dbeatty10.

Our team uses ephemeral workspaces that run the following commands at startup to ensure all dependencies are in sync with the lock files in our repositories:

poetry install
cd dbt && dbt deps

However, we noticed that the package-lock.yml file automatically changes at each startup, which seems like an unexpected behavior for a typical lock file. We considered adding it to .gitignore because of this.

The issue arises because we were only executing:

dbt deps --add-package <package-name>@<version>

before committing the new package.yml and package-lock.yml. Now we realize that if we also run dbt deps after package installation before committing the changes, the SHA1 hash will remain consistent on startup.

However, I feel that running the extra dbt deps command post-installation may not have been the intended solution to ensure a constant hash.

[dbeatty10]

Thanks for that additional information @simadma.

I'm glad you found a workaround by adding dbt deps after any commands that include --add-package.

This is probably right on the edge of what we'd consider a bug or not.

• On one hand, running dbt deps is required no matter what, and it does install all the correct packages in this scenario.
• On the other hand, the package-lock.yml file will be updated in a setup like yours and will show a file difference until it is committed (which is understandably not expected).

Agreed that the original intent in #8408 appears to me that the SHA1 hash would remain consistent regardless if another dbt deps was issued or not, so I will leave this categorized as a bug.

This looks likely to be a low-priority for us due to several available workarounds and because generating a new lock file is still "safe" -- dbt will operate without failure and install the correct packages.

Some workarouds

Here's a few separate workarounds:

1. Follow any commands that include --add-package with a regular dbt deps (no flags) to update the package-lock.yml file (like you chose to do).
2. Manually edit + commit packages into packages.yml rather than use the --add-package syntax.
3. Add package-lock.yml to your .gitignore file and remove it from source control in your git repo
4. At startup in your scenario, ignore the file differences for the package-lock.yml file after running `dbt deps

[dbeatty10]

Implementation ideas to resolve this issue

• Add the expected output of package-lock.yml to the functional test here
• Either:
  1. Figure out how to get the unrendered portion of PackageSpec populated in this scenario (ideal)
  2. Remove the unrendered portion prior to committing the hash (less ideal because there will be a one-time change of hashes for everyone).