1.4 - terraform init does not populate .terraform.lock.hcl with hashes for all platforms

[twbecker]

Terraform Version

Terraform v1.4.0
on darwin_amd64

Terraform Configuration Files

N/A

Debug Output

N/A

Expected Behavior

According to a comment in #28041:

In Terraform v1.4 something similar to this behavior will now happen automatically: when populating the lock file entry for the first time, terraform init will access the origin registry to obtain the full set of checksums for that provider and write them all into the lock file just as if the plugin cache directory were not present at all. On future runs, with the lock file already populated, Terraform will then use the plugin cache directory as long as its contents match a checksum in the lock file.

Actual Behavior

This doesn't seem to be true. Running terraform init still only records the h1 checksums for the current platform. This is true regardless of whether a lockfile currently exists or not.

Steps to Reproduce

terraform init Notice any new lockfiles generated only include h1 hashes for the current platform. If there is an existing lockfile, and you -upgraded, hashes for other platforms are dropped.

Additional Context

From what I can see, Terraform 1.4 behaves identically to 1.3 in this regard, and my main impetus for upgrading was the quoted comment above. My team uses Terraform on multiple platforms with the plugin cache and having to do a terraform providers lock after every provider change is terribly error prone.

References

• terraform init and lock file platform #28041
• terraform init does not populate .terraform.lock.hcl with hashes for all platforms #29794

[crw]

Thanks for this report!

[apparentlymart]

Hi @twbecker! Thanks for reporting this.

Can you share some more information about what you tried? In particular it would be helpful to see the result of running terraform init with the TF_LOG=trace environment variable set -- following the instructions under the "Debug Output" heading in the bug report form -- to see what Terraform's provider installer is doing and why it wasn't able to determine the full set of checksums despite the improvements in Terraform v1.4.

Thanks again!

[twbecker]

Thanks for the response. Here's a gist with the (redacted) trace output: https://gist.github.com/twbecker/46847d3521b20902d7cdf51328e4eb8d

[samcrop]

I'm having a similar problem with Terraform version 1.4.0 on linux_amd64 with the terragrunt wrapper and shared cache, only downgrading to 1.3.9 fixed the issue.

I would constantly receive the error:

Error: Required plugins are not installed. The installed provider plugins are not consistent with the packages selected in the dependency lock file: registry.terraform.io/hashicorp/aws: the cached package for registry.terraform.io/hashicorp/aws 4.58.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file. Terraform uses external plugins to integrate with a variety of different infrastructure services. To download the plugins required for this configuration, run: terraform init

Running terragrunt init -upgrade would be successful but any other command would result in the error above, even terraform providers lock ........

[apparentlymart]

Thanks for sharing that, @twbecker.

That trace contains the steps I would expect to see as a result of the v1.4 change:

2023-03-10T07:33:57.410-0500 [DEBUG] GET https://releases.hashicorp.com/terraform-provider-aws/4.57.1/terraform-provider-aws_4.57.1_SHA256SUMS
2023-03-10T07:33:57.410-0500 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-aws/4.57.1/terraform-provider-aws_4.57.1_SHA256SUMS
2023-03-10T07:33:57.585-0500 [DEBUG] GET https://releases.hashicorp.com/terraform-provider-aws/4.57.1/terraform-provider-aws_4.57.1_SHA256SUMS.72D7468F.sig
2023-03-10T07:33:57.585-0500 [TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-aws/4.57.1/terraform-provider-aws_4.57.1_SHA256SUMS.72D7468F.sig
...
2023-03-10T07:34:01.232-0500 [DEBUG] Provider signed by 34365D9472D7468F HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>

This shows Terraform CLI fetching the official signed signatures for this provider and verifying that the signature is valid, so indeed there doesn't seem to be any reason why the generated lock file shouldn't include all of the checksums from that terraform-provider-aws_4.57.1_SHA256SUMS file.

Just so I can get a sense of all of what you're seeing, can you also share the content of the generated .terraform.lock.hcl file? I realize this probably seems silly cause you've already told me there is just one checksum in there, but I'd like to see it in case something in there I'm not thinking to ask about can give me an idea about what might be causing this so that I can try to reproduce it.

Thanks!

[apparentlymart]

Hi @samcrop,

What you are seeing seems like it might be a little different than what this report was covering. Could you open a new bug report issue and share all of the information requested in the bug report form and then I'll try to see what these two issues have in common. It would also help to share in your new issue the contents of the generated .terraform.lock.hcl file, even though that isn't something the bug report form directly requests.

[twbecker]

Thanks @apparentlymart. Contents of .terraform.lock.hcl:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "4.57.1"
  constraints = "4.57.1"
  hashes = [
    "h1:lyiJFRB0nKUS/OkS8OSqxAYZuLWVBIPpN67VGoDyYak=",
    "zh:44200c213ddb138df80d2a5ad86c2ebadbb5fd1d08cd7e4fc56ec6dca927659b",
    "zh:469e6fe6a9e99e60cb168d32f05e2e9a83cf161f39160d075ff96f7674c510e1",
    "zh:6110ba2c15a2268652ec9ea3797dd0216de84ece428055c49eaf9caa2be1ed62",
    "zh:62ed7348acca44f64fc087e879e01cfa4e084c7600cc91e8bb7683f8065a9c79",
    "zh:7a80e6fa9b35be178bb566093f7984dd6ffb7ad9d40b9dd5d5907f054f0c3e60",
    "zh:8793043c8575a598c1a7cbefcb65ee1776b0061eba719098e552a3adc88f3090",
    "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
    "zh:a777a0082114e273b7b3eb14095a3f6f6e703c1aff61ffb1f0846bb869e6dfc7",
    "zh:b060c3b2973097f2087a98ac6aad7c9c89fe80f7cf3027019049feafc3f8305b",
    "zh:e7035e74563f4486848ea1feb60852175353790bc374e0e97e241a88dc0908f7",
    "zh:eaaa8e9eba09ada41e13116d53d4baece04fead8fcf3eab68cca3a67ed738e18",
    "zh:ec52d8f95a84fad8fe1aae169c89d0c54d5401f75caae0869ad8182c6b6db65b",
    "zh:f0e33174025b1b57ecfbdd09f2a59c2559ee94d7681e5ae09079e2822ec54ecf",
    "zh:f69790a21380e5aab9303a252564737333e1e95b5d25567681630e49b17e3ec7",
    "zh:ff6053942c40a99904bd407f3c082c1fa8f927ecce0374566eb7e8ee8145e582",
  ]
}

My understanding is that the h1: hashes are the ones that matter.

[apparentlymart]

Hi @twbecker,

The zh: hashes are the ones that provider developers sign, so this lock file seems to have a full complement of checksums required to authenticate packages coming from Terraform Registry, and that's the scope of what the changes in v1.4 were intended to achieve. zh: and h1: checksums are both equally valid for verifying the zip files that provider plugins are distributed in; the h1: checksum format additionally supports verifying already-extracted packages such as those in the plugin cache.

Terraform Registry cannot provide h1: hashes, so Terraform CLI still calculates those ones locally and that will remain the case until Terraform Registry is able to provide signed checksums using the h1: scheme. However, the automatic checksum upgrade behavior will generate h1: checksums for packages matching the zh: ones automatically now, whereas before folks who were using the provider plugin cache would get stuck with a lock file that didn't contain enough information to do that automatically.

Given that, it seems like the behavior you've observed is the intended behavior and that you had an expectation beyond what this change was intended to address. Making Terraform Registry be able to return new-style checksums is the blocker for the extra behavior you wanted here, but that's not something we'll be able to improve only through changes in this repository. (The main challenge is actually not updating the registry itself but rather to change the provider publishing workflows to generate hashes in the new format and include them with releases; each provider codebase has its own publication workflow, so it unfortunately requires every single provider codebase to change before Terraform would be able to record h1: checksums .)

Given that, I think this might be a "working as designed" situation and the remaining work is already represented by #27264, where I wrote a more verbose version of the explanation above about the external factors that are blocking a complete solution.

Given that, I'm going to close this just to keep the discussion consolidated over in that existing issue. That issue is the one that will see updates if the situation with the registry protocol and provider signing improves in future. Thanks!

[twbecker]

Thanks for the response. While I appreciate the level of technical detail you provided, what I'm really trying to determine is simply whether or not the lock file I posted will work for users on other platforms. I should point out that although we use the plugin cache, we only do so locally, not across machines.

[twbecker]

After reading the link provided and testing I see that although the generated lock file does work on other platforms, the checksum upgrade behavior means that using it on another platform will result in modifications to the lock file that then have to be checked in. So unfortunately the need to run terraform providers lock after provider changes remains.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.