Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Add a new flag terraform init -lockfile=readonly to suppress updating .terraform.lock.hcl #27506

Closed
minamijoyo opened this issue Jan 14, 2021 · 5 comments · Fixed by #27630
Labels
enhancement new new issue not yet triaged

Comments

@minamijoyo
Copy link
Contributor

This feature request was originally proposed in #27264 (comment) . To clarify the scope, I'll split it to a new issue.

Current Terraform Version

Terraform v0.14.4

Use-cases

I have 200+ root modules and I'm automating a provider version up workflow in CI with tfupdate, which updates all version constraints in Terraform configurations recursively. My laptop is macOS and CI is Linux, so I want to pre-populate hash values for all platforms I need in the workflow to avoid a checksum mismatch error.

I'm looking for an efficient way to maintain .terraform.lock.hcl for multiple root modules and platforms environments.

To use it on multiple platforms, a lock file should be generated in advance with the terraform providers lock command. In addition, I use a local filesystem mirror and cache to avoid redundant duplicate downloads. In such case, only h1 hashes are recorded in .terraform.lock.hcl.

However, the state of the mirror and cache depends on environments in which the terraform command is executed and the current implementation of the terraform init command will add zh hashes if neither the mirror nor the cache exists.

Since .terraform.lock.hcl is intended to commit to a VCS such as Git, the unnecessary change is not desirable. It would be great if we could suppress the lock file change.

For more details, see #27264

Attempted Solutions

  1. Save the following configuration as main.tf:
$ cat main.tf
terraform {
  required_version = "0.14.4"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "3.23.0"
    }
  }
}
  1. Create a local filesystem mirror
$ FS_MIRROR="/tmp/terraform.d/plugins"
$ terraform providers mirror -platform=linux_amd64 -platform=darwin_amd64 "${FS_MIRROR}"
- Mirroring hashicorp/aws...
  - Selected v3.23.0 to meet constraints 3.23.0
  - Downloading package for linux_amd64...
  - Package authenticated: signed by HashiCorp
  - Downloading package for darwin_amd64...
  - Package authenticated: signed by HashiCorp
  1. Generate a lock file
$ terraform providers lock -fs-mirror="${FS_MIRROR}" -platform=linux_amd64 -platform=darwin_amd64
- Fetching hashicorp/aws 3.23.0 for linux_amd64...
- Obtained hashicorp/aws checksums for linux_amd64 (unauthenticated)
- Fetching hashicorp/aws 3.23.0 for darwin_amd64...
- Obtained hashicorp/aws checksums for darwin_amd64 (unauthenticated)

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

The lock file contains only h1 hashes.

$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "3.23.0"
  constraints = "3.23.0"
  hashes = [
    "h1:GugGr7igctZkUUt0im9b0CbdinTRxb4dNXvmGuN2gZ8=",
    "h1:tSznQxPJvolDnmqqaTK9SsJ0bluTws7OAWcnc1t0ABs=",
  ]
}
  1. Run terraform init
$ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v3.23.0...
- Installed hashicorp/aws v3.23.0 (signed by HashiCorp)

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The zh hashes were added to the lock file.

$ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "3.23.0"
  constraints = "3.23.0"
  hashes = [
    "h1:GugGr7igctZkUUt0im9b0CbdinTRxb4dNXvmGuN2gZ8=",
    "h1:tSznQxPJvolDnmqqaTK9SsJ0bluTws7OAWcnc1t0ABs=",
    "zh:30b0733027c00472618da998bc77967c692e238ae117c07e046fdd7336b83fa3",
    "zh:3677550a8bef8e01c67cb615407dc8a69d32f4e36017033cd6f71a831c99d5de",
    "zh:3c2fb4c14bfd43cf20ee25d0068ce09f1d48758408b8f1c88a096cea243612b3",
    "zh:5577543322003693c4fe24a69ed0d47e58f867426fd704fac94cf5c16d3d6153",
    "zh:6771f09d76ad01ffc04baa3bce7a3eed09f6a8a949274ffbd9d11756a58a4329",
    "zh:7a57b79d304d17cf52ee3ddce91679f6b4289c5bdda2e31b763bf7d512e542d9",
    "zh:815fb027e17bfe754b05367d20bd0694726a95a99b81e8d939ddd44e2b1f05a9",
    "zh:a3d67db5ec0f4e9750eb19676a9a1aff36b0721e276a4ba789f42b991bf5951c",
    "zh:cd67ff33860ad578172c19412ce608ba818e7590083197df2b793f870d6f50a3",
    "zh:fbe0835055d1260fb77ad19a32a8726248ba7ac187f6c463ded90737b4cea8e6",
  ]
}

Of course, we can discard the change by using git checkout .terraform.lock.hcl, but obviously it's not the best solution.

Proposal

Add a new flag terraform init -lockfile=readonly to suppress updating .terraform.lock.hcl.

See the comment below for detailed behavior considerations
#27264 (comment)

References

This feature request was originally proposed as a partial workaround for #27264 (comment)
It was inspired by the discussion in #27241 (comment)
So, it may be also useful for the read-only filesystem.

@minamijoyo minamijoyo added enhancement new new issue not yet triaged labels Jan 14, 2021
minamijoyo added a commit to minamijoyo/terraform that referenced this issue Jan 29, 2021
Fixes hashicorp#27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in hashicorp#27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.
@wyardley
Copy link

wyardley commented Feb 1, 2021

We have a similar case, where we use https://github.com/renovatebot/renovate/ to update providers (it may eventually update the lockfiles, but for now, we .gitignore them). Since we're already pinning to specific versions of the provider, terraform, etc., the lockfile is not very useful in this case, and it means that terraform update will now bail unless you use the -uprade flag.

Would a flag to ignore the lockfile when terraform init is run, and / or to avoid creating (vs updating) a .terraform.lock.hcl file) be a separate feature request? I looked to see if there was an env var / flag that did this already, but did not find one

minamijoyo added a commit to minamijoyo/terraform that referenced this issue Feb 2, 2021
Fixes hashicorp#27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in hashicorp#27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.
@iniinikoski
Copy link

We have a similar case, where we use https://github.com/renovatebot/renovate/ to update providers (it may eventually update the lockfiles, but for now, we .gitignore them). Since we're already pinning to specific versions of the provider, terraform, etc., the lockfile is not very useful in this case, and it means that terraform update will now bail unless you use the -uprade flag.

Would a flag to ignore the lockfile when terraform init is run, and / or to avoid creating (vs updating) a .terraform.lock.hcl file) be a separate feature request? I looked to see if there was an env var / flag that did this already, but did not find one

We would also like to not use the lockfile - I also don't see why we should complicate anything as we use absolute locking already as Hashicorp strongly recommends. And, to make this work also with Terraform Cloud, it'd be great to ignore the new behaviour with an environment variable.

@richardj-bsquare
Copy link

We also .gitignore the .terraform.lock.hcl file, as we have CI/CD systems using linux/amd64 arch, but engineers using darwin/amd64 and even Windows to do local plans during development. The hashes are what bite us in this respect. Everything is locked down by fairly tight versioning anyway. This grates:

Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future.

... Because it encourages engineers to commit the file and break the deployments.

@wyardley
Copy link

wyardley commented Feb 4, 2021

One other point (could make a new feature request) is that requiring terraform init -upgrade to be run when a version is requested via ==x.y.z is really irritating. It seems to me that it should be reasonable to continue updating the version without being explicitly requested when an exact version is being requested.

I think, across the board, this feature was rolled out a bit hastily, and seems to have been (to the point of the previous post) a bit over-sold.

minamijoyo added a commit to minamijoyo/terraform that referenced this issue Mar 8, 2021
Fixes hashicorp#27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in hashicorp#27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.
alisdair added a commit that referenced this issue Mar 9, 2021
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
alisdair added a commit that referenced this issue Mar 9, 2021
Fixes #27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in #27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
minamijoyo added a commit to minamijoyo/terraform that referenced this issue Mar 10, 2021
Fixes hashicorp#27506

Add a new flag `-lockfile=readonly` to `terraform init`.
It would be useful to allow us to suppress dependency lockfile changes
explicitly.

The type of the `-lockfile` flag is string rather than bool, leaving
room for future extensions to other behavior variants.

The readonly mode suppresses lockfile changes, but should verify
checksums against the information already recorded. It should conflict
with the `-upgrade` flag.

Note: In the original use-case described in hashicorp#27506, I would like to
suppress adding zh hashes, but a test code here suppresses adding h1
hashes because it's easy for testing.

Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>
@ghost
Copy link

ghost commented Apr 9, 2021

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked as resolved and limited conversation to collaborators Apr 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement new new issue not yet triaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants