Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature parity with all of Terraform's built-in file-reading functions #137

Closed
1 task done
apparentlymart opened this issue Aug 31, 2022 · 5 comments · Fixed by #142
Closed
1 task done

Feature parity with all of Terraform's built-in file-reading functions #137

apparentlymart opened this issue Aug 31, 2022 · 5 comments · Fixed by #142

Comments

@apparentlymart
Copy link
Contributor

apparentlymart commented Aug 31, 2022

Terraform CLI and Provider Versions

Terraform v1.3.0-beta1, and hashicorp/local v2.2.3

Use Cases or Problem Statement

We originally established the local_file data source as a way to model reading a file as an explicit side-effect in the Terraform language, whereas the builtin functions like file are a bit dishonest and pretend that reading a file is a pure function (as a measure of pragmatism to support the common case of reading files that are included statically as part of a module package).

It's currently our recommended approach for whenever a Terraform configuration needs to make use of the content of a file that is instead created dynamically as part of the apply step, because resources are how we model side-effects in the Terraform language and so that allows Terraform to model the fact that reading the file depends on some other side-effects to have completed first.

The current mapping between built-in functions and local_file features is:

  • The file function corresponds to the content attribute.
  • The filebase64 function corresponds to the content_base64 attribute.

However, in the meantime we've accumulated some new functions in the core language that directly read static files from disk as a part of their work, and there's no reasonable dynamic analog to these in the hashicorp/local provider:

  • fileexists: Determines whether a particular file actually exists on disk.
  • fileset: Returns a set of filenames matching a particular glob pattern in a particular base directory.
  • The various "read binary data from a file and then calculate a hash of it" variations:

Of these, filemd5 and filebase64sha256 are both associated with specific functionality in other providers:

  • aws_s3_object has a design pattern of providing the filemd5 result for change detection without re-reading the entire object content from the S3 API.
  • aws_lambda_function has a design pattern of providing the filebase64sha256 result for change detection without downloading the original source code archive.

Implementing dynamic analogs to these functions in the hashicorp/local provider would be a pragmatic solution to allow for variations of these documented patterns for files that are dynamically-generated, rather than being included as static files in the module package.

(For completeness: there is also templatefile which reads a template from disk and renders it. The dynamic equivalent of that one was originally template_file in the hashicorp/template provider, but we deprecated that because it was a common usability hazard where people didn't quite understand how to double-escape their templates. I don't propose adding a provider analog to that just yet; we have an ongoing discussion about the tradeoffs for that particular case over in hashicorp/terraform#30616 , and it seems likely we'll try to solve that in the Terraform language itself because of how tightly coupled the template language is with the broader Terraform language.)

Proposal

First I propose adding new attributes to the local_file data source which export various checksums of the file that was read. This matches some precedent elsewhere, such as archive_file in the hashicorp/archive provider which exports SHA-1 hex, MD5 hex, and SHA-256 base64 checksums of the generated file to meet similar dynamic file checksum use-cases. (archive_file is commonly used with aws_lambda_function to dynamically construct source code archives at plan or apply time.)

It's less clear to me whether it's important to match the fileexists and fileset functionality in this provider, but if we do then I expect that each would need to be a separate data source each because the shape of their results isn't compatible with the existing interface and behavior of local_file:

  • A hypothetical local_file_set could take a base directory and a glob pattern as arguments and export a set of string attribute for its result.
  • A hypothetical local_file_exists could take the same arguments as local_file but export a boolean attribute as its result, instead of reading and returning the actual content of the file.

One possible use for these two new data sources would be in conjunction with Preconditions and Postconditions, to represent a rule like "the directory given in var.source_directory must contain a file named package.json".

How much impact is this issue causing?

Medium

Additional Information

I became aware of this thanks to a comment on hashicorp/terraform#22036, which is focused specifically on the aws_lambda_function change-detection pattern.

Code of Conduct

  • I agree to follow this project's Code of Conduct
zachwhaley added a commit to zachwhaley/terraform-provider-local that referenced this issue Sep 9, 2022
@zachwhaley
Copy link
Contributor

I've opened a draft PR to start this solution :)

#142

zachwhaley added a commit to zachwhaley/terraform-provider-archive that referenced this issue Sep 13, 2022
Per hashicorp/terraform-provider-local#137 this brings the provider up to feature parity with all of Terraform's built-in file-reading functions

The following outputs are added to `data_archive_file`:

* `output_sha256` - SHA256 checksum of archive content.
* `output_sha512` - SHA512 checksum of archive content.
* `output_base64sha512` - Base64 encoded SHA512 checksum of archive content.
@zachwhaley
Copy link
Contributor

Not to be impatient, but what is the typical turnaround time on PRs getting reviewed? Some of the PRs in the archive and local provider have been opened for years 🙁

Is there anything I could do to help expedite the process?

Thanks!

@bookshelfdave
Copy link
Contributor

bookshelfdave commented Sep 16, 2022

hi @zachwhaley thank you for the PR! The team is strongly focused on some project work for the next few weeks, but I'm adding it to our board to review when we have some free cycles.

Not to be impatient, but what is the typical turnaround time on PRs getting reviewed?

I don't have a solid answer for you here, but it's something that we're working on so we can communicate better with the community in the (near) future.

@glerchundi
Copy link

This would solve an issue we're seeing with local_file and uploading the file to Fastly Compute@Edge with fastly_service_compute.

The source_code_hash must be a SHA512 of the content of a package we load from another source, in this case from Google Cloud.

So this change would allow us to do the following:

data "google_storage_bucket_object_content" "this" {
  name   = "component/package.tar.gz.b64"
  bucket = "my-fastly-repository"
}

resource "local_file" "this" {
  content_base64 = data.google_storage_bucket_object_content.key.content
  filename       = "package.tar.gz"
}

resource "fastly_service_compute" "service_compute" {
  name = "Service compute"
  package {
    filename         = "package.tar.gz"
    source_code_hash = local_file.output_sha512
  }
}

Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants