Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose raw hcl.File objects to rules #745

Merged
merged 4 commits into from
May 19, 2020

Conversation

bendrucker
Copy link
Member

@bendrucker bendrucker commented May 11, 2020

This PR adds an initial proof of concept of reading files, parsing as HCL/JSON, and then traversing the content based on a partial schema for exactly the attributes we want. It would close #741 and close #744.

Happy to factor the parser/parsing into the runner as well. Just wanted to get this up and see if you're interested in this direction for this and similar rules.

@mveitas
Copy link
Contributor

mveitas commented May 11, 2020

Happy to factor the parser/parsing into the runner as well. Just wanted to get this up and see if you're interested in this direction for this and similar rules.

👍 This would be a great addition and open up the potential for allowing for additional linting capabilities.

@wata727
Copy link
Member

wata727 commented May 11, 2020

Yeah, this seems like a very good idea.

The way I was thinking was to add a low-level HCL files field (like map[string]*hcl.File) to the Runner. This allows to each rule to have access to TFConfig or low-level HCL APIs as needed.

In such cases, it is better the Loader loads object from a file system. Variable files and annotations are read in that way and build the runner.
https://github.com/terraform-linters/tflint/blob/v0.15.5/cmd/inspect.go#L95-L113

This change can be a bit painful, so it's not a bad option to add such file-reading functionality to the Runner once.

@bendrucker
Copy link
Member Author

Sounds good! Happy to work on the surrounding implementation a bit and move these pieces into the Loader.

Looks like LoadHCLFile takes care of HCL + JSON so that should be a good fit:

https://github.com/hashicorp/terraform/blob/v0.12.24/configs/parser.go#L65

@bendrucker
Copy link
Member Author

Ok, updates!

  • Added a Files method to the loader that returns a map of hcl.File objects that the loader has seen
    • Terraform has a cache of *hcl.File instead of []bytes but it's not exported in a way we can access
    • It seems like LoadAnnotations should also use Sources() rather than loading from disk again? Happy to do that in a separate PR.
  • Added a files field to Runner
  • Called loader.Files when constructing a runner to set files
  • Added a File method to Runner that returns a file from that map

Rules can call that method, presumably with a DeclRange.Filename or something similar, to get the raw HCL representation of any config file they want to inspect.

@bendrucker bendrucker requested a review from wata727 May 15, 2020 02:52
@bendrucker bendrucker changed the title Use hclparse to check variable types Expose raw hcl.File objects to rules May 15, 2020
@@ -186,7 +188,7 @@ func NewModuleRunners(parent *Runner) ([]*Runner, error) {
}
}

runner, err := NewRunner(parent.config, parent.annotations, cfg)
runner, err := NewRunner(parent.config, parent.files, parent.annotations, cfg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the module inspection, the second argument should be the files of the module to be parsed, not the parent module. This means that we need to use Loader here.

However, it would not be a good design for Runner to rely on Loader internally. One idea is to preload all files for all modules on the first load. Maybe it's a bit of a daunting task, what do you think?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooh yes good catch. I think this map will actually contain all files loaded by the loader, since BuildConfig will walk all of the child modules:

cfg, diags := configs.BuildConfig(rootMod, l.moduleWalker())

After doing a bit of poking, I don't think this will break anything, since any path references Terraform has should be relative to the parent dir. However, it does mean that a runner can potentially return a file used outside of its module.

Rather than try to call parser.ConfigDirFiles and essentially mix in loader behavior to the runner, a simple fix here would be to continue to use the same map of files in every runner but validate that path provided has the same directory as the module.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I was misunderstanding. I thought this map would only contain root module files. As you say, it seems that all files are included.

Certainly, it's possible that we can refer to files outside the module, but I don't think there's a big problem. Thank you for the explanation.

Copy link
Member

@wata727 wata727 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good! Is it okay to merge?

@@ -186,7 +188,7 @@ func NewModuleRunners(parent *Runner) ([]*Runner, error) {
}
}

runner, err := NewRunner(parent.config, parent.annotations, cfg)
runner, err := NewRunner(parent.config, parent.files, parent.annotations, cfg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I was misunderstanding. I thought this map would only contain root module files. As you say, it seems that all files are included.

Certainly, it's possible that we can refer to files outside the module, but I don't think there's a big problem. Thank you for the explanation.

@bendrucker
Copy link
Member Author

Yes feel free to merge!

@wata727 wata727 merged commit 16b3d95 into terraform-linters:master May 19, 2020
@wata727
Copy link
Member

wata727 commented May 19, 2020

@bendrucker Thank you again!

By the way, are you interested in joining maintainers? I would be very happy if you help this project based on your deep experience with Terraform and HCL.

@bendrucker
Copy link
Member Author

Absolutely!

@wata727
Copy link
Member

wata727 commented May 20, 2020

Thank you! I added you to the maintainers team 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

terraform_typed_variables - unable to distinguish between missing type and "any" type
3 participants