Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Option to use Terraform Configuration in TestStep #150

Closed
bendbennett opened this issue Jul 13, 2023 · 1 comment · Fixed by #153
Closed

Add Option to use Terraform Configuration in TestStep #150

bendbennett opened this issue Jul 13, 2023 · 1 comment · Fixed by #153
Labels
enhancement New feature or request
Milestone

Comments

@bendbennett
Copy link
Contributor

terraform-plugin-testing version

v1.3.0

Use cases

An individual acceptance test makes use of a TestStep which includes a Config field containing a string representation of Terraform configuration, supplied as either HCL or JSON.

One of the consequences of the Config field using a string is that provider developers can decide on how they pass in Terraform configuration to the tests, for example:

  • Usage of raw string syntax (backticks in Go).
  • Calling Go standard library functions, such as fmt.Sprintf(), to dynamically generate the string.
  • Usage of a Go constant or variable, with a value from either above form.
  • Creation a Go function which returns a string (and may or may not take input parameters to make it more dynamic).
  • Manually implement a Go function which can read an existing Terraform configuration file.

In all of the above cases except for the last, this can result in varying degrees of consistency in the provider codebase with respect to how Terraform configuration is created and supplied to the tests. Furthermore the usage of a Go string for Terraform configuration means that existing Terraform ecosystem tooling cannot be leveraged. This includes, but is not limited to:

  • terraform fmt
  • terraform validate
  • Editor code syntax highlighting.
  • Tooling which can rewrite configurations.
  • Cost estimation of running all acceptance tests.
  • Collection of data regarding which tests cover a specific resource or data source.

Attempted solutions

Some of the solutions that have been tried are outlined above, such as manually implementing a Go function that can read an existing Terraform configuration file. However, this solution requires implementing by each provider developer and that fact is one of the motivations for this proposal

Proposal

The provider developer experience can be improved by making the development and maintenance of acceptance tests easier through native support of a self-contained Terraform configuration directory. This would require modification of the TestStep struct to include a new ConfigDirectory field which would provide a relative path to a directory containing a Terraform configuration file (*.tf file):

func TestAccExampleThing_basic(t *testing.T) {
    resource.ParallelTest(t, resource.TestCase{
        /* ... */
        Steps: []resource.TestStep{
            {
                ConfigDirectory: "testdata/example_thing/basic",
                Check: /* ... */,
            },
        },
    })
}

To support dynamically updated Terraform configurations, new support for Terraform variable values defined within Go testing code can be injected into the “self-contained” files or directories through the usage of an additional new ConfigVariables field on the TestStep struct:

func TestAccExampleThing_basic(t *testing.T) {
    resource.ParallelTest(t, resource.TestCase{
        /* ... */
        Steps: []resource.TestStep{
            {
                ConfigDirectory: "testdata/example_thing/basic",
                ConfigVariables: map[string]config.Variable{
                    /* ... */

This would allow for the generation of an *.auto.tfvars.json during the execution of an acceptance test, which would then be used within the Terraform configuration file(s) at runtime.

@bendbennett bendbennett added the enhancement New feature or request label Jul 13, 2023
bendbennett added a commit that referenced this issue Jul 17, 2023
…s of TestStep.Config or TestStep.Directory can be encapsulated in teststep.config struct (#150)
bendbennett added a commit that referenced this issue Jul 17, 2023
…instantiating implementations of teststep.Config (#150)
bendbennett added a commit that referenced this issue Jul 17, 2023
…p.Config interface for applied configuration (#150)
bendbennett added a commit that referenced this issue Jul 18, 2023
…lock and TestStep.mergedConfig() to configuration.HasProviderBlock(), configuration.hasTerraformBlock and configuration.MergedConfig(), respectively (#150)
bendbennett added a commit that referenced this issue Jul 18, 2023
…configuration struct to simplify processing during GetRaw() and the equivalent function(s) for processing Directory and File (#150)
bendbennett added a commit that referenced this issue Jul 19, 2023
…gement of writing test case provider config or test step provider config simpler when handling copying of files from configuration.directory (#150)
bendbennett added a commit that referenced this issue Jul 19, 2023
…Case or TestStep are being used when TestStep.ConfigDirectory is specified (#150)
bendbennett added a commit that referenced this issue Jul 19, 2023
…f provider blocks in configuration files within the configuration directory (#150)
bendbennett added a commit that referenced this issue Jul 19, 2023
bendbennett added a commit that referenced this issue Jul 20, 2023
…ined within testCaseProviderConfig and testStepProvider config fields) when processing config directory as the expectation is that external providers will be specified directly in the terraform configuration files within the config directory (#150)
bendbennett added a commit that referenced this issue Jul 20, 2023
… validate that when ConfigDirectory is defined that ExternalProviders cannot be specified for either TestCase or TestStep (#150)
bendbennett added a commit that referenced this issue Jul 20, 2023
…blocks are only written when using TestStep.Config. The expectation is that when using TestStep.ConfigDirectoy, the terraform files within the configuration directory will specify the terraform and/or provider blocks as necessary (#150)
bendbennett added a commit that referenced this issue Jul 21, 2023
…aform configuration files in TestStep.ConfigDirectory (#150)
bendbennett added a commit that referenced this issue Jul 21, 2023
bendbennett added a commit that referenced this issue Jul 24, 2023
bendbennett added a commit that referenced this issue Jul 26, 2023
bendbennett added a commit that referenced this issue Jul 26, 2023
  * Configuration() now returns the Config interface to accommodate the different types that could be returned (e.g., configurationDirectory, configurationString)
@bflad bflad added this to the v1.5.0 milestone Jul 26, 2023
bendbennett added a commit that referenced this issue Jul 27, 2023
…ethods/funcs in teststep/config.go (#150)

  * Includes refactoring of Configuration func to remove error returned as this was always nil and the returned Config interface needs to be checked for nil in any case.
bendbennett added a commit that referenced this issue Jul 28, 2023
…figurationFile HasProviderBlock, HasTerraformBlock and Write methods (#150)

  * Adding test coverage for configurationDirectory and configurationFile HasProviderBlock method
bendbennett added a commit that referenced this issue Jul 28, 2023
bendbennett added a commit that referenced this issue Jul 28, 2023
… configurationFile and configurationString Write methods (#150)
bendbennett added a commit that referenced this issue Jul 28, 2023
bendbennett added a commit that referenced this issue Jul 31, 2023
…amically to determine the directory containing the Terraform configuration (#150)
bendbennett added a commit that referenced this issue Jul 31, 2023
bendbennett added a commit that referenced this issue Jul 31, 2023
…s prior to copy configuration and variables for current test step (#150)
bendbennett added a commit that referenced this issue Jul 31, 2023
bendbennett added a commit that referenced this issue Aug 31, 2023
* Adding ConfigDirectory to TestStep (#150)

* Add teststep.Config interface so that logic that examines the contents of TestStep.Config or TestStep.Directory can be encapsulated in teststep.config struct (#150)

* Update TestStep.Validate() to use teststep.Config interface implementation (#150)

* Switching to using a teststep.ConfigurationRequest struct for use in instantiating implementations of teststep.Config (#150)

* Refactoring to use type implementing teststep.Config interface (#150)

* Refactoring testStepNewImportState() to use type implementing teststep.Config interface for applied configuration (#150)

* Switching to using type implementing teststep.Config interface (#150)

* Moved TestStep.configHasProviderBlock(), TestStep.configHasTerraformBlock and TestStep.mergedConfig() to configuration.HasProviderBlock(), configuration.hasTerraformBlock and configuration.MergedConfig(), respectively (#150)

* Fix validation error message to include ConfigDirectory (#150)

* Added testCaseProviderConfig and testStepProviderConfig as fields on configuration struct to simplify processing during GetRaw() and the equivalent function(s) for processing Directory and File (#150)

* Add initial implementation of copying files from ConfigDirectory into wd.baseDir (#150)

* Switch to using WriteDirectory() method on configuration to make management of writing test case provider config or test step provider config simpler when handling copying of files from configuration.directory (#150)

* Adding tests to verify that the ExternalProviders specified in a TestCase or TestStep are being used when TestStep.ConfigDirectory is specified (#150)

* Extending implementation of HasProviderBlock() to include detection of provider blocks in configuration files within the configuration directory (#150)

* Moving writing of raw config or copy of config directory files internally to configuration struct. (#150)

* Removing writing of configuration files for external providers (contained within testCaseProviderConfig and testStepProvider config fields) when processing config directory as the expectation is that external providers will be specified directly in the terraform configuration files within the config directory (#150)

* Adding HasConfigurationFiles() func to Config interface to be able to validate that when ConfigDirectory is defined that ExternalProviders cannot be specified for either TestCase or TestStep (#150)

* Reinstating TestStep.mergedConfig() method as terraform and provider blocks are only written when using TestStep.Config. The expectation is that when using TestStep.ConfigDirectoy, the terraform files within the configuration directory will specify the terraform and/or provider blocks as necessary (#150)

* Adding a couple of tests to verify behaviour when using multiple terraform configuration files in TestStep.ConfigDirectory (#150)

* Switching to using TestStepConfigFunc type for TestStep.ConfigDirectory (#150)

* Adding acceptance test coverage for StaticDirectory(), TestNameDirectory() and TestStepDirectory() (#150)

* Adding Exec() method to TestStepConfigFunc type and removing ExecuteTestStepConfigFunc (#150)

* Adding ConfigVariables to allow defining of Terraform variables within a TestStep (#150)

* Adding docs for config directory and variables (#150)

* Adding tests for TestStepConfigFunc.Exec() method (#150)

* Adding tests for config<Type>Variable and Variables.Write() func (#150)

* Adding copyright headers (#150)

* Unmarshalling before comparing (#150)

* Apply suggestions from code review

Co-authored-by: Brian Flad <bflad417@gmail.com>

* Adding TestName to TestStepConfigRequest and updating TestStepDirectory() func to use TestName (#150)

* Switching to supplying t.Name() to TestStepConfigRequest (#150)

  * Moving all test fixtures for ConfigDirectory to resource/testdata

* Marshalling Variables directly (#150)

* Adding comment to explain the usage of Unwrap() (#150)

* Adding FloatVariable and IntegerVariable and removing NumberVariable (#150)

* Changing ConfigurationRequest fields Directory and Raw to *string to make it clear that they are optional (#150)

* Adding Validate() func to ConfigurationRequest along with test coverage (#150)

* Adding configurationDirectory and configurationString types (#150)

  * Configuration() now returns the Config interface to accommodate the different types that could be returned (e.g., configurationDirectory, configurationString)

* Renaming raw.go to string.go to match type defined therein (#150)

* Adding support for TestStep.ConfigFile field (#150)

* Updating docs for config pkg (#150)

* Adding documentation into teststep/config.go and tests for exported methods/funcs in teststep/config.go (#150)

  * Includes refactoring of Configuration func to remove error returned as this was always nil and the returned Config interface needs to be checked for nil in any case.

* Adding checks for absolute filepath in configurationDirectory and configurationFile HasProviderBlock, HasTerraformBlock and Write methods (#150)

  * Adding test coverage for configurationDirectory and configurationFile HasProviderBlock method

* Adding tests configurationDirectory, configurationFile and configurationString HasTerraformBlock methods (#150)

* Adding test coverage for error conditions and configurationDirectory, configurationFile and configurationString Write methods (#150)

* Adding empty dir for tests to git (#150)

* Adding a page on usage of Terraform configuration to the website docs (#150)

* Adding changelog entries (#150)

* Adding test to demonstrate that the TestStep number is being used dynamically to determine the directory containing the Terraform configuration (#150)

* Removing unneeded elsif conditional (#150)

* Updating docs (#150)

* Removing any configuration or variables files from previous test steps prior to copy configuration and variables for current test step (#150)

* Updating docs (#150)

* Apply suggestions from code review

Co-authored-by: Brian Flad <bflad417@gmail.com>

* Update .changes/unreleased/FEATURES-20230728-152822.yaml

Co-authored-by: Brian Flad <bflad417@gmail.com>

* Removing unneeded changelog entries (#150)

* Reusing stepNumber variable (#150)

---------

Co-authored-by: Brian Flad <bflad417@gmail.com>
@github-actions
Copy link

github-actions bot commented Oct 1, 2023

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 1, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants