tfversion: Consider Ignoring Terraform Prerelease Version Information in Certain Checks

[bflad]

terraform-plugin-testing version

v1.7.0

Use cases

When Terraform introduces new functionality, it will appear in prereleases of the minor version before general availability. This is true for provider-facing features, the Terraform built-in provider features, and purely configuration-based features.

The terraform-plugin-testing Go module supports automatic test skipping and test failures based on the Terraform version via the TestCase.TerraformVersionChecks field and the various built-in check implementations in the tfversion package.

In this example, tests will automatically be skipped if the Terraform version is below 1.8.0 (e.g. 1.7.x, 1.6.x, etc.):

TerraformVersionChecks: []tfversion.TerraformVersionCheck{
	tfversion.SkipBelow(tfversion.Version1_8_0),
},

Especially when it comes to provider-facing features though, there is a time period during these Terraform prereleases where provider developers may want to implement the new features with acceptance testing, but also ensure that testing only runs on the upcoming version or later. For example, many of the HashiCorp-owned "utility" providers explicitly run their testing against every minor version of Terraform supported to detect potential regressions or version-specific quirks so these version checks working are important to not raise false positives or unexpectedly pass testing by skipping it.

Given the above check, running the testing while using Terraform 1.8.0-beta1 will skip the test:

$ TF_ACC_TERRAFORM_PATH=/Users/bflad/bin/terraform1.8.0-beta1 TF_ACC=1 go test -count=1 -run='TestRFC3339Parse_UTC' -v ./internal/provider
=== RUN   TestRFC3339Parse_UTC
    function_rfc3339_parse_test.go:18: Terraform CLI version 1.8.0-beta1 is below minimum version 1.8.0: skipping test
--- SKIP: TestRFC3339Parse_UTC (1.18s)

For certain version checks, it may be desirable to "allow"/"ignore" prerelease versions given that the intention of these checks is to detect versions based on the core version information.

Attempted solutions

As a workaround, it is possible to inject prerelease information into tfversion package checks:

TerraformVersionChecks: []tfversion.TerraformVersionCheck{
	// TODO: Replace with the stable v1.8.0 release when available
	tfversion.SkipBelow(version.Must(version.NewVersion("1.8.0-beta1"))),
},

This requires updates until the final release occurs.

Proposal

In at least SkipBelow and RequireAbove checks, update the logic to ignore prerelease information when performing the comparison. github.com/hashicorp/go-version supports calling (*version.Version).Core() for example, which could be used against the Terraform version. If compatibility concerns are absolutely important in this case, then the Core() call could potentially be ignored if (*version.Version).Prerelease() is non-empty for the version given to the check.

Another idea could be to introduce separate checks with the slightly different prerelease behavior, although it seems like explicit Terraform prerelease handling should be treated as exceptional or less common use given how Terraform is versioned in practice.

References

• function/rfc3339_parse: Add RFC3339 parsing function terraform-provider-time#280 (comment)

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.