aws_cur_report_definition - Parquet, Athena and more #12428
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
added
needs-triage
robbruce
changed the title
|
Very similar to #11567, but includes refresh and version options as well as data resource. Didn't know 11567 existed until after creating the pull request! |
|
currently using this in a stage env to test (in the hope this gets merged some time in the future and we can go back to proper master...) hit a couple of things when trying to use, might be worth adding them to the docs: |
|
👍 |
|
Can we please get this merged? |
…definition_updates
|
Updated and re-ran acceptance tests: |
|
@robbruce Thanks for the contribution.
|
…definition_updates
…iginal and new functionality into separate test
|
@ewbankkit changes requested have been implemented. Note, this also fixes #14827 (a bug introduced by #14432) Unit tests re-ran (had to reduce parallel tests as an AWS limit was reached on our account as other reports already existed!) |
ewbankkit
approved these changes
Aug 28, 2020
There was a problem hiding this comment.
LGTM.
--- PASS: TestAccDataSourceAwsCurReportDefinition_additional (14.50s)
--- PASS: TestAccDataSourceAwsCurReportDefinition_basic (14.71s)
--- PASS: TestAccAwsCurReportDefinition_basic (15.66s)
--- PASS: TestAccAwsCurReportDefinition_refresh (15.66s)
--- PASS: TestAccAwsCurReportDefinition_parquet (14.35s)
--- PASS: TestAccAwsCurReportDefinition_athena (14.13s)
--- PASS: TestAccAwsCurReportDefinition_textOrCsv (13.74s)
--- PASS: TestAccAwsCurReportDefinition_overwrite (14.16s)
ewbankkit
reviewed
Aug 28, 2020
Comment on lines
125
to
194
| hasAthena := false | ||
|
|
||
| // perform various combination checks, AWS API unhelpfully just returns an empty ValidationException | ||
| // these combinations have been determined from the Create Report AWS Console Web Form | ||
| for _, artifact := range additionalArtifacts { | ||
| if *artifact == costandusagereportservice.AdditionalArtifactAthena { | ||
| hasAthena = true | ||
| break | ||
| } | ||
| } | ||
|
|
||
| if hasAthena { | ||
| if len(additionalArtifacts) > 1 { | ||
| return fmt.Errorf( | ||
| "When %s exists within additional_artifacts, no other artifact type can be declared", | ||
| costandusagereportservice.AdditionalArtifactAthena, | ||
| ) | ||
| } | ||
|
|
||
| if len(*prefix) == 0 { | ||
| return fmt.Errorf( | ||
| "When %s exists within additional_artifacts, prefix cannot be empty", | ||
| costandusagereportservice.AdditionalArtifactAthena, | ||
| ) | ||
| } | ||
|
|
||
| if *reportVersioning != costandusagereportservice.ReportVersioningOverwriteReport { | ||
| return fmt.Errorf( | ||
| "When %s exists within additional_artifacts, report_versioning must be %s", | ||
| costandusagereportservice.AdditionalArtifactAthena, | ||
| costandusagereportservice.ReportVersioningOverwriteReport, | ||
| ) | ||
| } | ||
| } | ||
|
|
||
| if *format == costandusagereportservice.ReportFormatParquet { | ||
| if *compression != costandusagereportservice.CompressionFormatParquet { | ||
| return fmt.Errorf( | ||
| "When format is %s, compression must also be %s", | ||
| costandusagereportservice.ReportFormatParquet, | ||
| costandusagereportservice.CompressionFormatParquet, | ||
| ) | ||
| } | ||
| } else { | ||
| if *compression == costandusagereportservice.CompressionFormatParquet { | ||
| return fmt.Errorf( | ||
| "When format is %s, format must not be %s", | ||
| *format, | ||
| costandusagereportservice.CompressionFormatParquet, | ||
| ) | ||
| } | ||
| } | ||
|
|
||
| if hasAthena && (*format != costandusagereportservice.ReportFormatParquet) { | ||
| return fmt.Errorf( | ||
| "When %s exists within additional_artifacts, both format and compression must be %s", | ||
| costandusagereportservice.AdditionalArtifactAthena, | ||
| costandusagereportservice.ReportFormatParquet, | ||
| ) | ||
| } | ||
|
|
||
| if !hasAthena && len(additionalArtifacts) > 1 && (*format == costandusagereportservice.ReportFormatParquet) { | ||
| return fmt.Errorf( | ||
| "When additional_artifacts includes %s and/or %s, format must not be %s", | ||
| costandusagereportservice.AdditionalArtifactQuicksight, | ||
| costandusagereportservice.AdditionalArtifactRedshift, | ||
| costandusagereportservice.ReportFormatParquet, | ||
| ) | ||
| } | ||
| // end checks |
There was a problem hiding this comment.
Minor suggestion to move this logic into a separate function that can then be unit tested.
There was a problem hiding this comment.
@ewbankkit makes sense
…ce to external function
|
@ewbankkit just as well you mentioned that as the unit test did highlight a bug if !hasAthena && len(additionalArtifacts) > 1 && (*format == costandusagereportservice.ReportFormatParquet) {should have been! if !hasAthena && len(additionalArtifacts) > 0 && (*format == costandusagereportservice.ReportFormatParquet) { |
maryelizbeth
added
enhancement
ewbankkit
approved these changes
Aug 28, 2020
There was a problem hiding this comment.
LGTM.
$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAwsCurReportDefinition_basic'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAwsCurReportDefinition_basic -timeout 120m
=== RUN TestAccAwsCurReportDefinition_basic
=== PAUSE TestAccAwsCurReportDefinition_basic
=== CONT TestAccAwsCurReportDefinition_basic
--- PASS: TestAccAwsCurReportDefinition_basic (14.42s)
PASS
ok github.com/terraform-providers/terraform-provider-aws/aws 14.466s
bflad
approved these changes
Sep 2, 2020
There was a problem hiding this comment.
Thank you for these updates, @robbruce 🚀
Output from acceptance testing:
--- PASS: TestAccAwsCurReportDefinition_overwrite (37.09s)
--- PASS: TestAccAwsCurReportDefinition_athena (37.35s)
--- PASS: TestAccAwsCurReportDefinition_refresh (38.27s)
--- PASS: TestAccAwsCurReportDefinition_textOrCsv (38.99s)
--- PASS: TestAccAwsCurReportDefinition_basic (39.75s)
--- PASS: TestAccAwsCurReportDefinition_parquet (40.34s)
--- PASS: TestAccDataSourceAwsCurReportDefinition_basic (40.55s)
--- PASS: TestAccDataSourceAwsCurReportDefinition_additional (40.62s)
| additionalArtifactsList = append(additionalArtifactsList, *additionalArtifacts[i]) | ||
| } | ||
|
|
||
| err := checkAwsCurReportDefinitionPropertyCombination( |
There was a problem hiding this comment.
Please note that if this function causes more bug reports than its solving, the maintainers prefer to not include logic like this and will remove its maintenance burden. API error messaging concerns should be handled by the upstream service team via AWS Support cases.
| @@ -107,15 +317,13 @@ func testAccPreCheckAWSCur(t *testing.T) { | |||
| func testAccAwsCurReportDefinitionConfig_basic(reportName string, bucketName string) string { | |||
| return fmt.Sprintf(` | |||
| provider "aws" { | |||
| region = "us-east-1" | |||
| region = "us-east-1" | |||
There was a problem hiding this comment.
Nit: Tabs versus spaces here and below
bflad
added a commit
that referenced
this pull request
Sep 2, 2020
|
This has been released in version 3.5.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change updates aws_cur_report_definition resource and data source to match options available on AWS.
Added:
When using the AWS cur API, it very unhelpfully returns an empty ValidationException when certain combinations of options conflict with each other. The allowed combination of options has been added as part of the create step. For example, if Athena is being used the output format and compression must be Parquet and the s3 prefix cannot be empty!
Community Note
Relates #8765
Release note for CHANGELOG:
Some slight changes were made on the acceptance tests because some race conditions can occur when the bucket policy had not finished applying (even with a depends_on) and so a ValidationException occurs.
Output from acceptance testing: