Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add configuration support for EMR Instance Groups #10426

Merged
merged 1 commit into from
Oct 10, 2019
Merged

Add configuration support for EMR Instance Groups #10426

merged 1 commit into from
Oct 10, 2019

Conversation

joelthompson
Copy link
Contributor

This adds a configurations_json property for the aws_emr_instance_group
resource similar to the configurations_json property for the
aws_emr_cluster resource as they are processed the same by AWS.

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Closes #9342

Release note for CHANGELOG:

Support configurations_json property for aws_emr_instance_group resources

Output from acceptance testing:

Note that two tests failed the first time -- one due to resource limits in my test AWS account and the other looks to have been due to a flaky test due to a race condition with AWS's EMR cluster provisioning process. Rerunning both of them individually succeeded.

$ make testacc TESTARGS='-run=TestAccAWSEMRInstanceGroup'         
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -count 1 -parallel 20 -run=TestAccAWSEMRInstanceGroup -timeout 120m
go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible
go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSEMRInstanceGroup_basic
=== PAUSE TestAccAWSEMRInstanceGroup_basic
=== RUN   TestAccAWSEMRInstanceGroup_BidPrice
=== PAUSE TestAccAWSEMRInstanceGroup_BidPrice
=== RUN   TestAccAWSEMRInstanceGroup_ConfigurationsJson
=== PAUSE TestAccAWSEMRInstanceGroup_ConfigurationsJson
=== RUN   TestAccAWSEMRInstanceGroup_AutoScalingPolicy
=== PAUSE TestAccAWSEMRInstanceGroup_AutoScalingPolicy
=== RUN   TestAccAWSEMRInstanceGroup_InstanceCount
=== PAUSE TestAccAWSEMRInstanceGroup_InstanceCount
=== RUN   TestAccAWSEMRInstanceGroup_EbsConfig_EbsOptimized
=== PAUSE TestAccAWSEMRInstanceGroup_EbsConfig_EbsOptimized
=== CONT  TestAccAWSEMRInstanceGroup_basic
=== CONT  TestAccAWSEMRInstanceGroup_InstanceCount
=== CONT  TestAccAWSEMRInstanceGroup_EbsConfig_EbsOptimized
=== CONT  TestAccAWSEMRInstanceGroup_AutoScalingPolicy
=== CONT  TestAccAWSEMRInstanceGroup_BidPrice
=== CONT  TestAccAWSEMRInstanceGroup_ConfigurationsJson
--- FAIL: TestAccAWSEMRInstanceGroup_InstanceCount (6.38s)
    testing.go:569: Step 0 error: errors during apply:
        
        Error: Error creating internet gateway: InternetGatewayLimitExceeded: The maximum number of internet gateways has been reached.
                status code: 400, request id: 7b3c326b-971f-441c-bede-d3a0e94b7936
        
          on /tmp/tf-test590806217/main.tf line 38:
          (source code not available)
        
        
--- PASS: TestAccAWSEMRInstanceGroup_EbsConfig_EbsOptimized (545.38s)
--- FAIL: TestAccAWSEMRInstanceGroup_BidPrice (552.05s)
    testing.go:569: Step 1 error: ImportStateVerify attributes not equivalent. Difference is shown below. Top is actual, bottom is expected.
        
        (map[string]string) (len=1) {
         (string) (len=6) "status": (string) (len=8) "RESIZING"
        }
        
        
        (map[string]string) (len=1) {
         (string) (len=6) "status": (string) (len=12) "PROVISIONING"
        }
        
--- PASS: TestAccAWSEMRInstanceGroup_AutoScalingPolicy (556.20s)
--- PASS: TestAccAWSEMRInstanceGroup_basic (591.83s)
--- PASS: TestAccAWSEMRInstanceGroup_ConfigurationsJson (839.93s)
FAIL
FAIL    github.com/terraform-providers/terraform-provider-aws/aws       839.969s
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/flatmap      0.030s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/keyvaluetags 0.004s [no tests to run]
FAIL
make: *** [testacc] Error 1

$ make testacc TESTARGS='-run=TestAccAWSEMRInstanceGroup_BidPrice'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -count 1 -parallel 20 -run=TestAccAWSEMRInstanceGroup_BidPrice -timeout 120m
go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible
go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSEMRInstanceGroup_BidPrice
=== PAUSE TestAccAWSEMRInstanceGroup_BidPrice
=== CONT  TestAccAWSEMRInstanceGroup_BidPrice
--- PASS: TestAccAWSEMRInstanceGroup_BidPrice (569.24s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       569.287s
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/flatmap      0.002s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/keyvaluetags 0.004s [no tests to run]

$ make testacc TESTARGS='-run=TestAccAWSEMRInstanceGroup_InstanceCount'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./... -v -count 1 -parallel 20 -run=TestAccAWSEMRInstanceGroup_InstanceCount -timeout 120m
go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible
go: finding github.com/terraform-providers/terraform-provider-tls v2.1.1+incompatible
?       github.com/terraform-providers/terraform-provider-aws   [no test files]
=== RUN   TestAccAWSEMRInstanceGroup_InstanceCount
=== PAUSE TestAccAWSEMRInstanceGroup_InstanceCount
=== CONT  TestAccAWSEMRInstanceGroup_InstanceCount
--- PASS: TestAccAWSEMRInstanceGroup_InstanceCount (593.90s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       593.921s
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/flatmap      0.007s [no tests to run]
testing: warning: no tests to run
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws/internal/keyvaluetags 0.005s [no tests to run]

@joelthompson
Add configuration support for EMR Instance Groups
1040810 
This adds a configurations_json property for the aws_emr_instance_group
resource similar to the configurations_json property for the
aws_emr_cluster resource as they are processed the same by AWS.

Fixes hashicorp#9342
@joelthompson joelthompson requested a review from a team October 8, 2019 18:36
@ghost ghost added size/M Managed by automation to categorize the size of a PR. service/emr Issues and PRs that pertain to the emr service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. documentation Introduces or discusses updates to documentation. labels Oct 8, 2019
@bflad bflad added the enhancement Requests to existing resources that expand the functionality or scope. label Oct 10, 2019
bflad
bflad approved these changes Oct 10, 2019
View reviewed changes
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @joelthompson 🚀

--- PASS: TestAccAWSEMRInstanceGroup_basic (545.24s)
--- PASS: TestAccAWSEMRInstanceGroup_InstanceCount (572.37s)
--- PASS: TestAccAWSEMRInstanceGroup_EbsConfig_EbsOptimized (591.95s)
--- PASS: TestAccAWSEMRInstanceGroup_BidPrice (605.60s)
--- PASS: TestAccAWSEMRInstanceGroup_AutoScalingPolicy (611.06s)
--- PASS: TestAccAWSEMRInstanceGroup_ConfigurationsJson (769.64s)

aws/resource_aws_emr_instance_group.go
"configurations_json": {
Type: schema.TypeString,
Optional: true,
ForceNew: false,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: ForceNew: false is extraneous 😉

aws/resource_aws_emr_instance_group.go
@@ -177,6 +200,19 @@ func resourceAwsEMRInstanceGroupRead(d *schema.ResourceData, meta interface{}) e
return fmt.Errorf("error reading EMR Instance Group (%s): %s", d.Id(), err)
}

switch {
case len(ig.Configurations) > 0:
configOut, err := flattenConfigurationJson(ig.Configurations)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally not for this pull request, but seems like it would be nice if flattenConfigurationJson just did

if len(config) == 0 {
  return "", nil
}

so we didn't need the surrounding conditional logic. 👍

@bflad bflad added this to the v2.32.0 milestone Oct 10, 2019
@bflad bflad merged commit 1585f05 into hashicorp:master Oct 10, 2019
bflad added a commit that referenced this pull request Oct 10, 2019
@bflad
Update CHANGELOG for #10426
0bc1023
@joelthompson joelthompson deleted the emr_instance_group_config branch October 10, 2019 03:58
@joelthompson
Copy link
Contributor Author

Thanks @bflad !

@ghost
Copy link

ghost commented Oct 10, 2019

This has been released in version 2.32.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Nov 9, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 9, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@bflad bflad bflad approved these changes

Assignees
No one assigned
Labels
documentation Introduces or discusses updates to documentation. enhancement Requests to existing resources that expand the functionality or scope. service/emr Issues and PRs that pertain to the emr service. size/M Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v2.32.0
Development

Successfully merging this pull request may close these issues.

Support Configurations Block for Instance Groups
2 participants
@joelthompson @bflad