aws_emr_cluster "New resource required" while updating instance_count inside instance_groups

[ghost]

This issue was originally opened by @aashisn as hashicorp/terraform#18405. It was migrated here as a result of the provider split. The original body of the issue is below.

---

I did quick search over existing issues and didnt see anything relevent; if its duplicate feel free to link existing ticket and close.

It seems terraform destroys all cluster or (creates from sctrach) even when changing number of node within instance_groups.

Either we should be able to update those count without destroying existing cluster as aws cli allows to do.

Or, aws_emr_instance_group, should accept bid_price.

This is problem; when you want to have Master/Core in ondemand and have all task instances in spot.

Terraform Version

Terraform v0.11.4
+ provider.aws v1.26.0
+ provider.terraform v1.0.2

Terraform Configuration Files

resource "aws_emr_cluster" "etl-cluster" {
  name             = "emr1"
  release_label = "emr-5.12.0"
  applications  =  ["Spark"]

  ec2_attributes {
    key_name                          = "my_key"
    subnet_id                         = "subnet"
    emr_managed_master_security_group = "${aws_security_group.emr_master_sec_group.id}"
    emr_managed_slave_security_group  = "${aws_security_group.emr_slave_sec_group.id}"

    additional_master_security_groups = "${var.additional_security_groups}"
    additional_slave_security_groups  = "${var.additional_security_groups}"




    service_access_security_group     = "${aws_security_group.emr_service_sec_group.id}"

    instance_profile                  = "${aws_iam_instance_profile.emr_profile.arn}"
  }

instance_groups = [
    {
      name           = "MasterInstanceGroup"
      instance_role  = "MASTER"
      instance_type  = "m3.xlarge"
      instance_count = "1"
    },
    {
      name           = "CoreInstanceGroup"
      instance_role  = "CORE"
      instance_type  = "m3.xlarge"
      instance_count = "1"
    },
    {
      name           = "TaskInstanceGroup"
      instance_role  = "TASK"
      instance_type  = "m3.xlarge"
      instance_count = "1"
      bid_price      = "0.2"
    }
  ]

  tags {
    Name = "${var.global_product}.${var.tier}.${var.instance_role}"
  }

  bootstrap_action {
    path = "${var.provisioning_script}"
    name = "emr_provision"
  }

  service_role = "${aws_iam_role.iam_emr_service_role.arn}"

  log_uri = "${var.log_uri}"

}

Expected Behavior

Should be able to update count without re-creating or destroying cluster;

like:
aws emr add-instance-groups --cluster-id j-XXXXXXXXXX --instance-groups file://add-task.json --region eu-west-1

aws emr modify-instance-groups --cluster-id j-XXXXXXXXXX --instance-groups file://update.json --region eu-west-1

Actual Behavior

-/+ module.dev-emr-push.aws_emr_cluster.etl-cluster (new resource required)
      id:                                                 "j-1C9J2PFS0Z26B" => <computed> (forces new resource)
      applications.#:                                     "1" => "1"
      applications.1557323817:                            "Spark" => "Spark"
      bootstrap_action.#:                                 "1" => "1"
      bootstrap_action.547413520.args.#:                  "0" => "0"
      bootstrap_action.547413520.name:                    "emr_provision" => "emr_provision"
      bootstrap_action.547413520.path:                    "s3://mybucket/emr_bootstrap" => "s3://mybucket/emr_bootstrap"
      cluster_state:                                      "WAITING" => <computed>
      core_instance_type:                                 "m3.xlarge" => <computed>
      ec2_attributes.#:                                   "1" => "1"
      ec2_attributes.0.emr_managed_master_security_group: "sg-e8708294" => "sg-e8708294"
      ec2_attributes.0.emr_managed_slave_security_group:  "sg-fe7b8982" => "sg-fe7b8982"
      ec2_attributes.0.instance_profile:                  "arn:aws:iam::xxxxxxxx:instance-profile/instance-p" => "arn:aws:iam::xxxxxxxx:instance-profile/instance-p"
      ec2_attributes.0.key_name:                          "my-key" => "my-key"
      ec2_attributes.0.service_access_security_group:     "sg-927587ee" => "sg-927587ee"
      ec2_attributes.0.subnet_id:                         "subnet-f9f0ee9e" => "subnet-f9f0ee9e"
      instance_group.#:                                   "3" => "3"
      instance_group.1492522993.autoscaling_policy:       "" => ""
      instance_group.1492522993.bid_price:                "0.2" => ""
      instance_group.1492522993.ebs_config.#:             "0" => "0"
      instance_group.1492522993.instance_count:           "1" => "0"
      instance_group.1492522993.instance_role:            "TASK" => ""
      instance_group.1492522993.instance_type:            "m3.xlarge" => "" (forces new resource)
      instance_group.1492522993.name:                     "TaskInstanceGroup" => "" (forces new resource)
      instance_group.1789981495.autoscaling_policy:       "" => ""
      instance_group.1789981495.bid_price:                "0.2" => "0.2"
      instance_group.1789981495.ebs_config.#:             "0" => "0"
      instance_group.1789981495.instance_count:           "1" => "1"
      instance_group.1789981495.instance_role:            "MASTER" => "MASTER"
      instance_group.1789981495.instance_type:            "m3.xlarge" => "m3.xlarge"
      instance_group.1789981495.name:                     "MasterInstanceGroup" => "MasterInstanceGroup"
      instance_group.4155555735.autoscaling_policy:       "" => ""
      instance_group.4155555735.bid_price:                "" => "0.2"
      instance_group.4155555735.ebs_config.#:             "" => "0"
      instance_group.4155555735.instance_count:           "" => "2"
      instance_group.4155555735.instance_role:            "" => "TASK"
      instance_group.4155555735.instance_type:            "" => "m3.xlarge" (forces new resource)
      instance_group.4155555735.name:                     "" => "TaskInstanceGroup" (forces new resource)
      instance_group.4288292731.autoscaling_policy:       "" => ""
      instance_group.4288292731.bid_price:                "0.2" => "0.2"
      instance_group.4288292731.ebs_config.#:             "0" => "0"
      instance_group.4288292731.instance_count:           "1" => "1"
      instance_group.4288292731.instance_role:            "CORE" => "CORE"
      instance_group.4288292731.instance_type:

[tikejhy]

Tested with 1.22 it's the same.

[szpuni]

I have same issue here. When you want add or remove instance terraform just decides to kill entire cluster.
This makes it kind of usable only to create new cluster but any sort of maintenance is not possible.

Another weird issue is if I add instance manually in AWS Console terraform doens't recognise that change.
To reproduce:

• Create cluster with terraform with Core instance count 1.
• Use AWS Console to increase Core Instances count and add 1 additional instance.
• Wait until instance is successfully added.
• Run "terraform refresh" to pull changes from AWS into your State file.
• Update instance group in your code to same amount as in AWS.
• Run "terraform plan"

What you will see is that plan will try to remove entire cluster even tho your remote state should have 2 instances in Core instance Group and your code have exact same value.
Plan in this case should return that no changes are required but it doesn't and still tries to remove entire cluster.

Is this something we can change? I see that GO API have call for changing instance groups but I'm not sure why this require to kill entire cluster.

[aneagoe]

I've also stumbled across this. Looking at the cloudformation documentation, it looks like it requires a recreate for this resource type:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-emr-cluster-jobflowinstancesconfig-instancegroupconfig.html
However, it's not exactly clear how come the console and aws cli work and can increase the instance count without any issues.

[szpuni]

I'm not sure either how they got it working but I think we could just recreate instance_group rather than entire cluster which basically require destroying all your work and start again....

[opsline-radek]

I just run a quick test with Cloudformation. I was able to change the count of instances without destroying the cluster. However, changing the instance type requires a replacement.

I created a stack with https://github.com/awslabs/aws-cloudformation-templates/blob/master/aws/services/EMR/EMRCLusterGangliaWithSparkOrS3backedHbase.json

[bflad]

Hi folks 👋 Version 3.0.0 of the Terraform AWS Provider removed the problematic instance_groups configuration blocks in preference of tailored master_instance_group and core_instance_group configuration blocks. Please reach out if you have any issues with the newer configurations.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!