When using the latest ecs-cli and the new tags option, container instances will not launch.

[rroller]

Summary

This PR added support to the ecs-cli for tags: #670

When using the new --tags option, container instances will not launch. Removing the tags option, container instances will launch.

Description

• Which specific command was run?

        ecs-cli up \
        --force \
        -c "$deployment_id" \
        --keypair somekeypair \
        --aws-profile "$AWS_PROFILE" \
        --instance-type "$INSTANCE_TYPE" \
        --region us-east-1 \
        --instance-role somerole \
        --security-group "$security_group" \
        --vpc "$vpc" \
        --tags "cost_resource_group=$service" \
        --subnets "$subnets" \
        --extra-user-data something.sh

• Which version of the CLI you are using? (Run: ecs-cli --version)

$ ecs-cli --version
ecs-cli version 1.13.0 (2d98b5e)

• Which version of Go are you using? (Run: go version)

$ go version
go version go1.12 darwin/amd64

• What platform are you using to run ECS CLI commands? (E.g. Linux, MacOSX, Windows)
  MacOSX

Config files

Expected Behavior

Container instances should launch

Observed Behavior

Container instances did not launch

[SoManyHs]

Hi @rroller, could you include the output when you run the above command?

[PettitWesley]

@rroller, @SoManyHs and I have reproduced this bug. Its late in our time zone, but we'll have an update in the morning with an explanation of the bug, and (I think) a possible work-around.

Thanks for alerting us to this problem!

[rroller]

Thanks! Sounds like you don't need the output of my command then. If you do please let me know.

[PettitWesley]

@rroller We've diagnosed the issue; I put up a PR to fix it.

Root Cause

1. ECS CLI adds tags to container instances using User Data. The following line is written to /etc/ecs/ecs.config: ECS_CONTAINER_INSTANCE_PROPAGATE_TAGS_FROM=ec2_instance
2. As noted in the documentation, you must have ec2:DescribeTags permissions in your instance role so that the agent can register the container instance with the tags.
3. This permission is not included in the managed policy, as stated in our docs, it must be added inline: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html
4. Container instance roles created by the ECS CLI lack this permission. Thus, the agent can not register the container instance. It crashes with the following output:

Agent Log:

2019-03-09T08:59:55Z [CRITICAL] Error calling DescribeTags API: UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: 1eac4219-99b9-4888-82c1-606b195c3820

ECS Init log:

2019-03-09T08:54:18Z [INFO] Starting Amazon Elastic Container Service Agent
2019-03-09T08:54:18Z [INFO] Agent exited with code 5
2019-03-09T08:54:18Z [ERROR] agent exited with terminal exit code
2019-03-09T08:54:18Z [INFO] stop

Unfortunately, it looks like in #724 I tested container instance tagging using a custom instance role: #724 (comment)

I missed the other case: the CLI creating the instance role. Which unfortunately is the default case.

Work Arounds for 1.13.0

1. Disable containerInstanceLongArnFormat. The ECS CLI will auto-detect that this setting is not enabled, and it will not try to give container instances tags.

aws ecs put-account-setting --name containerInstanceLongArnFormat --value disabled --region <REGION>

Then you will get a message from ecs-cli up:

WARN[0000] Disabling container instance tagging because containerInstanceLongArnFormat is not enabled for your identity, arn:aws:iam::144718711470:user/Wesley3.0. You can use the PutAccountSettingDefault API to change your account default.

2. Use a custom instance role with ecs-cli up using the --instance-role flag. Make sure this role has ec2:DescribeTags permissions.

[rroller]

Thanks for the quick response and details.

Does this mean the next version of the CLI things will work as expected without applying any of the work around?

[PettitWesley]

@rroller

Does this mean the next version of the CLI things will work as expected without applying any of the work around?

Yeah. While its against our policy to give exact dates for releases, we do believe in quickly releasing fixes for bugs like this. Once #745 is merged, tagging in ecs-cli up will work as intended.

[PettitWesley]

@rroller a fix was just released in v1.13.1!

[rroller]

I'm trying it out, but I still get no container instances.

$ ecs-cli --version
ecs-cli version 1.13.1 (78c4de9)

[PettitWesley]

@rroller can you give us full command output and etc? We tested this functionality pretty thoroughly prior to releasing.

[rroller]

Do you have somewhere private I can send it?

[PettitWesley]

@rroller please email wppttt@amazon.com

[rroller]

WARN[0001] Enabling container instance tagging because containerInstanceLongArnFormat
is enabled for your identity, <REMOVED>. If this is not your account default setting, your
instances will fail to join your cluster. You can use the PutAccountSettingDefault API to
change your account default.

I suspect that's why.

[PettitWesley]

Yeah, its an annoying issue. The container instance is registered by the EC2 instance role, and so that IAM Role must be opted in to long ARNs. And the only way to do make that happen is to set the default for your account.

[rroller]

I ran:

aws ecs put-account-setting-default --name serviceLongArnFormat \
  --value enabled \
  --region us-east-1 \
  --profile my-profile-name

Then deployed again. Same issue. I'll email you my logs.

[rroller]

Per the offline email, I needed...

 aws ecs put-account-setting-default \
  --name containerInstanceLongArnFormat \
  --value enabled \
  --region us-east-1 --profile my-profile-name

After that, things worked!

[rroller]

I also needed

aws ecs put-account-setting-default \
  --name taskLongArnFormat \
  --value enabled \
  --region us-east-1 --profile my-profile-name

[miketheman]

Thanks for the help with these commands!
After you've updated the account setting defaults, when you run an ecs-cli up --tags.... do you get a similar log emitted?

WARN[0000] Enabling container instance tagging because containerInstanceLongArnFormat is enabled for your identity, arn:aws:iam::[REDACTED]:root. If this is not your account default setting, your instances will fail to join your cluster. You can use the PutAccountSettingDefault API to change your account default.

Previously, it would show my IAM username, now it shows root - which is mildly confusing, since my credentials are not root.