Allow Private Registry Authentication

[2opremio]

ECS has an easy way to set up Private Registry Authentication through User Data but the ECS CLI still doesn't support it (see #16 )

[vultron81]

+1

[abest0]

👍

[alvarowolfx]

+1

[miw0129]

Hi All( @2opremio @vultron81 @abest0 @alvarowolfx @samuelkarp ),
I wanted to manage ECS nodes using ecs-cli command with my private DockerHub repository. So, I made a prototype of modified version (private fork) of ecs-cli last night. Then I found the Issue this morning and am happy to know other people also want this feature.

I patched the source code for PoC: add “User Data” entries into CloudFormation settings to put credentials into /etc/ecs/ecs.config following the official ECS guide.
I confirmed that I could successfully access my private repository via ecs-cli compose up command using ECS Cluster/EC2 instances launched with patched version of ecs-cli (ecs-cli up).

I am creating a Pull-Request on this issue to make authentication easy, but wondering how would we put credentials for DockerHub.
I think there are mostly three options here.

1. Specify auth and email of DockerHub in ecs-cli configure command and put them into current user’s base settings file
2. Specify auth and email of DockerHub in ecs-cli up command options
   eg) ecs-cli up --keypair someservice --capability-iam --size 2 --instance-type t2.micro --vpc vpc-abcd1234 --subnets subnet-abcdef01, subnet-10fedcba --docker-hub-auth xxxxxxx --docker-hub-email xxxxx@example.com
3. Specify auth and email of DockerHub in ecs-cli configure command to store them into current user’s base settings file, to make it overridable using ecs-cli up command options(option 1 + option 2)

I think that the option 2 is most appropriate in this situation from the viewpoint of security.
Because DockerHub’s access-keys are in most cases credentials for people(yes they are). I would welcome option 1 or 3 (saving credentials into user-wide settings file) if DockerHub supports something like AWS’s IAM for account role delegation but it doesn’t (at the moment AFAIK). So, I think we should set security over convenience by policy.

Of course it’s only my opinion and I want to know how other people think. Any comments are welcomed.

[2opremio]

@miw0129 I would expect a more generic solution (i.e. not specific to private registry credentials), in which you can append whatever User Data you want during instance creation, specified when invoking ecs up. For instance, through a new ecs up flag called --extra-user-data, see #16.

This should ultimately be approved by the maintainers, of course (@samuelkarp I think).

In the meantime and in case it's useful for anybody else, I use this workaround to configure Private Registry Authentication after creating a cluster with the ECS CLI. I use the AWS CLI to add credentials to all the container instances in the cluster.

In the example below

1. I create a cluster named foo
2. I add credentials to each container instance in the for accessing a quay.io private repository.

Tweak EXTRA_ECS_CONFIG to your particular Private Registry usecase.

The example assumes that you have created a key pair named bar and the pem key file (bar.pem) exists in your working directory.

#!/bin/bash

set -e

EXTRA_ECS_CONFIG='ECS_ENGINE_AUTH_TYPE=dockercfg\nECS_ENGINE_AUTH_DATA={"quay.io":{"auth":"SECRET=","email":""}}'
CLUSTER_NAME=foo
CLUSTER_SIZE=3
INSTANCE_TYPE=t2.micro
REGION=us-east-1
KEY_NAME=bar
KEY_FILE=bar.pem

print_instance_ips() {
    INSTANCE_ARNS=$(aws --region $REGION ecs list-container-instances --cluster $CLUSTER_NAME --query 'containerInstanceArns[*]' --output text)
    INSTANCE_IDS=$(aws --region $REGION ecs describe-container-instances --cluster $CLUSTER_NAME --container-instances $INSTANCE_ARNS --query 'containerInstances[*].ec2InstanceId' --output text)
    aws --region $REGION ec2 describe-instances --instance-ids $INSTANCE_IDS --query 'Reservations[*].Instances[*].PublicIpAddress' --output text
}

ecs-cli configure --region $REGION --cluster $CLUSTER_NAME
ecs-cli up --port 22 --keypair $KEY_NAME --capability-iam --size $CLUSTER_SIZE --instance-type $INSTANCE_TYPE
echo -n "Waiting for instances to be up to add auth credentials (this may take a while) ..."
while ! INSTANCE_IPS=$(print_instance_ips 2> /dev/null) ||
        [ "$(echo $INSTANCE_IPS | wc -w)" -ne $CLUSTER_SIZE ] ; do
    sleep 5
done
echo "done"
SSH_ARGS="-i $KEY_FILE -o StrictHostKeyChecking=no ec2-user@"
for IP in $INSTANCE_IPS; do
    ssh $SSH_ARGS$IP -t sudo bash -c "'echo -e $(printf %q $EXTRA_ECS_CONFIG)  >> /etc/ecs/ecs.config'"
    ssh $SSH_ARGS$IP -t sudo stop ecs
    ssh $SSH_ARGS$IP -t sudo start ecs
done

[abest0]

@2opremio 🍔 for you. Fantastic script that should satisfy most people's needs for now.

[miw0129]

@2opremio
Thank you for a very useful opinion.
I agree to take the generic solution for flexibility. So,I think this change may be divided into two phases.

1. make ecs-cli command UserData-aware
2. consider about shorthand for DockerHub private repositories

About (1), I want to discuss about how ecs-cli up subcommand handles the parameter. I’ll post my opinion on # 16 later.

About (2), in my opinion, ecs-cli up command should provide users the simple (and special) way to provide authentication information of DockerHub because:

1. the authentication information of DockerHub it is expected to be used frequently
2. it’s a bit of pain to create specific authentication JSON entry (or the whole shell script) only for DockerHub access credentials.

By the way, thanks for your workaround shell script. It looks useful 🍣.
I think there might be some tricks to improve this script🎃. When certain EC2 instance launched on ECS cluster is somehow shutten down and automatically destroyed & relaunched by CloudFormation settings, the new instance doesn’t contain modified ecs.config. So, maybe it’s also useful to extract ecs.config tweaking part and make it idempotent (creating some “already under control” file somewhere and perform a check before tweaking ecs.config would be enough).

[jessesanford]

I agree that there should be a more general solution for this, however I imagine they are holding off until amazon's own private registry service is available. Then they don't have to worry about creds. You will just have to push your images to the amazon hosted private registry and reference the image there inside your docker-compose.yml. This is similar to the way tutum and google container services work.

[2opremio]

I think there might be some tricks to improve this script🎃. When certain EC2 instance launched on ECS cluster is somehow shutten down and automatically destroyed & relaunched by CloudFormation settings, the new instance doesn’t contain modified ecs.config

Right, that's why it's just a workaround :)

I agree that there should be a more general solution for this, however I imagine they are holding off until amazon's own private registry service is available.

Maybe, but the ECS CLI should support other registries, not just Amazon's just like ECS already does.

[pleerock]

+1

[sobit]

👍

[pleerock]

@2opremio for some reason the script is hanging at "Waiting for instances to be up to add auth credentials (this may take a while) ..." on me

[aidanbon]

@pleerock
Try replacing the != sign with -ne in [ "$(echo $INSTANCE_IPS | wc -w)" != $CLUSTER_SIZE ].

i.e. Change the above condition to [ "$(echo $INSTANCE_IPS | wc -w)" -ne $CLUSTER_SIZE ]

-ne is for numeric "not equal", and != is for string comparison.

And again, thanks @2opremio for the script!!

[pleerock]

@aidanbon thanks, thats worked

[2opremio]

@aidanbon Thanks! I have edited the original script with your improvement.

[bappelt]

+1

[ShahBinoy]

+1

[robpc]

+1

[jesucarr]

@2opremio thanks for sharing your script. You forgot to replace us-east-1 with $REGION in the last line of print_instance_ips

[2opremio]

@jesucarr Thanks for spotting it, I've corrected the script.

[SoManyHs]

Related to #16

[allisaurus]

Hi all,

We released support for private registry authentication using AWS Secrets Manager secrets in 1.8.0. (see #573).

A more generic solution re: adding extra user data is being tracked in #16.

[allisaurus]

Hi all - FYI, enhanced support for private registry credentials was added in 1.10.0. Now the ecs-cli can store your creds with AWS Secrets Manager via the new registry-creds up command, and automatically add them to your tasks or service with the compose commands!

[j-mendez]

@allisaurus how are you auto adding them to the tasks with the registry-creds up? Followed the steps but, I still have to manually add the secrets after every deploy for private repos.

[allisaurus]

@j-mendez registry-creds up will produce an output file containing the SSM parameters for your secrets that can be ingested by the compose commands. So you need to run registry-creds up to produce the output file before you run compose so that the secrets can be incorporated into the task definition of your task/service.

Here are a couple examples to show how that would work with different inputs.

If you already have an output file, compose should pick it up automatically if it lives in the same directory as your docker-compose files, but you can also specify it explicitly with the --registry-creds flag.

Or if you already have your secrets in SSM parameters, you can also just pass them in directly via the credentials_parameter field in the ecs-params.yml file. Hope this helps!