Keeping instance type list up-to-date per region

[SoManyHs]

We need a better way of keeping our list of available EC2 instances up-to-date in our CloudFormation template (related pull requests: #339 and #340), as well as communicate which instance types are valid in a given region.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#AvailableInstanceTypes

The ECS Console currently only lists instance types available in all regions in the Create Cluster wizard dropdown, but the CLI has always allowed all instance types regardless of region. If an instance type that is not supported in a given region is specified, the request will fail:

~/go/src/github.com/aws/amazon-ecs-cli(update_instance_types)$ ecs-cli up --instance-type f1.2xlarge --capability-iam --keypair my-fra-keypair --region eu-central-1 --force
INFO[0001] Created cluster                               cluster=test region=eu-central-1
INFO[0001] Waiting for your CloudFormation stack resources to be deleted... 
INFO[0002] Cloudformation stack status                   stackStatus="DELETE_IN_PROGRESS"
INFO[0034] Waiting for your cluster resources to be created... 
INFO[0034] Cloudformation stack status                   stackStatus="CREATE_IN_PROGRESS"
INFO[0101] Cloudformation stack status                   stackStatus="CREATE_IN_PROGRESS"
INFO[0163] Cloudformation stack status                   stackStatus="CREATE_IN_PROGRESS"
ERRO[0194] Failure event                                 reason="The specified instance type f1.2xlarge is not valid" resourceType="AWS::AutoScaling::LaunchConfiguration"
FATA[0194] Error executing 'up': Cloudformation failure waiting for 'CREATE_COMPLETE'. State is 'ROLLBACK_IN_PROGRESS' 

Ideally, we should know which instance types are valid per region and fail fast rather than waiting for the issue to bubble up through Cloud Formation.

[kingcaubalejo]

Is there any updates on this one?

[made2591]

I think I can work on this in the next days

[SoManyHs]

Hi @made2591!

Thanks for your interest! If you end up working on this, could you please submit a design document so we can discuss options?

Thanks!

[made2591]

Sure! I will work on it and share with you before starting: I already have an idea about how to solve it, but I didn’t have time yet to formalize it. Stay tuned, I will post it here asap!

[made2591]

Hi @SoManyHs,

It seems there's no API for getting available family instance types in respective region (am I wrong?). The idea I had in mind was to load them from AWS exposed price endpoints: in fact, as soon as a new instance family is available in a region, AWS is obliged to expose the price for it as well. These prices are available here for each region with the specified endpoint (you can even ask for all of them in one time, but it would take too much time to download ~700Mb of data). The endpoints for each region are formatted in this way:

...
"ap-south-1" : {
    "regionCode" : "ap-south-1",
    "currentVersionUrl" : "/offers/v1.0/aws/AmazonEC2/20190619214951/ap-south-1/index.json"
},

So, given a region, you can build the endpoint with price list for each family for the specific region (example: eu-west-1):

...
"products" : {
"UEG7KUTR8Q9ZQQN3" : {
    "sku" : "UEG7KUTR8Q9ZQQN3",
    "productFamily" : "Compute Instance",
    "attributes" : {
        "servicecode" : "AmazonEC2",
        "location" : "EU (Ireland)",
        "locationType" : "AWS Region",
        "instanceType" : "m5d.24xlarge",
        ...
    ...
...

and thus by parsing this information inside a proper struct, you can have an always-updated list of available family instances in a given region. As a side effect, this will cost the time of making two HTTP get requests: the first one is pretty fast to complete because data to be downloaded are in the order of 4kb. The second call will require more time - unfortunately, you cannot project the only information you want (GraphQL would be perfect for this, imho) - but still would be faster then waiting for cloudformation fails I guess 🤔. If this approach works, we can even then extended it with a cache mechanism that updated a local store after failure, to avoid the download of prices in every deploy and let the process be fast until the next failure.

For what concerns how to implement this solution, I just started having a look at the code: I don't know if this can be integrated as a service in cloudformation module and then call it during the retrieve/validation chain of the parameters (like in the function retrieveInstanceType). Makes sense? It's just an idea, I didn't explore it or made a test still, just want to share with you the overall approach ^^

What do you think about this?

[SoManyHs]

Resolved by #958