Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle empty ECS Clusters properly #2170

Merged
merged 3 commits into from
Sep 29, 2017

Conversation

jeffreykoetsier
Copy link
Contributor

@jeffreykoetsier jeffreykoetsier commented Sep 23, 2017

This PR fixes a bug in the ECS provider.

When the ECS Provider is configured to poll multiple clusters, it'll show unexpected behavior when one of the clusters has 'zero' tasks running.

  1. If the first cluster Traefik encounters has no running tasks the listInstances function is ended immediately by returning an empty slice of ecsInstances. Successive clusters are simply ignored.

  2. If the 2nd or up cluster has no tasks running the listInstances function will continue trying to fetch data for these non-existing-tasks. The 'early return' condition will not detect this due to counting the length of taskArns slice. This slice var is used by each cluster discovery without being emptied, so it'll be filled with taskArns from the previous cluster. This eventually results in incomplete API calls to AWS ECS.

In the 2nd case Traefik (lookupEc2Instances) sends a 'DescribeContainerInstances' request to AWS with an 'empty' list of containerArns. AWS returns a 4xx.

time="2017-09-19T14:27:49Z" level=error msg="Provider connection error InvalidParameterException: Container instance cannot be empty.
        status code: 400, request id: b6c38d74-9d46-11e7-9fc8-3d440bc8c23f, retrying in 388.555963ms"

Fix:

  • Move initialization of taskArns slice into the for loop. Each iteration of the loop now starts with an empty slice.
  • Do not return from the function when len(taskArns) == 0. Just continue to the next iteration of the loop.

Fixes #2168

Copy link
Member

@mmatur mmatur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

Copy link
Contributor

@ldez ldez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@juliens juliens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@emilevauge emilevauge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants