Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws/request: Treat ConcurrentUpdateException as retryable #1570

Closed
wants to merge 1 commit into from
Closed

aws/request: Treat ConcurrentUpdateException as retryable #1570

wants to merge 1 commit into from

Conversation

radeksimko
Copy link
Contributor

I have received the following error when trying to create two autoscaling targets at the same time:

ConcurrentUpdateException: You already have a pending update to an Auto Scaling resource.
			status code: 400, request id: 12bcfbd4-a8ed-11e7-ab84-f5312ac9a28e

@xibz
Copy link
Contributor

xibz commented Oct 4, 2017

@radeksimko - I don't think this should be retryable. The only real concern I have is that operations would be nondeterministic. What is your use case? Is it for the describe calls?

@radeksimko
Copy link
Contributor Author

To be fair it would be best if the API was just capable of handling more than 1 request at a time 😜
Since that's not the reality there are IMO two ways of dealing with this:

  1. Implement mutex on the API client side
  2. Retry - which is what I'm proposing

Admittedly the first option may be a bit more "correct" - is that what you're suggesting?

@xibz
Copy link
Contributor

xibz commented Oct 4, 2017

@radeksimko - Yea, the first option I think is the better way of going about this. Is there any issues with the first approach?

@radeksimko
Copy link
Contributor Author

My only issue is that it feels a bit dirty to do this. My feeling is that API maintainer (AWS in this case) is pushing the responsibility for dealing with the internal problem up to the user and it should be either hidden in the API (best case) or SDK.

As far as I'm aware the SDK doesn't have any concept of mutexes for any particular methods? Would there be any interest in adding those to the SDK or do you expect users of the SDK to implement this themselves?

@jasdel
Copy link
Contributor

jasdel commented Oct 4, 2017

I don't think it would be good for the SDK to add murexes around these API operations. Mostly because it would only partially workaround the issue. In a distributed system environment a mutex within a single application instance won't be much help if two or more instances try to make the change close enough to each other.

If this is an issue with the service restrict any resource change to only a one at a time. I think this is feedback we should forward on to the service. Especially in the distributed environment, this issue should be handled either by the service allowing multiple concurrent changes, or the client application infrastructure limiting the changes to only a single one at a time.

I'm curious if there is anyway to identify that a resource change is pending. This might be a good place to have a waiter that will wait for a pending resource.

@xibz
Copy link
Contributor

xibz commented Oct 4, 2017

@jasdel - Agreed, I was thinking client side above the SDK as clarification. I did not see a waiter for this particular operation, unfortunately.

@radeksimko
Copy link
Contributor Author

I'm curious if there is anyway to identify that a resource change is pending. This might be a good place to have a waiter that will wait for a pending resource.

That would be awesome.

In a distributed system environment a mutex within a single application instance won't be much help if two or more instances try to make the change close enough to each other.

Indeed - which is actually argument for retry 🙂 although I agree with @xibz that it's not a nice solution, to blindly retry.

@adriantodorov
Copy link

Hey @radeksimko , thank you for this. Our Jenkins pipeline is hanging because of this bug. is there a timeline for a new release for the plugin for provider "aws"?

@jasdel
Copy link
Contributor

jasdel commented Nov 9, 2017

Thanks for creating this PR @radeksimko, but I don't think the SDK should retry for this API call. Since this is a API resource limit of the service API a waiter might be a better option, or for the application making the API call to manage the concurrent resource updates.

The AWS EC2 Forums is the best place to reach out to the Autoscaling and request the service support a higher rate of resource updates.

@jasdel jasdel closed this Nov 9, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants