fix(jenkins): Don't kick off multiples of the same build #863
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The issue here is that when an operation takes a long time (e.g. talking to jenkins to kick off a build) this prolonged time can cause orca to try to resubmit the request multiple times thus kicking off multiple (or infinite number of) builds. We solve this, but setting a flag that a given request (keyed of X-SPINNAKER-REQUEST-ID and master/job/param) is being processed so if an identical request comes in the controller can detect that and respond with 202 ACCEPTED. This will cause orca to retry (corresponding PR coming). When the job is finally kicked off, the resulting build number will be stored in the cache and next time orca queries it will get the build number back
marchello2000
added a commit
to marchello2000/orca
that referenced
this pull request
Sep 18, 2020
This a companion PR to spinnaker/igor#863 The issue here is that when kicking off a build takes a long time (e.g. when jenkins has a lot of job configs) orca might kick off a lot of builds instead of just one (and might still fail!) Igor now adds support for "pending" operation so if the same request is received twice it will return 202. This change adds the corresponding support to orca, on 202, we just need to retry, because eventually igor will give us the build number
|
IntelliJ decided to reformat the spacing.. so review with white space off : https://github.com/spinnaker/igor/pull/863/files?diff=split&w=1 |
jonsie
reviewed
Sep 21, 2020
igor-web/src/main/groovy/com/netflix/spinnaker/igor/build/BuildController.groovy
Outdated
Show resolved
Hide resolved
thank you! |
mergify bot
added a commit
to spinnaker/orca
that referenced
this pull request
Sep 22, 2020
* fix(igor): Handle timeouts kicking off builds This a companion PR to spinnaker/igor#863 The issue here is that when kicking off a build takes a long time (e.g. when jenkins has a lot of job configs) orca might kick off a lot of builds instead of just one (and might still fail!) Igor now adds support for "pending" operation so if the same request is received twice it will return 202. This change adds the corresponding support to orca, on 202, we just need to retry, because eventually igor will give us the build number * fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fixup! fixup! fix(igor): Handle timeouts kicking off builds Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
amanya
pushed a commit
to amanya/orca
that referenced
this pull request
Oct 23, 2020
* fix(igor): Handle timeouts kicking off builds This a companion PR to spinnaker/igor#863 The issue here is that when kicking off a build takes a long time (e.g. when jenkins has a lot of job configs) orca might kick off a lot of builds instead of just one (and might still fail!) Igor now adds support for "pending" operation so if the same request is received twice it will return 202. This change adds the corresponding support to orca, on 202, we just need to retry, because eventually igor will give us the build number * fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fixup! fixup! fix(igor): Handle timeouts kicking off builds Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
KathrynLewis
pushed a commit
to KathrynLewis/orca
that referenced
this pull request
Jan 31, 2021
* fix(igor): Handle timeouts kicking off builds This a companion PR to spinnaker/igor#863 The issue here is that when kicking off a build takes a long time (e.g. when jenkins has a lot of job configs) orca might kick off a lot of builds instead of just one (and might still fail!) Igor now adds support for "pending" operation so if the same request is received twice it will return 202. This change adds the corresponding support to orca, on 202, we just need to retry, because eventually igor will give us the build number * fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fixup! fix(igor): Handle timeouts kicking off builds * fixup! fixup! fixup! fixup! fix(igor): Handle timeouts kicking off builds Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The issue here is that when an operation takes a long time (e.g. talking to jenkins to kick off a build)
the prolonged time can cause
orcato try to resubmit the request multiple times thus kicking off multiple (or infinite number of) builds.We solve this, by setting a flag that a given request (keyed of X-SPINNAKER-REQUEST-ID and master/job/param) is being processed so if an identical request comes in the controller can detect it and respond with
202 ACCEPTED. This will cause orca to retry (see spinnaker/orca#3915).When the job is finally kicked off, the resulting build number will be stored in the cache and next time orca queries it will get the build number back