fix(deadline): Windows WorkerInstanceFleets with Deadline 10.1.14 installed fail deployment #354
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jusiskin
changed the title
ddneilson
requested changes
Mar 22, 2021
| ' Restart-Service "deadline10launcherservice"', | ||
| '} Else {', | ||
| ' & "$DEADLINE_PATH/deadlinelauncher.exe" -shutdownall 2>&1', | ||
| ' & "$DEADLINE_PATH/deadlinelauncher.exe" 2>&1', |
There was a problem hiding this comment.
I see below that you added the missing -nogui options. Add it here as well?
There was a problem hiding this comment.
Good catch - fixed
jusiskin
force-pushed
the
fix_windows_workers_service
branch
from
8b3018f
to
a4409c9
Compare
horsmand
previously approved these changes
Mar 22, 2021
ddneilson
previously approved these changes
Mar 22, 2021
jusiskin
force-pushed
the
fix_windows_workers_service
branch
2 times, most recently
from
9486877
to
7e1872e
Compare
jusiskin
force-pushed
the
fix_windows_workers_service
branch
from
7e1872e
to
5063e81
Compare
ddneilson
approved these changes
Mar 22, 2021
horsmand
approved these changes
Mar 22, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Primarily #353, but also fixes #312.
The real issue is a Deadline bug where stopping or restarting the launcher Windows service while it's still starting up can get stuck. The windows service gets stuck in a
stoppingstate and the only way to unlock the service is to kill the process.RFDK restarts the launcher twice when configuring workers in relatively quick succession. There is a race condition between restarting the service the first time and the second restart. Once Deadline 10.1.14 was available to RFDK, the race became more probable to fail. The result is that the user data generated by RFDK to configure Windows workers never completes to send the CloudFormation signal and deployments fail.
Solution
Modified the
RenderQueueConnectionsub-classes to accept an optionalrestartLauncherproperty. The default was made true to preserve backwards-compatible behavior when this API is called directly. When called byWorkerInstanceConfiguration, this flag is set tofalsebecause it needs to restart the Deadline Launcher again later after configuring the remote command listener ports. By reducing the number of launcher service restarts to one, we avoid the race condition.Testing
Ran the integration tests using this branch and they now all pass.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license