-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow number of simultaneous workers to be configured #4257
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved, but have reservations about ignoring invalid user input for --workers
flag. I think rejecting would be better.
@@ -282,3 +291,12 @@ func (ri *ResourceImporter) importResource( | |||
|
|||
return result | |||
} | |||
|
|||
// desiredWorkers returns the number of workers to use for importing resources. | |||
func (ri *ResourceImporter) desiredWorkers() int { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor: This method isn't hurting but my understanding is that it's not actually possible to take this path unless the user explicitly passes a negative or nonzero value for the cmdline option, because if the user doesn't specify anything the default of 4 is used at the cmdline flag level (thus not hitting the default of 4 here).
I'm wondering if it's better to add a validate
helper to importAzureResourceOptions
, and then call that at the start of importAzureResource
and fail the cmd if the user passes a negative value? Could also be extended to check the provided labels for validity (length, allowed number of characters?).
Oh wait I see you already parse the labels it's just down below after the resources are imported. Maybe move the parsing of the labels up front, and validate input of workers too to fail fast on bad user input, then run import (and use the actual parsed labels if configured?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The default will always be 4
if used from the command-line, but that's not the only way to get here.
With the relocation under pkg
we've allowed for reuse. The value here is an option provided in the ResourceImporterOptions
struct, which might not be set by a caller.
What this PR does / why we need it:
Does what it says on the box: makes the number of concurrent workers configurable.
More workers can make the import process complete more quickly - but not in all cases.
The default of 4 seems to be a good compromise, ensuring a single slow resource doesn't stall the entire process too much. Making this configurable gives consumers more control when required.
Closes #4251
How does this PR make you feel