-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Testing Platform] Command line arguments file source #3499
Comments
We have it partially done but that does need to be finished and documented (cc @MarcoRossignoli). |
Actually is not yet implemented but it's doable and a good idea, maybe a section like {
"testingPlatform" :
{
"additionalCommandLine" : "--someoption file --anotherone"
}
} That we can parse and add, we need to understand what to do in case of duplicates (override, merge, throw) and ordering of the apply. |
I'd probably take the file as the base values, and then any actual command line options override them |
👍 This is how the current vstest works. It takes config file, then answer file, then commandline arguments, then inline settings (settings specified on commandline after One great addition to that would imho be also applying settings file over a settings file, so we can have slightly different settings in CI, similar to my.configy.json, my.dev.config.json e.g. --settings my-settings.json --settings my-settings.ci.json
@MarcoRossignoli I think there is a good concept hidden in there, IMHO the settings and commandline parameters should be closely related and there should be a definitive step where all settings are finalized and are applied back to a "runsettings" object. This object is then the source of truth for the local run, or a remote run. In less abstract terms we should be able to write a plugin / execution mode that takes given command line parameters and turns them into runsettings. |
My main problem are complex configuration, in command line you cannot express too much and the configuration file in future could change the format like toml. One thing that we can do is maybe override the config with a convention "--overrideconfig-mstest-parallelize true" for scalar value {
...
"mstest":
{
"parallelize" : false
}
} here in command line we have only strings without type safety. |
That is the direction I am proposing. Command Line -> config file, not the other way around. So not being able to handle all settings in commandline is okay. What I would like to avoid is the situation in VSTest where we say that runsettings are the source of truth, but we cannot easily convert from commandline parameters to runsettings, and so distributed runs don't allow all providing additional commandline parameters, and users have to convert to runsettings manually. |
Or maybe allow choice of ordering like asp/generic host builder? |
1 similar comment
Or maybe allow choice of ordering like asp/generic host builder? |
What you mean? |
The Microsoft host builder model allows you to choose overriding settings by extension methods of .AddEnvVariables or .AddJsonFile. you could let the framework decide that if using automatic hook registrations, or if the framework allows the user to build their own entry main method, they could set up the order themselves. Would be more work but give more flexibility. Just a thought. |
From what I can tell options are only supported currently from the command line.
It might be useful to try to find a well-known file that can store options within it. This way it can be source controlled by teams, and provide a more deterministic result between local development and CI builds without developers having to know which various flags to set.
E.g.
test-options.json
The text was updated successfully, but these errors were encountered: