Support for flexible configuration

[dzhoshkun]

We have decided to migrate the format of NiftyNet configuration files from INI to YAML, as the latter is more flexible. As part of this migration, we have also decided to break the configuration parameters down into a file hierarchy, rather than presenting the user with a single, long configuration file. This with the aim of supporting a wider spectrum of users; such that beginner-level users will not have to deal with advanced configuration parameters, while more advanced users will have much more control.

The CMICLab branch has sample files (under the config/issue-189 folder) demonstrating the proposed new configuration hierarchy. In addition, the CMICLab issue has a long discussion thread with useful pointers. However as GitHub is the main development platform for NiftyNet, we are going to continue the relevant development work here. Also now the configuration parameters are documented on RTD.

This will have a large impact on NiftyNet users. As such, a staged approach may be more suitable to deal with it:

• deprecate the current configuration file format => Migrate configuration files to YAML #168
• migrate all current configuration files to YAML => Migrate configuration files to YAML #168
• support for nesting configuration into multiple files => Support for nested configuration files #169
• move relevant parameter sets into the nested hierarchy, as proposed within the CMICLab issue
• document new configuration file format on RTD => Revise documentation of configuration parameters #182

[dzhoshkun]

@wyli We stalled the development on this due to the refactoring of the NiftyNet engine architecture. Is now a good time to resume it?

Adding @csudre to this discussion as well.

[wyli]

Thanks for restarting this @dzhoshkun. The proposed new config looks really nice.

The next step (to implement the first two stages you mentioned) would be creating a new parser in parallel to the user parameters parser. Potentially we could have both .ini and .yaml supported during the transition?

At the moment "sections" in .ini are just dictionaries: loading here: (https://github.com/NifTK/NiftyNet/blob/v0.3.0/niftynet/utilities/user_parameters_parser.py#L159) and parsing here: (https://github.com/NifTK/NiftyNet/blob/v0.3.0/niftynet/utilities/user_parameters_parser.py#L255)

[dzhoshkun]

Yes, we can support both file formats during the transition. I will add a deprecation warning to the current parser.

[dzhoshkun]

As this is quite extensive work, I've broken it down into multiple issues (all referenced within this issue's description above). However it makes sense to complete all the issues, and then merge the changes back, as we would like to switch directly to the new nested configuration file hierarchy instead of first switching the current configuration file structure to YAML, and then switching to the nested hierarchy. The latter approach might confuse users.

[dzhoshkun]

@ericspod Please could you discuss with me before assigning tickets to me. Thanks.

[atbenmurray]

Note that this is active WRT the hackathon