diff --git a/docs/en_US/MnistExamples.md b/docs/en_US/MnistExamples.md index 916c32f153..5393a0eff5 100644 --- a/docs/en_US/MnistExamples.md +++ b/docs/en_US/MnistExamples.md @@ -51,7 +51,7 @@ This example is to show how to use hyperband to tune the model. There is one mor This example is to show that NNI also support nested search space. The search space file is an example of how to define nested search space. -`code directory: examples/trials/mnist-cascading-search-space/` +`code directory: examples/trials/mnist-nested-search-space/` **distributed MNIST (tensorflow) using kubeflow** diff --git a/docs/en_US/SearchSpaceSpec.md b/docs/en_US/SearchSpaceSpec.md index 0a5f06737f..843c77200b 100644 --- a/docs/en_US/SearchSpaceSpec.md +++ b/docs/en_US/SearchSpaceSpec.md @@ -29,7 +29,7 @@ All types of sampling strategies and their parameter are listed here: * Which means the variable's value is one of the options. Here 'options' should be a list. Each element of options is a number of string. It could also be a nested sub-search-space, this sub-search-space takes effect only when the corresponding element is chosen. The variables in this sub-search-space could be seen as conditional variables. - * An simple [example](../../examples/trials/mnist-cascading-search-space/search_space.json) of [nested] search space definition. If an element in the options list is a dict, it is a sub-search-space, and for our built-in tuners you have to add a key '_name' in this dict, which helps you to identify which element is chosen. Accordingly, here is a [sample](../../examples/trials/mnist-cascading-search-space/sample.json) which users can get from nni with nested search space definition. Tuners which support nested search space is as follows: + * An simple [example](https://github.com/microsoft/nni/tree/master/examples/trials/mnist-nested-search-space/search_space.json) of [nested] search space definition. If an element in the options list is a dict, it is a sub-search-space, and for our built-in tuners you have to add a key '_name' in this dict, which helps you to identify which element is chosen. Accordingly, here is a [sample](https://github.com/microsoft/nni/tree/master/examples/trials/mnist-nested-search-space/sample.json) which users can get from nni with nested search space definition. Tuners which support nested search space is as follows: - Random Search - TPE @@ -86,9 +86,19 @@ All types of sampling strategies and their parameter are listed here: | Hyperband Advisor | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | Metis Tuner | ✓ | ✓ | ✓ | ✓ | | | | | | | -Note that In Grid Search Tuner, for users' convenience, the definition of `quniform` and `qloguniform` change, where q here specifies the number of values that will be sampled. Details about them are listed as follows -* Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it. -* Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back. +Known Limitations: -Note that Metis Tuner only support numerical `choice` now +* Note that In Grid Search Tuner, for users' convenience, the definition of `quniform` and `qloguniform` change, where q here specifies the number of values that will be sampled. Details about them are listed as follows + + * Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it. + + * Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back. + +* Note that Metis Tuner only supports numerical `choice` now + +* Note that for nested search space: + + * Only Random Search/TPE/Anneal/Evolution tuner supports nested search space + + * We do not support nested search space "Hyper Parameter" parallel graph now, the enhancement is being considered in #1110(https://github.com/microsoft/nni/issues/1110), any suggestions or discussions or contributions are warmly welcomed diff --git a/docs/zh_CN/MnistExamples.md b/docs/zh_CN/MnistExamples.md index 2268d79c5b..c1402f0edc 100644 --- a/docs/zh_CN/MnistExamples.md +++ b/docs/zh_CN/MnistExamples.md @@ -51,7 +51,7 @@ 此样例演示了 NNI 如何支持嵌套的搜索空间。 搜索空间文件示了如何定义嵌套的搜索空间。 -`代码目录: examples/trials/mnist-cascading-search-space/` +`代码目录: examples/trials/mnist-nested-search-space/` **用 Kubeflow 运行分布式的 MNIST (tensorflow)** diff --git a/docs/zh_CN/SearchSpaceSpec.md b/docs/zh_CN/SearchSpaceSpec.md index d7379cd0d0..c8c7829ec3 100644 --- a/docs/zh_CN/SearchSpaceSpec.md +++ b/docs/zh_CN/SearchSpaceSpec.md @@ -29,7 +29,7 @@ * 表示变量的值是选项之一。 这里的 'options' 是一个数组。 选项的每个元素都是字符串。 也可以是嵌套的子搜索空间。此子搜索空间仅在相应的元素选中后才起作用。 该子搜索空间中的变量可看作是条件变量。 - * 这是个简单的 [nested] 搜索空间定义的[示例](../../examples/trials/mnist-cascading-search-space/search_space.json)。 如果选项列表中的元素是 dict，则它是一个子搜索空间，对于内置的 Tuner，必须在此 dict 中添加键 “_name”，这有助于标识选中的元素。 相应的，这是从 NNI 中获得的嵌套搜索空间定义的[示例](../../examples/trials/mnist-cascading-search-space/sample.json)。 以下 Tuner 支持嵌套搜索空间： + * 这是个简单的 [nested] 搜索空间定义的[示例](https://github.com/microsoft/nni/tree/master/examples/trials/mnist-nested-search-space/search_space.json)。 如果选项列表中的元素是 dict，则它是一个子搜索空间，对于内置的 Tuner，必须在此 dict 中添加键 “_name”，这有助于标识选中的元素。 相应的，这是从 NNI 中获得的嵌套搜索空间定义的[示例](https://github.com/microsoft/nni/tree/master/examples/trials/mnist-nested-search-space/sample.json)。 以下 Tuner 支持嵌套搜索空间： * Random Search（随机搜索） * TPE diff --git a/examples/trials/mnist-cascading-search-space/config.yml b/examples/trials/mnist-nested-search-space/config.yml similarity index 86% rename from examples/trials/mnist-cascading-search-space/config.yml rename to examples/trials/mnist-nested-search-space/config.yml index 1c2a4643a4..7c1715b97c 100644 --- a/examples/trials/mnist-cascading-search-space/config.yml +++ b/examples/trials/mnist-nested-search-space/config.yml @@ -1,5 +1,5 @@ authorName: default -experimentName: mnist-cascading-search-space +experimentName: mnist-nested-search-space trialConcurrency: 2 maxExecDuration: 1h maxTrialNum: 100 diff --git a/examples/trials/mnist-cascading-search-space/mnist.py b/examples/trials/mnist-nested-search-space/mnist.py similarity index 99% rename from examples/trials/mnist-cascading-search-space/mnist.py rename to examples/trials/mnist-nested-search-space/mnist.py index a90c7640cb..ed892ff2b5 100644 --- a/examples/trials/mnist-cascading-search-space/mnist.py +++ b/examples/trials/mnist-nested-search-space/mnist.py @@ -14,7 +14,7 @@ import nni -logger = logging.getLogger('mnist_cascading_search_space') +logger = logging.getLogger('mnist_nested_search_space') FLAGS = None class MnistNetwork(object): diff --git a/examples/trials/mnist-cascading-search-space/requirments.txt b/examples/trials/mnist-nested-search-space/requirments.txt similarity index 100% rename from examples/trials/mnist-cascading-search-space/requirments.txt rename to examples/trials/mnist-nested-search-space/requirments.txt diff --git a/examples/trials/mnist-cascading-search-space/sample.json b/examples/trials/mnist-nested-search-space/sample.json similarity index 100% rename from examples/trials/mnist-cascading-search-space/sample.json rename to examples/trials/mnist-nested-search-space/sample.json diff --git a/examples/trials/mnist-cascading-search-space/search_space.json b/examples/trials/mnist-nested-search-space/search_space.json similarity index 100% rename from examples/trials/mnist-cascading-search-space/search_space.json rename to examples/trials/mnist-nested-search-space/search_space.json diff --git a/test/config_test/examples/mnist-cascading-search-space.test.yml b/test/config_test/examples/mnist-nested-search-space.test.yml similarity index 72% rename from test/config_test/examples/mnist-cascading-search-space.test.yml rename to test/config_test/examples/mnist-nested-search-space.test.yml index 1904afac64..accbd8b88f 100644 --- a/test/config_test/examples/mnist-cascading-search-space.test.yml +++ b/test/config_test/examples/mnist-nested-search-space.test.yml @@ -3,7 +3,7 @@ experimentName: default_test maxExecDuration: 5m maxTrialNum: 4 trialConcurrency: 2 -searchSpacePath: ../../../examples/trials/mnist-cascading-search-space/search_space.json +searchSpacePath: ../../../examples/trials/mnist-nested-search-space/search_space.json tuner: #choice: TPE, Random, Anneal, Evolution @@ -13,7 +13,7 @@ assessor: classArgs: optimize_mode: maximize trial: - codeDir: ../../../examples/trials/mnist-cascading-search-space + codeDir: ../../../examples/trials/mnist-nested-search-space command: python3 mnist.py --batch_num 100 gpuNum: 0