Analysis utils #2435

zheng-ningxin · 2020-05-15T06:19:45Z

Add several analysis tools for the neural network: (1) sensitivity analysis, (2) topology analysis

Sensitivity Analysis Tool

'sensitivity_analysis.py' automatically analyze the sensitivity to the prune ratio for each layer in the model. In addition, the sensitivity results can also be visualized for user observation.

Topology Analysis Tools

1 'shape_dependency.py' analyze the network architecture and find the layers that have shape dependencies.
2 'mask_conflict.py' this module can fix the mask conflict between the layers has channel dependency. This module should be called before the 'speedup' ,so that, the speedup module can handle the models with residual connections.

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

model should be set to eval mode before the jit.trace call. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

In the original way, addmm will also triger the dependency set searching, which may lead to a wrong dependency set. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

mask_conflict can fix the mask conflict of the layers that has channel dependency. This part should be called before the speedup function, so that, the speedup module can handle the model with residual connection/concat operations. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

update the interface. if we alreay have the traced graph of the model we donnot need to trace the model again. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add unittest for tools in analysis_utils to verify the correctness of the visulization, channel dependency, and mask conflict. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

QuanluZhang · 2020-05-29T06:54:17Z

Thanks @zheng-ningxin , could you add doc (e.g., how to use the sensitivity tool, the requirement of this tool) under https://github.com/microsoft/nni/tree/master/docs/en_US/Compressor?

setup.py

src/sdk/pynni/nni/analysis_utils/sensitivity/__init__.py

src/sdk/pynni/nni/analysis_utils/sensitivity/torch/sensitivity_analysis.py

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

src/sdk/pynni/nni/compression/torch/utils/sensitivity_analysis.py

QuanluZhang · 2020-06-16T00:06:37Z

docs/en_US/Compressor/CompressionUtils.md

+s_analyzer.export(os.path.join(outdir, filename))
+```
+
+Two key parameters of SensitivityAnalysis are model, and val_func. 'model' is the neural network that to be analyzed and the 'val_func' is the validation function that returns the model accuracy/loss/ or other metrics on the validation dataset. Due to different scenarios may have different ways to calculate the loss/accuracy, so users should prepare a function that returns the model accuracy/loss on the dataset and pass it to SensitivityAnalysis.


'model' -> model
'val_func' -> val_func

Thanks~ I have fixed it, please review on the latest version~

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

QuanluZhang · 2020-06-16T02:41:21Z

docs/en_US/Compressor/CompressionUtils.md

+Two key parameters of SensitivityAnalysis are model, and val_func. model is the neural network that to be analyzed and the val_func is the validation function that returns the model accuracy/loss/ or other metrics on the validation dataset. Due to different scenarios may have different ways to calculate the loss/accuracy, so users should prepare a function that returns the model accuracy/loss on the dataset and pass it to SensitivityAnalysis.
+SensitivityAnalysis can export the sensitivity results as a csv file usage is shown in the example above.
+
+Futhermore, users can specify the sparsities values used to prune for each layer by optinal parameter 'sparsities'.


optinal -> optional

I have fixed several grammar errors, please review on the latest version, thanks~

QuanluZhang · 2020-06-16T02:46:10Z

docs/en_US/Compressor/CompressionUtils.md

+minimize: The analysis stops when the validation metric return by the val_func lower than early_stop_value.
+maximize: The analysis stops when the validation metric return by the val_func larger than early_stop_value.
+dropped: The analysis stops when the validation metric has dropped by early_stop_value.
+raised: The analysis stops when the validation metric has raised by early_stop_value.


better to itemize these four values. you can check the rendered version.

get confused about dropped and raised. what is the difference between dropped/raised and minimize/maximize?

dropped is used in the cases that you want to stop when the accuracy has dropped by 5%, for example.
minimize is used in the cases that you want to keep the accuracy larger than a specified threshold during the analysis.

QuanluZhang · 2020-06-16T02:53:49Z

docs/en_US/Compressor/CompressionUtils.md

+```python
+s_analyzer = SensitivityAnalysis(model=net, val_func=val, sparsities=[0.25, 0.5, 0.75], early_stop_mode='dropped', early_stop_value=0.1)
+```
+If users only want to analyze several specified convolutional layers, users can specify the target conv layers by the 'sepcified_layers' parameter in analysis function. For example


how to specify a conv layer, using module name or module type?

Sensitivity analyisis only analyze the conv layers, so we use the module name to specify the layers.

then better to make it clear that the layers are specified through PyTorch module name.

Sure, I'll update the doc and make this clear, thanks.

QuanluZhang · 2020-06-16T02:56:10Z

docs/en_US/Compressor/CompressionUtils.md

+features.10,0.55468,0.5394,0.49576,0.4291,0.3591,0.28138,0.14256,0.05446,0.01578
+```
+
+## Topology


-> Topology Analysis

QuanluZhang · 2020-06-16T02:56:55Z

docs/en_US/Compressor/CompressionUtils.md

@@ -0,0 +1,109 @@
+# Analysis Utils for Model Compression


could add

.. contents::

after this line

in order for users to easily get what is the content of this doc

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Ningxin added 9 commits May 14, 2020 01:26

Add analysis tools for sensitivity and topology.

0a4b7b0

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Reformat the code and add several small new features.

712d982

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add the flops information rendering for the visulization.

202593c

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add the depedency rendering feature.

8a7a799

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Update the interface of the SensitivityAnalysis

362441a

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add sensitivity rendering feature.

e69e78f

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add copyright and license.

5823276

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Remove the unrelated files.

4a70d79

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Fix some typos.

fc95dd7

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

QuanluZhang requested review from chicm-ms, ultmaster and QuanluZhang May 15, 2020 08:57

Ningxin added 13 commits May 18, 2020 04:04

Fix a small issue.

a90c35e

model should be set to eval mode before the jit.trace call. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Fix a small issue.

1909ff0

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Fix bug.

2d13dda

In the original way, addmm will also triger the dependency set searching, which may lead to a wrong dependency set. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add compatibility with versions prior to torch-1.4.0.

0e79624

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Update the interface.

6029603

update the interface. if we alreay have the traced graph of the model we donnot need to trace the model again. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add unit test for analysis_utils.

9beb1e2

Add unittest for tools in analysis_utils to verify the correctness of the visulization, channel dependency, and mask conflict. Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Fix the format warnings from pylint.

6b25ff3

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Add dependencies.

d0bda49

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

comment the visualization test temporarily.

4154cf0

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

update

83f0b26

Skip the test when the torch version is too old.

388056c

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

update

ccbcc6c

scarlett2018 mentioned this pull request May 28, 2020

NNI Backlog - welcome to comment and suggest #1917

Closed

100 tasks

QuanluZhang reviewed May 29, 2020

View reviewed changes

setup.py Outdated Show resolved Hide resolved

QuanluZhang reviewed May 29, 2020

View reviewed changes

src/sdk/pynni/nni/analysis_utils/sensitivity/__init__.py Outdated Show resolved Hide resolved

QuanluZhang reviewed May 29, 2020

View reviewed changes

src/sdk/pynni/nni/analysis_utils/sensitivity/torch/sensitivity_analysis.py Outdated Show resolved Hide resolved

Ningxin added 4 commits June 15, 2020 06:28

Update according the review comments.

b7671da

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

Rename the unit test.

1b9705b

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

update docs

9d0519e

fix pylint errors

f563802

zheng-ningxin requested review from QuanluZhang, colorjam and chicm-ms June 15, 2020 07:29

chicm-ms reviewed Jun 15, 2020

View reviewed changes

src/sdk/pynni/nni/compression/torch/utils/sensitivity_analysis.py Outdated Show resolved Hide resolved

colorjam approved these changes Jun 15, 2020

View reviewed changes

QuanluZhang reviewed Jun 16, 2020

View reviewed changes

Ningxin added 2 commits June 16, 2020 01:35

Update.

a24acd0

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

update

91d5f49

chicm-ms approved these changes Jun 16, 2020

View reviewed changes

QuanluZhang reviewed Jun 16, 2020

View reviewed changes

Ningxin added 6 commits June 16, 2020 02:57

fix grammar

33178a2

update doc

7cab808

Update the docs.

e8d4c31

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

update doc

3351cef

update Docs.

db0ff63

Signed-off-by: Ningxin <Ningxin.Zheng@microsoft.com>

remove unnecessray comments

7153bd7

zheng-ningxin requested review from QuanluZhang and colorjam June 16, 2020 06:13

QuanluZhang approved these changes Jun 16, 2020

View reviewed changes

QuanluZhang merged commit c3d7b4c into microsoft:master Jun 16, 2020

chicm-ms mentioned this pull request Jul 1, 2020

June release end game #2621

Closed

24 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Analysis utils #2435

Analysis utils #2435

zheng-ningxin commented May 15, 2020 •

edited

Loading

QuanluZhang commented May 29, 2020

QuanluZhang Jun 16, 2020

zheng-ningxin Jun 16, 2020

QuanluZhang Jun 16, 2020

zheng-ningxin Jun 16, 2020

QuanluZhang Jun 16, 2020

QuanluZhang Jun 16, 2020

zheng-ningxin Jun 16, 2020

QuanluZhang Jun 16, 2020

zheng-ningxin Jun 16, 2020

QuanluZhang Jun 16, 2020

zheng-ningxin Jun 16, 2020

QuanluZhang Jun 16, 2020

QuanluZhang Jun 16, 2020

QuanluZhang Jun 16, 2020

Analysis utils #2435

Analysis utils #2435

Conversation

zheng-ningxin commented May 15, 2020 • edited Loading

Sensitivity Analysis Tool

Topology Analysis Tools

QuanluZhang commented May 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zheng-ningxin commented May 15, 2020 •

edited

Loading