Add API docs for ONNX Runtime and Intel #100

lewtun · 2022-03-17T17:06:11Z

What does this PR do?

This PR adds the API docs for the intel and onnxruntime packages. It depends on the fixes to doc-builder in this PR, so the CI should pass once that's merged.

The docs are structured as follows:

docs
└── source
    ├── index.mdx  # The landing page / README
    ├── intel  # API docs go under their own subpackage
    │   ├── configuration.mdx  # I use the same naming conventions across subpackages
    │   ├── optimization.mdx
    │   ├── pruning.mdx
    │   ├── quantization.mdx
    │   └── trainer.mdx
    ├── onnxruntime
    │   ├── configuration.mdx
    │   ├── optimization.mdx
    │   ├── quantization.mdx
    │   └── trainer.mdx
    ├── quickstart.mdx  # Guides go in the root of source/
    └── _toctree.yml  # The table of contents

While putting together the docs, I ran into a few optimum quirks that I would like feedback on:

Unlike transformers, optimum doesn't have a dev install that includes the intel and onnxruntime packages. Is this by design? Would it make sense to have pip install '.[dev]' install all subpackages?
Some of the modules have inconsistent naming across intel and onnxruntime. For example, we have intel.neural_compressor.optimizer and onnxruntime.optimization. I've standardised this in the docs to have an "optimization" section per subpackage, and am happy to rename the intel module name if you want.
To build the docs, I had to include the intel.neural_compressor imports in the root init of the intel subpackage. Is this OK?

lewtun · 2022-03-17T21:33:46Z

.github/workflows/build_dev_documentation.yml

@@ -48,7 +48,7 @@ jobs:
          cd ..

          cd optimum
-          pip install .[dev]
+          pip install .[dev,onnxruntime,intel]


I have to include all the dependencies here to build the docs

lewtun · 2022-03-17T21:34:36Z

optimum/intel/__init__.py

@@ -11,3 +11,9 @@
 #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
+
+from .neural_compressor.config import IncConfig, IncPruningConfig, IncQuantizationConfig


Here I have to include the neural_compressor imports in the root init of the intel subpackage to build the docs

lewtun · 2022-03-17T21:38:02Z

docs/source/intel/optimization.mdx

+
+## IncOptimizer
+
+[[autodoc]] intel.neural_compressor.optimizer.IncOptimizer


For consistency with the onnxruntime package, I suggest we rename this module to intel.neural_compressor.optimization. I realise this is a breaking change, but I think it will help optimum users more easily navigate the API.

We can discuss this in a separate issue / thread since it doesn't relate to this PR very strongly

Yes, that's a good point, I was also thinking to rename intel.neural_compressor.config to intel.neural_compressor.configuration for the same reason. I agree that this is an important change that should be done as soon as possible (before our next release, which should be soon)

OK great, let's tackle that in a separate PR :)

echarlaix · 2022-03-22T17:18:22Z

docs/source/intel/optimization.mdx

+
+## IncOptimizer
+
+[[autodoc]] intel.neural_compressor.optimizer.IncOptimizer


Yes, that's a good point, I was also thinking to rename intel.neural_compressor.config to intel.neural_compressor.configuration for the same reason. I agree that this is an important change that should be done as soon as possible (before our next release, which should be soon)

echarlaix · 2022-03-22T17:22:01Z

docs/source/intel/configuration.mdx

+# Trainer
+
+## IncTrainer
+
+[[autodoc]] intel.neural_compressor.trainer_inc.IncTrainer


There seems to be a mismatch with docs/source/intel/trainer.mdx

Yes, I wasn't sure whether to use the same hierarchy as we have in the onnxruntime docs. Happy to nest it though so it lines up with the source code!

What I meant is that IncTrainer is displayed in configuration.mdx whileIncOptimizer is displayed in trainer.mdx

Good catch, fixed now!

echarlaix · 2022-03-22T17:26:22Z

Great addition, thanks a lot @lewtun !!

This reverts commit 9d59bc9.

michaelbenayoun · 2022-03-23T09:59:46Z

.github/workflows/build_dev_documentation.yml

@@ -48,7 +48,7 @@ jobs:
          cd ..

          cd optimum
-          pip install .[dev,onnxruntime,intel]
+          pip install -e .[dev,onnxruntime,intel]


Sorry, I am not very knowledgeable on the subject, but why do we need to install the packages in development mode?

Sorry, that was me trying to debug the doc-builder in the GitHub actions - it's removed now!

lewtun · 2022-03-23T10:17:32Z

.github/workflows/build_dev_documentation.yml

@@ -88,6 +88,7 @@ jobs:
          NODE_OPTIONS: --max-old-space-size=6656
        run: |
          cd doc-build-dev && git pull
+          python -c "from optimum import intel; print(dir(intel))"


This is a temporary debug for the GH actions - will remove before merging

lewtun added 7 commits March 17, 2022 16:21

Add ORT and INC to ToC

753473d

Add dummy files for testing

2a2cf5b

Add API docs for INC

3156ad7

Add imports to root init of INC

25c8590

Add deps to doc builds

fe3058e

Add ORT docs

67ffbe5

Remove dummy text

daacd5c

lewtun requested review from mfuntowicz, echarlaix and michaelbenayoun March 17, 2022 21:32

lewtun marked this pull request as ready for review March 17, 2022 21:33

lewtun commented Mar 17, 2022

View reviewed changes

Fix style

2d8c818

lewtun commented Mar 17, 2022

View reviewed changes

lewtun added 3 commits March 22, 2022 16:26

Fix copyright year

2ec877e

Merge branch 'main' into add-api-docs

2daa86a

Fix docs

aed8ff1

echarlaix approved these changes Mar 22, 2022

View reviewed changes

lewtun added 3 commits March 22, 2022 17:34

Refactor intel docs structure to match codebase

5d0e907

Debug actions

9d59bc9

Revert "Debug actions"

6806c8e

This reverts commit 9d59bc9.

lewtun mentioned this pull request Mar 22, 2022

Test api docs #104

Closed

lewtun added 6 commits March 22, 2022 21:06

Debug action

982bc9c

Debug action

556b98d

Debug action

3e64351

Debug actions

c7dc300

Slice out Intel docs for debug

08fddb4

Slice out Intel for realz

28b8746

lewtun added 2 commits March 23, 2022 09:49

Un nest Intel docs

bbea067

Fix Intel Trainer docs

92b6d8e

michaelbenayoun reviewed Mar 23, 2022

View reviewed changes

Check Intel installed in action

5e6fcca

lewtun commented Mar 23, 2022

View reviewed changes

lewtun added 4 commits March 23, 2022 10:26

Add opencv dep

cedeccf

Aarg, autoinstall opencv

b9eab48

Add opencv dep to main doc build

62cab06

Fix ORT section name

831f6b2

lewtun merged commit 4a83d95 into main Mar 23, 2022

lewtun deleted the add-api-docs branch March 23, 2022 15:35

lewtun mentioned this pull request Mar 23, 2022

Use sudo for doc builder installs #105

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add API docs for ONNX Runtime and Intel #100

Add API docs for ONNX Runtime and Intel #100

lewtun commented Mar 17, 2022 •

edited

Loading

lewtun Mar 17, 2022

lewtun Mar 17, 2022

lewtun Mar 17, 2022 •

edited

Loading

echarlaix Mar 22, 2022

lewtun Mar 22, 2022

echarlaix Mar 22, 2022

echarlaix Mar 22, 2022

lewtun Mar 22, 2022

echarlaix Mar 23, 2022

lewtun Mar 23, 2022

echarlaix commented Mar 22, 2022

michaelbenayoun Mar 23, 2022

lewtun Mar 23, 2022

lewtun Mar 23, 2022


		## IncOptimizer

		[[autodoc]] intel.neural_compressor.optimizer.IncOptimizer

Add API docs for ONNX Runtime and Intel #100

Add API docs for ONNX Runtime and Intel #100

Conversation

lewtun commented Mar 17, 2022 • edited Loading

What does this PR do?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewtun Mar 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

echarlaix commented Mar 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewtun commented Mar 17, 2022 •

edited

Loading

lewtun Mar 17, 2022 •

edited

Loading