MKLDNN header missing in mxnet-cu102==1.7.0 pip package

[jasperzhong]

I encountered an compilation error showing that <mkldnn.h> is not found when I tried to compile BytePS with mxnet-cu102==1.7.0. I checked the mxnet package and indeed there is no MKLDNN header.

However, interestingly I found that it supports MKLDNN. And later I found MKLDNN is enabled by default in 1.7.0. #16899

So I suggest including the headers in the pip package.

[bartekkuncer]

Hi @vycezhong,
I saw that there was simillar issue in the past: #18120, which has been fixed here: #18310 and #18355. I will check why this issue is occurring again.
Thanks for reporting.

[bartekkuncer]

I have checked that the fix has been applied to branch v1.7.x but not to the tag 1.7.0 which apparently is used to build the pip package.

[bartekkuncer]

Please try using: https://repo.mxnet.io/dist/python/cu102/mxnet_cu102-1.7.0b20200813-py2.py3-none-manylinux2014_x86_64.whl. I believe it has all necessary headers.

[jasperzhong]

Please try using: https://repo.mxnet.io/dist/python/cu102/mxnet_cu102-1.7.0b20200813-py2.py3-none-manylinux2014_x86_64.whl. I believe it has all necessary headers.

Thanks for your help.

[bartekkuncer]

Please try using: https://repo.mxnet.io/dist/python/cu102/mxnet_cu102-1.7.0b20200813-py2.py3-none-manylinux2014_x86_64.whl. I believe it has all necessary headers.

Thanks for your help.

@vycezhong I understand that the package I proposed works for you? :)

@szha @leezu Should we release patch v1.7.1 which will fix the lacking headers or just wait for official minor v1.8.0 release?

[jasperzhong]

Yes it works for me. Thanks.

[szha]

@bartekkuncer I think we should. At least it should be fixed with a post-release

[bartekkuncer]

@bartekkuncer I think we should. At least it should be fixed with a post-release

@szha Ok, I will look into it. What is the post-release flow?

[szha]

@bartekkuncer it's just about repackaging and releasing on PyPI as another version. Since there is some ongoing license issue that needs to be sorted out first, we are putting changes to PyPI changes on hold. Will revisit once the issue is resolved.

[EnricoMi]

Is there an ETA on releasing v1.7.1 or v1.7.0.post1 for cu101 and cu102?

[bartekkuncer]

Is there an ETA on releasing v1.7.1 or v1.7.0.post1 for cu101 and cu102?

@szha

[szha]

Thanks for the ping. We resolved the license issue and I can get to it this week.

[EnricoMi]

Awesome, thanks for picking it up!

[szha]

Patched wheels are now available:
https://pypi.org/project/mxnet-cu101/#files
https://pypi.org/project/mxnet-cu102/#files

[EnricoMi]

@szha thanks for the cu101 and cu102 releases, is a release with MKLDNN headers planned for mxnet as well? Version mxnet-1.7.0.post1 does not include headers.

[EnricoMi]

With mxnet-cu101==1.7.0.post0, I get the following error:

    In file included from /usr/local/lib/python3.8/dist-packages/mxnet/include/mkldnn/mkldnn.hpp:24:0,
                     from /usr/local/lib/python3.8/dist-packages/mxnet/include/mxnet/ndarray.h:41,
                     from /tmp/pip-req-build-n16mfgkd/horovod/mxnet/mpi_ops.h:24,
                     from /tmp/pip-req-build-n16mfgkd/horovod/mxnet/mpi_ops.cc:21:
    /usr/local/lib/python3.8/dist-packages/mxnet/include/mkldnn/dnnl.hpp:23:10: fatal error: dnnl_config.h: No such file or directory

With mxnet-cu101==1.7.0b20200813, everything worked fine. Are there any changes between those two versions that I have to incorporate?

[leezu]

It's a bug in the post0. I'm reopening the issue

[szha]

Sorry, the two dnnl_* files were missed in post0. I posted 1.7.0.post1 to cu101/cu102 wheels which should include the necessary mkldnn headers (thanks @leezu for verifying). Let me look into including the headers for the CPU versions.

[EnricoMi]

I can confirm 1.7.0.post1 for cu101 works for Horovod. Is a release of 1.7.0.post1 for cu110 also planned for the near future?

[szha]

@EnricoMi our 1.8.0 release is in the pipeline so I think it's more likely to have cu110 release for that.

[szha]

I uploaded the post2 versions of the CPU wheels to include mkldnn headers.

[EnricoMi]

@szha that post2 version for CPU works for Horovod, thanks!