Deploying multiple models at the same time will raise MKLDNN error

[juncaipeng]

• Info

C++ API
Ubuntu 16.04
CPU MKLDNN
GCC 8.2.0

• Prepare

Please contact danqing to download the demo. The demo only has two group models for test.

Download model_test.cc.zip, unzip model_test.cc.zip, use the new model_test.cc to update the old model_test.cc file in the demo.

Make and install paddle release2.0 (commit id: c7a6a1f9610a9ee018c19d89950d76b38f33aed1).

cmake -DCMAKE_BUILD_TYPE=Release -DWITH_PYTHON=OFF -DWITH_MKL=ON -DWITH_GPU=OFF -DON_INFER=ON .. && make -j && make inference_lib_dist -j

Set LIB_DIR as the path of PaddleInference in build.sh.

Run sh build.sh.

Run ulimit -c unlimited enable save core file.

• The first test

Run ./build/model_test --test_groups=0 --single_instance=true , it does not raise error.
If set single_instance as true, every model only has one predictor. Otherwise, some models will have several predictors by calling predictor.clone().

Run ./build/model_test --test_groups=0 --single_instance=false, it raises segmentation fault error.
Run gdb ./build/model_test core_file get the following error.

• The second test

Run ./build/model_test --test_groups=1 --single_instance=true , it does not raise error.

Run ./build/model_test --test_groups=1 --single_instance=false, it also raises segmentation fault error.

Sometimes, the above demo raises different error, such as

• The third test

Run ./build/model_test --test_groups="4 5 6" --single_instance=true. The demo loads several group models and every model has one predictor, and it also raises error as following. The demo only has two group models for now, we will provide other models later.

[paddle-bot-old]

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

[jczaja]

@lidanqing-intel , @juncaipeng We are investigating that issue. Currently We reproduced it on develop branch. Candidate fix was made (#32136) and we are testing it now.

[jczaja]

@juncaipeng

1. Could you please test this PR [oneDNN] Candidate fix to #31992 #32136 (develop) if this solves problem for you?
2. We miss some multi-instance unit tests (Improve Multi-threading, multi-instance UT #32087) , would it be possible to turn this issue's test into UT ?

[juncaipeng]

@jczaja
For the latest develop branch, the demo still raises error when run the following command:
./build/model_test --test_groups="0 1" --single_instance=false
Sometimes, the demo also raises error when run the following command:
./build/model_test --test_groups="0" --single_instance=false
./build/model_test --test_groups="1" --single_instance=false

Should I use the inference library of release/2.0?

These models can not be used in UT, so you should find some other models.

[jczaja]

@juncaipeng This is cherry-pick for release/2.0 #32163 . It works fine on my setup but @lidanqing-intel that every other run there is some crash on her setup so I will test it further.

[jczaja]

@juncaipeng I have made some more changes (develop PR: #32309). Could you please test them and report problems if any?

[juncaipeng]

@jczaja I have tested all the models in the demo and don't have problems. The customer will use the new inference library to test in their project. If there's any news, I'll give it back.

[juncaipeng]

@jczaja The customers reported that there is no problem with the new inference library (develop PR: #32309) for now.

[jczaja]

@juncaipeng I have implemented alternative fix : #32499 . That is the one I would like to merge. Could you please test it against this issue

[juncaipeng]

@jczaja The inference library (develop PR: #32499) also passed all tests.

[lidanqing-intel]

@juncaipeng Could this issue be closed

[paddle-bot-old]

Are you satisfied with the resolution of your issue?

YES
No