Parallel build are failed on master branch

[iMasaruOki]

Description

I tried to build SONiC with PLATFORM=broadcom and SONIC_CONFIG_BUILD_JOBS = 8 on master branch, it got build failure.
Other platform except for vs has same problem, I think.
Build successful with SONIC_CONFIG_BUILD_JOBS = 1 but build speed are too slow.

Steps to reproduce the issue:
1 edit rules/config, SONIC_CONFIG_BUILD_JOBS = 8
2. make init
3. make configure PLATFORM=broadcom
4. make all

Describe the results you received:

[ 04 ] [ target/debs/stretch/socat_1.7.3.1-2+deb9u1_amd64.deb ]
[ 05 ] [ target/debs/stretch/iptables_1.6.0+snapshot20161117-6_amd64.deb ]
[ 06 ] [ target/debs/stretch/smartmontools_6.6-1_amd64.deb ]
[ 07 ] [ target/python-wheels/redis_dump_load-1.1-py2-none-any.whl ]
[ 08 ] [ target/python-debs/python-sonic-ledd_1.1-1_all.deb ]
[ FAIL LOG START ] [ target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-uninstall ]
target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb does not exist
[  FAIL LOG END  ] [ target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-uninstall ]
slave.mk:504: recipe for target 'target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-uninstall' failed
make: *** [target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-uninstall] Error 1
make: *** Waiting for unfinished jobs....
[ 01 ] [ target/debs/stretch/libnl-3-200_3.5.0-1_amd64.deb ]
[ 02 ] [ target/debs/stretch/swig3.0_3.0.12-2_amd64.deb ]

Describe the results you expected:

build successful.

Additional information you deem important (e.g. issue happens only occasionally):

My build environment: RYZEN 5 3600, Ubuntu 20.04LTS

tested on commit ec0c826
affected by commit 0a260b7 , I think.

[lguohan]

can you provide the log file for this one? libsaivs-dev_1.0.0_amd64.deb-uninstall.log

it should be under target/debs/stretch/

[iMasaruOki]

masaru@tama:~/src/Azure/sonic-buildimage$ cat target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-uninstall.log 
target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb does not exist
masaru@tama:~/src/Azure/sonic-buildimage$ 

[aravindmani-1]

Team,

I've also faced the same issue where the parallel build was failing.
Compilation succeeded only when changing SONIC_CONFIG_BUILD_JOBS=1.

Logs:
[ FAIL LOG START ] [ target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-install ]
Selecting previously unselected package libsaivs-dev.
(Reading database ... 163266 files and directories currently installed.)
Preparing to unpack .../libsaivs-dev_1.0.0_amd64.deb ...
Unpacking libsaivs-dev (1.0.0) ...
dpkg: error processing archive target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb (--install):
trying to overwrite '/usr/include/sai/sai.h', which is also in package libsaibcm-dev 3.7.5.1-1
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb
[ FAIL LOG END ] [ target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-install ]
slave.mk:483: recipe for target 'target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-install' failed
make: *** [target/debs/stretch/libsaivs-dev_1.0.0_amd64.deb-install] Error 1
make: *** Waiting for unfinished jobs....
Makefile.work:224: recipe for target 'stretch' failed
make[1]: *** [stretch] Error 2
make[1]: Leaving directory '/home/am1/sonic/master/sonic-buildimage'
Makefile:23: recipe for target 'stretch' failed
make: *** [stretch] Error 2

[lguohan]

it is likely due to this pr.

The pr that introduce this issue is likely this following PR.

58632e6#diff-54252fe595f09e268f275810298a4beeR494

I did put a lock here. 58632e6#diff-54252fe595f09e268f275810298a4beeR508

So, I haven’t figure out the root cause yet.

[lguohan]

when parallel build is enabled, both docker-fpm-frr and docker-syncd-brcm is built at the same time, docker-fpm-frr requires swss which requires to install libsaivs-dev. docker-syncd-brcm requires syncd package which requires to install libsaibcm-dev.

since libsaivs-dev and libsaibcm-dev install the sai header in the same location, these two packages cannot be installed at the same time. Therefore, we need to serialize the build between these two packages. The previous solution is not good enough. Now, the new solution is to have one package wait for another package to be uninstalled.

For example, if syncd is built first, then it will install libsaibcm-dev. Meanwhile, if the swss build job starts, and tries to install libsaivs-dev, it will first try to query if libsaibcm-dev is installed or not. if it is installed, then it will wait until libsaibcm-dev is uninstalled. After syncd job is finished, it will uninstall libsaibcm-dev and swss build job will be unblocked.

To solve this issue, _UNINSTALLS is introduced to uninstall a package that is no longer needed and to allow parallel job to go on.