SONiC Build Improvements #942
Conversation
Build Improvements
|
|
||
| - When multiple builds are triggered on the same build server that creates parallel building issue because all the build jobs are trying to create the same docker with latest tag. This happens only when sonic dockers are built using native host dockerd for sonic docker image creation. | ||
|
|
||
| - This feature creates all sonic dockers as user sonic dockers and then, whilesaving and loading the user sonic dockers, it rename the user sonic dockers into correct sonic dockers with tag as latest. |
There was a problem hiding this comment.
What if single user builds multiple SONiC branches or same branch different hashes?
There was a problem hiding this comment.
- We will enhance this to use it as a user tag which includes username with SHA value of docker control files(Dockerfile.j2, etc)
- Eg: docker-database-<user>-<SHA>:latest
- Different user tag is generated for a different branch of the same user.
| ### Version components | ||
|
|
||
| - Sonic build downloads lots of component from external web which includes | ||
| - Source code files |
There was a problem hiding this comment.
"Source code files" and "Git source code" - why not make those sources part of the sonic-buildimage repository or submodules?
There was a problem hiding this comment.
- These source codes are maintained by external users.
- It can undergo a frequent update or release.
- Maintaining these source codes as part of SONiC repo, might add extra overhead.
| - Prebuilt debian packages | ||
| - Python PIP packages | ||
| - Git source code | ||
| - Docker images |
There was a problem hiding this comment.
Docker image caching. Docker supports build cache which is disabled from day one, do you plan to enable it?
There was a problem hiding this comment.
- Currently, it is not supported.
- Will try to enable SONIC_USE_DOCKER_BUILDKIT and validate it for docker image layer caching.
| - Bootstrap generation | ||
| - ROOTFS installation | ||
| - SONiC packages installation |
There was a problem hiding this comment.
Step 3 can be split into two parts:
Step 3.1 General packages, such as curl, vim, sudo, python3, ...
Step 3.1 Sonic relative packages, some of the targets built or copied from sonic repo.
Step 1 & step 3.1 can be generated a base image, when can be run in parallel with the other targets, before build image step.
Benifits:
- High hit rate, for less dependencies.
- Reduce the cache size.
- Improve the concurrency when cache not hit, the step has small dependencies, can be run with any other steps.
But it needs to do some code change, move some build code, to build a new target.
There was a problem hiding this comment.
Thanks for the input, updated the document.
Review comments addressed
|
@Kalimuthu-Velappan can you please update the Code PR list? Current one I think is out of date and below is the new list. Thanks. sonic-net/sonic-buildimage#10352 merged |
@Kalimuthu-Velappan kindly ping.. |
|
Update the Code PR list.
#941
…On Thu, Dec 8, 2022 at 4:33 AM Yanzhao Zhang ***@***.***> wrote:
@Kalimuthu-Velappan <https://github.com/Kalimuthu-Velappan> can you
please update the Code PR list? Current one I think is out of date and
below is the new list. Thanks.
sonic-net/sonic-buildimage#10352
<sonic-net/sonic-buildimage#10352> merged
sonic-net/sonic-buildimage#12000
<sonic-net/sonic-buildimage#12000> merged
sonic-net/sonic-buildimage#12001
<sonic-net/sonic-buildimage#12001>
sonic-net/sonic-buildimage#12003
<sonic-net/sonic-buildimage#12003>
sonic-net/sonic-buildimage#12004
<sonic-net/sonic-buildimage#12004>
sonic-net/sonic-buildimage#12005
<sonic-net/sonic-buildimage#12005>
@Kalimuthu-Velappan <https://github.com/Kalimuthu-Velappan> kindly ping..
—
Reply to this email directly, view it on GitHub
<#942 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AM2UC2TBACRICDC3HIZSAF3WMEJS7ANCNFSM5OBZZN6Q>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
This electronic communication and the information and any files transmitted
with it, or attached to it, are confidential and are intended solely for
the use of the individual or entity to whom it is addressed and may contain
information that is confidential, legally privileged, protected by privacy
laws, or otherwise restricted from disclosure to anyone else. If you are
not the intended recipient or the person responsible for delivering the
e-mail to the intended recipient, you are hereby notified that any use,
copying, distributing, dissemination, forwarding, printing, or copying of
this e-mail is strictly prohibited. If you received this e-mail in error,
please return the e-mail to the sender, delete it from your computer, and
destroy any printed copy of it.
|
|
The code PRs need be updated to the proper ones. @Kalimuthu-Velappan @adyeung |
Back porting of version caching Code PRs into 202211 branch sonic-net/SONiC#942 1. Restructuring of Makefile.work 2. Build cache framework 3. Environment settings 4. Debug framework 5. Docker cache framework
Build Improvements
Build cache framework
Environment settings
Debug framework
Docker cache framework
wget caching support
PIP caching support