[Build] Support j2 template for debian sources #12557

xumia · 2022-10-31T07:45:16Z

Why I did it

Unify the Debian mirror sources
Make easy to upgrade to the next Debian release, not source url code change required.
Support to customize the Debian mirror sources during the build

Relative issue: #12523

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

kv-y · 2022-10-31T13:52:27Z

files/apt/sources.list.j2

+{% if DISTRIBUTION == 'stretch' %}
+deb {{ mirror_url }} {{ DISTRIBUTION }}/updates main contrib non-free
+deb-src {{ mirror_url }} {{ DISTRIBUTION }}/updates main contrib non-free
+{% else %}
+deb {{ mirror_url }} {{ DISTRIBUTION }}-security main contrib non-free
+deb-src {{ mirror_url }} {{ DISTRIBUTION }}-security main contrib non-free
+{% endif %}


Format was changed in bullseye: https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#security-archive
Buster also has format {{ DISTRIBUTION }}/updates for security mirrors.

@kv-y , fixed, thanks for you comment.

kv-y · 2022-10-31T14:43:27Z

scripts/build_mirror_config.sh

+TEMPALTE=files/apt/sources.list.j2
+[ -f files/apt/sources.list.$ARCHITECTURE.j2 ] && TEMPALTE=files/apt/sources.list.$ARCHITECTURE.j2
+[ -f $CONFIG_PATH/sources.list.j2 ] && TEMPALTE=$CONFIG_PATH/sources.list.j2
+[ -f $CONFIG_PATH/sources.list.$ARCHITECTURE.j2 ] && TEMPALTE=$CONFIG_PATH/sources.list.$ARCHITECTURE.j2
+
+j2 $TEMPALTE > $CONFIG_PATH/sources.list.$ARCHITECTURE


Fixed, thanks.

kv-y · 2022-10-31T15:47:03Z

scripts/build_mirror_config.sh

+DEFAULT_MIRROR_URLS=http://debian-archive.trafficmanager.net/debian/,http://cdn1.packages.trafficmanager.net/snapshot/debian/latest/
+DEFAULT_MIRROR_SECURITY_URLS=http://debian-archive.trafficmanager.net/debian-security/,http://cdn1.packages.trafficmanager.net/snapshot/debian-security/latest/


Need fix for repr build?
Now REPR_MIRROR_URL_PATTERN='http://packages.trafficmanager.net/debian'

Yes, we need to support repr build, fixed, thanks.

xumia · 2022-11-05T05:40:48Z

@kv-y , do you have any more comments?

xumia · 2022-11-05T05:41:35Z

@qiluo-msft , could you please help review it? Thanks.

kv-y · 2022-11-05T09:50:57Z

@xumia

Could you please retest a customization of Debian mirrors?
I mean something like this:

MIRROR_URLS="http://deb.debian.org/debian/,http://packages.trafficmanager.net/debian/debian/" MIRROR_SECURITY_URLS="http://deb.debian.org/debian-security,http://packages.trafficmanager.net/debian/debian-security/" make configure PLATFORM=vs

MIRROR_URLS="http://deb.debian.org/debian/,http://packages.trafficmanager.net/debian/debian/" MIRROR_SECURITY_URLS="http://deb.debian.org/debian-security,http://packages.trafficmanager.net/debian/debian-security/" make target/sonic-vs.img.gz

I got correct sources.lists for sonic-slaves.
But for docker-base-xxx and for rootfs (files/apt) I got debian-archive.trafficmanager.net instead of expected deb.debian.org.

I think you forgot to pass these variables from Makefile.work to slave.mk.

kv-y · 2022-11-06T16:57:22Z

scripts/build_mirror_config.sh

+# The debian-archive.trafficmanager.net does not support armhf, use debian.org instead
+if [ "$ARCHITECTURE" == "armhf" ]; then
+    DEFAULT_MIRROR_URLS=http://deb.debian.org/debian/,http://packages.trafficmanager.net/debian/debian/
+    DEFAULT_MIRROR_SECURITY_URLS=http://security.debian.org/,http://packages.trafficmanager.net/debian/debian-security/


I think we can use here http://deb.debian.org/debian-security/ instead of http://security.debian.org/ for unification.

There are multiple variants:
security.debian.org
security.debian.org/debian-security
deb.debian.org/debian-security
and all work.

Related question: https://unix.stackexchange.com/questions/387006/sources-list-repository-url-for-debian-security-updates-with-or-without-debian

Documentation recommends to use deb.debian.org/debian-security or security.debian.org/debian-security.
Official debian docker images use deb.debian.org/debian-security since buster.

Fixed, "deb.debian.org/debian-security" is better than "security.debian.org".

Using the URLs as below:

http://deb.debian.org/debian/ http://deb.debian.org/debian-security/

kv-y · 2022-11-06T21:35:11Z

Also I tried to test repr build with this patch, but seem like repr build is broken for master.
Probably need this fix in master branch.

xumia · 2022-11-07T01:54:12Z

@xumia

Could you please retest a customization of Debian mirrors? I mean something like this:

MIRROR_URLS="http://deb.debian.org/debian/,http://packages.trafficmanager.net/debian/debian/" MIRROR_SECURITY_URLS="http://deb.debian.org/debian-security,http://packages.trafficmanager.net/debian/debian-security/" make configure PLATFORM=vs

MIRROR_URLS="http://deb.debian.org/debian/,http://packages.trafficmanager.net/debian/debian/" MIRROR_SECURITY_URLS="http://deb.debian.org/debian-security,http://packages.trafficmanager.net/debian/debian-security/" make target/sonic-vs.img.gz

I got correct sources.lists for sonic-slaves. But for docker-base-xxx and for rootfs (files/apt) I got debian-archive.trafficmanager.net instead of expected deb.debian.org.

I think you forgot to pass these variables from Makefile.work to slave.mk.

Fixed, thanks, missing to pass the variables.

xumia · 2022-11-07T01:57:54Z

Also I tried to test repr build with this patch, but seem like repr build is broken for master. Probably need this fix in master branch.

The repr only works on the release branches, such as 202012, 202111, 202205, etc, not on master branch today, but we have a plan to support it.

…hanged (#12702) Why I did it After #12557 sources.lists are generated. So we need to recalculate SLAVE_BASE_TAG if mirrors were changed. Also we need to rebuild DPKG cache in this case. How I did it Use generated sources.list for SLAVE_BASE_TAG Add MIRROR_URLS and MIRROR_SECURITY_URLS to SONIC_COMMON_FLAGS_LIST

…hanged (sonic-net#12702) Why I did it After sonic-net#12557 sources.lists are generated. So we need to recalculate SLAVE_BASE_TAG if mirrors were changed. Also we need to rebuild DPKG cache in this case. How I did it Use generated sources.list for SLAVE_BASE_TAG Add MIRROR_URLS and MIRROR_SECURITY_URLS to SONIC_COMMON_FLAGS_LIST

Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: #12523 How I did it How to verify it

Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: sonic-net#12523

commit 69f74304e2c848d4ca4c7767c7b0b228f5351de6 Author: xumia <59720581+xumia@users.noreply.github.com> Date: Wed Nov 9 08:09:53 2022 +0800 [Build] Support j2 template for debian sources (sonic-net#12557) Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: sonic-net#12523

Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: sonic-net#12523

commit 69f74304e2c848d4ca4c7767c7b0b228f5351de6 Author: xumia <59720581+xumia@users.noreply.github.com> Date: Wed Nov 9 08:09:53 2022 +0800 [Build] Support j2 template for debian sources (sonic-net#12557) Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: sonic-net#12523

Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: sonic-net#12523

…lity (#14663) Why I did it Cherry-pick commits from master to support the snapshot based mirror, and fix the code conflicts. ad162ae [Build] Optimize the version control for Debian packages (#14557) 38c5d7f [Build] Support j2 template for debian sources for docker ptf (#13198) 5e4826e [Ci] Support to use the same snapshot for all platform builds (#13913) 8206925 [Build] Change the default mirror version config file (#13786) 5e4a866 [Build] Support Debian snapshot mirror to improve build stability (#13097) ac5d89c [Build] Support j2 template for debian sources (#12557) Work item tracking Microsoft ADO (number only): 18018114 How I did it How to verify it

…lity (#14664) Why I did it Cherry-pick commits from master to support the snapshot based mirror, and fix the code conflicts. ad162ae [Build] Optimize the version control for Debian packages (#14557) 38c5d7f [Build] Support j2 template for debian sources for docker ptf (#13198) 5e4826e [Ci] Support to use the same snapshot for all platform builds (#13913) 8206925 [Build] Change the default mirror version config file (#13786) 5e4a866 [Build] Support Debian snapshot mirror to improve build stability (#13097) ac5d89c [Build] Support j2 template for debian sources (#12557) Work item tracking Microsoft ADO (number only): 18018114 How I did it How to verify it

Why I did it Unify the Debian mirror sources Make easy to upgrade to the next Debian release, not source url code change required. Support to customize the Debian mirror sources during the build Relative issue: sonic-net#12523

…lity (#14558) #### Why I did it Cherry-pick commits from master to support the snapshot based mirror, and fix the code conflicts. And add the last commit to fix the build broken issue according to the mirror change. ad162ae [Build] Optimize the version control for Debian packages (#14557) 38c5d7f [Build] Support j2 template for debian sources for docker ptf (#13198) 5e4826e [Ci] Support to use the same snapshot for all platform builds (#13913) 8206925 [Build] Change the default mirror version config file (#13786) 5e4a866 [Build] Support Debian snapshot mirror to improve build stability (#13097) ac5d89c [Build] Support j2 template for debian sources (#12557)

xumia added the Build label Oct 31, 2022

xumia requested review from qiluo-msft and lguohan as code owners October 31, 2022 07:45

xumia force-pushed the uniq-debian-sources branch from 989b275 to dc38589 Compare October 31, 2022 09:29

xumia marked this pull request as draft October 31, 2022 12:26

kv-y reviewed Oct 31, 2022

View reviewed changes

xumia force-pushed the uniq-debian-sources branch from 9eb8215 to d436286 Compare November 2, 2022 04:27

xumia mentioned this pull request Nov 2, 2022

Add the debian mirror template to sonic base docker image #12523

Closed

kv-y reviewed Nov 6, 2022

View reviewed changes

xumia added 6 commits November 7, 2022 01:44

Support j2 template for debian sources

e634846

Fix mirror url issue

1ddd72a

Pretty print

7e78024

Fix comment issue in the script

e1a042c

Remove duplicate empty lines

967cc26

Pass the mirror url relative parameters to slave

50eebf4

xumia force-pushed the uniq-debian-sources branch from d2d798f to 50eebf4 Compare November 7, 2022 02:22

xumia marked this pull request as ready for review November 7, 2022 03:50

kv-y approved these changes Nov 7, 2022

View reviewed changes

xumia requested review from liushilongbuaa and saiarcot895 November 8, 2022 01:40

richardyu-ms approved these changes Nov 8, 2022

View reviewed changes

xumia merged commit ac5d89c into sonic-net:master Nov 9, 2022

xumia deleted the uniq-debian-sources branch November 9, 2022 00:09

kv-y mentioned this pull request Nov 14, 2022

[Build] Update SLAVE_BASE_TAG and DPKG cache if Debian mirrors were changed #12702

Merged

xumia added a commit to xumia/sonic-buildimage-1 that referenced this pull request Dec 28, 2022

[Build] Support j2 template for debian sources (sonic-net#12557)

54f0272

xumia mentioned this pull request Apr 7, 2023

[Build][202012] Support Debian snapshot mirror to improve build stability #14558

Merged

8 tasks

xumia mentioned this pull request Apr 14, 2023

[Build][202106] Support Debian snapshot mirror to improve build stability #14663

Merged

10 tasks

xumia mentioned this pull request Apr 14, 2023

[Build][202111] Support Debian snapshot mirror to improve build stability #14664

Merged

10 tasks

xumia mentioned this pull request Apr 14, 2023

Support sn 202012 3 #14666

Draft

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build] Support j2 template for debian sources #12557

[Build] Support j2 template for debian sources #12557

xumia commented Oct 31, 2022 •

edited

Loading

kv-y Oct 31, 2022

xumia Nov 1, 2022

kv-y Oct 31, 2022

xumia Nov 1, 2022

kv-y Oct 31, 2022

xumia Nov 1, 2022

xumia commented Nov 5, 2022

xumia commented Nov 5, 2022

kv-y commented Nov 5, 2022 •

edited

Loading

kv-y Nov 6, 2022 •

edited

Loading

xumia Nov 7, 2022 •

edited

Loading

kv-y commented Nov 6, 2022

xumia commented Nov 7, 2022

xumia commented Nov 7, 2022

		DEFAULT_MIRROR_URLS=http://debian-archive.trafficmanager.net/debian/,http://cdn1.packages.trafficmanager.net/snapshot/debian/latest/
		DEFAULT_MIRROR_SECURITY_URLS=http://debian-archive.trafficmanager.net/debian-security/,http://cdn1.packages.trafficmanager.net/snapshot/debian-security/latest/

[Build] Support j2 template for debian sources #12557

[Build] Support j2 template for debian sources #12557

Conversation

xumia commented Oct 31, 2022 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

kv-y Oct 31, 2022

Choose a reason for hiding this comment

xumia Nov 1, 2022

Choose a reason for hiding this comment

kv-y Oct 31, 2022

Choose a reason for hiding this comment

xumia Nov 1, 2022

Choose a reason for hiding this comment

kv-y Oct 31, 2022

Choose a reason for hiding this comment

xumia Nov 1, 2022

Choose a reason for hiding this comment

xumia commented Nov 5, 2022

xumia commented Nov 5, 2022

kv-y commented Nov 5, 2022 • edited Loading

kv-y Nov 6, 2022 • edited Loading

Choose a reason for hiding this comment

xumia Nov 7, 2022 • edited Loading

Choose a reason for hiding this comment

kv-y commented Nov 6, 2022

xumia commented Nov 7, 2022

xumia commented Nov 7, 2022

xumia commented Oct 31, 2022 •

edited

Loading

kv-y commented Nov 5, 2022 •

edited

Loading

kv-y Nov 6, 2022 •

edited

Loading

xumia Nov 7, 2022 •

edited

Loading