Add a build mode to the installer to facilitate partition dump #846

padmanarayana · 2017-07-26T17:27:05Z

Add a new "build" mode in addition to the sonic & onie modes in the installer. If the installer is run on a build server, it should create a SONiC partition dump that can be used to 'dd' the partition instead of going thru ONIE or sonic install. This is useful in case a user wants to migrate into SONiC from another 3rd party OS.
Use the installer to create the 8GB blob for Dell's Z9100 & S6100
Once the SONiC boots up the 'firsttime' post a migration from a 3rd party OS, it performs certain initializations from rc.local like reconstructing machine.conf from ONIE.

Note: The installer currently checks the /etc/lsb-release to determine if it's a onie environment. The assumption is that DISTRIB_ID=onie will be present in all the ONIE builds across platforms.

…on in build environment

msftclas · 2017-07-26T17:27:09Z

@padmanarayana,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla.microsoft.com.

It will cover your contributions to all Microsoft-managed open source projects.
Thanks,
Microsoft Pull Request Bot

marian-pritsak · 2017-07-27T12:10:53Z

build_image.sh

+
+    ## Generate a compressed 8GB partition dump that can be used to 'dd' in-lieu of using the onie-nos-installer
+    ## The 'build' install mode of the installer is used to generate this dump.
+    if [[ "$TARGET_MACHINE" == "broadcom" ]] && [[ -n "$DELL_Z9100_PLATFORM_MODULE_VERSION" || -n "$DELL_S6100_PLATFORM_MODULE_VERSION" ]]; then


Why only for this machine?
Can we make it more generic to support any type of machine?

…nstaller build

padmanarayana · 2017-07-28T12:29:10Z

Removed the platform check as requested. Currently, the installer is platform agnostic - so it cannot determine the dump partition size : which is set to 8GB. I presume this would suffice for most platforms.

Resolved

msftclas · 2017-08-01T11:48:03Z

@padmanarayana, thanks for signing the contribution license agreement. We will now validate the agreement and then the pull request.

Thanks, Microsoft Pull Request Bot

lguohan · 2017-08-07T15:22:04Z

installer/x86_64/install.sh


-[ -b "$blk_dev" ] || {
+if [ "$install_env" != "build" ] && [ ! -b "$blk_dev" ]; then
    echo "Error: Unable to determine block device of ONIE install"
    exit 1


move this into previous block?

lguohan · 2017-08-08T01:40:37Z

I like the user command to be make target/sonic-broadcom.raw, and it will generate the dd image you have created. If user only specify target/sonic-broadcom.bin, then you do not need to build the raw image.

By default, the raw image will be gzipped.

Also, you can specify the raw image disk size, default is 8G.

you can add two variables in onie-image.conf

OUTPUT_RAW_IMAGE=target/sonic-$TARGET_MACHINE.raw

# Raw image disk size in MB
RAW_IMAGE_DISK_SIZE=8192

lguohan · 2017-08-08T08:54:43Z

files/image_config/platform/rc.local

+    if [ ! -e /host/machine.conf ]; then
+        onie_dev=$(blkid | grep ONIE-BOOT | head -n 1 | awk '{print $1}' |  sed -e 's/:.*$//')
+        mkdir -p /mnt/onie-boot
+        mount $onie_dev /mnt/onie-boot


mount $onie_dev /mnt/onie-boot [](start = 8, length = 30)

please umount /mnt/onie-boot after done.

lguohan · 2017-08-08T09:05:22Z

can you help to understand why we still have 6.6 space give the /var/log is a 4G file under /host/disk-img/var-log.ext4

admin@str-s6100-acs-3:/host/disk-img$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
root-aufs       7.8G  830M  6.6G  12% /
tmpfs           1.6G   21M  1.6G   2% /run
/dev/sda8       7.8G  830M  6.6G  12% /host
/dev/loop1      3.9G   16M  3.6G   1% /var/log
devtmpfs         10M     0   10M   0% /dev
tmpfs           3.9G  804K  3.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2       120M   12M  100M  11% /mnt/onie-boot

admin@str-s6100-acs-3:/host/disk-img$ ls -lh
total 137M
-rw-r--r-- 1 root root    0 Aug  8 16:48 hehe
-rw-r--r-- 1 root root 4.0G Aug  8 16:50 var-log.ext4

lguohan · 2017-08-08T09:08:56Z

installer/x86_64/install.sh

@@ -559,7 +600,13 @@ $onie_menuentry
 EOF
 fi

-cp $grub_cfg $onie_initrd_tmp/$demo_mnt/grub/grub.cfg
+if [ "$install_env" = "build" ]; then
+    cp $grub_cfg $demo_mnt/$image_dir/platform/grub.cfg.migration


what is your plan for this grub.cfg.migration? I do not see it is used anywhere.

Removed all changes pertaining to grub.cfg.migration since /host/machine.conf can be used instead.

The migration flag was meant to be used to install the grub for SONiC post migration. Since machine.conf can be used, it is now removed.

lguohan · 2017-08-08T09:13:30Z

if [ "$IMAGE_TYPE" = "onie" ]; then

if $IMAGE_TYPE == "onie" or $IMAGE_TYPE == "raw"; then

Refers to: build_image.sh:17 in 9c6221f. [](commit_id = 9c6221f, deletion_comment = False)

lguohan · 2017-08-08T09:15:38Z

build_image.sh

@@ -35,6 +35,18 @@ if [ "$IMAGE_TYPE" = "onie" ]; then
    ./onie-mk-demo.sh $TARGET_PLATFORM $TARGET_MACHINE $TARGET_PLATFORM-$TARGET_MACHINE-$ONIEIMAGE_VERSION \
          installer platform/$TARGET_MACHINE/platform.conf $OUTPUT_ONIE_IMAGE OS $IMAGE_VERSION $ONIE_IMAGE_PART_SIZE \
          $ONIE_INSTALLER_PAYLOAD
+


in line 17, $IMAGE_TYPE = "onie" or $IMAGE_TYPE = "raw"

Introduced IMAGE_TYPE "raw".

lguohan · 2017-08-08T09:17:23Z

build_image.sh

@@ -35,6 +35,18 @@ if [ "$IMAGE_TYPE" = "onie" ]; then
    ./onie-mk-demo.sh $TARGET_PLATFORM $TARGET_MACHINE $TARGET_PLATFORM-$TARGET_MACHINE-$ONIEIMAGE_VERSION \
          installer platform/$TARGET_MACHINE/platform.conf $OUTPUT_ONIE_IMAGE OS $IMAGE_VERSION $ONIE_IMAGE_PART_SIZE \
          $ONIE_INSTALLER_PAYLOAD
+
+    ## Generate a compressed 8GB partition dump that can be used to 'dd' in-lieu of using the onie-nos-installer


if IMAGE_TYPE == "raw", then after we build the onie image, we need to install it in a raw device. Here, in the build_image.sh, we should use fallocate to create the disk file and in onie installer, we only do installation in the disk file, after that, we should do gzip here in the build_image.sh.

Done. Note that the build of sonic-broadcom.raw will build the sonic-broadcom.bin and then use it to build the sonic-broadcom.raw.

lguohan · 2017-08-08T09:18:02Z

installer/x86_64/install.sh

-cp $grub_cfg $onie_initrd_tmp/$demo_mnt/grub/grub.cfg
+if [ "$install_env" = "build" ]; then
+    cp $grub_cfg $demo_mnt/$image_dir/platform/grub.cfg.migration
+    gzip /tmp/sonic-${TARGET_MACHINE}_8GB_dd.img


do the gzip in the build_image.sh

lguohan · 2017-08-08T09:28:58Z

installer/x86_64/install.sh

+# to facilitate migration from a 3rd party OS into SONiC
+if [ "$install_env" = "build" ]; then
+   touch $demo_mnt/$image_dir/platform/migration
+fi


I prefer not to introduce this migration flag. In the rc.local, if it is the first time, check if there is /host/machine.conf or not first, if there is no /host/machine.conf, then mount onie and copy the file. We do not really need the migration flag here.

Removed migration flag.

lguohan

as comments

padmanarayana · 2017-08-21T13:03:33Z

can you help to understand why we still have 6.6 space give the /var/log is a 4G file under /host/disk->> img/var-log.ext4

This seems to be the expected behaviour as shown in the test on my build server:

root@ubuntu-16:/work#
root@ubuntu-16:/work# df -lh | grep 'work|test'
/dev/sdb2 20G 44M 19G 1% /work
root@ubuntu-16:/work# pwd
/work
root@ubuntu-16:/work# ls -lt
total 16
drwx------ 2 root root 16384 Aug 15 08:56 lost+found
root@ubuntu-16:/work# dd if=/dev/zero of=4G.fs bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 15.6807 s, 274 MB/s
root@ubuntu-16:/work# df -lh | grep 'work|test'
/dev/sdb2 20G 4.1G 15G 22% /work
root@ubuntu-16:/work# mkfs.ext4 -q 4G.fs -F
root@ubuntu-16:/work#
root@ubuntu-16:/work# df -lh | grep 'work|test'
/dev/sdb2 20G 174M 19G 1% /work
root@ubuntu-16:/work# ls -lt
total 132288
-rw-r--r-- 1 root root 4294967296 Aug 18 02:01 4G.fs
drwx------ 2 root root 16384 Aug 15 08:56 lost+found
root@ubuntu-16:/work# mkdir test_mnt
root@ubuntu-16:/work# mount -t auto -o loop 4G.fs test_mnt
root@ubuntu-16:/work# df -lh | grep 'work|test'
/dev/sdb2 20G 174M 19G 1% /work
/dev/loop0 3.9G 8.0M 3.6G 1% /work/test_mnt
root@ubuntu-16:/work#
root@ubuntu-16:/work# cd test_mnt
root@ubuntu-16:/work/test_mnt# df -lh | grep 'work|test'
/dev/sdb2 20G 174M 19G 1% /work
/dev/loop0 3.9G 8.0M 3.6G 1% /work/test_mnt
root@ubuntu-16:/work/test_mnt#
root@ubuntu-16:/work/test_mnt#
root@ubuntu-16:/work/test_mnt# dd if=/dev/urandom of=my_big_file.bin bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 139.81 s, 15.4 MB/s
root@ubuntu-16:/work/test_mnt#
root@ubuntu-16:/work/test_mnt#
root@ubuntu-16:/work/test_mnt#
root@ubuntu-16:/work/test_mnt# df -lh | grep 'work|test'
/dev/sdb2 20G 2.1G 17G 12% /work
/dev/loop0 3.9G 2.1G 1.6G 56% /work/test_mnt
root@ubuntu-16:/work/test_mnt#

Note that once a child ext4 partition (4G.fs) is carved out of the parent, the parent does not immediately account for the child's partition size as used. However, once the child partition usage increases, it gets accounted both in the child as well as the parent appropriately.

padmanarayana · 2017-08-21T13:04:59Z

Please refer to #901

* d0f8091 2020-03-22 | Revert "add support for MCLAG (sonic-net#453)" (sonic-net#849) (HEAD -> master, origin/master, origin/HEAD) [lguohan] * 6f54e8c 2020-03-22 | Revert "return list for _get_optional_services() (sonic-net#822)" (sonic-net#848) [lguohan] * f1c79d5 2020-03-22 | return list for _get_optional_services() (sonic-net#822) (HEAD -> master, origin/master, origin/HEAD) [shine4chen] * 28ea21a 2020-03-21 | Fix kernel panic for irq after fast-reboot (sonic-net#823) [byu343] * 727b499 2020-03-22 | [decode-syseeprom] fix getattribute check for sime platforms (sonic-net#835) [Mykola F] * db78cb6 2020-03-21 | Update Command Reference with sFlow section (sonic-net#841) [padmanarayana] * 780673c 2020-03-21 | explicitly specify command with underscores (sonic-net#846) [lguohan] * 07dc201 2020-03-21 | [db_migrator]Do DB migration for buffer pool size change on Mellanox platform (sonic-net#833) [Kebo Liu] * 9a94955 2020-03-20 | [sonic_installer] Enable ARM64 arch (sonic-net#811) [arheneus@marvell.com] * 92b30c2 2020-03-18 | [config]: add syslog messages to config load_minigraph/reload (sonic-net#843) [lguohan] * 4389ffe 2020-03-17 | [intfutil] set speed to 0 when interface speed is not available (sonic-net#839) [Ying Xie] * 45c6c68 2020-03-17 | [Mellanox] add document for thermal control related cli (sonic-net#832) [Junchao-Mellanox] * 7105400 2020-03-12 | Add kdump support for Aboot platforms (sonic-net#824) [byu343] * c5c5ffc 2020-03-01 | [fwutil]: Set default socket timeout for FW download to 30 sec. (sonic-net#821) [Nazarii Hnydyn] * 81c5930 2020-03-01 | Update config/show to include PFC Watchdog commands (sonic-net#736) [Andriy Moroz] * 66e9dfb 2020-02-28 | [MultiDB] sonic-utilities - replace redis-cli/redis-dump with sonic-db-cli/sonic-db-dump (sonic-net#810) [Dong Zhang] * 8aea564 2020-02-24 | add support for MCLAG (sonic-net#453) [shine4chen] * 118620f 2020-02-23 | [reboot] make sure the reboot happens even if platform reboot failed (sonic-net#819) [Ying Xie] * 40eff82 2020-02-22 | Multi-Db changes for NAT feature. (sonic-net#818) [Akhilesh Samineni] * a4cb4dd 2020-02-21 | [Command-Reference.md] Unify Usage statments and Examples (including sample prompts) (sonic-net#816) [Joe LeVeque] Signed-off-by: Guohan Lu <lguohan@gmail.com>

Signed-off-by: Shu0T1an ChenG <shuche@microsoft.com>

* [submodule]: update sonic-utilities * d0f8091 2020-03-22 | Revert "add support for MCLAG (#453)" (#849) (HEAD -> master, origin/master, origin/HEAD) [lguohan] * 6f54e8c 2020-03-22 | Revert "return list for _get_optional_services() (#822)" (#848) [lguohan] * f1c79d5 2020-03-22 | return list for _get_optional_services() (#822) (HEAD -> master, origin/master, origin/HEAD) [shine4chen] * 28ea21a 2020-03-21 | Fix kernel panic for irq after fast-reboot (#823) [byu343] * 727b499 2020-03-22 | [decode-syseeprom] fix getattribute check for sime platforms (#835) [Mykola F] * db78cb6 2020-03-21 | Update Command Reference with sFlow section (#841) [padmanarayana] * 780673c 2020-03-21 | explicitly specify command with underscores (#846) [lguohan] * 07dc201 2020-03-21 | [db_migrator]Do DB migration for buffer pool size change on Mellanox platform (#833) [Kebo Liu] * 9a94955 2020-03-20 | [sonic_installer] Enable ARM64 arch (#811) [arheneus@marvell.com] * 92b30c2 2020-03-18 | [config]: add syslog messages to config load_minigraph/reload (#843) [lguohan] * 4389ffe 2020-03-17 | [intfutil] set speed to 0 when interface speed is not available (#839) [Ying Xie] * 45c6c68 2020-03-17 | [Mellanox] add document for thermal control related cli (#832) [Junchao-Mellanox] * 7105400 2020-03-12 | Add kdump support for Aboot platforms (#824) [byu343] * c5c5ffc 2020-03-01 | [fwutil]: Set default socket timeout for FW download to 30 sec. (#821) [Nazarii Hnydyn] * 81c5930 2020-03-01 | Update config/show to include PFC Watchdog commands (#736) [Andriy Moroz] * 66e9dfb 2020-02-28 | [MultiDB] sonic-utilities - replace redis-cli/redis-dump with sonic-db-cli/sonic-db-dump (#810) [Dong Zhang] * 8aea564 2020-02-24 | add support for MCLAG (#453) [shine4chen] * 118620f 2020-02-23 | [reboot] make sure the reboot happens even if platform reboot failed (#819) [Ying Xie] * 40eff82 2020-02-22 | Multi-Db changes for NAT feature. (#818) [Akhilesh Samineni] * a4cb4dd 2020-02-21 | [Command-Reference.md] Unify Usage statments and Examples (including sample prompts) (#816) [Joe LeVeque] Signed-off-by: Guohan Lu <lguohan@gmail.com>

[fwutil]: Use overlay driver when mounting next image filesystem (#825) Fix for adding L3 interface to Vlan group (#826)Fix for adding L3 interface to Vlan group (#826) [db_migrator]Do DB migration for buffer pool size change on Mellanox platform (#833) explicitly specify command with underscores (#846) [intfutil] set speed to 0 when interface speed is not available (#839)

[fwutil]: Use overlay driver when mounting next image filesystem (sonic-net#825) Fix for adding L3 interface to Vlan group (sonic-net#826)Fix for adding L3 interface to Vlan group (sonic-net#826) [db_migrator]Do DB migration for buffer pool size change on Mellanox platform (sonic-net#833) explicitly specify command with underscores (sonic-net#846) [intfutil] set speed to 0 when interface speed is not available (sonic-net#839)

Starting click 7.0. The default behavior of a command with under scores will be replace with dashes. this is to address the above default behavior change, so that the command remains the same. more details can be found: pallets/click#1123

6cfb3ecb0248768da0a91e5f7fb4477c5da7eb4e (HEAD -> 201911, origin/201911) [build]: allow to use extra inc/lib location to build the package (#595) 40d34872d3b7f354adac67f084eebf6ee467f779 Merge pull request #846 from xumia/azp-201911 76ac50f147a7d820b19d8d7628a67f2fe4f5159b Disable the build test 6c9cf655b8b5b152cab1d578e05eddf8238b81b0 Fix branch reference error ca8d81d37a9b0294098f161b036d330d9ff461e0 [ci]: download artifacts from master branch (#768) 0cbf4d55c67a9f8f52715f95536f3588acf06c4a [ci]: use sonicbld pool (#766) b6f1265ee9bd86f8a5e909a6f1e9b2384497c906 [ci]: add build for arm64 and armhf (#757) 9ec0a7da64d479b124815edc5b505fb88b2532a0 CI: add azure pipeline CI/CD (#754) 1436dbe02cd3c56f796c6b3398d4075cd05d97e0 Fix RIF issue (#835) Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>

Add a build mode to the installer to facilitate partition dump creati…

db8c34a

…on in build environment

msftclas added the cla-required label Jul 26, 2017

marian-pritsak previously requested changes Jul 27, 2017

View reviewed changes

Remove platform check while generating the partition dump post onie i…

9c6221f

…nstaller build

msftclas added cla-signed and removed cla-required labels Aug 1, 2017

padmanarayana mentioned this pull request Aug 1, 2017

Script to create the 8GB blob that can be used in lieu of onie installer #806

Closed

lguohan reviewed Aug 7, 2017

View reviewed changes

lguohan reviewed Aug 8, 2017

View reviewed changes

lguohan requested changes Aug 8, 2017

View reviewed changes

padmanarayana closed this Aug 21, 2017

lguohan mentioned this pull request Aug 22, 2017

Address review comments for pull request #846 #901

Merged

padmanarayana deleted the sonic_install_env_build_mode branch August 29, 2017 09:43

lguohan mentioned this pull request Mar 22, 2020

[submodule]: update sonic-utilities #4308

Merged

madhanmellanox pushed a commit to madhanmellanox/sonic-buildimage that referenced this pull request Mar 23, 2020

[routeorch]: Move extra log into INFO level (sonic-net#846)

f8be024

Signed-off-by: Shu0T1an ChenG <shuche@microsoft.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a build mode to the installer to facilitate partition dump #846

Add a build mode to the installer to facilitate partition dump #846

padmanarayana commented Jul 26, 2017

msftclas commented Jul 26, 2017

marian-pritsak Jul 27, 2017

padmanarayana commented Jul 28, 2017

msftclas commented Aug 1, 2017

lguohan Aug 7, 2017

padmanarayana Aug 21, 2017

lguohan commented Aug 8, 2017 •

edited

Loading

lguohan Aug 8, 2017

padmanarayana Aug 21, 2017

lguohan commented Aug 8, 2017

lguohan Aug 8, 2017

padmanarayana Aug 21, 2017

padmanarayana Aug 21, 2017

lguohan commented Aug 8, 2017 •

edited

Loading

lguohan Aug 8, 2017

padmanarayana Aug 21, 2017

lguohan Aug 8, 2017

padmanarayana Aug 21, 2017

lguohan Aug 8, 2017

lguohan Aug 8, 2017

padmanarayana Aug 21, 2017

lguohan left a comment

padmanarayana commented Aug 21, 2017

padmanarayana commented Aug 21, 2017

Add a build mode to the installer to facilitate partition dump #846

Add a build mode to the installer to facilitate partition dump #846

Conversation

padmanarayana commented Jul 26, 2017

msftclas commented Jul 26, 2017

Choose a reason for hiding this comment

padmanarayana commented Jul 28, 2017

msftclas commented Aug 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan commented Aug 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan commented Aug 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan commented Aug 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan left a comment

Choose a reason for hiding this comment

padmanarayana commented Aug 21, 2017

padmanarayana commented Aug 21, 2017

lguohan commented Aug 8, 2017 •

edited

Loading

lguohan commented Aug 8, 2017 •

edited

Loading