From 5650762f2c12b6228f4a1b643a7728f418f2d31f Mon Sep 17 00:00:00 2001 From: ganglv <88995770+ganglyu@users.noreply.github.com> Date: Sat, 17 Sep 2022 06:08:10 +0800 Subject: [PATCH 01/43] Fix dhcp option buffer issue (#12033) Why I did it Current isc-dhcp uses below code to remove DHCP option: memmove(sp, op, op[1] + 2); sp += op[1] + 2; sp points to the option to be stripped, we can call it as option S. op points to the option after options S, we can call it as option O. DHCP option is a typical type-length-value structure, the first byte is type, the second byte is length, and remain parts are value. In this case, option O length is bigger than option S, and more than 2 bytes, after the memmove, we will get this result: Now Option S and Option O are overwritten, op[1] was the length of Option O, and it's modified after memmove. But current implementation is still using op[1] as length to update sp (sp+=op[1]+2), so we get the wrong sp. How I did it Create patch from https://github.com/isc-projects/dhcp The new impelementation use mlen to store the length of Option O before memmove, that's how it fixed the bug. size_t mlen = op[1] + 2; memmove(sp, op, mlen); sp += mlen; How to verify it I have a PR for sonic-mgmt to cover this issue: sonic-net/sonic-mgmt#6330 Signed-off-by: Gang Lv ganglv@microsoft.com --- ...ay-agent-option-buffer-pointer-logic.patch | 53 +++++++++++++++++++ src/isc-dhcp/patch/series | 1 + 2 files changed, 54 insertions(+) create mode 100644 src/isc-dhcp/patch/0013-Fix-dhcrelay-agent-option-buffer-pointer-logic.patch diff --git a/src/isc-dhcp/patch/0013-Fix-dhcrelay-agent-option-buffer-pointer-logic.patch b/src/isc-dhcp/patch/0013-Fix-dhcrelay-agent-option-buffer-pointer-logic.patch new file mode 100644 index 000000000000..051b58966cae --- /dev/null +++ b/src/isc-dhcp/patch/0013-Fix-dhcrelay-agent-option-buffer-pointer-logic.patch @@ -0,0 +1,53 @@ +From 0a2f9a62bceb90b0d30461add2e25c4ce7a24547 Mon Sep 17 00:00:00 2001 +From: Thomas Markwalder +Date: Fri, 20 Dec 2019 10:11:54 -0500 +Subject: [PATCH] [#71] Fix dhcrelay agent option buffer pointer logic + +relay/dhcrelay.c + strip_relay_agent_options() + strip_relay_agent_options() + - corrected buffer pointer logic + +--- + relay/dhcrelay.c | 18 ++++++++++++++---- + 1 file changed, 14 insertions(+), 4 deletions(-) + +diff --git a/relay/dhcrelay.c b/relay/dhcrelay.c +index 896e1e2e..980dacae 100644 +--- a/relay/dhcrelay.c ++++ b/relay/dhcrelay.c +@@ -1238,8 +1238,13 @@ strip_relay_agent_options(struct interface_info *in, + return (0); + + if (sp != op) { +- memmove(sp, op, op[1] + 2); +- sp += op[1] + 2; ++ size_t mlen = op[1] + 2; ++ memmove(sp, op, mlen); ++ sp += mlen; ++ if (sp > max) { ++ return (0); ++ } ++ + op = nextop; + } else + op = sp = nextop; +@@ -1620,8 +1620,13 @@ add_relay_agent_options(struct interface_info *ip, struct dhcp_packet *packet, + end_pad = NULL; + + if (sp != op) { +- memmove(sp, op, op[1] + 2); +- sp += op[1] + 2; ++ size_t mlen = op[1] + 2; ++ memmove(sp, op, mlen); ++ sp += mlen; ++ if (sp > max) { ++ return (0); ++ } ++ + op = nextop; + } else + op = sp = nextop; +-- +2.17.1 + diff --git a/src/isc-dhcp/patch/series b/src/isc-dhcp/patch/series index 5397aa0c6e06..b9efee0192f5 100644 --- a/src/isc-dhcp/patch/series +++ b/src/isc-dhcp/patch/series @@ -11,3 +11,4 @@ 0010-Bugfix-correctly-set-interface-netmask.patch 0011-dhcp-relay-Prevent-Buffer-Overrun.patch 0012-add-option-si-to-support-using-src-intf-ip-in-relay.patch +0013-Fix-dhcrelay-agent-option-buffer-pointer-logic.patch From 1effff983610ab38e9b862d4bb4e85d2291c3b94 Mon Sep 17 00:00:00 2001 From: Zhaohui Sun <94606222+ZhaohuiS@users.noreply.github.com> Date: Sat, 17 Sep 2022 13:33:53 +0800 Subject: [PATCH 02/43] Enable system-site-packages for ptf docker and install thrift for test_qos_sai (#12094) Why I did it test_sai_qos failed because of the following error: "stderr_lines": [ "Traceback (most recent call last):", " File \"/usr/bin/ptf\", line 522, in ", " test_modules = load_test_modules(config)", " File \"/usr/bin/ptf\", line 413, in load_test_modules", " mod = imp.load_module(modname, *imp.find_module(modname, [root]))", " File \"saitests/switch.py\", line 19, in ", " import switch_sai_thrift", "ImportError: No module named switch_sai_thrift" ], It's because test_sai_qos runs ptf script which imports switch_sai_thrift, switch_sai_thrift is installed from python-saithrift_0.9.4_amd64.deb. For master image, the deb file is for python3, but ptf only has virtual python3 environment, that's why we add --system-site-packages to allow virtual env to access system site-packeges. Add thrift package in docker ptf virtual python3 env, because currently env-python3 doesn't have thrift module which is needed in switch_sai_thrift. How I did it Enable --system-site-packages for virtual py3 env in ptf docker and install thrift for test_qos_sai How to verify it load and login ptf conatiner dpkg - i python-saithrift_0.9.4_amd64.deb source /root/env-python3/bin/activate python import switch_sai_thrift.switch_sai_rpc Signed-off-by: Zhaohui Sun --- dockers/docker-ptf/Dockerfile.j2 | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/dockers/docker-ptf/Dockerfile.j2 b/dockers/docker-ptf/Dockerfile.j2 index c73d12f3245a..35c85fd1eac4 100644 --- a/dockers/docker-ptf/Dockerfile.j2 +++ b/dockers/docker-ptf/Dockerfile.j2 @@ -136,7 +136,7 @@ RUN rm -rf /debs \ && cd /opt \ && wget https://raw.githubusercontent.com/p4lang/ptf/master/ptf_nn/ptf_nn_agent.py -RUN python3 -m venv env-python3 +RUN python3 -m venv --system-site-packages env-python3 # Activating a virtualenv. The virtualenv automatically works for RUN, ENV and CMD. ENV VIRTUAL_ENV=/root/env-python3 @@ -173,7 +173,8 @@ RUN python3 -m pip install setuptools \ && pip3 install itsdangerous \ && pip3 install retrying \ && pip3 install jinja2 \ - && pip3 install scapy==2.4.5 + && pip3 install scapy==2.4.5 \ + && pip3 install thrift {% if docker_ptf_whls.strip() -%} # Copy locally-built Python wheel dependencies From de68f10923219b5a7c8ecd41c9aa9e3ce6883424 Mon Sep 17 00:00:00 2001 From: "Richard.Yu" Date: Mon, 19 Sep 2022 13:34:38 +0800 Subject: [PATCH 03/43] [submodule] Advance sairedis head (#12098) Advance sairedis head Signed-off-by: richardyu-ms Signed-off-by: richardyu-ms --- src/sonic-sairedis | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sonic-sairedis b/src/sonic-sairedis index 854d54e07fd8..228e37c5e996 160000 --- a/src/sonic-sairedis +++ b/src/sonic-sairedis @@ -1 +1 @@ -Subproject commit 854d54e07fd8eeff9cae0410e8e7f04d6385400b +Subproject commit 228e37c5e9968278280202ba77cb85714ba2d8fa From c1d2e88de971f1dec89ca9fb32f6b4fb6e2ec4b9 Mon Sep 17 00:00:00 2001 From: ganglv <88995770+ganglyu@users.noreply.github.com> Date: Mon, 19 Sep 2022 14:54:08 +0800 Subject: [PATCH 04/43] Replace configuration parameter for gnmi write (#11780) Why I did it Replace configuration parameter for gnmi write, and we will add other gnmi write features in the future. How I did it Update rules/config and other Makefile. How to verify it Build sonic image. --- Makefile.work | 4 ++-- rules/config | 4 ++-- slave.mk | 2 +- src/sonic-gnmi | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Makefile.work b/Makefile.work index 139d15ed2f59..6171a05c5192 100644 --- a/Makefile.work +++ b/Makefile.work @@ -34,7 +34,7 @@ # * Default: yes # * Values: yes, no # * KERNEL_PROCURE_METHOD: Specifying method of obtaining kernel Debian package: download or build -# * TELEMETRY_WRITABLE: Enable write/config operations via the gNMI interface. +# * ENABLE_TRANSLIB_WRITE: Enable translib write/config operations via the gNMI interface. # * Default: unset # * Values: y # * SONIC_DPKG_CACHE_METHOD: Specifying method of obtaining the Debian packages from cache: none or cache @@ -395,7 +395,7 @@ SONIC_BUILD_INSTRUCTION := make \ INCLUDE_MACSEC=$(INCLUDE_MACSEC) \ SONIC_INCLUDE_RESTAPI=$(INCLUDE_RESTAPI) \ SONIC_INCLUDE_MUX=$(INCLUDE_MUX) \ - TELEMETRY_WRITABLE=$(TELEMETRY_WRITABLE) \ + ENABLE_TRANSLIB_WRITE=$(ENABLE_TRANSLIB_WRITE) \ EXTRA_DOCKER_TARGETS=$(EXTRA_DOCKER_TARGETS) \ BUILD_LOG_TIMESTAMP=$(BUILD_LOG_TIMESTAMP) \ SONIC_ENABLE_IMAGE_SIGNATURE=$(ENABLE_IMAGE_SIGNATURE) \ diff --git a/rules/config b/rules/config index 380b28bc7cd5..888c470cdee2 100644 --- a/rules/config +++ b/rules/config @@ -156,9 +156,9 @@ INCLUDE_P4RT = n # ENABLE_AUTO_TECH_SUPPORT - Enable the configuration for event-driven techsupport & coredump mgmt feature ENABLE_AUTO_TECH_SUPPORT = y -# TELEMETRY_WRITABLE - Enable write/config operations via the gNMI interface. +# ENABLE_TRANSLIB_WRITE - Enable translib write/config operations via the gNMI interface. # Uncomment to enable: -# TELEMETRY_WRITABLE = y +# ENABLE_TRANSLIB_WRITE = y # INCLUDE_MACSEC - build docker-macsec for macsec support INCLUDE_MACSEC = y diff --git a/slave.mk b/slave.mk index f720061b2e52..34af3169cfef 100644 --- a/slave.mk +++ b/slave.mk @@ -391,7 +391,7 @@ $(info "INCLUDE_MUX" : "$(INCLUDE_MUX)") $(info "INCLUDE_BOOTCHART : "$(INCLUDE_BOOTCHART)") $(info "ENABLE_BOOTCHART : "$(ENABLE_BOOTCHART)") $(info "ENABLE_FIPS_FEATURE" : "$(ENABLE_FIPS_FEATURE)") -$(info "TELEMETRY_WRITABLE" : "$(TELEMETRY_WRITABLE)") +$(info "ENABLE_TRANSLIB_WRITE" : "$(ENABLE_TRANSLIB_WRITE)") $(info "ENABLE_AUTO_TECH_SUPPORT" : "$(ENABLE_AUTO_TECH_SUPPORT)") $(info "PDDF_SUPPORT" : "$(PDDF_SUPPORT)") $(info "MULTIARCH_QEMU_ENVIRON" : "$(MULTIARCH_QEMU_ENVIRON)") diff --git a/src/sonic-gnmi b/src/sonic-gnmi index 14f91214fe2b..437fc357fd95 160000 --- a/src/sonic-gnmi +++ b/src/sonic-gnmi @@ -1 +1 @@ -Subproject commit 14f91214fe2b4ba7575bffff58989dbf7df71c48 +Subproject commit 437fc357fd95a0b25715b8ab869b8cf8a7a22681 From c243af0cce96432423377eef7a5f8d57eef4e9a8 Mon Sep 17 00:00:00 2001 From: Volodymyr Boiko Date: Mon, 19 Sep 2022 17:25:10 +0300 Subject: [PATCH 05/43] [bgp][service] Start bgp service after interfaces-config service (#11827) - Why I did it interfaces-config service restarts networking service, during the restart loopback interface address is being removed and reassigned back, leaving loopback without an ipv4 address for a while. On SONiC startup and config reload interfaces-config and bgp services start in parallel and sometimes fpmsyncd in bgp attempts bind to loopback while it does not have an address, fails with the log Exception "Cannot assign requested address" had been thrown in daemon and exits with rc 0. root@sonic:/# supervisorctl status fpmsyncd EXITED Jul 20 05:04 AM zebra RUNNING pid 35, uptime 6:15:05 zsocket EXITED Jul 20 05:04 AM docker logs bgp INFO exited: fpmsyncd (exit status 0; expected) With fpmsyncd dead, configured routes do not appear in the database. - How I did it Added ordering dependency on interfaces-config service into bgp.config - How to verify it Itself the issue reproduces quite rarely, but one can gain the time interval between networking down and networking up in interfaces-config.sh like this: diff --git a/files/image_config/interfaces/interfaces-config.sh b/files/image_config/interfaces/interfaces-config.sh index f6aa4147a..87caceeff 100755 --- a/files/image_config/interfaces/interfaces-config.sh +++ b/files/image_config/interfaces/interfaces-config.sh @@ -63,7 +63,11 @@ done # Read sysctl conf files again sysctl -p /etc/sysctl.d/90-dhcp6-systcl.conf -systemctl restart networking +# systemctl restart networking + +systemctl start networking +sleep 10 +systemctl stop networking # Clean-up created files rm -f /tmp/ztp_input.json /tmp/ztp_port_data.json with this change the issue reproduces on every config reload. Signed-off-by: Volodymyr Boyko --- files/build_templates/per_namespace/bgp.service.j2 | 1 + 1 file changed, 1 insertion(+) diff --git a/files/build_templates/per_namespace/bgp.service.j2 b/files/build_templates/per_namespace/bgp.service.j2 index 0c9f01fe8b68..61eb71c4db9a 100644 --- a/files/build_templates/per_namespace/bgp.service.j2 +++ b/files/build_templates/per_namespace/bgp.service.j2 @@ -7,6 +7,7 @@ After=updategraph.service BindsTo=sonic.target After=sonic.target Before=ntp-config.service +After=interfaces-config.service StartLimitIntervalSec=1200 StartLimitBurst=3 From e662008f724882f07e985863712389e1861a60ce Mon Sep 17 00:00:00 2001 From: Stepan Blyshchak <38952541+stepanblyschak@users.noreply.github.com> Date: Mon, 19 Sep 2022 19:34:33 +0300 Subject: [PATCH 06/43] [services] kill container on stop in warm/fast mode (#10510) - Why I did it To optimize stop on warm boot. - How I did it Added kill for containers --- files/build_templates/docker_image_ctl.j2 | 13 ++- .../build_templates/mgmt-framework.service.j2 | 6 +- .../per_namespace/database.service.j2 | 6 +- files/build_templates/snmp.service.j2 | 6 +- .../build_templates/sonic_debian_extension.j2 | 4 + files/build_templates/telemetry.service.j2 | 6 +- files/scripts/database.sh | 1 + files/scripts/mgmt-framework.sh | 1 + files/scripts/radv.sh | 85 +------------------ files/scripts/service_mgmt.sh | 80 +++++++++++++++++ files/scripts/snmp.sh | 1 + files/scripts/telemetry.sh | 1 + 12 files changed, 113 insertions(+), 97 deletions(-) create mode 120000 files/scripts/database.sh create mode 120000 files/scripts/mgmt-framework.sh mode change 100755 => 120000 files/scripts/radv.sh create mode 100755 files/scripts/service_mgmt.sh create mode 120000 files/scripts/snmp.sh create mode 120000 files/scripts/telemetry.sh diff --git a/files/build_templates/docker_image_ctl.j2 b/files/build_templates/docker_image_ctl.j2 index 296b04dca69a..b4ff4a4379c9 100644 --- a/files/build_templates/docker_image_ctl.j2 +++ b/files/build_templates/docker_image_ctl.j2 @@ -594,6 +594,17 @@ stop() { {%- endif %} } +kill() { + {%- if docker_container_name == "database" %} + docker kill $DOCKERNAME + if [ "$DEV" ]; then + ip netns delete "$NET_NS" + fi + {%- else %} + /usr/local/bin/container kill $DOCKERNAME + {%- endif %} +} + DOCKERNAME={{docker_container_name}} OP=$1 DEV=$2 # namespace/device number to operate on @@ -621,7 +632,7 @@ fi [ -f /etc/sonic/sonic-environment ] && . /etc/sonic/sonic-environment case "$1" in - start|wait|stop) + start|wait|stop|kill) $1 ;; *) diff --git a/files/build_templates/mgmt-framework.service.j2 b/files/build_templates/mgmt-framework.service.j2 index 1eb983a897fb..ff99afe62bc1 100644 --- a/files/build_templates/mgmt-framework.service.j2 +++ b/files/build_templates/mgmt-framework.service.j2 @@ -8,6 +8,6 @@ Before=ntp-config.service [Service] User={{ sonicadmin_user }} -ExecStartPre=/usr/bin/{{docker_container_name}}.sh start -ExecStart=/usr/bin/{{docker_container_name}}.sh wait -ExecStop=/usr/bin/{{docker_container_name}}.sh stop +ExecStartPre=/usr/local/bin/{{docker_container_name}}.sh start +ExecStart=/usr/local/bin/{{docker_container_name}}.sh wait +ExecStop=/usr/local/bin/{{docker_container_name}}.sh stop diff --git a/files/build_templates/per_namespace/database.service.j2 b/files/build_templates/per_namespace/database.service.j2 index c42995d2054c..c5c8f267d972 100644 --- a/files/build_templates/per_namespace/database.service.j2 +++ b/files/build_templates/per_namespace/database.service.j2 @@ -14,9 +14,9 @@ StartLimitBurst=3 [Service] User=root -ExecStartPre=/usr/bin/{{docker_container_name}}.sh start{% if multi_instance == 'true' %} %i{% endif %} -ExecStart=/usr/bin/{{docker_container_name}}.sh wait{% if multi_instance == 'true' %} %i{% endif %} -ExecStop=/usr/bin/{{docker_container_name}}.sh stop{% if multi_instance == 'true' %} %i{% endif %} +ExecStartPre=/usr/local/bin/{{docker_container_name}}.sh start{% if multi_instance == 'true' %} %i{% endif %} +ExecStart=/usr/local/bin/{{docker_container_name}}.sh wait{% if multi_instance == 'true' %} %i{% endif %} +ExecStop=/usr/local/bin/{{docker_container_name}}.sh stop{% if multi_instance == 'true' %} %i{% endif %} RestartSec=30 [Install] diff --git a/files/build_templates/snmp.service.j2 b/files/build_templates/snmp.service.j2 index b27ac347007b..5c753dd651eb 100644 --- a/files/build_templates/snmp.service.j2 +++ b/files/build_templates/snmp.service.j2 @@ -10,7 +10,7 @@ StartLimitIntervalSec=1200 StartLimitBurst=3 [Service] -ExecStartPre=/usr/bin/{{docker_container_name}}.sh start -ExecStart=/usr/bin/{{docker_container_name}}.sh wait -ExecStop=/usr/bin/{{docker_container_name}}.sh stop +ExecStartPre=/usr/local/bin/{{docker_container_name}}.sh start +ExecStart=/usr/local/bin/{{docker_container_name}}.sh wait +ExecStop=/usr/local/bin/{{docker_container_name}}.sh stop RestartSec=30 diff --git a/files/build_templates/sonic_debian_extension.j2 b/files/build_templates/sonic_debian_extension.j2 index 58597bbdbcea..a62685dfbbe7 100644 --- a/files/build_templates/sonic_debian_extension.j2 +++ b/files/build_templates/sonic_debian_extension.j2 @@ -853,6 +853,10 @@ sudo LANG=C cp $SCRIPTS_DIR/bgp.sh $FILESYSTEM_ROOT/usr/local/bin/bgp.sh sudo LANG=C cp $SCRIPTS_DIR/teamd.sh $FILESYSTEM_ROOT/usr/local/bin/teamd.sh sudo LANG=C cp $SCRIPTS_DIR/lldp.sh $FILESYSTEM_ROOT/usr/local/bin/lldp.sh sudo LANG=C cp $SCRIPTS_DIR/radv.sh $FILESYSTEM_ROOT/usr/local/bin/radv.sh +sudo LANG=C cp $SCRIPTS_DIR/database.sh $FILESYSTEM_ROOT/usr/local/bin/database.sh +sudo LANG=C cp $SCRIPTS_DIR/snmp.sh $FILESYSTEM_ROOT/usr/local/bin/snmp.sh +sudo LANG=C cp $SCRIPTS_DIR/telemetry.sh $FILESYSTEM_ROOT/usr/local/bin/telemetry.sh +sudo LANG=C cp $SCRIPTS_DIR/mgmt-framework.sh $FILESYSTEM_ROOT/usr/local/bin/mgmt-framework.sh sudo LANG=C cp $SCRIPTS_DIR/asic_status.sh $FILESYSTEM_ROOT/usr/local/bin/asic_status.sh sudo LANG=C cp $SCRIPTS_DIR/asic_status.py $FILESYSTEM_ROOT/usr/local/bin/asic_status.py diff --git a/files/build_templates/telemetry.service.j2 b/files/build_templates/telemetry.service.j2 index 33affa62a748..ebdd484dc877 100644 --- a/files/build_templates/telemetry.service.j2 +++ b/files/build_templates/telemetry.service.j2 @@ -10,7 +10,7 @@ StartLimitBurst=3 [Service] User={{ sonicadmin_user }} -ExecStartPre=/usr/bin/{{docker_container_name}}.sh start -ExecStart=/usr/bin/{{docker_container_name}}.sh wait -ExecStop=/usr/bin/{{docker_container_name}}.sh stop +ExecStartPre=/usr/local/bin/{{docker_container_name}}.sh start +ExecStart=/usr/local/bin/{{docker_container_name}}.sh wait +ExecStop=/usr/local/bin/{{docker_container_name}}.sh stop RestartSec=30 diff --git a/files/scripts/database.sh b/files/scripts/database.sh new file mode 120000 index 000000000000..ce97295f0364 --- /dev/null +++ b/files/scripts/database.sh @@ -0,0 +1 @@ +service_mgmt.sh \ No newline at end of file diff --git a/files/scripts/mgmt-framework.sh b/files/scripts/mgmt-framework.sh new file mode 120000 index 000000000000..ce97295f0364 --- /dev/null +++ b/files/scripts/mgmt-framework.sh @@ -0,0 +1 @@ +service_mgmt.sh \ No newline at end of file diff --git a/files/scripts/radv.sh b/files/scripts/radv.sh deleted file mode 100755 index 1047808e4e44..000000000000 --- a/files/scripts/radv.sh +++ /dev/null @@ -1,84 +0,0 @@ -#!/bin/bash - -function debug() -{ - /usr/bin/logger $1 - /bin/echo `date` "- $1" >> ${DEBUGLOG} -} - -function check_warm_boot() -{ - SYSTEM_WARM_START=`$SONIC_DB_CLI STATE_DB hget "WARM_RESTART_ENABLE_TABLE|system" enable` - SERVICE_WARM_START=`$SONIC_DB_CLI STATE_DB hget "WARM_RESTART_ENABLE_TABLE|${SERVICE}" enable` - if [[ x"$SYSTEM_WARM_START" == x"true" ]] || [[ x"$SERVICE_WARM_START" == x"true" ]]; then - WARM_BOOT="true" - else - WARM_BOOT="false" - fi -} - -function check_fast_boot () -{ - if [[ $($SONIC_DB_CLI STATE_DB GET "FAST_REBOOT|system") == "1" ]]; then - FAST_BOOT="true" - else - FAST_BOOT="false" - fi -} - -start() { - debug "Starting ${SERVICE}$DEV service..." - - check_warm_boot - check_fast_boot - debug "Warm boot flag: ${SERVICE}$DEV ${WARM_BOOT}." - debug "Fast boot flag: ${SERVICE}$DEV ${FAST_BOOT}." - - # start service docker - /usr/bin/${SERVICE}.sh start $DEV - debug "Started ${SERVICE}$DEV service..." -} - -wait() { - /usr/bin/${SERVICE}.sh wait $DEV -} - -stop() { - debug "Stopping ${SERVICE}$DEV service..." - - check_warm_boot - check_fast_boot - debug "Warm boot flag: ${SERVICE}$DEV ${WARM_BOOT}." - debug "Fast boot flag: ${SERVICE}$DEV ${FAST_BOOT}." - - # For WARM/FAST boot do not perform service stop - if [[ x"$WARM_BOOT" != x"true" ]] && [[ x"$FAST_BOOT" != x"true" ]]; then - /usr/bin/${SERVICE}.sh stop $DEV - debug "Stopped ${SERVICE}$DEV service..." - else - debug "Killing Docker radv..." - /usr/bin/docker kill radv &> /dev/null || debug "Docker radv is not running ($?) ..." - fi -} - -DEV=$2 - -SERVICE="radv" -DEBUGLOG="/tmp/radv-debug$DEV.log" -NAMESPACE_PREFIX="asic" -if [ "$DEV" ]; then - NET_NS="$NAMESPACE_PREFIX$DEV" #name of the network namespace - SONIC_DB_CLI="sonic-db-cli -n $NET_NS" -else - SONIC_DB_CLI="sonic-db-cli" -fi - -case "$1" in - start|wait|stop) - $1 - ;; - *) - echo "Usage: $0 {start|wait|stop}" - exit 1 - ;; -esac diff --git a/files/scripts/radv.sh b/files/scripts/radv.sh new file mode 120000 index 000000000000..ce97295f0364 --- /dev/null +++ b/files/scripts/radv.sh @@ -0,0 +1 @@ +service_mgmt.sh \ No newline at end of file diff --git a/files/scripts/service_mgmt.sh b/files/scripts/service_mgmt.sh new file mode 100755 index 000000000000..d400c8472246 --- /dev/null +++ b/files/scripts/service_mgmt.sh @@ -0,0 +1,80 @@ +#!/bin/bash + +function debug() +{ + /usr/bin/logger $1 + /bin/echo `date` "- $1" >> ${DEBUGLOG} +} + +function check_warm_boot() +{ + SYSTEM_WARM_START=`$SONIC_DB_CLI STATE_DB hget "WARM_RESTART_ENABLE_TABLE|system" enable` + SERVICE_WARM_START=`$SONIC_DB_CLI STATE_DB hget "WARM_RESTART_ENABLE_TABLE|${SERVICE}" enable` + if [[ x"$SYSTEM_WARM_START" == x"true" ]] || [[ x"$SERVICE_WARM_START" == x"true" ]]; then + WARM_BOOT="true" + else + WARM_BOOT="false" + fi +} + +function check_fast_boot () +{ + if [[ $($SONIC_DB_CLI STATE_DB GET "FAST_REBOOT|system") == "1" ]]; then + FAST_BOOT="true" + else + FAST_BOOT="false" + fi +} + +start() { + debug "Starting ${SERVICE}$DEV service..." + + # start service docker + /usr/bin/${SERVICE}.sh start $DEV + debug "Started ${SERVICE}$DEV service..." +} + +wait() { + /usr/bin/${SERVICE}.sh wait $DEV +} + +stop() { + debug "Stopping ${SERVICE}$DEV service..." + + check_warm_boot + check_fast_boot + debug "Warm boot flag: ${SERVICE}$DEV ${WARM_BOOT}." + debug "Fast boot flag: ${SERVICE}$DEV ${FAST_BOOT}." + + # For WARM/FAST boot do not perform service stop + if [[ x"$WARM_BOOT" != x"true" ]] && [[ x"$FAST_BOOT" != x"true" ]]; then + /usr/bin/${SERVICE}.sh stop $DEV + debug "Stopped ${SERVICE}$DEV service..." + else + debug "Killing Docker ${SERVICE}${DEV}..." + /usr/bin/${SERVICE}.sh kill $DEV + fi +} + +DEV=$2 + +SCRIPT_NAME=$(basename -- "$0") +SERVICE="${SCRIPT_NAME%.*}" +DEBUGLOG="/tmp/$SERVICE-debug$DEV.log" +NAMESPACE_PREFIX="asic" +if [ "$DEV" ]; then + NET_NS="$NAMESPACE_PREFIX$DEV" #name of the network namespace + SONIC_DB_CLI="sonic-db-cli -n $NET_NS" +else + SONIC_DB_CLI="sonic-db-cli" +fi + +case "$1" in + start|wait|stop) + $1 + ;; + *) + echo "Usage: $0 {start|wait|stop}" + exit 1 + ;; +esac diff --git a/files/scripts/snmp.sh b/files/scripts/snmp.sh new file mode 120000 index 000000000000..ce97295f0364 --- /dev/null +++ b/files/scripts/snmp.sh @@ -0,0 +1 @@ +service_mgmt.sh \ No newline at end of file diff --git a/files/scripts/telemetry.sh b/files/scripts/telemetry.sh new file mode 120000 index 000000000000..ce97295f0364 --- /dev/null +++ b/files/scripts/telemetry.sh @@ -0,0 +1 @@ +service_mgmt.sh \ No newline at end of file From f30fc7627883c3ac150d712519e652a1157fdee9 Mon Sep 17 00:00:00 2001 From: Dror Prital <76714716+dprital@users.noreply.github.com> Date: Tue, 20 Sep 2022 10:22:33 +0300 Subject: [PATCH 07/43] Remove jinja2_cache (#11996) - Why I did it As part of Persistent log level HLD , LOGLEVEL_DB content is moved to CONFIG_DB. In addition, it was decided to remove jinja2_cache which currently appear on LOGLEVEL_DB This cache was added to speed up template rendering in start scripts. There were a lot of them rendered during system start. This caused a delay in warm boot LAG restore time. It was tested and verified that with and without the cache we don't see any difference in this timing now. It is probably due to a lot of other optimizations done to sonic-cfggen. Since there is no noticeable improvement made by j2 cache now it is safe to remove it. - How I did it Remove redis_bcc.py file and and remove the bytcode_cache from sonic-sfggen - How to verify it Warm boot was tested with \ without this jinja2_cache and it there is no difference in performance --- src/sonic-config-engine/redis_bcc.py | 29 ---------------------------- src/sonic-config-engine/setup.py | 1 - src/sonic-config-engine/sonic-cfggen | 6 ++---- 3 files changed, 2 insertions(+), 34 deletions(-) delete mode 100644 src/sonic-config-engine/redis_bcc.py diff --git a/src/sonic-config-engine/redis_bcc.py b/src/sonic-config-engine/redis_bcc.py deleted file mode 100644 index c9d3ebca239f..000000000000 --- a/src/sonic-config-engine/redis_bcc.py +++ /dev/null @@ -1,29 +0,0 @@ -import jinja2 - -from base64 import b64encode, b64decode - -class RedisBytecodeCache(jinja2.BytecodeCache): - """ A bytecode cache for jinja2 template that stores bytecode in Redis """ - - REDIS_HASH = 'JINJA2_CACHE' - - def __init__(self, client): - self._client = client - try: - self._client.connect(self._client.LOGLEVEL_DB, retry_on=False) - except Exception: - self._client = None - - def load_bytecode(self, bucket): - if self._client is None: - return - code = self._client.get(self._client.LOGLEVEL_DB, self.REDIS_HASH, bucket.key) - if code is not None: - bucket.bytecode_from_string(b64decode(code.encode())) - - def dump_bytecode(self, bucket): - if self._client is None: - return - self._client.set(self._client.LOGLEVEL_DB, self.REDIS_HASH, - bucket.key, b64encode(bucket.bytecode_to_string()).decode()) - diff --git a/src/sonic-config-engine/setup.py b/src/sonic-config-engine/setup.py index aa340995b60b..3f2b2a240a95 100644 --- a/src/sonic-config-engine/setup.py +++ b/src/sonic-config-engine/setup.py @@ -43,7 +43,6 @@ 'minigraph', 'openconfig_acl', 'portconfig', - 'redis_bcc', ] if sys.version_info.major == 3: # Python 3-only modules diff --git a/src/sonic-config-engine/sonic-cfggen b/src/sonic-config-engine/sonic-cfggen index ed7c3b616f3c..d5358f633dbf 100755 --- a/src/sonic-config-engine/sonic-cfggen +++ b/src/sonic-config-engine/sonic-cfggen @@ -31,10 +31,9 @@ from config_samples import generate_sample_config, get_available_config from functools import partial from minigraph import minigraph_encoder, parse_xml, parse_device_desc_xml, parse_asic_sub_role, parse_asic_switch_type from portconfig import get_port_config, get_breakout_mode -from redis_bcc import RedisBytecodeCache from sonic_py_common.multi_asic import get_asic_id_from_name, get_asic_device_id, is_multi_asic from sonic_py_common import device_info -from swsscommon.swsscommon import SonicV2Connector, ConfigDBConnector, SonicDBConfig, ConfigDBPipeConnector +from swsscommon.swsscommon import ConfigDBConnector, SonicDBConfig, ConfigDBPipeConnector PY3x = sys.version_info >= (3, 0) @@ -241,8 +240,7 @@ def _get_jinja2_env(paths): Retreive Jinj2 env used to render configuration templates """ loader = jinja2.FileSystemLoader(paths) - redis_bcc = RedisBytecodeCache(SonicV2Connector(host='127.0.0.1')) - env = jinja2.Environment(loader=loader, trim_blocks=True, bytecode_cache=redis_bcc) + env = jinja2.Environment(loader=loader, trim_blocks=True) env.filters['sort_by_port_index'] = sort_by_port_index env.filters['ipv4'] = is_ipv4 env.filters['ipv6'] = is_ipv6 From 63c14d2e9e7a6b9e7da3185b34c75e12de1061b0 Mon Sep 17 00:00:00 2001 From: Junhua Zhai Date: Tue, 20 Sep 2022 20:04:14 +0800 Subject: [PATCH 08/43] [PikeZ] Update port alias in Arista-720DT-48S (#12086) Fix #12037, by following HLD https://github.com/sonic-net/SONiC/blob/master/doc/sonic-port-name.md. --- .../Arista-720DT-48S/port_config.ini | 104 +++++++++--------- .../x86_64-arista_720dt_48s/platform.json | 104 +++++++++--------- 2 files changed, 104 insertions(+), 104 deletions(-) diff --git a/device/arista/x86_64-arista_720dt_48s/Arista-720DT-48S/port_config.ini b/device/arista/x86_64-arista_720dt_48s/Arista-720DT-48S/port_config.ini index 938f5588c59e..dfec6e5642f4 100644 --- a/device/arista/x86_64-arista_720dt_48s/Arista-720DT-48S/port_config.ini +++ b/device/arista/x86_64-arista_720dt_48s/Arista-720DT-48S/port_config.ini @@ -1,53 +1,53 @@ # name lanes alias index speed -Ethernet0 25 Ethernet1 1 1000 -Ethernet1 26 Ethernet2 2 1000 -Ethernet2 27 Ethernet3 3 1000 -Ethernet3 28 Ethernet4 4 1000 -Ethernet4 29 Ethernet5 5 1000 -Ethernet5 30 Ethernet6 6 1000 -Ethernet6 31 Ethernet7 7 1000 -Ethernet7 32 Ethernet8 8 1000 -Ethernet8 33 Ethernet9 9 1000 -Ethernet9 34 Ethernet10 10 1000 -Ethernet10 35 Ethernet11 11 1000 -Ethernet11 36 Ethernet12 12 1000 -Ethernet12 37 Ethernet13 13 1000 -Ethernet13 38 Ethernet14 14 1000 -Ethernet14 39 Ethernet15 15 1000 -Ethernet15 40 Ethernet16 16 1000 -Ethernet16 41 Ethernet17 17 1000 -Ethernet17 42 Ethernet18 18 1000 -Ethernet18 43 Ethernet19 19 1000 -Ethernet19 44 Ethernet20 20 1000 -Ethernet20 45 Ethernet21 21 1000 -Ethernet21 46 Ethernet22 22 1000 -Ethernet22 47 Ethernet23 23 1000 -Ethernet23 48 Ethernet24 24 1000 -Ethernet24 1 Ethernet25 25 1000 -Ethernet25 2 Ethernet26 26 1000 -Ethernet26 3 Ethernet27 27 1000 -Ethernet27 4 Ethernet28 28 1000 -Ethernet28 5 Ethernet29 29 1000 -Ethernet29 6 Ethernet30 30 1000 -Ethernet30 7 Ethernet31 31 1000 -Ethernet31 8 Ethernet32 32 1000 -Ethernet32 9 Ethernet33 33 1000 -Ethernet33 10 Ethernet34 34 1000 -Ethernet34 11 Ethernet35 35 1000 -Ethernet35 12 Ethernet36 36 1000 -Ethernet36 13 Ethernet37 37 1000 -Ethernet37 14 Ethernet38 38 1000 -Ethernet38 15 Ethernet39 39 1000 -Ethernet39 16 Ethernet40 40 1000 -Ethernet40 17 Ethernet41 41 1000 -Ethernet41 18 Ethernet42 42 1000 -Ethernet42 19 Ethernet43 43 1000 -Ethernet43 20 Ethernet44 44 1000 -Ethernet44 21 Ethernet45 45 1000 -Ethernet45 22 Ethernet46 46 1000 -Ethernet46 23 Ethernet47 47 1000 -Ethernet47 24 Ethernet48 48 1000 -Ethernet48 61 Ethernet49 49 10000 -Ethernet49 62 Ethernet50 50 10000 -Ethernet50 63 Ethernet51 51 10000 -Ethernet51 64 Ethernet52 52 10000 +Ethernet0 25 etp1 1 1000 +Ethernet1 26 etp2 2 1000 +Ethernet2 27 etp3 3 1000 +Ethernet3 28 etp4 4 1000 +Ethernet4 29 etp5 5 1000 +Ethernet5 30 etp6 6 1000 +Ethernet6 31 etp7 7 1000 +Ethernet7 32 etp8 8 1000 +Ethernet8 33 etp9 9 1000 +Ethernet9 34 etp10 10 1000 +Ethernet10 35 etp11 11 1000 +Ethernet11 36 etp12 12 1000 +Ethernet12 37 etp13 13 1000 +Ethernet13 38 etp14 14 1000 +Ethernet14 39 etp15 15 1000 +Ethernet15 40 etp16 16 1000 +Ethernet16 41 etp17 17 1000 +Ethernet17 42 etp18 18 1000 +Ethernet18 43 etp19 19 1000 +Ethernet19 44 etp20 20 1000 +Ethernet20 45 etp21 21 1000 +Ethernet21 46 etp22 22 1000 +Ethernet22 47 etp23 23 1000 +Ethernet23 48 etp24 24 1000 +Ethernet24 1 etp25 25 1000 +Ethernet25 2 etp26 26 1000 +Ethernet26 3 etp27 27 1000 +Ethernet27 4 etp28 28 1000 +Ethernet28 5 etp29 29 1000 +Ethernet29 6 etp30 30 1000 +Ethernet30 7 etp31 31 1000 +Ethernet31 8 etp32 32 1000 +Ethernet32 9 etp33 33 1000 +Ethernet33 10 etp34 34 1000 +Ethernet34 11 etp35 35 1000 +Ethernet35 12 etp36 36 1000 +Ethernet36 13 etp37 37 1000 +Ethernet37 14 etp38 38 1000 +Ethernet38 15 etp39 39 1000 +Ethernet39 16 etp40 40 1000 +Ethernet40 17 etp41 41 1000 +Ethernet41 18 etp42 42 1000 +Ethernet42 19 etp43 43 1000 +Ethernet43 20 etp44 44 1000 +Ethernet44 21 etp45 45 1000 +Ethernet45 22 etp46 46 1000 +Ethernet46 23 etp47 47 1000 +Ethernet47 24 etp48 48 1000 +Ethernet48 61 etp49 49 10000 +Ethernet49 62 etp50 50 10000 +Ethernet50 63 etp51 51 10000 +Ethernet51 64 etp52 52 10000 diff --git a/device/arista/x86_64-arista_720dt_48s/platform.json b/device/arista/x86_64-arista_720dt_48s/platform.json index 625e64335278..ce405b82f955 100644 --- a/device/arista/x86_64-arista_720dt_48s/platform.json +++ b/device/arista/x86_64-arista_720dt_48s/platform.json @@ -247,7 +247,7 @@ "lanes": "25", "breakout_modes": { "1x1G": [ - "Ethernet1" + "etp1" ] } }, @@ -256,7 +256,7 @@ "lanes": "26", "breakout_modes": { "1x1G": [ - "Ethernet2" + "etp2" ] } }, @@ -265,7 +265,7 @@ "lanes": "27", "breakout_modes": { "1x1G": [ - "Ethernet3" + "etp3" ] } }, @@ -274,7 +274,7 @@ "lanes": "28", "breakout_modes": { "1x1G": [ - "Ethernet4" + "etp4" ] } }, @@ -283,7 +283,7 @@ "lanes": "29", "breakout_modes": { "1x1G": [ - "Ethernet5" + "etp5" ] } }, @@ -292,7 +292,7 @@ "lanes": "30", "breakout_modes": { "1x1G": [ - "Ethernet6" + "etp6" ] } }, @@ -301,7 +301,7 @@ "lanes": "31", "breakout_modes": { "1x1G": [ - "Ethernet7" + "etp7" ] } }, @@ -310,7 +310,7 @@ "lanes": "32", "breakout_modes": { "1x1G": [ - "Ethernet8" + "etp8" ] } }, @@ -319,7 +319,7 @@ "lanes": "33", "breakout_modes": { "1x1G": [ - "Ethernet9" + "etp9" ] } }, @@ -328,7 +328,7 @@ "lanes": "34", "breakout_modes": { "1x1G": [ - "Ethernet10" + "etp10" ] } }, @@ -337,7 +337,7 @@ "lanes": "35", "breakout_modes": { "1x1G": [ - "Ethernet11" + "etp11" ] } }, @@ -346,7 +346,7 @@ "lanes": "36", "breakout_modes": { "1x1G": [ - "Ethernet12" + "etp12" ] } }, @@ -355,7 +355,7 @@ "lanes": "37", "breakout_modes": { "1x1G": [ - "Ethernet13" + "etp13" ] } }, @@ -364,7 +364,7 @@ "lanes": "38", "breakout_modes": { "1x1G": [ - "Ethernet14" + "etp14" ] } }, @@ -373,7 +373,7 @@ "lanes": "39", "breakout_modes": { "1x1G": [ - "Ethernet15" + "etp15" ] } }, @@ -382,7 +382,7 @@ "lanes": "40", "breakout_modes": { "1x1G": [ - "Ethernet16" + "etp16" ] } }, @@ -391,7 +391,7 @@ "lanes": "41", "breakout_modes": { "1x1G": [ - "Ethernet17" + "etp17" ] } }, @@ -400,7 +400,7 @@ "lanes": "42", "breakout_modes": { "1x1G": [ - "Ethernet18" + "etp18" ] } }, @@ -409,7 +409,7 @@ "lanes": "43", "breakout_modes": { "1x1G": [ - "Ethernet19" + "etp19" ] } }, @@ -418,7 +418,7 @@ "lanes": "44", "breakout_modes": { "1x1G": [ - "Ethernet20" + "etp20" ] } }, @@ -427,7 +427,7 @@ "lanes": "45", "breakout_modes": { "1x1G": [ - "Ethernet21" + "etp21" ] } }, @@ -436,7 +436,7 @@ "lanes": "46", "breakout_modes": { "1x1G": [ - "Ethernet22" + "etp22" ] } }, @@ -445,7 +445,7 @@ "lanes": "47", "breakout_modes": { "1x1G": [ - "Ethernet23" + "etp23" ] } }, @@ -454,7 +454,7 @@ "lanes": "48", "breakout_modes": { "1x1G": [ - "Ethernet24" + "etp24" ] } }, @@ -463,7 +463,7 @@ "lanes": "1", "breakout_modes": { "1x1G": [ - "Ethernet25" + "etp25" ] } }, @@ -472,7 +472,7 @@ "lanes": "2", "breakout_modes": { "1x1G": [ - "Ethernet26" + "etp26" ] } }, @@ -481,7 +481,7 @@ "lanes": "3", "breakout_modes": { "1x1G": [ - "Ethernet27" + "etp27" ] } }, @@ -490,7 +490,7 @@ "lanes": "4", "breakout_modes": { "1x1G": [ - "Ethernet28" + "etp28" ] } }, @@ -499,7 +499,7 @@ "lanes": "5", "breakout_modes": { "1x1G": [ - "Ethernet29" + "etp29" ] } }, @@ -508,7 +508,7 @@ "lanes": "6", "breakout_modes": { "1x1G": [ - "Ethernet30" + "etp30" ] } }, @@ -517,7 +517,7 @@ "lanes": "7", "breakout_modes": { "1x1G": [ - "Ethernet31" + "etp31" ] } }, @@ -526,7 +526,7 @@ "lanes": "8", "breakout_modes": { "1x1G": [ - "Ethernet32" + "etp32" ] } }, @@ -535,7 +535,7 @@ "lanes": "9", "breakout_modes": { "1x1G": [ - "Ethernet33" + "etp33" ] } }, @@ -544,7 +544,7 @@ "lanes": "10", "breakout_modes": { "1x1G": [ - "Ethernet34" + "etp34" ] } }, @@ -553,7 +553,7 @@ "lanes": "11", "breakout_modes": { "1x1G": [ - "Ethernet35" + "etp35" ] } }, @@ -562,7 +562,7 @@ "lanes": "12", "breakout_modes": { "1x1G": [ - "Ethernet36" + "etp36" ] } }, @@ -571,7 +571,7 @@ "lanes": "13", "breakout_modes": { "1x1G": [ - "Ethernet37" + "etp37" ] } }, @@ -580,7 +580,7 @@ "lanes": "14", "breakout_modes": { "1x1G": [ - "Ethernet38" + "etp38" ] } }, @@ -589,7 +589,7 @@ "lanes": "15", "breakout_modes": { "1x1G": [ - "Ethernet39" + "etp39" ] } }, @@ -598,7 +598,7 @@ "lanes": "16", "breakout_modes": { "1x1G": [ - "Ethernet40" + "etp40" ] } }, @@ -607,7 +607,7 @@ "lanes": "17", "breakout_modes": { "1x1G": [ - "Ethernet41" + "etp41" ] } }, @@ -616,7 +616,7 @@ "lanes": "18", "breakout_modes": { "1x1G": [ - "Ethernet42" + "etp42" ] } }, @@ -625,7 +625,7 @@ "lanes": "19", "breakout_modes": { "1x1G": [ - "Ethernet43" + "etp43" ] } }, @@ -634,7 +634,7 @@ "lanes": "20", "breakout_modes": { "1x1G": [ - "Ethernet44" + "etp44" ] } }, @@ -643,7 +643,7 @@ "lanes": "21", "breakout_modes": { "1x1G": [ - "Ethernet45" + "etp45" ] } }, @@ -652,7 +652,7 @@ "lanes": "22", "breakout_modes": { "1x1G": [ - "Ethernet46" + "etp46" ] } }, @@ -661,7 +661,7 @@ "lanes": "23", "breakout_modes": { "1x1G": [ - "Ethernet47" + "etp47" ] } }, @@ -670,7 +670,7 @@ "lanes": "24", "breakout_modes": { "1x1G": [ - "Ethernet48" + "etp48" ] } }, @@ -679,7 +679,7 @@ "lanes": "61", "breakout_modes": { "1x10G": [ - "Ethernet49" + "etp49" ] } }, @@ -688,7 +688,7 @@ "lanes": "62", "breakout_modes": { "1x10G": [ - "Ethernet50" + "etp50" ] } }, @@ -697,7 +697,7 @@ "lanes": "63", "breakout_modes": { "1x10G": [ - "Ethernet51" + "etp51" ] } }, @@ -706,7 +706,7 @@ "lanes": "64", "breakout_modes": { "1x10G": [ - "Ethernet52" + "etp52" ] } } From 5ff45c584682ba3f16a21f07d0fdbf720c8b302f Mon Sep 17 00:00:00 2001 From: Samuel Angebault Date: Wed, 21 Sep 2022 08:56:14 +0200 Subject: [PATCH 09/43] Implement ssd_util plugin for Arista products (#11981) Why I did it Some Arista products do not have an SSD but use an eMMC instead. The SsdUtil plugin is therefore extended to support both. How I did it Implemented ssd_util.py platform plugin loaded by ssdutil. This plugin fallback to the default SONiC implementation if the arista one can't be found. How to verify it Run show platform ssdhealth on a product with an eMMC --- device/arista/x86_64-arista_common/plugins/ssd_util.py | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 device/arista/x86_64-arista_common/plugins/ssd_util.py diff --git a/device/arista/x86_64-arista_common/plugins/ssd_util.py b/device/arista/x86_64-arista_common/plugins/ssd_util.py new file mode 100644 index 000000000000..c9ca1e2a722c --- /dev/null +++ b/device/arista/x86_64-arista_common/plugins/ssd_util.py @@ -0,0 +1,5 @@ +# pylint: disable=unused-import +try: + from arista.utils.sonic_ssd import SsdUtil +except ImportError: + from sonic_platform_base.sonic_ssd.ssd_generic import SsdUtil From 8211c850f163c0e25047b550b1e8199f9b3a1e48 Mon Sep 17 00:00:00 2001 From: Liu Shilong Date: Wed, 21 Sep 2022 15:50:30 +0800 Subject: [PATCH 10/43] [ci] Update docker sonic slave pipeline to build slave base docker (#11908) * [ci] Update docker sonic slave pipeline to build slave base docker --- .azure-pipelines/docker-sonic-slave-arm64.yml | 52 ---------- .azure-pipelines/docker-sonic-slave-armhf.yml | 52 ---------- .../docker-sonic-slave-template.yml | 94 +++++-------------- .azure-pipelines/docker-sonic-slave.yml | 44 +++++---- 4 files changed, 52 insertions(+), 190 deletions(-) delete mode 100644 .azure-pipelines/docker-sonic-slave-arm64.yml delete mode 100644 .azure-pipelines/docker-sonic-slave-armhf.yml diff --git a/.azure-pipelines/docker-sonic-slave-arm64.yml b/.azure-pipelines/docker-sonic-slave-arm64.yml deleted file mode 100644 index 6350d8fd3b5d..000000000000 --- a/.azure-pipelines/docker-sonic-slave-arm64.yml +++ /dev/null @@ -1,52 +0,0 @@ -# Starter pipeline -# Start with a minimal pipeline that you can customize to build and deploy your code. -# Add steps that build, run tests, deploy, and more: -# https://aka.ms/yaml -# Build and push sonic-slave-[buster|jessie|stretch] images for amd64/armhf/arm64 -resources: - repositories: - - repository: buildimage - type: github - name: sonic-net/sonic-buildimage - ref: master - endpoint: sonic-net - -schedules: -- cron: "0 8 * * *" - branches: - include: - - master - - 202012 - always: true - -trigger: none -pr: - branches: - include: - - master - paths: - include: - - sonic-slave-jessie - - sonic-slave-stretch - - sonic-slave-buster - - sonic-slave-bullseye - - .azure-pipelines - -parameters: -- name: 'dists' - type: object - default: - - bullseye - - buster - - stretch - -stages: -- stage: Build - jobs: - - ${{ each dist in parameters.dists }}: - - ${{ if contains(variables['Build.DefinitionName'], dist) }}: - - template: docker-sonic-slave-template.yml - parameters: - pool: sonicbld-arm64 - arch: arm64 - dist: ${{ dist }} diff --git a/.azure-pipelines/docker-sonic-slave-armhf.yml b/.azure-pipelines/docker-sonic-slave-armhf.yml deleted file mode 100644 index fcdad5f2cd19..000000000000 --- a/.azure-pipelines/docker-sonic-slave-armhf.yml +++ /dev/null @@ -1,52 +0,0 @@ -# Starter pipeline -# Start with a minimal pipeline that you can customize to build and deploy your code. -# Add steps that build, run tests, deploy, and more: -# https://aka.ms/yaml -# Build and push sonic-slave-[buster|jessie|stretch] images for amd64/armhf/arm64 -resources: - repositories: - - repository: buildimage - type: github - name: sonic-net/sonic-buildimage - ref: master - endpoint: sonic-net - -schedules: -- cron: "0 8 * * *" - branches: - include: - - master - - 202012 - always: true - -trigger: none -pr: - branches: - include: - - master - paths: - include: - - sonic-slave-jessie - - sonic-slave-stretch - - sonic-slave-buster - - sonic-slave-bullseye - - .azure-pipelines - -parameters: -- name: 'dists' - type: object - default: - - bullseye - - buster - - stretch - -stages: -- stage: Build - jobs: - - ${{ each dist in parameters.dists }}: - - ${{ if contains(variables['Build.DefinitionName'], dist) }}: - - template: docker-sonic-slave-template.yml - parameters: - pool: sonicbld-armhf - arch: armhf - dist: ${{ dist }} diff --git a/.azure-pipelines/docker-sonic-slave-template.yml b/.azure-pipelines/docker-sonic-slave-template.yml index 2cf06638a4cc..1a7b983b5c1f 100644 --- a/.azure-pipelines/docker-sonic-slave-template.yml +++ b/.azure-pipelines/docker-sonic-slave-template.yml @@ -10,6 +10,9 @@ parameters: - amd64 - armhf - arm64 +- name: march + type: string + default: '' - name: dist type: string values: @@ -32,89 +35,44 @@ parameters: - sonicbld-armhf jobs: -- job: Build_${{ parameters.dist }}_${{ parameters.arch }} +- job: Build_${{ parameters.dist }}_${{ parameters.march }}${{ parameters.arch }} timeoutInMinutes: 360 + variables: + - template: .azure-pipelines/template-variables.yml@buildimage + - template: .azure-pipelines/azure-pipelines-repd-build-variables.yml@buildimage pool: ${{ parameters.pool }} steps: - template: cleanup.yml - - ${{ if eq(variables['Build.Reason'], 'PullRequest') }}: - - template: template-clean-sonic-slave.yml - - ${{ else }}: - - template: '/.azure-pipelines/template-clean-sonic-slave.yml@buildimage' + - template: .azure-pipelines/template-clean-sonic-slave.yml@buildimage - checkout: self clean: true submodules: recursive + - task: Docker@2 + displayName: Login to ACR + inputs: + command: login + containerRegistry: ${{ parameters.registry_conn }} - bash: | set -ex + image_tag=$(BLDENV=${{ parameters.dist }} make -f Makefile.work showtag PLATFORM=generic PLATFORM_ARCH=${{ parameters.arch }} | grep sonic-slave | tail -n 1) + image_latest=$(echo $(echo $image_tag | awk -F: '{print$1}'):latest) + docker rmi $image_tag || true - SLAVE_DIR=sonic-slave-${{ parameters.dist }} - if [ x${{ parameters.pool }} == x"sonicbld" ]; then - if [ x${{ parameters.arch }} == x"amd64" ]; then - SLAVE_BASE_IMAGE=${SLAVE_DIR} - SLAVE_BASE_IMAGE_UPLOAD=${SLAVE_DIR} - elif [ x${{ parameters.pool }} == x"sonicbld" ]; then - SLAVE_BASE_IMAGE=${SLAVE_DIR}-march-${{ parameters.arch }} - SLAVE_BASE_IMAGE_UPLOAD=${SLAVE_DIR}-march-${{ parameters.arch }} - fi - elif [[ x${{ parameters.pool }} == x"sonicbld-armhf" && x${{ parameters.arch }} == x"armhf" ]]; then - SLAVE_BASE_IMAGE=${SLAVE_DIR} - SLAVE_BASE_IMAGE_UPLOAD=${SLAVE_DIR}-armhf - elif [[ x${{ parameters.pool }} == x"sonicbld-arm64" && x${{ parameters.arch }} == x"arm64" ]]; then - SLAVE_BASE_IMAGE=${SLAVE_DIR} - SLAVE_BASE_IMAGE_UPLOAD=${SLAVE_DIR}-arm64 - else - echo "do not support build ${{ parameters.arch }} on ${{ parameters.pool }}" - exit 1 + if [[ "$(Build.Reason)" =~ [a-zA-Z]*CI ]] && docker pull ${{ parameters.registry_url }}/${image_tag};then + exit 0 fi - if [ x"$(Build.SourceBranchName)" == x"202012" ]; then - BUILD_OPTIONS = 'SONIC_VERSION_CONTROL_COMPONENTS=deb,py2,py3,web,git,docker' + DOCKER_DATA_ROOT_FOR_MULTIARCH=/data/march/docker make configure PLATFORM=generic PLATFORM_ARCH=${{ parameters.arch }} $args || docker image ls $image_tag + if [[ "$(Build.Reason)" == "PullRequest" ]];then + exit 0 fi - tmpfile=$(mktemp) - - echo ${{ parameters.arch }} > .arch - - DOCKER_DATA_ROOT_FOR_MULTIARCH=/data/march/docker BLDENV=${{ parameters.dist }} $(BUILD_OPTIONS) make -f Makefile.work sonic-slave-build | tee $tmpfile - SLAVE_BASE_TAG=$(grep "^Checking sonic-slave-base image:" $tmpfile | awk -F ':' '{print $3}') - SLAVE_TAG=$(grep "^Checking sonic-slave image:" $tmpfile | awk -F ':' '{print $3}') - - mkdir -p target - - docker tag $SLAVE_BASE_IMAGE:$SLAVE_BASE_TAG $REGISTRY_SERVER/$SLAVE_BASE_IMAGE_UPLOAD:latest - docker tag $SLAVE_BASE_IMAGE:$SLAVE_BASE_TAG $REGISTRY_SERVER/$SLAVE_BASE_IMAGE_UPLOAD:$SLAVE_BASE_TAG - if [ "$SLAVE_BASE_IMAGE_UPLOAD" != "$SLAVE_DIR" ]; then - docker tag $SLAVE_BASE_IMAGE:$SLAVE_BASE_TAG $REGISTRY_SERVER/$SLAVE_DIR:latest-${{ parameters.arch }} - docker tag $SLAVE_BASE_IMAGE:$SLAVE_BASE_TAG $REGISTRY_SERVER/$SLAVE_DIR:$SLAVE_BASE_TAG + docker tag ${image_tag} ${REGISTRY_SERVER}/${image_tag} + docker push ${REGISTRY_SERVER}/${image_tag} + if [[ "${{ parameters.arch }}" == "amd64" ]];then + docker tag ${image_tag} ${REGISTRY_SERVER}/${image_latest} + docker push ${REGISTRY_SERVER}/${image_latest} fi - set +x - echo "##vso[task.setvariable variable=VARIABLE_SLAVE_BASE_IMAGE]$SLAVE_BASE_IMAGE_UPLOAD" - echo "##vso[task.setvariable variable=VARIABLE_SLAVE_BASE_TAG]$SLAVE_BASE_TAG" env: REGISTRY_SERVER: ${{ parameters.registry_url }} displayName: Build sonic-slave-${{ parameters.dist }}-${{ parameters.arch }} - - - task: Docker@2 - condition: ne(variables['Build.Reason'], 'PullRequest') - displayName: Upload image - inputs: - containerRegistry: ${{ parameters.registry_conn }} - repository: $(VARIABLE_SLAVE_BASE_IMAGE) - command: push - ${{ if eq(variables['Build.SourceBranchName'], 'master') }}: - tags: | - $(VARIABLE_SLAVE_BASE_TAG) - latest - ${{ else }}: - tags: | - $(VARIABLE_SLAVE_BASE_TAG) - - ${{ if ne(parameters.arch, 'amd64') }}: - - task: Docker@2 - condition: ne(variables['Build.Reason'], 'PullRequest') - displayName: Upload image ${{ parameters.dist }} - inputs: - containerRegistry: ${{ parameters.registry_conn }} - repository: "sonic-slave-${{ parameters.dist }}" - command: push - tags: | - $(VARIABLE_SLAVE_BASE_TAG) diff --git a/.azure-pipelines/docker-sonic-slave.yml b/.azure-pipelines/docker-sonic-slave.yml index cd8a6df4aff4..dd7a11d2c531 100644 --- a/.azure-pipelines/docker-sonic-slave.yml +++ b/.azure-pipelines/docker-sonic-slave.yml @@ -12,26 +12,28 @@ resources: endpoint: sonic-net schedules: -- cron: "0 8 * * *" +- cron: "0 0 * * 0" + displayName: Weekly build branches: include: - master - - 202012 + - 202??? always: true -trigger: none -pr: +pr: none +trigger: + batch: true branches: include: - master + - 202??? paths: include: - - sonic-slave-jessie - - sonic-slave-stretch - - sonic-slave-buster - - sonic-slave-bullseye + - sonic-slave-* - src/sonic-build-hooks - - .azure-pipelines + - files/build/versions + - Makefile + - Makefile.work parameters: - name: 'arches' @@ -60,15 +62,21 @@ stages: - ${{ each dist in parameters.dists }}: - ${{ if endswith(variables['Build.DefinitionName'], dist) }}: - ${{ each arch in parameters.arches }}: - - ${{ if eq(variables['Build.Reason'], 'PullRequest') }}: - - template: docker-sonic-slave-template.yml - parameters: - pool: sonicbld - arch: ${{ arch }} - dist: ${{ dist }} - - ${{ else }}: - - template: '/.azure-pipelines/docker-sonic-slave-template.yml@buildimage' + - template: .azure-pipelines/docker-sonic-slave-template.yml@buildimage + parameters: + pool: sonicbld + arch: ${{ arch }} + dist: ${{ dist }} +- stage: Build_march + dependsOn: [] + jobs: + - ${{ each dist in parameters.dists }}: + - ${{ if endswith(variables['Build.DefinitionName'], dist) }}: + - ${{ each arch in parameters.arches }}: + - ${{ if ne(arch, 'amd64') }}: + - template: .azure-pipelines/docker-sonic-slave-template.yml@buildimage parameters: - pool: sonicbld + pool: sonicbld-${{ arch }} arch: ${{ arch }} dist: ${{ dist }} + march: march_ From fd6a1b0ce28f07ac22254bfce8b0ad4a8dd92ea2 Mon Sep 17 00:00:00 2001 From: Zain Budhwani <99770260+zbud-msft@users.noreply.github.com> Date: Wed, 21 Sep 2022 09:20:53 -0700 Subject: [PATCH 11/43] Add events to host and create rsyslog_plugin deb pkg (#12059) Why I did it Create rsyslog plugin deb for other containers/host to install Add events for bgp and host events --- dockers/docker-eventd/supervisord.conf | 2 +- dockers/docker-fpm-frr/Dockerfile.j2 | 8 +- files/build_templates/events_info.json | 17 ++ files/build_templates/init_cfg.json.j2 | 2 +- files/build_templates/monit_regex.json | 17 ++ files/build_templates/rsyslog_plugin.conf.j2 | 2 +- .../build_templates/sonic_debian_extension.j2 | 14 +- files/build_templates/sshd_regex.json | 7 + files/build_templates/systemd_regex.json | 7 + files/image_config/monit/container_checker | 15 ++ rules/docker-eventd.mk | 10 - rules/docker-fpm-frr.mk | 4 +- rules/eventd.mk | 4 +- slave.mk | 3 +- src/sonic-eventd/Makefile | 20 +- src/sonic-eventd/debian/control | 8 +- src/sonic-eventd/debian/sonic-eventd.install | 3 + .../debian/sonic-rsyslog-plugin.install | 3 + src/sonic-eventd/tools/events_monit_test.py | 226 ++++++++++++++++++ src/sonic-eventd/tools/monit_events | 6 + 20 files changed, 343 insertions(+), 35 deletions(-) create mode 100644 files/build_templates/events_info.json create mode 100644 files/build_templates/monit_regex.json create mode 100644 files/build_templates/sshd_regex.json create mode 100644 files/build_templates/systemd_regex.json create mode 100644 src/sonic-eventd/debian/sonic-eventd.install create mode 100644 src/sonic-eventd/debian/sonic-rsyslog-plugin.install create mode 100755 src/sonic-eventd/tools/events_monit_test.py create mode 100644 src/sonic-eventd/tools/monit_events diff --git a/dockers/docker-eventd/supervisord.conf b/dockers/docker-eventd/supervisord.conf index 5d9a50bca2ae..be51f922c120 100644 --- a/dockers/docker-eventd/supervisord.conf +++ b/dockers/docker-eventd/supervisord.conf @@ -41,7 +41,7 @@ dependent_startup_wait_for=rsyslogd:running [program:eventd] -command=/usr/sbin/eventd +command=/usr/bin/eventd priority=3 autostart=false autorestart=false diff --git a/dockers/docker-fpm-frr/Dockerfile.j2 b/dockers/docker-fpm-frr/Dockerfile.j2 index 25e191bc338f..c42f001b830f 100644 --- a/dockers/docker-fpm-frr/Dockerfile.j2 +++ b/dockers/docker-fpm-frr/Dockerfile.j2 @@ -56,14 +56,14 @@ COPY ["TS", "/usr/bin/TS"] COPY ["files/supervisor-proc-exit-listener", "/usr/bin"] COPY ["zsocket.sh", "/usr/bin/"] COPY ["*.json", "/etc/rsyslog.d/"] -# COPY ["files/rsyslog_plugin.conf.j2", "/etc/rsyslog.d/"] +COPY ["files/rsyslog_plugin.conf.j2", "/etc/rsyslog.d/"] RUN chmod a+x /usr/bin/TSA && \ chmod a+x /usr/bin/TSB && \ chmod a+x /usr/bin/TSC && \ chmod a+x /usr/bin/zsocket.sh -# RUN j2 -f json /etc/rsyslog.d/rsyslog_plugin.conf.j2 /etc/rsyslog.d/events_info.json > /etc/rsyslog.d/bgp_events.conf -# RUN rm -f /etc/rsyslog.d/rsyslog_plugin.conf.j2* -RUN rm -f /etc/rsyslog.d/events_info.json* +RUN j2 -f json /etc/rsyslog.d/rsyslog_plugin.conf.j2 /etc/rsyslog.d/events_info.json > /etc/rsyslog.d/bgp_events.conf +RUN rm -f /etc/rsyslog.d/rsyslog_plugin.conf.j2 +RUN rm -f /etc/rsyslog.d/events_info.json ENTRYPOINT ["/usr/bin/docker_init.sh"] diff --git a/files/build_templates/events_info.json b/files/build_templates/events_info.json new file mode 100644 index 000000000000..d2e2eb151b48 --- /dev/null +++ b/files/build_templates/events_info.json @@ -0,0 +1,17 @@ +{ + "yang_module": "sonic-events-host", + "proclist": [ + { + "name": "monit", + "parse_json": "monit_regex.json" + }, + { + "name": "sshd", + "parse_json": "sshd_regex.json" + }, + { + "name": "systemd", + "parse_json": "systemd_regex.json" + } + ] +} diff --git a/files/build_templates/init_cfg.json.j2 b/files/build_templates/init_cfg.json.j2 index 8e92807f4e2c..38bd7c2e43a6 100644 --- a/files/build_templates/init_cfg.json.j2 +++ b/files/build_templates/init_cfg.json.j2 @@ -39,7 +39,7 @@ ("pmon", "enabled", false, "enabled"), ("radv", "enabled", false, "enabled"), ("snmp", "enabled", true, "enabled"), - ("eventd", "enabled", true, "enabled"), + ("eventd", "enabled", false, "enabled"), ("swss", "enabled", false, "enabled"), ("syncd", "enabled", false, "enabled"), ("teamd", "enabled", false, "enabled")] %} diff --git a/files/build_templates/monit_regex.json b/files/build_templates/monit_regex.json new file mode 100644 index 000000000000..e7384a5434d4 --- /dev/null +++ b/files/build_templates/monit_regex.json @@ -0,0 +1,17 @@ +[ + { + "tag": "disk-usage", + "regex": ".([a-zA-Z0-9-_]*). space usage (\\d+\\.\\d+)% matches resource limit .space usage.(\\d+\\.\\d+)%.", + "params": [ "fs", "usage", "limit" ] + }, + { + "tag": "memory-usage", + "regex": "mem usage of (\\d+\\.\\d+)% matches resource limit .mem usage>(\\d+\\.\\d+)%.", + "params": [ "usage", "limit" ] + }, + { + "tag": "cpu-usage", + "regex": "cpu user usage of (\\d+\\.\\d+)% matches resource limit .cpu user usage>(\\d+\\.\\d+)%.", + "params": [ "usage", "limit" ] + } +] diff --git a/files/build_templates/rsyslog_plugin.conf.j2 b/files/build_templates/rsyslog_plugin.conf.j2 index ec19c62a78f6..fdf661fc2d65 100644 --- a/files/build_templates/rsyslog_plugin.conf.j2 +++ b/files/build_templates/rsyslog_plugin.conf.j2 @@ -12,7 +12,7 @@ $ModLoad omprog {% for proc in proclist %} if re_match($programname, "{{ proc.name }}") then { action(type="omprog" - binary="/usr/share/sonic/scripts/rsyslog_plugin -r /etc/rsyslog.d/{{ proc.parse_json }} -m {{ yang_module }}" + binary="/usr/bin/rsyslog_plugin -r /etc/rsyslog.d/{{ proc.parse_json }} -m {{ yang_module }}" output="/var/log/rsyslog_plugin.log" template="prog_msg") } diff --git a/files/build_templates/sonic_debian_extension.j2 b/files/build_templates/sonic_debian_extension.j2 index a62685dfbbe7..08098b9c13ed 100644 --- a/files/build_templates/sonic_debian_extension.j2 +++ b/files/build_templates/sonic_debian_extension.j2 @@ -319,6 +319,16 @@ sudo dpkg --root=$FILESYSTEM_ROOT -i {{deb}} || sudo LANG=C DEBIAN_FRONTEND=noni sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/sonic-db-cli_*.deb || \ sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f +# Install sonic-rsyslog-plugin +sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/sonic-rsyslog-plugin_*.deb || \ + sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f + +# Generate host conf for rsyslog_plugin +j2 -f json $BUILD_TEMPLATES/rsyslog_plugin.conf.j2 $BUILD_TEMPLATES/events_info.json | sudo tee $FILESYSTEM_ROOT_ETC/rsyslog.d/host_events.conf +sudo cp $BUILD_TEMPLATES/monit_regex.json $FILESYSTEM_ROOT_ETC/rsyslog.d/ +sudo cp $BUILD_TEMPLATES/sshd_regex.json $FILESYSTEM_ROOT_ETC/rsyslog.d/ +sudo cp $BUILD_TEMPLATES/systemd_regex.json $FILESYSTEM_ROOT_ETC/rsyslog.d/ + # Install custom-built monit package and SONiC configuration files sudo dpkg --root=$FILESYSTEM_ROOT -i $debs_path/monit_*.deb || \ sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f @@ -799,10 +809,6 @@ sudo bash -c "echo { > $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_name {% endfor %} sudo bash -c "echo } >> $FILESYSTEM_ROOT_USR_SHARE_SONIC_TEMPLATES/ctr_image_names.json" -# copy rsyslog plugin binary for use by all dockers that use plugin to publish events. -# sudo mkdir -p ${FILESYSTEM_ROOT_USR_SHARE_SONIC_SCRIPTS} -# sudo cp ${files_path}/rsyslog_plugin ${FILESYSTEM_ROOT_USR_SHARE_SONIC_SCRIPTS}/ - {% for script in installer_start_scripts.split(' ') -%} if [ -f $TARGET_MACHINE"_{{script}}" ]; then sudo cp $TARGET_MACHINE"_{{script}}" $FILESYSTEM_ROOT/usr/bin/{{script}} diff --git a/files/build_templates/sshd_regex.json b/files/build_templates/sshd_regex.json new file mode 100644 index 000000000000..dd664bd1ea75 --- /dev/null +++ b/files/build_templates/sshd_regex.json @@ -0,0 +1,7 @@ +[ + { + "tag": "event-sshd", + "regex": "auth fail: Password Incorrect. user:.([a-zA-Z0-9-_]*)", + "params": [ "username" ] + } +] diff --git a/files/build_templates/systemd_regex.json b/files/build_templates/systemd_regex.json new file mode 100644 index 000000000000..d6ea5619eed9 --- /dev/null +++ b/files/build_templates/systemd_regex.json @@ -0,0 +1,7 @@ +[ + { + "tag": "event-stopped-ctr", + "regex": "Stopped ([a-zA-Z-_\\s]*) container", + "params": [ "ctr-name" ] + } +] diff --git a/files/image_config/monit/container_checker b/files/image_config/monit/container_checker index 0958368559ad..6d7f1403d7ae 100755 --- a/files/image_config/monit/container_checker +++ b/files/image_config/monit/container_checker @@ -22,6 +22,9 @@ import sys from sonic_py_common import multi_asic, device_info from swsscommon import swsscommon +EVENTS_PUBLISHER_SOURCE = "sonic-events-host" +EVENTS_PUBLISHER_TAG = "event-down-ctr" + def get_expected_running_containers(): """ @summary: This function will get the expected running & always-enabled containers by following the rule: @@ -150,6 +153,17 @@ def get_current_running_containers(always_running_containers): return current_running_containers +def publish_events(lst): + events_handle = swsscommon.events_init_publisher(EVENTS_PUBLISHER_SOURCE) + params = swsscommon.FieldValueMap() + + for ctr in lst: + params["name"] = ctr; + swsscommon.event_publish(events_handle, EVENTS_PUBLISHER_TAG, params) + + swsscommon.events_deinit_publisher(events_handle) + + def main(): """ @summary: This function will compare the difference between the current running containers @@ -162,6 +176,7 @@ def main(): expected_running_containers |= always_running_containers not_running_containers = expected_running_containers.difference(current_running_containers) if not_running_containers: + publish_events(not_running_containers) print("Expected containers not running: " + ", ".join(not_running_containers)) sys.exit(3) diff --git a/rules/docker-eventd.mk b/rules/docker-eventd.mk index ec333bf66048..304f295e2a4b 100644 --- a/rules/docker-eventd.mk +++ b/rules/docker-eventd.mk @@ -36,13 +36,3 @@ $(DOCKER_EVENTD)_RUN_OPT += -v /etc/sonic:/etc/sonic:ro SONIC_BULLSEYE_DOCKERS += $(DOCKER_EVENTD) SONIC_BULLSEYE_DBG_DOCKERS += $(DOCKER_EVENTD_DBG) - -$(DOCKER_EVENTD)_FILESPATH = $($(SONIC_EVENTD)_SRC_PATH)/rsyslog_plugin - -$(DOCKER_EVENTD)_PLUGIN = rsyslog_plugin -$($(DOCKER_EVENTD)_PLUGIN)_PATH = $($(DOCKER_EVENTD)_FILESPATH) - -SONIC_COPY_FILES += $($(DOCKER_EVENTD)_PLUGIN) -# Some builds fails to find this file. Remove until we root cause it. -# $(DOCKER_EVENTD)_SHARED_FILES = $($(DOCKER_EVENTD)_PLUGIN) - diff --git a/rules/docker-fpm-frr.mk b/rules/docker-fpm-frr.mk index 861e1d8e8b41..79d5a927d20f 100644 --- a/rules/docker-fpm-frr.mk +++ b/rules/docker-fpm-frr.mk @@ -7,10 +7,10 @@ DOCKER_FPM_FRR_DBG = $(DOCKER_FPM_FRR_STEM)-$(DBG_IMAGE_MARK).gz $(DOCKER_FPM_FRR)_PATH = $(DOCKERS_PATH)/$(DOCKER_FPM_FRR_STEM) $(DOCKER_FPM_FRR)_PYTHON_WHEELS += $(SONIC_BGPCFGD) $(SONIC_FRR_MGMT_FRAMEWORK) -$(DOCKER_FPM_FRR)_DEPENDS += $(FRR) $(FRR_SNMP) $(SWSS) $(LIBYANG2) +$(DOCKER_FPM_FRR)_DEPENDS += $(FRR) $(FRR_SNMP) $(SWSS) $(LIBYANG2) $(SONIC_RSYSLOG_PLUGIN) $(DOCKER_FPM_FRR)_DBG_DEPENDS = $($(DOCKER_SWSS_LAYER_BULLSEYE)_DBG_DEPENDS) $(DOCKER_FPM_FRR)_DBG_DEPENDS += $(SWSS_DBG) $(LIBSWSSCOMMON_DBG) \ - $(FRR_DBG) $(FRR_SNMP_DBG) $(LIBYANG2_DBG) + $(FRR_DBG) $(FRR_SNMP_DBG) $(LIBYANG2_DBG) $(SONIC_RSYSLOG_PLUGIN) $(DOCKER_FPM_FRR)_DBG_IMAGE_PACKAGES = $($(DOCKER_SWSS_LAYER_BULLSEYE)_DBG_IMAGE_PACKAGES) diff --git a/rules/eventd.mk b/rules/eventd.mk index 9eea21a4cfb5..02a6e3729829 100644 --- a/rules/eventd.mk +++ b/rules/eventd.mk @@ -11,9 +11,9 @@ SONIC_DPKG_DEBS += $(SONIC_EVENTD) SONIC_EVENTD_DBG = sonic-$(SONIC_EVENTD_PKG_NAME)-dbgsym_$(SONIC_EVENTD_VERSION)_$(CONFIGURED_ARCH).deb $(eval $(call add_derived_package,$(SONIC_EVENTD),$(SONIC_EVENTD_DBG))) - +SONIC_RSYSLOG_PLUGIN = sonic-rsyslog-plugin_$(SONIC_EVENTD_VERSION)_$(CONFIGURED_ARCH).deb +$(eval $(call add_derived_package,$(SONIC_EVENTD),$(SONIC_RSYSLOG_PLUGIN))) # The .c, .cpp, .h & .hpp files under src/{$DBG_SRC_ARCHIVE list} # are archived into debug one image to facilitate debugging. # DBG_SRC_ARCHIVE += sonic-eventd - diff --git a/slave.mk b/slave.mk index 34af3169cfef..c521b3807356 100644 --- a/slave.mk +++ b/slave.mk @@ -1129,6 +1129,7 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \ $(PYTHON_SWSSCOMMON) \ $(PYTHON3_SWSSCOMMON) \ $(SONIC_DB_CLI) \ + $(SONIC_RSYSLOG_PLUGIN) \ $(SONIC_UTILITIES_DATA) \ $(SONIC_HOST_SERVICES_DATA) \ $(BASH) \ @@ -1292,8 +1293,6 @@ $(addprefix $(TARGET_PATH)/, $(SONIC_INSTALLERS)) : $(TARGET_PATH)/% : \ $(if $($(docker:-dbg.gz=.gz)_MACHINE),\ mv $($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh $($(docker:-dbg.gz=.gz)_MACHINE)_$($(docker:-dbg.gz=.gz)_CONTAINER_NAME).sh ) - $(foreach file, $($(docker)_SHARED_FILES), \ - { cp $($(file)_PATH)/$(file) $(FILES_PATH)/ $(LOG) || exit 1 ; } ; ) ) # Exported variables are used by sonic_debian_extension.sh diff --git a/src/sonic-eventd/Makefile b/src/sonic-eventd/Makefile index 00d3199a65bc..ebc1aa7ee9e8 100644 --- a/src/sonic-eventd/Makefile +++ b/src/sonic-eventd/Makefile @@ -5,6 +5,9 @@ EVENTD_TOOL := tools/events_tool EVENTD_PUBLISH_TOOL := tools/events_publish_tool.py RSYSLOG-PLUGIN_TARGET := rsyslog_plugin/rsyslog_plugin RSYSLOG-PLUGIN_TEST := rsyslog_plugin_tests/tests +EVENTD_MONIT := tools/events_monit_test.py +EVENTD_MONIT_CONF := tools/monit_events + CP := cp MKDIR := mkdir CC := g++ @@ -68,15 +71,18 @@ rsyslog-plugin-tests: $(RSYSLOG-PLUGIN-TEST_OBJS) @echo ' ' install: - $(MKDIR) -p $(DESTDIR)/usr/sbin - $(CP) $(EVENTD_TARGET) $(DESTDIR)/usr/sbin - $(CP) $(EVENTD_TOOL) $(DESTDIR)/usr/sbin - $(CP) $(EVENTD_PUBLISH_TOOL) $(DESTDIR)/usr/sbin + $(MKDIR) -p $(DESTDIR)/usr/bin + $(MKDIR) -p $(DESTDIR)/etc/monit/conf.d + $(CP) $(EVENTD_TARGET) $(DESTDIR)/usr/bin + $(CP) $(EVENTD_TOOL) $(DESTDIR)/usr/bin + $(CP) $(EVENTD_PUBLISH_TOOL) $(DESTDIR)/usr/bin + $(CP) $(RSYSLOG-PLUGIN_TARGET) $(DESTDIR)/usr/bin + $(CP) $(EVENTD_MONIT) $(DESTDIR)/usr/bin + $(CP) $(EVENTD_MONIT_CONF) $(DESTDIR)/etc/monit/conf.d deinstall: - $(RM) $(DESTDIR)/usr/sbin/$(EVENTD_TARGET) - $(RM) $(DESTDIR)/usr/sbin/$(RSYSLOG-PLUGIN_TARGET) - $(RM) -rf $(DESTDIR)/usr/sbin + $(RM) -rf $(DESTDIR)/usr + $(RM) -rf $(DESTDIR)/etc clean: -@echo ' ' diff --git a/src/sonic-eventd/debian/control b/src/sonic-eventd/debian/control index 95ae6fd76452..9ee3d9a563e9 100644 --- a/src/sonic-eventd/debian/control +++ b/src/sonic-eventd/debian/control @@ -2,7 +2,7 @@ Source: sonic-eventd Section: devel Priority: optional Maintainer: Renuka Manavalan -Build-Depends: debhelper (>= 12.0.0), libevent-dev, libboost-thread-dev, libboost-system-dev, libswsscommon-dev +Build-Depends: debhelper (>= 12.0.0), libevent-dev, libboost-thread-dev, libboost-system-dev, libswsscommon-dev, liblua5.1-0 Standards-Version: 3.9.3 Homepage: https://github.com/Azure/sonic-buildimage XS-Go-Import-Path: github.com/Azure/sonic-buildimage @@ -12,3 +12,9 @@ Architecture: any Built-Using: ${misc:Built-Using} Depends: ${shlibs:Depends} Description: SONiC event service + +Package: sonic-rsyslog-plugin +Architecture: any +Built-Using: ${misc:Built-Using} +Depends: ${shlibs:Depends} +Description: SONiC rsyslog plugin service diff --git a/src/sonic-eventd/debian/sonic-eventd.install b/src/sonic-eventd/debian/sonic-eventd.install new file mode 100644 index 000000000000..bdba566c77b3 --- /dev/null +++ b/src/sonic-eventd/debian/sonic-eventd.install @@ -0,0 +1,3 @@ +usr/bin/eventd +usr/bin/events_tool +usr/bin/events_publish_tool.py diff --git a/src/sonic-eventd/debian/sonic-rsyslog-plugin.install b/src/sonic-eventd/debian/sonic-rsyslog-plugin.install new file mode 100644 index 000000000000..bc7cd548ab93 --- /dev/null +++ b/src/sonic-eventd/debian/sonic-rsyslog-plugin.install @@ -0,0 +1,3 @@ +usr/bin/rsyslog_plugin +usr/bin/events_monit_test.py +etc/monit/conf.d/monit_events diff --git a/src/sonic-eventd/tools/events_monit_test.py b/src/sonic-eventd/tools/events_monit_test.py new file mode 100755 index 000000000000..dea7cbdd4c11 --- /dev/null +++ b/src/sonic-eventd/tools/events_monit_test.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 + +from inspect import getframeinfo, stack +from swsscommon.swsscommon import events_init_publisher, event_publish, FieldValueMap +from swsscommon.swsscommon import event_receive_op_t, event_receive, events_init_subscriber +from swsscommon.swsscommon import events_deinit_subscriber, events_deinit_publisher +import argparse +import os +import threading +import time +import syslog +import uuid + +chk_log_level = syslog.LOG_ERR + +test_source = "sonic-host" +test_event_tag = "device-test-event" +test_event_key = "{}:{}".format(test_source, test_event_tag) +test_event_params = { + "sender": os.path.basename(__file__), + "reason": "monit periodic test", + "batch-id": str(uuid.uuid1()), + "index": "0" +} + +# Async connection wait time in seconds. +ASYNC_CONN_WAIT = 0.3 +RECEIVE_TIMEOUT = 1000 + +# Thread results +rc_test_receive = -1 +publish_cnt = 0 + +def _log_msg(lvl, pfx, msg): + if lvl <= chk_log_level: + caller = getframeinfo(stack()[2][0]) + fmsg = "{}:{}:{}".format(caller.function, caller.lineno, msg) + print("{}: {}".format(pfx, fmsg)) + syslog.syslog(lvl, fmsg) + +def log_err(m): + _log_msg(syslog.LOG_ERR, "Err", m) + +def log_info(m): + _log_msg(syslog.LOG_INFO, "Info", m) + +def log_notice(m): + _log_msg(syslog.LOG_NOTICE, "Notice", m) + +def log_debug(m): + _log_msg(syslog.LOG_DEBUG, "Debug", m) + + +def map_dict_fvm(s, d): + for k, v in s.items(): + d[k] = v + + +# Invoked in a separate thread +def test_receiver(event_obj, cnt): + global rc_test_receive + + sh = events_init_subscriber(False, RECEIVE_TIMEOUT, None) + + # Sleep ASYNC_CONN_WAIT to ensure async connectivity is complete. + time.sleep(ASYNC_CONN_WAIT) + + exp_params = dict(test_event_params) + + # Signal main thread that subscriber is ready to receive + event_obj.set() + cnt_done = 0 + + while cnt_done < cnt: + p = event_receive_op_t() + rc = event_receive(sh, p) + + if rc > 0 and publish_cnt < 2: + # ignore timeout as test code has not yet published an event yet + continue + + if rc != 0: + log_notice("Failed to receive. {}/{} rc={}".format(cnt_done, cnt, rc)) + break + + if test_event_key != p.key: + # received a different event than published + continue + + exp_params["index"] = str(cnt_done) + rcv_params = {} + map_dict_fvm(p.params, rcv_params) + + for k, v in exp_params.items(): + if k in rcv_params: + if (rcv_params[k] != v): + log_notice("key:{} exp:{} != exist:{}".format( + k, v, rcv_params[k])) + rc = -1 + else: + log_notice("key:{} is missing".format(k)) + rc = -1 + + if (rc != 0): + log_notice("params mismatch {}/{}".format(cnt_done, cnt)) + break + + if p.missed_cnt != 0: + log_notice("Expect missed_cnt {} == 0 {}/{}".format(p.missed_cnt, cnt_done, cnt)) + break + + if p.publish_epoch_ms == 0: + log_notice("Expect publish_epoch_ms != 0 {}/{}".format(cnt_done, cnt)) + break + + cnt_done += 1 + log_debug("Received {}/{}".format(cnt_done + 1, cnt)) + + if (cnt_done == cnt): + rc_test_receive = 0 + else: + log_notice("test receive abort {}/{}".format(cnt_done, cnt)) + + # Signal main thread that subscriber thread is done + event_obj.set() + events_deinit_subscriber(sh) + + +def publish_events(cnt): + global publish_cnt + rc = -1 + ph = events_init_publisher(test_source) + if not ph: + log_notice("Failed to get publisher handle") + return rc + + # Sleep ASYNC_CONN_WAIT to ensure async connectivity is complete. + # Messages published before connection are silently dropped by ZMQ. + time.sleep(ASYNC_CONN_WAIT) + + pub_params = dict(test_event_params) + + for i in range(cnt): + pd = FieldValueMap() + pub_params["index"] = str(i) + map_dict_fvm(pub_params, pd) + + rc = event_publish(ph, test_event_tag, pd) + if (rc != 0): + log_notice("Failed to publish. {}/{} rc={}".format(i, cnt, rc)) + break + + log_debug("published: {}/{}".format(i+1, cnt)) + publish_cnt += 1 + + # Sleep ASYNC_CONN_WAIT to ensure publish complete, before closing channel. + time.sleep(ASYNC_CONN_WAIT) + + events_deinit_publisher(ph) + + log_debug("publish_events Done. cnt={}".format(cnt)) + + return rc + + + +def run_test(cnt): + global rc_test_receive + + # Initialising event objects + event_sub = threading.Event() + + # Start subscriber thread + thread_sub = threading.Thread(target=test_receiver, args=(event_sub, cnt)) + thread_sub.start() + + # Wait until subscriber thread completes the async subscription + # Any event published prior to that could get lost + # Subscriber would wait for ASYNC_CONN_WAIT. Wait additional 200ms + # for signal from test_receiver as ready. + event_sub.wait(ASYNC_CONN_WAIT + 0.2) + event_sub.clear() + + rc_pub = publish_events(cnt) + if (rc_pub != 0): + log_notice("Failed in publish_events") + else: + # Wait for subscriber to complete with 1 sec timeout. + event_sub.wait(1) + if (rc_test_receive != 0): + log_notice("Failed to receive events") + + log_debug("run_test_DONE rc_pub={} rc_test_receive={}".format( + rc_pub, rc_test_receive)) + + if (rc_pub != 0): + return rc_pub + + if (rc_test_receive == 0): + return rc_test_receive + + return 0 + + +def main(): + global chk_log_level + + parser=argparse.ArgumentParser( + description="Check events from publish to receive via gNMI") + parser.add_argument('-l', "--loglvl", default=syslog.LOG_ERR, type=int, + help="log level") + parser.add_argument('-n', "--cnt", default=5, type=int, + help="count of events to publish/receive") + args = parser.parse_args() + + chk_log_level = args.loglvl + rc = run_test(args.cnt) + + if(rc == 0): + log_info("eventd test succeeded") + else: + log_notice("eventd monit test failed rc={}".format(rc)) + + +if __name__ == "__main__": + main() diff --git a/src/sonic-eventd/tools/monit_events b/src/sonic-eventd/tools/monit_events new file mode 100644 index 000000000000..ebb9acce4b90 --- /dev/null +++ b/src/sonic-eventd/tools/monit_events @@ -0,0 +1,6 @@ +############################################################################### +## Monit configuration for telemetry container +############################################################################### +check program container_eventd with path "/usr/bin/events_monit_test.py" + every 5 cycles + if status != 0 for 3 cycle then alert repeat every 1 cycles From f8494d10ad33c2ed3dd2e8f2c40e83000ad88b51 Mon Sep 17 00:00:00 2001 From: Hua Liu <58683130+liuh-80@users.noreply.github.com> Date: Thu, 22 Sep 2022 09:25:29 +0800 Subject: [PATCH 12/43] Improve SSHD config to use more secure settings (#12109) Improve SSHD config to use more secure settings #### Why I did it According to Sonic OS review result, SSHD config file /etc/ssh/sshd_config using insecure settings. #### How I did it Change build_debian.sh script to set following settings to /etc/ssh/sshd_config: ClientAliveInterval is set to 300 MaxAuthTries is set to default of 3 Banner set to /etc/issue LogLevel is set to VERBOSE #### How to verify it Pass all E2E test case. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Improve SSHD config to use more secure settings #### Link to config_db schema for YANG module changes #### A picture of a cute animal (not mandatory but encouraged) --- build_debian.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/build_debian.sh b/build_debian.sh index 8dcd8596f684..01a10d78d678 100755 --- a/build_debian.sh +++ b/build_debian.sh @@ -467,10 +467,16 @@ rm /files/etc/ssh/sshd_config/ClientAliveInterval rm /files/etc/ssh/sshd_config/ClientAliveCountMax touch /files/etc/ssh/sshd_config/EmptyLineHack rename /files/etc/ssh/sshd_config/EmptyLineHack "" -set /files/etc/ssh/sshd_config/ClientAliveInterval 900 +set /files/etc/ssh/sshd_config/ClientAliveInterval 300 set /files/etc/ssh/sshd_config/ClientAliveCountMax 1 ins #comment before /files/etc/ssh/sshd_config/ClientAliveInterval -set /files/etc/ssh/sshd_config/#comment[following-sibling::*[1][self::ClientAliveInterval]] "Close inactive client sessions after 15 minutes" +set /files/etc/ssh/sshd_config/#comment[following-sibling::*[1][self::ClientAliveInterval]] "Close inactive client sessions after 5 minutes" +rm /files/etc/ssh/sshd_config/MaxAuthTries +set /files/etc/ssh/sshd_config/MaxAuthTries 3 +rm /files/etc/ssh/sshd_config/LogLevel +set /files/etc/ssh/sshd_config/LogLevel VERBOSE +rm /files/etc/ssh/sshd_config/Banner +set /files/etc/ssh/sshd_config/Banner /etc/issue save quit EOF From 8af369a7c94d461338ee0aa63202b11252742e2c Mon Sep 17 00:00:00 2001 From: Xichen96 Date: Thu, 22 Sep 2022 13:57:52 +0800 Subject: [PATCH 13/43] Enable swap for haliburton device. (#11746) Signed-off-by: Xichen Lin Signed-off-by: Xichen Lin --- .../debian/platform-modules-haliburton.init | 22 +++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/platform/broadcom/sonic-platform-modules-cel/debian/platform-modules-haliburton.init b/platform/broadcom/sonic-platform-modules-cel/debian/platform-modules-haliburton.init index 07a6542109b5..3f6358bf4ab4 100644 --- a/platform/broadcom/sonic-platform-modules-cel/debian/platform-modules-haliburton.init +++ b/platform/broadcom/sonic-platform-modules-cel/debian/platform-modules-haliburton.init @@ -11,6 +11,26 @@ # Short-Description: Setup Haliburton board. ### END INIT INFO +setup_swap () { + SWAPFILE=/host/myswapfile + + if [ ! -f $SWAPFILE ]; then + availspace=`df -h --output=avail /host | sed '1d;s/\s//g;s/[^0-9].*//g'` + diff=$(( availspace - 2*$1 )) + if [ $diff -gt 0 ]; then + fallocate -l ${1}G $SWAPFILE + chmod 600 $SWAPFILE + echo "swap file created successfully" + else + echo "not enough disk space to turn on swap." + return + fi + fi + mkswap $SWAPFILE + swapon $SWAPFILE + echo "swap on successfully" +} + case "$1" in start) echo -n "Setting up board... " @@ -74,6 +94,8 @@ start) /bin/sh /usr/local/bin/platform_api_mgnt.sh init + setup_swap 2 + echo "done." ;; From 283efeeaccd386f99e3f9242f54d76bfd2901f44 Mon Sep 17 00:00:00 2001 From: Mai Bui Date: Thu, 22 Sep 2022 06:40:42 -0700 Subject: [PATCH 14/43] [sonic-py-common] Add getstatusoutput_noshell() functions to general module (#12065) Signed-off-by: maipbui #### Why I did it `getstatusoutput()` function from `subprocess` module has shell injection issue because it includes `shell=True` in the implementation Eliminate duplicate code #### How I did it Reimplement `getstatusoutput_noshell()` and `getstatusoutput_noshell_pipe()` functions with `shell=False` Add `check_output_pipe()` function #### How to verify it Pass UT --- .../sonic_py_common/general.py | 68 +++++++++++++++++++ src/sonic-py-common/tests/test_general.py | 31 +++++++++ 2 files changed, 99 insertions(+) create mode 100644 src/sonic-py-common/tests/test_general.py diff --git a/src/sonic-py-common/sonic_py_common/general.py b/src/sonic-py-common/sonic_py_common/general.py index 9e04f3e214ee..4fea165de9fb 100644 --- a/src/sonic-py-common/sonic_py_common/general.py +++ b/src/sonic-py-common/sonic_py_common/general.py @@ -1,4 +1,5 @@ import sys +from subprocess import Popen, STDOUT, PIPE, CalledProcessError, check_output def load_module_from_source(module_name, file_path): @@ -23,3 +24,70 @@ def load_module_from_source(module_name, file_path): sys.modules[module_name] = module return module + + +def getstatusoutput_noshell(cmd): + """ + This function implements getstatusoutput API from subprocess module + but using shell=False to prevent shell injection. + Ref: https://github.com/python/cpython/blob/3.10/Lib/subprocess.py#L602 + """ + try: + output = check_output(cmd, universal_newlines=True, stderr=STDOUT) + exitcode = 0 + except CalledProcessError as ex: + output = ex.output + exitcode = ex.returncode + if output[-1:] == '\n': + output = output[:-1] + return exitcode, output + + +def getstatusoutput_noshell_pipe(cmd0, *args): + """ + This function implements getstatusoutput API from subprocess module + but using shell=False to prevent shell injection. Input command + includes two or more commands connected by shell pipe(s). + """ + popens = [Popen(cmd0, stdout=PIPE, universal_newlines=True)] + i = 0 + while i < len(args): + popens.append(Popen(args[i], stdin=popens[i].stdout, stdout=PIPE, universal_newlines=True)) + popens[i].stdout.close() + i += 1 + output = popens[-1].communicate()[0] + if output[-1:] == '\n': + output = output[:-1] + + exitcodes = [0] * len(popens) + while popens: + last = popens.pop(-1) + exitcodes[len(popens)] = last.wait() + + return (exitcodes, output) + + +def check_output_pipe(cmd0, *args): + """ + This function implements check_output API from subprocess module. + Input command includes two or more commands connected by shell pipe(s) + """ + popens = [Popen(cmd0, stdout=PIPE, universal_newlines=True)] + i = 0 + while i < len(args): + popens.append(Popen(args[i], stdin=popens[i].stdout, stdout=PIPE, universal_newlines=True)) + popens[i].stdout.close() + i += 1 + output = popens[-1].communicate()[0] + + i = 0 + args_list = [cmd0] + list(args) + while popens: + current = popens.pop(0) + exitcode = current.wait() + if exitcode != 0: + raise CalledProcessError(returncode=exitcode, cmd=args_list[i], output=current.stdout) + i += 1 + + return output + diff --git a/src/sonic-py-common/tests/test_general.py b/src/sonic-py-common/tests/test_general.py new file mode 100644 index 000000000000..a395cf9aeb6b --- /dev/null +++ b/src/sonic-py-common/tests/test_general.py @@ -0,0 +1,31 @@ +import sys +import pytest +import subprocess +from sonic_py_common.general import getstatusoutput_noshell, getstatusoutput_noshell_pipe, check_output_pipe + + +def test_getstatusoutput_noshell(tmp_path): + exitcode, output = getstatusoutput_noshell(['echo', 'sonic']) + assert (exitcode, output) == (0, 'sonic') + + exitcode, output = getstatusoutput_noshell([sys.executable, "-c", "import sys; sys.exit(6)"]) + assert exitcode != 0 + +def test_getstatusoutput_noshell_pipe(): + exitcode, output = getstatusoutput_noshell_pipe(['echo', 'sonic'], ['awk', '{print $1}']) + assert (exitcode, output) == ([0, 0], 'sonic') + + exitcode, output = getstatusoutput_noshell_pipe([sys.executable, "-c", "import sys; sys.exit(6)"], [sys.executable, "-c", "import sys; sys.exit(8)"]) + assert exitcode == [6, 8] + +def test_check_output_pipe(): + output = check_output_pipe(['echo', 'sonic'], ['awk', '{print $1}']) + assert output == 'sonic\n' + + with pytest.raises(subprocess.CalledProcessError) as e: + check_output_pipe([sys.executable, "-c", "import sys; sys.exit(6)"], [sys.executable, "-c", "import sys; sys.exit(0)"]) + assert e.returncode == [6, 0] + + with pytest.raises(subprocess.CalledProcessError) as e: + check_output_pipe([sys.executable, "-c", "import sys; sys.exit(0)"], [sys.executable, "-c", "import sys; sys.exit(6)"]) + assert e.returncode == [0, 6] From 57ff7a230827102f2d04cfdae8c3fcf242b09b50 Mon Sep 17 00:00:00 2001 From: "Marty Y. Lok" <76118573+mlok-nokia@users.noreply.github.com> Date: Thu, 22 Sep 2022 19:39:31 -0400 Subject: [PATCH 15/43] [chassis][supervisor] show system-health summary fails on the supervisor card (#10631) Fix the command "sudo show system-health summary" shows the following error on the supervisor card. Fixes #10630 --- src/system-health/health_checker/service_checker.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/system-health/health_checker/service_checker.py b/src/system-health/health_checker/service_checker.py index 33eeb4dfb707..674ad9944280 100644 --- a/src/system-health/health_checker/service_checker.py +++ b/src/system-health/health_checker/service_checker.py @@ -127,11 +127,11 @@ def get_critical_process_list_from_file(self, container, critical_processes_file self.bad_containers.add(container) logger.log_error('Invalid syntax in critical_processes file of {}'.format(container)) continue - - identifier_key = match.group(2).strip() - identifier_value = match.group(3).strip() - if identifier_key == "program" and identifier_value: - critical_process_list.append(identifier_value) + if match.group(1) is not None: + identifier_key = match.group(2).strip() + identifier_value = match.group(3).strip() + if identifier_key == "program" and identifier_value: + critical_process_list.append(identifier_value) return critical_process_list From 27032bfb84de0db2733dd024e54f0f0f83a0c769 Mon Sep 17 00:00:00 2001 From: Samuel Angebault Date: Fri, 23 Sep 2022 02:52:40 +0200 Subject: [PATCH 16/43] Add BUILD_DATE to SWI (#11915) Add the BUILD_DATE to the SWI version info, as this is a requirement of Secure Boot. --- build_image.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/build_image.sh b/build_image.sh index ddf134e845ce..9c70713866a4 100755 --- a/build_image.sh +++ b/build_image.sh @@ -191,6 +191,7 @@ elif [ "$IMAGE_TYPE" = "aboot" ]; then zip -g $ABOOT_BOOT_IMAGE .imagehash rm .imagehash echo "SWI_VERSION=42.0.0" > version + echo "BUILD_DATE=$(date -d "${build_date}" -u +%Y%m%dT%H%M%SZ)" >> version echo "SWI_MAX_HWEPOCH=2" >> version echo "SWI_VARIANT=US" >> version zip -g $OUTPUT_ABOOT_IMAGE version From c968114a36dacd46b3dfea0e61b52a72daac4737 Mon Sep 17 00:00:00 2001 From: Liu Shilong Date: Fri, 23 Sep 2022 12:54:57 +0800 Subject: [PATCH 17/43] [ci] Use absolute template file path in docker-sonic-slave pipeline. (#12153) --- .azure-pipelines/docker-sonic-slave-template.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.azure-pipelines/docker-sonic-slave-template.yml b/.azure-pipelines/docker-sonic-slave-template.yml index 1a7b983b5c1f..142ff5fa3f6f 100644 --- a/.azure-pipelines/docker-sonic-slave-template.yml +++ b/.azure-pipelines/docker-sonic-slave-template.yml @@ -38,12 +38,12 @@ jobs: - job: Build_${{ parameters.dist }}_${{ parameters.march }}${{ parameters.arch }} timeoutInMinutes: 360 variables: - - template: .azure-pipelines/template-variables.yml@buildimage - - template: .azure-pipelines/azure-pipelines-repd-build-variables.yml@buildimage + - template: /.azure-pipelines/template-variables.yml@buildimage + - template: /.azure-pipelines/azure-pipelines-repd-build-variables.yml@buildimage pool: ${{ parameters.pool }} steps: - template: cleanup.yml - - template: .azure-pipelines/template-clean-sonic-slave.yml@buildimage + - template: /.azure-pipelines/template-clean-sonic-slave.yml@buildimage - checkout: self clean: true submodules: recursive @@ -62,7 +62,7 @@ jobs: exit 0 fi - DOCKER_DATA_ROOT_FOR_MULTIARCH=/data/march/docker make configure PLATFORM=generic PLATFORM_ARCH=${{ parameters.arch }} $args || docker image ls $image_tag + DOCKER_DATA_ROOT_FOR_MULTIARCH=/data/march/docker BLDENV=${{ parameters.dist }} make -f Makefile.work configure PLATFORM=generic PLATFORM_ARCH=${{ parameters.arch }} $args || docker image ls $image_tag if [[ "$(Build.Reason)" == "PullRequest" ]];then exit 0 fi From cc0781b40b7e442677eb4ad21d72a29f16a666b3 Mon Sep 17 00:00:00 2001 From: Hua Liu <58683130+liuh-80@users.noreply.github.com> Date: Sun, 25 Sep 2022 03:37:35 +0800 Subject: [PATCH 18/43] Build swss-common with libyang (#12087) Build swss-common with libyang #### Why I did it sonic-swss-common lib add dependency to libyang recently, so need update make file before update sonic-swss-common submodule. #### How I did it Add dependency to libyang in rules/swss-common.mk #### How to verify it Pass all E2E test case. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Add new Redis database PROFILE_DB #### Link to config_db schema for YANG module changes #### A picture of a cute animal (not mandatory but encouraged) --- rules/swss-common.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rules/swss-common.mk b/rules/swss-common.mk index 9ab49b672b6c..d2c4390ef4ff 100644 --- a/rules/swss-common.mk +++ b/rules/swss-common.mk @@ -9,9 +9,9 @@ $(LIBSWSSCOMMON)_VERSION = $(LIBSWSSCOMMON_VERSION) $(LIBSWSSCOMMON)_NAME = $(LIBSWSSCOMMON_NAME) $(LIBSWSSCOMMON)_DEPENDS += $(LIBHIREDIS_DEV) $(LIBNL3_DEV) $(LIBNL_GENL3_DEV) \ $(LIBNL_ROUTE3_DEV) $(LIBNL_NF3_DEV) \ - $(LIBNL_CLI_DEV) + $(LIBNL_CLI_DEV) $(LIBYANG_DEV) $(LIBYANG) $(LIBSWSSCOMMON)_RDEPENDS += $(LIBHIREDIS) $(LIBNL3) $(LIBNL_GENL3) \ - $(LIBNL_ROUTE3) $(LIBNL_NF3) $(LIBNL_CLI) + $(LIBNL_ROUTE3) $(LIBNL_NF3) $(LIBNL_CLI) $(LIBYANG) SONIC_DPKG_DEBS += $(LIBSWSSCOMMON) LIBSWSSCOMMON_DEV = $(LIBSWSSCOMMON_NAME)-dev_$(LIBSWSSCOMMON_VERSION)_$(CONFIGURED_ARCH).deb From f50dc28789cd2fdec99131284cc4ccc5192b3806 Mon Sep 17 00:00:00 2001 From: Xin Wang Date: Mon, 26 Sep 2022 10:48:02 +0800 Subject: [PATCH 19/43] [docker-sonic-mgmt] Deprecate azure-kusto-data & azure-kusto-ingest for py2 (#12143) Why I did it The python packages azure-kusto-data and azure-kusto-ingest packages for python2 are too old and not really used. The python3 environment has newer version of these packages installed. This change is to deprecate these two packages for python2 in docker-sonic-mgmt image. How I did it Removed the lines for installing old version of packages azure-kusto-data and azure-kusto-ingest in python2 in the Dockerfile template. Signed-off-by: Xin Wang --- dockers/docker-sonic-mgmt/Dockerfile.j2 | 4 ---- 1 file changed, 4 deletions(-) diff --git a/dockers/docker-sonic-mgmt/Dockerfile.j2 b/dockers/docker-sonic-mgmt/Dockerfile.j2 index 42aaa3525a64..fec662dff0ed 100755 --- a/dockers/docker-sonic-mgmt/Dockerfile.j2 +++ b/dockers/docker-sonic-mgmt/Dockerfile.j2 @@ -114,10 +114,6 @@ RUN apt-get update \ # Install Azure CLI RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash -# Install Microsoft Azure Kusto Library for Python -RUN pip install azure-kusto-data==0.0.13 \ - azure-kusto-ingest==0.0.13 - RUN pip install wheel==0.33.6 ## Copy and install sonic-mgmt docker dependencies From 2c10ebb4fe978bded6499365d3471e8343573d8c Mon Sep 17 00:00:00 2001 From: Aryeh Feigin <101218333+arfeigin@users.noreply.github.com> Date: Mon, 26 Sep 2022 19:01:49 +0300 Subject: [PATCH 20/43] Use warm-boot infrastructure for fast-boot (#11594) This PR should be merged together with the sonic-utilities PR (sonic-net/sonic-utilities#2286) and sonic-sairedis PR (sonic-net/sonic-sairedis#1100). Use redis contents from dump file in fast-reboot. Improve fast-reboot flow by utilizing the warm-reboot infrastructure. This followes https://github.com/sonic-net/SONiC/blob/master/doc/fast-reboot/Fast-reboot_Flow_Improvements_HLD.md --- files/build_templates/docker_image_ctl.j2 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/files/build_templates/docker_image_ctl.j2 b/files/build_templates/docker_image_ctl.j2 index b4ff4a4379c9..ace39df4e546 100644 --- a/files/build_templates/docker_image_ctl.j2 +++ b/files/build_templates/docker_image_ctl.j2 @@ -87,7 +87,7 @@ function preStartAction() {%- if docker_container_name == "database" %} WARM_DIR=/host/warmboot if [ "$DATABASE_TYPE" != "chassisdb" ]; then - if [[ ("$BOOT_TYPE" == "warm" || "$BOOT_TYPE" == "fastfast") && -f $WARM_DIR/dump.rdb ]]; then + if [[ ("$BOOT_TYPE" == "warm" || "$BOOT_TYPE" == "fastfast" || "$BOOT_TYPE" == "fast") && -f $WARM_DIR/dump.rdb ]]; then # Load redis content from /host/warmboot/dump.rdb docker cp $WARM_DIR/dump.rdb database$DEV:/var/lib/redis/dump.rdb else @@ -218,7 +218,7 @@ function postStartAction() ($(docker exec -i database$DEV sonic-db-cli PING | grep -c PONG) -gt 0) ]]; do sleep 1; done - if [[ ("$BOOT_TYPE" == "warm" || "$BOOT_TYPE" == "fastfast") && -f $WARM_DIR/dump.rdb ]]; then + if [[ ("$BOOT_TYPE" == "warm" || "$BOOT_TYPE" == "fastfast" || "$BOOT_TYPE" == "fast") && -f $WARM_DIR/dump.rdb ]]; then # retain the dump file from last boot for debugging purposes mv $WARM_DIR/dump.rdb $WARM_DIR/dump.rdb.old else From 1b50a2b72111593af51906f9bcaea89f1f763126 Mon Sep 17 00:00:00 2001 From: Tal Berlowitz <100570204+talber-nvidia@users.noreply.github.com> Date: Mon, 26 Sep 2022 19:30:38 +0300 Subject: [PATCH 21/43] Patch ifupdown2 (#9630) (#11548) --- ...alue-of-utils._execute_subprocess-me.patch | 21 +++++++++++++++++++ src/ifupdown2/patch/series | 1 + 2 files changed, 22 insertions(+) create mode 100644 src/ifupdown2/patch/0003-Fix-the-return-value-of-utils._execute_subprocess-me.patch diff --git a/src/ifupdown2/patch/0003-Fix-the-return-value-of-utils._execute_subprocess-me.patch b/src/ifupdown2/patch/0003-Fix-the-return-value-of-utils._execute_subprocess-me.patch new file mode 100644 index 000000000000..39cd481099ca --- /dev/null +++ b/src/ifupdown2/patch/0003-Fix-the-return-value-of-utils._execute_subprocess-me.patch @@ -0,0 +1,21 @@ +Fix the return value of utils._execute_subprocess method for empty strings +--- + ifupdown2/ifupdown/utils.py | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/ifupdown2/ifupdown/utils.py b/ifupdown2/ifupdown/utils.py +index d638fe9..0c5d8ce 100644 +--- a/ifupdown2/ifupdown/utils.py ++++ b/ifupdown2/ifupdown/utils.py +@@ -380,7 +380,7 @@ class utils(): + finally: + utils.disable_subprocess_signal_forwarding(signal.SIGINT) + +- cmd_output_string = cmd_output.decode() if cmd_output else cmd_output ++ cmd_output_string = cmd_output.decode() if cmd_output is not None else cmd_output + + if cmd_returncode != 0: + raise Exception(cls._format_error(cmd, +-- +2.14.1 + diff --git a/src/ifupdown2/patch/series b/src/ifupdown2/patch/series index c4e412bbe83f..7d0fa15ded61 100644 --- a/src/ifupdown2/patch/series +++ b/src/ifupdown2/patch/series @@ -1,2 +1,3 @@ 0001-fix-broadcast-addr-encoding.patch 0002-disable-checks-when-using-no-wait.patch +0003-Fix-the-return-value-of-utils._execute_subprocess-me.patch From 60c80ad26d17312b868a66ea32ccce3a1de2c8dc Mon Sep 17 00:00:00 2001 From: xumia <59720581+xumia@users.noreply.github.com> Date: Tue, 27 Sep 2022 06:55:19 +0800 Subject: [PATCH 22/43] [Build] Fix the build unstalbe issue caused by the kvm not ready (#12180) Why I did it Fix the build unstable issue caused by the kvm 9000 port is not ready to use in 2 seconds. 2022-09-02T10:57:30.8122304Z + /usr/bin/kvm -m 8192 -name onie -boot order=cd,once=d -cdrom target/files/bullseye/onie-recovery-x86_64-kvm_x86_64_4_asic-r0.iso -device e1000,netdev=onienet -netdev user,id=onienet,hostfwd=:0.0.0.0:3041-:22 -vnc 0.0.0.0:0 -vga std -drive file=target/sonic-6asic-vs.img,media=disk,if=virtio,index=0 -drive file=./sonic-installer.img,if=virtio,index=1 -serial telnet:127.0.0.1:9000,server 2022-09-02T10:57:30.8123378Z + sleep 2.0 2022-09-02T10:57:30.8123889Z + '[' -d /proc/284923 ']' 2022-09-02T10:57:30.8124528Z + echo 'to kill kvm: sudo kill 284923' 2022-09-02T10:57:30.8124994Z to kill kvm: sudo kill 284923 2022-09-02T10:57:30.8125362Z + ./install_sonic.py 2022-09-02T10:57:30.8125720Z Trying 127.0.0.1... 2022-09-02T10:57:30.8126041Z telnet: Unable to connect to remote host: Connection refused How I did it Waiting more time until the tcp port 9000 is ready, waiting for 60 seconds in maximum. --- scripts/build_kvm_image.sh | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/scripts/build_kvm_image.sh b/scripts/build_kvm_image.sh index 44009ed013f4..f3936a548299 100755 --- a/scripts/build_kvm_image.sh +++ b/scripts/build_kvm_image.sh @@ -49,6 +49,19 @@ prepare_installer_disk() umount $tmpdir } +wait_kvm_ready() +{ + local count=30 + local waiting_in_seconds=2.0 + for ((i=1; i<=$count; i++)); do + sleep $waiting_in_seconds + echo "$(date) [$i/$count] waiting for the port $KVM_PORT ready" + if netstat -l | grep -q ":$KVM_PORT"; then + break + fi + done +} + apt-get install -y net-tools create_disk prepare_installer_disk @@ -86,7 +99,7 @@ echo "Installing SONiC" kvm_pid=$! -sleep 2.0 +wait_kvm_ready [ -d "/proc/$kvm_pid" ] || { echo "ERROR: kvm died." @@ -114,7 +127,7 @@ echo "Booting up SONiC" kvm_pid=$! -sleep 2.0 +wait_kvm_ready [ -d "/proc/$kvm_pid" ] || { echo "ERROR: kvm died." From 1995540758cccbf4f3ff4c081a745db835993222 Mon Sep 17 00:00:00 2001 From: ShiyanWangMS Date: Tue, 27 Sep 2022 09:15:48 +0800 Subject: [PATCH 23/43] Upgrade docker-sonic-mgmt base image from Ubuntu18.04 to 20.04 (#12056) Upgrade docker-sonic-mgmt base image from Ubuntu18.04 to 20.04 --- dockers/docker-sonic-mgmt/Dockerfile.j2 | 27 ++++++++++++++++--------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/dockers/docker-sonic-mgmt/Dockerfile.j2 b/dockers/docker-sonic-mgmt/Dockerfile.j2 index fec662dff0ed..e803508e855f 100755 --- a/dockers/docker-sonic-mgmt/Dockerfile.j2 +++ b/dockers/docker-sonic-mgmt/Dockerfile.j2 @@ -1,5 +1,5 @@ {% set prefix = DEFAULT_CONTAINER_REGISTRY %} -FROM {{ prefix }}ubuntu:18.04 +FROM {{ prefix }}ubuntu:20.04 ENV DEBIAN_FRONTEND=noninteractive @@ -21,8 +21,6 @@ RUN apt-get update && apt-get install -y build-essential \ psmisc \ python \ python-dev \ - python-scapy \ - python-pip \ python3-pip \ python3-venv \ rsyslog \ @@ -31,10 +29,20 @@ RUN apt-get update && apt-get install -y build-essential \ sudo \ tcpdump \ telnet \ - vim + vim \ + python-is-python2 \ + software-properties-common + +RUN add-apt-repository -y universe +RUN curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py \ + && python2 get-pip.py + +RUN curl -L http://archive.ubuntu.com/ubuntu/pool/universe/s/scapy/python-scapy_2.3.3-3_all.deb \ + --output python-scapy_2.3.3-3_all.deb \ + && dpkg -i python-scapy_2.3.3-3_all.deb RUN pip install setuptools==44.1.1 -RUN pip install cffi==1.10.0 \ +RUN pip install cffi==1.12.0 \ contextlib2==0.6.0.post1 \ cryptography==3.3.2 \ "future>=0.16.0" \ @@ -96,7 +104,7 @@ RUN pip install cffi==1.10.0 \ && rm -f 1.0.0.tar.gz \ && pip install nnpy \ && pip install dpkt \ - && pip install scapy==2.4.5 --upgrade + && pip install scapy==2.4.5 --upgrade --ignore-installed # Install docker-ce-cli RUN apt-get update \ @@ -189,8 +197,7 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH" ENV LANG=C.UTF-8 LC_ALL=C.UTF-8 PYTHONIOENCODING=UTF-8 -RUN python3 -m pip install --upgrade --ignore-installed pip setuptools==58.4.0 - +RUN python3 -m pip install --upgrade --ignore-installed pip setuptools==58.4.0 wheel==0.33.6 RUN python3 -m pip install setuptools-rust \ aiohttp \ defusedxml \ @@ -206,6 +213,7 @@ RUN python3 -m pip install setuptools-rust \ ixnetwork-restpy==1.0.64 \ ixnetwork-open-traffic-generator==0.0.79 \ snappi[ixnetwork,convergence]==0.7.44 \ + markupsafe==2.0.1 \ jinja2==2.7.2 \ jsonpatch \ lxml \ @@ -233,7 +241,6 @@ RUN python3 -m pip install setuptools-rust \ tabulate \ textfsm==1.1.2 \ virtualenv \ - wheel==0.33.6 \ pysubnettree \ nnpy \ dpkt \ @@ -248,5 +255,5 @@ RUN python3 -m pip install setuptools-rust \ celery[redis]==4.4.7 \ msrest==0.6.21 -# Deactivating a virtualenv. +# Deactivating a virtualenv ENV PATH="$BACKUP_OF_PATH" From 9c602320c3df3c8e84d5819c207b939b7ea84ff3 Mon Sep 17 00:00:00 2001 From: Ye Jianquan Date: Wed, 28 Sep 2022 11:38:41 +0800 Subject: [PATCH 24/43] install missed package python-dateutil (#12197) Why I did it Fix issue of can't import dateutil.parser in show_techsupport/test_auto_techsupport.py How I did it install python-dateutil --- dockers/docker-sonic-mgmt/Dockerfile.j2 | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/dockers/docker-sonic-mgmt/Dockerfile.j2 b/dockers/docker-sonic-mgmt/Dockerfile.j2 index e803508e855f..7676341cbaed 100755 --- a/dockers/docker-sonic-mgmt/Dockerfile.j2 +++ b/dockers/docker-sonic-mgmt/Dockerfile.j2 @@ -86,6 +86,7 @@ RUN pip install cffi==1.12.0 \ allure-pytest==2.8.22 \ celery[redis]==4.4.7 \ msrest==0.6.21 \ + python-dateutil \ && git clone https://github.com/p4lang/scapy-vxlan.git \ && cd scapy-vxlan \ && python setup.py install \ @@ -253,7 +254,8 @@ RUN python3 -m pip install setuptools-rust \ ptf \ scapy==2.4.5 \ celery[redis]==4.4.7 \ - msrest==0.6.21 + msrest==0.6.21 \ + python-dateutil \ # Deactivating a virtualenv ENV PATH="$BACKUP_OF_PATH" From 7666af94030f44e48da1b4e8a8f265c32d0f2220 Mon Sep 17 00:00:00 2001 From: Ye Jianquan Date: Wed, 28 Sep 2022 14:39:33 +0800 Subject: [PATCH 25/43] Fix pip install error (#12198) Fix the error of pip install introduced in PR #12197 --- dockers/docker-sonic-mgmt/Dockerfile.j2 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dockers/docker-sonic-mgmt/Dockerfile.j2 b/dockers/docker-sonic-mgmt/Dockerfile.j2 index 7676341cbaed..c6b4d1600a8e 100755 --- a/dockers/docker-sonic-mgmt/Dockerfile.j2 +++ b/dockers/docker-sonic-mgmt/Dockerfile.j2 @@ -255,7 +255,7 @@ RUN python3 -m pip install setuptools-rust \ scapy==2.4.5 \ celery[redis]==4.4.7 \ msrest==0.6.21 \ - python-dateutil \ + python-dateutil # Deactivating a virtualenv ENV PATH="$BACKUP_OF_PATH" From f890606d8252c0e181bf16ea0c83846d9a37bbbf Mon Sep 17 00:00:00 2001 From: Junchao-Mellanox <57339448+Junchao-Mellanox@users.noreply.github.com> Date: Wed, 28 Sep 2022 15:15:26 +0800 Subject: [PATCH 26/43] Revert "[Mellanox] Redirect ethtool stderr to subprocess for better error log (#12038)" (#12183) This reverts commit 9750cb4. There is a PR to handle 202205 branch revert: #12184 - Why I did it The PR to be reverted introduced many notice logs every 1 minute if SFP is not plugged: Cannot get module EEPROM information: Input/output error Before the "bad" PR, the message format is like this: INFO pmon#supervisord: xcvrd Cannot get module EEPROM information: Input/output error It was truncated by rsyslog because every message is the same. However, the "bad" PR introduces SFP index to the message: NOTICE pmon#xcvrd: Failed to get EEPROM data for sfp 39: Cannot get module EEPROM information: Input/output error Rsyslog no longer truncate such log and many such messages are flooded to syslog. - How I did it Revert the PR - How to verify it Manual test --- platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py b/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py index d373b6a8f705..617b4f33d636 100644 --- a/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py +++ b/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py @@ -359,8 +359,7 @@ def _read_eeprom_specific_bytes(self, offset, num_bytes): try: output = subprocess.check_output(ethtool_cmd, shell=True, - universal_newlines=True, - stderr=subprocess.PIPE) + universal_newlines=True) output_lines = output.splitlines() first_line_raw = output_lines[0] if "Offset" in first_line_raw: @@ -368,7 +367,6 @@ def _read_eeprom_specific_bytes(self, offset, num_bytes): line_split = line.split() eeprom_raw = eeprom_raw + line_split[1:] except subprocess.CalledProcessError as e: - logger.log_notice("Failed to get EEPROM data for sfp {}: {}".format(self.index, e.stderr)) return None eeprom_raw = list(map(lambda h: int(h, base=16), eeprom_raw)) From 4d317aff9491a56c094f96966bb0e3ff9d009611 Mon Sep 17 00:00:00 2001 From: Stephen Sun <5379172+stephenxs@users.noreply.github.com> Date: Wed, 28 Sep 2022 16:09:18 +0800 Subject: [PATCH 27/43] [Mellanox] Fix typo in platform API (#12136) - Why I did it Fix a typo in chassis platform API which causes the following error >>> import sonic_platform as P >>> c = P.platform.Platform().get_chassis() >>> sl = c.get_all_sfps() >>> sl[0].get_lpmode() Sep 28 07:48:33 INFO LOG: Initializing SX log with STDOUT as output file. False >>> del c Exception ignored in: Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/sonic_platform/chassis.py", line 126, in __del__ self.sfp_module.deinitialize_sdk_handle(sfp_module.SFP.shared_sdk_handle) NameError: name 'sfp_module' is not defined - How I did it Use self while using the SDK handle - How to verify it Manual test Signed-off-by: Stephen Sun --- platform/mellanox/mlnx-platform-api/sonic_platform/chassis.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform/mellanox/mlnx-platform-api/sonic_platform/chassis.py b/platform/mellanox/mlnx-platform-api/sonic_platform/chassis.py index b9fa2593174c..96a7b9e2315a 100644 --- a/platform/mellanox/mlnx-platform-api/sonic_platform/chassis.py +++ b/platform/mellanox/mlnx-platform-api/sonic_platform/chassis.py @@ -123,7 +123,7 @@ def __del__(self): if self._sfp_list: if self.sfp_module.SFP.shared_sdk_handle: - self.sfp_module.deinitialize_sdk_handle(sfp_module.SFP.shared_sdk_handle) + self.sfp_module.deinitialize_sdk_handle(self.sfp_module.SFP.shared_sdk_handle) @property def RJ45_port_list(self): From 8c7e0f8e02233dc4236f8f7f048b4281ab9d31b8 Mon Sep 17 00:00:00 2001 From: vijayvyasm <112528485+vijayvyasm@users.noreply.github.com> Date: Wed, 28 Sep 2022 18:37:33 -0700 Subject: [PATCH 28/43] Support for serdes platform library debian installation for Innovium SONiC image (#11920) Signed-off-by: vijayvyasm vijayvyasm@marvell.com Signed-off-by: vijayvyasm vijayvyasm@marvell.com --- platform/innovium/invm-sai.mk | 4 +++- platform/innovium/one-image.mk | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/platform/innovium/invm-sai.mk b/platform/innovium/invm-sai.mk index 25e490e44f83..734184e35b0c 100755 --- a/platform/innovium/invm-sai.mk +++ b/platform/innovium/invm-sai.mk @@ -5,11 +5,13 @@ INVM_SAI_ONLINE = https://github.com/Innovium/SONiC/raw/master/debian/master INVM_LIBSAI = isai.deb INVM_HSAI = saihdr.deb INVM_DRV = ipd.deb +INVM_SERDES_PLATFORM_LIBRARY = ivm_serdes_pltfm.deb $(INVM_LIBSAI)_URL = $(INVM_SAI_ONLINE)/$(INVM_LIBSAI) $(INVM_HSAI)_URL = $(INVM_SAI_ONLINE)/$(INVM_HSAI) $(INVM_DRV)_URL = $(INVM_SAI_ONLINE)/$(INVM_DRV) +$(INVM_SERDES_PLATFORM_LIBRARY)_URL = $(INVM_SAI_ONLINE)/$(INVM_SERDES_PLATFORM_LIBRARY) $(eval $(call add_conflict_package,$(INVM_HSAI),$(LIBSAIVS_DEV))) -SONIC_ONLINE_DEBS += $(INVM_LIBSAI) $(INVM_HSAI) $(INVM_DRV) +SONIC_ONLINE_DEBS += $(INVM_LIBSAI) $(INVM_HSAI) $(INVM_DRV) $(INVM_SERDES_PLATFORM_LIBRARY) diff --git a/platform/innovium/one-image.mk b/platform/innovium/one-image.mk index 2cae779d71ac..39e2e9fc8246 100755 --- a/platform/innovium/one-image.mk +++ b/platform/innovium/one-image.mk @@ -8,6 +8,6 @@ $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(CEL_MIDSTONE_200I_PLATFORM_MODULE) $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(DELTA_PLATFORM_MODULE) $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(NETBERG_AURORA_715_PLATFORM_MODULE) $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(WISTRON_PLATFORM_MODULE) -$(SONIC_ONE_IMAGE)_INSTALLS += $(INVM_DRV) +$(SONIC_ONE_IMAGE)_INSTALLS += $(INVM_DRV) $(INVM_SERDES_PLATFORM_LIBRARY) $(SONIC_ONE_IMAGE)_DOCKERS += $(SONIC_INSTALL_DOCKER_IMAGES) SONIC_INSTALLERS += $(SONIC_ONE_IMAGE) From 1f9c89a8d34ab94cd871d0dd798d991d5e4cf00e Mon Sep 17 00:00:00 2001 From: Hua Liu <58683130+liuh-80@users.noreply.github.com> Date: Thu, 29 Sep 2022 10:27:57 +0800 Subject: [PATCH 29/43] [sonic-py-common] porting sonic_db_dump_load.py from sonic-py-swsssdk to sonic-py-common (#12185) Porting sonic_db_dump_load.py from sonic-py-swsssdk to sonic-py-common. #### Why I did it sonic-py-swsssdk will be deprecate, so porting sonic_db_dump_load.py to sonic-py-common. #### How I did it Copy sonic_db_dump_load.py to sonic-py-common, and fix minor API different. #### How to verify it Pass all E2E test. The platform_tests/test_advanced_reboot.py::test_warm_reboot will cover this script. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 #### Description for the changelog Porting sonic_db_dump_load.py from sonic-py-swsssdk to sonic-py-common. #### Ensure to add label/tag for the feature raised. example - [PR#2174](https://github.com/sonic-net/sonic-utilities/pull/2174) where, Generic Config and Update feature has been labelled as GCU. #### Link to config_db schema for YANG module changes #### A picture of a cute animal (not mandatory but encouraged) --- src/sonic-py-common/setup.py | 6 + .../sonic_py_common/sonic_db_dump_load.py | 139 ++++++++++++++++++ 2 files changed, 145 insertions(+) create mode 100755 src/sonic-py-common/sonic_py_common/sonic_db_dump_load.py diff --git a/src/sonic-py-common/setup.py b/src/sonic-py-common/setup.py index f12c0d83cb77..144cf61f52c4 100644 --- a/src/sonic-py-common/setup.py +++ b/src/sonic-py-common/setup.py @@ -27,6 +27,12 @@ 'pytest', 'mock==3.0.5' # For python 2. Version >=4.0.0 drops support for py2 ], + entry_points={ + 'console_scripts': [ + 'sonic-db-load = sonic_py_common.sonic_db_dump_load:sonic_db_dump_load', + 'sonic-db-dump = sonic_py_common.sonic_db_dump_load:sonic_db_dump_load', + ], + }, classifiers=[ 'Intended Audience :: Developers', 'Operating System :: Linux', diff --git a/src/sonic-py-common/sonic_py_common/sonic_db_dump_load.py b/src/sonic-py-common/sonic_py_common/sonic_db_dump_load.py new file mode 100755 index 000000000000..126dd2bc112e --- /dev/null +++ b/src/sonic-py-common/sonic_py_common/sonic_db_dump_load.py @@ -0,0 +1,139 @@ +## ref: https://github.com/p/redis-dump-load/blob/7bbdb1eaea0a51ed4758d3ce6ca01d497a4e7429/redisdl.py + +def sonic_db_dump_load(): + import optparse + import os.path + import re + import sys + from redisdl import dump, load + from swsscommon.swsscommon import SonicDBConfig + + DUMP = 1 + LOAD = 2 + + def options_to_kwargs(options): + args = {} + if options.password: + args['password'] = options.password + if options.encoding: + args['encoding'] = options.encoding + # dump only + if hasattr(options, 'pretty') and options.pretty: + args['pretty'] = True + if hasattr(options, 'keys') and options.keys: + args['keys'] = options.keys + # load only + if hasattr(options, 'use_expireat') and options.use_expireat: + args['use_expireat'] = True + if hasattr(options, 'empty') and options.empty: + args['empty'] = True + if hasattr(options, 'backend') and options.backend: + args['streaming_backend'] = options.backend + if hasattr(options, 'dbname') and options.dbname: + if options.conntype == 'tcp': + args['host'] = SonicDBConfig.getDbHostname(options.dbname) + args['port'] = SonicDBConfig.getDbPort(options.dbname) + args['db'] = SonicDBConfig.getDbId(options.dbname) + args['unix_socket_path'] = None + elif options.conntype == "unix_socket": + args['host'] = None + args['port'] = None + args['db'] = SonicDBConfig.getDbId(options.dbname) + args['unix_socket_path'] = SonicDBConfig.getDbSock(options.dbname) + else: + raise TypeError('redis connection type is tcp or unix_socket') + + return args + + def do_dump(options): + if options.output: + output = open(options.output, 'w') + else: + output = sys.stdout + + kwargs = options_to_kwargs(options) + dump(output, **kwargs) + + if options.output: + output.close() + + def do_load(options, args): + if len(args) > 0: + input = open(args[0], 'rb') + else: + input = sys.stdin + + kwargs = options_to_kwargs(options) + load(input, **kwargs) + + if len(args) > 0: + input.close() + + script_name = os.path.basename(sys.argv[0]) + if re.search(r'load(?:$|\.)', script_name): + action = help = LOAD + elif re.search(r'dump(?:$|\.)', script_name): + action = help = DUMP + else: + # default is dump, however if dump is specifically requested + # we don't show help text for toggling between dumping and loading + action = DUMP + help = None + + if help == LOAD: + usage = "Usage: %prog [options] [FILE]" + usage += "\n\nLoad data from FILE (which must be a JSON dump previously created" + usage += "\nby redisdl) into specified or default redis." + usage += "\n\nIf FILE is omitted standard input is read." + elif help == DUMP: + usage = "Usage: %prog [options]" + usage += "\n\nDump data from specified or default redis." + usage += "\n\nIf no output file is specified, dump to standard output." + else: + usage = "Usage: %prog [options]" + usage += "\n %prog -l [options] [FILE]" + usage += "\n\nDump data from redis or load data into redis." + usage += "\n\nIf input or output file is specified, dump to standard output and load" + usage += "\nfrom standard input." + parser = optparse.OptionParser(usage=usage) + parser.add_option('-w', '--password', help='connect with PASSWORD') + if help == DUMP: + parser.add_option('-n', '--dbname', help='dump DATABASE (APPL_DB/ASIC_DB...)') + parser.add_option('-t', '--conntype', help='indicate redis connection type (tcp[default] or unix_socket)', default='tcp') + parser.add_option('-k', '--keys', help='dump only keys matching specified glob-style pattern') + parser.add_option('-o', '--output', help='write to OUTPUT instead of stdout') + parser.add_option('-y', '--pretty', help='split output on multiple lines and indent it', action='store_true') + parser.add_option('-E', '--encoding', help='set encoding to use while decoding data from redis', default='utf-8') + elif help == LOAD: + parser.add_option('-n', '--dbname', help='dump DATABASE (APPL_DB/ASIC_DB...)') + parser.add_option('-t', '--conntype', help='indicate redis connection type (tcp[default] or unix_socket)', default='tcp') + parser.add_option('-e', '--empty', help='delete all keys in destination db prior to loading', action='store_true') + parser.add_option('-E', '--encoding', help='set encoding to use while encoding data to redis', default='utf-8') + parser.add_option('-B', '--backend', help='use specified streaming backend') + parser.add_option('-A', '--use-expireat', help='use EXPIREAT rather than TTL/EXPIRE', action='store_true') + else: + parser.add_option('-l', '--load', help='load data into redis (default is to dump data from redis)', action='store_true') + parser.add_option('-n', '--dbname', help='dump DATABASE (APPL_DB/ASIC_DB/COUNTERS_DB/LOGLEVEL_DB/CONFIG_DB...)') + parser.add_option('-t', '--conntype', help='indicate redis connection type (tcp[default] or unix_socket)', default='tcp') + parser.add_option('-k', '--keys', help='dump only keys matching specified glob-style pattern') + parser.add_option('-o', '--output', help='write to OUTPUT instead of stdout (dump mode only)') + parser.add_option('-y', '--pretty', help='split output on multiple lines and indent it (dump mode only)', action='store_true') + parser.add_option('-e', '--empty', help='delete all keys in destination db prior to loading (load mode only)', action='store_true') + parser.add_option('-E', '--encoding', help='set encoding to use while decoding data from redis', default='utf-8') + parser.add_option('-A', '--use-expireat', help='use EXPIREAT rather than TTL/EXPIRE', action='store_true') + parser.add_option('-B', '--backend', help='use specified streaming backend (load mode only)') + options, args = parser.parse_args() + + if hasattr(options, 'load') and options.load: + action = LOAD + + if action == DUMP: + if len(args) > 0: + parser.print_help() + exit(4) + do_dump(options) + else: + if len(args) > 1: + parser.print_help() + exit(4) + do_load(options, args) From d9c9c70fb543ed4e6c8bfff24e17e65613409e72 Mon Sep 17 00:00:00 2001 From: Dmytro Lytvynenko Date: Fri, 30 Sep 2022 01:12:01 +0300 Subject: [PATCH 30/43] [BFN] Move qsfp eeprom reading to new cached api (#9909) * Move qsfp eeprom reading to new cached api * provide reading multiple pages in recursive manner * workaround with flat memory on cmis * remove workaround with memory model * Remove unused imports --- .../sonic_platform/platform_thrift_client.py | 11 ++-- .../sonic_platform/pltfm_mgr_rpc/ttypes.py | 18 ------ .../sonic_platform/sfp.py | 62 +++++++++++++------ 3 files changed, 48 insertions(+), 43 deletions(-) diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_thrift_client.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_thrift_client.py index dff16577de74..8490d132a2df 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_thrift_client.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_thrift_client.py @@ -1,12 +1,10 @@ #!/usr/bin/env python try: - import os - import sys import time - import importlib - sys.path.append(os.path.dirname(__file__)) + from sonic_platform.pltfm_mgr_rpc.pltfm_mgr_rpc import Client + from sonic_platform.pltfm_mgr_rpc.pltfm_mgr_rpc import InvalidPltfmMgrOperation from thrift.transport import TSocket from thrift.transport import TTransport @@ -25,9 +23,8 @@ def open(self): self.transport = TTransport.TBufferedTransport(self.transport) bprotocol = TBinaryProtocol.TBinaryProtocol(self.transport) - self.pltfm_mgr_module = importlib.import_module(".".join(["pltfm_mgr_rpc", "pltfm_mgr_rpc"])) pltfm_mgr_protocol = TMultiplexedProtocol.TMultiplexedProtocol(bprotocol, "pltfm_mgr_rpc") - self.pltfm_mgr = self.pltfm_mgr_module.Client(pltfm_mgr_protocol) + self.pltfm_mgr = Client(pltfm_mgr_protocol) self.transport.open() return self @@ -59,7 +56,7 @@ def pltfm_mgr_try(func, default=None, thrift_attempts=35): def pm_cb_run(client): try: return (None, func(client.pltfm_mgr)) - except client.pltfm_mgr_module.InvalidPltfmMgrOperation as ouch: + except InvalidPltfmMgrOperation as ouch: return (ouch.code, default) return thrift_try(pm_cb_run) diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py index 501596941664..391d4bd5377b 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py @@ -16,24 +16,6 @@ all_structs = [] -class qsfp_eeprom_page_t(object): - PAGE0_LOWER = 0 - PAGE0_UPPER = 1 - PAGE3 = 2 - - _VALUES_TO_NAMES = { - 0: "PAGE0_LOWER", - 1: "PAGE0_UPPER", - 2: "PAGE3", - } - - _NAMES_TO_VALUES = { - "PAGE0_LOWER": 0, - "PAGE0_UPPER": 1, - "PAGE3": 2, - } - - class pltfm_mgr_sys_tmp_t(object): """ Attributes: diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/sfp.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/sfp.py index f5d800b749c9..6a5534d8ba6b 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/sfp.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/sfp.py @@ -11,29 +11,38 @@ SFP_TYPE = "SFP" QSFP_TYPE = "QSFP" QSFP_DD_TYPE = "QSFP_DD" +EEPROM_PAGE_SIZE = 128 +try: + from thrift.Thrift import TApplicationException + + def cached_num_bytes_get(client): + return client.pltfm_mgr.pltfm_mgr_qsfp_cached_num_bytes_get(1, 0, 0, 0) + thrift_try(cached_num_bytes_get, 1) + EEPROM_CACHED_API_SUPPORT = True +except TApplicationException as e: + EEPROM_CACHED_API_SUPPORT = False class Sfp(SfpOptoeBase): """ BFN Platform-specific SFP class """ - SFP_EEPROM_PATH = "/var/run/platform/sfp/" - def __init__(self, port_num): SfpOptoeBase.__init__(self) self.index = port_num self.port_num = port_num self.sfp_type = QSFP_TYPE + self.SFP_EEPROM_PATH = "/var/run/platform/sfp/" - if not os.path.exists(self.SFP_EEPROM_PATH): - try: - os.makedirs(self.SFP_EEPROM_PATH) - except OSError as e: - if e.errno != errno.EEXIST: - raise - - self.eeprom_path = self.SFP_EEPROM_PATH + "sfp{}-eeprom-cache".format(self.index) + if not EEPROM_CACHED_API_SUPPORT: + if not os.path.exists(self.SFP_EEPROM_PATH): + try: + os.makedirs(self.SFP_EEPROM_PATH) + except OSError as e: + if e.errno != errno.EEXIST: + raise + self.eeprom_path = self.SFP_EEPROM_PATH + "sfp{}-eeprom-cache".format(self.index) def get_presence(self): """ @@ -47,7 +56,7 @@ def qsfp_presence_get(client): try: presence = thrift_try(qsfp_presence_get) except Exception as e: - print( e.__doc__) + print(e.__doc__) print(e.message) return presence @@ -75,14 +84,31 @@ def get_eeprom_path(self): def qsfp_info_get(client): return client.pltfm_mgr.pltfm_mgr_qsfp_info_get(self.index) - if self.get_presence(): - eeprom_hex = thrift_try(qsfp_info_get) - eeprom_raw = bytearray.fromhex(eeprom_hex) - with open(self.eeprom_path, 'wb') as fp: - fp.write(eeprom_raw) - return self.eeprom_path + eeprom_hex = thrift_try(qsfp_info_get) + eeprom_raw = bytearray.fromhex(eeprom_hex) + with open(self.eeprom_path, 'wb') as fp: + fp.write(eeprom_raw) + return self.eeprom_path + + def read_eeprom(self, offset, num_bytes): + if not self.get_presence(): + return None + + if not EEPROM_CACHED_API_SUPPORT: + return super().read_eeprom(offset, num_bytes) + + def cached_num_bytes_get(page, offset, num_bytes): + def qsfp_cached_num_bytes_get(client): + return client.pltfm_mgr.pltfm_mgr_qsfp_cached_num_bytes_get(self.index, page, offset, num_bytes) + return bytearray.fromhex(thrift_try(qsfp_cached_num_bytes_get)) + + page_offset = offset % EEPROM_PAGE_SIZE + if page_offset + num_bytes > EEPROM_PAGE_SIZE: + curr_page_num_bytes_left = EEPROM_PAGE_SIZE - page_offset + curr_page_bytes = cached_num_bytes_get(offset // EEPROM_PAGE_SIZE, page_offset, curr_page_num_bytes_left) + return curr_page_bytes + self.read_eeprom(offset + curr_page_num_bytes_left, num_bytes - curr_page_num_bytes_left) - return None + return cached_num_bytes_get(offset // EEPROM_PAGE_SIZE, page_offset, num_bytes) def write_eeprom(self, offset, num_bytes, write_buffer): # Not supported at the moment From d08fcc971cbd7b27c7140eaa7b1baaad433ddba8 Mon Sep 17 00:00:00 2001 From: Dmytro Lytvynenko Date: Fri, 30 Sep 2022 01:13:46 +0300 Subject: [PATCH 31/43] [BFN] Updated syseeprom platform plugin to use onie-eeprom (#10556) * Align system eeprom info with ONIE * revert linked sonic_platform implementation * refactor into one class * refactor after review --- .../sonic_platform/chassis.py | 32 ++-- .../sonic_platform/eeprom.py | 155 +++++++++------- .../pltfm_mgr_rpc/pltfm_mgr_rpc.py | 175 ++++++++++++++++++ .../sonic_platform/pltfm_mgr_rpc/ttypes.py | 62 +++++++ 4 files changed, 336 insertions(+), 88 deletions(-) diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/chassis.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/chassis.py index 6d0e8b8c120c..1041561db423 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/chassis.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/chassis.py @@ -13,8 +13,8 @@ from sonic_platform.psu import psu_list_get from sonic_platform.fan_drawer import fan_drawer_list_get from sonic_platform.thermal import thermal_list_get - from eeprom import Eeprom - from platform_utils import file_create + from sonic_platform.platform_utils import file_create + from sonic_platform.eeprom import Eeprom from sonic_platform.platform_thrift_client import pltfm_mgr_ready from sonic_platform.platform_thrift_client import thrift_try @@ -40,7 +40,10 @@ class Chassis(ChassisBase): def __init__(self): ChassisBase.__init__(self) - self.__eeprom = None + self._eeprom = Eeprom() + self.__tlv_bin_eeprom = self._eeprom.get_raw_data() + self.__tlv_dict_eeprom = self._eeprom.get_data() + self.__fan_drawers = None self.__fan_list = None self.__thermals = None @@ -57,16 +60,6 @@ def __init__(self): file_create(config_dict['handlers']['file']['filename'], '646') logging.config.dictConfig(config_dict) - @property - def _eeprom(self): - if self.__eeprom is None: - self.__eeprom = Eeprom() - return self.__eeprom - - @_eeprom.setter - def _eeprom(self, value): - pass - @property def _fan_drawer_list(self): if self.__fan_drawers is None: @@ -152,7 +145,7 @@ def get_name(self): Returns: string: The name of the chassis """ - return self._eeprom.modelstr() + return self._eeprom.modelstr(self.__tlv_bin_eeprom) def get_presence(self): """ @@ -168,7 +161,7 @@ def get_model(self): Returns: string: Model/part number of chassis """ - return self._eeprom.part_number_str() + return self._eeprom.part_number_str(self.__tlv_bin_eeprom) def get_serial(self): """ @@ -176,7 +169,7 @@ def get_serial(self): Returns: string: Serial number of chassis """ - return self._eeprom.serial_number_str() + return self._eeprom.serial_number_str(self.__tlv_bin_eeprom) def get_revision(self): """ @@ -184,7 +177,8 @@ def get_revision(self): Returns: string: Revision number of chassis """ - return self._eeprom.revision_str() + return self.__tlv_dict_eeprom.get( + "0x{:X}".format(Eeprom._TLV_CODE_LABEL_REVISION), 'N/A') def get_sfp(self, index): """ @@ -225,7 +219,7 @@ def get_base_mac(self): A string containing the MAC address in the format 'XX:XX:XX:XX:XX:XX' """ - return self._eeprom.base_mac_addr() + return self._eeprom.base_mac_addr(self.__tlv_bin_eeprom) def get_system_eeprom_info(self): """ @@ -236,7 +230,7 @@ def get_system_eeprom_info(self): OCP ONIE TlvInfo EEPROM format and values are their corresponding values. """ - return self._eeprom.system_eeprom_info() + return self.__tlv_dict_eeprom def __get_transceiver_change_event(self, timeout=0): forever = False diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/eeprom.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/eeprom.py index 2335c02863d9..4b5c1e3051fb 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/eeprom.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/eeprom.py @@ -2,7 +2,8 @@ import os import sys import datetime - import re + import logging + import logging.config sys.path.append(os.path.dirname(__file__)) @@ -13,13 +14,15 @@ from sonic_platform_base.sonic_eeprom import eeprom_base from sonic_platform_base.sonic_eeprom import eeprom_tlvinfo - from platform_utils import file_create - from platform_thrift_client import thrift_try + from sonic_py_common import device_info + + from sonic_platform.platform_thrift_client import thrift_try + from sonic_platform.platform_utils import file_create + except ImportError as e: raise ImportError (str(e) + "- required module not found") - _platform_eeprom_map = { "prod_name" : ("Product Name", "0x21", 12), "odm_pcba_part_num" : ("Part Number", "0x22", 13), @@ -44,25 +47,55 @@ class Eeprom(eeprom_tlvinfo.TlvInfoDecoder): def __init__(self): file_create(_EEPROM_SYMLINK, '646') file_create(_EEPROM_STATUS, '646') - with open(_EEPROM_STATUS, 'w') as f: - f.write("initializing..") + super(Eeprom, self).__init__(_EEPROM_SYMLINK, 0, _EEPROM_STATUS, True) - self.eeprom_path = _EEPROM_SYMLINK - super(Eeprom, self).__init__(self.eeprom_path, 0, _EEPROM_STATUS, True) - - def sys_eeprom_get(client): - return client.pltfm_mgr.pltfm_mgr_sys_eeprom_get() + self._eeprom_bin = bytearray() + self.report_status("initializing..") try: - platform_eeprom = thrift_try(sys_eeprom_get) - except Exception: - raise RuntimeError("eeprom.py: Initialization failed") + try: + if device_info.get_platform() in ["x86_64-accton_as9516_32d-r0", + "x86_64-accton_as9516bf_32d-r0"]: + def tlv_eeprom_get(client): + return client.pltfm_mgr.pltfm_mgr_tlv_eeprom_get() + try: + self._eeprom_bin = bytearray.fromhex( + thrift_try(tlv_eeprom_get, 1).raw_content_hex) + except TApplicationException as e: + raise RuntimeError("api is not supported") + except Exception as e: + self._eeprom_bin = bytearray.fromhex( + thrift_try(tlv_eeprom_get).raw_content_hex) + else: + raise RuntimeError("platform is not supported") + + except RuntimeError as e: + logging.warning("Tlv eeprom fetching failed: %s, using OpenBMC" % (str(e))) + + def sys_eeprom_get(client): + return client.pltfm_mgr.pltfm_mgr_sys_eeprom_get() + + eeprom_params = self.platfrom_eeprom_to_params(thrift_try(sys_eeprom_get)) + stdout_stream = sys.stdout + sys.stdout = open(os.devnull, 'w') + self._eeprom_bin = self.set_eeprom(self._eeprom_bin, [eeprom_params]) + sys.stdout.close() + sys.stdout = stdout_stream + try: + self.write_eeprom(self._eeprom_bin) + self.report_status("ok") + except IOError as e: + logging.error("Failed to write eeprom: %s" % (str(e))) - self.__eeprom_init(platform_eeprom) + except Exception as e: + logging.error("eeprom.py: Initialization failed: %s" % (str(e))) + raise RuntimeError("eeprom.py: Initialization failed: %s" % (str(e))) - def __eeprom_init(self, platform_eeprom): - with open(_EEPROM_STATUS, 'w') as f: - f.write("ok") + self._system_eeprom_info = dict() + visitor = EepromContentVisitor(self._system_eeprom_info) + self.visit_eeprom(self._eeprom_bin, visitor) + @staticmethod + def platfrom_eeprom_to_params(platform_eeprom): eeprom_params = "" for attr, val in platform_eeprom.__dict__.items(): if val is None: @@ -86,57 +119,41 @@ def __eeprom_init(self, platform_eeprom): if len(eeprom_params) > 0: eeprom_params += "," eeprom_params += "{0:s}={1:s}".format(elem[1], value) + return eeprom_params - orig_stdout = sys.stdout - sys.stdout = StringIO() - try: - eeprom_data = eeprom_tlvinfo.TlvInfoDecoder.set_eeprom(self, "", [eeprom_params]) - finally: - decode_output = sys.stdout.getvalue() - sys.stdout = orig_stdout - - eeprom_base.EepromDecoder.write_eeprom(self, eeprom_data) - self.__eeprom_tlv_dict = self.__parse_output(decode_output) + def get_data(self): + return self._system_eeprom_info - def __parse_output(self, decode_output): - EEPROM_DECODE_HEADLINES = 6 - lines = decode_output.replace('\0', '').split('\n') - lines = lines[EEPROM_DECODE_HEADLINES:] - res = dict() - - for line in lines: - try: - # match whitespace-separated tag hex, length and value (value is mathced with its whitespaces) - match = re.search('(0x[0-9a-fA-F]{2})([\s]+[\S]+[\s]+)([\S]+[\s]*[\S]*)', line) - if match is not None: - code = match.group(1) - value = match.group(3).rstrip('\0') - res[code] = value - except Exception: - pass - return res - - def __tlv_get(self, code): - return self.__eeprom_tlv_dict.get("0x{:X}".format(code), 'N/A') - - def system_eeprom_info(self): - return self.__eeprom_tlv_dict - - def serial_number_str(self): - return self.__tlv_get(self._TLV_CODE_SERIAL_NUMBER) - - def serial_str(self): - return self.serial_number_str() - - def base_mac_addr(self): - return self.__tlv_get(self._TLV_CODE_MAC_BASE) - - def part_number_str(self): - return self.__tlv_get(self._TLV_CODE_PART_NUMBER) - - def modelstr(self): - return self.__tlv_get(self._TLV_CODE_PRODUCT_NAME) - - def revision_str(self): - return self.__tlv_get(self._TLV_CODE_LABEL_REVISION) + def get_raw_data(self): + return self._eeprom_bin + def report_status(self, status): + status_file = None + try: + status_file = open(_EEPROM_STATUS, "w") + status_file.write(status) + except IOError as e: + logging.error("Failed to report state: %s" % (str(e))) + finally: + if status_file is not None: + status_file.close() + +class EepromContentVisitor(eeprom_tlvinfo.EepromDefaultVisitor): + def __init__(self, content_dict): + self.content_dict = content_dict + + def visit_tlv(self, name, code, length, value): + if code != Eeprom._TLV_CODE_VENDOR_EXT: + self.content_dict["0x{:X}".format(code)] = value.rstrip('\0') + else: + if value: + value = value.rstrip('\0') + if value: + code = "0x{:X}".format(code) + if code not in self.content_dict: + self.content_dict[code] = [value] + else: + self.content_dict[code].append(value) + + def set_error(self, error): + logging.error("EepromContentVisitor error: %s" % (str(error))) diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/pltfm_mgr_rpc.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/pltfm_mgr_rpc.py index 0fa03d58b31a..b671be1be313 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/pltfm_mgr_rpc.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/pltfm_mgr_rpc.py @@ -33,6 +33,9 @@ def pltfm_mgr_sys_tmp_get(self): def pltfm_mgr_sys_eeprom_get(self): pass + def pltfm_mgr_tlv_eeprom_get(self): + pass + def pltfm_mgr_pwr_supply_present_get(self, ps_num): """ Parameters: @@ -403,6 +406,34 @@ def recv_pltfm_mgr_sys_eeprom_get(self): raise result.ouch raise TApplicationException(TApplicationException.MISSING_RESULT, "pltfm_mgr_sys_eeprom_get failed: unknown result") + def pltfm_mgr_tlv_eeprom_get(self): + self.send_pltfm_mgr_tlv_eeprom_get() + return self.recv_pltfm_mgr_tlv_eeprom_get() + + def send_pltfm_mgr_tlv_eeprom_get(self): + self._oprot.writeMessageBegin('pltfm_mgr_tlv_eeprom_get', TMessageType.CALL, self._seqid) + args = pltfm_mgr_tlv_eeprom_get_args() + args.write(self._oprot) + self._oprot.writeMessageEnd() + self._oprot.trans.flush() + + def recv_pltfm_mgr_tlv_eeprom_get(self): + iprot = self._iprot + (fname, mtype, rseqid) = iprot.readMessageBegin() + if mtype == TMessageType.EXCEPTION: + x = TApplicationException() + x.read(iprot) + iprot.readMessageEnd() + raise x + result = pltfm_mgr_tlv_eeprom_get_result() + result.read(iprot) + iprot.readMessageEnd() + if result.success is not None: + return result.success + if result.ouch is not None: + raise result.ouch + raise TApplicationException(TApplicationException.MISSING_RESULT, "pltfm_mgr_tlv_eeprom_get failed: unknown result") + def pltfm_mgr_pwr_supply_present_get(self, ps_num): """ Parameters: @@ -1579,6 +1610,7 @@ def __init__(self, handler): self._processMap["pltfm_mgr_dummy"] = Processor.process_pltfm_mgr_dummy self._processMap["pltfm_mgr_sys_tmp_get"] = Processor.process_pltfm_mgr_sys_tmp_get self._processMap["pltfm_mgr_sys_eeprom_get"] = Processor.process_pltfm_mgr_sys_eeprom_get + self._processMap["pltfm_mgr_tlv_eeprom_get"] = Processor.process_pltfm_mgr_tlv_eeprom_get self._processMap["pltfm_mgr_pwr_supply_present_get"] = Processor.process_pltfm_mgr_pwr_supply_present_get self._processMap["pltfm_mgr_pwr_supply_info_get"] = Processor.process_pltfm_mgr_pwr_supply_info_get self._processMap["pltfm_mgr_pwr_rail_info_get"] = Processor.process_pltfm_mgr_pwr_rail_info_get @@ -1710,6 +1742,32 @@ def process_pltfm_mgr_sys_eeprom_get(self, seqid, iprot, oprot): oprot.writeMessageEnd() oprot.trans.flush() + def process_pltfm_mgr_tlv_eeprom_get(self, seqid, iprot, oprot): + args = pltfm_mgr_tlv_eeprom_get_args() + args.read(iprot) + iprot.readMessageEnd() + result = pltfm_mgr_tlv_eeprom_get_result() + try: + result.success = self._handler.pltfm_mgr_tlv_eeprom_get() + msg_type = TMessageType.REPLY + except TTransport.TTransportException: + raise + except InvalidPltfmMgrOperation as ouch: + msg_type = TMessageType.REPLY + result.ouch = ouch + except TApplicationException as ex: + logging.exception('TApplication exception in handler') + msg_type = TMessageType.EXCEPTION + result = ex + except Exception: + logging.exception('Unexpected exception in handler') + msg_type = TMessageType.EXCEPTION + result = TApplicationException(TApplicationException.INTERNAL_ERROR, 'Internal error') + oprot.writeMessageBegin("pltfm_mgr_tlv_eeprom_get", msg_type, seqid) + result.write(oprot) + oprot.writeMessageEnd() + oprot.trans.flush() + def process_pltfm_mgr_pwr_supply_present_get(self, seqid, iprot, oprot): args = pltfm_mgr_pwr_supply_present_get_args() args.read(iprot) @@ -2954,6 +3012,123 @@ def __ne__(self, other): ) +class pltfm_mgr_tlv_eeprom_get_args(object): + + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('pltfm_mgr_tlv_eeprom_get_args') + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) +all_structs.append(pltfm_mgr_tlv_eeprom_get_args) +pltfm_mgr_tlv_eeprom_get_args.thrift_spec = ( +) + + +class pltfm_mgr_tlv_eeprom_get_result(object): + """ + Attributes: + - success + - ouch + + """ + + + def __init__(self, success=None, ouch=None,): + self.success = success + self.ouch = ouch + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 0: + if ftype == TType.STRUCT: + self.success = pltfm_mgr_tlv_sys_eeprom_t() + self.success.read(iprot) + else: + iprot.skip(ftype) + elif fid == 1: + if ftype == TType.STRUCT: + self.ouch = InvalidPltfmMgrOperation.read(iprot) + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('pltfm_mgr_tlv_eeprom_get_result') + if self.success is not None: + oprot.writeFieldBegin('success', TType.STRUCT, 0) + self.success.write(oprot) + oprot.writeFieldEnd() + if self.ouch is not None: + oprot.writeFieldBegin('ouch', TType.STRUCT, 1) + self.ouch.write(oprot) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) +all_structs.append(pltfm_mgr_tlv_eeprom_get_result) +pltfm_mgr_tlv_eeprom_get_result.thrift_spec = ( + (0, TType.STRUCT, 'success', [pltfm_mgr_tlv_sys_eeprom_t, None], None, ), # 0 + (1, TType.STRUCT, 'ouch', [InvalidPltfmMgrOperation, None], None, ), # 1 +) + + class pltfm_mgr_pwr_supply_present_get_args(object): """ Attributes: diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py index 391d4bd5377b..ad686b888029 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/pltfm_mgr_rpc/ttypes.py @@ -460,6 +460,63 @@ def __ne__(self, other): return not (self == other) +class pltfm_mgr_tlv_sys_eeprom_t(object): + """ + Attributes: + - raw_content_hex + + """ + + + def __init__(self, raw_content_hex=None,): + self.raw_content_hex = raw_content_hex + + def read(self, iprot): + if iprot._fast_decode is not None and isinstance(iprot.trans, TTransport.CReadableTransport) and self.thrift_spec is not None: + iprot._fast_decode(self, iprot, [self.__class__, self.thrift_spec]) + return + iprot.readStructBegin() + while True: + (fname, ftype, fid) = iprot.readFieldBegin() + if ftype == TType.STOP: + break + if fid == 1: + if ftype == TType.STRING: + self.raw_content_hex = iprot.readString().decode('utf-8', errors='replace') if sys.version_info[0] == 2 else iprot.readString() + else: + iprot.skip(ftype) + else: + iprot.skip(ftype) + iprot.readFieldEnd() + iprot.readStructEnd() + + def write(self, oprot): + if oprot._fast_encode is not None and self.thrift_spec is not None: + oprot.trans.write(oprot._fast_encode(self, [self.__class__, self.thrift_spec])) + return + oprot.writeStructBegin('pltfm_mgr_tlv_sys_eeprom_t') + if self.raw_content_hex is not None: + oprot.writeFieldBegin('raw_content_hex', TType.STRING, 1) + oprot.writeString(self.raw_content_hex.encode('utf-8') if sys.version_info[0] == 2 else self.raw_content_hex) + oprot.writeFieldEnd() + oprot.writeFieldStop() + oprot.writeStructEnd() + + def validate(self): + return + + def __repr__(self): + L = ['%s=%r' % (key, value) + for key, value in self.__dict__.items()] + return '%s(%s)' % (self.__class__.__name__, ', '.join(L)) + + def __eq__(self, other): + return isinstance(other, self.__class__) and self.__dict__ == other.__dict__ + + def __ne__(self, other): + return not (self == other) + + class pltfm_mgr_pwr_supply_info_t(object): """ Attributes: @@ -1380,6 +1437,11 @@ def __ne__(self, other): (21, TType.STRING, 'location', 'UTF8', None, ), # 21 (22, TType.I16, 'crc8', None, None, ), # 22 ) +all_structs.append(pltfm_mgr_tlv_sys_eeprom_t) +pltfm_mgr_tlv_sys_eeprom_t.thrift_spec = ( + None, # 0 + (1, TType.STRING, 'raw_content_hex', 'UTF8', None, ), # 1 +) all_structs.append(pltfm_mgr_pwr_supply_info_t) pltfm_mgr_pwr_supply_info_t.thrift_spec = ( None, # 0 From 9bb0a7f33cd99f769063dd4bb798f0b59985e619 Mon Sep 17 00:00:00 2001 From: Andriy Kokhan Date: Fri, 30 Sep 2022 01:18:43 +0300 Subject: [PATCH 32/43] [BFN] Canceling PSU platform API calls on SIGTERM (#10720) * [BFN] Canceling PSU platform API calls on SIGTERM Signed-off-by: Andriy Kokhan * [BFN] Fixed SONiC fwutil exec time (#31) Signed-off-by: Taras Keryk Signed-off-by: Andriy Kokhan Signed-off-by: Taras Keryk Co-authored-by: Taras Keryk --- .../sonic_platform/component.py | 2 + .../sonic_platform/platform_utils.py | 60 ++++++++++++++++++- .../sonic_platform/psu.py | 30 +++++++++- 3 files changed, 89 insertions(+), 3 deletions(-) diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/component.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/component.py index 47a0993bf3e5..a7f236cb42a4 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/component.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/component.py @@ -6,6 +6,7 @@ import json from collections import OrderedDict from sonic_py_common import device_info + from platform_utils import limit_execution_time except ImportError as e: raise ImportError(str(e) + "- required module not found") @@ -24,6 +25,7 @@ def get_bios_version(): except subprocess.CalledProcessError as e: raise RuntimeError("Failed to get BIOS version") +@limit_execution_time(1) def get_bmc_version(): """ Retrieves the firmware version of the BMC diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_utils.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_utils.py index 81e78ee01041..2f7b5aecb6d0 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_utils.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/platform_utils.py @@ -3,11 +3,19 @@ try: import os import subprocess + import signal + from functools import wraps except ImportError as e: raise ImportError(str(e) + "- required module not found") def file_create(path, mode=None): + """ + Ensure that file is created with the appropriate permissions + Args: + path: full path of a file + mode: file permission in octal representation + """ def run_cmd(cmd): if os.geteuid() != 0: cmd.insert(0, 'sudo') @@ -18,5 +26,55 @@ def run_cmd(cmd): run_cmd(['mkdir', '-p', file_path]) if not os.path.isfile(path): run_cmd(['touch', path]) - if (mode is not None): + if (mode is not None): run_cmd(['chmod', mode, path]) + +def cancel_on_sigterm(func): + """ + Wrapper for a function which has to be cancel on SIGTERM + """ + @wraps(func) + def wrapper(*args, **kwargs): + def handler(sig, frame): + if sigterm_handler: + sigterm_handler(sig, frame) + raise Exception("Canceling {}() execution...".format(func.__name__)) + + sigterm_handler = signal.getsignal(signal.SIGTERM) + signal.signal(signal.SIGTERM, handler) + result = None + try: + result = func(*args, **kwargs) + finally: + signal.signal(signal.SIGTERM, sigterm_handler) + return result + return wrapper + +def limit_execution_time(execution_time_secs: int): + """ + Wrapper for a function whose execution time must be limited + Args: + execution_time_secs: maximum execution time in seconds, + after which the function execution will be stopped + """ + def wrapper(func): + @wraps(func) + def execution_func(*args, **kwargs): + def handler(sig, frame): + if sigalrm_handler: + sigalrm_handler(sig, frame) + raise Exception("Canceling {}() execution...".format(func.__name__)) + + sigalrm_handler = signal.getsignal(signal.SIGALRM) + signal.signal(signal.SIGALRM, handler) + signal.alarm(execution_time_secs) + result = None + try: + result = func(*args, **kwargs) + finally: + signal.alarm(0) + + return result + return execution_func + return wrapper + diff --git a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/psu.py b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/psu.py index fb9bce50e071..fbd83d6496ae 100644 --- a/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/psu.py +++ b/platform/barefoot/sonic-platform-modules-bfn-montara/sonic_platform/psu.py @@ -4,18 +4,26 @@ import os import sys import time + import signal + import syslog sys.path.append(os.path.dirname(__file__)) from .platform_thrift_client import thrift_try from sonic_platform_base.psu_base import PsuBase + from platform_utils import cancel_on_sigterm + except ImportError as e: raise ImportError (str(e) + "- required module not found") class Psu(PsuBase): """Platform-specific PSU class""" + sigterm = False + sigterm_default_handler = None + cls_inited = False + def __init__(self, index): PsuBase.__init__(self) self.__index = index @@ -24,6 +32,21 @@ def __init__(self, index): # STUB IMPLEMENTATION self.color = "" + syslog.syslog(syslog.LOG_INFO, "Created PSU #{} instance".format(self.__index)) + if not Psu.cls_inited: + Psu.sigterm_default_handler = signal.getsignal(signal.SIGTERM) + signal.signal(signal.SIGTERM, Psu.signal_handler) + if Psu.sigterm_default_handler: + syslog.syslog(syslog.LOG_INFO, "Default SIGTERM handler overridden!!") + Psu.cls_inited = True + + @classmethod + def signal_handler(cls, sig, frame): + if cls.sigterm_default_handler: + cls.sigterm_default_handler(sig, frame) + syslog.syslog(syslog.LOG_INFO, "Canceling PSU platform API calls...") + cls.sigterm = True + ''' Units of returned info object values: vin - V @@ -33,20 +56,23 @@ def __init__(self, index): fspeed - RPM ''' def __info_get(self): + @cancel_on_sigterm def psu_info_get(client): return client.pltfm_mgr.pltfm_mgr_pwr_supply_info_get(self.__index) # Update cache once per 2 seconds - if self.__ts + 2 < time.time(): + if self.__ts + 2 < time.time() and not Psu.sigterm: self.__info = None try: self.__info = thrift_try(psu_info_get, attempts=1) + except Exception as e: + if "Canceling" in str(e): + syslog.syslog(syslog.LOG_INFO, "{}".format(e)) finally: self.__ts = time.time() return self.__info return self.__info - @staticmethod def get_num_psus(): """ From 179882398c0c035f3bf2a522c07ad5c958453d80 Mon Sep 17 00:00:00 2001 From: Prince George <45705344+prgeor@users.noreply.github.com> Date: Thu, 29 Sep 2022 17:12:20 -0700 Subject: [PATCH 33/43] Revert "Support for serdes platform library debian installation for Innovium SONiC image (#11920)" (#12227) This reverts commit 8c7e0f8e02233dc4236f8f7f048b4281ab9d31b8. --- platform/innovium/invm-sai.mk | 4 +--- platform/innovium/one-image.mk | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/platform/innovium/invm-sai.mk b/platform/innovium/invm-sai.mk index 734184e35b0c..25e490e44f83 100755 --- a/platform/innovium/invm-sai.mk +++ b/platform/innovium/invm-sai.mk @@ -5,13 +5,11 @@ INVM_SAI_ONLINE = https://github.com/Innovium/SONiC/raw/master/debian/master INVM_LIBSAI = isai.deb INVM_HSAI = saihdr.deb INVM_DRV = ipd.deb -INVM_SERDES_PLATFORM_LIBRARY = ivm_serdes_pltfm.deb $(INVM_LIBSAI)_URL = $(INVM_SAI_ONLINE)/$(INVM_LIBSAI) $(INVM_HSAI)_URL = $(INVM_SAI_ONLINE)/$(INVM_HSAI) $(INVM_DRV)_URL = $(INVM_SAI_ONLINE)/$(INVM_DRV) -$(INVM_SERDES_PLATFORM_LIBRARY)_URL = $(INVM_SAI_ONLINE)/$(INVM_SERDES_PLATFORM_LIBRARY) $(eval $(call add_conflict_package,$(INVM_HSAI),$(LIBSAIVS_DEV))) -SONIC_ONLINE_DEBS += $(INVM_LIBSAI) $(INVM_HSAI) $(INVM_DRV) $(INVM_SERDES_PLATFORM_LIBRARY) +SONIC_ONLINE_DEBS += $(INVM_LIBSAI) $(INVM_HSAI) $(INVM_DRV) diff --git a/platform/innovium/one-image.mk b/platform/innovium/one-image.mk index 39e2e9fc8246..2cae779d71ac 100755 --- a/platform/innovium/one-image.mk +++ b/platform/innovium/one-image.mk @@ -8,6 +8,6 @@ $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(CEL_MIDSTONE_200I_PLATFORM_MODULE) $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(DELTA_PLATFORM_MODULE) $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(NETBERG_AURORA_715_PLATFORM_MODULE) $(SONIC_ONE_IMAGE)_LAZY_INSTALLS += $(WISTRON_PLATFORM_MODULE) -$(SONIC_ONE_IMAGE)_INSTALLS += $(INVM_DRV) $(INVM_SERDES_PLATFORM_LIBRARY) +$(SONIC_ONE_IMAGE)_INSTALLS += $(INVM_DRV) $(SONIC_ONE_IMAGE)_DOCKERS += $(SONIC_INSTALL_DOCKER_IMAGES) SONIC_INSTALLERS += $(SONIC_ONE_IMAGE) From 5510d9c03b6f2b1b07198f3c421ab4ce7bf5a0ef Mon Sep 17 00:00:00 2001 From: Ye Jianquan Date: Fri, 30 Sep 2022 08:17:01 +0800 Subject: [PATCH 34/43] Make t0 part1 and part2 be able to be rerun if failed (#12221) Why I did it With continueOnError: true, a failed job returns the result: partiallySuccess, which cause it can't be rerun, since AZP consider it as passed. Then we can't only rerun t0 jobs when it fails. How I did it Mark t0 part1 and part2 as continueOnError: false. How to verify it The pipeline will verify it. --- azure-pipelines.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 7856915f9b60..26a86dffa01e 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -137,7 +137,7 @@ stages: pool: sonictest displayName: "kvmtest-t0-part1" timeoutInMinutes: 360 - continueOnError: true + continueOnError: false steps: - template: .azure-pipelines/run-test-template.yml parameters: @@ -151,7 +151,7 @@ stages: pool: sonictest displayName: "kvmtest-t0-part2" timeoutInMinutes: 360 - continueOnError: true + continueOnError: false steps: - template: .azure-pipelines/run-test-template.yml parameters: From 1d69f0916eccc3961cdaa2c680141f57207396a1 Mon Sep 17 00:00:00 2001 From: Junchao-Mellanox <57339448+Junchao-Mellanox@users.noreply.github.com> Date: Fri, 30 Sep 2022 14:38:05 +0800 Subject: [PATCH 35/43] [Mellanox] Provide dummy implementation for get_rx_los and get_tx_fault (#12231) - Why I did it get_rx_los and get_tx_fault is not supported via the exisitng interface used, need provide dummy implementation for them. NOTE: in later releases we will get them back via different interface. - How I did it Return False * lane_num for get_rx_los and get_tx_fault - How to verify it Added unit test --- .../mlnx-platform-api/sonic_platform/sfp.py | 32 +++++++++++++++++++ .../mlnx-platform-api/tests/test_sfp.py | 14 ++++++++ 2 files changed, 46 insertions(+) diff --git a/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py b/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py index 617b4f33d636..d35b869e9a29 100644 --- a/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py +++ b/platform/mellanox/mlnx-platform-api/sonic_platform/sfp.py @@ -755,6 +755,38 @@ def get_error_description(self): error_description = "Unknow SFP module status ({})".format(oper_status) return error_description + def get_rx_los(self): + """Accessing rx los is not supproted, return all False + + Returns: + list: [False] * channels + """ + api = self.get_xcvr_api() + return [False] * api.NUM_CHANNELS if api else None + + def get_tx_fault(self): + """Accessing tx fault is not supproted, return all False + + Returns: + list: [False] * channels + """ + api = self.get_xcvr_api() + return [False] * api.NUM_CHANNELS if api else None + + def get_xcvr_api(self): + """ + Retrieves the XcvrApi associated with this SFP + + Returns: + An object derived from XcvrApi that corresponds to the SFP + """ + if self._xcvr_api is None: + self.refresh_xcvr_api() + if self._xcvr_api is not None: + self._xcvr_api.get_rx_los = self.get_rx_los + self._xcvr_api.get_tx_fault = self.get_tx_fault + return self._xcvr_api + class RJ45Port(NvidiaSFPCommon): """class derived from SFP, representing RJ45 ports""" diff --git a/platform/mellanox/mlnx-platform-api/tests/test_sfp.py b/platform/mellanox/mlnx-platform-api/tests/test_sfp.py index f599e0241d25..b72a5f3ed4aa 100644 --- a/platform/mellanox/mlnx-platform-api/tests/test_sfp.py +++ b/platform/mellanox/mlnx-platform-api/tests/test_sfp.py @@ -119,3 +119,17 @@ def test_is_port_admin_status_up(self, mock_port_status): mock_port_status.return_value = (0, False) assert not SFP.is_port_admin_status_up(None, None) + + @mock.patch('sonic_platform.sfp.SFP.get_xcvr_api') + def test_dummy_apis(self, mock_get_xcvr_api): + mock_api = mock.MagicMock() + mock_api.NUM_CHANNELS = 4 + mock_get_xcvr_api.return_value = mock_api + + sfp = SFP(0) + assert sfp.get_rx_los() == [False] * 4 + assert sfp.get_tx_fault() == [False] * 4 + + mock_get_xcvr_api.return_value = None + assert sfp.get_rx_los() is None + assert sfp.get_tx_fault() is None From 92bd6dae281977c93bbf39e4456a62857a485dfb Mon Sep 17 00:00:00 2001 From: Volodymyr Samotiy Date: Fri, 30 Sep 2022 09:40:12 +0300 Subject: [PATCH 36/43] [Mellanox] Update SAI to v2205.22.1.19 and SDK/FW to v4.5.3168/v2010.3170 (#12205) - Why I did it To include latest fixes and new functionality SAI fixes and new features fix #3205239, incorrect object type returned for SG child list Fix VRF-VNI map entries remove issue ECC health event and logging [Port Buffers] restore default queue and pg configuration when all user pools are deleted Fix EVPN type3 error on removal of uc/bc flood group Fix EVPN type2 MAC move from local to remote results in SAI failure Fix Disable learning on VXLAN tunnel Fix error on VXLAN v6 tunnel removal Fix port cannot apply schedule group when it is a lag member Fix BFD add more detailed message on BFD packet not related to any existing session gcc10 compilation fixes Disable learning on VXLAN tunnel Support BFD remote-disc exchange in negotiation stage Tunnel Loopback packet action attribute implementation (for Dual TOR) Add KVD resources MIN/MAX functionality (pending CRM issue with MIN only) Support for CRC2 hash algorithm Bulk counter support for PGs, queues Support mirror sample rate attribute (SPC2+) [Functional] [QoS] | Unable to remove SCHEDULE profile table even if there is no object referencing it Next hop group optimized bulk API Reduce verbosity of shared database already exists print Span mirror policer (SPC2+), optimize pipeline for acl mirror action with policer on SPC2+ use same size descriptor pool for rx/tx fix bfd - notify Sonic for admin-down event 2201 - empty list for supported fec for RJ45 ports Fix don't disable used tunnel underlay interfaces SDK fixes 100GbE FCI DAC (10137628-4050LF/HPE PN: 845408-B21) was recognized by mistake as supporting "cable burning' which caused the switch firmware to read page 0x9f (which unsupported in the cable) and to report this cable as having "bad eeprom". Added remote peer UDP port information in BFD packet event. After editing an ECMP, the resilient ECMP next-hop counter may not count correctly. Fixed potential memory leaks in some APIs related to LPM If TTL_CMD_COPY is used in Encap direction for a packet with no TTL, then the value passed in the ttl data structure will be used if non-zero (default 255 if zero). In SN2201: When configuring Force mode, user should configure Speed and FEC on both sides In Flex Tunnel encapsulation flow, if the encapsulation is with an IPv6 header, the flow label field may not be updated as expected. In some cases, when changing speed to 400GbE over 8 lanes, the first few packets would be dropped. In some traffic patterns involving small packets, the PortRcvErrors counter may mistakenly count events of local physical errors due to an internal flow in the hardware that involves link packets. On Spectrum systems, sometimes during link failure, not all previous firmware indications cleared properly, potentially affecting the next link up attempt. On the NVIDIA Spectrum-2 switch, when receiving a packet with Symbol Errors on ports that are configured to cut-thought mode, a pipeline might get stuck. PCI calibration changes from a static to a dynamic mechanism. SDK debug dump shows "Unknown" Counter in RFC3635 Counter Group. SDK debug dump shows "Unknown" Counter in the PPCNT Traffic Class Counter Group. SDK Dump missing column headers in some GC tables may result in difficulty understanding the dump. SLL configuration is missing in SDK dump. Spectrum-2 systems, do no support 1GbE on supported 40GbE modules. When binding a UDP port which is already in use for BFD TX session, the error message appears incorrectly. When Flex Tunnel was used, Flex Modifier sometimes experienced a brief mis-configuration during ISSU. When many ports are active (e.g. 70 ports up), and the configuration of shared buffer is applied on the fly, occasionally, the firmware might get stuck. When running 1GbE speeds on SN4600 system, the port remained active while peer side was closed. When toggling many ports of the Spectrum devices while raising 10GbE link up and link maintenance is enabled, the switch may get stuck and may need to be rebooted. When trying to reconfigure the Flex Parser header and Flex transition parameters after ISSU, the switch will returned an error even if the configuration was identical to that done before performing the ISSU. While toggling the cable, and the low power mode is set to ON, an unexpected PMPE event error is received. - How I did it Updated SDK/SAI submodule and relevant makefiles with the required versions. - How to verify it Build an image and run tests from "sonic-mgmt". Signed-off-by: Volodymyr Samotiy --- platform/mellanox/fw.mk | 6 +++--- platform/mellanox/mlnx-sai.mk | 2 +- platform/mellanox/mlnx-sai/SAI-Implementation | 2 +- platform/mellanox/sdk-src/sx-kernel/Switch-SDK-drivers | 2 +- platform/mellanox/sdk.mk | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/platform/mellanox/fw.mk b/platform/mellanox/fw.mk index 962ea0ae31e1..92aed0663311 100644 --- a/platform/mellanox/fw.mk +++ b/platform/mellanox/fw.mk @@ -27,17 +27,17 @@ else FW_FROM_URL = n endif -MLNX_SPC_FW_VERSION = 13.2010.2320 +MLNX_SPC_FW_VERSION = 13.2010.3170 MLNX_SPC_FW_FILE = fw-SPC-rel-$(subst .,_,$(MLNX_SPC_FW_VERSION))-EVB.mfa $(MLNX_SPC_FW_FILE)_PATH = $(MLNX_FW_BASE_PATH) $(MLNX_SPC_FW_FILE)_URL = $(MLNX_FW_BASE_URL)/$(MLNX_SPC_FW_FILE) -MLNX_SPC2_FW_VERSION = 29.2010.2320 +MLNX_SPC2_FW_VERSION = 29.2010.3170 MLNX_SPC2_FW_FILE = fw-SPC2-rel-$(subst .,_,$(MLNX_SPC2_FW_VERSION))-EVB.mfa $(MLNX_SPC2_FW_FILE)_PATH = $(MLNX_FW_BASE_PATH) $(MLNX_SPC2_FW_FILE)_URL = $(MLNX_FW_BASE_URL)/$(MLNX_SPC2_FW_FILE) -MLNX_SPC3_FW_VERSION = 30.2010.2320 +MLNX_SPC3_FW_VERSION = 30.2010.3170 MLNX_SPC3_FW_FILE = fw-SPC3-rel-$(subst .,_,$(MLNX_SPC3_FW_VERSION))-EVB.mfa $(MLNX_SPC3_FW_FILE)_PATH = $(MLNX_FW_BASE_PATH) $(MLNX_SPC3_FW_FILE)_URL = $(MLNX_FW_BASE_URL)/$(MLNX_SPC3_FW_FILE) diff --git a/platform/mellanox/mlnx-sai.mk b/platform/mellanox/mlnx-sai.mk index 6eaa1dcf80b4..90ca7430b0ad 100644 --- a/platform/mellanox/mlnx-sai.mk +++ b/platform/mellanox/mlnx-sai.mk @@ -1,6 +1,6 @@ # Mellanox SAI -MLNX_SAI_VERSION = SAIRel1.21.2.0 +MLNX_SAI_VERSION = SAIBuild2205.22.1.19 export MLNX_SAI_VERSION diff --git a/platform/mellanox/mlnx-sai/SAI-Implementation b/platform/mellanox/mlnx-sai/SAI-Implementation index f9a21df71363..82274ffaef77 160000 --- a/platform/mellanox/mlnx-sai/SAI-Implementation +++ b/platform/mellanox/mlnx-sai/SAI-Implementation @@ -1 +1 @@ -Subproject commit f9a21df713636fe648b8bb190698e4494a0f5239 +Subproject commit 82274ffaef7748120b7657362f7875fb7d6e6f5f diff --git a/platform/mellanox/sdk-src/sx-kernel/Switch-SDK-drivers b/platform/mellanox/sdk-src/sx-kernel/Switch-SDK-drivers index 5650c3519b55..8b1f1c0f1164 160000 --- a/platform/mellanox/sdk-src/sx-kernel/Switch-SDK-drivers +++ b/platform/mellanox/sdk-src/sx-kernel/Switch-SDK-drivers @@ -1 +1 @@ -Subproject commit 5650c3519b55051124810a4625f8269694b1e592 +Subproject commit 8b1f1c0f11647749f79ebc4e823c157513067412 diff --git a/platform/mellanox/sdk.mk b/platform/mellanox/sdk.mk index b620b07ee2f8..5a6864bc1e4a 100644 --- a/platform/mellanox/sdk.mk +++ b/platform/mellanox/sdk.mk @@ -16,7 +16,7 @@ # MLNX_SDK_BASE_PATH = $(PLATFORM_PATH)/sdk-src/sx-kernel/Switch-SDK-drivers/bin/ MLNX_SDK_PKG_BASE_PATH = $(MLNX_SDK_BASE_PATH)/$(BLDENV)/$(CONFIGURED_ARCH)/ -MLNX_SDK_VERSION = 4.5.2320 +MLNX_SDK_VERSION = 4.5.3168 MLNX_SDK_ISSU_VERSION = 101 MLNX_SDK_DEB_VERSION = $(subst -,.,$(subst _,.,$(MLNX_SDK_VERSION))) From eea8ebd0a9ee977765097666d1d89961a466d891 Mon Sep 17 00:00:00 2001 From: Volodymyr Samotiy Date: Fri, 30 Sep 2022 09:48:40 +0300 Subject: [PATCH 37/43] [Mellanox] Update MFT to v4.21.0-100 (#11758) - Why I did it To update MFT package to the latest version. - How I did it Updated MFT_VERSION & MFT_REVISION in platform/mellanox/mft.mk. - How to verify it Build an image and deploy to the switch Check MFT version by dpkg -l | grep mft Verify that all the SONiC services up and running Run regression testing using tests from sonic-mgmt Signed-off-by: Volodymyr Samotiy --- platform/mellanox/mft.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/platform/mellanox/mft.mk b/platform/mellanox/mft.mk index eb702a85a1d1..047e87b09086 100644 --- a/platform/mellanox/mft.mk +++ b/platform/mellanox/mft.mk @@ -16,8 +16,8 @@ # # Mellanox SAI -MFT_VERSION = 4.20.0 -MFT_REVISION = 34 +MFT_VERSION = 4.21.0 +MFT_REVISION = 100 export MFT_VERSION MFT_REVISION From 004a8b6eae203cb1496beb8b192b087cc2a32a66 Mon Sep 17 00:00:00 2001 From: Hua Liu <58683130+liuh-80@users.noreply.github.com> Date: Fri, 30 Sep 2022 15:56:46 +0800 Subject: [PATCH 38/43] [AzurePipeline] Fix vstest step failed by libyang missing. (#12240) Why I did it Fix PR merge failed because 'vstest' step does not install libyang. How I did it Install libyang in azure pipeline. How to verify it Pass vstest step. --- azure-pipelines.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index 26a86dffa01e..a9004af559ef 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -113,6 +113,8 @@ stages: - script: | set -x + sudo apt-get update + sudo apt-get install libyang0.16 -y sudo dpkg -i --force-confask,confnew ../libswsscommon_1.0.0_amd64.deb sudo dpkg -i ../python3-swsscommon_1.0.0_amd64.deb sudo docker load -i ../target/docker-sonic-vs.gz From 18850e4e28bb5d536abc9c455138c57aad69811d Mon Sep 17 00:00:00 2001 From: Samuel Angebault Date: Fri, 30 Sep 2022 10:03:40 +0200 Subject: [PATCH 39/43] [Arista] Update platform submodules (#12225) Implement input power psu API Report DC power output via API Add bootloader Component in API Fix issue where naming was not unique for Component --- platform/barefoot/sonic-platform-modules-arista | 2 +- platform/broadcom/sonic-platform-modules-arista | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/platform/barefoot/sonic-platform-modules-arista b/platform/barefoot/sonic-platform-modules-arista index e12a04b24c5f..11180c37fa17 160000 --- a/platform/barefoot/sonic-platform-modules-arista +++ b/platform/barefoot/sonic-platform-modules-arista @@ -1 +1 @@ -Subproject commit e12a04b24c5f752a9ca789d62bb7b94c563e1c4b +Subproject commit 11180c37fa17421afdeef346b3896552872a2721 diff --git a/platform/broadcom/sonic-platform-modules-arista b/platform/broadcom/sonic-platform-modules-arista index e12a04b24c5f..11180c37fa17 160000 --- a/platform/broadcom/sonic-platform-modules-arista +++ b/platform/broadcom/sonic-platform-modules-arista @@ -1 +1 @@ -Subproject commit e12a04b24c5f752a9ca789d62bb7b94c563e1c4b +Subproject commit 11180c37fa17421afdeef346b3896552872a2721 From 0a2743d5e472d25209a846e49f032998ee093261 Mon Sep 17 00:00:00 2001 From: jingwenxie Date: Sat, 1 Oct 2022 11:36:55 -0700 Subject: [PATCH 40/43] [submodule] update sonic-utilities (#12138) 0a7557bd9 [minigraph] add option to specify golden path in load_minigraph (#2350) 322aefc37 [GCU]Remove GCU unique lane check for duplicate lanes platforms (#2343) 7099fffa7 [fastboot] fastboot enhancement: Use warm-boot infrastructure for fast-boot (#2286) 09026edbb [warm-reboot] fix warm-reboot when /tmp/cache is missing (#2367) a3c404c74 Fix typo in platform_sfputil_helper.is_rj45_port (#2374) 637d834ce Vnet_route_check Vxlan tunnel route update. (#2281) 29a3e5180 Added support for tunnel route status in show vnet routes all. (#2341) 1ac584bb3 Use 'default' VRF when VRF name is not provided (#2368) 4d377a620 [subinterface]Added additional checks in portchannel and subinterface commands (#2345) bbcdf2ed7 disk_check: Publish event for RO state (#2320) 3fd537b0a Support the bandit check by GitHub Action (#2358) 491d3d380 [generate dump]Added error message when saisdkdump fails (#2356) 6830e01ec [counterpoll]Fixing counterpoll show for tunnel and acl stats (#2355) 3be2ad7de [fast-reboot]Avoid stopping masked services during fast-reboot (#2335) 0e1b0cf20 [GCU] Fix missing backend in dry run (#2347) 676c31bd0 Add verification for override (#2305) 48997c266 Add Password Hardening CLI support (#2338) 414e239ea update unit tests for swap allocator a91a4922f consider swap checking memory in installer f0ce58635 [route_check]: Ignore standalone tunnel routes (#2325) --- src/sonic-utilities | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sonic-utilities b/src/sonic-utilities index 3af8ba4acc2b..0a7557bd9162 160000 --- a/src/sonic-utilities +++ b/src/sonic-utilities @@ -1 +1 @@ -Subproject commit 3af8ba4acc2bbc77d17be0d67943703021c7d1e1 +Subproject commit 0a7557bd9162eae40f5d4c4f6fbab92dbad7204b From 8c10851c2aebb44099611248f91da350b16df1dc Mon Sep 17 00:00:00 2001 From: Muhammad Danish <88161975+mdanish-kh@users.noreply.github.com> Date: Sun, 2 Oct 2022 11:02:10 +0500 Subject: [PATCH 41/43] Update azure.github.io links to sonic-net.github.io (#12209) Why I did it azure.github.io/SONiC/ no longer works and returns 404 Not Found. Updated it to the correct sonic-net.github.io/SONiC/ --- files/image_config/environment/motd | 2 +- platform/vs/sonic-gns3a.sh | 4 ++-- src/sonic-device-data/README.md | 2 +- src/sonic-host-services-data/README.md | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/files/image_config/environment/motd b/files/image_config/environment/motd index 8562e330fe2c..0d857e5c5f94 100644 --- a/files/image_config/environment/motd +++ b/files/image_config/environment/motd @@ -10,5 +10,5 @@ You are on Unauthorized access and/or use are prohibited. All access and/or use are subject to monitoring. -Help: http://azure.github.io/SONiC/ +Help: https://sonic-net.github.io/SONiC/ diff --git a/platform/vs/sonic-gns3a.sh b/platform/vs/sonic-gns3a.sh index 41e39cd8686a..2a772ce5a332 100644 --- a/platform/vs/sonic-gns3a.sh +++ b/platform/vs/sonic-gns3a.sh @@ -41,9 +41,9 @@ echo " \"category\": \"router\", \"description\": \"SONiC Virtual Switch/Router\", \"vendor_name\": \"SONiC\", - \"vendor_url\": \"https://azure.github.io/SONiC/\", + \"vendor_url\": \"https://sonic-net.github.io/SONiC/\", \"product_name\": \"SONiC\", - \"product_url\": \"https://azure.github.io/SONiC/\", + \"product_url\": \"https://sonic-net.github.io/SONiC/\", \"registry_version\": 3, \"status\": \"experimental\", \"maintainer\": \"SONiC\", diff --git a/src/sonic-device-data/README.md b/src/sonic-device-data/README.md index e8ccad58b819..d9d403758a86 100644 --- a/src/sonic-device-data/README.md +++ b/src/sonic-device-data/README.md @@ -1,4 +1,4 @@ # sonic-device-data Device-specific data for the SONiC project -See the [SONiC Website](http://azure.github.io/SONiC/) for more information about the SONiC project. +See the [SONiC Website](https://sonic-net.github.io/SONiC/) for more information about the SONiC project. diff --git a/src/sonic-host-services-data/README.md b/src/sonic-host-services-data/README.md index 93af66a83d6b..0b9e714932d2 100644 --- a/src/sonic-host-services-data/README.md +++ b/src/sonic-host-services-data/README.md @@ -16,4 +16,4 @@ dpkg-buildpackage -rfakeroot -Tclean --- -See the [SONiC Website](http://azure.github.io/SONiC/) for more information about the SONiC project. +See the [SONiC Website](https://sonic-net.github.io/SONiC/) for more information about the SONiC project. From 44356fa8d758fffd6e023ebfb7e1c8646a81ede1 Mon Sep 17 00:00:00 2001 From: Dror Prital <76714716+dprital@users.noreply.github.com> Date: Sun, 2 Oct 2022 11:34:24 +0300 Subject: [PATCH 42/43] [Mellanox] Add NVIDIA copyright header for NVIDIA added files (#12130) - Why I did it Add NVIDIA Copyright header for new "NVIDIA" files - How I did it Add the copyright header as remark at the head of the file --- .../ACS-MSN2700/buffers_defaults_objects.j2 | 17 +++++++++++++++++ .../buffers_defaults_objects.j2 | 17 +++++++++++++++++ .../Mellanox-SN4700-C128/port_config.ini | 17 +++++++++++++++++ .../Mellanox-SN4700-C128/sai_4700_128x100g.xml | 17 +++++++++++++++++ .../tests/input_platform/__init__.py | 16 ++++++++++++++++ .../tests/input_platform/output_sfp.py | 17 +++++++++++++++++ platform/mellanox/zero_profiles.j2 | 17 +++++++++++++++++ 7 files changed, 118 insertions(+) diff --git a/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/buffers_defaults_objects.j2 b/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/buffers_defaults_objects.j2 index 29a3c74e5233..e8edeca556d1 100644 --- a/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/buffers_defaults_objects.j2 +++ b/device/mellanox/x86_64-mlnx_msn2700-r0/ACS-MSN2700/buffers_defaults_objects.j2 @@ -1,3 +1,20 @@ +{# + Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. + Apache-2.0 + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +#} + {%- macro generate_buffer_pool_and_profiles_with_inactive_ports(port_names_inactive) %} "BUFFER_POOL": { {% if dynamic_mode is not defined and port_names_inactive|length > 0 -%} diff --git a/device/mellanox/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/buffers_defaults_objects.j2 b/device/mellanox/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/buffers_defaults_objects.j2 index f0b0e3993bd4..6bf657d1fb7b 100644 --- a/device/mellanox/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/buffers_defaults_objects.j2 +++ b/device/mellanox/x86_64-mlnx_msn2700-r0/Mellanox-SN2700-D48C8/buffers_defaults_objects.j2 @@ -1,3 +1,20 @@ +{# + Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. + Apache-2.0 + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +#} + {%- macro generate_buffer_pool_and_profiles_with_inactive_ports(port_names_inactive) %} "BUFFER_POOL": { {% if dynamic_mode is not defined and port_names_inactive|length > 0 -%} diff --git a/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/port_config.ini b/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/port_config.ini index d64b66b0b691..0d67f9b366fc 100644 --- a/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/port_config.ini +++ b/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/port_config.ini @@ -1,3 +1,20 @@ +## +## Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. +## Apache-2.0 +## +## Licensed under the Apache License, Version 2.0 (the "License"); +## you may not use this file except in compliance with the License. +## You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## + # name lanes alias index speed Ethernet0 0,1 etp1a 1 100000 Ethernet2 2,3 etp1b 1 100000 diff --git a/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/sai_4700_128x100g.xml b/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/sai_4700_128x100g.xml index f5d49f8b86ab..2575b49f3fa0 100644 --- a/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/sai_4700_128x100g.xml +++ b/device/mellanox/x86_64-mlnx_msn4700-r0/Mellanox-SN4700-C128/sai_4700_128x100g.xml @@ -1,3 +1,20 @@ + + diff --git a/platform/mellanox/mlnx-platform-api/tests/input_platform/__init__.py b/platform/mellanox/mlnx-platform-api/tests/input_platform/__init__.py index e69de29bb2d1..07ebf17a113e 100644 --- a/platform/mellanox/mlnx-platform-api/tests/input_platform/__init__.py +++ b/platform/mellanox/mlnx-platform-api/tests/input_platform/__init__.py @@ -0,0 +1,16 @@ +# +# Copyright (c) 2017-2022 NVIDIA CORPORATION & AFFILIATES. +# Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# diff --git a/platform/mellanox/mlnx-platform-api/tests/input_platform/output_sfp.py b/platform/mellanox/mlnx-platform-api/tests/input_platform/output_sfp.py index 20a09d1b54f6..170b0246430f 100644 --- a/platform/mellanox/mlnx-platform-api/tests/input_platform/output_sfp.py +++ b/platform/mellanox/mlnx-platform-api/tests/input_platform/output_sfp.py @@ -1,3 +1,20 @@ +# +# Copyright (c) 2022 NVIDIA CORPORATION & AFFILIATES. +# Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + """ module holding the correct values for the sfp_test.py """ diff --git a/platform/mellanox/zero_profiles.j2 b/platform/mellanox/zero_profiles.j2 index a953c18409b2..007f19c83a0a 100644 --- a/platform/mellanox/zero_profiles.j2 +++ b/platform/mellanox/zero_profiles.j2 @@ -1,3 +1,20 @@ +{# + Copyright (c) 2021-2022 NVIDIA CORPORATION & AFFILIATES. + Apache-2.0 + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +#} + [ { "BUFFER_POOL_TABLE:ingress_zero_pool": { From 2f46689a059d3d7146b8ab27a0293e4b42383ec7 Mon Sep 17 00:00:00 2001 From: andywongarista <78833093+andywongarista@users.noreply.github.com> Date: Sun, 2 Oct 2022 22:53:34 -0700 Subject: [PATCH 43/43] [Arista] Add components for 720DT-48S (#12217) Why I did it Add components data for sonic-mgmt testing How I did it Update platform.json and add platform_components.json How to verify it Ran sonic-mgmt tests (test_chassis and test_component) --- device/arista/x86_64-arista_720dt_48s/platform.json | 9 ++++++++- .../x86_64-arista_720dt_48s/platform_components.json | 10 ++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) create mode 100644 device/arista/x86_64-arista_720dt_48s/platform_components.json diff --git a/device/arista/x86_64-arista_720dt_48s/platform.json b/device/arista/x86_64-arista_720dt_48s/platform.json index ce405b82f955..c560f8d739f1 100644 --- a/device/arista/x86_64-arista_720dt_48s/platform.json +++ b/device/arista/x86_64-arista_720dt_48s/platform.json @@ -1,7 +1,14 @@ { "chassis": { "name": "CCS-720DT-48S", - "components": [], + "components": [ + { + "name": "Aboot()" + }, + { + "name": "Scd(addr=0000:00:18.7)" + } + ], "fan_drawers": [ { "name": "fixed1", diff --git a/device/arista/x86_64-arista_720dt_48s/platform_components.json b/device/arista/x86_64-arista_720dt_48s/platform_components.json new file mode 100644 index 000000000000..ea8bbb5e3346 --- /dev/null +++ b/device/arista/x86_64-arista_720dt_48s/platform_components.json @@ -0,0 +1,10 @@ +{ + "chassis": { + "CCS-720DT-48S": { + "component": { + "Aboot()": {}, + "Scd(addr=0000:00:18.7)": {} + } + } + } +}