[Mellanox] Facilitate automatic integration of new hw-mgmt (#14594)

- Why I did it
Facilitate Automatic integration of new hw-mgmt version into SONiC.

Inputs to the Script:

MLNX_HW_MANAGEMENT_VERSION Eg: 7.0040.5202
CREATE_BRANCH: (y|n) Creates a branch instead of a commit (optional, default: n)
BRANCH_SONIC: Only relevant when CREATE_BRANCH is y. Default: master.
Note: These should be provided through SONIC_OVERRIDE_BUILD_VARS  parameter

Output:

Script creates a commit (in each of sonic-buildimage, sonic-linux-kernel) with all the changes required for upgrading the hw-management version to a version provided by MLNX_HW_MANAGEMENT_VERSION
Brief Summary of the changes made:

MLNX_HW_MANAGEMENT_VERSION flag in the hw-management.mk file
hw-mgmt submodule is updated to the corresponding version
Updates are made to non-upstream-patches/patches and series.patch file
series, kconfig-inclusion and kconfig-exclusion files can be updated in the sonic-linux-kernel repo
sonic-linux-kernel/patches folder is updated with the corresponding upstream patches
Based on the inputs, there could be a branch seen in the local for each of the repo's. Branch is named as <branch>_<parent_commit>_integrate_<hw_mgmt_version>

- How I did it
Added a new make target which can be invoked by calling make integrate-mlnx-hw-mgmt
user@server:/sonic-buildimage$ git rev-parse --abbrev-ref HEAD
master_23193446a_integrate_7.0020.5052
user@server:/sonic-buildimage$ git log --oneline -n 2
f66e01867 (HEAD -> master_23193446a_integrate_V.7.0020.5052, show) Intgerate HW-MGMT V.7.0020.5052 Changes
2319344 (master_intg_hw_mgmt) Update logic

user@server:/sonic-buildimage/src/sonic-linux-kernel$ git rev-parse --abbrev-ref HEAD
master_6847319_integrate_7.0020.4104
user@server:/sonic-buildimage/src/sonic-linux-kernel$ git log --oneline -n 2
6094f71 (HEAD -> master_6847319_integrate_V.7.0020.5052) Intgerate HW-MGMT V.7.0020.5052 Changes
6847319 (origin/master, origin/HEAD) Read ID register for optoe1 to find pageable bit in optoe driver  (#308)
Changes made will be summarized under sonic-buildimage/integrate-mlnx-hw-mgmt_user.out file. Debugging and troubleshooting output is written to sonic-buildimage/integrate-mlnx-hw-mgmt.log files

User output file & stdout file:

log_files.tar.gz

Limitations:
Assumes the changes would only work for amd64
Assumes the non-upstream patches in mellanox only belong to hw-mgmt

- How to verify it
Build the Kernel

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

platform/mellanox/hw-management/hwmgmt_nonup_patches

@@ -0,0 +1,63 @@
+# Current non-upstream patch list, should be updated by hwmgmt_kernel_patches.py script
+0099-mlxsw-core_hwmon-Fix-variable-names-for-hwmon-attrib.patch
+0100-mlxsw-core_thermal-Rename-labels-according-to-naming.patch
+0101-mlxsw-core_thermal-Remove-obsolete-API-for-query-res.patch
+0102-mlxsw-reg-Add-mgpir_-prefix-to-MGPIR-fields-comments.patch
+0103-mlxsw-core-Remove-unnecessary-asserts.patch
+0104-mlxsw-reg-Extend-MTMP-register-with-new-slot-number-.patch
+0105-mlxsw-reg-Extend-MTBR-register-with-new-slot-number-.patch
+0106-mlxsw-reg-Extend-MCIA-register-with-new-slot-number-.patch
+0107-mlxsw-reg-Extend-MCION-register-with-new-slot-number.patch
+0108-mlxsw-reg-Extend-PMMP-register-with-new-slot-number-.patch
+0109-mlxsw-reg-Extend-MGPIR-register-with-new-slot-fields.patch
+0110-mlxsw-core_env-Pass-slot-index-during-PMAOS-register.patch
+0111-mlxsw-reg-Add-new-field-to-Management-General-Periph.patch
+0112-mlxsw-core-Extend-interfaces-for-cable-info-access-w.patch
+0113-mlxsw-core-Extend-port-module-data-structures-for-li.patch
+0114-mlxsw-core-Move-port-module-events-enablement-to-a-s.patch
+0115-mlxsw-core_hwmon-Split-gearbox-initialization.patch
+0116-mlxsw-core_hwmon-Extend-internal-structures-to-suppo.patch
+0117-mlxsw-core_hwmon-Introduce-slot-parameter-in-hwmon-i.patch
+0118-mlxsw-core_hwmon-Extend-hwmon-device-with-gearbox-ma.patch
+0119-mlxsw-core_thermal-Extend-internal-structures-to-sup.patch
+0120-mlxsw-core_thermal-Split-gearbox-initialization.patch
+0121-mlxsw-core_thermal-Extend-thermal-area-with-gearbox-.patch
+0122-mlxsw-core_thermal-Add-line-card-id-prefix-to-line-c.patch
+0123-mlxsw-core_thermal-Use-exact-name-of-cooling-devices.patch
+0124-mlxsw-core_thermal-Use-common-define-for-thermal-zon.patch
+0125-devlink-add-support-to-create-line-card-and-expose-t.patch
+0126-devlink-implement-line-card-provisioning.patch
+0127-devlink-implement-line-card-active-state.patch
+0128-devlink-add-port-to-line-card-relationship-set.patch
+0129-devlink-introduce-linecard-info-get-message.patch
+0130-devlink-introduce-linecard-info-get-message.patch
+0131-mlxsw-reg-Add-Ports-Mapping-event-Configuration-Regi.patch
+0132-mlxsw-reg-Add-Management-DownStream-Device-Query-Reg.patch
+0133-mlxsw-reg-Add-Management-DownStream-Device-Control-R.patch
+0134-mlxsw-reg-Add-Management-Binary-Code-Transfer-Regist.patch
+0135-mlxsw-core_linecards-Add-line-card-objects-and-imple.patch
+0136-mlxsw-core_linecards-Implement-line-card-activation-.patch
+0137-mlxsw-core-Extend-driver-ops-by-remove-selected-port.patch
+0138-mlxsw-spectrum-Add-port-to-linecard-mapping.patch
+0139-mlxsw-reg-Introduce-Management-Temperature-Extended-.patch
+0140-mlxsw-core-Add-APIs-for-thermal-sensor-mapping.patch
+0141-mlxsw-reg-Add-Management-DownStream-Device-Tunneling.patch
+0142-mlxsw-core_linecards-Probe-devices-for-provisioned-l.patch
+0143-mlxsw-core_linecards-Expose-device-FW-version-over-d.patch
+0144-mlxsw-core-Introduce-flash-update-components.patch
+0145-mlxfw-Get-the-PSID-value-using-op-instead-of-passing.patch
+0146-mlxsw-core_linecards-Implement-line-card-device-flas.patch
+0147-mlxsw-core_linecards-Introduce-ops-for-linecards-sta.patch
+0148-mlxsw-core-Add-interfaces-for-line-card-initializati.patch
+0149-mlxsw-core_thermal-Add-interfaces-for-line-card-init.patch
+0150-mlxsw-core_hwmon-Add-interfaces-for-line-card-initia.patch
+0151-mlxsw-minimal-Prepare-driver-for-modular-system-supp.patch
+0152-mlxsw-core-Extend-bus-init-function-with-event-handl.patch
+0153-mlxsw-i2c-Add-support-for-system-events-handling.patch
+0154-mlxsw-core-Export-line-card-API.patch
+0155-mlxsw-minimal-Add-system-event-handler.patch
+0156-mlxsw-minimal-Add-interfaces-for-line-card-initializ.patch
+0163-platform-mellanox-Introduce-support-for-rack-manager.patch
+0176-platform-mellanox-fix-reset_pwr_converter_fail-attri.patch
+0177-Documentation-ABI-fix-description-of-fix-reset_pwr_c.patch
+0178-platform-mellanox-Introduce-support-for-next-generat.patch

platform/mellanox/integration-scripts.mk

@@ -0,0 +1,135 @@
+#
+# Copyright (c) 2016-2022 NVIDIA CORPORATION & AFFILIATES.
+# Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Mellanox Integration Scripts
+
+# override this for other branches
+BRANCH_SONIC = master
+# set this flag to y to create a branch instead of commit
+CREATE_BRANCH = n
+
+TEMP_HW_MGMT_DIR = /tmp/hw_mgmt
+PTCH_DIR = $(TEMP_HW_MGMT_DIR)/patch_dir/
+NON_UP_PTCH_DIR = $(TEMP_HW_MGMT_DIR)/non_up_patch_dir/
+PTCH_LIST  = $(TEMP_HW_MGMT_DIR)/series
+KCFG_LIST = $(TEMP_HW_MGMT_DIR)/kconfig
+HWMGMT_NONUP_LIST = $(BUILD_WORKDIR)/$($(MLNX_HW_MANAGEMENT)_SRC_PATH)/hwmgmt_nonup_patches
+HWMGMT_USER_OUTFILE = $(BUILD_WORKDIR)/integrate-mlnx-hw-mgmt_user.out
+TMPFILE_OUT := $(shell mktemp)
+SB_HEAD = $(shell git rev-parse --short HEAD)
+SLK_HEAD = $(shell cd src/sonic-linux-kernel; git rev-parse --short HEAD)
+
+integrate-mlnx-hw-mgmt:
+	$(FLUSH_LOG)
+	rm -rf $(TEMP_HW_MGMT_DIR) $(TMPFILE_OUT)
+	mkdir -p $(PTCH_DIR) $(NON_UP_PTCH_DIR)
+	touch $(PTCH_LIST) $(KCFG_LIST)
+
+	# clean up existing untracked files
+	pushd $(BUILD_WORKDIR); git clean -f -- platform/mellanox/
+ifeq ($(CREATE_BRANCH), y)
+	git checkout -B "$(BRANCH_SONIC)_$(SB_HEAD)_integrate_$(MLNX_HW_MANAGEMENT_VERSION)" HEAD
+	echo $(BRANCH_SONIC)_$(SB_HEAD)_integrate_$(MLNX_HW_MANAGEMENT_VERSION) branch created in sonic-buildimage
+endif
+	popd
+
+	pushd $(BUILD_WORKDIR)/src/sonic-linux-kernel; git clean -f -- patch/
+ifeq ($(CREATE_BRANCH), y)
+	git checkout -B "$(BRANCH_SONIC)_$(SLK_HEAD)_integrate_$(MLNX_HW_MANAGEMENT_VERSION)" HEAD
+	echo $(BRANCH_SONIC)_$(SLK_HEAD)_integrate_$(MLNX_HW_MANAGEMENT_VERSION) branch created in sonic-linux-kernel
+endif
+	popd
+
+	echo "#### Integrate HW-MGMT $(MLNX_HW_MANAGEMENT_VERSION) Kernel Patches into SONiC" > ${HWMGMT_USER_OUTFILE}
+	pushd $(BUILD_WORKDIR)/$(PLATFORM_PATH) $(LOG_SIMPLE)
+
+	# Run tests
+	pushd integration-scripts/tests; pytest-3 -v; popd
+
+	# Checkout to the corresponding hw-mgmt version and update mk file
+	pushd hw-management/hw-mgmt; git checkout V.${MLNX_HW_MANAGEMENT_VERSION}; popd
+	sed -i "s/\(^MLNX_HW_MANAGEMENT_VERSION = \).*/\1${MLNX_HW_MANAGEMENT_VERSION}/g" hw-management.mk
+
+	# Pre-processing before runing hw_mgmt script
+	integration-scripts/hwmgmt_kernel_patches.py pre \
+							--config_inclusion $(KCFG_LIST) \
+							--build_root $(BUILD_WORKDIR) $(LOG_SIMPLE)
+
+	$(BUILD_WORKDIR)/$($(MLNX_HW_MANAGEMENT)_SRC_PATH)/hw-mgmt/recipes-kernel/linux/deploy_kernel_patches.py \
+							--dst_accepted_folder $(PTCH_DIR) \
+							--dst_candidate_folder $(NON_UP_PTCH_DIR) \
+							--series_file $(PTCH_LIST) \
+							--config_file $(KCFG_LIST) \
+							--kernel_version $(KERNEL_VERSION) \
+							--os_type sonic $(LOG_SIMPLE)
+
+	# Post-processing
+	integration-scripts/hwmgmt_kernel_patches.py post \
+							--patches $(PTCH_DIR) \
+							--non_up_patches $(NON_UP_PTCH_DIR) \
+							--config_inclusion $(KCFG_LIST) \
+							--series $(PTCH_LIST) \
+							--current_non_up_patches $(HWMGMT_NONUP_LIST) \
+							--build_root $(BUILD_WORKDIR) $(LOG_SIMPLE)
+
+	# Commit the changes in linux kernel and and log the diff
+	pushd $(BUILD_WORKDIR)/src/sonic-linux-kernel
+	git add -- patch/
+
+	echo -en "\n###-> series file changes in sonic-linux-kernel <-###\n" >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- patch/series >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en "\n###-> kconfig-inclusions file changes in sonic-linux-kernel <-###\n" >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- patch/kconfig-inclusions >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en "\n###-> kconfig-exclusions file changes in sonic-linux-kernel <-###\n" >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- patch/kconfig-exclusions >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en '\n###-> Summary of files updated in sonic-linux-kernel <-###\n' >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged --stat --output=${TMPFILE_OUT}
+	cat ${TMPFILE_OUT} | tee -a ${HWMGMT_USER_OUTFILE}
+
+	git diff --staged --quiet || git commit -m "Intgerate HW-MGMT ${MLNX_HW_MANAGEMENT_VERSION} Changes";
+	popd
+
+	# Commit the changes in buildimage and log the diff
+	pushd $(BUILD_WORKDIR)
+	git add -- $($(MLNX_HW_MANAGEMENT)_SRC_PATH)
+	git add -- $(PLATFORM_PATH)/non-upstream-patches/
+	git add -- $(PLATFORM_PATH)/hw-management.mk
+
+	echo -en '\n###-> Non Upstream series.patch changes <-###\n' >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- $(PLATFORM_PATH)/non-upstream-patches/series.patch >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en '\n###-> Non Upstream patch list file <-###\n' >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- $($(MLNX_HW_MANAGEMENT)_SRC_PATH)/hwmgmt_nonup_patches >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en '\n###-> hw-mgmt submodule update <-###\n' >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- $($(MLNX_HW_MANAGEMENT)_SRC_PATH)/hw-mgmt >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en '\n###-> hw-management make file version change <-###\n' >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged -- $(PLATFORM_PATH)/hw-management.mk >> ${HWMGMT_USER_OUTFILE}
+
+	echo -en '\n###-> Summary of buildimage changes <-###\n' >> ${HWMGMT_USER_OUTFILE}
+	git diff --no-color --staged --stat --output=${TMPFILE_OUT} -- $(PLATFORM_PATH)
+	cat ${TMPFILE_OUT} | tee -a ${HWMGMT_USER_OUTFILE}
+
+	git diff --staged --quiet || git commit -m "Intgerate HW-MGMT ${MLNX_HW_MANAGEMENT_VERSION} Changes";
+	popd
+
+	popd $(LOG_SIMPLE)
+
+SONIC_PHONY_TARGETS += integrate-mlnx-hw-mgmt

platform/mellanox/integration-scripts/helper.py

@@ -0,0 +1,171 @@
+import os
+import glob
+import re
+
+MARK_ID = "###->"
+MLNX_KFG_MARKER = "mellanox"
+HW_MGMT_MARKER = "mellanox_hw_mgmt"
+SLK_PATCH_LOC = "src/sonic-linux-kernel/patch/"
+SLK_KCONFIG = SLK_PATCH_LOC + "kconfig-inclusions"
+SLK_KCONFIG_EXCLUDE = SLK_PATCH_LOC + "kconfig-exclusions"
+SLK_SERIES = SLK_PATCH_LOC + "series"
+NON_UP_PATCH_DIR = "platform/mellanox/non-upstream-patches/"
+NON_UP_PATCH_LOC = NON_UP_PATCH_DIR + "patches"
+NON_UP_PATCH_DIFF = NON_UP_PATCH_DIR + "series.patch"
+KCFG_HDR_RE = "\[(.*)\]"
+ # kconfig_inclusion headers to consider
+HDRS = ["common", "amd64"]
+
+class FileHandler:
+
+    @staticmethod
+    def write_lines(path, lines, raw=False):
+        # Create the dir if it doesn't exist already
+        os.makedirs(os.path.dirname(path), exist_ok=True)
+        with open(path, 'w') as f:
+            for line in lines:
+                if raw:
+                    f.write(f"{line}")
+                else:
+                    f.write(f"{line}\n")
+
+    @staticmethod
+    def read_raw(path):
+        # Read the data line by line into a list
+        data = []
+        with open(path) as im:
+            data = im.readlines()
+        return data
+
+    @staticmethod
+    def read_strip(path, as_string=False):
+        # Read the data line by line into a list and strip whitelines
+        data = FileHandler.read_raw(path)
+        data = [d.strip() for d in data]
+        if as_string:
+            return "\n".join(data)
+        return data
+
+    @staticmethod
+    def read_strip_minimal(path, as_string=False, ignore_start_with="#"):
+        # Read the data line by line into a list, strip spaces and ignore comments
+        data = FileHandler.read_raw(path)
+        filtered_data = []
+        for l in data:
+            l = l.strip()
+            if l and not l.startswith(ignore_start_with):
+                filtered_data.append(l)
+        if as_string:
+            return "\n".join(filtered_data)
+        return filtered_data
+
+    @staticmethod
+    def read_dir(path, ext="*") -> list:
+        return [os.path.basename(f) for f in glob.glob(os.path.join(path, ext))]
+
+    @staticmethod
+    def find_marker_indices(lines: list, marker=None) -> tuple:
+        i_start = -1
+        i_end = len(lines)
+        # print("TEST", marker, lines)
+        if marker:
+            for index, line in enumerate(lines):
+                # assumes one unique marker per file
+                # if multiple marker sections are present, reads the first one
+                if line.strip().startswith(MARK_ID):
+                    if marker+"-start" in line:
+                        i_start = index
+                    elif marker+"-end" in line:
+                        i_end = index
+        # print(i_start, i_end)             
+        return (i_start, i_end)
+
+    @staticmethod
+    def read_kconfig_inclusion(path, marker=MLNX_KFG_MARKER):
+        lines = FileHandler.read_strip(path)
+        if not marker:
+            return lines
+        i_start, i_end = FileHandler.find_marker_indices(lines, marker)
+
+        if i_start < 0 or i_end >= len(lines):
+            print("-> WARNING No Marker Found")
+            return []
+
+        return lines[i_start+1:i_end]
+
+    @staticmethod
+    def write_lines_marker(path, writable_opts: list, marker=None):
+        # if marker is none, just write the opts into the file,
+        # otherwise write the data only b/w the marker
+        curr_data = FileHandler.read_raw(path)
+        i_start, i_end = FileHandler.find_marker_indices(curr_data, marker)
+        newline_writ_opts = [opt + "\n" for opt in writable_opts]
+        if i_start < 0 or i_end >= len(curr_data):
+            print("-> WARNING No Marker Found, writing data at the end of file")
+            curr_data.extend(["\n"])
+            curr_data.extend(newline_writ_opts)
+        else:
+            curr_data = curr_data[0:i_start+1] + newline_writ_opts + curr_data[i_end:]
+
+        print("-> INFO Written the following opts: \n{}".format("".join(FileHandler.read_raw(path))))
+        FileHandler.write_lines(path, curr_data, True)
+
+    @staticmethod
+    def read_kconfig_parser(path) -> dict:
+        # kconfig_inclusion output formatted to {"no_parent", "common":[,], "amd64": [,], "arm64": [,]}
+        lines = FileHandler.read_strip_minimal(path)
+        ret = dict({"no_parent":[]})
+        curr_hdr = ""
+        for line in lines:
+            match = re.search(KCFG_HDR_RE, line)
+            if match:
+                curr_hdr = match.group(1)
+                ret[curr_hdr] = []
+            elif curr_hdr in ret:
+                ret[curr_hdr].append(line)
+            else:
+                ret["no_parent"].append(line)
+        return ret
+
+
+class KCFG:
+
+    @staticmethod
+    def parse_opt_str(opt: str) -> tuple:
+        if not opt.startswith("CONFIG"):
+            print("-> DEBUG: Malformed kconfig opt, {}".format(opt))
+            return ()
+
+        tmp = opt.split("=")
+        if len(tmp) != 2:
+            print("-> DEBUG: Malformed kconfig opt, {}".format(opt))
+            return ()
+
+        return (tmp[0], tmp[1])
+
+    @staticmethod
+    def parse_opts_strs(kcfg_sec: list) -> list(tuple()):
+        opts = [] # list of tuples (CONFIG_*, "m|y|n")
+        for kcfg in kcfg_sec:
+            tmp = KCFG.parse_opt_str(kcfg)
+            if tmp: 
+                opts.append(tmp)
+        return opts
+
+    @staticmethod
+    def get_writable_opts(opts):
+        lines = []
+        for opt in opts:
+            lines.append("{}={}".format(opt[0].upper(), opt[1]))
+        return lines
+
+
+class Action():
+    def __init__(self, args):
+        self.args = args
+
+    def perform(self):
+        pass
+
+    def write_user_out(self):
+        pass