show_techsupport & saidump errors during testbed testing

[JunhongMao]

Error information

show_techsupport/test_techsupport.py::test_techsupport[acl-ixre-egl-board15]::setup

2024 May 28 00:59:35.593889 ixre-egl-board15 ERR syncd0#saidump: :- dumpFromRedisRdbJson: JSON file /var/run/redis0/dump.json is invalid.
2024 May 28 00:59:35.593889 ixre-egl-board15 ERR syncd0#saidump: :- dumpFromRedisRdbJson: JSON parsing error: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal.
2024 May 28 00:59:36.595630 ixre-egl-board15 ERR syncd1#saidump: :- dumpFromRedisRdbJson: JSON file /var/run/redis1/dump.json is invalid.
2024 May 28 00:59:36.595630 ixre-egl-board15 ERR syncd1#saidump: :- dumpFromRedisRdbJson: JSON parsing error: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal.
         

The details are shown below.
show_techsupport.html.pdf

admin@ixre-egl-board30:~$ show ver

Image version

SONiC Software Version: SONiC.HEAD.728118-nokia-master-69fa72b6
SONiC OS Version: 12
Distribution: Debian 12.5
Kernel: 6.1.0-11-2-amd64
Build commit: 69fa72b6
Build date: Wed May 29 00:19:20 UTC 2024
Built by: gitlab-runner@sonic-bld2

[kcudnik]

@JunhongMao you recently added json parser to saidump (previous version didn't support json) #1288, please investigate

[JunhongMao]

@kcudnik I found the root reason below:
The latest redis-rdb-tools-0.1.15 doesn't support Redis 7.0, which was released in 2020.

https://github.com/sripathikrishnan/redis-rdb-tools

I tested the PR: sripathikrishnan/redis-rdb-tools#193, and it works perfectly. But unfortunately, this PR is still in open status.

So I plan to add this PR's code by patch command. I.e., to update dockers/docker-base-bullseye/Dockerfile.j2 of repo sonic-buildimage by add a command, such as:

# Install rdbtools
RUN pip3 install 'rdbtools==0.1.15'
+ RUN patch -p1 -d /usr/local/lib/python3.9/dist-packages/rdbtools < /tmp/redis-rdb-tools-0.1.15-redis7.patch

Dockerfile.j2 was included in PR:
https://github.com/sonic-net/sonic-buildimage/pull/16466/files

But I'm not sure where the patch file should be in the repo https://github.com/sonic-net/sonic-buildimage. I have made the patch file below.
redis-rdb-tools-0.1.15-redis7.patch

Any comments are welcome and appreciated. Thank you.

[kcudnik]

is this a bug in rdb-tools or format changes or what ? maybe our code could woakaround that instead of waiting for rdb-tools ? for example if parrsing format fails, lets try other new one ?

[JunhongMao]

@kcudnik @mlok-nokia
On Redis 7.0, the RDB file format was upgraded. So, the rdb-tools no longer support it because it's far behind the Redis 7.0. By further investigation, I found the tool librdb is a better choice for this issue. Please see quote from https://github.com/redis/librdb.

Motivation behind this project
There is a genuine need by the Redis community for a versatile RDB file parser that can export data, perform data analysis, or merely extract raw data from RDB and RESTORE it against a live Redis server. However, available parsers have shortcomings in some aspects such as lack of long-term support, lagging far behind the latest Redis release, and usually not being optimized for memory, performance, or high-traffic streaming for production environments. Additionally, most of them are not written in C, which limits the reuse of Redis components and potential to contribute back to Redis repo. To address these issues, it is worthwhile to develop a new parser with a modern architecture, that maybe can also challenge the current integrated RDB parser of Redis and even replace it in the future.

I tested librdb, it works well. But how to add the librdb into the syncd0 container? Please give some suggestions. Thanks.

[kcudnik]

depending on which container you are talking, if az pipeline then you need to add librdb-dev https://packages.ubuntu.com/search?keywords=librbd-dev probably somwwhere here: https://github.com/sonic-net/sonic-sairedis/blob/master/.azure-pipelines/build-template.yml#L91

[JunhongMao]

It's librdb other than librbd-dev. They are different. The only way to install it is by building from source codes https://github.com/redis/librdb.git.

I'm trying the below method for repo sonic-buildimage.

1. Add a new folder src/librdb. Add src/librdb/Makefile below:

.ONESHELL:
SHELL = /bin/bash
.SHELLFLAGS += -e

MAIN_TARGET = rdb-cli

$(addprefix $(DEST)/, $(MAIN_TARGET)): $(DEST)/% :
	# Remove any stale files
	rm -rf ./librdb
	# Obtain librdb
	git clone https://github.com/redis/librdb.git
	# Build source
	pushd librdb
	git submodule update --init --recursive
	make test -j$(SONIC_CONFIG_MAKE_JOBS)
	make -j$(SONIC_CONFIG_MAKE_JOBS)
	popd
	mv librdb/bin/rdb-cli $(DEST)/

2. Add rules/librdb.mk

LIBRDB_VERSION = 0.0.1
LIBRDB = rdb-cli
$(LIBRDB)_SRC_PATH = $(SRC_PATH)/librdb
SONIC_MAKE_DEBS += $(LIBRDB)

export LIBRDB_VERSION
export LIBRDB

3. Add rules/librdb.dep

SPATH       := $($(LIBRDB)_SRC_PATH)
DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/librdb.mk rules/librdb.dep   
DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
DEP_FILES   += $(shell git ls-files $(SPATH))

$(LIBRDB)_CACHE_MODE  := GIT_CONTENT_SHA 
$(LIBRDB)_DEP_FLAGS   := $(SONIC_COMMON_FLAGS_LIST)
$(LIBRDB)_DEP_FILES   := $(DEP_FILES)

But this method doesn't work. How to do it? Thanks.

[kcudnik]

then you will need to probably install it manually from source in the docker

[kcudnik]

What do you mean that method dont work ? what errors do you get ?

[JunhongMao]

It means that after building the image and installing it on a board, the rdb-tool could not be found. Please see below.

admin@ixre-egl-board30:~$ rdb-tool
-bash: rdb-tool: command not found
admin@ixre-egl-board30:~$ docker exec -it syncd0 bash
root@ixre-egl-board30:/# rdb-tool
bash: rdb-tool: command not found
root@ixre-egl-board30:/#

[JunhongMao]

then you will need to probably install it manually from source in the docker

My question is how to add it into the repo sonic-buildimage.

[JunhongMao]

I have created two PRs for this issue. Please help to review them.
sonic-net/sonic-buildimage#19268
#1391

[kcudnik]

you probably need to do that in Dockerfile.j2 file, depends on which version you are building for example ./sonic-slave-buster/Dockerfile.j2:, if you take a look there are many different packages installed from source (tar.gz) in that docker, but probalby you can also add this src as submodule in sonic-buildimage/src and build those from source as packages let say in similar way as sonic-swss-common, and then install that in docker