To fix the issue: show_techsupport & saidump errors during testbed testing by replacing redis-rdb-tool with rdb-cli #1391

JunhongMao · 2024-06-11T02:37:45Z

Why I did it

Fix issue: #1387
The latest redis-rdb-tools-0.1.15 doesn't support Redis 7.0. Redis 7.0 was released in 2020 and adopted by SONiC's latest version. So, this issue turned out.

https://github.com/sripathikrishnan/redis-rdb-tools

I.e., the rdb-tools is far behind the Redis 7.0. The librdb can perfectly fix this issue. Please see quote from https://github.com/redis/librdb.

Motivation behind this project
There is a genuine need by the Redis community for a versatile RDB file parser that can export data, perform data analysis, or merely extract raw data from RDB and RESTORE it against a live Redis server. However, available parsers have shortcomings in some aspects such as lack of long-term support, lagging far behind the latest Redis release, and usually not being optimized for memory, performance, or high-traffic streaming for production environments. Additionally, most of them are not written in C, which limits the reuse of Redis components and the potential to contribute back to Redis repo. To address these issues, it is worthwhile to develop a new parser with a modern architecture, that maybe can also challenge the current integrated RDB parser of Redis and even replace it in the future.

So, the below PRS are to replace rdbtools with librdb's tool rdb-cli.
sonic-net/sonic-buildimage#19268
#1391

The above PRs are based on the below PRs.
#1288
#1298
sonic-net/sonic-buildimage#16466
sonic-net/sonic-utilities#2972

How I did it

To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it.

(1) Updated sonic-buildimage repo's platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, 
 install rdb-cli into the syncd containter.
(2) Updated sonic-sairedis repo's script file: files/scripts/saidump.sh, replace rdbtools with rdb-cli.
(3) Updated sonic-sairedis repo's saidump/saidump.cpp, to process the rdb-cli's ouput json file.

How to verify it

Method 1:
On T2 setup with more than 96K routes, execute CLI command -- generate_dump
No error should be shown

Method 2:
Go into syncd0 container.
admin@ixre-egl-board30:~$ docker exec -it syncd0 bash
Run below command, get saidump's sha1sum value.
root@ixre-egl-board30:/# saidump | sha1sum
1893097ecaeae72383432a54bfe4b285d05b23e5 -
Run below command, get saidump.sh's sha1sum value.
root@ixre-egl-board30:/# saidump.sh | sha1sum
1893097ecaeae72383432a54bfe4b285d05b23e5 -
The two sha1sum values should be the same and no error should be shown.
Method 3:
To run the testbed show_techsupport test case.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

JunhongMao · 2024-06-11T03:01:39Z

@mlok-nokia @kcudnik, please help to review, thanks.

kcudnik · 2024-06-11T07:27:20Z

Please fix coverage

saidump/main.cpp

saidump/saidump.h

saidump/Makefile.am

kcudnik · 2024-06-13T07:56:19Z

manyfiles have changes permissions from 644 to 755, please revert

saidump/saidump.cpp

saidump/saidump.h

tests/checkwhitespace.sh

unittest/saidump/main.cpp

JunhongMao · 2024-09-16T13:06:24Z

Hello, @kcudnik. Please help to review and approve. It's been a long time since this PR was created.

…sting sonic-net#1387

JunhongMao · 2024-10-07T14:34:09Z

Hello @deepak-singhal0408 , please help to review and approve. Thanks.

JunhongMao · 2024-10-07T14:36:27Z

Hello @deepak-singhal0408 , please help to review and approve. Thanks.

deepak-singhal0408 · 2024-10-08T19:39:53Z

syncd/scripts/saidump.sh

@@ -10,7 +10,8 @@ save_saidump_by_rdb()
 import json
 with open('$filepath') as json_file:
  data = json.load(json_file)
-  print(data['INSTANCES']['redis']['hostname'], data['INSTANCES']['redis']['port'], data['INSTANCES']['redis']['unix_socket_path'])")
+  print(data['INSTANCES']['redis']['hostname'], data['INSTANCES']['redis']['port'], data['INSTANCES']['redis']['unix_socket_path'])
+  json_file.close()")


nit: this is redundant as you are using with open..

deepak-singhal0408 · 2024-10-08T20:01:15Z

overall looks fine to me.. thanks.

deepak-singhal0408 · 2024-10-08T20:24:41Z

saidump/SaiDump.cpp

@@ -49,22 +31,21 @@ void printUsage()
    std::cout << "    -g --dumpGraph:" << std::endl;
    std::cout << "        Dump current graph" << std::endl;
    std::cout << "    -r --rdb:" << std::endl;
-    std::cout << "        Dump by parsing the RDB JSON file, which is created by rdbtools based on Redis dump.rdb that is generated by Redis SAVE command" << std::endl;
+    std::cout << "        Dump by parsing the RDB JSON file, which is created based on Redis dump.rdb that is generated by Redis SAVE command" << std::endl;


Can you update the description as we no longer use redis SAVE command.

abdosi · 2024-10-09T18:03:40Z

@deepak-singhal0408 : please create MSFT ADO

deepak-singhal0408 · 2024-10-22T16:45:08Z

MSFT ADO: 29975607

JunhongMao · 2024-11-11T20:28:21Z

@deepak-singhal0408, this PR may have a conflict to cherry-pick to 202405 branch. If so, please notify me. I will create a PR to cherry-pick it. Thanks!

bingwang-ms · 2024-11-15T16:54:38Z

Hi @JunhongMao, what's the impact if this change is missing?

JunhongMao · 2024-11-15T18:54:06Z

Hi @JunhongMao, what's the impact if this change is missing?

The issue #1387 exists.

JunhongMao changed the title ~~To fix the issue: show_techsupport & saidump errors during testbed testing by replacing rdb-tool with rdb-cli~~ To fix the issue: show_techsupport & saidump errors during testbed testing by replacing redis-rdb-tool with rdb-cli Jun 11, 2024

This was referenced Jun 11, 2024

To fix the issue: show_techsupport & saidump errors during testbed testing by replacing redis-rdb-tool with rdb-cli sonic-net/sonic-buildimage#19268

Merged

show_techsupport & saidump errors during testbed testing #1387

Closed

kcudnik reviewed Jun 12, 2024

View reviewed changes

saidump/main.cpp Outdated Show resolved Hide resolved

kcudnik reviewed Jun 12, 2024

View reviewed changes

saidump/saidump.h Outdated Show resolved Hide resolved

JunhongMao force-pushed the master branch 3 times, most recently from 033e6a2 to 1b9d9d5 Compare June 13, 2024 02:54