[generate_dump] remove secrets from dump files #1886

liuh-80 · 2021-10-19T08:46:58Z

Remove secrets from dump files.

What I did

Add bash functions to remove secrets from dump files.

How I did it

For tacacs key, radius key, snmp community srring, use sed command with regex to remove user secrets from dump files.
For certs, update tar command exclude list to remove those certs from dump file.

How to verify it

Run 'show techsupport' command and check secrets removed from dump files.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

liuh-80 · 2021-10-21T03:08:29Z

azp /run

liuh-80 · 2021-10-21T03:09:12Z

/azp run

azure-pipelines · 2021-10-21T03:09:37Z

Azure Pipelines successfully started running 1 pipeline(s).

lguohan · 2021-10-22T02:15:21Z

need to back port this to 202012, 202006, 201911

qiluo-msft · 2021-10-22T05:03:38Z

scripts/generate_dump

@@ -180,6 +181,13 @@ save_cmd() {
        redirect_eval=""
    fi

+    # Cleanup metthod need do post procress on dump file.


metthod

typo #Closed

qiluo-msft · 2021-10-22T05:09:02Z

scripts/generate_dump

+    echo "Remove secret from '$filepath' files."
+
+    # Remove tacacs & radius passkey from config DB
+    sed -i -E 's/\"passkey\"\s*:\s*\"([^\"]*)\"/\"passkey\":\"\"/g' $filepath


passkey":""

Do you leave an emtpy passkey?
Is it better to replace with a fixed length of ****?
One benefit is indicating there is passkey cofigured. #Closed

qiluo-msft · 2021-10-22T05:10:33Z

scripts/generate_dump

+    echo "Remove secret from etc files."
+    # Remove tacacs passkey from tacplus_nss.conf
+    local secret_regex='s/(secret=)([^,|\S]*)(.*)/\1***\3/g'
+    sed -i -E $secret_regex $TARDIR/etc/tacplus_nss.conf


$TARDIR

Better to use function parameter instread of global var. #Closed

qiluo-msft · 2021-10-22T05:13:23Z

scripts/generate_dump

@@ -155,6 +154,7 @@ save_bcmcmd_all_ns() {
 #  cmd: The command to run. Make sure that arguments with spaces have quotes
 #  filename: the filename to save the output as in $BASE/dump
 #  do_gzip: (OPTIONAL) true or false. Should the output be gzipped
+#  cleanup_method: (OPTIONAL) the cleanup method to procress dump file after it generated.


cleanup_method

Is it possible to accept both do_gzip=true and cleanup_method=something? #Closed

Currently when do_gzip=true, dump output been redirect to gzip command with pipeline.
However, sed can work with pipeline, so I will investigate if we can invoke cleanup method in pipeline.

Fixed, UT failed because azure pipeline issue.

qiluo-msft · 2021-10-22T05:15:28Z

There is no 202006 branch. Is it a typo or do you really want creating that branch and backport?

In reply to: 949225407

liuh-80 · 2021-10-26T02:40:06Z

/azp run

azure-pipelines · 2021-10-26T02:40:19Z

Azure Pipelines successfully started running 1 pipeline(s).

qiluo-msft · 2021-10-28T02:58:18Z

Could you add some test cases in sonic-mgmt? I see some files in tests/show_techsupport/

qiluo-msft · 2021-11-05T00:50:00Z

This commit could not be cleanly cherry-picked to 202012. Please submit another PR.

…to 202012 #### What I did Add bash functions to remove secrets from dump files. #### How I did it For tacacs key, radius key, snmp community srring, use sed command with regex to remove user secrets from dump files. For certs, update tar command exclude list to remove those certs from dump file. #### How to verify it Run 'show techsupport' command and check secrets removed from dump files. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

…12 (#1938) #### What I did Add bash functions to remove secrets from dump files. #### How I did it For tacacs key, radius key, snmp community srring, use sed command with regex to remove user secrets from dump files. For certs, update tar command exclude list to remove those certs from dump file. #### How to verify it Run 'show techsupport' command and check secrets removed from dump files. #### Previous command output (if the output of a command-line utility has changed) #### New command output (if the output of a command-line utility has changed)

Remove secrets from dump files. What I did Add bash functions to remove secrets from dump files. How I did it For tacacs key, radius key, snmp community srring, use sed command with regex to remove user secrets from dump files. For certs, update tar command exclude list to remove those certs from dump file. How to verify it Run 'show techsupport' command and check secrets removed from dump files. Previous command output (if the output of a command-line utility has changed) New command output (if the output of a command-line utility has changed) # Conflicts: # scripts/generate_dump

Backport [generate_dump] remove secrets from dump files #1886 to 201911 What I did Add bash functions to remove secrets from dump files. How I did it For tacacs key, radius key, snmp community srring, use sed command with regex to remove user secrets from dump files. For certs, update tar command exclude list to remove those certs from dump file. How to verify it Run 'show techsupport' command and check secrets removed from dump files.

* c3691d3 [202012][pfcwd] Convert polling interval from ms to us in LUA scripts (sonic-net#1909) * 549c804 Mux state order change (sonic-net#1902) * 6b0b2c4 Update acl type check logic (sonic-net#1886) Signed-off-by: Volodymyr Samotiy <volodymyrs@nvidia.com>

Submodule update for sonic-utilities with following change: ec9e5ee Backport [generate_dump] remove secrets from dump files sonic-net#1886 to 202012 (sonic-net#1938) ce3b856 [fdbshow]: Handle FDB cleanup gracefully. (sonic-net#1926) 1437bf2 [202012] Add DHCPv6 Relay counter and ipv6 helper CLI (sonic-net#1917)

liuh-80 and others added 5 commits October 18, 2021 17:24

Remove secret from show techsupport dump.

a093607

Improve script.

db73f56

Improve script.

b63bfbf

Improve scripts.

739f3f1

Improve script

e28d5f7

liuh-80 marked this pull request as ready for review October 22, 2021 01:55

lguohan requested a review from qiluo-msft October 22, 2021 02:14

Revert exclude files change for mlnx and mft.

856d1b4

qiluo-msft added Request for 201911 Branch Request for 202012 Branch labels Oct 22, 2021

qiluo-msft reviewed Oct 22, 2021

View reviewed changes

liuh-80 and others added 3 commits October 22, 2021 22:52

Fix PR comments.

200c962

Fix PR comments, improve cleanup method to work with pipeline and gzip.

5cc7774

Fix script issue.

6866899

qiluo-msft approved these changes Oct 27, 2021

View reviewed changes

liuh-80 merged commit 9f123c0 into sonic-net:master Oct 28, 2021

liuh-80 mentioned this pull request Nov 22, 2021

Backport [generate_dump] remove secrets from dump files #1886 to 202012 #1938

Merged

qiluo-msft removed the Request for 202012 Branch label Nov 23, 2021

qiluo-msft mentioned this pull request Nov 30, 2021

[test_techsupport]: Add new test cases for remove password and certs from show techsupport result. sonic-net/sonic-mgmt#4767

Merged

3 tasks

This was referenced Dec 6, 2021

Backport [generate_dump] remove secrets from dump files #1886 to 201911 #1959

Closed

[generate_dump][201911] Remove secrets from dump files #1960

Merged

vivekrnv mentioned this pull request Jan 24, 2022

[techsupport] failures seen in generate_dump stdout while handling /etc/ folder sonic-net/sonic-buildimage#9843

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[generate_dump] remove secrets from dump files #1886

[generate_dump] remove secrets from dump files #1886

liuh-80 commented Oct 19, 2021 •

edited

Loading

liuh-80 commented Oct 21, 2021

liuh-80 commented Oct 21, 2021

azure-pipelines bot commented Oct 21, 2021

lguohan commented Oct 22, 2021

qiluo-msft Oct 22, 2021 •

edited

Loading

liuh-80 Oct 22, 2021

qiluo-msft Oct 22, 2021 •

edited

Loading

liuh-80 Oct 22, 2021

qiluo-msft Oct 22, 2021 •

edited

Loading

liuh-80 Oct 22, 2021

qiluo-msft Oct 22, 2021 •

edited

Loading

liuh-80 Oct 22, 2021

liuh-80 Oct 26, 2021

qiluo-msft commented Oct 22, 2021

liuh-80 commented Oct 26, 2021

azure-pipelines bot commented Oct 26, 2021

qiluo-msft commented Oct 28, 2021

qiluo-msft commented Nov 5, 2021

[generate_dump] remove secrets from dump files #1886

[generate_dump] remove secrets from dump files #1886

Conversation

liuh-80 commented Oct 19, 2021 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

liuh-80 commented Oct 21, 2021

liuh-80 commented Oct 21, 2021

azure-pipelines bot commented Oct 21, 2021

lguohan commented Oct 22, 2021

qiluo-msft Oct 22, 2021 • edited Loading

Choose a reason for hiding this comment

liuh-80 Oct 22, 2021

Choose a reason for hiding this comment

qiluo-msft Oct 22, 2021 • edited Loading

Choose a reason for hiding this comment

liuh-80 Oct 22, 2021

Choose a reason for hiding this comment

qiluo-msft Oct 22, 2021 • edited Loading

Choose a reason for hiding this comment

liuh-80 Oct 22, 2021

Choose a reason for hiding this comment

qiluo-msft Oct 22, 2021 • edited Loading

Choose a reason for hiding this comment

liuh-80 Oct 22, 2021

Choose a reason for hiding this comment

liuh-80 Oct 26, 2021

Choose a reason for hiding this comment

qiluo-msft commented Oct 22, 2021

liuh-80 commented Oct 26, 2021

azure-pipelines bot commented Oct 26, 2021

qiluo-msft commented Oct 28, 2021

qiluo-msft commented Nov 5, 2021

liuh-80 commented Oct 19, 2021 •

edited

Loading

qiluo-msft Oct 22, 2021 •

edited

Loading

qiluo-msft Oct 22, 2021 •

edited

Loading

qiluo-msft Oct 22, 2021 •

edited

Loading

qiluo-msft Oct 22, 2021 •

edited

Loading