NAT bulking and SAI NAT aging notification #2091

arvbb · 2021-12-27T17:22:10Z

This PR covers:

Changes in sonic-swss to handle Aging notification from SAI. Related PRs:
SAI NAT aging notification opencomputeproject/SAI#1365
SAI NAT aging notification sonic-sairedis#987
Bulking the NAT messages between Natorch and Syncd.

These changes are aimed to improve the SONiC NAT session setup rate.

The changes were verified by measuring the time taken to setup a large number of dynamic NAT entries (closer to programmable hardware upper limit) on the SONiC device.

As described in SAI PR, the frequent polling of dynamic NAT entries using Hitbit timer is not efficient and instead the Aging mechanism is used by SAI to notify SONiC when NAT entry is aged out. Natorch processes these notifications to cleanup the aged out NAT entries.

Bulk changes use the bulker mechanism previously used for bulking routes, etc. This change decreased the time to program NAT entries at both SONiC and SAI level.

rlhui · 2021-12-28T17:59:25Z

Thanks. Would we please put more description for the PR (fill in the template).

arlakshm · 2022-01-11T20:32:18Z

@AkhileshSamineni , can you review this PR

arlakshm · 2022-01-11T20:37:01Z

@arvbb can you add Unit tests for this ?

arlakshm · 2022-01-11T20:37:29Z

/Azp run Azure.sonic-swss

azure-pipelines · 2022-01-11T20:37:39Z

Azure Pipelines successfully started running 1 pipeline(s).

arvbb · 2022-01-31T07:28:11Z

@arvbb can you add Unit tests for this ?

Hi @arlakshm, thanks for review.. Added UT logs..

root@sonic:/home/admin# show nat config
 
Global Values
 
Admin Mode     : enabled
Global Timeout : 600 secs
TCP Timeout    : 86400 secs
UDP Timeout    : 300 secs
Static Entries
 
Nat Type    IP Protocol    Global IP    Global Port    Local IP    Local Port     Twice-NAT Id
----------  -------------  -----------  -------------  ----------  ------------  ---------------
Pool Entries
 
Pool Name    Global IP Range    Global Port Range
-----------  -----------------  -------------------
test_pool    10.10.10.10        2000-42000
NAT Bindings
 
Binding Name    Pool Name    Access-List     Nat Type    Twice-NAT Id
--------------  -----------  --------------  ----------  --------------
test_binding    test_pool    test_acl_table  snat        ---
NAT Zones
 
Port           Zone
-----------  ------
Ethernet0         0
Ethernet100       0
Ethernet104       0
Ethernet108       0
Ethernet112       0
Ethernet116       0
Ethernet12        0
Ethernet120       0
Ethernet124       0
Ethernet16        0
Ethernet20        0
Ethernet24        0
Ethernet28        0
Ethernet32        0
Ethernet36        0
Ethernet4         0
Ethernet40        0
Ethernet44        0
Ethernet48        0
Ethernet52        0
Ethernet56        0
Ethernet60        0
Ethernet64        0
Ethernet68        0
Ethernet72        0
Ethernet76        0
Ethernet8         0
Ethernet80        1
Ethernet84        0
Ethernet88        0
Ethernet92        0
Ethernet96        0
Loopback0         0
Loopback10        1
root@sonic:/home/admin# show nat statistics
 
Protocol    Source    Destination    Packets    Bytes
----------  --------  -------------  ---------  -------
root@sonic:/home/admin# show nat translations
 
Static NAT Entries         ..................... 0
Static NAPT Entries        ..................... 0
Dynamic NAT Entries        ..................... 0
Dynamic NAPT Entries       ..................... 0
Static Twice NAT Entries   ..................... 0
Static Twice NAPT Entries  ..................... 0
Dynamic Twice NAT Entries  ..................... 0
Dynamic Twice NAPT Entries ..................... 0
Total SNAT/SNAPT Entries   ..................... 0
Total DNAT/DNAPT Entries   ..................... 0
Total Entries              ..................... 0
 
Protocol    Source    Destination    Translated Source    Translated Destination
----------  --------  -------------  -------------------  ------------------------
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin#
root@sonic:/home/admin# config nat set tcp-timeout 300
root@sonic:/home/admin#
 
 
 
After sending TCP packets…
 
 
 
swss.rec
2022-01-31.08:58:26.169025|NAPT_TABLE:TCP:10.10.10.10:20056|SET|nat_type:dnat|entry_type:dynamic|translated_ip:1.1.1.1|translated_l4_port:20056
2022-01-31.08:58:26.169138|NAPT_TABLE:TCP:1.1.1.1:20056|SET|nat_type:snat|entry_type:dynamic|translated_ip:10.10.10.10|translated_l4_port:20056
 
sairedis.rec
2022-01-31.08:58:26.170092|C|SAI_OBJECT_TYPE_NAT_ENTRY||{"nat_data":{"key":{"dst_ip":"10.10.10.10","l4_dst_port":"20056","l4_src_port":"0","proto":"6","src_ip":"0.0.0.0"},"mask":{"dst_ip":"255.255.255.255","l4_dst_port":"65535","l4_src_port":"0","proto":"255","src_ip":"0.0.0.0"}},"nat_type":"SAI_NAT_TYPE_DESTINATION_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}|SAI_NAT_ENTRY_ATTR_DST_IP=1.1.1.1|SAI_NAT_ENTRY_ATTR_DST_IP_MASK=255.255.255.255|SAI_NAT_ENTRY_ATTR_L4_DST_PORT=20056|SAI_NAT_ENTRY_ATTR_ENABLE_PACKET_COUNT=true|SAI_NAT_ENTRY_ATTR_ENABLE_BYTE_COUNT=true|SAI_NAT_ENTRY_ATTR_AGING_TIME=300||{"nat_data":{"key":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"20056","proto":"6","src_ip":"1.1.1.1"},"mask":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"65535","proto":"255","src_ip":"255.255.255.255"}},"nat_type":"SAI_NAT_TYPE_SOURCE_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}|SAI_NAT_ENTRY_ATTR_SRC_IP=10.10.10.10|SAI_NAT_ENTRY_ATTR_SRC_IP_MASK=255.255.255.255|SAI_NAT_ENTRY_ATTR_L4_SRC_PORT=20056|SAI_NAT_ENTRY_ATTR_ENABLE_PACKET_COUNT=true|SAI_NAT_ENTRY_ATTR_ENABLE_BYTE_COUNT=true|SAI_NAT_ENTRY_ATTR_AGING_TIME=300
 
 
syslog
Jan 31 08:58:26.160179 sonic NOTICE nat#natsyncd: :- addNatEntry: SNAPT entry with key TCP:1.1.1.1:20056 added to APP_DB
Jan 31 08:58:26.160900 sonic NOTICE nat#natsyncd: :- addNatEntry: Implicit DNAPT entry with key TCP:10.10.10.10:20056 added to APP_DB
Jan 31 08:58:26.162549 sonic NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 1.1.1.1, 68:05:ca:a3:98:fd on Ethernet72
Jan 31 08:58:26.165374 sonic NOTICE swss#orchagent: :- addNextHop: Created next hop 1.1.1.1 on Ethernet72
Jan 31 08:58:26.167471 sonic NOTICE swss#orchagent: :- addNeighbor: Created neighbor ip 2.2.2.2, 68:05:ca:a3:98:fc on Ethernet80
Jan 31 08:58:26.168600 sonic NOTICE swss#orchagent: :- addNextHop: Created next hop 2.2.2.2 on Ethernet80
Jan 31 08:58:26.169564 sonic NOTICE swss#orchagent: :- addHwSnaptEntry: Created dynamic SNAT NAPT entry with ip 1.1.1.1, port 20056, prototype TCP and it's translated ip 10.10.10.10, translated port 20056
Jan 31 08:58:26.169796 sonic NOTICE swss#orchagent: :- addHwDnaptEntry: Created dynamic DNAT NAPT entry with ip 10.10.10.10, port 20056, prototype TCP and it's translated ip 1.1.1.1, translated port 20056
Jan 31 08:58:26.178902 sonic INFO nat#/supervisord: natmgrd conntrack v1.4.5 (conntrack-tools): connection tracking table has been emptied.
Jan 31 08:58:26.178902 sonic INFO nat#/supervisord: natmgrd conntrack v1.4.5 (conntrack-tools): 1 flow entries have been updated.
 
 
root@sonic:/home/admin#
root@sonic:/home/admin# show nat statistics
 
Protocol    Source         Destination          Packets    Bytes
----------  -------------  -----------------  ---------  -------
tcp         ---            10.10.10.10:20056          0        0
tcp         1.1.1.1:20056  ---                        0        0
root@sonic:/home/admin# show nat translations
 
Static NAT Entries         ..................... 0
Static NAPT Entries        ..................... 0
Dynamic NAT Entries        ..................... 0
Dynamic NAPT Entries       ..................... 2
Static Twice NAT Entries   ..................... 0
Static Twice NAPT Entries  ..................... 0
Dynamic Twice NAT Entries  ..................... 0
Dynamic Twice NAPT Entries ..................... 0
Total SNAT/SNAPT Entries   ..................... 1
Total DNAT/DNAPT Entries   ..................... 1
Total Entries              ..................... 2
 
Protocol    Source         Destination        Translated Source    Translated Destination
----------  -------------  -----------------  -------------------  ------------------------
tcp         1.1.1.1:20056  ---                10.10.10.10:20056    ---
tcp         ---            10.10.10.10:20056  ---                  1.1.1.1:20056
root@sonic:/home/admin#
 
 
 



After aging timeout, SAI notifies for delete
 
swss.rec
2022-01-31.09:03:41.492676|NAPT_TABLE:TCP:1.1.1.1:20056|DEL
2022-01-31.09:03:41.497017|NAPT_TABLE:TCP:10.10.10.10:20056|DEL
 
 
sairedis.rec
2022-01-31.09:03:41.469201|n|nat_event|[{"nat_entry":"{\"nat_data\":{\"key\":{\"dst_ip\":\"10.10.10.10\",\"l4_dst_port\":\"20056\",\"l4_src_port\":\"0\",\"proto\":\"6\",\"src_ip\":\"0.0.0.0\"},\"mask\":{\"dst_ip\":\"255.255.255.255\",\"l4_dst_port\":\"65535\",\"l4_src_port\":\"0\",\"proto\":\"255\",\"src_ip\":\"0.0.0.0\"}},\"nat_type\":\"SAI_NAT_TYPE_DESTINATION_NAT\",\"switch_id\":\"oid:0x21000000000000\",\"vr\":\"oid:0x3000000000048\"}","nat_event":"SAI_NAT_EVENT_AGED"}]|
2022-01-31.09:03:41.475089|n|nat_event|[{"nat_entry":"{\"nat_data\":{\"key\":{\"dst_ip\":\"0.0.0.0\",\"l4_dst_port\":\"0\",\"l4_src_port\":\"20056\",\"proto\":\"6\",\"src_ip\":\"1.1.1.1\"},\"mask\":{\"dst_ip\":\"0.0.0.0\",\"l4_dst_port\":\"0\",\"l4_src_port\":\"65535\",\"proto\":\"255\",\"src_ip\":\"255.255.255.255\"}},\"nat_type\":\"SAI_NAT_TYPE_SOURCE_NAT\",\"switch_id\":\"oid:0x21000000000000\",\"vr\":\"oid:0x3000000000048\"}","nat_event":"SAI_NAT_EVENT_AGED"}]|
2022-01-31.09:03:41.493851|R|SAI_OBJECT_TYPE_NAT_ENTRY||{"nat_data":{"key":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"20056","proto":"6","src_ip":"1.1.1.1"},"mask":{"dst_ip":"0.0.0.0","l4_dst_port":"0","l4_src_port":"65535","proto":"255","src_ip":"255.255.255.255"}},"nat_type":"SAI_NAT_TYPE_SOURCE_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}
2022-01-31.09:03:41.497821|R|SAI_OBJECT_TYPE_NAT_ENTRY||{"nat_data":{"key":{"dst_ip":"10.10.10.10","l4_dst_port":"20056","l4_src_port":"0","proto":"6","src_ip":"0.0.0.0"},"mask":{"dst_ip":"255.255.255.255","l4_dst_port":"65535","l4_src_port":"0","proto":"255","src_ip":"0.0.0.0"}},"nat_type":"SAI_NAT_TYPE_DESTINATION_NAT","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000048"}
 
 
syslog
Jan 31 09:03:41.492598 sonic NOTICE nat#natsyncd: :- addNatEntry: SNAPT entry with key TCP:1.1.1.1:20056 deleted from APP_DB
Jan 31 09:03:41.493032 sonic INFO nat#/supervisord: natmgrd conntrack v1.4.5 (conntrack-tools): 0 flow entries have been updated.
Jan 31 09:03:41.493245 sonic NOTICE swss#orchagent: :- removeHwSnaptEntry: Removed dynamic SNAT NAPT entry with ip 1.1.1.1, port 20056, prototype TCP and it's translated ip 10.10.10.10, translated port 20056
Jan 31 09:03:41.493410 sonic NOTICE nat#natsyncd: :- addNatEntry: Implicit DNAPT entry with key TCP:10.10.10.10:20056 deleted from APP_DB
Jan 31 09:03:41.493960 sonic INFO nat#supervisord 2022-01-31 09:03:41,493 INFO reaped unknown pid 166 (exit status 1)
Jan 31 09:03:41.497493 sonic NOTICE swss#orchagent: :- removeHwDnaptEntry: Removed dynamic DNAT NAPT entry with ip 10.10.10.10, port 20056, prototype TCP and it's translated ip 1.1.1.1, translated port 20056
 
 
 
root@sonic:/home/admin# show nat statistics
 
Protocol    Source    Destination    Packets    Bytes
----------  --------  -------------  ---------  -------
root@sonic:/home/admin# show nat translations
 
Static NAT Entries         ..................... 0
Static NAPT Entries        ..................... 0
Dynamic NAT Entries        ..................... 0
Dynamic NAPT Entries       ..................... 0
Static Twice NAT Entries   ..................... 0
Static Twice NAPT Entries  ..................... 0
Dynamic Twice NAT Entries  ..................... 0
Dynamic Twice NAPT Entries ..................... 0
Total SNAT/SNAPT Entries   ..................... 0
Total DNAT/DNAPT Entries   ..................... 0
Total Entries              ..................... 0
 
Protocol    Source    Destination    Translated Source    Translated Destination
----------  --------  -------------  -------------------  ------------------------
root@sonic:/home/admin#

kperumalbfn · 2022-03-08T01:50:23Z

@AkhileshSamineni @arlakshm could you pls review the PR. SAI PR has been merged with new set of attributes for this feature. Sonic build will be successful after SAI refpoint update on sonic-sairedis repo.

sunesh · 2022-05-09T23:45:16Z

/Azp run Azure.sonic-swss

azure-pipelines · 2022-05-09T23:45:21Z

Commenter does not have sufficient privileges for PR 2091 in repo Azure/sonic-swss

arvbb · 2022-05-10T01:38:07Z

/azp run

azure-pipelines · 2022-05-10T01:38:12Z

Commenter does not have sufficient privileges for PR 2091 in repo Azure/sonic-swss

kperumalbfn · 2022-05-10T05:18:47Z

/azp run

azure-pipelines · 2022-05-10T05:18:52Z

Commenter does not have sufficient privileges for PR 2091 in repo Azure/sonic-swss

arvbb · 2022-05-10T19:18:12Z

/azpw run

mssonicbld · 2022-05-10T19:18:13Z

/AzurePipelines run

azure-pipelines · 2022-05-10T19:18:25Z

Azure Pipelines successfully started running 1 pipeline(s).

arlakshm · 2022-05-24T22:08:15Z

Please add some unit tests

arvbb · 2022-09-26T19:25:44Z

test_nat.py

collected 18 items

test_nat.py::TestNat::test_NatGlobalTable PASSED [ 5%]
test_nat.py::TestNat::test_NatInterfaceZone PASSED [ 11%]
test_nat.py::TestNat::test_AddNatStaticEntry PASSED [ 16%]
test_nat.py::TestNat::test_DelNatStaticEntry PASSED [ 22%]
test_nat.py::TestNat::test_AddNaPtStaticEntry PASSED [ 27%]
test_nat.py::TestNat::test_DelNaPtStaticEntry PASSED [ 33%]
test_nat.py::TestNat::test_AddTwiceNatEntry SKIPPED [ 38%]
test_nat.py::TestNat::test_DelTwiceNatStaticEntry PASSED [ 44%]
test_nat.py::TestNat::test_AddTwiceNaPtEntry PASSED [ 50%]
test_nat.py::TestNat::test_DelTwiceNaPtStaticEntry PASSED [ 55%]
test_nat.py::TestNat::test_VerifyConntrackTimeoutForNatEntry PASSED [ 61%]
test_nat.py::TestNat::test_DoNotNatAclAction PASSED [ 66%]
test_nat.py::TestNat::test_CrmSnatAndDnatEntryUsedCount PASSED [ 72%]
test_nat.py::TestNat::test_AddSnatDynamicEntry PASSED [ 77%]
test_nat.py::TestNat::test_AddDnatDynamicEntry PASSED [ 83%]
test_nat.py::TestNat::test_AddSnaptDynamicEntry PASSED [ 88%]
test_nat.py::TestNat::test_AddDnaptDynamicEntry PASSED [ 94%]
test_nat.py::test_nonflaky_dummy PASSED [100%]

============================================ 17 passed, 1 skipped in 181.18 seconds ============================================

lgtm-com · 2022-11-08T22:18:52Z

This pull request introduces 1 alert when merging b815d86 into ab0e474 - view on LGTM.com

new alerts:

1 for Unused local variable

arvbb · 2022-12-08T08:16:00Z

Please add some unit tests

@arlakshm -Hi, added UT, could you please take a look, thanks.

NAT bulking and SAI NAT aging notification

1a4f67b

arvbb requested a review from prsunny as a code owner December 27, 2021 17:22

rlhui assigned arlakshm Dec 28, 2021

arvbb force-pushed the master branch from 0e4e5d2 to 1a4f67b Compare March 11, 2022 04:34

prsunny requested a review from arlakshm May 26, 2022 04:59

arvbb added 2 commits September 25, 2022 15:35

Merge branch 'master' into master

47a9905

Added UT for dynamic NAT and Aging

5275e6c

arvbb added 5 commits October 7, 2022 10:13

Merge branch 'sonic-net:master' into master

f2e755a

Merge branch 'sonic-net:master' into master

98e74f2

Merge branch 'sonic-net:master' into master

bd99ed7

Merge branch 'sonic-net:master' into master

adf67f4

Merge branch 'sonic-net:master' into master

177c3c3

Added UT for dynamic NAT notification

b815d86

arvbb added 7 commits November 8, 2022 14:34

Fixing unused var

8cc620a

Merge branch 'sonic-net:master' into master

6c74ffb

Fixed missing import

ed6b0e0

Merge branch 'master' of github.com:arvbb/sonic-swss

48b7da2

Merge branch 'sonic-net:master' into master

6056c60

Merge branch 'sonic-net:master' into master

ba50d2b

Merge branch 'sonic-net:master' into master

c5ff617

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NAT bulking and SAI NAT aging notification #2091

NAT bulking and SAI NAT aging notification #2091

arvbb commented Dec 27, 2021 •

edited

Loading

rlhui commented Dec 28, 2021

arlakshm commented Jan 11, 2022

arlakshm commented Jan 11, 2022

arlakshm commented Jan 11, 2022

azure-pipelines bot commented Jan 11, 2022

arvbb commented Jan 31, 2022

kperumalbfn commented Mar 8, 2022

sunesh commented May 9, 2022

azure-pipelines bot commented May 9, 2022

arvbb commented May 10, 2022

azure-pipelines bot commented May 10, 2022

kperumalbfn commented May 10, 2022

azure-pipelines bot commented May 10, 2022

arvbb commented May 10, 2022

mssonicbld commented May 10, 2022

azure-pipelines bot commented May 10, 2022

arlakshm commented May 24, 2022

arvbb commented Sep 26, 2022

lgtm-com bot commented Nov 8, 2022

arvbb commented Dec 8, 2022

NAT bulking and SAI NAT aging notification #2091

Are you sure you want to change the base?

NAT bulking and SAI NAT aging notification #2091

Conversation

arvbb commented Dec 27, 2021 • edited Loading

rlhui commented Dec 28, 2021

arlakshm commented Jan 11, 2022

arlakshm commented Jan 11, 2022

arlakshm commented Jan 11, 2022

azure-pipelines bot commented Jan 11, 2022

arvbb commented Jan 31, 2022

kperumalbfn commented Mar 8, 2022

sunesh commented May 9, 2022

azure-pipelines bot commented May 9, 2022

arvbb commented May 10, 2022

azure-pipelines bot commented May 10, 2022

kperumalbfn commented May 10, 2022

azure-pipelines bot commented May 10, 2022

arvbb commented May 10, 2022

mssonicbld commented May 10, 2022

azure-pipelines bot commented May 10, 2022

arlakshm commented May 24, 2022

arvbb commented Sep 26, 2022

test_nat.py

lgtm-com bot commented Nov 8, 2022

arvbb commented Dec 8, 2022

arvbb commented Dec 27, 2021 •

edited

Loading