Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCU] Optimize full key deletion/addition, to be done in a single step rather than multiple #2099

Closed
ghooo opened this issue Mar 9, 2022 · 0 comments · Fixed by #2120
Closed

Comments

@ghooo
Copy link
Contributor

ghooo commented Mar 9, 2022

Description

Example:

admin@vlab-01:~/everflow$ cat add_session.json2
[ {
  "op": "add",
  "path": "/MIRROR_SESSION",
  "value": {
   "mirror_session_dscp": {
    "dscp": "5",
    "dst_ip": "2.2.2.2",
    "src_ip": "1.1.1.1",
    "ttl": "32",
    "type": "ERSPAN"
   }
  }
 }
 ]
admin@vlab-01:~/everflow$ sudo config apply-patch add_session.json2
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "add", "path": "/MIRROR_SESSION", "value": {"mirror_session_dscp": {"dscp": "5", "dst_ip": "2.2.2.2", "src_ip": "1.1.1.1", "ttl": "32", "type": "ERSPAN"}}}]
Patch Applier: Getting current config db.
Patch Applier: Simulating the target full config after applying the patch.
Patch Applier: Validating target config does not have empty tables, since they do not show up in ConfigDb.
...
Patch Applier: Applying 5 changes in order:
Patch Applier:   * [{"op": "add", "path": "/MIRROR_SESSION", "value": {"mirror_session_dscp": {"dscp": "5"}}}]
Patch Applier:   * [{"op": "add", "path": "/MIRROR_SESSION/mirror_session_dscp/dst_ip", "value": "2.2.2.2"}]
Patch Applier:   * [{"op": "add", "path": "/MIRROR_SESSION/mirror_session_dscp/src_ip", "value": "1.1.1.1"}]
Patch Applier:   * [{"op": "add", "path": "/MIRROR_SESSION/mirror_session_dscp/ttl", "value": "32"}]
Patch Applier:   * [{"op": "add", "path": "/MIRROR_SESSION/mirror_session_dscp/type", "value": "ERSPAN"}]
Patch Applier: Verifying patch updates are reflected on ConfigDB.
Patch Applier: Patch application completed.
Patch applied successfully.

The generated steps are 5 changes. This can be optimized to only 1 step.
i.e.

Patch Applier: Applying 1 changes in order:
Patch Applier:   * [{"op": "add", "path": "/MIRROR_SESSION", "value": {"mirror_session_dscp": {"dscp": "5", "dst_ip": "2.2.2.2", "src_ip": "1.1.1.1", "ttl": "32", "type": "ERSPAN"}}}]
Patch Applier: Verifying patch updates are reflected on ConfigDB.
Patch Applier: Patch application completed.
Patch applied successfully.

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

Additional information you deem important (e.g. issue happens only occasionally)

Output of show version

(paste your output here)
ghooo added a commit that referenced this issue Apr 1, 2022
#### What I did
Fixes #2099

Grouping all fields under tables/keys to be added/deleted in 1 JsonChange.


**BEFORE** how did we generate moves to try?
- Start with the low level differences, low level means differences that has no children
- Then extend this low level by different extenders, the most used is the `UpperLevelMoveExtender`
- Extended the already extended moves to go upper in level, and do that multiple times until there are no possible moves to extend

The diagram below shows:
- Generation (low level):
```json
{"op":"replace", "path":"/ACL_RULE/SNMP_ACL|RULE_1/SRC_IP", "value": "1.1.1.1/21"}
{"op":"remove", "path":"/ACL_RULE/SNMP_ACL|RULE_2/PRIORITY"}
{"op":"remove", "path":"/ACL_RULE/SNMP_ACL|RULE_2/SRC_IP"}
{"op":"remove", "path":"/ACL_RULE/SNMP_ACL|RULE_2/IP_PROTOCOL"}
{"op":"remove", "path":"/ACL_RULE/SNMP_ACL|RULE_2/PACKET_ACTION"}
{"op":"add", "path":"/ACL_RULE/SNMP_ACL|RULE_3/PRIORITY", "value": "9997"}
{"op":"add", "path":"/ACL_RULE/SNMP_ACL|RULE_3/SRC_IP", "value": "3.3.3.3/20"}
{"op":"add", "path":"/ACL_RULE/SNMP_ACL|RULE_3/IP_PROTOCOL", "value": "17"}
{"op":"add", "path":"/ACL_RULE/SNMP_ACL|RULE_3/PACKET_ACTION", "value": "ACCEPT"}
```
- Extension - 1st level
```json
{"op":"replace", "path":"/ACL_RULE/SNMP_ACL|RULE_1", "value": {"PRIORITY": "9999","SRC_IP": "2.2.2.2/21","SRC_IP": "1.1.1.1/21","IP_PROTOCOL": "17","PACKET_ACTION": "ACCEPT"}}
{"op":"remove", "path":"/ACL_RULE/SNMP_ACL|RULE_2"}
{"op":"add", "path":"/ACL_RULE/SNMP_ACL|RULE_3", "value": {"PRIORITY": "9997","SRC_IP": "3.3.3.3/20","IP_PROTOCOL": "17","PACKET_ACTION": "ACCEPT"}}
```
- Extension - 2nd level
```json
{"op":"replace", "path":"/ACL_RULE", "value": {"SNMP_ACL|RULE_1": {"PRIORITY": "9999","SRC_IP": "1.1.1.1/21","IP_PROTOCOL": "17","PACKET_ACTION": "ACCEPT"},"SNMP_ACL|RULE_3": {"PRIORITY": "9997","SRC_IP": "3.3.3.3/20","IP_PROTOCOL": "17","PACKET_ACTION": "ACCEPT"},"SNMP_ACL|DEFAULT_RULE": {"PRIORITY": "1","ETHER_TYPE": "2048","PACKET_ACTION": "DROP"}}}
```
- Extension - 3rd level
 ```json
{"op":"replace", "path":"", "value": {"ACL_RULE": {"SNMP_ACL|RULE_1": {"PRIORITY": "9999","SRC_IP": "1.1.1.1/21","IP_PROTOCOL": "17","PACKET_ACTION": "ACCEPT"},"SNMP_ACL|RULE_3": {"PRIORITY": "9997","SRC_IP": "3.3.3.3/20","IP_PROTOCOL": "17","PACKET_ACTION": "ACCEPT"},"SNMP_ACL|DEFAULT_RULE": {"PRIORITY": "1","ETHER_TYPE": "2048","PACKET_ACTION": "DROP"}},"ACL_TABLE": {"SNMP_ACL": {"type": "CTRLPLANE","policy_desc": "SNMP_ACL","services": ["SNMP"]}}}}
```
![image](https://user-images.githubusercontent.com/3927651/160676723-29688656-3382-4247-8358-cb988cda5134.png)


**AFTER**
In this PR, we introduce a different kind of generator. The non-extendable generators i.e. generators that their moves do not get extended using the extenders. We added 2 generators:
- KeyLevelGenerator: if a whole key to be deleted or added, do that directly.
- TableLevelGenerator: if a whole table to be deleted or added, do that directly.

Only **enabled** KeyLevelGenerator, to enable **TableLevelGenerator** we have to confirm with Renuka if it is possible to update a whole table at once in the `ChangeApplier` (instead of just keys like it is today). We have to be able to update a table as a whole because within the table there can be internal dependencies between the object, so we have to guarantee all objects are deleted together. For the keys it is guaranteed for the whole key to be updated at once according to the `ChangeApplier`.

**Why non-extendable generators?**
Calling the non-extendable generators precedes the extendable ones, it is like checking for the low hanging fruits. If we use the same extenders for these new generators we will always go up the config tree. Imaging a move that tries to update a key but fails, then we call the extenders on this move. What will happen is the extenders will go up the config tree to the table level, will first try to replace the table, then try to delete the table. Deleting the table is a problem because it affects multiple more than intended and thus violates the minimum disruption guarantee. We decided to leave the extenders only for the LowLevelGenerator thus making sure we try all possible moves from the lowest levels and go up from there.

Another way to look at it is like this:
- Non-extendable generators do top-down search: Check tables, keys in this order
- Extendable generators and extenders do bottom-up approach: Check fields, keys, tables in this order

#### How I did it
Modifying the `MoveWrapper.Generate` method to allow for different type of move generators

#### How to verify it
Unit-test

Please check `tests/generic_config_updater/files/patch_sorter_test_success.json` to see how the moved generated got much compacted.

#### Previous command output (if the output of a command-line utility has changed)

#### New command output (if the output of a command-line utility has changed)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant