Simplify healing - determinate reselect state for whole chain

[denis-tingaikin]

Steps to reproduce

1. deploy nsm
2. deploy nse1 with service ns1
3. deploy nse2 with service ns1
4. deploy nsc
5. nsc connects to nse1
6. nse1 dies
7. nsc reselects nse
8. client's nsmgr misses close from reselect

Expected Behavior

All downstream components correctly handle reselect on next request from the nsc.

Current Behavior

All downstream components do something non-deterministic. (i.e. some component like forwarder may try to restore connection, some component may just refresh the connection)

Context

See at

sdk/pkg/networkservice/common/heal/eventloop.go

Line 131 in 6fa2f68

reselect := false

See at

sdk/pkg/networkservice/common/begin/event_factory.go

Line 102 in 6fa2f68

if o.reselect {

Initial ideas

1. Remove here path

   sdk/pkg/networkservice/common/begin/event_factory.go

   Line 107 in 6fa2f68

   request.GetConnection().Mechanism = nil

   and then all downstream components will think that it is a new connection
2. Add something into request proto to determidate reselect state by downstream components

Defenition of done

• Investigate the problem and suggest options on resolve the problem with pros/cons.
• Get approve from @edwarnicke
• Implement the solution.
• Pass linters/code review ~2h
• Pass integration tests. ~14h

[d-uzlov]

Current behavior

If Close reaches all apps in path, it clears all caches, and reselect works as expected.

If Close doesn't reach some apps, those apps try to restore old connection to avoid resource leaks.
Therefore, reselect doesn't work.

Solution 1

Changes:

• Clear request path.

When path is empty, updatePath will create new IDs.
All the caches that prevent reselect from working are linked to path segment ID.
With new ID we are guaranteed to get reselect.

Pros:

• Very simple
• Fully backward-compatible

Cons:

• The only way to clear leaked resources is using timeout

Solution 2 (Best solution)

Changes:

• Create a new state of connection: RESELECT_REQUESTED
  • https://github.com/networkservicemesh/api/blob/5d372416a0b4b0d70d37a801e6d24fe37cf5c432/pkg/api/networkservice/connection.proto#L33-L37
• Set this state in event factory when using reselect
• Add a new chain element, reselectcleanup, that will request Close if it sees RESELECT_REQUESTED on an existing connection

If Close from client has reached all apps in the connection path, everything will work as before.
If local Nsmgr has restarted, it's likely we go to the same forwarder, and forwarder will close the old connection.
If local forwarder has restarted, but we go to the same remote node, the remote forwarder or nsmgr will clear resources.

It is still possible to have resource leaks if we can't reach some apps in the path.
But this solutions will fix many common cases.

Pros:

• "Best effort" resource cleanup
• Still simple

Cons:

• A small change in API

Example when NSMgr restarted:

[denis-tingaikin]

Seems like done