docs: DR for clustered data-plane

[ndr-brt]

What this PR changes/adds

DR for implementing clustered data-plane.

Why it does that

Briefly state why the change was necessary.

Further notes

List other areas of code that have changed but are not necessarily linked to the main feature. This could be method
signature changes, package declarations, bugs that were encountered and were fixed inline, etc.

Linked Issue(s)

Part of: #4439 #4440

Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.

[jimmarino]

These will be propagated back to the control plane, correct?

[ndr-brt]

these are releated to DataFlow, because SUSPENDING and TERMINATING in the control-plane have different meaning.
only when the DataFlow would been correctly SUSPENDED or TERMINATED the control-plane will be advertised and it would transition the transfer from the SUSPENDING/TERMINATING state to SUSPENDED/TERMINATED

[jimmarino]

Yes, I meant the transition to Dataflow terminated, now that the operation is async. There's two ways to approach this:

1. Have the control plane transition to TERMINATING and wait until the DataFlow is TERMINATED before itself transitioning to TERMINATED
2. Have the control plane send a terminate message to the data plane and immediately transition to TERMINATED once the ack is received.

Appraoch 2 is a lot easier and straightforward without much downside. It does place the burden on the data plane to handle the terminate message correctly or perform some form of cleanup if it fails but that I think is acceptable.

What do you think? Perhaps this should be documented?

[ndr-brt]

I think 2 will be enough because termination of data flow it's just a cleanup operation, generally speaking it should be quick to complete. I will add some details to the DR