⚠️ RSDK-7903 Include unhealthy remotes in resource list with status

[maximpertsov]

⚠️ hold off on reviewing for now - the following issue needs to be addressed first

• https://github.com/viamrobotics/rdk/pull/4399/files#r1783017466

---

Summary

Keep remote resources in the ResourceNames and MachineStatus APIs when their remote gets disconnected. In the latter API, mark such resources with a state indicate that they are disconnected.

Prerequisites

To make reviews a bit easier, this PR is going to broken out into smaller chunks:

• ⚠️ RSDK-7903 Cache machine status in robot client #4399

Details

Robot clients now fetch and cache resource statuses via the GetMachineStatus API instead of resource names via ResourceNames API.

Open questions

• Should we add a new state for disconnected nodes? (e.g. NodeStateDisconnected), or is it sufficient to reuse the unhealthy state with a corresponding error (e.g. NodeStateUnhealthy[reason:disconnected])?
  • FWIW, I slightly prefer a new state given the proposed implementation, since a healthy but disconnected remote resource can appear as "disconnected" or "ready" depending on whether a client calls MachineStatus on the local or remote machine, respectively. Having a specialized state that can only appear for remote resources feels like the less confusing option in this case.
• Shall we bubble up remote update errors? (see this comment and this comment) - this will make the PR much noisier but might be the correct thing to do.

Depends on

• Expose unhealthy state + error: RSDK-8595 Unhealthy resource state #4257
  • Note that those changes are included in this PR as well - see this diff for changes excluding the dependent PR.
• RSDK-7843: Do not remove resources when a remote has a transient error. #4268 - specifically include the unit test from the PR since we want to satisfy the same invariant here.

[maximpertsov]

An alternative approach to the changes in this file is to define a new state (e.g. NodeStateDisconnected) - see the "Open questions" section in the PR description for more details and see the revert of this commit for a general idea of how this would look.

[maximpertsov]

the high-level change here is that a robot client now caches the results of MachineStatus rather than just ResourceNames. As a result, multiple tests that result on injecting a mock function for ResourceNames now need to mock MachineStatus.

[maximpertsov]

An alternative to defining a new state would be to mark disconnected resource nodes as "unhealthy" with an appropriate error message. See "Open questions" in the description of this PR for more details and revert this commit to see how this alternative approach would look.

[dgottlieb]

I hate to be that person, but is there any chance to split this PR up into 2 (or 3?) commits?

1. Code change to make resource names work
2. Code change to make MachineStatus work
3. Refactor changing client.resourceNames -> client.cachedMachineStatus structures

Or maybe 2 and 3 are the same and its only two commits.

I feel confident reviewing "commit 1", but not useful for the MachineStatus changes. And I know commit 1 really just changes 1/2 files + test files.

I'm also happy to step back if Cheuk/Benji are confident being the stakeholder for (1)

[maximpertsov]

I hate to be that person, but is there any chance to split this PR up into 2 (or 3?) commits?

1. Code change to make resource names work

2. Code change to make MachineStatus work

3. Refactor changing client.resourceNames -> client.cachedMachineStatus structures

Or maybe 2 and 3 are the same and its only two commits.

I feel confident reviewing "commit 1", but not useful for the MachineStatus changes. And I know commit 1 really just changes 1/2 files + test files.

I'm also happy to step back if Cheuk/Benji are confident being the stakeholder for (1)

This is a reasonable request, I was considering making the client cache change as it's own no-op PR anyway.

[maximpertsov]

3. Refactor changing client.resourceNames -> client.cachedMachineStatus structures

@dgottlieb @benjirewis @cheukt

#4399

[dgottlieb]

IIUC, in the resource graph we have:

• A node for each remote robot. Where the underlying resource is the robot client
• A node for each resource on a robot robot. Where the underlying resource is the component client (e.g: camera client).

Does this SetDisconnected state only apply to nodes representing the remote robot client in the resource graph? Or does it also apply to the remote resource objects in the resource graph?

Second:
And when a remote goes from working -> disconnected -> working, what state transition will that take? Is it ready -> disconnected -> ready? Or do we jump through any of the unconfigured/configuring states?

I think the answers would work well in the documentation.

[maximpertsov]

Does this SetDisconnected state only apply to nodes representing the remote robot client in the resource graph? Or does it also apply to the remote resource objects in the resource graph?

it applies to the remote resource objects. also note that resources representing full remote machine clients are excluded from the MachineStatus endpoint.

And when a remote goes from working -> disconnected -> working, what state transition will that take? Is it ready -> disconnected -> ready? Or do we jump through any of the unconfigured/configuring states?

thanks for drawing attention to this. the flow I want is for disconnected nodes to go directly back into the ready state (unless they are unhealthy for some other reason). however, I don't think the current logic actually transitions disconnected nodes back to ready, so let me address that.

I think the answers would work well in the documentation.

will do

[dgottlieb]

Wait, are you changing robots to use machine status instead of resource names to populate remote resources?