Liveliness

[OlivierHecart]

Liveliness

🔬 Unstable

The API, resource keys and value formats defined in this document are unstable. They may change it in a future release.

Context

The ability to check the liveliness of a Zenoh entity (application, publisher, subscriber, etc...) and the ability to be notified of liveliness changes of Zenoh entities (apparition/disappearance of an entity) is a functionnality that is needed in some use cases.

The ability to monitor the liveliness of any entity in the system at any time puts a lot of pressure on the infrastructure. Zenoh needs to anticipate the possibility that any application could monitor the liveliness of any other entity at any time even if this never happens. This will cause huge scalability issues in a large system with a lot of entities.

To offer such functionality without impacting scalability, users should be able to explicitely declare which entities may have their liveliness monitored by other applications or not.

Such a functionality is already offered by the group management feature in zenoh-ext, but the group management feature is quite resource consuming and may also suffer scalability issues. In particular each group member needs to periodically assert it's liveliness by broadcasting a HeartBeat message to all other members of the group.

It should be possible to build a far less resource consuming generic liveliness mechanism in the routing layer that would rely on the existing Transport Session liveliness.

The basics

Any Zenoh application can declare one or multiple liveliness tokens. Each liveliness token is declared on a key expression. A declared liveliness token will be seen as alive by any other Zenoh application in the system that monitors it while the liveliness token is not undeclared or dropped, while the Zenoh application that declared it is alive (didn't stop or crashed) and while the Zenoh application that declared the token has Zenoh connectivity with the Zenoh application that monitors it.

In the case of a network partition, the Zenoh applications that still have connectivity with the Zenoh application that declared the token still see it as alive while the Zenoh applications that loosed connectivity with the Zenoh application that declared the token see it as dropped.

Multiple liveliness tokens on the same key

If one or multiple Zenoh applications declare multiple liveliness tokens on the same key, the token will be seen as alive as asson as the first declaration occurs and will be seen as dropped when the last token is dropped.

Declaring liveliness tokens

A liveliness token can be declared on any key expression with the help of the Session declare_liveliness function:

pub fn declare_liveliness<'a, 'b, TryIntoKeyExpr>(
    &'a self,
    key_expr: TryIntoKeyExpr,
) -> LivelinessTokenBuilder<'a, 'b>;

Example:

let session = zenoh::open(Config::default()).res().unwrap();
let token = session
    .declare_liveliness("group1/member1")
    .res()
    .unwrap();

A liveliness token can be explicitely undeclared or dropped.

Example:

let session = zenoh::open(Config::default()).res().unwrap();
let token = session
    .declare_liveliness("group1/member1")
    .res()
    .unwrap();
token.undeclare();

Querying liveliness tokens

Each alive token can be accessed through queries at any pont in the system with the key prefix @/liveliness. So a token with key group1/member1 is accessible in key @/liveliness/group1/member1 and all alive tokens in the system can be accessed with the following key expression: @/liveliness/**.

Example

Using the z_get example:

$ z_get -s @/liveliness/**
Opening session...
Sending Query '@/liveliness/**'...
>> Received ('@/liveliness/group1/member1': '')

Notifications of liveliness changes

Each time a liveliness token is declared on a given key (assume group1/member1), a put is sent to matching subscribers:

• key: @/liveliness/group1/member1
• kind: Put
• value: empty

Each time a liveliness token is dropped (because the application that declared it stopped, crashed or loosed connectivity), a delete is sent to matching subscribers:

• key: @/liveliness/group1/member1
• kind: Delete
• value: empty

Example

Using the z_sub example:

$ z_sub -k @/liveliness/**
Opening session...
Declaring Subscriber on '@/liveliness/**'...
Enter 'q' to quit...
>> [Subscriber] Received PUT ('@/liveliness/group1/member1': '')
>> [Subscriber] Received DELETE ('@/liveliness/group1/member1': '')

Using the QueryingSubscriber

When monitoring liveliness changes by declaring a Subscriber, the Subscriber only receives changes that occured after it's declaration. In particular, the Subscriber will not see liveliness tokens declared before the Subscriber declaration. In order to access liveliness tokens declared before the Subscriber declaration as well as future connectivity changes, a zenoh-ext QueryingSubscriber can be used.

Example

Using the z_query_sub example:

$ z_query_sub -k @/liveliness/**
Opening session...
Declaring QueryingSubscriber on @/liveliness/** with an initial query on @/liveliness/**
Enter 'd' to issue the query again, or 'q' to quit...
>> [Subscriber] Received PUT ('@/liveliness/group1/member1': '')
>> [Subscriber] Received DELETE ('@/liveliness/group1/member1': '')

Client liveliness Subscribers loosing connectivity

When a client Zenoh application that subscribed to liveliness changes looses connectivity with the system (looses connectivity with it's router and fails to reesatblish connectivity), as it loosed connectivity with all remote applications a user could expect to receive a delete for all existing liveliness tokens. But this implies keeping a state of all tokens in the client itself which is undesirable.

So, in such situation, to indicate to the user that all tokens are dropped, the following delete is sent to matching subscribers:

• key: @/liveliness/**
• kind: Delete
• value: empty

Future improvements

Liveliness tokens associated value

It may be usefull to allow Zenoh applications to attach a value to liveliness tokens they declare. This attached value would be seen by applications that query or subscribe to liveliness tokens/changes.

The LivelinessTokenBuilder would offer a with_value funtion:

pub fn with_value<IntoValue>(mut self, value: IntoValue) -> Self
where IntoValue: Into<Value>;

Example:

let session = zenoh::open(Config::default()).res().unwrap();
let token = session
    .declare_liveliness("group1/member1")
    .with_value("value")
    .res()
    .unwrap();

Multiple associated values on the same key

This rises the question of which value is seen associated to a token when the token has been declared multiple times on the same key with different values. Two behaviors could be offered in such case:

• Undefined behavior: queriers/subscribers will receive any of the associeted values.
• Last value behavior: queriers/subscribers receive the latest associated value. This implies that tokens declaration are timestamped and implies more traffic as each time a token is declared again with the same key it needs to be repropagated in the whole system.

[kydos]

@OlivierHecart, I think it would be nice to also have the ability to support testaments. In other terms a value that is produced automatically by the infrastructure when the lifelines is lost. Have you thought about this?

[kydos]

But you should provide a testament as part of the token registration, in the same style the value is currently provided.

[Mallets]

That's correct. Which brings in some sort of encoding associated with the value, which could trigger different mechanisms either at infrastructure or application level.

[OlivierHecart]

Testaments is something we discussed and the main yet unsolved problem is: who will execute the testament ?

[kydos]

I don't think we should keep state in the infrastructure. The value (and the testament) should be kept on the receiving side and "attached" to the notifications when relevant. In other terms the testament could be provided as the value associated with the delete. Finally, we could use the value to allow for also providing a testament.

[Mallets]

I agree with @kydos .

Thinking better about the problem and various topologies we want to support, it would be very very complex to solve the problem of who sill execute the testament? Especially when system partitioning occurs.

[JEnoch]

Few comments:

1. It feels inconsistent to declare liveliness on group1/member1 and having to query/subscribe on @/liveliness/group1/member1. I understand having a separate data-space for liveliness is nice because it allows for instance to associate a liveliness with a queryable using the same key expression, and without conflict.
   But what about allowing (or forcing) declare_liveliness("@/liveliness/group1/member1") ?

2. Could we offer optional/automatic liveliness for some of the Zenoh primitives. e.g.:

   • let session = zenoh::open(Config::default()).with_liveliness().res().unwrap().
     => automatic liveliness declaration on @/liveliness/<uuid>
   • let q = zenoh.declare_queryable("demo/example/**").with_liveliness().res().await.unwrap()
     => automatic liveliness declaration on @/liveliness/<uuid>/queryable/demo/example/__2STARS__
   • ... same for Subscriber and Publisher

3. Following point 2, it could be useful for debug to activate via configuration the automatic liveliness declaration for all primitives. Thus, no need of code change to check what's declared/alive in a system.

[OlivierHecart]

1. My first feeling is that allowing/forcing declare_liveliness on "@/liveliness/<key>" will cause more problems than it will solve. Any one else has an opinion on this ?
2. , 3. Sure this could be added at the API level.

[Mallets]

1. I fairly dislike the mismatch between the various key expressions for the declarations and subscriptions. Moreover, the fact that we MIGHT access is it via the admin space is quite orthogonal to the problem. I also find that we are overloading a bit too much what the admin space is supposed to do. What I propose is to follow a similar semantics at API for the declare_subscriber and put. E.g.:
   • declare_token_liveness(key_expr) coupled with declare_token_notifier(key_expr)
     I know that the naming is not great, but I'm trying to convey the idea here. Under the hood we could temporarily leverage the admin space to have it work, while we don't expose it to the user.
2. and 3. I'm fine as general idea.

[Mallets]

Here it goes a proposal for the API:

• session.liveness().declare_token(key)
• session.liveness().declare_interest(key)
• session.liveness().get(key)

In this way, all the liveness-related methods are accessible via the liveness() accessor.
This is an API that would be self-contained and semantically well defined.
At the same time, liveness information could be accessed via the admin space on @/liveness/<key>.

[sanri]

That's a good feature.
In my current application, monitoring is achieved through a heartbeat variable.