Payload logging/events

[njhill]

For various reasons including monitoring by external system for things like drift / outlier detection etc.

It should support CloudEvents and be compatible with the logger in KServe "classic", so that it can be used in a similar way, as illustrated in these samples:

• https://github.com/kserve/kserve/tree/master/docs/samples/logger/basic
• https://github.com/kserve/kserve/tree/master/docs/samples/drift-detection/alibi-detect/cifar10 / https://github.com/kserve/kserve/tree/master/docs/samples/outlier-detection/alibi-detect/cifar10

Some considerations / possible complications:

• In KServe the logger can be configured per InferenceService. We need to decide whether we support this with model-mesh, or a simpler global configuration, or both. Another possibility could be allowing a logging destination to be configured globally and enabled/disabled per model.
• Model-mesh doesn't really touch the payloads currently, but it only routes gRPC/protobuf. So we could emit the raw protobuf messages but this would differ from the existing KServe case and so would not necessarily be compatible with the same integrations. We could transcode to json on the fly, but this would introduce processing overhead that may be undesirable and affect data path performance.
• The KServe examples are based on the V1 API, we should to check whether the existing logger works with the V2 API; the runtimes supported by model-mesh are primarily V2 based.

cc @rafvasq

[lizzzcai]

In KServe the logger can be configured per InferenceService. We need to decide whether we support this with model-mesh, or a simpler global configuration, or both. Another possibility could be allowing a logging destination to be configured globally and enabled/disabled per model.

Hi @njhill , what is the definition of payload logging here, does it include the logs in the runtime level? If not, is it possible to support it like adding a model_id identifier in the log line so that we can filter it per inferenceSerivce.

Currently we plan to expose the logs (runtime logs) on the servingRuntime level (actual deployment) as the inferenceService (model) is kind of a logical concept in modelmesh. However, in standard KServe flow, we expose the logs per inferenceService. If logging per inferenceService is possible in ModelMesh, it will be good to align the user experience. Thanks.

[njhill]

@lizzzcai noticing your question here after a long time, sorry about that.

This issue isn't about program logs, rather about the ability to send the contents of every request and response somewhere configurable (so that it can be logged/stored/analysed as needed).

[ckadner]

I was just revisiting some features in the ModelMesh v0.11.0 release to draft a release blog.

Re: CloudEvents ...

It should support CloudEvents and be compatible with the logger in KServe "classic", so that it can be used in a similar way, as illustrated in these samples: ...

As best as I can tell, this is not part of the implementation on kserve/modelmesh#84 but it could now be implemented as one new PayloadProcessor that emits CloudEvents?

@tteofili @njhill