WIP: initial EFK setup

[ryandawsonuk]

Part of #545

Able to run EFK and find log entries in kibana such as:

But need to consider how we'd (optionally) log the full request and how the content of the request would appear in kibana. Clearly we see full log and search by timestamp already. Would we need to be able to search by a feature value?

[ukclivecox]

Cool. I think by feature might be difficult but we do have a "names" field in the "default_data" which can give columns names. The core would be to get the request back and be able to load it as a SeldonMessage for retransimisison

[ryandawsonuk]

Initial attempt at request-response logging logs first SeldonMessage.toString and then full request body. Comes out like:

2019-05-31 15:04:15.151  INFO 1 --- [nio-8000-exec-2] i.s.e.api.rest.RestClientController      : {"data": {"names": ["f0", "f1"], "ndarray": [[0.12, 0.16]]}}
2019-05-31 15:04:15.157  INFO 1 --- [nio-8000-exec-2] i.s.e.api.rest.RestClientController      : {
  "meta": {
    "puid": "cecr3j2pl3mie72sohhbait7ti",
    "tags": {
    },
    "routing": {
    },
    "requestPath": {
      "classifier": "seldonio/mock_classifier:1.0"
    },
    "metrics": []
  },
  "data": {
    "names": ["proba"],
    "ndarray": [[0.05859816662313229]]
  }
}

Doesn't look great in kibana presently as it's multi-line and each line of logging is a separate entry. I think I need to figure out how to format it as single line and then figure out how to search for something more specific.

[ryandawsonuk]

It seems it is possible to get the logging to be parsed as structured, I think for that the log line might need to be entirely json and single-line.

We could do the logging in a general way in the java engine using a contextHandler but we might be better doing it explicitly as then we can deserialize the grpc.

[ryandawsonuk]

It looks better as single-line json but still not actually structured:

[ryandawsonuk]

Actually looking at it in the tabular view, it seems it has been parsed:

[ryandawsonuk]

Now able to filter on particular values from the request/response. Next step is to enrich the messages to indicate which is a request and which is a response. Going to do this in the PredictionService as there the request is enriched with puid and if we log that version then we'll be able to search for meta.puid to see linked request-response-feedback entries.

Note for the future - should we parse out the data and names entries and flatten them into a fieldname: fieldvalue format in the json? That would make them much easier to search for. Currently I can only see how to easily search for requests that contain a feature name or feature value but not how to search for a particular feature having a particular value.

[ryandawsonuk]

Now have puid in request and response.

Not sure about how we could log data in a nicely queryable way without running up against array querying limitations.

We could look at doing some custom parsing/transformation with fluentbit but this might be more complex than it's intended for. Other options might be if there's a simple way to just serialize the message differently or send the request to another agent to transform (e.g. through a message broker).

[ryandawsonuk]

Considering alternative idea of transforming data under #616

[ryandawsonuk]

Closing this in favour of #616

[seldondev]

failed to trigger Pull Request pipeline

• failed to create agent
• failed to calculate in repo config
• failed to load trigger config for repository SeldonIO/seldon-core for ref a9c51fb
• failed to find any lighthouse configuration files in repo SeldonIO/seldon-core at sha a9c51fb
• failed to process repo SeldonIO/seldon-core refref a9c51fb
• failed to list files in directory /var/tmp/gitrepo283418320/.lighthouse
• open /var/tmp/gitrepo283418320/.lighthouse
• no such file or directory