[Filebeat] http trace logger - truncate response bodies larger than request.tracer.maxsize

[andrewkroh]

Problem: When using request.tracer.filename in CEL or httpjson, if the logged message is larger than the maximum size allowed for a log file then the message is discarded by Zap.

Proposal: Provide the max size as a hint to the httplog package and when the http.{request,response}.body.content is larger than the max size (minus some safety margin to account for the rest of the log message body) then truncate the content and add a truncation indicator field. The missing response is confusing to developers. So including anything (even if truncated) is better than nothing.

It feels a little hacky to me, but I could not find an easy way to check for errors from the logger and react to that.

[bhapas]

Will it be possible to truncate the content value based on size? Like if http.response.body.content is 3MB , should the truncate happen based on size to cut off less than 1 MB?

[andrewkroh]

Yes, we control the value that goes into the http.response.body.content field. So if we want to truncate it we can do that at

beats/x-pack/filebeat/input/internal/httplog/roundtripper.go

Line 110 in 0f011b0

zap.ByteString("http.response.body.content", body),