[Filebeat] offset points to end of line

[andrewkroh]

The Filebeat documentation defines the offset field as

The file offset the reported line starts at.

But it appears that the offset is actually pointing to the end of the line. For example, when reading the first line of the file you would expect offset: 0, but this is not the case (tested with 6.2.2).

Should the docs or the code be changed?

[ruflin]

It seems we accidentially introduced this change in the 5.x releases. I tried 1.x where it's still as documented, but in 5.6 and 6.x it is as you reported. It probably happened because the offset in the registry needs to be at the end of the line and we move the calculation.

I'm thinking having the offset reported as the beginning of the line feels more natural from a user perspective. But as so far nobody complained I wonder if it matters?

[andrewkroh]

I'm thinking having the offset reported as the beginning of the line feels more natural from a user perspective.

I totally agree. I find having the offset point to the end of the line misleading.

This came up while I was helper a user debug a json decoding issue, and we used the offset to determine where the line began in the log file and the offset became a point of confusion.

Additionally, seeing offset: 0 would be a clear indicator that rotation occurred.

[ruflin]

The problem lies here:

beats/filebeat/input/log/harvester.go

Line 258 in bb6d3d8

state.Offset += int64(message.Bytes)

In filebeat 1.x this happened after creating the event / fields data. Now it happens afterwards:

beats/filebeat/input/log/harvester.go

Line 272 in bb6d3d8

"offset": state.Offset, // Offset here is the offset before the starting char.

Even the code comments are incorrect :-( A fix should be pretty simple by temporarely copy the previous offset and then use it for the event creation.

[dedemorton]

@ruflin So it sounds like a code (not doc) fix is required here? if so, maybe remove the doc label?

[ruflin]

@dedemorton Yes, removed.