Introduce close_eof harvster option #1914
Conversation
|
|
||
| # Closes the file handler when the end of the file is reached. This is useful for files where | ||
| # content is only written to once. In case further content is added to the file later again | ||
| # it will be picked up with the previous state latest after scan_frequency. |
There was a problem hiding this comment.
In case reading goes faster then writing, Isn't there always a chance of hitting EOF?
There was a problem hiding this comment.
Yes. But I assume for files with 1 event, the hole event is flushed at once.
There was a problem hiding this comment.
Perhaps I should not make an example here in the config file and only explain what it does. The example should be in the docs pages.
There was a problem hiding this comment.
I changed it to:
# Closes the file handler as soon as the harvesters reaches then end of the file.
# The file will be picked up again by the harvester at previous known state
# after scan_frequency in case the file can still be discovered by the prospector.
#close_eof: false
There was a problem hiding this comment.
Maybe add note of potential data loss if file is deleted within close and scan_frequency
There was a problem hiding this comment.
Will add it in #1915
|
@urso Changes applied and rebased. |
|
jenkins, test it |
|
@urso I triggered this build manually as Jenkins didn't want to start it initially: http://build-eu-00.elastic.co/job/Beats-PR/2217/console I will ping you when green. |
| @@ -176,6 +176,11 @@ filebeat.prospectors: | |||
| # Note: Potential data loss if file reading was not finished when file was removed. | |||
| #close_removed: false | |||
|
|
|||
| # Closes the file handler as soon as the harvesters reaches then end of the file. | |||
There was a problem hiding this comment.
@ruflin needs rephrase: Closes the file handler as soon as the harvester reaches end of the file
Depends on #1909