Stop following symlinks. #1767
Stop following symlinks. #1767
Conversation
Previously symlinks were followed. This had the consequence if a symlink and the file itself existed, the file was read twice. Now symlinks are not followed anymore. This closes elastic#1686
| @@ -79,11 +79,15 @@ func (p *ProspectorLog) getFiles() map[string]os.FileInfo { | |||
| } | |||
|
|
|||
| // Stat the file, following any symlinks. | |||
There was a problem hiding this comment.
The comment here is out of date, right? os.Lstat doesn't follow symlinks.
There was a problem hiding this comment.
Yes, will remove it.
|
|
||
| if os.name == "nt": | ||
| import win32file | ||
| win32file.CreateSymbolicLink(symlink_file, testfile, 1) |
There was a problem hiding this comment.
The link points to a file so the last parameter should be 0. https://msdn.microsoft.com/en-us/library/windows/desktop/aa363866(v=vs.85).aspx
Hopefully that will fix the issue with 7z on AppVeyor.
|
Can we add the option for a fail fast option so that if the path provided in the config is a symlink, that it fails to startup so that way a person knows right away of the issue? |
|
Alternatively why not default to it being disabled but allowing it to be enabled if one chooses. |
|
@djschny The plan was to introduce a config option as soon as we have a request for it as we were not sure if this is something even needed. See #1686 (comment) Is following symlink something you need? About the "fail fast". This is a little bit more complex as the symlink could also appear when Filebeat is already running and there is not difference in "first scan" of file and all the follow up scans. I prefer here the config option as I think it is more predictable and is closer to the principle we have so far. If one file is not readable, all the other files are still read. |
|
Cool, thanks @ruflin, I missed that comment linked issue. Sounds good. In regards to "fail fast" I don't think people would care much about the worry about if it changed "out from under" Filebeat, but if you don't have startup check code in place, I can understand how this would be a heavier implementation than what may be viewed externally. Thanks for fielding my comment. |
This implementation of symlinks takes a symlink and opens the original file. The only difference is that as Source the symlink path is reported. The advantage of this implementation is that everything related to file handling is identical with non symlink files as the original file is harvested. All close_* options work as expected. State information is stored for the original file, not the symlink itself. In case a symlink and original file are harvested, only one will be harvested as they share the same inode information. * Add test to verify that symlinks are disabled by default * Add test for symlink handling * Improve error handling in harvester. Return proper error messages instead of logging it. Prevents too many log messages. * Remove IsRegular file call as not needed anymore and leads to additional Stat calls * Add documentation See elastic#2277 and elastic#1767
This implementation of symlinks takes a symlink and opens the original file. The only difference is that as Source the symlink path is reported. The advantage of this implementation is that everything related to file handling is identical with non symlink files as the original file is harvested. All close_* options work as expected. State information is stored for the original file, not the symlink itself. In case a symlink and original file are harvested, only one will be harvested as they share the same inode information. * Add test to verify that symlinks are disabled by default * Add test for symlink handling * Improve error handling in harvester. Return proper error messages instead of logging it. Prevents too many log messages. * Remove IsRegular file call as not needed anymore and leads to additional Stat calls * Add documentation See #2277 and #1767
This implementation of symlinks takes a symlink and opens the original file. The only difference is that as Source the symlink path is reported. The advantage of this implementation is that everything related to file handling is identical with non symlink files as the original file is harvested. All close_* options work as expected. State information is stored for the original file, not the symlink itself. In case a symlink and original file are harvested, only one will be harvested as they share the same inode information. * Add test to verify that symlinks are disabled by default * Add test for symlink handling * Improve error handling in harvester. Return proper error messages instead of logging it. Prevents too many log messages. * Remove IsRegular file call as not needed anymore and leads to additional Stat calls * Add documentation See elastic#2277 and elastic#1767
Previously symlinks were followed. This had the consequence if a symlink and the file itself existed, the file was read twice. Now symlinks are not followed anymore.
This closes #1686