Allow filebeat to only run once #2456
Conversation
ruflin
added
in progress
| if *once { | ||
| logp.Debug("filebeat", "Running filebeat once. Waiting for completion ...") | ||
| spooler.Flush() | ||
| publisher.Publish() |
There was a problem hiding this comment.
@urso What I'm trying to do here is to make sure that all events which are in the queue are sent. Is there a better way to wait / check the publisher if all events were sent?
* Separate async and sync publisher into own files * Add Publish interface method which can be used to manually trigger publish (see elastic#2456) * Add getDataEvents function * Rename publish to publisher package
* Separate async and sync publisher into own files * Add Publish interface method which can be used to manually trigger publish (see #2456) * Add getDataEvents function * Rename publish to publisher package
This metric can be useful in combination with elastic#2456 to track how long the harvesting of a file took. Uptime is reported before shutdown.
This metric can be useful in combination with #2456 to track how long the harvesting of a file took. Uptime is reported before shutdown.
ruflin
force-pushed
the
run-once
branch
2 times, most recently
from
6830490 to
dfd46c9
Compare
| var once *bool | ||
|
|
||
| func init() { | ||
| once = flag.Bool("once", false, "Run filebeat only once until all harvesters reach EOF") |
ruflin
force-pushed
the
run-once
branch
2 times, most recently
from
690a9a9 to
41e9c9b
Compare
| wgEvents = &sync.WaitGroup{} | ||
| finishedLogger = newFinishedLogger(wgEvents) | ||
| } | ||
| //if fb.config.ShutdownTimeout > 0 || *once { |
There was a problem hiding this comment.
@urso do you see an issue with this? As long as we don't use wgEvents nothing will happen.
There was a problem hiding this comment.
it's correct. as long as wgEvent.Wait is not installed in sigWait there is no difference in initializing these always. I just didn't want to have all the wgEvent update by default, but overhead should be ok.
There was a problem hiding this comment.
Will leave it in as default for the moment and we can still make it optional in case it becomes a performance issue (I would not assume so)
| } | ||
|
|
||
| fb.sigWait.AddChan(fb.done) | ||
| fb.sigWait.Wait() |
There was a problem hiding this comment.
let's use a separate signalWait waiting for runOnce or fb.done to finished. This way the waiter waiting for shutdown and the one waiting for events published will not get mixed up. The fb.done channel can be installed multiple times. With channel being closed, it will send a signal immediately.
ruflin
force-pushed
the
run-once
branch
3 times, most recently
from
1593f58 to
a8a7100
Compare
When the `-once` flag is used, filebeat starts all configured harvesters and prospectors and runs each prospector until the harvesters are closed. It is recommended to use the flag in combination with `close_eof` so harvester directly close when the end of the file is reached. By default harvesters are closed after `close_inactive`.
| logp.Info("Shutdown output timer started. Waiting for max %v.", timeout) | ||
| waitEvents.Add(withLog(waitDuration(timeout), | ||
| "Continue shutdown: Time out waiting for events being published.")) | ||
| } |
There was a problem hiding this comment.
Add:
} else {
waitEvents.AddChan(fb.done)
}
This will ensure CTRL-C will be handled immediately/properly if run-once without shutdown_timeout is configured. If fb.done has been closed already, waitEvents.Wait will return immediately. If fb.done is closed while waiting for wgEvents.Wait, shutdown will continue.
When the
-onceflag is used, filebeat starts all configured harvesters and prospectors and runs each prospector until the harvesters are closed. It is recommended to use the flag in combination withclose_eofso harvester directly close when the end of the file is reached. By default harvesters are closed afterclose_inactive.Closes #880