Updating from remote master #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The stop-environment command now also removes volumes from contains and potential contaiiners which were only run once. This should eliminiate the tangling volume issue on Jenkins.
The new structure was introduced but the generator was not updated yet
Add fixed version numbers where releases exist
To build the docs, `make docs` can be used. To preview the docs, `make docs-preview` can be used. * Building the docs is now part of make testsuite * Make local docs build and preview easy by adding it to the makefile * Update CONTRIBUTING.md with new commands The generators are currently not part of the build as they require glide to be installed.
Logstash output to adapt window sizes faster by growing/shrinking windows per windowed batch being pushed instead of full batch.
Currently file truncation was handled by the reader. In case of truncation, the same harvester stayed open and the reader just continued reading from the beginning. This is the most efficient way to make sure the truncated file is immediately read. From a logical point of view, truncation of a file is very similar to creating a new file. Instead of letting the reader just continue, the harvester is closed and a completely new harvester is started with a new reader. This means the reader only needs to detect a truncation, but doesnt' have to decide what happens. The consequence of this change is that it can take up to `scan_frequency` to read the truncated file, but that is the same behaviour as for a new file. In addition the file handler will be closed when the old harvester is stopped.
Was "To make made it even ..." before.
…beats (#1883) During implementation of Travis CI for community beat I found out that some of the generated paths and options don't work for the beat. - make command doesn't need directory setting (-C directory) because we talk about single beat - paths for travis setup require beat paths and name and not 'beats' paths
Update dependencies to most recent version
Start a new harvester when file was truncated
This scripts removes the unnecessary files from the vendor directory. As we check in the vendor directory the goal is to keep the number of files to a minimum and if changes are made to the dependency make sure that it is easy to review the relevant changes.
When reading events from the ForwardedEvents log, Winlogbeat was always attempting to render the event locally event using messages files on the collector. This was wrong because the collector may not have the same message files as the source machine. It would result in a message_error if the collector machine was missing the application that originally logged the event on the source machine. This PR changes Winlogbeat to not attempt to render the event on the collector machine, but instead use the event as is. This requires that the subscription for the Windows event collector uses the "RenderedText" format which includes the message string and any description strings with the forwarded event.
* Rename shipper in docs * Remove runoptions doc from all beats except Packetbeat
Script for cleaning up vendor directory
* Update testcases to test timestamps with timezone specified This is the default @timestamp format used by kibana json log for example * Make timestamp conversation based on RFC3339 Allow timestamps without timezone. See #1831 for more details
This should remove the confusion that topbeat was still in the beats repo but no further development in master * We must make sure the docbuild still works. 1.x docs must stay around * Remove references in docs * Remove build file * Remove topbeat from doc examples * Move system tests from topbeat to system module. Remove obsolete tests. TBD: Update dev guide to remove Topbeat
* Structural changes * Move exported fields lower in the nav * Change order of topics in config section * Add config file name to each reference topic to set context * Move runconfig.asciidoc to packetbeat folder
NFS4 compound operations can be constructed from multiple single requests. Nevertheless, each compound request as a main operation. current code failing to detect main operation if it was in the middle of compound call. Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>
The overall goal is to decouple all the modules to make them better reusable and extendable. In addition the current organisation was not very intuitive. * Move registrar to its own package * Improve startup handling of filebeat * Rename state to states in registrar as this is more accurate * Add warn message for truncated files. See https://github.com/elastic/beats/pull/1882/files#r67523587 * Move ignore older to ProspectorLog as only used there * Remove Prospector reference from ProspectorStdin * Remove empty filebeat test file * Cleanup New function naming * Implement defer statements to stop running services * Clean up crawler / prospector stopping
Improve Filebeat organisiation and Cleanup
* Disable tests which use _ttl * Temporarly disable user / password for LS build because it breaks the build
Update dependencies to alpha4
* Cleanup defaults by moving them to config * Move readers and sources into their own packages * Add some minor comments * Move state to its own package * Move multiline and json to its own file * Rename FileState to State as now package name is included * Adjust test cases * Move multiline config to processor * Move json to processor * Separate limit and line processor * Improve error handling in registrar
* Fix failing test because of ucfg change * Add make env-logs to easier see environment logging information * Some command cleanup
Cleanup Harvester and Input parts of Filebeat
Update mysql version dependency to 5.7.12
Fix Metricbeat field names for redis aof
Fix for fail to remove state where file was deleted and clean_removed was set.
Forward ported fix from #2062
See #1913 * Make sure ignore_older is not only applied to new files but also existing files * Improve logging in prospector * Add expvar to prospector and harvester * Cleanup offset handling * Add truncation handling to prospector * Handle all offsets as part of the state * Remove path from harvester and use state.Source instead * Remove obsolete exclude / include options * Remove unnecessary getOffset() * Fix potential file handler leak. Close file handler in case of errors. * Implement cleaner file handler closing * Cleanup initOffset handling * Add build-image task to generate json in metricbeat as it was missing * Only send state if file was properly initialised * Only set offset if seek succeeds
* Always add conditionals to all processor actions processor registry automatically adds conditional support to registered actions by wrapping processor constructors with NewConditional. * filebeat: Update processors system tests * packetbeat: Update processors system tests * metricbeat: Update processors system tests
Add Unpack method to EventFormatString and MustCompile, in order to make EventFormatString usable with `config.Unpack(...)`
EventFormatString unpack for use with configurations
The "index" setting name has historical reasons, from the times that all outputs shared the same configuration options. Since the meaning and contents of the "index" setting are starting to vary per output, it's time to clean this up. The Redis "index" is now called "key". Before this change, the index was still somehow special, since it was set to the beatName outside of the output. This removes this hack, and sets index to beatName only in the Elasticsearch and Logstash modules. This also removes the `index` setting for the file output, but that was never documented. Part of #2074.
Add date formatter to libbeat/common/dtfmt. Pattern syntax supported is subset of joda DateTimeFormat
* Rename all processors to readers * Rename line struct to Message * Remove "reader" from all structs inside reader package No logic changes are done in this change.
Add timestamp formatting to EventStringFormatter
The change modifies the unmarshaling logic to try parsing the numbers in an int64 first, and only on error parse to a float64. In practice, this means that e.g. 1 will become an int64 and 1.0 will become a float64. Fixes #2038. Includes a system test to verify that ticket.
* Rename FileEvent to Event * Remove double information between FileEvent and State (Source, Fileinfo) * Remove readLine function as not needed anymore
Do not store evaluation context in formatter, but allocate context via shared sync.Pool.
Empty lines should not be sent as an event. To make sure this does not happen a test was added. This was probably broken in #2090
Currently it was possible, that a state was cleaned up also if a harvester was not finished yet. Now a state will not be cleaned up as long as a harvester is still running.
* Add ignore_older check * Add clean_inactive check
* Add dashboards for system module * Reduce spaces between widgets * Run make collect * Add dashboards for filesystem, filesystem per host, and overview for the system module
= State Removal With a very low scan_frequency it could potentially happen, that a state of a finished harvester was overwritten by the prospector which lead to the problem, that the state was never set to Finished. This is now fixed in that the prospector only sends a state when the state is set to Finished. = Sending empty lines Empty lines were still sent as the byte check was set to 0 and with the newline char it was always at least 1. The tests passed because filebeat was shutdown before the last line was written. Tests were now improved to tests both cases.
Add alpha5 build, remove alpha4.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.