From d4629f65ff631690842b10df02be781c6256c4ea Mon Sep 17 00:00:00 2001 From: Filip Hajny Date: Wed, 30 Nov 2016 22:17:10 +0000 Subject: [PATCH] Update beats to 5.0.2. Beats version 5.0.2 Metricbeat - Fix the `password` option in the MongoDB module. Filebeat - Fix registry cleanup issue when files falling under ignore_older after restart. Beats version 5.0.1 Metricbeat - Fix system.process.start_time on Windows. - Fix system.process.ppid on Windows. - Fix system process metricset for Windows XP and 2003. cmdline will be unavailable. - Fix access denied issues in system process metricset by enabling SeDebugPrivilege on Windows. - Fix system diskio metricset for Windows XP and 2003. - Add username and password config options to the PostgreSQL module. - Add username and password config options to the MongoDB module. - Add system core metricset for Windows. Packetbeat - Fix 'index out of bounds' bug in Packetbeat DNS protocol plugin. - Define client_geoip.location as geo_point in the mappings to be used by the GeoIP processor in the Ingest Node pipeline. Filebeat - Stop Filebeat on registrar loading error. Beats version 5.0.0-GA Affecting all Beats - Fix kafka output re-trying batches with too large events. - Fix kafka output protocol error if version: 0.10 is configured. - Fix kafka output connection closed by broker on SASL/PLAIN. - Add beat.version fields to all events. Metricbeat - Fix high CPU usage on macOS when encountering processes with long command lines. - Fix high value of system.memory.actual.free and system.memory.actual.used. - Change several OpenProcess calls on Windows to request the lowest possible access provilege. - Fix system.memory.actual.free high value on Windows. Filebeat - Fix issue when clean_removed and clean_inactive were used together that states were not directly removed from the registry. - Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. Beats version 5.0.0-rc1 Affecting all Beats - A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. - Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. - Fix ignoring all fields from drop_fields in case the first field is unknown. - Fix dynamic configuration int/uint to float type conversion. - Fix primitive types conversion if values are read from environment variables. Metricbeat - Fix default configuration file on Windows to not enabled the load metricset. Packetbeat - Fix the bpf_filter setting. Filebeat - Fix input buffer on encoding problem. - Fix registry cleanup issue when files falling under ignore_older after restart. Beats version 5.0.0-beta1 Affecting all Beats - Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. - Replace output.kafka.use_type by output.kafka.topic accepting a format string. - If the path specified by the -c flag is not absolute and -path.config is not specified, it is considered relative to the current working directory. - rename tls configurations section to ssl. - rename certificate_key configuration to key. - replace tls.insecure with ssl.verification_mode setting. - replace tls.min/max_version with ssl.supported_protocols setting requiring full protocol name. - Fix logstash output handles error twice when asynchronous sending fails. - Fix Elasticsearch structured error response parsing error. - Fixed the run script to allow the overriding of the configuration file. - Fix logstash output crash if no hosts are configured. - Fix array value support in -E CLI flag. - Fix merging array values if -c CLI flag is used multiple times. - Fix beats failing to start due to invalid duplicate key error in configuration file. - Fix panic on non writable logging directory. - Add script to generate the Kibana index-pattern from fields.yml. - Enhance Redis output key selection based on format string. - Configurable Redis keys using filters and format strings. - Add format string support to output.kafka.topic. - Add output.kafka.topics for more advanced kafka topic selection per event. - Add support for Kafka 0.10. - Add SASL/PLAIN authentication support to kafka output. - Make Kafka metadata update configurable. - Add Kafka version setting (optional) enabling kafka broker version support. - Add Kafka message timestamp if at least version 0.10 is configured. - Add configurable Kafka event key setting. - Add settings for configuring the kafka partitioning strategy. - Add partitioner settings reachable_only to ignore partitions not reachable by network. - Enhance contains condition to work on fields that are arrays of strings. - Lookup the configuration file relative to the -path.config CLI flag. - Re-write import_dashboards.sh in Golang. - Update to Go 1.7. - Log total non-zero internal metrics on shutdown. - Add support for encrypted private key files by introducing ssl.key_passphrase setting. - Add experimental symlink support with symlinks config - Topology map is deprecated. This applies to the settings: refresh_topology_freq, topology_expire, save_topology, host_topology, password_topology, db_topology. Metricbeat - Change field type system.process.cpu.start_time from keyword to date. - redis/info metricset fields were renamed up according to the naming conventions. - Fix module filters to work properly with drop_event filter. - Use the new scaled_float Elasticsearch type for the percentage values. - Add experimental cgroup metrics to the system/process MetricSet. - Added a PostgreSQL module. - Improve mapping by converting half_float to scaled_float and integers to long. - Add experimental haproxy module. Packetbeat - Group HTTP fields under http.request and http.response - Export http.request.body and http.response.body when configured under include_body_for - Move ignore_outgoing config to packetbeat.ignore_outgoing - Fix mapping for some Packetbeat flow metrics that were not marked as being longs. - Fix handling of messages larger than the maximum message size (10MB). - Add Cassandra protocol analyzer to Packetbeat. - Match connections with IPv6 addresses to processes - Add IP address to -devices command output - Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. Filebeat - Set close_inactive default to 5 minutes (was 1 hour before) - Set clean_removed and close_removed to true by default - Fix processor failure in Filebeat when using regex, contain, or equals with the message field. - Fix async publisher sending empty events - Fix potential issue with multiple harvester per file on large file numbers or slow output - Introduce close_timeout harvester options - Strip BOM from first message in case of BOM files - Add harvester_limit option Beats version 5.0.0-alpha5 Affecting all Beats - Rename the filters section to processors. - Introduce the condition with when in the processor configuration. - The Elasticsearch template is now loaded by default. - The Redis output index setting is renamed to key. index still works but it's deprecated. - The undocumented file output index setting was removed. Use filename instead. - Fix sync publisher PublishEvents return value if client is closed concurrently. - Periodically log internal metrics. - Add enabled setting to all output modules. - Command line flag -c can be used multiple times. - Add OR/AND/NOT to the condition associated with the processors. - Add -E CLI flag for overwriting single config options via command line. - Choose the mapping template file based on the Elasticsearch version. - Check stdout being available when console output is configured. Metricbeat - Create a separate metricSet for load under the system module and remove load information from CPU stats. - Add system.load.norm.1, system.load.norm.5 and system.load.norm.15. - Add threads fields to mysql module. - Do not send zero values when no value was present in the source. - Add pgid field to process information. https://github.com/elastic/beats/pull/ 2021[2021] Packetbeat - Set enabled ` in packetbeat.protocols.icmp configuration to true by default. - Add enabled setting to Packetbeat protocols. - Add enabled setting to Packetbeat network flows configuration. Filebeat - Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. - Fix open file handler issue. 2028 - Fix filtering of JSON events when using integers in conditions. - Introduce close_removed and close_renamed harvester options. - Introduce close_eof harvester option. - Add clean_removed and clean_inactive config option. - Deprecate close_older option and replace it with close_inactive. - Deprecate force_close_files option and replace it with close_removed and close_renamed. Beats version 5.0.0-alpha4 Affecting all Beats - The topology_expire option of the Elasticserach output was removed. - Reset backoff factor on partial ACK. - Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. - Fix logstash output with pipelining mode enabled not reconnecting. - Empty configuration sections become merge-able with variables containing full path. - Fix error message about required fields missing not printing the missing field name. - Improve error message if compiling regular expression from config files fails. - Compression support in the Elasticsearch output. Metricbeat - Fix the CPU values returned for each core. - Add MongoDB module. Packetbeat - Add missing nil-check to memcached GapInStream handler. - Fix NFSv4 Operation returning the first found first-class operation available in compound requests. - Fix TCP overlapping segments not being handled correctly. Filebeat - Stop following symlink. Symlinks are now ignored: Beats version 5.0.0-alpha3 Affecting all Beats - All configuration settings under shipper: are moved to be top level configuration settings. I.e. shipper.name: becomes name: in the configuration file. - Add conditions to generic filtering. - The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. Topbeat - Topbeat is replaced by Metricbeat. Metricbeat - First public release, containing the following modules: apache, mysql, nginx, redis, system, and zookeeper. Filebeat - The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled. - The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. Beats version 5.0.0-alpha2 Affecting all Beats - On DEB/RPM installations, the binary files are now found under /usr/share/{{beat_name}}/bin, not in /usr/bin. - The logs are written by default to self rotating files, instead of syslog. - Remove deprecated host option from elasticsearch, logstash and redis outputs. - Drain response buffers when pipelining is used by Redis output. - Unterminated environment variable expressions in config files will now cause an error - Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. - Fix bug affecting -cpuprofile, -memprofile, and -httpprof CLI flags - Fix race when multiple outputs access the same event with logstash output manipulating event 1410 - Seed random number generator using crypto.rand package. https://github.com/elastic/beats/pull/1503{1503] - Fix beats hanging in -configtest - Fix kafka log message output - Add support for TLS to Redis output. - Add SOCKS5 proxy support to Redis output. - Failover and load balancing support in redis output. - Multiple-worker per host support for redis output. - Added ability to escape ${x} in config files to avoid environment variable expansion - Configuration options and CLI flags for setting the home, data and config paths. - Configuration options and CLI flags for setting the default logs path. - Update to Go 1.6.2 - Add Elasticsearch template files compatible with Elasticsearch 2.x. - Add scripts for managing the dashboards of a single Beat Packetbeat - Configuration of redis topology support changed. - Move all Packetbeat configuration options under the packetbeat namespace - Fix compile issues for OpenBSD. Filebeat - Default location for the registry file was changed to be data/registry from the binary directory, rather than .filebeat in the current working directory. This affects installations for zip/tar.gz/source, the location for DEB and RPM packages stays the same. - Improvements in registrar dealing with file rotation. - Fix issue with JSON decoding where @timestamp or type keys with the wrong type could cause Filebeat to crash. - Fix issue with JSON decoding where values having null as values could crash Filebeat. - Multiline reader normalizing newline to use \n. Topbeat - Updated elastic/gosigar version so Topbeat can compile on OpenBSD. Beats version 5.0.0-alpha1 libbeat - Run function to start a Beat now returns an error instead of directly exiting. - The method signature of HandleFlags() was changed to allow returning an error - Require braces for environment variable expansion in config files Packetbeat - Rename output fields in the dns package. Former flag recursion_allowed becomes recursion_available. 803 Former SOA field ttl becomes minimum. - The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. - Remove the count field from the exported event Topbeat - Rename proc.cpu.user_p with proc.cpu.total_p as it includes CPU time spent in kernel space - Remove count field from the exported fields - Rename input top level config option to topbeat Filebeat - Scalar values in used in the fields configuration setting are no longer automatically converted to strings. - Count field was removed from event as not used in filebeat Beats version 1.3.1 Filebeat - Fix a concurrent bug on filebeat startup with a large number of prospectors defined. Packetbeat - Fix description for the -I CLI flag. Beats version 1.3.0 Affecting all Beats - Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. - Fix output modes backoff counter reset. 1803 1814 - Set logstash output default bulk_max_size to 2048. - Seed random number generator using crypto.rand package. - Check stdout being available when console output is configured. - Updated to Go 1.7 Packetbeat - Add missing nil-check to memcached GapInStream handler. - Fix NFSv4 Operation returning the first found first-class operation available in compound requests. - Fix TCP overlapping segments not being handled correctly. Filebeat - Undocumented support for following symlinks is deprecated. Filebeat will not follow symlinks in version 5.0. Beats version 1.2.3 Topbeat - Fix high CPU usage when using filtering under Windows. Filebeat - Fix rotation issue with ignore_older. - Prevent file opening for files which reached ignore_older. --- beats/Makefile | 16 ++++++++-------- beats/PLIST | 6 ++++++ beats/distinfo | 8 ++++---- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/beats/Makefile b/beats/Makefile index d36f04c95dc..c9c1bfffee7 100644 --- a/beats/Makefile +++ b/beats/Makefile @@ -1,7 +1,7 @@ # $NetBSD$ # -DISTNAME= beats-1.2.2 +DISTNAME= beats-5.0.2 CATEGORIES= sysutils MASTER_SITES= ${MASTER_SITE_GITHUB:=elastic/} GITHUB_PROJECT= beats @@ -19,9 +19,9 @@ USE_TOOLS+= gmake WRKSRC= ${WRKDIR}/src/github.com/elastic/beats -EGFILES= filebeat.template.json filebeat.yml -EGFILES+= packetbeat.template.json packetbeat.yml -#EGFILES+= topbeat/topbeat.template.json topbeat/topbeat.yml +EGFILES= filebeat.template-es2x.json filebeat.template.json filebeat.yml +EGFILES+= metricbeat.template-es2x.json metricbeat.template.json metricbeat.yml +EGFILES+= packetbeat.template-es2x.json packetbeat.template.json packetbeat.yml .for file in ${EGFILES} CONF_FILES+= share/examples/beats/${file} ${PKG_SYSCONFDIR}/${file} @@ -30,9 +30,9 @@ CONF_FILES+= share/examples/beats/${file} ${PKG_SYSCONFDIR}/${file} PKG_SYSCONFSUBDIR= beats MAKE_ENV+= GOPATH=${WRKDIR}:${PREFIX}/gopkg -BUILD_DIRS= libbeat packetbeat filebeat # topbeat +BUILD_DIRS= libbeat filebeat metricbeat packetbeat -BUILD_TARGET= build +BUILD_TARGET= # INSTALLATION_DIRS+= bin share/examples/beats @@ -43,10 +43,10 @@ post-extract: do-install: ${INSTALL_PROGRAM} ${WRKSRC}/filebeat/filebeat ${DESTDIR}${PREFIX}/bin ${INSTALL_PROGRAM} ${WRKSRC}/libbeat/libbeat ${DESTDIR}${PREFIX}/bin + ${INSTALL_PROGRAM} ${WRKSRC}/metricbeat/metricbeat ${DESTDIR}${PREFIX}/bin ${INSTALL_PROGRAM} ${WRKSRC}/packetbeat/packetbeat ${DESTDIR}${PREFIX}/bin -# ${INSTALL_PROGRAM} ${WRKSRC}/topbeat ${DESTDIR}${PREFIX}/bin .for file in ${EGFILES} - ${INSTALL_DATA} ${WRKSRC}/*/etc/${file} \ + ${INSTALL_DATA} ${WRKSRC}/*/${file} \ ${DESTDIR}${PREFIX}/share/examples/beats .endfor diff --git a/beats/PLIST b/beats/PLIST index 98f84e183cf..bf32af3658d 100644 --- a/beats/PLIST +++ b/beats/PLIST @@ -1,8 +1,14 @@ @comment $NetBSD$ bin/filebeat bin/libbeat +bin/metricbeat bin/packetbeat +share/examples/beats/filebeat.template-es2x.json share/examples/beats/filebeat.template.json share/examples/beats/filebeat.yml +share/examples/beats/metricbeat.template-es2x.json +share/examples/beats/metricbeat.template.json +share/examples/beats/metricbeat.yml +share/examples/beats/packetbeat.template-es2x.json share/examples/beats/packetbeat.template.json share/examples/beats/packetbeat.yml diff --git a/beats/distinfo b/beats/distinfo index 80f47f4fb69..8ecb2f16823 100644 --- a/beats/distinfo +++ b/beats/distinfo @@ -1,6 +1,6 @@ $NetBSD$ -SHA1 (beats-1.2.2.tar.gz) = e74fac4538350da9f380e42c00fc7034c1cb8291 -RMD160 (beats-1.2.2.tar.gz) = 1484fefd8418f458b74b685d68a0b574f9dbc1ac -SHA512 (beats-1.2.2.tar.gz) = 44682089b0f06c171672dbe3f5d41b8f5c3071dfc34a5a40fd0e329882b06d7c493b6fa99a1f88a220602bd794120a457fdd5139b95f67bc4f64704da4b6d347 -Size (beats-1.2.2.tar.gz) = 9918529 bytes +SHA1 (beats-5.0.2.tar.gz) = aa815924cf212aa080eb984e1d7bd660a4d09eef +RMD160 (beats-5.0.2.tar.gz) = b7f77ac0a8b71fa154a1180a88c04af6a33aafbe +SHA512 (beats-5.0.2.tar.gz) = 2481ed584efda67051e59a27b45ba52039d960d6dc335d9225e2370bc71e5167f5e4e9e7cc0cb7e689c9a73d5edf27655c4764b0acb6f1422f18bfc10fc44fe9 +Size (beats-5.0.2.tar.gz) = 15774694 bytes