Beats version 7.1.1

View commits

No changes in this release.

Beats version 7.1.0

View commits

Updates to support changes to licensing of security features.

Some Elastic Stack security features, such as encrypted communications, file and native authentication, and role-based access control, are now available in more subscription levels. For details, see https://www.elastic.co/subscriptions.

Beats version 7.0.1

View commits

Breaking changes

Metricbeat

Change cloud.provider from ec2 to aws and from gce to gcp in add_cloud_metadata to align with ECS. 10775 11687

Bugfixes

Affecting all Beats

Fix formatting for event.duration, "human readable" was not working well for this. 11675
Fix initialization of the TCP input logger. 11605

Auditbeat

Package dataset: Log error when Homebrew is not installed. 11667

Heartbeat

Fix NPE on some monitor configuration errors. 11910

Metricbeat

Change add_cloud_metadata processor to not overwrite cloud field when it already exist in the event. 11612 11305

Added

Auditbeat

Add support to the system package dataset for the SUSE OS family. 11634

Deprecated

Metricbeat

Prevent the docker/memory metricset from processing invalid events before container start 11676

libbeat/docs/release-notes/7.0.0.asciidoc

Beats version 7.0.0-GA

View commits

The list below covers the changes between 7.0.0-rc2 and 7.0.0 GA only.

Bugfixes

Affecting all Beats

Relax validation of the X-Pack license UID value. 11640
Fix a parsing error with the X-Pack license check on 32-bit system. 11650
Fix ILM policy always being overwritten. 11671
Fix template always being overwritten. 11671

Auditbeat

Package dataset: Nullify Librpm’s rpmsqEnable. 11628

Filebeat

Fix add_docker_metadata source matching, using log.file.path field now. 11577

Beats version 7.0.0-rc2

Check the HEAD diff

Breaking changes

Auditbeat

Process dataset: Only report processes with executable. 11232
Shorten entity IDs. 11405

Metricbeat

Add connection and request timeouts for HTTP helper. 11032

Bugfixes

Affecting all Beats

Fixed OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494
Allow 'ilm.rollover_alias' to expand global fields like agent.version. 12233

Auditbeat

Package dataset: dlopen versioned librpm shared objects. 11565

Filebeat

Don’t apply multiline rules in Logstash json logs. 11346
Fix panic in add_kubernetes_metadata processor when key log does not exist. 11543 11549
Fix goroutine leak happening when harvesters are dynamically stopped. 11263

Metricbeat

Add _bucket to histogram metrics in Prometheus Collector 11578

Added

Auditbeat

Login dataset: Add event category and type. 11339

Filebeat

Add support for MySQL 8.0, Percona 8.0 and MariaDB 10.3. 11417

Beats version 7.0.0-rc1

Check the HEAD diff

Breaking changes

Affecting all Beats

On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project info from the cloud.machine.type and cloud.availability_zone. 10968
Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905
Empty meta.json file will be treated as a missing meta file. 8558
Rename migration.enabled config to migration.6_to_7.enabled. 11284
Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706
Beats Xpack now checks for Basic license on connect. 11296

Auditbeat

Process dataset: Only report processes with executable. 11232

Filebeat

Set ecs: true in user_agent processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875

Metricbeat

Migrate docker module to ECS. 10927

Functionbeat

Correctly extract Kinesis Data field from the Kinesis Record. 11141

Bugfixes

Affecting all Beats

Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. 10988
Add missing host.* fields to fields.yml. 11016
Include ip and boolean type when generating index pattern. 10995
Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936
Cancelling enrollment of a beat will not enroll the beat. 10150
Allow to configure Kafka fetching strategy for the topic metadata. 10682

Auditbeat

Package: Disable librpm signal handlers. 10694
Login: Handle different bad login UTMP types. 10865
System module: Fix and unify bucket closing logic. 10897
User dataset: Numerous fixes to error handling. 10942

Filebeat

Fix errors in filebeat Zeek dashboard and README files. Add notice.log support. 10916
Fix a bug when converting NetFlow fields to snake_case. 10950
Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105
Fix issue preventing docker container events to be stored if the container has a network interface without ip address. 11225 11247
Change URLPATH grok pattern to support brackets. 11135 11252
Add support for iis log with different address format. 11255 11256

Heartbeat

Fix checks for TCP send/receive data 11118

Metricbeat

Migrate docker autodiscover to ECS. 10757 10862
Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946
Fix for not reusable http client leading to connection leaks in Jolokia module 11014
Fix parsing error using GET in Jolokia module. 11075 11071
Collect metrics when EC2 instances are not in running state. 11008 11023
Change ECS field cloud.provider to aws. 11023
Add documentation about jolokia autodiscover fields. 10925 10979
Add missing aws.ec2.instance.state.name into fields.yml. 11219 11221
Fix ec2 metricset to collect metrics from Cloudwatch with the same timestamp. 11142
Fix potential memory leak in stopped docker metricsets 11294

Packetbeat

Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

Prevent Winlogbeat from dropping events with invalid XML. 11006
Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357

Functionbeat

Added

Affecting all Beats

Add ip fields to default_field in Elasticsearch template. 11035

Auditbeat

Move System module to beta. 10800

Filebeat

Add ISO8601 timestamp support in syslog metricset. 8716 10736
Add support for loading custom NetFlow and IPFIX field definitions to netflow input. 10945 11223
Added categorization fields for SSH login events in the system/auth fileset. 11334

Metricbeat

Add filters and pie chart for AWS EC2 dashboard. 10596

Known Issue

Journalbeat

Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

Beats version 7.0.0-beta1

Check the HEAD diff

Breaking changes

Affecting all Beats

Embedded html is not escaped anymore by default. 9914
Remove port settings from Logstash and Redis output. 9934
Rename process.exe to process.executable in add_process_metadata to align with ECS. 9949
Import ECS change ecs#308: leaf field user.group is now the group field set. 10275
Update the code of Central Management to align with the new returned format. 10019
Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338
Remove --setup command line flag. 10138
Remove --version command line flag. 10138
Remove --configtest command line flag. 10138
Move output.elasticsearch.ilm settings to setup.ilm. 10347
ILM will be available by default if Elasticsearch > 7.0 is used. 10347

Auditbeat

Rename process.exe to process.executable in auditd module to align with ECS. 9949
Rename process.cwd to process.working_directory in auditd module to align with ECS. 10195
Change data type of process.pid and process.ppid to number in JSON output of the auditd module. 10195
Change data type of file.uid and file.gid to string in JSON output of the FIM module. 10195
Field file.origin changed type from text to keyword. 10544
Rename user fields to ECS in auditd module. 10456
Rename event.type to auditd.message_type in auditd module because event.type is reserved for future use by ECS. 10536
Rename auditd.messages to event.original and auditd.warnings to error.message. 10577

Filebeat

Rename many kibana.log.* fields to map to ECS. 9301
Modify apache/error dataset to follow ECS. 8963
Rename many traefik.access.* fields to map to ECS. 9005
Fix parsing of GC entries in elasticsearch server log. 9513 9810
Rename read_timestamp to event.created for Redis input. 9924
Rename a few elasticsearch.audit.* fields to map to ECS. 9293
Rename read_timestamp to event.created for all Filebeat modules using it. 10139
Rename many iis.error.* fields to map to ECS. 9955
Adjust fileset haproxy.log to map to ECS. 10143
Rename a few logstash.* fields to map to ECS, remove logstash.slowlog.message. 9935
Rename a few mongodb.* fields to map to ECS. 10009
Rename a few mysql.* fields to map to ECS. 10008
Rename a few nginx.error.* fields to map to ECS. 10007
Rename many auditd.log.* fields to map to ECS. 10192
Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001
Remove service.name from Elastcsearch module. Replace by service.type. 10042
Remove numeric coercions for user.id and group.id. IDs should be keyword. 10233
Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033
Now save the 'first seen' timestamp in event.created (previously read_timestamp), instead of saving the parsed date. Now aligned with event.created semantics elsewhere. 10139
Rename mysql.error.thread_id and mysql.slowlog.id to mysql.thread_id. 10161
Remove mysql.error.timestamp and mysql.slowlog.timestamp. 10161
Migrate multiple fields to event.duration, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", including http.response.elapsed_time (ECS). 10188, 10274
Rename multiple fields to http.response.body.bytes, from modules "apache", "iis", "kibana", "nginx" and "traefik", including http.response.content_length (ECS). 10188
Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, raw_request_line, mode. 10397
Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. 10401
Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 10352
Migrate Elasticsearch audit logs fields to ECS 10352
Several text fields in the Logstash module are now indexed as keyword fields with text multi-fields (ECS). 10417
Several text fields in the Elasticsearch module are now indexed as keyword fields with text multi-fields (ECS). 10414
Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. 10442
The elasticsearch/deprecation fileset now indexes the component field under elasticsearch instead of elasticsearch.server. 10445
Remove field kafka.log.trace.full from kafka.log fielset. 10398
Change field kafka.log.class for kafka.log fileset from text to keyword. 10398
Address add_kubernetes_metadata processor issue where old source field is still used for matcher. 10505 10506
Change type of haproxy.source from text to keyword. 10506
Rename event.type to suricata.eve.event_type in Suricata module because event.type is reserved for future use by ECS. 10575
Populate more ECS fields in the Suricata module. 10006
Rename setting filebeat.registry_flush to filebeat.registry.flush. 10504
Rename setting filebeat.registry_file_permission to filebeat.registry.file_permission. 10504
Remove setting filebeat.registry_file in favor of filebeat.registry.path. The registry file will be stored in a sub-directory by now. 10504

Heartbeat

Remove monitor generator script that was rarely used. 9648
monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the id property explicitly. 9697
A number of fields have been aliased to their relevant counterparts in the url.* field. Existing visualizations should mostly work. The fields that have been moved are monitor.scheme → url.scheme, monitor.host → url.domain, resolve.host → url.domain, http.url → url.full, tcp.port → url.port. In addition to these moves the new fields url.username, url.password, url.path, and url.query are now present. It should be noted that the url.password field does not contain actual password values, but rather the text <hidden> 9570.
The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294

Journalbeat

Rename read_timestamp to event.created to align with ECS. 10043, 10139
Rename host.name to host.hostname to align with ECS. 10043
Fix typo in the field name container.id_truncated. 10525
Rename container.image.tag to container.log.tag. 10561
Change type of text fields to keyword. 10542

Metricbeat

Migrate system process metricset fields to ECS. 10332
Refactor Prometheus metric mappings 9948
Removed Prometheus stats metricset in favor of just using Prometheus collector 9948
Migrate system socket metricset fields to ECS. 10339
Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339
Adjust Redis.info metricset fields to ECS. 10319
Change type of field docker.container.ip_addresses to ip instead of keyword. 10364
Rename http.request.body field to http.request.body.content. 10315
Adjust php_fpm.process metricset fields to ECS. 10366
Adjust mongodb.status metricset to to ECS. 10368
Refactor munin module to collect an event per plugin and to have more strict field mappings. namespace option has been removed, and will be replaced by service.name. 10322
Change the following fields from type text to keyword: 10318
ceph.osd_df.name
ceph.osd_tree.name
ceph.osd_tree.children
kafka.consumergroup.meta
kibana.stats.name
mongodb.metrics.replication.executor.network_interface
php_fpm.process.request_uri
php_fpm.process.script
Add service.name option to all modules to explicitly set service.name if it is unset. 10427
Update a few elasticsearch.* fields to map to ECS. 10350
Update a few logstash.* fields to map to ECS. 10350
Update a few kibana.* fields to map to ECS. 10350
Update rabbitmq.* fields to map to ECS. 10563
Update haproxy.* fields to map to ECS. 10558 10568
Collect all EC2 meta data from all instances in all states. 10628
Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715

Packetbeat

Adjust Packetbeat http fields to ECS Beta 2 9645
http.request.body moves to http.request.body.content
http.response.body moves to http.response.body.content
Changed Packetbeat fields to align with ECS. 7968
Removed trailing dot from domain names reported by the DNS protocol. 9941

Winlogbeat

Adjust Winlogbeat fields to map to ECS. 10333

Functionbeat

Correctly normalize Cloudformation resource name. 10087
Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]
Allow functionbeat to use the keystore. 9009

Bugfixes

Affecting all Beats

Fix config appender registration. 9873
Gracefully handle TLS options when enrolling a Beat. 9129
The backing off now implements jitter to better distribute the load. 10172
Fix TLS certificate DoS vulnerability. 10302
Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289
Fix encoding of timestamps when using disk spool. 10099
Fix stopping of modules started by kubernetes autodiscover. 10476
Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587
Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675
Fix exclude_labels when there are dotted keys 10154
Fix registry handle leak on Windows (elastic/go-sysinfo#33). 9920

Auditbeat

Enable System module config on Windows. 10237

Filebeat

Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.
Support haproxy log lines without captured headers. 9463 9958
Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135
Fix bad bytes count in docker input when filtering by stream. 10211
Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307
Ensure source.address is always populated by the nginx module (ECS). 10418
Support mysql 5.7.22 slowlog starting with time information. 7892 9647

Heartbeat

Made monitors.d configuration part of the default config. 9004
Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace. 9566

Journalbeat

Do not stop collecting events when journal entries change. 9994

Metricbeat

Fix panics in vsphere module when certain values where not returned by the API. 9784
Fix pod UID metadata enrichment in Kubernetes module. 10081
Fix issue that would prevent collection of processes without command line on Windows. 10196
Fixed data type for tags field in docker/container metricset 10307
Fixed data type for tags field in docker/image metricset 10307
Fixed data type for isr field in kafka/partition metricset 10307
Fixed data types for various hosts fields in mongodb/replstatus metricset 10307
Added function to close sql database connection. 10355
Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850
Fixed a crash when using af_packet capture 10477

Winlogbeat

Close handle on signalEvent. 9838

Functionbeat

Ensure that functionbeat is logging at info level not debug. 10262
Add the required permissions to the role when deployment SQS functions. 9152

Added

Affecting all Beats

Update field definitions for http to ECS Beta 2 9645
Add agent.id and agent.ephemeral_id fields to all beats. 9404
Add name config option to add_host_metadata processor. 9943
Add add_labels and add_tags processors. 9973
Add missing file encoding to readers. 10080
Introduce migration.enabled configuration. 9805
Add alias field support in Kibana index pattern. 10075
Add add_fields processor. 10119
Add Kibana field formatter to bytes fields. 10184
Document a few more auditd.log.* fields. 10192
Support Kafka 2.1.0. 10440
Add ILM mode auto to setup.ilm.enabled setting. This new default value detects if ILM is available 10347
Add support to read ILM policy from external JSON file. 10347
Add overwrite and check_exists settings to ILM support. 10347
Generate Kibana index pattern on demand instead of using a local file. 10478
Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]
Allow to unenroll a Beat from the UI. 9452
Release Jolokia autodiscover as GA. 9706
Allow Central Management to send events back to kibana. 9382

Auditbeat

Add system module. 9546
Add user.id (UID) and user.name for ECS. 10195
Add group.id (GID) and group.name for ECS. 10195
System module process dataset: Add user information to processes. 9963
Add system package dataset. 10225
Add system module login dataset. 9327
Add entity_id fields. 10500
Add seven dashboards for the system module. 10511

Filebeat

Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761
Added module for parsing Google Santa logs. 9540
Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399
Add option to modules.yml file to indicate that a module has been moved 9432.
Add support for ssl_request_log in apache2 module. 8088 9833
Add support for iis 7.5 log format. 9753 9967
Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with service.type config. 10042
Add support for MariaDB in the slowlog fileset of mysql module. 9731
Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273
Elasticsearch module’s slowlog now populates event.duration (ECS). 9293
HAProxy module now populates event.duration and http.response.bytes (ECS). 10143
Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137
Add convert_timezone to nginx module. 9839 10148
Add support for Percona in the slowlog fileset of mysql module. 6665 10227
Added support for ingesting structured Elasticsearch audit logs 10352
Added support for ingesting structured Elasticsearch slow logs 10445
Added support for ingesting structured Elasticsearch deprecation logs 10445
New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176
Added support for ingesting structured Elasticsearch server logs 10428
Populate more ECS fields in the Suricata module. 10006
Add module zeek. 9931 10034

Heartbeat

Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Journalbeat

Migrate registry from previously incorrect path. 10486

Metricbeat

Add key metricset to the Redis module. 9582 9657 9746
Add socket_summary metricset to system defaults, removing experimental tag and supporting Windows 9709
Add docker event metricset. 9856
Add 'performance' metricset to x-pack mssql module 9826
Add DeDot for kubernetes labels and annotations. 9860 9939
Add more meaningful metrics to 'performance' Metricset on 'MSSQL' module 10011
Rename some fields in performance Metricset on MSSQL module to match the updated documentation from Microsoft 10074
Add AWS EC2 module. 9257 9300
Release windows Metricbeat module as GA. 10163
Release traefik Metricbeat module as GA. 10166
Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094
List filesystems on Windows that have an access path but not an assigned letter 8916 10196
Add nats module. 10071
Release uswgi Metricbeat module GA. 10164
Release php_fpm module as GA. 10198
Release Memcached module as GA. 10199
Release etcd module as GA. 10200
Release Ceph module as GA. 10202
Release aerospike module as GA. 10203
Release kubernetes apiserver and event metricsets as GA 10212
Release Couchbase module as GA. 10201
Release RabbitMQ module GA. 10165
Release envoyproxy module GA. 10223
Release mongodb.metrics and mongodb.replstatus as GA. 10242
Release mysql.galera_status as GA. 10242
Release postgresql.statement as GA. 10242
Release RabbitMQ Metricbeat module GA. 10165
Release Dropwizard module as GA. 10240
Release Graphite module as GA. 10240
Release kvm module as beta. 10279
Release http.server metricset as GA. 10240
Release Nats module as GA. 10281
Release munin module as GA. 10311
Release Golang module as GA. 10312
Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222
Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261
Rename 'db' Metricset to 'transaction_log' in MSSQL Metricbeat module 10109
Add process arguments and the path to its executable file in the system process metricset 10332
Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341
Release AWS module as GA. 10345
Add overview dashboard to Zookeeper Metricbeat module 10379

Packetbeat

Add network.community_id to Packetbeat flow events. 10061
Add aliases for flow fields that were renamed. 7968 10063
Add support to decode mysql prepare statement command. 8084

Functionbeat

Mark Functionbeat as GA. 10564

Beats version 7.0.0-alpha2

Check the HEAD diff

Breaking changes

Affecting all Beats

Update add_cloud_metadata fields to adjust to ECS. 9265
Automaticall cap signed integers to 63bits. 8991
Rename beat.timezone to event.timezone. 9458
Use _doc as document type. 9056
Removed dashboards and index patterns generation for Kibana 5. 8927
On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. 8942.

Auditbeat

Remove warning for deprecated option: "filters". 9002

Filebeat

Allow beats to blacklist certain part of the configuration while using Central Management. 9099
Remove warnings for deprecated options: "spool_size", "publish_async", "idle_timeout". 9002
Rename many haproxy.* fields to map to ECS. 9117
Rename many iis.access.* fields to map to ECS. 9084
IIS module’s user agent string is no longer encoded (+ replaced with spaces). 9084
Rename many system.syslog.* fields to map to ECS. 9135
Rename many nginx.access.* fields to map to ECS. 9081
Rename many system.auth.* fields to map to ECS. 9138
Rename many apache2.access.* fields to map to ECS. 9245
Rename apache2 module to apache. 9402

Metricbeat

Allow beats to blacklist certain part of the configuration while using Central Management. 9099
Remove warning for deprecated option: "filters". 9002

Packetbeat

Renamed the flow event fields to follow Elastic Common Schema. 9121
Renamed several client and server fields. IP, port, and process metadata are now contained under the client and server namespaces. 9303

Functionbeat

The CLI will now log CloudFormation Stack events. 8912
Function concurrency is now set to 5 instead of unreserved. 8992

Bugfixes

Affecting all Beats

Propagate Sync error when running SafeFileRotate. 9069
Fix autodiscover configurations stopping when metadata is missing. 8851
Log events at the debug level when dropped by encoding problems. 9251
Refresh host metadata in add_host_metadata. 9359
When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. 6271 9383
Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016
Ignore non index fields in default_field for Elasticsearch. 9549
Update Kibana index pattern attributes for objects that are disabled. 9644
Enforce validation for the Central Management access token. 9621
Update to Golang 1.11.4. 9627

Auditbeat

Filebeat

Correctly parse December or Dec in the Syslog input. 9349
Fix installation of haproxy dashboard. 9307 9313
Don’t generate incomplete configurations when logs collection is disabled by hints. 9305
Stop runners disabled by hints after previously being started. 9305
Fix saved objects in filebeat haproxy dashboard. 9417
Use log.source.address instead of log.source.ip for network input sources. 9487
Rename many redis.log.* fields to map to ECS. 9315
Rename many icinga.* fields to map to ECS. 9294
Rename many postgresql.log.* fields to map to ECS. 9308
Rename many kafka.log.* fields to map to ECS. 9297
Add convert_timezone option to Logstash module to convert dates to UTC. 9756 9797

Metricbeat

Fix issue preventing diskio metrics collection for idle disks. 9124 9125
Fix panic on docker healthcheck collection on dockers without healthchecks. 9171
Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. 9179
The node.name field in the elasticsearch/node metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. 9209

Packetbeat

Fix issue with process monitor associating traffic to the wrong process. 9151 9443

Added

Affecting all Beats

Unify dashboard exporter tools. 9097
Add cache.ttl to add_host_metadata. 9359
Add support for index lifecycle management (beta). 7963
Always include Pod UID as part of Pod metadata. 9517
Autodiscovery no longer requires that the condition field be set. If left unset all configs will be matched. 9029
Add geo fields to add_host_metadata processor. 9392

Filebeat

Added the redirect_stderr option that allows panics to be logged to log files. 8430
Added detect_null_bytes selector to detect null bytes from a io.reader. 9210
Added syslog_host variable to HAProxy module to allow syslog listener to bind to configured host. 9366
Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format 8015 6111 8768.
Add support for multi-core thread_id in postgresql module 9156 9482

Heartbeat

Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022
Add central management support. 9254

Journalbeat

Add cursor_seek_fallback option. 9234

Metricbeat

Add settings to disable docker and cgroup cpu metrics per core. 9187 9194 9589
The elasticsearch/node metricset now reports the Elasticsearch cluster UUID. 8771
Add service.type field to Metricbeat. 8965
Support GET requests in Jolokia module. 8566 9226
Add freebsd support for the uptime metricset. 9413
Add host.os.name field to add_host_metadata processor. 8948 9405
Add more TCP statuses to socket_summary metricset. 9430
Remove experimental tag from ceph metricsets. 9708
Add MS SQL module to X-Pack elastic#9414[9414

Deprecated

Metricbeat

event.duration is now in nano and not microseconds anymore. 8941

Beats version 7.0.0-alpha1

View commits

Breaking changes

Affecting all Beats

Dissect syntax change, use * instead of ? when working with field reference. 8054

Auditbeat

Use initial_scan action for new paths. 7954
Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.
Rename source.hostname to source.domain in the auditd module. 9027

Filebeat

Rename fileset.name to event.name. 8879
Rename fileset.module to event.module. 8879
Rename source to log.file.path and log.source.ip 8902
Remove the deprecated prospector(s) option in the configuration use input(s) instead. 8909
Rename offset to log.offset. 8923
Rename source_ecs to source in the Filebeat Suricata module. 8983

Bugfixes

Affecting all Beats

Fixed -d CLI flag by trimming spaces from selectors. 7864
Fixed Support add_docker_metadata in Windows by identifying systems' path separator. 7797
Do not panic when no tokenizer string is configured for a dissect processor. 8895
Start autodiscover consumers before producers. 7926

Filebeat

Fixed a memory leak when harvesters are closed. 7820
Fix improperly set config for CRI Flag in Docker Input 8899
Just enabling the elasticsearch fileset and starting Filebeat no longer causes an error. 8891
Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]

Heartbeat

Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn’t used since the connection would be closed soon after the headers were sent, but before the entire body was. 8894
Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516

Metricbeat

Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789
Add missing namespace field in http server metricset 7890
Fix race condition when enriching events with kubernetes metadata. 9055 9067

Packetbeat

Fixed the mysql missing transactions if monitoring a connection from the start. 8173

Added

Affecting all Beats

Add field host.os.kernel to the add_host_metadata processor and to the internal monitoring data. 7807
Add debug check to logp.Logger 7965
Count HTTP 429 responses in the elasticsearch output 8056
Allow Bus to buffer events in case listeners are not configured. 8527
Dissect will now flag event on parsing error. 8751
add_cloud_metadata initialization is performed asynchronously to avoid delays on startup. 8845
Add DeDot method in add_docker_metadata processor in libbeat. 9350 9505

Filebeat

Make inputsource generic taking bufio.SplitFunc as input 7746
Add custom unpack to log hints config to avoid env resolution 7710
Make docker input check if container strings are empty 7960
Keep unparsed user agent information in user_agent.original. 8537
Allow to force CRI format parsing for better performance 8424

Heartbeat

Add automatic config file reloading. 8023

Journalbeat

Add the ability to check against JSON HTTP bodies with conditions. 8667

Metricbeat

Add metrics about cache size to memcached module 7740
Add experimental socket summary metricset to system module 6782
Collect custom cluster display_name in elasticsearch/cluster_stats metricset. 8445
Test etcd module with etcd 3.3. 9068
All elasticsearch metricsets now have module-level cluster.id and cluster.name fields. 8770 8771 9164 9165 9166 9168
All elasticsearch node-level metricsets now have node.id and node.name fields. 9168 9209

Packetbeat

Add support to decode HTTP bodies compressed with gzip and deflate. 7915
Added support to calculate certificates' fingerprints (MD5, SHA-1, SHA-256). 8180
Support new TLS version negotiation introduced in TLS 1.3. 8647.

Beats version 6.8.0

Updates to support changes to licensing of security features.

Some Elastic Stack security features, such as encrypted communications, file and native authentication, and role-based access control, are now available in more subscription levels. For details, see https://www.elastic.co/subscriptions.

Beats version 6.7.2

View commits

Bugfixes

Affecting all Beats

Relax validation of the X-Pack license UID value. 11640
Fix a parsing error with the X-Pack license check on 32-bit system. 11650
Fix OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494
Fix false positives reported in the host.containerized field added by add_host_metadata. 11494
Fix the add_host_metadata’s host.id field on older Linux versions. 11494

Auditbeat

Package dataset: dlopen versioned librpm shared objects. 11565
Package dataset: Nullify Librpm’s rpmsqEnable. 11628

Filebeat

Don’t apply multiline rules in Logstash json logs. 11346
Fix goroutine leak happening when harvesters are dynamically stopped. 11263
Fix initialization of the TCP input logger. 11605

Metricbeat

Prevent the docker/memory metricset from processing invalid events before container start 11676

Added

Auditbeat

Add support to the system package dataset for the SUSE OS family. 11634

Beats version 6.7.1

View commits

Breaking changes

Affecting all Beats

Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

Bugfixes

Affecting all Beats

Remove IP fields from default_field in Elasticsearch template. 11399

Beats version 6.7.0

View commits

Breaking changes

Affecting all Beats

Port settings have been deprecated in redis/logstash output and will be removed in 7.0. 9915
Update the code of Central Management to align with the new returned format. 10019
Allow Central Management to send events back to kibana. 9382
Fix panic if fields settting is used to configure hosts.x fields. 10824 10935
Introduce query.default_field as part of the template. 11205
Beats Xpack now checks for Basic license on connect. 11296

Filebeat

Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001
Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033
Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 8852
Remove ecs option from user_agent processors when loading pipelines with Filebeat 6.7.x into Elasticsearch < 6.7.0. 10655 11362

Heartbeat

Remove monitor generator script that was rarely used. 9648

Bugfixes

Affecting all Beats

Fix TLS certificate DoS vulnerability. 10303
Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289
Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016
Do not panic when no tokenizer string is configured for a dissect processor. 8895
Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587
Add ECS-like selectors and dedotting to docker autodiscover. 10757 10862
Fix encoding of timestamps when using disk spool. 10099
Include ip and boolean type when generating index pattern. 10995
Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936
Cancelling enrollment of a beat will not enroll the beat. 10150
Remove IP fields from default_field in Elasticsearch template. 11399

Auditbeat

Package: Disable librpm signal handlers. 10694
Login: Handle different bad login UTMP types. 10865
Fix hostname references in System module dashbords. 11064
User dataset: Numerous fixes to error handling. 10942

Filebeat

Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10029
Fix bad bytes count in docker input when filtering by stream. 10211
Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307
Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/pull/10846[10846]
Fix a bug with the convert_timezone option using the incorrect timezone field. 11055 11164
Change URLPATH grok pattern to support brackets. 11135 11252
Add support for iis log with different address format. 11255 11256
Add fix to parse syslog message with priority value 0. 11010

Heartbeat

Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516
Fix checks for TCP send/receive data 10777

Journalbeat

Do not stop collecting events when journal entries change. 9994

Metricbeat

Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715
Fix issue that would prevent collection of processes without command line on Windows. 10196
Fixed data type for tags field in docker/container metricset 10307
Fixed data type for tags field in docker/image metricset 10307
Fixed data type for isr field in kafka/partition metricset 10307
Fixed data types for various hosts fields in mongodb/replstatus metricset 10307
Added function to close sql database connection. 10355
Fix parsing error using GET in Jolokia module. 11075 11071

Winlogbeat

Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357

Functionbeat

Correctly extract Kinesis Data field from the Kinesis Record. 11141
Add the required permissions to the role when deployment SQS functions. 9152

Added

Affecting all Beats

Add ip fields to default_field in Elasticsearch template. 11035
Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905

Auditbeat

System module process dataset: Add user information to processes. 9963
Add system package dataset. 10225
Add system module login dataset. 9327
Add entity_id fields. 10500
Add seven dashboards for the system module. 10511

Filebeat

Add field log.source.address and log.file.path to replace source. 9435
Support mysql 5.7.22 slowlog starting with time information. 7892 9647
Add support for ssl_request_log in apache2 module. 8088 9833
Add support for iis 7.5 log format. 9753 9967
Add support for MariaDB in the slowlog fileset of mysql module. 9731
Add convert_timezone to nginx module. 9839 10148
Add support for Percona in the slowlog fileset of mysql module. 6665 10227
Added support for ingesting structured Elasticsearch audit logs 8852
New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176
Populate more ECS fields in the Suricata module. 10006

Heartbeat

Made monitors.d configuration part of the default config. 9004
Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Metricbeat

Add field event.dataset which is {module}.{metricset}.
Add more TCP statuses to socket_summary metricset. 9430
Remove experimental tag from ceph metricsets. 9708
Add key metricset to the Redis module. 9582 9657
Add DeDot for kubernetes labels and annotations. 9860 9939
Add docker event metricset. 9856
Release Ceph module as GA. 10202
Release windows Metricbeat module as GA. 10163
Release traefik Metricbeat module as GA. 10166
List filesystems on Windows that have an access path but not an assigned letter 8916 10196
Release uswgi Metricbeat module GA. 10164
Release php_fpm module as GA. 10198
Release Memcached module as GA. 10199
Release etcd module as GA. 10200
Release kubernetes apiserver and event metricsets as GA 10212
Release Couchbase module as GA. 10201
Release aerospike module as GA. 10203
Release envoyproxy module GA. 10223
Release mongodb.metrics and mongodb.replstatus as GA. 10242
Release mysql.galera_status as Beta. 10242
Release postgresql.statement as GA. 10242
Release RabbitMQ Metricbeat module GA. 10165
Release Dropwizard module as GA. 10240
Release Graphite module as GA. 10240
Release http.server metricset as GA. 10240
Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261
Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222
Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094
Add remaining memory metrics of pods in Kubernetes metricbeat module 10157
Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341
Add overview dashboard to Zookeeper Metricbeat module 10379

Functionbeat

Mark Functionbeat as GA. 10564
Functionbeat can now deploy a function for Kinesis. 10116
Allow functionbeat to use the keystore. 9009

Deprecated

Filebeat

Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435

Metricbeat

Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds.

Packetbeat

Support new TLS version negotiation introduced in TLS 1.3. 8647.

Known Issue

Journalbeat

Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

Beats version 6.6.2

View commits

Bugfixes

Auditbeat

System module: Fix and unify bucket closing logic. 10897

Filebeat

Fix a bug when converting NetFlow fields to snake_case. 10950

Metricbeat

Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946

Packetbeat

Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

Prevent Winlogbeat from dropping events with invalid XML. 11006

Beats version 6.6.1

View commits

Breaking changes

Affecting all Beats

Fix stopping of modules started by kubernetes autodiscover. 10476

Auditbeat

Enable System module config on Windows. 10237

Filebeat

Fix bad bytes count in docker input when filtering by stream. 10211
Add convert_timezone option to Logstash module to convert dates to UTC. 9756 9797
Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761
Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

Journalbeat

Fix fields.yml indentation of audit group which had the effect of creating an incomplete Elasticsearch index template. 10556

Metricbeat

Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

Fixed a crash when using af_packet capture 10477

Functionbeat

Ensure that functionbeat is logging at info level not debug. 10262

Added

Filebeat

Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

Journalbeat

Migrate registry from previously incorrect path. 10486

Beats version 6.6.0

View commits

Breaking changes

Affecting all Beats

Dissect syntax change, use * instead of ? when working with field reference. 8054

Filebeat

Allow beats to blacklist certain part of the configuration while using Central Management. 9099

Metricbeat

Allow beats to blacklist certain part of the configuration while using Central Management. 9099

Functionbeat

The CLI will now log CloudFormation Stack events. 8912
Correctly normalize Cloudformation resource name. 10087

Bugfixes

Affecting all Beats

Fix autodiscover configurations stopping when metadata is missing. 8851
Refresh host metadata in add_host_metadata. 9359
When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. 6271 9383
Ignore non index fields in default_field for Elasticsearch. 9549
Update Golang to 1.10.6. 9563
Update Kibana index pattern attributes for objects that are disabled. 9644
Enforce validation for the Central Management access token. 9621
Fix registry handle leak on Windows (elastic/go-sysinfo#33). 9920
Gracefully handle TLS options when enrolling a Beat. 9129
Allow to unenroll a Beat from the UI. 9452
The backing off now implements jitter to better distribute the load. 10172
Fix config appender registration. 9873
Fix TLS certificate DoS vulnerability. 10304

Filebeat

Fix improperly set config for CRI Flag in Docker Input 8899
Just enabling the elasticsearch fileset and starting Filebeat no longer causes an error. 8891
Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]
Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10030
Support haproxy log lines without captured headers. 9463 9958

Heartbeat

Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn’t used since the connection would be closed soon after the headers were sent, but before the entire body was. 8894

Metricbeat

Add missing namespace field in http server metricset 7890
Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. 9179
The node.name field in the elasticsearch/node metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. 9209
Fix panics in vsphere module when certain values where not returned by the API. 9784
Fix pod UID metadata enrichment in Kubernetes module. 10081

Packetbeat

Fix issue with process monitor associating traffic to the wrong process. 9151 9443
Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

Added

Affecting all Beats

Unify dashboard exporter tools. 9097
Dissect will now flag event on parsing error. 8751
Added the redirect_stderr option that allows panics to be logged to log files. 8430
Add cache.ttl to add_host_metadata. 9359
Add support for index lifecycle management (beta). 7963
Always include Pod UID as part of Pod metadata. {pull]9517[9517]
Release Jolokia autodiscover as GA. 9706

Auditbeat

Add system module. 9546

Filebeat - Added detect_null_bytes selector to detect null bytes from a io.reader. 9210 - Added syslog_host variable to HAProxy module to allow syslog listener to bind to configured host. 9366 - Allow to force CRI format parsing for better performance 8424 - Add event.dataset to module events. 9457 - Add field log.source.address and log.file.path to replace source. 9435 - Add support for multi-core thread_id in postgresql module 9156 9482 - Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

Journalbeat

Add the ability to check against JSON HTTP bodies with conditions. 8667
Add cursor_seek_fallback option. 9234

Metricbeat

Collect custom cluster display_name in elasticsearch/cluster_stats metricset. 8445
Test etcd module with etcd 3.3. 9068
All elasticsearch metricsets now have module-level cluster.id and cluster.name fields. 8770 8771 9164 9165 9166 9168
All elasticsearch node-level metricsets now have node.id and node.name fields. 9168 9209
Add settings to disable docker and cgroup cpu metrics per core. 9187 9194 9589
The elasticsearch/node metricset now reports the Elasticsearch cluster UUID. 8771
Support GET requests in Jolokia module. 8566 9226
Add freebsd support for the uptime metricset. 9413
Add host.os.name field to add_host_metadata processor. 8948 9405
Add field event.dataset which is {module}.{metricset). 9393

Deprecated

Filebeat - Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435

Metricbeat

Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds. 9393

Packetbeat

Support new TLS version negotiation introduced in TLS 1.3. 8647.

Beats version 6.5.4

View commits

Bugfixes

Affecting all Beats

Update Golang to 1.10.6. This fixes an issue in remote certificate validation CVE-2018-16875. 9563

Filebeat

Fix saved objects in filebeat haproxy dashboard. 9417
Fixed a memory leak when harvesters are closed. 7820

Added

Filebeat

Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format 8015 6111 8768.

Beats version 6.5.3

View commits

Bugfixes

Affecting all Beats

Log events at the debug level when dropped by encoding problems. 9251

Filebeat

Correctly parse December or Dec in the Syslog input. 9349
Don’t generate incomplete configurations when logs collection is disabled by hints. 9305
Stop runners disabled by hints after previously being started. 9305
Fix installation of haproxy dashboard. 9307 9313

Beats version 6.5.2

View commits

Bugfixes

Affecting all Beats

Propagate Sync error when running SafeFileRotate. 9069

Metricbeat

Fix panic on docker healthcheck collection on dockers without healthchecks. 9171
Fix issue preventing diskio metrics collection for idle disks. 9124 9125

Beats version 6.5.1

View commits

Bugfixes

Affecting all Beats - Fix windows binaries not having an enroll command. 9096 8836

Journalbeat - Fix journalbeat sometimes hanging if output is unavailable. 9106

Metricbeat - Fix race condition when enriching events with kubernetes metadata. 9055 9067

Added

Journalbeat - Add minimal kibana dashboard. 9106

Beats version 6.5.0

View commits

Bugfixes

Affecting all Beats

Fixed add_host_metadata not initializing correctly on Windows. 7715
Fixed missing file unlock in spool file on Windows, so file can be reopened and locked. 7859
Fix spool file opening/creation failing due to file locking on Windows. 7859
Fix size of maximum mmaped read area in spool file on Windows. 7859
Fix potential data loss on OS X in spool file by using fcntl with F_FULLFSYNC. 7859
Improve fsync on linux, by assuming the kernel resets error flags of failed writes. 7859
Remove unix-like permission checks on Windows, so files can be opened. 7849
Replace index patterns in TSVB visualizations. 7929
Deregister pipeline loader callback when inputsRunner is stopped. 7893[7893]
Add backoff support to x-pack monitoring outputs. 7966
Removed execute permissions systemd unit file. 7873
Fix a race condition with the add_host_metadata and the event serialization. 8223 8653
Enforce that data used by k8s or docker doesn’t use any reference. 8240
Switch to different UUID lib due to to non-random generated UUIDs. 8485
Fix race condition when publishing monitoring data. 8646
Fix bug in loading dashboards from zip file. 8051
Fix in-cluster kubernetes configuration on IPv6. 8754
The export config subcommand should not display real value for field reference. 8769
The setup command will not fail if no dashboard is available to import. 8977
Fix central management configurations reload when a configuration is removed in Kibana. 9010

Auditbeat

Fixed a crash in the file_integrity module under Linux. 7753
Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075
Fixed a concurrent map write panic in the auditd module. 8158
Fixed a data race in the file_integrity module. 8009
Fixed a deadlock in the file_integrity module. 8027

Filebeat

Fix date format in Mongodb Ingest pipeline. 7974
Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]
Update CRI format to support partial/full tags. 8265
Fix some errors happening when stopping syslog input. 8347
Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346
Mark the TCP and UDP input as GA. 8125
Support multiline logs in logstash/log fileset of Filebeat. 8562
Support different timestamp format in postgresql module. 9494 9650

Heartbeat

Fixed bug where HTTP responses with larger bodies would incorrectly report connection errors. 8660

Metricbeat

Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789
Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075
Fixed the location of the modules.d dir in Deb and RPM packages. 8104
Add docker diskio stats on Windows. 6815 8126
Fix incorrect type conversion of average response time in Haproxy dashboards 8404
Added io disk read and write times to system module 8473 8508
Avoid mapping issues in kubernetes module. 8487
Recover metrics for old apache versions removed by mistake on #6450. 7871
Fix dropwizard module parsing of metric names. 8365 8385
Fix issue that would prevent kafka module to find a proper broker when port is not set 8613
Fix range colors in multiple visualizations. 8633 8634
Fix incorrect header parsing on http metricbeat module 8564 8585
Fixed a panic when the kvm module cannot establish a connection to libvirtd. 7792.

Packetbeat

Fixed a seccomp related error where the fcntl64 syscall was not permitted on 32-bit Linux and the sniffer failed to start. 7839
Added missing cmdline and client_cmdline fields to index template. 8258

Added

Affecting all Beats

Added time-based log rotation. 8349
Add backoff on error support to redis output. 7781
Allow for cloud-id to specify a custom port. This makes cloud-id work in ECE contexts. 7887
Add support to grow or shrink an existing spool file between restarts. 7859
Make kubernetes autodiscover ignore events with empty container IDs 7971
Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961
Add DNS processor with support for performing reverse lookups on IP addresses. 7770
Support for Kafka 2.0.0 in kafka output 8399
Add setting setup.kibana.space.id to support Kibana Spaces 7942
Better tracking of number of open file descriptors. 7986
Report number of open file handles on Windows. 8329
Added the add_process_metadata processor to enrich events with process information. 6789
Add Beats Central Management 8559
Report configured queue type. 8091
Enable host and cloud metadata processors by default. 8596

Filebeat

Add tag "truncated" to "log.flags" if incoming line is longer than configured limit. 7991
Add haproxy module. 8014
Add tag "multiline" to "log.flags" if event consists of multiple lines. 7997
Release docker input as GA. 8328
Keep unparsed user agent information in user_agent.original. 7832
Added default and TCP parsing formats to HAproxy module 8311 8637
Add Suricata IDS/IDP/NSM module. 8153 8693
Support for Kafka 2.0.0 8853

Heartbeat

Heartbeat is marked as GA.
Add automatic config file reloading. 8023
Added autodiscovery support 8415
Added support for extra TLS/x509 metadata. 7944
Added stats and state metrics for number of monitors and endpoints started. 8621
Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022

Journalbeat

Add journalbeat. 8703

Metricbeat

Add replstatus metricset to MongoDB module 7604
Add experimental socket summary metricset to system module 6782
Move common kafka fields (broker, topic and partition.id) to the module level to facilitate events correlation 7767
Add fields for memory fragmentation, memory allocator stats, copy on write, master-slave status, and active defragmentation to info metricset of Redis module. 7695
Increase ignore_above for system.process.cmdline to 2048. 8100
Add support to renamed fields planned for redis 5.0. 8167
Allow TCP helper to support delimiters and graphite module to accept multiple metrics in a single payload. 8278
Added 'died' PID state to process_system metricset on system module 8275
Add metrics metricset to MongoDB module. 7611
Added ccr metricset to Elasticsearch module. 8335
Support for Kafka 2.0.0 8399
Added support for query params in configuration 8286 8292
Add container image for docker metricsets. 8214 8438
Precalculate composed id fields for kafka dashboards. 8504
Add support for full status page output for php-fpm module as a separate metricset called process. 8394
Add Kafka dashboard. 8457
Release Kafka module as GA. 8854

Packetbeat

Added DHCP protocol support. 7647

Functionbeat

Initial version of Functionbeat. 8678

Deprecated

Heartbeat

watch.poll_file is now deprecated and superceded by automatic config file reloading.

Metricbeat

Redis info replication.master_offset has been deprecated in favor of replication.master.offset.https://github.com/elastic/beats/pull/7695[7695]
Redis info clients fields longest_output_list and biggest_input_buf have been renamed to max_output_buffer and max_input_buffer based on the names they will have in Redis 5.0, both fields will coexist during a time with the same value 8167.
Move common kafka fields (broker, topic and partition.id) to the module level 7767.

Beats version 6.4.3

View commits

Bugfixes

Affecting all Beats

Fix a race condition with the add_host_metadata and the event serialization. 8223 8653
Fix race condition when publishing monitoring data. 8646
Fix bug in loading dashboards from zip file. 8051
The export config subcommand should not display real value for field reference. 8769

Filebeat

Fix typo in Filebeat IIS Kibana visualization. 8604

Metricbeat

Recover metrics for old Apache versions removed by mistake on #6450. 7871
Avoid mapping issues in Kubernetes module. 8487
Fixed a panic when the KVM module cannot establish a connection to libvirtd. 7792

Beats version 6.4.2

View commits

Bugfixes

Filebeat

Fix some errors happening when stopping syslog input. 8347
Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346

Metricbeat

Fix incorrect type conversion of average response time in Haproxy dashboards 8404
Fix dropwizard module parsing of metric names. 8365 8385

Beats version 6.4.1

View commits

Bugfixes

Affecting all Beats

Add backoff support to x-pack monitoring outputs. 7966
Removed execute permissions systemd unit file. 7873
Fix a race condition with the add_host_metadata and the event serialization. 8223
Enforce that data used by k8s or docker doesn’t use any reference. 8240
Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961
Make kubernetes autodiscover ignore events with empty container IDs 7971

Auditbeat

Fixed a concurrent map write panic in the auditd module. 8158
Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075

Filebeat

Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]
Update CRI format to support partial/full tags. 8265

Metricbeat

Fixed the location of the modules.d dir in Deb and RPM packages. 8104
Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075
Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

Packetbeat

Added missing cmdline and client_cmdline fields to index template. 8258

Beats version 6.4.0

View commits

Known issue

Due to a packaging mistake, the modules.d configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This issue results in an empty list when you run metricbeat modules list and failures when you try to enable or disable modules. To work around this issue, run the following command:

sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/

This issue affects all new installations on DEB and RPM. Upgrades will run, but use old configurations defined in the modules.d directory from the previous installation.

The issue will be fixed in the 6.4.1 release.

Breaking changes

Affecting all Beats

Set default kafka version to 1.0.0 in kafka output. Older versions are still supported by configuring the version setting. Minimally supported version is 0.11 (older versions might work, but are untested). 7025

Heartbeat

Rename http.response.status to http.response.status_code to align with ECS. 7274
Remove type field as not needed. 7307

Metricbeat

Fixed typo in values for state_container status.phase, from terminate to terminated. 6916
RabbitMQ management plugin path is now configured at the module level instead of having to do it in each of the metricsets. New management_path_prefix option should be used now 7074
RabbitMQ node metricset only collects metrics of the instance it connects to, node.collect: cluster can be used to collect all nodes as before. 6556 6971
Change http/server metricset to put events by default under http.server and prefix config options with server.. 7100
Disable dedotting in docker module configuration. This will change the out-of-the-box behaviour, but not the one of already configured instances. 7485
Fix typo in etcd/self metricset fields from *.bandwithrate to *.bandwidthrate. 7456
Changed the definition of the system.cpu.total.pct and system.cpu.total.norm.cou fields to exclude the IOWait time. 7691

Bugfixes

Affecting all Beats

Error out on invalid Autodiscover template conditions settings. 7200
Allow to override the ignore_above option when defining new field with the type keyword. 7238
Fix a panic on the Dissect processor when we have data remaining after the last delimiter. 7449
When we fail to build a Kubernetes' indexer or matcher we produce a warning but we don’t add them to the execution. 7466
Fix default value for logging.files.keepfiles. It was being set to 0 and now it’s set to the documented value of 7. 7494
Retain compatibility with older Docker server versions. 7542
Fix errors unpacking configs modified via CLI by ignoring -E key=value pairs with missing value. 7599

Auditbeat

Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111
Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

Fix a data race between stopping and starting of the harvesters. 6879
Fix an issue when parsing ISO8601 dates with timezone definition 7367
Fix Grok pattern of MongoDB module. 7568
Fix registry duplicates and log resending on upgrade. 7634

Metricbeat

Fix Windows service metricset when using a 32-bit binary on a 64-bit OS. 7294
Do not report Metricbeat container host as hostname in Kubernetes deployment. 7199
Ensure metadata updates don’t replace existing pod metrics. 7573
Fix kubernetes pct fields reporting. 7677
Add support for new kube_node_status_condition in Kubernetes state_node. 7699

Added

Affecting all Beats

Add dissect processor. 6925
Add IP-addresses and MAC-addresses to add_host_metadata. 6878
Added a seccomp (secure computing) filter on Linux that whitelists the necessary system calls used by each Beat. 5213
Ship fields.yml as part of the binary 4834
Added options to dev-tools/cmd/dashboards/export_dashboard.go: -indexPattern to include index-pattern in output, -quiet to be quiet. 7101
Add Indexer indexing by pod uid. Enable pod uid metadata gathering in add_kubernetes_metadata. Extended Matcher log_path matching to support volume mounts 7072
Add default_fields to Elasticsearch template when connecting to Elasticsearch >= 7.0. 7015
Add support for loading a template.json file directly instead of using fields.yml. 7039
Add support for keyword multifields in field.yml. 7131
Add experimental Jolokia Discovery autodiscover provider. 7141
Add owner object info to Kubernetes metadata. 7231
Add Beat export dashboard command. 7239
Add support for docker autodiscover to monitor containers on host network 6708
Add ability to define input configuration as stringified JSON for autodiscover. 7372
Add processor definition support for hints builder 7386
Add support to disable html escaping in outputs. 7445
Refactor error handing in schema.Apply(). 7335
Add additional types to Kubernetes metadata 7457
Add module state reporting for Beats Monitoring. 7075
Release the rename processor as GA. 7656
Add support for Openstack Nova in add_cloud_metadata processor. 7663
Add support to set Beats services to automatic-delayed start on Windows. 8711

Auditbeat

Added XXH64 hash option for file integrity checks. 7311
Added the show auditd-rules and show auditd-status commands to show kernel rules and status. 7114
Add Kubernetes specs for auditbeat file integrity monitoring 7642

Filebeat

Add Kibana module with log fileset. 7052
Support MySQL 5.7.19 by mysql/slowlog 6969
Correctly join partial log lines when using docker input. 6967
Add support for TLS with client authentication to the TCP input 7056
Converted part of pipeline from treafik/access metricSet to dissect to improve efficiency. 7209
Add GC fileset to the Elasticsearch module. 7305
Add Audit log fileset to the Elasticsearch module. 7365
Add Slow log fileset to the Elasticsearch module. 7473
Add deprecation fileset to the Elasticsearch module. 7474
Add convert_timezone option to Kafka module to convert dates to UTC. 7546 7578
Add patterns for kafka 1.1 logs. 7608
Move debug messages in tcp input source 7712

Metricbeat

Add experimental Elasticsearch index metricset. 6881
Add dashboards and visualizations for haproxy metrics. 6934
Add Jolokia agent in proxy mode. 6475
Add message rates to the RabbitMQ queue metricset 6442 6606
Add exchanges metricset to the RabbitMQ module 6442 6607
Add Elasticsearch index_summary metricset. 6918
Add shard metricset to Elasticsearch module. 7006
Add apiserver metricset to Kubernetes module. 7059
Add maxmemory to redis info metricset. 7127
Set guest as default user in RabbitMQ module. 7107
Add postgresql statement metricset. 7048 7060
Update state_container metricset to support latest kube-state-metrics version. 7216
Add TLS support to MongoDB module. 7401
Added Traefik module with health metricset. 7413
Add Elasticsearch ml_job metricsets. 7196
Add support for bearer token files to HTTP helper. 7527
Add Elasticsearch index recovery metricset. 7225
Add locks, global_locks, oplatencies and process fields to status metricset of MongoDB module. 7613
Run Kafka integration tests on version 1.1.0 7616
Release raid and socket metricset from system module as GA. 7658
Release elasticsearch module and all its metricsets as beta. 7662
Release munin and traefik module as beta. 7660
Add envoyproxy module. 7569
Release prometheus collector metricset as GA. 7660
Add Elasticsearch cluster_stats metricset. 7638
Added basepath setting for HTTP-based metricsets 7700
Add couchdb module. 9406

Packetbeat

The process monitor now reports the command-line for all processes, under Linux and Windows. 7135
Updated the TLS protocol parser with new cipher suites added to TLS 1.3. 7455
Flows are enriched with process information using the process monitor. 7507
Added UDP support to process monitor. 7571

Deprecated

Metricbeat

Kubernetes state_container cpu.limit.nanocores and cpu.request.nanocores have been deprecated in favor of cpu.*.cores. 6916

Beats version 6.3.1

View commits

Bugfixes

Affecting all Beats

Allow index-pattern only setup when setup.dashboards.only_index=true. 7285
Preserve the event when source matching fails in add_docker_metadata. 7133
Negotiate Docker API version from our client instead of using a hardcoded one. 7165
Fix duplicating dynamic_fields in template when overwriting the template. 7352

Auditbeat

Fixed parsing of AppArmor audit messages. 6978

Filebeat

Comply with PostgreSQL database name format 7198
Optimize PostgreSQL ingest pipeline to use anchored regexp and merge multiple regexp into a single expression. 7269
Keep different registry entry per container stream to avoid wrong offsets. 7281
Fix offset field pointing at end of a line. 6514
Commit registry writes to stable storage to avoid corrupt registry files. 6792

Metricbeat

Fix field mapping for the system process CPU ticks fields. 7230
Ensure canonical naming for JMX beans is disabled in Jolokia module. 7047
Fix Jolokia attribute mapping when using wildcards and MBean names with multiple properties. 7321

Packetbeat

Fix an out of bounds access in HTTP parser caused by malformed request. 6997
Fix missing type for http.response.body field. 7169

Added

Auditbeat

Added caching of UID and GID values to auditd module. 6978
Updated syscall tables for Linux 4.16. 6978
Added better error messages for when the auditd module fails due to the Linux kernel not supporting auditing (CONFIG_AUDIT=n). 7012

Metricbeat

Collect accumulated docker network metrics and mark old ones as deprecated. 7253

Beats version 6.3.0

View commits

Breaking changes

Affecting all Beats

De dot keys of labels and annotations in kubernetes meta processors to prevent collisions. 6203
Rename beat.cpu..time metrics to beat.cpu..time.ms. 6449
Add host.name field to all events, to avoid mapping conflicts. This could be breaking Logstash configs if you rely on the host field being a string. 7051

Filebeat

Add validation for Stdin, when Filebeat is configured with Stdin and any other inputs, Filebeat will now refuse to start. 6463
Mark system.syslog.message and system.auth.message as text instead of keyword. 6589

Metricbeat

De dot keys in kubernetes/event metricset to prevent collisions. 6203
Add config option for windows/perfmon metricset to ignore non existent counters. 6432
Refactor docker CPU calculations to be more consistent with docker stats. 6608
Update logstash.node_stats metricset to write data under logstash.node.stats.*. 6714

Bugfixes

Affecting all Beats

Fix panic when Events containing a float32 value are normalized. 6129
Fix setup.dashboards.always_kibana when using Kibana 5.6. 6090
Fix for Kafka logger. 6430
Remove double slashes in Windows service script. 6491
Ensure Kubernetes labels/annotations don’t break mapping 6490
Ensure that the dashboard zip files can’t contain files outside of the kibana directory. 6921
Fix map overwrite panics by cloning shared structs before doing the update. 6947
Fix delays on autodiscovery events handling caused by blocking runner stops. 7170
Do not emit Kubernetes autodiscover events for Pods without IP address. 7235
Fix self metrics when containerized 6641

Auditbeat

Add hex decoding for the name field in audit path records. 6687
Fixed a deadlock in the file_integrity module under Windows. 6864
Fixed parsing of AppArmor audit messages. 6978
Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111
Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

Fix panic when log prospector configuration fails to load. 6800
Fix memory leak in log prospector when files cannot be read. 6797
Add raw JSON to message field when JSON parsing fails. 6516
Commit registry writes to stable storage to avoid corrupt registry files. 6877
Fix a parsing issue in the syslog input for RFC3339 timestamp and time with nanoseconds. 7046
Fix an issue with an overflowing wait group when using the TCP input. 7202

Heartbeat

Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. 6616

Metricbeat

Fix the default configuration for Logstash to include the default port. 6279
Fix dealing with new process status codes in Linux kernel 4.14+. 6306
Add filtering option by exact device names in system.diskio. diskio.include_devices. 6085
Add connections metricset to RabbitMQ module 6548
Fix panic in http dependent modules when invalid config was used. 6205
Fix system.filesystem.used.pct value to match what df reports. 5494
Fix namespace disambiguation in Kubernetes state_* metricsets. 6281
Fix Windows perfmon metricset so that it sends metrics when an error occurs. 6542
Fix Kubernetes calculated fields store. https://github.com/elastic/beats/pull/6564{6564}
Exclude bind mounts in fsstat and filesystem metricsets. 6819
Don’t stop Metricbeat if aerospike server is down. 6874
disk reads and write count metrics in RabbitMQ queue metricset made optional. 6876
Add mapping for docker metrics per cpu. 6843

Winlogbeat

Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247
Fix config validation to allow event_logs.processors. [pull]6217[6217]

Added

Affecting all Beats

Update Golang 1.9.4 6326
Add the ability to log to the Windows Event Log. 5913
The node name can be discovered automatically by machine-id matching when beat deployed outside Kubernetes cluster. 6146
Panics will be written to the logger before exiting. 6199
Add builder support for autodiscover and annotations builder 6408
Add plugin support for autodiscover builders, providers 6457
Preserve runtime from container statuses in Kubernetes autodiscover 6456
Experimental feature setup.template.append_fields added. 6024
Add appender support to autodiscover 6469
Add add_host_metadata processor 5968
Retry configuration to load dashboards if Kibana is not reachable when the beat starts. 6560
Add has_fields conditional to filter events based on the existence of all the given fields. 6285 6653
Add support for spooling to disk to the beats event publishing pipeline. 6581
Added logging of system info at Beat startup. 5946
Do not log errors if X-Pack Monitoring is enabled but Elastisearch X-Pack is not. 6627
Add rename processor. 6292
Allow override of dynamic template match_mapping_type for fields with object_type. 6691

Filebeat

Add IIS module to parse access log and error log. 6127
Renaming of the prospector type to the input type and all prospectors are now moved to the input folder, to maintain backward compatibility type aliasing was used to map the old type to the new one. This change also affect YAML configuration. 6078
Addition of the TCP input 6700
Add option to convert the timestamps to UTC in the system module. 5647
Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481
Add stream filtering when using docker prospector. 6057
Add support for CRI logs format. 5630
Add json.ignore_decoding_error config to not log json decoding erors. 6547
Make registry file permission configurable. 6455
Add MongoDB module. 6238
Add Ingest pipeline loading to setup. 6814
Add support of log_format combined to NGINX access logs. 6858
Release config reloading feature as GA.
Add support human friendly size for the UDP input. 6886
Add Syslog input to ingest RFC3164 Events via TCP and UDP 6842
Remove the undefined username option from the Redis input and clarify the documentation. 6662

Heartbeat

Made the URL field of Heartbeat aggregateable. 6263
Use match.Matcher for checking Heartbeat response bodies with regular expressions. 6539

Metricbeat

Support apache status pages for versions older than 2.4.16. 6450
Add support for huge pages on Linux. 6436
Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5970
Add graphite protocol metricbeat module. 4734
Add http server metricset to support push metrics via http. 4770
Make config object public for graphite and http server 4820
Add system uptime metricset. 4848
Add experimental queue metricset to RabbitMQ module. 4788
Add additional php-fpm pool status kpis for Metricbeat module 5287
Add etcd module. 4970
Add ip address of docker containers to event. 5379
Add ceph osd tree information to metricbeat 5498
Add ceph osd_df to metricbeat 5606
Add basic Logstash module. 5540
Add dashboard for Windows service metricset. 5603
Add pct calculated fields for Pod and container CPU and memory usages. 6158
Add statefulset support to Kubernetes module. 6236
Refactor prometheus endpoint parsing to look similar to upstream prometheus 6332
Making the http/json metricset GA. 6471
Add support for array in http/json metricset. 6480
Making the jolokia/jmx module GA. 6143
Making the MongoDB module GA. 6554
Allow to disable labels dedot in Docker module, in favor of a safe way to keep dots. 6490
Add experimental module to collect metrics from munin nodes. 6517
Add support for wildcards and explicit metrics grouping in jolokia/jmx. 6462
Set collector as default metricset in Prometheus module. 6636 6747
Set mntr as default metricset in Zookeeper module. 6674
Set default metricsets in vSphere module. 6676
Set status as default metricset in Apache module. 6673
Set namespace as default metricset in Aerospike module. 6669
Set service as default metricset in Windows module. 6675
Set all metricsets as default metricsets in uwsgi module. 6688
Allow autodiscover to monitor unexposed ports 6727
Mark kubernetes.event metricset as beta. 6715
Set all metricsets as default metricsets in couchbase module. 6683
Mark uwsgi module and metricset as beta. 6717
Mark Golang module and metricsets as beta. 6711
Mark system.raid metricset as beta. 6710
Mark http.server metricset as beta. 6712
Mark metricbeat logstash module and metricsets as beta. 6713
Set all metricsets as default metricsets in Ceph module. 6676
Set container, cpu, diskio, healthcheck, info, memory and network in docker module as default. 6718
Set cpu, load, memory, network, process and process_summary as default metricsets in system module. 6689
Set collector as default metricset in Dropwizard module. 6669
Set info and keyspace as default metricsets in redis module. 6742
Set connection as default metricset in rabbitmq module. 6743
Set all metricsets as default metricsets in Elasticsearch module. 6755
Set all metricsets as default metricsets in Etcd module. 6756
Set server metricsets as default in Graphite module. 6757
Set all metricsets as default metricsets in HAProxy module. 6758
Set all metricsets as default metricsets in Kafka module. 6759
Set all metricsets as default metricsets in postgresql module. 6761
Set status metricsets as default in Kibana module. 6762
Set all metricsets as default metricsets in Logstash module. 6763
Set container, node, pod, system, volume as default in Kubernetes module. https://github.com/elastic/beats/pull/ 6764[6764]
Set stats as default in memcached module. 6765
Set all metricsets as default metricsets in Mongodb module. 6766
Set pool as default metricset for php_fpm module. 6768
Set status as default metricset for mysql module. https://github.com/elastic/beats/pull/ 6769[6769]
Set stubstatus as default metricset for nginx module. 6770
Added support for haproxy 1.7 and 1.8. 6793
Add accumulated I/O stats to diskio in the line of docker stats. 6701
Ignore virtual filesystem types by default in system module. 6819
Release config reloading feature as GA. 6891
Kubernetes deployment: Add ServiceAccount config to system metricbeat. 6824
Kubernetes deployment: Add DNS Policy to system metricbeat. 6656

Packetbeat

Add support for condition on bool type 5659 5954
Fix high memory usage on HTTP body if body is not published. 6680
Allow to capture the HTTP request or response bodies independently. 6784
HTTP publishes an Error event for unmatched requests or responses. 6794

Winlogbeat

Use bookmarks to persist the last published event. 6150

Beats version 6.2.3

View commits

Breaking changes

Affecting all Beats

Fix conditions checking on autodiscover Docker labels. 6412

Bugfixes

Affecting all Beats

Avoid panic errors when processing nil Pod events in add_kubernetes_metadata. 6372
Fix infinite failure on Kubernetes watch 6504

Metricbeat

Fix Kubernetes overview dashboard views for non default time ranges. https://github.com/elastic/beats/issues/6395{6395}

Beats version 6.2.2

View commits

Bugfixes

Affecting all Beats

Add logging when monitoring cannot connect to Elasticsearch. 6365
Fix infinite loop when event unmarshal fails in Kubernetes pod watcher. 6353

Filebeat

Fix a conversion issue for time related fields in the Logstash module for the slowlog fileset. 6317

Beats version 6.2.1

View commits

No changes in this release.

Beats version 6.2.0

View commits

Breaking changes

Affecting all Beats

The log format may differ due to logging library changes. 5901
The default value for pipelining is reduced to 2 to avoid high memory in the Logstash beats input. 6250

Auditbeat

Split the audit.kernel and audit.file metricsets into their own modules named auditd and file_integrity, respectively. This change requires existing users to update their config. 5422
Renamed file_integrity module fields. 5423 5995
Renamed auditd module fields. 5423 6080

Metricbeat

Rename golang.heap.system.optained field to golang.heap.system.obtained. 5703
De dot keys in jolokia/jmx metricset to prevent collisions. 5957

Bugfixes

Auditbeat

Fixed an issue where the proctitle value was being truncated. 6080
Fixed an issue where values were incorrectly interpreted as hex data. 6080
Fixed parsing of the key value when multiple keys are present. 6080
Fix possible resource leak if file_integrity module is used with config reloading on Windows or Linux. 6198

Filebeat

Fix variable name for convert_timezone in the system module. 5936

Metricbeat

Fix error datastore '*' not found in Vsphere module. 4879
Fix error NotAuthenticated in Vsphere module. 4673
Fix mongodb session consistency mode to allow command execution on secondary nodes. 4689
Fix kubernetes state_pod status.phase so that the active phase is returned instead of unknown. 5980
Fix error collecting network_names in Vsphere module. 5962
Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp. 6033
Fix kafka OffsetFetch request missing topic and partition parameters. 5880

Packetbeat

Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

Added

Affecting all Beats

Adding a local keystore to allow user to obfuscate password 5687
Add autodiscover for kubernetes. 6055
Add Beats metrics reporting to Xpack. 3422
Update the command line library cobra and add support for zsh completion 5761
Update to Golang 1.9.2
Moved ip_port indexer for add_kubernetes_metadata to all beats. 5707
ip_port indexer now index both IP and IP:port pairs. 5721
Add the ability to write structured logs. 5901
Use structured logging for the metrics that are periodically logged via the logging.metrics feature. 5915
Improve Elasticsearch output metrics to count number of dropped and duplicate (if event ID is given) events. 5811
Add the ability for the add_docker_metadata process to enrich based on process ID. 6100
The add_docker_metadata and add_kubernetes_metadata processors are now GA, instead of Beta. 6105
Update go-ucfg library to support top level key reference and cyclic key reference for the keystore 6098

Auditbeat

Auditbeat is marked as GA, no longer Beta. 5432
Add support for BLAKE2b hash algorithms to the file integrity module. 5926
Add support for recursive file watches. 5575 5833

Filebeat

Add Osquery module. 5971
Add stream filtering when using docker prospector. 6057

Metricbeat

Add ceph osd_df to metricbeat 5606
Add field network_names of hosts and virtual machines. 5646
Add experimental system/raid metricset. 5642
Add a dashboard for the Nginx module. 5991
Add experimental mongodb/collstats metricset. 5852
Update the MySQL dashboard to use the Time Series Visual Builder. 5996
Add experimental uwsgi module. 6006
Docker and Kubernetes modules are now GA, instead of Beta. 6105
Support haproxy stats gathering using http (additionally to tcp socket). 5819
Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5957

Packetbeat

Configure good defaults for add_kubernetes_metadata. 5707

Beats version 6.1.3

View commits

No changes in this release.

Beats version 6.1.2

View commits

Bugfixes

Auditbeat

Add an error check to the file integrity scanner to prevent a panic when there is an error reading file info via lstat. 6005

Added

Filebeat

Switch to docker prospector in sample manifests for Kubernetes deployment 5963

Beats version 6.1.1

View commits

No changes in this release.

Beats version 6.1.0

View commits

Breaking changes

Auditbeat

Changed audit.file.path to be a multi-field so that path is searchable. 5625

Metricbeat

Rename heap_init field to heap.init in the Elasticsearch module. 5320
Rename http.response.status_code field to http.response.code in the HTTP module. 5521

Bugfixes

Affecting all Beats

Remove ID() from Runner interface 5153
Correctly send configured Host header to the remote server. 4842
Change add_kubernetes_metadata to attempt detection of namespace. 5482
Avoid double slash when join url and path 5517
Fix console color output for Windows. 5611
Fix logstash output debug message. https://github.com/elastic/beats/pull/5799{5799]
Fix isolation of modules when merging local and global field settings. 5795
Report ephemeral ID and uptime in monitoring events on all platforms 6501

Filebeat

Add support for adding string tags 5395
Fix race condition when limiting the number of harvesters running in parallel 5458
Fix relative paths in the prospector definitions. 5443
Fix recursive_globe.enabled option. 5443

Metricbeat

Change field type of http header from nested to object 5258
Fix the fetching of process information when some data is missing under MacOS X. 5337
Change MySQL active connections visualization title to MySQL total connections. 4812
Fix ProcState on Linux and FreeBSD when process names contain parentheses. 5775
Fix incorrect Mem.Used calculation under linux. 5775
Fix open_file_descriptor_count and max_file_descriptor_count lost in zookeeper module 5902
Fix system process metricset for kernel processes. 5700
Change kubernetes.node.cpu.allocatable.cores to float. 6130

Packetbeat

Fix http status phrase parsing not allow spaces. 5312
Fix http parse to allow to parse get request with space in the URI. 5495
Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572
Fix corruption when parsing repeated headers in an HTTP request or response. 6325
Fix panic when parsing partial AMQP messages. 6384
Fix out of bounds access to slice in MongoDB parser. 6256
Fix sniffer hanging on exit under Linux. 6535
Fix bounds check error in http parser causing a panic. 6750

Winlogbeat

Fix the registry file. It was not correctly storing event log names, and upon restart it would begin reading at the start of each event log. 5813

Added

Affecting all Beats

Support dashboard loading without Elasticsearch 5653
Changed the hashbang used in the beat helper script from /bin/bash to /usr/bin/env bash. 5051
Changed beat helper script to use exec when running the beat. 5051
Fix reloader error message to only print on actual error 5066
Add support for enabling TLS renegotiation. 4386
Add Azure VM support for add_cloud_metadata processor 5355
Add output.file.permission config option. 4638
Refactor add_kubernetes_metadata to support autodiscovery 5434
Improve custom flag handling and CLI flags usage message. 5543
Add number_of_routing_shards config set to 30 5570
Set log level for kafka output. 5397
Move TCP UDP start up into server.Start() 4903
Update to Golang 1.9.2

Auditbeat

Add support for SHA3 hash algorithms to the file integrity module. 5345
Add dashboards for Linux audit framework events (overview, executions, sockets). 5516

Filebeat

Add PostgreSQL module with slowlog support. 4763
Add Kafka log module. 4885
Add support for /var/log/containers/ log path in add_kubernetes_metadata processor. 4981
Remove error log from runnerfactory as error is returned by API. 5085
Add experimental Docker json-file prospector . 5402
Add experimental Docker autodiscover functionality. 5245
Add option to convert the timestamps to UTC in the system module. 5647
Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481

Metricbeat

Add graphite protocol metricbeat module. 4734
Add http server metricset to support push metrics via http. 4770
Make config object public for graphite and http server 4820
Add system uptime metricset. 4848
Add experimental queue metricset to RabbitMQ module. 4788
Add additional php-fpm pool status kpis for Metricbeat module 5287
Add etcd module. 4970
Add ip address of docker containers to event. 5379
Add ceph osd tree information to Metricbeat 5498
Add basic Logstash module. 5540
Add dashboard for Windows service metricset. 5603
Add experimental Docker autodiscover functionality. 5245
Add Windows service metricset in the windows module. 5332
Update gosigar to v0.6.0. 5775

Packetbeat

Add support for decoding the TLS envelopes. 5476
HTTP parses successfully on empty status phrase. 6176
HTTP parser supports broken status line. 6631

Beats version 6.0.1

View commits

Bugfixes

Affecting all Beats

Fix documentation links in README.md files. 5710
Fix add_docker_metadata dropping some containers. 5788

Heartbeat

Fix the "HTTP up status" visualization. 5564

Metricbeat

Fix map overwrite in docker diskio module. 5582
Fix connection leak in mongodb module. 5688
Fix the include top N processes feature for cases where there are fewer processes than N. 5729

libbeat/docs/release-notes/6.0.0.asciidoc

Beats version 6.0.0-GA

View commits

The list below covers the changes between 6.0.0-rc2 and 6.0.0 GA only.

Bugfixes

Filebeat

Fix machine learning jobs setup for dynamic modules. 5509

Packetbeat

Fix missing length check in the PostgreSQL module. 5457
Fix panic in ACK handler if event is dropped on blocked queue 5524

Added

Filebeat

Add Kubernetes manifests to deploy Filebeat. 5349
Add container short ID matching to add_docker_metadata. 6172

Metricbeat

Add Kubernetes manifests to deploy Metricbeat. 5349

Beats version 6.0.0-rc2

View commits

Breaking changes

Packetbeat

Remove not-working runoptions.uid and runoptions.gid options in Packetbeat. 5261

Bugfixes

Affecting all Beats

Fix data race accessing watched containers. 5147
Do not require template if index change and template disabled 5319
Fix missing ACK in redis output. 5404

Filebeat

Fix default paths for redis 4.0.1 logs on macOS 5173
Fix Filebeat not starting if command line and modules configs are used together. 5376
Fix double @timestamp field when JSON decoding was used. 5436

Metricbeat

Use beat.name instead of beat.hostname in the Host Overview dashboard. 5340
Fix the loading of 5.x dashboards. 5277

Added

Metricbeat

Auto-select a hostname (based on the host on which the Beat is running) in the Host Overview dashboard. 5340

Deprecated

Filebeat

The filebeat.config_dir option is deprecated. Use filebeat.config.prospector options instead. 5321

Beats version 6.0.0-rc1

View commits

Bugfixes

Affecting all Beats

Fix the /usr/bin/beatname script to accept -d "*" as a parameter. 5040
Combine fields.yml properties when they are defined in different sources. 5075
Keep Docker & Kubernetes pod metadata after container dies while they are needed by processors. 5084
Fix fields.yml lookup when using export template with a custom path.config param. 5089
Remove runner creation from every reload check 5141
Fix add_kubernetes_metadata matcher registry lookup. 5159

Metricbeat

Fix a memory allocation issue where more memory was allocated than needed in the windows-perfmon metricset. 5035
Don’t start metricbeat if external modules config is wrong and reload is disabled 5053
The MongoDB module now connects on each fetch, to avoid stopping the whole Metricbeat instance if MongoDB is not up when starting. 5120
Fix kubernetes events module to be able to index time fields properly. 5093
Fixed cmd_set and cmd_get being mixed in the Memcache module. 5189

Added

Affecting all Beats

Enable flush timeout by default. 5150
Add @metadata.version to events send to Logstash. 5166

Auditbeat

Changed the number of shards in the default configuration to 3. 5095
Add support for receiving audit events using a multicast socket. 4850

Filebeat

Changed the number of shards in the default configuration to 3. 5095
Don’t start filebeat if external modules/prospectors config is wrong and reload is disabled 5053
Add filebeat.registry_flush setting, to delay the registry updates. 5146

Heartbeat

Changed the number of shards in the default configuration to 1. 5095

Packetbeat

Changed the number of shards in the default configuration to 3. 5095

Winlogbeat

Changed the number of shards in the default configuration to 3. 5095

Beats version 6.0.0-beta2

View commits

Breaking changes

Affecting all Beats

The log directory (path.log) for Windows services is now set to C:\ProgramData\[beatname]\logs. 4764
The _all field is disabled in Elasticsearch 6.0. This means that searching by individual words only work on text fields. 4901
Fail if removed setting output.X.flush_interval is explicitly configured.
Rename the /usr/bin/beatname.sh script (e.g. metricbeat.sh) to /usr/bin/beatname. 4933
Beat does not start if elasticsearch index pattern was modified but not the template name and pattern. 4769
Fail if removed setting output.X.flush_interval is explicitly configured. 4880

Bugfixes

Affecting all Beats

Register kubernetes field_format matcher and remove logger in Encode API 4888
Fix go plugins not loaded when beat starts 4799
Add support for initContainers in add_kubernetes_metadata processor. 4825
Eliminate deprecated default mapping in 6.x 4864
Fix pod name indexer to use both namespace, pod name to frame index key 4775

Filebeat

Fix issue where the fileset.module could have the wrong value. 4761

Heartbeat

Fix monitor.name being empty by default. 4852
Fix wrong event timestamps. 4851

Metricbeat

Added missing mongodb configuration file to the modules.d folder. 4870
Fix wrong MySQL CRUD queries timelion visualization 4857
Add new metrics to CPU metricset 4969

Packetbeat

Update flow timestamp on each packet being received. 4895

Added

Affecting all Beats

Add setting to enable/disable the slow start in logstash output. 4972
Update init scripts to use the test config subcommand instead of the deprecated -configtest flag. 4600
Get by default the credentials for connecting to Kibana from the Elasticsearch output configuration. 4867
Added cloud.id and cloud.auth settings, for simplifying using Beats with the Elastic Cloud. 4959
Add lz4 compression support to kafka output. 4977
Add newer kafka versions to kafka output. 4977
Configure the index name when loading the dashboards and the index pattern. 4949

Metricbeat

Add filesystem.ignore_types to system module for ignoring filesystem types. 4685
Add support to exclude labels from kubernetes pod metadata. 4757

Beats version 6.0.0-beta1

View commits

Breaking changes

Affecting all Beats

Rename kubernetes processor to add_kubernetes_metadata. 4473
Rename .full.yml config files to .reference.yml. 4563
The scripts/import_dashboards is removed from packages. Use the setup command instead. 4586
Change format of the saved kibana dashboards to have a single JSON file for each dashboard 4413
Rename configtest command to test config. 4590
Remove setting queue_size and bulk_queue_size. 4650
Remove setting dashboard.snapshot and dashboard.snapshot_url. They are no longer needed because the dashboards are included in the packages by default. 4675
Beats can no longer be launched from Windows Explorer (GUI), command line is required. 4420

Auditbeat

Changed file metricset config to make file.paths a list instead of a dictionary. 4796

Heartbeat

Renamed the heartbeat RPM/DEB name to heartbeat-elastic. 4601

Metricbeat

Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544
Remove filters setting from metricbeat modules. 4699
Added type field to filesystem metrics. 4717

Packetbeat

Remove the already unsupported pf_ring sniffer option. 4608

Bugfixes

Affecting all Beats

Don’t stop with error loading the ES template if the ES output is not enabled. 4436
Fix race condition in internal logging rotator. 4519
Normalize all times to UTC to ensure proper index naming. 4569
Fix issue with loading dashboards to ES 6.0 when .kibana index did not already exist. 4659

Auditbeat

Fix file.max_file_size config option for the audit file metricset. 4796

Filebeat

Fix issue where the fileset.module could have the wrong value. 4761

Metricbeat

Fix issue affecting Windows services timing out at startup. 4491
Fix incorrect docker.diskio.total metric calculation. 4507
Vsphere module: used memory field corrected. 4461

Packetbeat

Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Winlogbeat

Removed validation of top-level config keys. This behavior was inconsistent with other Beats and caused maintainability issues. 4657

Added

Affecting all Beats

New cli subcommands interface. 4420
Allow source path matching in add_docker_metadata processor. 4495
Add support for analyzers and multifields in fields.yml. 4574
Add support for JSON logging. 4523
Add test output command, to test Elasticsearch and Logstash output settings. 4590
Introduce configurable event queue settings: queue.mem.events, queue.mem.flush.min_events and queue.mem.flush.timeout. 4650
Enable pipelining in Logstash output by default. 4650
Added 'result' field to Elasticsearch QueryResult struct for compatibility with 6.x Index and Delete API responses. {issue]4661[4661]
The sample dashboards are now included in the Beats packages. 4675
Add pattern option to be used in the fields.yml to specify the pattern for a number field. 4731

Auditbeat

Added file.hash_types config option for controlling the hash types. 4796
Added the ability to specify byte unit suffixes to file.max_file_size. 4796

Filebeat

Add experimental Redis module. 4441
Nginx module: use the first not-private IP address as the remote_ip. 4417
Load Ingest Node pipelines when the Elasticsearch connection is established, instead of only once at startup. 4479
Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609
Add udp prospector type. 4452
Enabled Cgo which means libc is dynamically compiled. 4546
Add Beta module config reloading mechanism 4566
Remove spooler and publisher components and settings. 4644

Heartbeat

Enabled Cgo which means libc is dynamically compiled. 4546

Metricbeat

Add random startup delay to each metricset to avoid the thundering herd problem. 4010
Add the ability to configure audit rules to the kernel module. 4482
Add the ability to configure kernel’s audit failure mode. 4516
Add experimental Aerospike module. 4560
Vsphere module: collect custom fields from virtual machines. 4464
Add test modules command, to test modules expected output. 4656
Add processors setting to metricbeat modules. 4699
Support npipe protocol (Windows) in Docker module. 4751

Winlogbeat

Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Auditbeat

Add file integrity metricset to the audit module. 4486

Beats version 6.0.0-alpha2

View commits

Breaking changes

Filebeat

Rename input_type field to prospector.type 4294
The @metadata.type field, added by the Logstash output, is now hardcoded to doc and will be removed in future versions. 4331.

Bugfixes

Affecting all Beats

Fix importing the dashboards when the limit for max open files is too low. 4244
Fix configuration documentation for kubernetes processor 4313
Fix misspelling in add_locale configuration option for abbreviation.

Filebeat

Fix race condition on harvester stopping with reloading enabled. 3779
Fix recursive glob config parsing and resolution across restarts. 4269
Allow string characters in user agent patch version (NGINX and Apache) 4415
Fix grok pattern in filebeat module system/auth without hostname. 4224

Metricbeat

Set correct format for percent fields in memory module. 4619
Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264
Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316
Fix panic when events were dropped by filters. 4327
Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370
Remove unnecessary print statement in schema apis. 4355
Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat - Enable memcache filtering only if a port is specified in the config file. 4335 - Enable memcache filtering only if a port is specified in the config file. 4335

Added

Affecting all Beats

Upgraded to Golang 1.8.3. 4401
Added the possibility to set Elasticsearch mapping template settings from the Beat configuration file. 4284 4317
Add a variable to the SysV init scripts to make it easier to change the user. 4340
Add the option to write the generated Elasticsearch mapping template into a file. 4323
Add instance_name in GCE add_cloud_metadata processor. 4414
Add add_docker_metadata processor. 4352
Add logging.files permissions option. 4295

Filebeat - Added ability to sort harvested files. 4374 - Add experimental Redis slow log prospector type. 4180

Metricbeat

Add macOS implementation of the system diskio metricset. 4144
Add process_summary metricset that records high level metrics about processes. 4231
Add kube-state-metrics based metrics to kubernetes module 4253
Add debug logging to Jolokia JMX metricset. 4341
Add events metricset for kubernetes metricbeat module 4315
Change Metricbeat default configuration file to be better optimized for most users. 4329
Add experimental RabbitMQ module. 4394
Add Kibana dashboard for the Kubernetes modules. 4138

Packetbeat

Winlogbeat

Deprecated

Affecting all Beats

The @metadata.type field, added by the Logstash output, is deprecated, hardcoded to doc and will be removed in future versions. 4331.

Filebeat

Deprecate input_type prospector config. Use type config option instead. 4294

Known Issue

If the Elasticsearch output is not enabled, but setup.template options are present (like it’s the case in the default Metricbeat configuration), the Beat stops with an error: "Template loading requested but the Elasticsearch output is not configured/enabled". To avoid this error, disable the template loading explicitly setup.template.enabled: false.

Beats version 6.0.0-alpha1

View commits

Breaking changes

Affecting all Beats

Introduce beat version in the Elasticsearch index and mapping template 3527
Usage of field _type is now ignored and hardcoded to doc. 3757
Change vendor manager from glide to govendor. 3851
Rename error field to error.message. 3987
Change dashboards. config options to setup.dashboards.. 3921
Change outputs.elasticsearch.template.* to `setup.template.* 4080

Filebeat

Remove code to convert states from 1.x. 3767
Remove deprecated config options force_close_files and close_older. 3768
Change clean_removed behaviour to also remove states for files which cannot be found anymore under the same name. 3827
Remove document_type config option. Use fields instead. 4204
Move json_error under error.message and error.key. 4167

Packetbeat

Remove deprecated geoip. 3766
Replace waitstop command line argument by shutdown_timeout in configuration file. 3588

Winlogbeat

Remove metrics endpoint. Replaced by http endpoint in libbeat (see #3717). 3901

Bugfixes

Affecting all Beats

Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat

Fix the Mysql slowlog parsing of IP addresses. 4183
Fix issue that new prospector was not reloaded on conflict 4128

Heartbeat

Use IP type of elasticsearch for ip field. 3926

Metricbeat

Support common.Time in mapstriface.toTime() 3812
Fix MongoDB dbstats fields mapping. 4025
Fixing prometheus collector to aggregate metrics based on metric family. 4075
Fixing multiEventFetch error reporting when no events are returned 4153

Added

Affecting all Beats

Initialize a beats UUID from file on startup. 3615
Add new add_locale processor to export the local timezone with an event. 3902
Add http endpoint. 3717
Updated to Go 1.8.1. 4033
Add kubernetes processor 3888
Add support for include_labels and include_annotations in kubernetes processor 4043
Support new index_patterns field when loading templates for Elasticsearch >= 6.0 4056
Adding goimports support to make check and fmt 4114
Make kubernetes indexers/matchers pluggable 4151
Abstracting pod interface in kubernetes plugin to enable easier vendoring 4152

Filebeat

Restructure input.Event to be inline with outputs.Data 3823
Add base for supporting prospector level processors 3853
Add filebeat.config.path as replacement for config_dir. 4051
Add a recursive_glob.enabled setting to expand ** in patterns. 3980
Add Icinga module. 3904
Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address.

Heartbeat

Event format and field naming changes in Heartbeat and sample Dashboard. 4091

Metricbeat

Add experimental metricset perfmon to Windows module. 3758
Add memcached module with stats metricset. 3693
Add the process.cmdline.cache.enabled config option to the System Process Metricset. 3891
Add new MetricSet interfaces for developers (Closer, ReportingFetcher, and PushMetricSet). 3908
Add kubelet module 3916
Add dropwizard module 4022
Adding query APIs for metricsets and modules from metricbeat registry 4102
Fixing nil pointer on prometheus collector when http response is nil 4119
Add http module with json metricset. 4092
Add the option to the system module to include only the first top N processes by CPU and memory. 4127.
Add experimental Vsphere module. 4028
Add experimental Elasticsearch module. 3903
Add experimental Kibana module. 3895
Move elasticsearch metricset node_stats under node.stats namespace. 4142
Make IP port indexer constructor public 4434

Packetbeat

Add fields and fields_under_root to Packetbeat protocols configurations. 3518
Add list style Packetbeat protocols configurations. This change supports specifying multiple configurations of the same protocol analyzer. 3518

Winlogbeat

Deprecated

Affecting all Beats

Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Packetbeat

Deprecate dictionary style protocols configuration. 3518

Winlogbeat

Known Issue

Filebeat

Prospector reloading only works properly with new files. 3546

Beats version 5.6.14

View commits

No changes in this version.

Beats version 5.6.13

View commits

No changes in this version.

Beats version 5.6.12

View commits

No changes in this version.

Beats version 5.6.11

View commits

No changes in this version.

Beats version 5.6.10

View commits

Bugfixes

Packetbeat

Fix an out of bounds access in HTTP parser caused by malformed request. 6997

Beats version 5.6.9

View commits

Bugfixes

Affecting all Beats

Fix a type issue when specifying certicate authority when using the import_dashboards command. 6678

Packetbeat

Fix http status phrase parsing not allow spaces. 5312
Fix http parse to allow to parse get request with space in the URI. 5495
Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572
Fix corruption when parsing repeated headers in an HTTP request or response. 6325
Fix panic when parsing partial AMQP messages. 6384
Fix out of bounds access to slice in MongoDB parser. 6256
Fix sniffer hanging on exit under Linux. 6535
Fix bounds check error in http parser causing a panic. 6750
HTTP parses successfully on empty status phrase. 6176
HTTP parser supports broken status line. 6631

Beats version 5.6.8

View commits

Bugfixes

Winlogbeat

Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247

Beats version 5.6.7

View commits

No changes in this release.

Beats version 5.6.6

View commits

No changes in this release.

Beats version 5.6.5

View commits

Bugfixes

Affecting all Beats

Fix duplicate batches of events in retry queue. 5520

Metricbeat

Clarify meaning of percentages reported by system core metricset. 5565
Fix map overwrite in docker diskio module. 5582

Beats version 5.6.4

View commits

Bugfixes

Affecting all Beats

Fix race condition in internal logging rotator. 4519

Packetbeat

Fix missing length check in the PostgreSQL module. 5457

Added

Affecting all Beats

Add support for enabling TLS renegotiation. 4386
Add setting to enable/disable the slow start in logstash output. 5400

Beats version 5.6.3

View commits

No changes in this release.

Beats version 5.6.2

View commits

No changes in this release.

Beats version 5.6.1

View commits

No changes in this release.

Beats version 5.6.0

View commits

Breaking changes

Affecting all Beats

The _all.norms setting in the Elasticsearch template is no longer disabled. This increases the storage size with one byte per document, but allows for a better upgrade experience to 6.0. 4901

Bugfixes

Filebeat

Fix issue where the fileset.module could have the wrong value. 4761

Packetbeat

Update flow timestamp on each packet being received. 4895

Metricbeat

Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264
Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316
Fix panic when events were dropped by filters. 4327

Added

Affecting all Beats

Add option to the import_dashboards script to load the dashboards via Kibana API. 4682

Filebeat

Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609
Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address. 4351

Metricbeat

Add filesystem.ignore_types to system module for ignoring filesystem types. 4685

Deprecated

Affecting all Beats

Loading more than one output is deprecated and will be removed in 6.0. 4907

Beats version 5.5.3

View commits

No changes in this release.

Beats version 5.5.2

View commits

No changes in this release.

Beats version 5.5.1

View commits

Bugfixes

Affecting all Beats

Normalize all times to UTC to ensure proper index naming. 4569

Beats version 5.5.0

View commits

Breaking changes

Affecting all Beats

Usage of field _type is now ignored and hardcoded to doc. 3757

Metricbeat - Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544

Bugfixes

Affecting all Beats

Fix console output. 4045

Filebeat

Allow string characters in user agent patch version (NGINX and Apache) 4415

Metricbeat

Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat

Fix packetbeat.interface options that contain underscores (e.g. with_vlans or bpf_filter). 4378
Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Deprecated

Filebeat

Deprecate document_type prospector config option as _type is removed in elasticsearch 6.0. Use fields instead. 4225

Winlogbeat

Deprecated metrics endpoint. It is superseded by a libbeat feature that can serve metrics on an HTTP endpoint. 4145

Beats version 5.4.2

View commits

Bugfixes

Affecting all Beats

Removed empty sections from the template files, causing indexing errors for array objects. 4488

Metricbeat

Fix issue affecting Windows services timing out at startup. 4491
Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

Packetbeat

Clean configured geoip.paths before attempting to open the database. 4306

Beats version 5.4.1

View commits

Bugfixes

Affecting all Beats

Fix importing the dashboards when the limit for max open files is too low. 4244
Fix console output. 4045

Filebeat

Fix issue that new prospector was not reloaded on conflict. 4128
Fix grok pattern in filebeat module system/auth without hostname. 4224
Fix the Mysql slowlog parsing of IP addresses. 4183

Added

Affecting all Beats

Binaries upgraded to Go 1.7.6 which contains security fixes. 4400

Winlogbeat

Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Beats version 5.4.0

View commits

Bugfixes

Affecting all Beats

Improve error message when downloading the dashboards fails. 3805
Fix potential Elasticsearch output URL parsing error if protocol scheme is missing. 3671
Downgrade Elasticsearch per batch item failure log to debug level. 3953
Make @timestamp accessible from format strings. 3721

Filebeat

Allow log lines without a program name in the Syslog fileset. 3944
Don’t stop Filebeat when modules are used with the Logstash output. 3929

Metricbeat

Fixing panic on the Prometheus collector when label has a comma. 3947
Make system process metricset honor the cpu_ticks config option. 3590

Winlogbeat

Fix null terminators include in raw XML string when include_xml is enabled. 3943

Added

Affecting all Beats

Update index mappings to support future Elasticsearch 6.X. 3778

Filebeat

Add auditd module for reading audit logs on Linux. 3750 3941
Add fileset for the Linux authorization logs. 3669

Heartbeat

Add default ports in HTTP monitor. 3924

Metricbeat

Add beta Jolokia module. 3844
Add dashboard for the MySQL module. 3716
Module configuration reloading is now beta instead of experimental. 3841
Marked http fields from the HAProxy module optional to improve compatibility with 1.5. 3788
Add support for custom HTTP headers and TLS for the Metricbeat modules. 3945

Packetbeat

Add DNS dashboard for an overview the DNS traffic. 3883
Add DNS Tunneling dashboard to highlight domains with large numbers of subdomains or high data volume. 3884

Beats version 5.3.2

View commits

Bugfixes

Filebeat

Properly shut down crawler in case one prospector is misconfigured. 4037
Fix panic in JSON decoding code if the input line is "null". 4042

Beats version 5.3.1

View commits

Bugfixes

Affecting all Beats

Fix panic when testing regex-AST to match against date patterns. 3889
Fix panic due to race condition in kafka output. 4098

Filebeat

Fix modules default file permissions. 3879
Allow - in Apache access log byte count. 3863

Metricbeat

Avoid errors when some Apache status fields are missing. 3074

Beats version 5.3.0

View commits

Breaking changes

Affecting all Beats

Configuration files must be owned by the user running the Beat or by root, and they must not be writable by others. 3544 3689
Change Beat generator. Use $GOPATH/src/github.com/elastic/beats/script/generate.py to generate a beat. 3452

Filebeat

Always use absolute path for event and registry. This can lead to issues when relative paths were used before. 3328

Metricbeat

Linux cgroup metrics are now enabled by default for the system process metricset. The configuration option for the feature was renamed from cgroups to process.cgroups.enabled. 3519
Change field names couchbase.node.couch..actual_disk_size. to couchbase.node.couch..disk_size. 3545

Bugfixes

Affecting all Beats

Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat - Always use absolute path for event and registry. 3328 - Raise an exception in case there is a syntax error in one of the configuration files available under filebeat.config_dir. 3573 - Fix empty registry file on machine crash. 3537

Metricbeat

Add error handling to system process metricset for when Linux cgroups are missing from the kernel. 3692
Add labels to the Docker healthcheck metricset output. 3707

Winlogbeat

Fix handling of empty strings in event_data. 3705

Added

Affecting all Beats

Files created by Beats (logs, registry, file output) will have 0600 permissions. 3387.
RPM/deb packages will now install the config file with 0600 permissions. 3382
Add the option to pass custom HTTP headers to the Elasticsearch output. 3400
Unify regexp and contains conditionals, for both to support array of strings and convert numbers to strings if required. 3469
Add the option to load the sample dashboards during the Beat startup phase. 3506
Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. 3528
Using environment variables in the configuration file is now GA, instead of experimental. 3525

Filebeat

Add Filebeat modules for system, apache2, mysql, and nginx. 3159
Add the pipeline config option at the prospector level, for configuring the Ingest Node pipeline ID. 3433
Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. 3469
The symlinks and harvester_limit settings are now GA, instead of experimental. 3525
close_timeout is also applied when the output is blocking. 3511
Improve handling of different path variants on Windows. 3781
Add multiline.flush_pattern option, for specifying the 'end' of a multiline pattern 4019

Heartbeat

Add tags, fields and fields_under_root in monitors configuration. 3623

Metricbeat

Add experimental dbstats metricset to MongoDB module. 3228
Use persistent, direct connections to the configured nodes for MongoDB module. 3228
Add dynamic configuration reloading for modules. 3281
Add docker health metricset 3357
Add docker image metricset 3467
System module uses new matchers for white-listing processes. 3469
Add Beta CEPH module with health metricset. 3311
Add Beta php_fpm module with pool metricset. 3415
The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. 3525
The HAProxy module is now GA, instead of experimental. 3525
Add the ability to collect the environment variables from system processes. 3337

Deprecated

Affecting all Beats

Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Filebeat

The experimental publish_async option is now deprecated and is planned to be removed in 6.0. 3525

Beats version 5.2.2

View commits

Metricbeat

Fix bug docker module hanging when docker container killed. 3610
Set timeout to period instead of 1s by default as documented. 3612

Beats version 5.2.1

View commits

Bugfixes

Metricbeat

Fix go routine leak in docker module. 3492

Packetbeat

Fix error in the NFS sample dashboard. 3548

Winlogbeat

Fix error in the Winlogbeat sample dashboard. 3548

Beats version 5.2.0

View commits

Bugfixes

Affecting all Beats

Fix overwriting explicit empty config sections. 2918

Filebeat

Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. 3273

Metricbeat

Fix service times-out at startup. 3056
Kafka module case sensitive host name matching. 3193
Fix interface conversion panic in couchbase module 3272

Packetbeat

Fix issue where some Cassandra visualizations were showing data from all protocols. 3314

Added

Affecting all Beats

Add support for passing list and dictionary settings via -E flag.
Support for parsing list and dictionary setting from environment variables.
Added new flags to import_dashboards (-cacert, -cert, -key, -insecure). 3139 3163
The limit for the number of fields is increased via the mapping template. 3275
Updated to Go 1.7.4. 3277
Added a NOTICE file containing the notices and licenses of the dependencies. 3334.

Heartbeat

First release, containing monitors for ICMP, TCP, and HTTP.

Filebeat

Add enabled config option to prospectors. 3157
Add target option for decoded_json_field. 3169

Metricbeat

Kafka module broker matching enhancements. 3129
Add a couchbase module with metricsets for node, cluster and bucket. 3081
Export number of cores for CPU module. 3192
Experimental Prometheus module. 3202
Add system socket module that reports all TCP sockets. 3246
Kafka consumer groups metricset. 3240
Add jolokia module with dynamic jmx metricset. 3570

Winlogbeat

Reduced amount of memory allocated while reading event log records. 3113 3118

Beats version 5.1.2

View commits

Bugfixes

Filebeat

Fix registry migration issue from old states where files were only harvested after second restart. 3322

Packetbeat

Fix error on importing dashboards due to colons in the Cassandra dashboard. 3140
Fix error on importing dashboards due to the wrong type for the geo_point fields. 3147

Winlogbeat

Fix for "The array bounds are invalid" error when reading large events. 3076

Beats version 5.1.1

View commits

Breaking changes

Metricbeat

Change data structure of experimental haproxy module. 3003

Filebeat

If a file is falling under ignore_older during startup, offset is now set to end of file instead of 0. With the previous logic the whole file was sent in case a line was added and it was inconsistent with files which were harvested previously. 2907
tail_files is now only applied on the first scan and not for all new files. 2932

Bugfixes

Affecting all Beats

Fix empty benign errors logged by processor actions. 3046

Metricbeat

Calculate the fsstat values per mounting point, and not filesystem. 2777

Added

Affecting all Beats

Add add_cloud_metadata processor for collecting cloud provider metadata. 2728
Added decode_json_fields processor for decoding fields containing JSON strings. 2605
Add Tencent Cloud provider for add_cloud_metadata processor. 4023
Add Alibaba Cloud provider for add_cloud_metadata processor. 4111

Metricbeat

Add experimental Docker module. Provided by Ingensi and @douaejeouit based on dockbeat.
Add a sample Redis Kibana dashboard. 2916
Add support for MongoDB 3.4 and WiredTiger metrics. 2999
Add experimental kafka module with partition metricset. 2969
Add raw config option for mysql/status metricset. 3001
Add command fields for mysql/status metricset. 3251

Filebeat

Add command line option -once to run Filebeat only once and then close. 2456
Only load matching states into prospector to improve state handling 2840
Reset all states ttl on startup to make sure it is overwritten by new config 2840
Persist all states for files which fall under ignore_older to have consistent behaviour 2859
Improve shutdown behaviour with large number of files. 3035

Winlogbeat

Add event_logs.batch_read_size configuration option. 2641

Beats version 5.1.0 (skipped)

Version 5.1.0 doesn’t exist because, for a short period of time, the Elastic Yum and Apt repositories included unreleased binaries labeled 5.1.0. To avoid confusion and upgrade issues for the people that have installed these without realizing, we decided to skip the 5.1.0 version and release 5.1.1 instead.

Beats version 5.0.2

View commits

Bugfixes

Metricbeat

Fix the password option in the MongoDB module. 2995

Beats version 5.0.1

View commits

Bugfixes

Metricbeat

Fix system.process.start_time on Windows. 2848
Fix system.process.ppid on Windows. 2860
Fix system process metricset for Windows XP and 2003. cmdline will be unavailable. 1704
Fix access denied issues in system process metricset by enabling SeDebugPrivilege on Windows. 1897
Fix system diskio metricset for Windows XP and 2003. 2885

Packetbeat

Fix 'index out of bounds' bug in Packetbeat DNS protocol plugin. 2872

Filebeat

Fix registry cleanup issue when files falling under ignore_older after restart. 2818

Added

Metricbeat

Add username and password config options to the PostgreSQL module. 2890
Add username and password config options to the MongoDB module. 2889
Add system core metricset for Windows. 2883

Packetbeat

Define client_geoip.location as geo_point in the mappings to be used by the GeoIP processor in the Ingest Node pipeline. 2795

Filebeat

Stop Filebeat on registrar loading error. 2868

libbeat/docs/release-notes/5.0.0.asciidoc

Beats version 5.0.0-GA

View commits

The list below covers the changes between 5.0.0-rc1 and 5.0.0 GA only.

Bugfixes

Affecting all Beats

Fix kafka output re-trying batches with too large events. 2735
Fix kafka output protocol error if version: 0.10 is configured. 2651
Fix kafka output connection closed by broker on SASL/PLAIN. 2717

Metricbeat

Fix high CPU usage on macOS when encountering processes with long command lines. 2747
Fix high value of system.memory.actual.free and system.memory.actual.used. 2653
Change several OpenProcess calls on Windows to request the lowest possible access privilege. 1897
Fix system.memory.actual.free high value on Windows. 2653

Filebeat

Fix issue when clean_removed and clean_inactive were used together that states were not directly removed from the registry.
Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. 2792

Added

Affecting all Beats

Add beat.version fields to all events.

Beats version 5.0.0-rc1

View commits

Breaking changes

Affecting all Beats

A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. 2688

Bugfixes

Affecting all Beats

Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. 2627
Fix ignoring all fields from drop_fields in case the first field is unknown. 2685
Fix dynamic configuration int/uint to float type conversion. 2698
Fix primitive types conversion if values are read from environment variables. 2698

Metricbeat

Fix default configuration file on Windows to not enabled the load metricset. 2632

Packetbeat

Fix the bpf_filter setting. 2660

Filebeat

Fix input buffer on encoding problem. 2416

Deprecated

Affecting all Beats

Setting port has been deprecated in Redis and Logstash outputs. 2620

Beats version 5.0.0-beta1

View commits

Breaking changes

Affecting all Beats

Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. 2119
Replace output.kafka.use_type by output.kafka.topic accepting a format string. 2188
If the path specified by the -c flag is not absolute and -path.config is not specified, it is considered relative to the current working directory. 2245
rename tls configurations section to ssl. 2330
rename certificate_key configuration to key. 2330
replace tls.insecure with ssl.verification_mode setting. 2330
replace tls.min/max_version with ssl.supported_protocols setting requiring full protocol name. 2330

Metricbeat

Change field type system.process.cpu.start_time from keyword to date. 1565
redis/info metricset fields were renamed up according to the naming conventions.

Packetbeat

Group HTTP fields under http.request and http.response 2167
Export http.request.body and http.response.body when configured under include_body_for 2167
Move ignore_outgoing config to packetbeat.ignore_outgoing 2393

Filebeat

Set close_inactive default to 5 minutes (was 1 hour before)
Set clean_removed and close_removed to true by default

Bugfixes

Affecting all Beats

Fix logstash output handles error twice when asynchronous sending fails. 2441
Fix Elasticsearch structured error response parsing error. 2229
Fixed the run script to allow the overriding of the configuration file. 2171
Fix logstash output crash if no hosts are configured. 2325
Fix array value support in -E CLI flag. 2521
Fix merging array values if -c CLI flag is used multiple times. 2521
Fix beats failing to start due to invalid duplicate key error in configuration file. 2521
Fix panic on non writable logging directory. 2571

Metricbeat

Fix module filters to work properly with drop_event filter. 2249

Packetbeat

Fix mapping for some Packetbeat flow metrics that were not marked as being longs. 2177
Fix handling of messages larger than the maximum message size (10MB). 2470

Filebeat

Fix processor failure in Filebeat when using regex, contain, or equals with the message field. 2178
Fix async publisher sending empty events 2455
Fix potential issue with multiple harvester per file on large file numbers or slow output 2541

Winlogbeat

Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Added

Affecting all Beats

Add script to generate the Kibana index-pattern from fields.yml. 2122
Enhance Redis output key selection based on format string. 2169
Configurable Redis keys using filters and format strings. 2169
Add format string support to output.kafka.topic. 2188
Add output.kafka.topics for more advanced kafka topic selection per event. 2188
Add support for Kafka 0.10. 2190
Add SASL/PLAIN authentication support to kafka output. 2190
Make Kafka metadata update configurable. 2190
Add Kafka version setting (optional) enabling kafka broker version support. 2190
Add Kafka message timestamp if at least version 0.10 is configured. 2190
Add configurable Kafka event key setting. 2284
Add settings for configuring the kafka partitioning strategy. 2284
Add partitioner settings reachable_only to ignore partitions not reachable by network. 2284
Enhance contains condition to work on fields that are arrays of strings. 2237
Lookup the configuration file relative to the -path.config CLI flag. 2245
Re-write import_dashboards.sh in Golang. 2155
Update to Go 1.7. 2306
Log total non-zero internal metrics on shutdown. 2349
Add support for encrypted private key files by introducing ssl.key_passphrase setting. 2330
Add experimental symlink support with symlinks config 2478
Improve validation of registry file on startup.

Metricbeat

Use the new scaled_float Elasticsearch type for the percentage values. 2156
Add experimental cgroup metrics to the system/process MetricSet. 2184
Added a PostgreSQL module. 2253
Improve mapping by converting half_float to scaled_float and integers to long. 2430
Add experimental haproxy module. 2384
Add Kibana dashboard for cgroups data 2555

Packetbeat

Add Cassandra protocol analyzer to Packetbeat. 1959
Match connections with IPv6 addresses to processes 2254
Add IP address to -devices command output 2327
Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. 2470

Filebeat

Introduce close_timeout harvester options 1926
Strip BOM from first message in case of BOM files 2351
Add harvester_limit option 2417

Deprecated

Affecting all Beats

Topology map is deprecated. This applies to the settings: refresh_topology_freq, topology_expire, save_topology, host_topology, password_topology, db_topology.

Beats version 5.0.0-alpha5

View commits

Breaking changes

Affecting all Beats

Rename the filters section to processors. 1944
Introduce the condition with when in the processor configuration. 1949
The Elasticsearch template is now loaded by default. 1993
The Redis output index setting is renamed to key. index still works but it’s deprecated. 2077
The undocumented file output index setting was removed. Use filename instead. 2077

Metricbeat

Create a separate metricSet for load under the system module and remove load information from CPU stats. 2101
Add system.load.norm.1, system.load.norm.5 and system.load.norm.15. 2101
Add threads fields to mysql module. 2484

Packetbeat

Set enabled ` in packetbeat.protocols.icmp configuration to true by default. 1988

Bugfixes

Affecting all Beats

Fix sync publisher PublishEvents return value if client is closed concurrently. 2046

Metricbeat

Do not send zero values when no value was present in the source. 1972

Filebeat

Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. 2041
Fix open file handler issue. 2028 2020
Fix filtering of JSON events when using integers in conditions. 2038

Winlogbeat

Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. 2041

Added

Affecting all Beats

Periodically log internal metrics. 1955
Add enabled setting to all output modules. 1987
Command line flag -c can be used multiple times. 1985
Add OR/AND/NOT to the condition associated with the processors. 1983
Add -E CLI flag for overwriting single config options via command line. 1986
Choose the mapping template file based on the Elasticsearch version. 1993
Check stdout being available when console output is configured. 2035

Metricbeat

Add pgid field to process information. https://github.com/elastic/beats/pull/ 2021[2021]

Packetbeat

Add enabled setting to Packetbeat protocols. 1988
Add enabled setting to Packetbeat network flows configuration. 1988

Filebeat

Introduce close_removed and close_renamed harvester options. 1600
Introduce close_eof harvester option. 1600
Add clean_removed and clean_inactive config option. 1600

Deprecated

Filebeat

Deprecate close_older option and replace it with close_inactive. 2051
Deprecate force_close_files option and replace it with close_removed and close_renamed. 1600

Beats version 5.0.0-alpha4

View commits

Breaking changes

Affecting all Beats

The topology_expire option of the Elasticsearch output was removed. 1907

Filebeat

Stop following symlink. Symlinks are now ignored: 1686

Bugfixes

Affecting all Beats

Reset backoff factor on partial ACK. 1803
Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829
Fix logstash output with pipelining mode enabled not reconnecting. 1876
Empty configuration sections become merge-able with variables containing full path. 1900
Fix error message about required fields missing not printing the missing field name. 1900

Metricbeat

Fix the CPU values returned for each core. 1863

Packetbeat

Add missing nil-check to memcached GapInStream handler. 1162
Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821
Fix TCP overlapping segments not being handled correctly. 1898

Winlogbeat

Fix issue with rendering forwarded event log records. 1891

Added

Affecting all Beats

Improve error message if compiling regular expression from config files fails. 1900
Compression support in the Elasticsearch output. 1835

Metricbeat

Add MongoDB module. 1837

Beats version 5.0.0-alpha3

View commits

Breaking changes

Affecting all Beats

All configuration settings under shipper: are moved to be top level configuration settings. I.e. shipper.name: becomes name: in the configuration file. 1570

Topbeat

Topbeat is replaced by Metricbeat.

Filebeat

The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled.

Bugfixes

Winlogbeat

Adding missing argument to the "Stop processing" log message. 1590

Added

Affecting all Beats

Add conditions to generic filtering. 1623

Metricbeat

First public release, containing the following modules: apache, mysql, nginx, redis, system, and zookeeper.

Filebeat

The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. 1703

Deprecated

Affecting all Beats

The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. 1601

Beats version 5.0.0-alpha2

View commits

Breaking changes

Affecting all Beats

On DEB/RPM installations, the binary files are now found under /usr/share/{{beat_name}}/bin, not in /usr/bin. 1385
The logs are written by default to self rotating files, instead of syslog. 1371
Remove deprecated host option from elasticsearch, logstash and redis outputs. 1474

Packetbeat

Configuration of redis topology support changed. 1353
Move all Packetbeat configuration options under the packetbeat namespace 1417

Filebeat

Default location for the registry file was changed to be data/registry from the binary directory, rather than .filebeat in the current working directory. This affects installations for zip/tar.gz/source, the location for DEB and RPM packages stays the same. 1373

Bugfixes

Affecting all Beats

Drain response buffers when pipelining is used by Redis output. 1353
Unterminated environment variable expressions in config files will now cause an error 1389
Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. 1321
Fix bug affecting -cpuprofile, -memprofile, and -httpprof CLI flags 1415
Fix race when multiple outputs access the same event with logstash output manipulating event 1410 1428
Seed random number generator using crypto.rand package. https://github.com/elastic/beats/pull/1503{1503]
Fix beats hanging in -configtest 1213
Fix kafka log message output 1516

Filebeat

Improvements in registrar dealing with file rotation. 1281
Fix issue with JSON decoding where @timestamp or type keys with the wrong type could cause Filebeat to crash. 1378
Fix issue with JSON decoding where values having null as values could crash Filebeat. 1466
Multiline reader normalizing newline to use \n. 1552

Winlogbeat

Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498
Fix panic that occurs when reading a large events on Windows Vista and newer. 1499

Added

Affecting all Beats

Add support for TLS to Redis output. 1353
Add SOCKS5 proxy support to Redis output. 1353
Failover and load balancing support in redis output. 1353
Multiple-worker per host support for redis output. 1353
Added ability to escape ${x} in config files to avoid environment variable expansion 1389
Configuration options and CLI flags for setting the home, data and config paths. 1373
Configuration options and CLI flags for setting the default logs path. 1437
Update to Go 1.6.2 1447
Add Elasticsearch template files compatible with Elasticsearch 2.x. 1501
Add scripts for managing the dashboards of a single Beat 1359

Packetbeat

Fix compile issues for OpenBSD. 1347

Topbeat

Updated elastic/gosigar version so Topbeat can compile on OpenBSD. 1403

Beats version 5.0.0-alpha1

View commits

Breaking changes

libbeat

Run function to start a Beat now returns an error instead of directly exiting. 771
The method signature of HandleFlags() was changed to allow returning an error 1249
Require braces for environment variable expansion in config files 1304

Packetbeat

Rename output fields in the dns package. Former flag recursion_allowed becomes recursion_available. 803 Former SOA field ttl becomes minimum. 803
The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. 803
Remove the count field from the exported event 1210

Topbeat

Rename proc.cpu.user_p with proc.cpu.total_p as it includes CPU time spent in kernel space 631
Remove count field from the exported fields 1207
Rename input top level config option to topbeat

Filebeat

Scalar values in used in the fields configuration setting are no longer automatically converted to strings. 1092
Count field was removed from event as not used in filebeat 778

Winlogbeat

The message_inserts field was replaced with the event_data field 1053
The category field was renamed to task to better align with the Windows Event Log API naming 1053
Remove the count field from the exported event 1218

Bugfixes

Affecting all Beats

Logstash output will not retry events that are not JSON-encodable 927

Packetbeat

Create a proper BPF filter when ICMP is the only enabled protocol 757
Check column length in pgsql parser. 565
Harden pgsql parser. 565

Topbeat

Fix issue with cpu.system_p being greater than 1 on Windows 1128

Filebeat

Stop filebeat if started without any prospectors defined or empty prospectors 644 647
Improve shutdown of crawler and prospector to wait for clean completion 720
Omit fields from Filebeat events when null 899

Winlogbeat

Added

Affecting all Beats

Update builds to Golang version 1.6
Add option to Elasticsearch output to pass http parameters in index operations 805
Improve Logstash and Elasticsearch backoff behavior. 927
Add experimental Kafka output. 942
Add config file option to configure GOMAXPROCS. 969
Improve shutdown handling in libbeat. 1075
Add fields and fields_under_root options under the shipper configuration 1092
Add the ability to use a SOCKS5 proxy with the Logstash output 823
The -configtest flag will now print "Config OK" to stdout on success 1249

Packetbeat

Change the DNS library used throughout the dns package to github.com/miekg/dns. 803
Add support for NFS v3 and v4. 1231
Add support for EDNS and DNSSEC. 1292

Topbeat

Add username to processes 845

Filebeat

Add the ability to set a list of tags for each prospector 1092
Add JSON decoding support 1143

Winlogbeat

Add caching of event metadata handles and the system render context for the wineventlog API 888
Improve config validation by checking for unknown top-level YAML keys. 1100
Add the ability to set tags, fields, and fields_under_root as options for each event log 1092
Add additional data to the events published by Winlogbeat. The new fields are activity_id, event_data, keywords, opcode, process_id, provider_guid, related_activity_id, task, thread_id, user_data, and version. 1053
Add event_id, level, and provider configuration options for filtering events 1218
Add include_xml configuration option for including the raw XML with the event 1218

Known issues

All Beats can hang or panic on shutdown if the next server in the pipeline (e.g. Elasticsearch or Logstash) is not reachable. 1319
When running the Beats as a service on Windows, you need to manually load the Elasticsearch mapping template. 1315
The ES template automatic load doesn’t work if Elasticsearch is not available when the Beat is starting. 1321

Beats version 1.3.1

View commits

Bugfixes

Filebeat

Fix a concurrent bug on filebeat startup with a large number of prospectors defined. 2509

Packetbeat

Fix description for the -I CLI flag. 2480

Winlogbeat

Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Beats version 1.3.0

View commits

Deprecated

Filebeat

Undocumented support for following symlinks is deprecated. Filebeat will not follow symlinks in version 5.0. 1767

Bugfixes

Affecting all Beats

Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829
Fix output modes backoff counter reset. 1803 1814 1818
Set logstash output default bulk_max_size to 2048. 1662
Seed random number generator using crypto.rand package. 1503
Check stdout being available when console output is configured. 2063

Packetbeat

Add missing nil-check to memcached GapInStream handler. 1162
Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821
Fix TCP overlapping segments not being handled correctly. 1917

Added

Affecting all Beats

Updated to Go 1.7

Beats version 1.2.3

View commits

Bugfixes

Topbeat

Fix high CPU usage when using filtering under Windows. 1598

Filebeat

Fix rotation issue with ignore_older. 1528

Winlogbeat

Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498

Added

Filebeat

Prevent file opening for files which reached ignore_older. 1649

Beats version 1.2.2

View commits

Bugfixes

Affecting all Beats

Fix race when multiple outputs access the same event with Logstash output manipulating event. 1410
Fix go-daemon (supervisor used in init scripts) hanging when executed over SSH. 1394

Filebeat

Improvements in registrar dealing with file rotation. 1281

Beats version 1.2.1

View commits

Breaking changes

Affecting all Beats

Require braces for environment variable expansion in config files 1304
Removed deprecation warning for the Redis output. 1282

Topbeat

Fixed name of the setting stats.proc to stats.process in the default configuration file. 1343
Fix issue with cpu.system_p being greater than 1 on Windows 1128

Added

Topbeat

Add username to processes 845

Beats version 1.2.0

View commits

Breaking changes

Filebeat

Default config for ignore_older is now infinite instead of 24h, means ignore_older is disabled by default. Use close_older to only close file handlers.

Bugfixes

Packetbeat

Split real_ip_header value when it contains multiple IPs 1241

Winlogbeat

Fix invalid event_id on Windows XP and Windows 2003 1227

Added

Affecting all Beats

Add ability to override configuration settings using environment variables 114
Libbeat now always exits through a single exit method for proper cleanup and control 736
Add ability to create Elasticsearch mapping on startup 639

Topbeat

Add the command line used to start processes 533

Filebeat

Add close_older configuration option to complete ignore_older 181

Beats version 1.1.2

View commits

Bugfixes

Filebeat

Fix registrar bug for rotated files 1010

Beats version 1.1.1

View commits

Bugfixes

Affecting all Beats

Fix logstash output loop hanging in infinite loop on too many output errors. 944
Fix critical bug in filebeat and winlogbeat potentially dropping events. 953

Beats version 1.1.0

View commits

Bugfixes

Affecting all Beats

Fix logging issue with file based output where newlines could be misplaced during concurrent logging 650
Reduce memory usage by separate queue sizes for single events and bulk events. 649 516
Set default default bulk_max_size value to 2048 628

Packetbeat

Fix setting direction to out and use its value to decide when dropping events if ignore_outgoing is enabled 557
Fix logging issue with file-based output where newlines could be misplaced during concurrent logging 650
Reduce memory usage by having separate queue sizes for single events and bulk events. 649 516
Set default bulk_max_size value to 2048 628
Fix logstash window size of 1 not increasing. 598

Packetbeat

Fix the condition that determines whether the direction of the transaction is set to "outgoing". Packetbeat uses the direction field to determine which transactions to drop when dropping outgoing transactions. 557
Allow PF_RING sniffer type to be configured using pf_ring or pfring 671

Filebeat

Set spool_size default value to 2048 628

Added

Affecting all Beats

Add include_fields and drop_fields as part of generic filtering 1120
Make logstash output compression level configurable. 630
Some publisher options refactoring in libbeat 684
Move event preprocessor applying GeoIP to packetbeat 772

Packetbeat

Add support for capturing DNS over TCP network traffic. 486 554

Topbeat

Group all CPU usage per core statistics and export them optionally if cpu_per_core is configured 496

Filebeat

Add multiline support for combining multiple related lines into one event. 461
Add exclude_lines and include_lines options for regexp based line filtering. 430
Add exclude_files configuration option. 563
Add experimental option to enable filebeat publisher pipeline to operate asynchronously 782

Winlogbeat

First public release of Winlogbeat

Beats version 1.0.1

Check 1.0.1 diff

Bugfixes

Filebeat

Fix force_close_files in case renamed file appeared very fast. 302

Packetbeat

Improve MongoDB message correlation. 377
Improve redis parser performance. 422
Fix panic on nil in redis protocol parser. 384
Fix errors redis parser when messages are split in multiple TCP segments. 402
Fix errors in redis parser when length prefixed strings contain sequences of CRLF. 402
Fix errors in redis parser when dealing with nested arrays. 402

Beats version 1.0.0

Check 1.0.0 diff

Breaking changes

Topbeat

Change proc type to process #138

Bugfixes

Affecting all Beats

Fix random panic on shutdown by calling shutdown handler only once. elastic/filebeat#204
Fix credentials are not send when pinging an elasticsearch host. elastic/filebeat#287

Filebeat

Fix problem that harvesters stopped reading after some time and filebeat stopped processing events #257
Fix line truncating by internal buffers being reused by accident #258
Set default ignore_older to 24 hours #282

Beats version 1.0.0-rc2

Check 1.0.0-rc2 diff

Breaking changes

Affecting all Beats

The shipper output field is renamed to beat.name. #285
Use of enabled as a configuration option for outputs (elasticsearch, logstash, etc.) has been removed. #264
Use of disabled as a configuration option for tls has been removed. #264
The -test command line flag was renamed to -configtest. #264
Disable geoip by default. To enable it uncomment in config file. #305

Filebeat

Removed utf-16be-bom encoding support. Support will be added with fix for #205
Rename force_close_windows_files to force_close_files and make it available for all platforms.

Bugfixes

Affecting all Beats

Disable logging to stderr after configuration phase. #276
Set the default file logging path when not set in config. #275
Fix bug silently dropping records based on current window size. elastic/filebeat#226
Fix direction field in published events. #300
Fix elasticsearch structured errors breaking error handling. #309

Packetbeat

Packetbeat will now exit if a configuration error is detected. #357
Fixed an issue handling DNS requests containing no questions. #369

Topbeat

Fix leak of Windows handles. #98
Fix memory leak of process information. #104

Filebeat

Filebeat will now exit if a configuration error is detected. #198
Fix to enable prospector to harvest existing files that are modified. #199
Improve line reading and encoding to better keep track of file offsets based on encoding. #224
Set input_type by default to "log"

Added

Affecting all Beats

Added beat.hostname to contain the hostname where the Beat is running on as returned by the operating system. #285
Added timestamp for file logging. #291

Filebeat

Handling end of line under windows was improved #233

Beats version 1.0.0-rc1

Check 1.0.0-rc1 diff

Breaking changes

Affecting all Beats

Rename timestamp field with @timestamp. #237

Packetbeat

Rename timestamp field with @timestamp. #343

Topbeat

Rename timestamp field with @timestamp for a better integration with Logstash. #80

Filebeat

Rename the timestamp field with @timestamp #168
Rename tail_on_rotate prospector config to tail_files
Removal of line field in event. Line number was not correct and does not add value. #217

Bugfixes

Affecting all Beats

Use stderr for console log output. #219
Handle empty event array in publisher. #207
Respect '*' debug selector in IsDebug. #226 (elastic#339)
Limit number of workers for Elasticsearch output. elastic#226
On Windows, remove service related error message when running in the console. #242
Fix waitRetry no configured in single output mode configuration. elastic/filebeat#144
Use http as the default scheme in the elasticsearch hosts #253
Respect max bulk size if bulk publisher (collector) is disabled or sync flag is set.
Always evaluate status code from Elasticsearch responses when indexing events. #192
Use bulk_max_size configuration option instead of bulk_size. #256
Fix max_retries=0 (no retries) configuration option. #266
Filename used for file based logging now defaults to beat name. #267

Packetbeat

Close file descriptors used to monitor processes. #337
Remove old RPM spec file. It moved to elastic/beats-packer. #334

Topbeat

Don’t wait for one period until shutdown #75

Filebeat

Omit 'fields' from event JSON when null. #126
Make offset and line value of type long in elasticsearch template to prevent overflow. #140
Fix locking files for writing behaviour. #156
Introduce 'document_type' config option per prospector to define document type for event stored in elasticsearch. #133
Add 'input_type' field to published events reporting the prospector type being used. #133
Fix high CPU usage when not connected to Elasticsearch or Logstash. #144
Fix issue that files were not crawled anymore when encoding was set to something other then plain. #182

Added

Affecting all Beats

Add Console output plugin. #218
Add timestamp to log messages #245
Send @metadata.beat to Logstash instead of @metadata.index to prevent possible name clashes and give user full control over index name used for Elasticsearch
Add logging messages for bulk publishing in case of error #229
Add option to configure number of parallel workers publishing to Elasticsearch or Logstash.
Set default bulk size for Elasticsearch output to 50.
Set default http timeout for Elasticsearch to 90s.
Improve publish retry if sync flag is set by retrying only up to max bulk size events instead of all events to be published.

Filebeat

Introduction of backoff, backoff_factor, max_backoff, partial_line_waiting, force_close_windows_files config variables to make crawling more configurable.
All Godeps dependencies were updated to master on 2015-10-21 [#122]
Set default value for ignore_older config to 10 minutes. #164
Added the fields_under_root setting to optionally store the custom fields top level in the output dictionary. #188
Add more encodings by using x/text/encodings/htmlindex package to select encoding by name.

Beats version 1.0.0-beta4

Check 1.0.0-beta4 diff

Breaking changes

Affecting all Beats

Update tls config options naming from dash to underline #162
Feature/output modes: Introduction of PublishEvent(s) to be used by beats #118 #115

Packetbeat

Renamed http module config file option 'strip_authorization' to 'redact_authorization'
Save_topology is set to false by default
Rename elasticsearch index to [packetbeat-]YYYY.MM.DD

Topbeat

Percentage fields (e.g user_p) are exported as a float between 0 and 1 #34

Bugfixes

Affecting all Beats

Determine Elasticsearch index for an event based on UTC time #81
Fixing ES output’s defaultDeadTimeout so that it is 60 seconds #103
ES outputer: fix timestamp conversion #91
Fix TLS insecure config option #239
ES outputer: check bulk API per item status code for retransmit on failure.

Packetbeat

Support for lower-case header names when redacting http authorization headers
Redact proxy-authorization if redact-authorization is set
Fix some multithreading issues #203
Fix negative response time #216
Fix memcache TCP connection being nil after dropping stream data. #299
Add missing DNS protocol configuration to documentation #269

Topbeat

Don’t divide the reported memory by an extra 1024 #60

Added

Affecting all Beats

Add logstash output plugin #151
Integration tests for Beat → Logstash → Elasticsearch added #195 #188 #168 #137 #128 #112
Large updates and improvements to the documentation
Add direction field to publisher output to indicate inbound/outbound transactions #150
Add tls configuration support to elasticsearch and logstash outputers #139
All external dependencies were updated to the latest version. Update to Golang 1.5.1 #162
Guarantee ES index is based in UTC time zone #164
Cache: optional per element timeout #144
Make it possible to set hosts in different ways. #135
Expose more TLS config options #124
Use the Beat name in the default configuration file path #99

Packetbeat

add [.editorconfig file](http://editorconfig.org/)
add (experimental/unsupported?) saltstack files
Sample config file cleanup
Moved common documentation to [libbeat repository](https://github.com/elastic/libbeat)
Update build to go 1.5.1
Adding device descriptions to the -device output.
Generate coverage for system tests
Move go-daemon dependency to beats-packer
Rename integration tests to system tests
Made the -devices option more user friendly in case sudo is not used. Issue #296.
Publish expired DNS transactions #301
Update protocol guide to libbeat changes
Add protocol registration to new protocol guide
Make transaction timeouts configurable #300
Add direction field to the exported fields #317

Topbeat

Document fields in a standardized format (etc/fields.yml) #34
Updated to use new libbeat Publisher #37 #41
Update to go 1.5.1 #43
Updated configuration files with comments for all options #65
Documentation improvements

Deprecated

Affecting all Beats

Redis output was deprecated #169 #145
Host and port configuration options are deprecated. They are replaced by the hosts configuration option. #141

Files

CHANGELOG.asciidoc

Latest commit

History

CHANGELOG.asciidoc

File metadata and controls

Beats version 7.1.1

Beats version 7.1.0

Beats version 7.0.1

Breaking changes

Bugfixes

Added

Deprecated

Beats version 7.0.0-GA

Bugfixes

Beats version 7.0.0-rc2

Breaking changes

Bugfixes

Added

Beats version 7.0.0-rc1

Breaking changes

Bugfixes

Added

Known Issue

Beats version 7.0.0-beta1

Breaking changes

Bugfixes

Added

Beats version 7.0.0-alpha2

Breaking changes

Bugfixes

Added

Deprecated

Beats version 7.0.0-alpha1

Breaking changes

Bugfixes

Added

Beats version 6.8.0

Beats version 6.7.2

Bugfixes

Added

Beats version 6.7.1

Breaking changes

Bugfixes

Beats version 6.7.0

Breaking changes

Bugfixes

Added

Deprecated

Known Issue

Beats version 6.6.2

Bugfixes

Beats version 6.6.1

Breaking changes

Added

Beats version 6.6.0

Breaking changes

Bugfixes

Added

Deprecated

Beats version 6.5.4

Bugfixes

Added

Beats version 6.5.3

Bugfixes

Beats version 6.5.2

Bugfixes

Beats version 6.5.1

Bugfixes

Added

Beats version 6.5.0

Bugfixes

Added

Deprecated

Beats version 6.4.3

Bugfixes

Beats version 6.4.2

Bugfixes

Beats version 6.4.1

Bugfixes