Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into s3_test
Browse files Browse the repository at this point in the history
  • Loading branch information
@kaiyan-sheng
kaiyan-sheng committed Feb 8, 2021
2 parents 1670a9b + 35e2e99 commit 5bc2d55
Show file tree
Hide file tree
Showing 35 changed files with 637 additions and 96 deletions.
98 changes: 95 additions & 3 deletions .ci/packaging.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ pipeline {
parameters {
booleanParam(name: 'macos', defaultValue: false, description: 'Allow macOS stages.')
booleanParam(name: 'linux', defaultValue: true, description: 'Allow linux stages.')
booleanParam(name: 'arm', defaultValue: true, description: 'Allow ARM stages.')
}
stages {
stage('Filter build') {
Expand Down Expand Up @@ -83,12 +84,13 @@ pipeline {
}
}
setEnvVar("GO_VERSION", readFile("${BASE_DIR}/.go-version").trim())
// Stash without any build/dependencies context to support different architectures.
stashV2(name: 'source', bucket: "${JOB_GCS_BUCKET_STASH}", credentialsId: "${JOB_GCS_CREDENTIALS}")
withMageEnv(){
dir("${BASE_DIR}"){
setEnvVar('BEAT_VERSION', sh(label: 'Get beat version', script: 'make get-version', returnStdout: true)?.trim())
}
}
stashV2(name: 'source', bucket: "${JOB_GCS_BUCKET_STASH}", credentialsId: "${JOB_GCS_CREDENTIALS}")
}
}
stage('Build Packages'){
Expand Down Expand Up @@ -172,12 +174,73 @@ pipeline {
}
steps {
withGithubNotify(context: "Packaging MacOS ${BEATS_FOLDER}") {
deleteDir()
deleteWorkspace()
withMacOSEnv(){
release()
}
}
}
post {
always {
// static workers require this
deleteWorkspace()
}
}
}
}
}
}
stage('Build Packages ARM'){
matrix {
axes {
axis {
name 'BEATS_FOLDER'
values (
'auditbeat',
'filebeat',
'heartbeat',
'journalbeat',
'metricbeat',
'packetbeat',
'x-pack/auditbeat',
'x-pack/elastic-agent',
'x-pack/filebeat',
'x-pack/heartbeat',
'x-pack/metricbeat',
'x-pack/packetbeat'
)
}
}
stages {
stage('Package Docker images for linux/arm64'){
agent { label 'arm' }
options { skipDefaultCheckout() }
when {
beforeAgent true
expression {
return params.arm
}
}
environment {
HOME = "${env.WORKSPACE}"
PACKAGES = "docker"
PLATFORMS = [
'linux/arm64',
].join(' ')
}
steps {
withGithubNotify(context: "Packaging linux/arm64 ${BEATS_FOLDER}") {
deleteWorkspace()
release()
pushCIDockerImages()
}
}
post {
always {
// static workers require this
deleteWorkspace()
}
}
}
}
}
Expand Down Expand Up @@ -408,14 +471,43 @@ def getBeatsName(baseDir) {
}

def withBeatsEnv(Closure body) {
unstashV2(name: 'source', bucket: "${JOB_GCS_BUCKET_STASH}", credentialsId: "${JOB_GCS_CREDENTIALS}")
fixPermissions()
withMageEnv(){
withEnv([
"PYTHON_ENV=${WORKSPACE}/python-env"
]) {
unstashV2(name: 'source', bucket: "${JOB_GCS_BUCKET_STASH}", credentialsId: "${JOB_GCS_CREDENTIALS}")
dir("${env.BASE_DIR}"){
body()
}
}
}
}

/**
* This method fixes the filesystem permissions after the build has happenend. The reason is to
* ensure any non-ephemeral workers don't have any leftovers that could cause some environmental
* issues.
*/
def deleteWorkspace() {
catchError(buildResult: 'SUCCESS', stageResult: 'SUCCESS') {
fixPermissions()
deleteDir()
}
}

def fixPermissions() {
if(isUnix()) {
catchError(buildResult: 'SUCCESS', stageResult: 'SUCCESS') {
dir("${env.BASE_DIR}") {
if (fileExists('script/fix_permissions.sh')) {
sh(label: 'Fix permissions', script: """#!/usr/bin/env bash
set +x
source ./dev-tools/common.bash
docker_setup
script/fix_permissions.sh ${WORKSPACE}""", returnStatus: true)
}
}
}
}
}
227 changes: 227 additions & 0 deletions CHANGELOG.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,233 @@
:issue: https://github.com/elastic/beats/issues/
:pull: https://github.com/elastic/beats/pull/

[[release-notes-7.11.0]]
=== Beats version 7.11.0
https://github.com/elastic/beats/compare/v7.10.2...v7.11.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs. {pull}21179[21179]
- Update to ECS 1.7.0. {pull}22571[22571]
- Add support for SCRAM-SHA-512 and SCRAM-SHA-256 in Kafka output. {pull}12867[12867]

*Auditbeat*

- Use ECS 1.7 ingress/egress network directions instead of inbound/outbound for system/socket. {pull}22991[22991]
- Use ingress/egress instead of inbound/outbound for ECS 1.7 in auditd module. {pull}23000[23000]

*Filebeat*

- Add fileset to ingest Kibana's ECS audit logs. {pull}22696[22696]
- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095]
- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571]
- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]

*Heartbeat*
- Adds negative body match. {pull}20728[20728]

*Metricbeat*

- Change cloud.provider from googlecloud to gcp. {pull}21775[21775]
- Rename googlecloud module to gcp module. {pull}22246[22246]
- Use ingress/egress instead of inbound/outbound for system/socket metricset. {pull}22992[22992]
- Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. {pull}23335[23335]

*Packetbeat*

- Update how Packetbeat classifies network directionality to bring it in line with ECS 1.7 {pull}22996[22996]

*Winlogbeat*

- Use ECS 1.7 ingress/egress instead of inbound/outbound network.direction in sysmon. {pull}22997[22997]

==== Bugfixes

*Affecting all Beats*

- Fix memory leak and events duplication in docker autodiscover and add_docker_metadata. {pull}21851[21851]
- Fix duplicated pod events in kubernetes autodiscover for pods with init or ephemeral containers. {pull}22438[22438]
- Fix FileVersion contained in Windows exe files. {pull}22581[22581]
- Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure {issue}12211[12211], {pull}13387[13387]
- Periodic metrics in logs will now report `libbeat.output.events.active` and `beat.memstats.rss` as gauges (rather than counters). {pull}22877[22877]
- Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service {pull}22874[22874]
- Fix reporting of cgroup metrics when running under Docker {pull}22879[22879]
- Fix typo in config docs {pull}23185[23185]
- Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419]
- Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484]

*Auditbeat*

- file_integrity: stop monitoring excluded paths {issue}21278[21278] {pull}21282[21282]
- Note incompatibility of system/socket on ARM. {pull}23381[23381]

*Filebeat*

- Fix Zeek dashboard reference to `zeek.ssl.server.name` field. {pull}21696[21696]
- Fix network.direction logic in zeek connection fileset. {pull}22967[22967]
- Fix aws s3 overview dashboard. {pull}23045[23045]
- Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072]
- Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966]
- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126]
- Simplify regex for organization custom prefix in AWS/CloudTrail fileset. {issue}23203[23203] {pull}23204[23204]
- Fix syslog header parsing in infoblox module. {issue}23272[23272] {pull}23273[23273]
- Fix concurrent modification exception in Suricata ingest node pipeline. {pull}23534[23534]
- Fix handling of ModifiedProperties field in Office 365. {pull}23777[23777]

*Heartbeat*

- Fixed missing `tls` fields when connecting to https via proxy. {issue}15797[15797] {pull}22190[22190]

*Metricbeat*

- Change Session ID type from int to string {pull}22359[22359]
- Fix filesystem types on Windows in filesystem metricset. {pull}22531[22531]
- Fix failiures caused by custom beat names with more than 15 characters {pull}22550[22550]
- Update NATS dashboards to leverage connection and route metricsets {pull}22646[22646]
- Fix rate metrics in Kafka broker metricset by using last minute rate instead of mean rate. {pull}22733[22733]
- Update config in `windows.yml` file. {issue}23027[23027]{pull}23327[23327]
- Fix metric grouping for windows/perfmon module {issue}23489[23489] {pull}23505[23505]

*Packetbeat*

- Fix SIP parser logic related to line length check. {pull}23411[23411]


*Winlogbeat*

- Protect against accessing an undefined variable in Security module. {pull}22937[22937]
- Add source.ip validation for event ID 4778 in the Security module. {issue}19627[19627]

==== Added

*Affecting all Beats*

- Add istiod metricset. {pull}21519[21519]
- Add support for OpenStack SSL metadata APIs in `add_cloud_metadata`. {pull}21590[21590]
- Add cloud.account.id for GCP into add_cloud_metadata processor. {pull}21776[21776]
- Add proxy metricset for istio module. {pull}21751[21751]
- Add kubernetes.node.hostname metadata of Kubernetes node. {pull}22189[22189]
- Enable always add_resource_metadata for Pods and Services of kubernetes autodiscovery. {pull}22189[22189]
- Add add_resource_metadata option setting (always enabled) for add_kubernetes_metadata setting. {pull}22189[22189]
- Add support for ephemeral containers in kubernetes autodiscover and `add_kubernetes_metadata`. {pull}22389[22389] {pull}22439[22439]
- Added support for wildcard fields and keyword fallback in beats setup commands. {pull}22521[22521]
- Fix polling node when it is not ready and monitor by hostname {pull}22666[22666]
- Add `expand_keys` option to `decode_json_fields` processor and `json` input, to recusively de-dot and expand json keys into hierarchical object structures {pull}22849[22849]
- Update k8s client and release k8s leader lock gracefully {pull}22919[22919]
- Improve event normalization performance {pull}22974[22974]
- Add tini as init system in docker images {pull}22137[22137]
- Added "detect_mime_type" processor for detecting mime types {pull}22940[22940]
- Added "add_network_direction" processor for determining perimeter-based network direction. {pull}23076[23076]
- Added new `rate_limit` processor for enforcing rate limits on event throughput. {pull}22883[22883]
- Allow node/namespace metadata to be disabled on kubernetes metagen and ensure add_kubernetes_metadata honors host {pull}23012[23012]
- Improve equals check. {pull}22778[22778]

*Auditbeat*

- Add several improvements for auditd module for improved ECS field mapping {pull}22647[22647]
- Add ECS 1.7 `configuration` categorization in certain events in auditd module. {pull}23000[23000]

*Filebeat*


- Adding support for Oracle Database Audit Logs {pull}21991[21991]
- Add max_number_of_messages config into s3 input. {pull}21993[21993]
- Add SSL option to checkpoint module {pull}19560[19560]
- Added support for MySQL Enterprise audit logs. {pull}22273[22273]
- Rename googlecloud module to gcp module. {pull}22214[22214]
- Rename awscloudwatch input to aws-cloudwatch. {pull}22228[22228]
- Rename google-pubsub input to gcp-pubsub. {pull}22213[22213]
- Copy tag names from MISP data into events. {pull}21664[21664]
- Added TLS JA3 fingerprint, certificate not_before/not_after, certificate SHA1 hash, and certificate subject fields to Zeek SSL dataset. {pull}21696[21696]
- Add platform logs in the azure filebeat module. {pull}22371[22371]
- Added `event.ingested` field to data from the Netflow module. {pull}22412[22412]
- Improve panw ECS url fields mapping. {pull}22481[22481]
- Improve Nats filebeat dashboard. {pull}22726[22726]
- Add support for UNIX datagram sockets in `unix` input. {issues}18632[18632] {pull}22699[22699]
- Add `http.request.mime_type` for Elasticsearch audit log fileset. {pull}22975[22975]
- Add new httpjson input features and mark old config ones for deprecation {pull}22320[22320]
- Add configuration option to set external and internal networks for panw panos fileset {pull}22998[22998]
- Add `subbdomain` fields for rsa2elk modules. {pull}23035[23035]
- Add subdomain enrichment for suricata/eve fileset. {pull}23011[23011]
- Add subdomain enrichment for zeek/dns fileset. {pull}23011[23011]
- Add `event.category` "configuration" to auditd module events. {pull}23010[23010]
- Add `event.category` "configuration" to gsuite module events. {pull}23010[23010]
- Add `event.category` "configuration" to o365 module events. {pull}23010[23010]
- Add `event.category` "configuration" to zoom module events. {pull}23010[23010]
- Add `network.direction` to auditd/log fileset. {pull}23041[23041]
- Add logic for external network.direction in sophos xg fileset {pull}22973[22973]
- Preserve AWS CloudTrail eventCategory in aws.cloudtrail.event_category. {issue}22776[22776] {pull}22805[22805]
- Add top_level_domain enrichment for suricata/eve fileset. {pull}23046[23046]
- Add top_level_domain enrichment for zeek/dns fileset. {pull}23046[23046]
- Add `observer.egress.zone` and `observer.ingress.zone` for cisco/asa and cisco/ftd filesets. {pull}23068[23068]
- Allow cisco/asa and cisco/ftd filesets to override network directionality based off of zones. {pull}23068[23068]
- Allow cef and checkpoint modules to override network directionality based off of zones {pull}23066[23066]
- Add `network.direction` to netflow/log fileset. {pull}23052[23052]
- Add the ability to override `network.direction` based on interfaces in Fortinet/firewall fileset. {pull}23072[23072]
- Add `network.direction` override by specifying `internal_networks` in gcp module. {pull}23081[23081]
- Migrate microsoft/defender_atp to httpjson v2 config {pull}23017[23017]
- Migrate microsoft/m365_defender to httpjson v2 config {pull}23018[23018]
- Migrate okta to httpjson v2 config {pull}23059[23059]
- Add support for Snyk Vulnerability and Audit API. {pull}22677[22677]
- Misp improvements: Migration to httpjson v2 config, pagination and deduplication ID {pull}23070[23070]
- Add Google Workspace module and mark Gsuite module as deprecated {pull}22950[22950]
- Mark m365 defender, defender atp, okta and google workspace modules as GA {pull}23113[23113]
- Added `alternative_host` option to google pubsub input {pull}23215[23215]

*Heartbeat*

- Add mime type detection for http responses. {pull}22976[22976]

*Metricbeat*

- Move s3_daily_storage and s3_request metricsets to use cloudwatch input. {pull}21703[21703]
- Duplicate system.process.cmdline field with process.command_line ECS field name. {pull}22325[22325]
- Add awsfargate module task_stats metricset to monitor AWS ECS Fargate. {pull}22034[22034]
- Add connection and route metricsets for nats metricbeat module to collect metrics per connection/route. {pull}22445[22445]
- Add unit file states to system/service {pull}22557[22557]
- `kibana` module: `stats` metricset no-longer collects usage-related data. {pull}22732[22732]
- Add more TCP states to Metricbeat system socket_summary. {pull}14347[14347]
- Add io.ops in fields exported by system.diskio. {pull}22066[22066]
- Adjust the Apache status fields in the fleet mode. {pull}22821[22821]
- Add AWS Fargate overview dashboard. {pull}22941[22941]
- Add process.state, process.cpu.pct, process.cpu.start_time and process.memory.pct. {pull}22845[22845]
- Move IIS module to GA and map fields. {issue}22609[22609] {pull}23024[23024]
- Apache: convert status.total_kbytes to status.total_bytes in fleet mode. {pull}23022[23022]
- Release MSSQL as GA {pull}23146[23146]

*Packetbeat*

- Add support for overriding the published index on a per-protocol/flow basis. {pull}22134[22134]
- Change build process for x-pack distribution {pull}21979[21979]
- Tuned the internal queue size to reduce the chances of events being dropped. {pull}22650[22650]
- Add support for "http.request.mime_type" and "http.response.mime_type". {pull}22940[22940]

*Winlogbeat*

- Add file.pe and process.pe fields to ProcessCreate & LoadImage events in Sysmon module. {issue}17335[17335] {pull}22217[22217]
- Add dns.question.subdomain fields for sysmon DNS events. {pull}22999[22999]
- Add additional event categorization for security and sysmon modules. {pull}22988[22988]
- Add dns.question.top_level_domain fields for sysmon DNS events. {pull}23046[23046]

*Elastic Log Driver*

- Add new winlogbeat security dashboard {pull}18775[18775]

==== Deprecated

*Filebeat*

- The experimental modules for Citrix Netscaler and Symantec Endpoint Protection have been removed.
As we continue to expand our coverage of common security data sources, we may consider supporting
Citrix Netscaler and Symantec Endpoint Protection in a future release. {issue}23129[23129] {pull}23130[23130]

==== Known Issue



[[release-notes-7.10.2]]
=== Beats version 7.10.2
https://github.com/elastic/beats/compare/v7.10.1\...v7.10.2[View commits]
Expand Down
Loading

0 comments on commit 5bc2d55

Please sign in to comment.