Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix panic due to no type #4331

Merged
merged 2 commits into from
May 17, 2017
Merged

Fix panic due to no type #4331

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
46d392a
Fix panic due to no type
May 17, 2017
71e472a
Added breaking + deprecation changelog and docs updates
May 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha1...master[Check the HEAD d

*Filebeat*
- Rename `input_type` field to `prospector.type` {pull}4294[4294]
- The `@metadata.type` field, added by the Logstash output, is now hardcoded to `doc` and will be removed in future versions. {pull}4331[4331].

*Heartbeat*

Expand Down Expand Up @@ -70,6 +71,8 @@ https://github.com/elastic/beats/compare/v6.0.0-alpha1...master[Check the HEAD d

*Affecting all Beats*

- The `@metadata.type` field, added by the Logstash output, is deprecated, hardcoded to `doc` and will be removed in future versions. {pull}4331[4331].

*Filebeat*
- Deprecate `input_type` prospector config. Use `type` config option instead. {pull}4294[4294]

Expand Down
13 changes: 8 additions & 5 deletions libbeat/docs/outputconfig.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ use in Logstash for indexing and filtering:
...
"@metadata": { <1>
"beat": "{beatname_lc}", <2>
"type": "<event type>" <3>
"type": "doc" <3>
}
}
------------------------------------------------------------------------------
Expand All @@ -329,7 +329,12 @@ events sent from Logstash. See the
for more about the `@metadata` field.
<2> The default is {beatname_lc}. To change this value, set the
<<logstash-index,`index`>> option in the {beatname_uc} config file.
<3> The value of `type` varies depending on the event type.
<3> The value of `type` is currently hardcoded to `doc`. It was used by previous
Logstash configs to set the type of the document in Elasticsearch.


WARNING: The `@metadata.type` field, added by the Logstash output, is
deprecated, hardcoded to `doc`, and will be removed in {beatname_uc} 7.0.

You can access this metadata from within the Logstash config file to set values
dynamically based on the contents of the metadata.
Expand All @@ -351,20 +356,18 @@ output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" <1>
document_type => "%{[@metadata][type]}" <2>
}
}
------------------------------------------------------------------------------
<1> `%{[@metadata][beat]}` sets the first part of the index name to the value
of the `beat` metadata field, and `%{+YYYY.MM.dd}` sets the second part of the
name to a date based on the Logstash `@timestamp` field. For example:
+{beatname_lc}-2017.03.29+.
<2> `%{[@metadata][type]}` sets the document type based on the value of the `type`
metadata field.

Events indexed into Elasticsearch with the Logstash configuration shown here
will be similar to events directly indexed by Beats into Elasticsearch.


==== Compatibility

This output works with all compatible versions of Logstash. See "Supported Beats Versions" in the https://www.elastic.co/support/matrix#show_compatibility[Elastic Support Matrix].
Expand Down
3 changes: 0 additions & 3 deletions libbeat/outputs/kafka/kafka_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,6 @@ func TestKafkaPublish(t *testing.T) {
single(common.MapStr{
"@timestamp": common.Time(time.Now()),
"host": "test-host",
"type": "log",
"message": id,
}),
},
Expand All @@ -80,7 +79,6 @@ func TestKafkaPublish(t *testing.T) {
single(common.MapStr{
"@timestamp": common.Time(time.Now()),
"host": "test-host",
"type": "log",
"message": id,
}),
},
Expand All @@ -91,7 +89,6 @@ func TestKafkaPublish(t *testing.T) {
randMulti(5, 100, common.MapStr{
"@timestamp": common.Time(time.Now()),
"host": "test-host",
"type": "log",
}),
},
{
Expand Down
2 changes: 0 additions & 2 deletions libbeat/outputs/kafka/partition_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,6 @@ func partTestSimple(N int, makeKey bool) partTestScenario {

event := common.MapStr{
"@timestamp": common.Time(ts),
"type": "test",
"message": randString(20),
}

Expand Down Expand Up @@ -263,7 +262,6 @@ func partTestHashInvariant(N int) partTestScenario {

event := common.MapStr{
"@timestamp": common.Time(ts),
"type": "test",
"message": randString(20),
}

Expand Down
9 changes: 3 additions & 6 deletions libbeat/outputs/logstash/json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,11 @@ func makeLogstashEventEncoder(beat string) (func(interface{}) ([]byte, error), e

buf.WriteRune('{')
if _, hasMeta := event["@metadata"]; !hasMeta {
typ := event["type"].(string)
buf.WriteString(`"@metadata":{"type":`)
encodeString(buf, typ)
buf.WriteString(`"@metadata":{"type":"doc","beat":`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if there is a type like we have in packetbeat? I think it is ok to just have it outside the meta info.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, i think it's safer to hardcode the metadata one to doc, as otherwise existing Logstash configs will put it in _type.

buf.Write(beatName)
buf.WriteString(`},`)
}

buf.WriteString(`,"beat":`)
buf.Write(beatName)
buf.WriteString(`},`)
err := enc.encodeKeyValues(event)
if err != nil {
logp.Err("jsonEncode failed with: %v", err)
Expand Down
1 change: 0 additions & 1 deletion libbeat/outputs/logstash/logstash_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,7 +266,6 @@ func testSendMessageViaLogstash(t *testing.T, name string, tls bool) {
event := outputs.Data{Event: common.MapStr{
"@timestamp": common.Time(time.Now()),
"host": "test-host",
"type": "log",
"message": "hello world",
}}
ls.PublishEvent(nil, testOptions, event)
Expand Down