UDP prospector type with plain harvester #4452

ruflin · 2017-06-02T22:06:40Z

Opening this prospector mainly to show this is WIP and so I don't forget about it :-)

TODO

Changelog
Docs
Tests
Code cleanup

And even looks as following

{
  "@timestamp": "2017-05-26T11:57:17.68711727+02:00",
  "beat": {
    "hostname": "ruflin",
    "name": "ruflin",
    "version": "6.0.0-alpha2"
  },
  "message": "Hello, World!99",
  "prospector": {
    "harvester": "plain",
    "type": "udp"
  }
}

ruflin · 2017-06-07T13:24:16Z

filebeat/prospector/udp/harvester.go

+	defer listener.Close()
+
+	//logp.Info("Now listening for logs via %s on %s", ll.config.Protocol, address)
+	buffer := make([]byte, 1000)


@hartfordfive Any recommendation on the value here?

@ruflin I realize in theory you could have up to 65507 bytes of data in IPv4 (removing UDP and IP header overhead of 20 and 8 bytes respectively) although I know that you risk having fragmentation of this because of MTU (which I guess is generally between 1400 and 1500 - not taking into account jumbo frames). Considering this should only be done locally on the server, i figured this shouldn't happen, so thus why I chose a slightly larger 2048 or 4096 bytes buffer size.

I know many applications tend to limit to <= 1024 bytes for a UDP buffer (especially for cross-host communication), although from what I've tested so far, I haven't had messages that were cut off, but I'm aware this could happen, especially if you're going form one host to another even on the same rack. You could potentially stick with the 1024 byte buffer as a default but allow users to specify the port and the buffer size via the prospector config parameters.

I think it would maybe be nice to have someone else give some feedback on this, especially from the networking stack side to confirm if this is a reasonable assumption or not. What do you think?

Thanks for sharing the thoughts. I set it 1024 for the moment and made it configurable. Time will tell and we can still change the default.

ruflin · 2017-06-08T13:44:51Z

filebeat/prospector/udp/harvester.go

+		event := common.MapStr{
+			"message": string(buffer[:length]),
+			"prospector": common.MapStr{
+				"harvester": "plain",


@tsg I'm asking myself if this should be code? Other options here could be json.

Do we need a "harverster" key? Maybe we add it when we have more of them.

I'm thinking that, for example, JSON can be accomplished with the processor, so I'm not sure harvester support for it is needed.

We don't need it at the moment an can still add it later.

tsg · 2017-06-20T12:01:23Z

filebeat/_meta/common.full.p2.yml

+#- type: udp
+
+  # Maximum size of each message sent over udp
+  #max_message_size: 1024


Two things here:

I think it would be more clear if it would say "Maximum size of the message received over UDP".

1024 seems low, UDP can go up to 65K. How about 10K, which should be large enough for most usecases but still small enough to not be noticeable in the mem usage?

I set it to 10240 now. 1024 was based on the discussion here.

tsg

Left a minor comment.

exekias

Left a comment on default value, LGTM apart from that

exekias · 2017-06-23T15:15:21Z

filebeat/prospector/udp/config.go

+	ForwarderConfig: harvester.ForwarderConfig{
+		Type: "udp",
+	},
+	MaxMessageSize: 1024,


Should this be 10240?

changed, thanks for spotting.

@timestamp

…get about it :-) TODO * [x] Changelog * [ ] Docs * [x] Tests * [x] Code cleanup And even looks as following ``` { "@timestamp": "2017-05-26T11:57:17.68711727+02:00", "beat": { "hostname": "ruflin", "name": "ruflin", "version": "6.0.0-alpha2" }, "message": "Hello, World!99", "prospector": { "harvester": "plain", "type": "udp" } } ```

exekias · 2017-06-23T15:26:30Z

WFG

dedemorton · 2017-09-18T20:55:05Z

Can I remove the needs_docs tag? Is #4940 sufficient, or is additional doc required?

ruflin · 2017-09-20T07:44:54Z

@dedemorton I was hoping for more docs that also shows how to configure the host for example. But I don't know how this fits into our current doc model of prospectors. Perhaps worth to discuss quickly over zoom?

dedemorton · 2018-01-22T18:20:30Z

I'm going to remove the needs_docs label here in favor of tracking the work in #5632

ruflin added Filebeat Filebeat in progress Pull request is currently in progress. labels Jun 2, 2017

ruflin force-pushed the udp-prospector branch from 7c51d2e to 366fd97 Compare June 2, 2017 22:09

ruflin commented Jun 7, 2017

View reviewed changes

ruflin force-pushed the udp-prospector branch from 366fd97 to 4e02617 Compare June 8, 2017 13:43

ruflin commented Jun 8, 2017

View reviewed changes

ruflin force-pushed the udp-prospector branch 3 times, most recently from ce9a97e to f0ce934 Compare June 14, 2017 07:19

ruflin changed the title ~~[WIP] UDP prospector type with plain harvester~~ UDP prospector type with plain harvester Jun 14, 2017

ruflin force-pushed the udp-prospector branch from f0ce934 to a428449 Compare June 14, 2017 07:23

ruflin added review and removed in progress Pull request is currently in progress. labels Jun 14, 2017

ruflin force-pushed the udp-prospector branch from a428449 to a2d3d06 Compare June 19, 2017 19:37

tsg reviewed Jun 20, 2017

View reviewed changes

tsg approved these changes Jun 20, 2017

View reviewed changes

elastic deleted a comment from codecov bot Jun 20, 2017

ruflin force-pushed the udp-prospector branch 2 times, most recently from 42100f1 to cfac655 Compare June 23, 2017 05:50

ruflin added the needs_docs label Jun 23, 2017

exekias approved these changes Jun 23, 2017

View reviewed changes

ruflin force-pushed the udp-prospector branch from cfac655 to ec16164 Compare June 23, 2017 15:18

Merge branch 'master' into udp-prospector

a9d8ec5

exekias merged commit 51d8637 into elastic:master Jun 23, 2017

ruflin deleted the udp-prospector branch June 27, 2017 06:18

dedemorton mentioned this pull request Jul 25, 2017

Beats doc updates in 6.0 #4540

Closed

42 tasks

dedemorton mentioned this pull request Aug 18, 2017

[Docs] Add udp prospector to list of types #4940

Merged

dedemorton mentioned this pull request Jan 22, 2018

Beats doc updates in 6.x #5632

Closed

37 tasks

dedemorton removed the needs_docs label Jan 22, 2018

dedemorton mentioned this pull request Mar 13, 2018

Refactor input docs #6537

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UDP prospector type with plain harvester #4452

UDP prospector type with plain harvester #4452

ruflin commented Jun 2, 2017 •

edited

Loading

ruflin Jun 7, 2017

hartfordfive Jun 7, 2017 •

edited

Loading

ruflin Jun 8, 2017

ruflin Jun 8, 2017

tsg Jun 20, 2017

ruflin Jun 20, 2017

tsg Jun 20, 2017

ruflin Jun 20, 2017

tsg left a comment

exekias left a comment

exekias Jun 23, 2017

ruflin Jun 23, 2017

exekias commented Jun 23, 2017

dedemorton commented Sep 18, 2017

ruflin commented Sep 20, 2017

dedemorton commented Jan 22, 2018

UDP prospector type with plain harvester #4452

UDP prospector type with plain harvester #4452

Conversation

ruflin commented Jun 2, 2017 • edited Loading

Choose a reason for hiding this comment

hartfordfive Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg left a comment

Choose a reason for hiding this comment

exekias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exekias commented Jun 23, 2017

dedemorton commented Sep 18, 2017

ruflin commented Sep 20, 2017

dedemorton commented Jan 22, 2018

ruflin commented Jun 2, 2017 •

edited

Loading

hartfordfive Jun 7, 2017 •

edited

Loading