Add more info about flows to the doc #4197

dedemorton · 2017-05-03T21:09:30Z

First cut of the content on flows.

I also moved up a couple of items in the TOC because they seemed important.

Please respond to the comments that I've added inline.

monicasarbu · 2017-05-03T21:32:47Z

packetbeat/docs/flows.asciidoc

+
+//REVIEWERS: Is it OK for me to show all these IP addresses (not sure what might be sensitive)? I don't worry about that as much as I should, but now that I know Mr. Robot is watching...perhaps I should be more careful. Do we have a better image to use? This one is actually from a v5.3.
+
+image:./images/flows.png[]


@dedemorton I am not able to find flows.png. Did you forget to add it to the PR?

@monicasarbu whoops! just added it.

monicasarbu · 2017-05-03T21:36:03Z

packetbeat/docs/flows.asciidoc

+Here’s an example of a flow event sent by Packetbeat. See
+<<exported-fields-flows_event>> for a description of each field.
+
+//REVIEWERS: As always, I sit on the fence about whether to show the output event because it may change.


Maybe we can include the content of the https://github.com/elastic/beats/blob/master/metricbeat/module/system/diskio/_meta/data.json. Unfortunately this exists only for the Metricbeat metricsets, and not for all Beats.

monicasarbu · 2017-05-03T21:49:16Z

packetbeat/docs/flows.asciidoc

+
+* IP/TCP
+* UDP
+* SSL


I think SSL is not done yet. There is an open issue for it: #3604

monicasarbu · 2017-05-03T21:49:45Z

packetbeat/docs/flows.asciidoc

+* UDP
+* SSL
+
+//REVEIWER: Not sure this list is complete. should it also say Application? Anything else?


No for Application.

andrewkroh · 2017-05-03T21:33:59Z

packetbeat/docs/flows.asciidoc

+flow.
+
+Packetbeat collects and reports statistics per flow on the following network
+layers:


When I hear network layers I think of the OSI model. Flows collect the mac address and vlan ID from layer 2 (data-link), the ip address from layer 3, and udp/tcp port info from layer 4 (transport).

It doesn't do anything higher level layers yet (like SSL).

@andrewkroh Do you think users will want to know this level of detail? If not, then I can remove lines 16-21.

I think it's enough to say it collects info up to and including the transport layer. The details aren't that important since they can jump over to the exported field docs.

andrewkroh · 2017-05-03T21:40:40Z

packetbeat/docs/flows.asciidoc

+Here’s an example of a flow event sent by Packetbeat. See
+<<exported-fields-flows_event>> for a description of each field.
+
+//REVIEWERS: As always, I sit on the fence about whether to show the output event because it may change.


I like seeing the output in docs.

But I would only show the contents of the _source field (the other stuff is specific to ES). To be consistent with some other example docs, change the hostname and name to host.example.com. You could swap out the IP addresses with ones reserved for documentation purposes (see IPv4 Address Blocks Reserved for Documentation).

andrewkroh · 2017-05-03T21:43:40Z

packetbeat/docs/flows.asciidoc

+For each flow, Packetbeat reports the number of packets and the total number of
+bytes sent from the source to the destination. Each flow event also contains
+information about the source and destination hosts, such as their IP address and
+location. For bi-directional flows, Packetbeat reports statistics for the reverse


Location is not provided by Beat, but we do have the field in our index template. We should explain how to get these fields populated with data using geoip lookups.

@andrewkroh I've created a PR to track this work because I don't have time to do it right now. #4238

andrewkroh · 2017-05-03T21:50:54Z

packetbeat/docs/flows.asciidoc

+      "name": "My-MacBook-Pro.local",
+      "version": "{stack-version}"
+    },
+    "connection_id": "AQAAAAAAAAA=",


I think the field documentation needs more details too, but we probably need urso to help us out. Some questions I had about the data:

What is connection_id and icmp_id and where to they come from? How can I use them?

What data contributes to the flow_id? Under what conditions would a flow ID be reused (or when can collisions occur).

Clarify (and confirm) that net_bytes_total includes the total amount of bytes across all layers. Does this count retransmits?

Clarify if net_packets_total includes retransmits.

transport does not appear in the field docs. What are the possible values of the field?

I've created a separate issue to track this list of issues: #4236

andrewkroh · 2017-05-03T21:57:23Z

packetbeat/docs/flows.asciidoc

+* IP/TCP
+* UDP
+* SSL
+


Somewhere in here we should explain the final flag and how the packetbeat sends intermediate reports about each flow that it is tracking with the final = false, then when the flow completes it sends one last event with final = true. Users that wish to aggregate sums of traffic will want to know this because they should be filtering on final:true (or using some other technique to get only the latest update from each flow).

And if period: -1 is in the config, then the intermediate reports are disabled.

Just wanted to point out the is not period: -1 the correct setting is period: -1s

It is also incorrect in the existing docs. I ran into this a couple weeks ago.

@robcowart Thanks and could you please open a new issue for this. Otherwise I'm afraid it might get lost in the noise. Plus DeDe is on vacation this week so even more change it might not be seen right away by her.

dedemorton · 2017-05-06T01:02:02Z

This PR is ready for a final review.

andrewkroh

I modified the JSON output a bit to drop the _source key and some other ES/Kibana specific fields.

* Add more info about Packetbeat flows to the docs. * Add dashboard image.

Add more info about flows to the doc

aba4fb7

dedemorton added docs review labels May 3, 2017

dedemorton mentioned this pull request May 3, 2017

Beats Doc Updates for 5.4 #4021

Closed

9 tasks

monicasarbu reviewed May 3, 2017

View reviewed changes

andrewkroh reviewed May 3, 2017

View reviewed changes

Add image

3daf980

dedemorton mentioned this pull request May 5, 2017

Improve the field documentation about packetbeat flows #4236

Closed

Add fixes from the review

4d7495f

Remove extraneous fields from example JSON output

cb62e6e

andrewkroh approved these changes May 9, 2017

View reviewed changes

andrewkroh merged commit 1af914d into elastic:master May 9, 2017

dedemorton added a commit to dedemorton/beats that referenced this pull request May 17, 2017

Add more info about Packetbeat flows to the doc (elastic#4197)

47081ac

* Add more info about Packetbeat flows to the docs. * Add dashboard image.

dedemorton mentioned this pull request May 17, 2017

Backport: Cherrypick #4140, #4320, and #4197 into 5.4. #4342

Merged

ruflin pushed a commit that referenced this pull request May 18, 2017

Add more info about Packetbeat flows to the doc (#4197)

ccd3563

* Add more info about Packetbeat flows to the docs. * Add dashboard image.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more info about flows to the doc #4197

Add more info about flows to the doc #4197

dedemorton commented May 3, 2017 •

edited

Loading

monicasarbu May 3, 2017

dedemorton May 3, 2017

monicasarbu May 3, 2017

monicasarbu May 3, 2017

monicasarbu May 3, 2017

andrewkroh May 3, 2017

dedemorton May 3, 2017

andrewkroh May 3, 2017 •

edited

Loading

andrewkroh May 3, 2017

andrewkroh May 3, 2017

dedemorton May 6, 2017

andrewkroh May 3, 2017 •

edited

Loading

dedemorton May 5, 2017

andrewkroh May 3, 2017 •

edited

Loading

robcowart May 10, 2017

andrewkroh May 10, 2017

dedemorton commented May 6, 2017

andrewkroh left a comment


		//REVIEWERS: Is it OK for me to show all these IP addresses (not sure what might be sensitive)? I don't worry about that as much as I should, but now that I know Mr. Robot is watching...perhaps I should be more careful. Do we have a better image to use? This one is actually from a v5.3.

		image:./images/flows.png[]

Add more info about flows to the doc #4197

Add more info about flows to the doc #4197

Conversation

dedemorton commented May 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh May 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh May 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh May 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedemorton commented May 6, 2017

andrewkroh left a comment

Choose a reason for hiding this comment

dedemorton commented May 3, 2017 •

edited

Loading

andrewkroh May 3, 2017 •

edited

Loading

andrewkroh May 3, 2017 •

edited

Loading

andrewkroh May 3, 2017 •

edited

Loading