[metricbeat] Add windows memory data to docker/memory #12172

fearful-symmetry · 2019-05-10T17:48:33Z

Leaving this as a draft since I'm not 100% sure how to handle vastly different stats across various operating systems. Will probably add more commits later as I tinker around.

This addresses #11971 by adding the memory stats that are only available on windows. The change itself is fairly simple, but I have no practical experience with docker/go on windows, so I would appreciate help from any other folks. I added @narph to the reviewers list as well.

metricbeat/module/docker/memory/_meta/fields.yml

jsoriano

This looks quite good to me if we cannot consider that Windows metrics are more or less equivalent to Linux ones.

jsoriano · 2019-05-11T19:52:44Z

metricbeat/module/docker/memory/helper.go

@@ -66,5 +70,9 @@ func (s *MemoryService) getMemoryStats(myRawStat docker.Stat, dedot bool) Memory
 		TotalRssP: float64(totalRSS) / float64(myRawStat.Stats.MemoryStats.Limit),
 		Usage:     myRawStat.Stats.MemoryStats.Usage,
 		UsageP:    float64(myRawStat.Stats.MemoryStats.Usage) / float64(myRawStat.Stats.MemoryStats.Limit),


I wonder if these percentages are NaN because of 0/0 on Windows, we should probably check for these limits not being 0. Though it wouldn't be so important now if we send different fields for Windows.

jsoriano · 2019-05-11T20:02:43Z

metricbeat/module/docker/memory/data.go

+				"pct":   memoryData.UsageP,
+				"max":   memoryData.MaxUsage,
+			},
+		}


I am fine with this approach if we cannot consider these metrics equivalent to others on Linux. The private working set could be similar to the RSS, and commit and commit peak could be similar to usage and max usage, but not sure.
Probably @narph or @andrewkroh can shed more light about this.
If they are equivalent or similar enough we should send them in the same fields.

Commit - memory space the application has reserved for use, so this will be allocated but not necessarily used. (This is composed of main memory (RAM) and disk (pagefiles).)
CommitPeak - the highest amount that the commit has reached since the operating system was last started or session.
PrivateWorkingSet - the amount of memory used by a process.
Not sure the memory stats are matching here so I would go for separate naming, we might want to look into that as well for the other memory related metricsets

jsoriano · 2019-05-11T20:04:36Z

metricbeat/module/docker/memory/data.go

+		fields = common.MapStr{
+			"commit":              memoryData.Commit,
+			"commit_peak":         memoryData.CommitPeak,
+			"private_working_set": memoryData.PrivateWorkingSet,


Follow the structure of Linux metrics, in case some day docker supports limits in Windows and we can calculate percentages.

fearful-symmetry · 2019-05-13T12:59:50Z

Adjusted things a tad so now we have commited bytes in usage at least.

Regarding @jsoriano 's comments: I agree that we should make these more consistent, and that things like rss on linux and Private Working Set on Windows are probably more or less equivalent. My concern is finding some sort of naming scheme that doesn't conflate the two, as technically RSS is a linux concept, not a windows one. I worry that if we're not careful and we just cram metrics together, we might confuse people with linux-sounding Windows metrics and so on. I adjusted the format slightly so windows has usage now, but I would like input from some windows experts to see what else we can do to smooth out the naming.

jsoriano · 2019-05-16T17:24:39Z

I worry that if we're not careful and we just cram metrics together, we might confuse people with linux-sounding Windows metrics and so on.

Yeah, that's actually a point to discuss. The good thing of having all the equivalent metrics in the same fields is that you can build dashboards or queries showing this information together more easily. One option can be to keep OS-specific fields, but also duplicate these values into common fields. This could be added afterwars in any case.

fearful-symmetry · 2019-05-16T19:52:27Z

One option can be to keep OS-specific fields, but also duplicate these values into common fields. This could be added afterwars in any case.

That's one option, yah! We did something similar with the RAID metricset, where we had a detail section, and a normalized section for RAID states.

narph · 2019-05-23T08:44:27Z

@fearful-symmetry , I have not commented on the initial naming as I found it to be clear enough.

"commit":              memoryData.Commit,
"commit_peak":         memoryData.CommitPeak,
"private_working_set": memoryData.PrivateWorkingSet,

You could go for Commit size instead of Commit here.

I would stir away from usage or memory usage when talking about private working set as there are more types here (of memory usage): working set, private working set, working set will also include the memory shared by other processes.
https://answers.microsoft.com/en-us/windows/forum/windows_10-performance/physical-and-virtual-memory-in-windows-10/e36fb5bc-9ac8-49af-951c-e7d39b979938
http://blogs.microsoft.co.il/sasha/2016/01/05/windows-process-memory-usage-demystified/

@jsoriano, Is there a way we can store only the os specific information and utilize the field alias to create a common visualization/dashboard for the user instead of going for duplication? (I have not tried this)

fearful-symmetry · 2019-05-23T14:27:23Z

Thanks for your added input @narph.
I definitely like the alias idea, but I have absolutely no idea how to do that.

jsoriano · 2019-05-24T08:36:02Z

@jsoriano, Is there a way we can store only the os specific information and utilize the field alias to create a common visualization/dashboard for the user instead of going for duplication? (I have not tried this)

I don't think we can use the alias field for this. Aliases are defined in the index mapping (not per document), and can only be defined in a 1 to 1 way, so we couldn't create an alias pointing to several OS-specific metrics.

I'd keep both OS-specific metrics in the events by now as they are different things (thanks @narph for the clarifications!), and we can evaluate later what to do if we create visualizations for this. Maybe this can be solved in the visualization or the dashboard itself.

fearful-symmetry · 2019-05-24T14:33:59Z

I'd keep both OS-specific metrics in the events by now

@jsoriano So we're good as-is? Should I take this out of draft?

jsoriano · 2019-05-24T16:52:48Z

I'd keep both OS-specific metrics in the events by now

@jsoriano So we're good as-is? Should I take this out of draft?

Yeah, I think this can go for review. Only let me know what you think about this comment https://github.com/elastic/beats/pull/12172/files#r283109849

It would be to change the metrics to something like this:

			"usage": common.MapStr{
				"commit": common.MapStr{
                                    "total": memoryData.Commit,
				    "peak":  memoryData.CommitPeak,
                                },
			},
			"private_working_set": common.MapStr{
                            "total": memoryData.PrivateWorkingSet,
                        },

In case some day we add also percentages or other aggregations.

fearful-symmetry · 2019-05-24T16:55:48Z

Ahh, alright, I wasn't 100% sure what you meant by that, so I figured the usage dict was what you meant. Yah, I like that idea.

fearful-symmetry · 2019-05-24T17:34:06Z

Okay, I think I'm confusing myself with the fields.yml format, but it should be good now.

jsoriano · 2019-05-28T10:58:39Z

@fearful-symmetry are you going to do the change for commit and commit peak too?

It would be to change the metrics to something like this:

			"usage": common.MapStr{
				"commit": common.MapStr{
                                    "total": memoryData.Commit,
				    "peak":  memoryData.CommitPeak,
                                },
			},

fearful-symmetry · 2019-05-28T13:05:21Z

Oh, is that how we want to do it? I figured we could just use the existing pct type in the commit object.

fearful-symmetry · 2019-05-28T14:47:48Z

Okay. Hopefully this makes sense now.

We are discussing offline about the proper location of the commit metrics

jsoriano

A couple of comments on documentation, for the rest it LGTM, sorry for the back and forth on field names!

metricbeat/module/docker/memory/_meta/fields.yml

fearful-symmetry added :Windows Metricbeat Metricbeat Team:Integrations Label for the Integrations team labels May 10, 2019

fearful-symmetry requested review from narph and a team May 10, 2019 17:48

fearful-symmetry self-assigned this May 10, 2019

fearful-symmetry changed the title ~~[metricbeat] WIP of adding windows memory data~~ [metricbeat] Add windows memory data to docker/memory May 10, 2019

kaiyan-sheng reviewed May 10, 2019

View reviewed changes

metricbeat/module/docker/memory/_meta/fields.yml Show resolved Hide resolved

jsoriano reviewed May 11, 2019

View reviewed changes

fearful-symmetry added 3 commits May 13, 2019 15:48

WIP of adding windows memory data

ef9c7f4

just send different events for windows/linux

80af3f1

adding commit data to usage, changed type to bytes

bd22180

fearful-symmetry force-pushed the docker-windows-add-memory-data branch from 2ab8496 to bd22180 Compare May 13, 2019 19:48

andresrc added the [zube]: In Review label May 20, 2019

fearful-symmetry added 2 commits May 24, 2019 12:22

move private working set to a new map

8638336

fix up field types/format

b18cac7

fearful-symmetry marked this pull request as ready for review May 24, 2019 18:58

fearful-symmetry requested a review from a team as a code owner May 24, 2019 18:58

fearful-symmetry requested a review from a team May 24, 2019 18:58

change fields of commit data

c93a44b

jsoriano previously approved these changes May 28, 2019

View reviewed changes

change fields of commit data, one last time

39d440f

jsoriano reviewed May 29, 2019

View reviewed changes

metricbeat/module/docker/memory/_meta/fields.yml Outdated Show resolved Hide resolved

metricbeat/module/docker/memory/_meta/fields.yml Outdated Show resolved Hide resolved

one last cleanup of fields.yml

3f742c7

jsoriano approved these changes May 29, 2019

View reviewed changes

narph approved these changes May 29, 2019

View reviewed changes

fearful-symmetry merged commit 1c75c66 into elastic:master May 30, 2019

zube bot added [zube]: Done and removed [zube]: In Review labels May 30, 2019

andresrc removed the [zube]: Done label Jun 3, 2019

jsoriano mentioned this pull request Apr 29, 2021

Use working set bytes if usage bytes is zero #25428

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metricbeat] Add windows memory data to docker/memory #12172

[metricbeat] Add windows memory data to docker/memory #12172

fearful-symmetry commented May 10, 2019 •

edited by andresrc

Loading

jsoriano left a comment

jsoriano May 11, 2019

jsoriano May 11, 2019

narph May 20, 2019

jsoriano May 11, 2019

narph May 20, 2019

fearful-symmetry commented May 13, 2019

jsoriano commented May 16, 2019

fearful-symmetry commented May 16, 2019

narph commented May 23, 2019

fearful-symmetry commented May 23, 2019

jsoriano commented May 24, 2019 •

edited

Loading

fearful-symmetry commented May 24, 2019 •

edited

Loading

jsoriano commented May 24, 2019

fearful-symmetry commented May 24, 2019

fearful-symmetry commented May 24, 2019

jsoriano commented May 28, 2019

fearful-symmetry commented May 28, 2019

fearful-symmetry commented May 28, 2019

jsoriano left a comment

[metricbeat] Add windows memory data to docker/memory #12172

[metricbeat] Add windows memory data to docker/memory #12172

Conversation

fearful-symmetry commented May 10, 2019 • edited by andresrc Loading

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano May 11, 2019

Choose a reason for hiding this comment

jsoriano May 11, 2019

Choose a reason for hiding this comment

narph May 20, 2019

Choose a reason for hiding this comment

jsoriano May 11, 2019

Choose a reason for hiding this comment

narph May 20, 2019

Choose a reason for hiding this comment

fearful-symmetry commented May 13, 2019

jsoriano commented May 16, 2019

fearful-symmetry commented May 16, 2019

narph commented May 23, 2019

fearful-symmetry commented May 23, 2019

jsoriano commented May 24, 2019 • edited Loading

fearful-symmetry commented May 24, 2019 • edited Loading

jsoriano commented May 24, 2019

fearful-symmetry commented May 24, 2019

fearful-symmetry commented May 24, 2019

jsoriano commented May 28, 2019

fearful-symmetry commented May 28, 2019

fearful-symmetry commented May 28, 2019

jsoriano left a comment

Choose a reason for hiding this comment

fearful-symmetry commented May 10, 2019 •

edited by andresrc

Loading

jsoriano commented May 24, 2019 •

edited

Loading

fearful-symmetry commented May 24, 2019 •

edited

Loading