Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[metricbeat] Add windows memory data to docker/memory #12172

Merged
merged 8 commits into from
May 30, 2019
Merged

[metricbeat] Add windows memory data to docker/memory #12172

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ef9c7f4
WIP of adding windows memory data
fearful-symmetry May 10, 2019
80af3f1
just send different events for windows/linux
fearful-symmetry May 10, 2019
bd22180
adding commit data to usage, changed type to bytes
fearful-symmetry May 13, 2019
8638336
move private working set to a new map
fearful-symmetry May 24, 2019
b18cac7
fix up field types/format
fearful-symmetry May 24, 2019
c93a44b
change fields of commit data
fearful-symmetry May 28, 2019
39d440f
change fields of commit data, one last time
fearful-symmetry May 28, 2019
3f742c7
one last cleanup of fields.yml
fearful-symmetry May 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4539,6 +4539,49 @@ Memory metrics.



[float]
== commit fields

Committed bytes on Windows



*`docker.memory.commit.total`*::
+
--
type: long

format: bytes

Total bytes


--

*`docker.memory.commit.peak`*::
+
--
type: long

format: bytes

peak bytes


--

*`docker.memory.private_working_set.total`*::
+
--
type: long

format: bytes

private working sets on Windows


--

*`docker.memory.fail.count`*::
+
--
Expand Down Expand Up @@ -4599,6 +4642,18 @@ Usage memory stats.



*`docker.memory.usage.commit_peak`*::
+
--
type: long

format: bytes

Peak committed bytes on Windows


--

*`docker.memory.usage.max`*::
+
--
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/module/docker/fields.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 26 additions & 1 deletion metricbeat/module/docker/memory/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,26 @@
Memory metrics.
release: ga
fields:

- name: commit
type: group
description: >
Committed bytes on Windows
fields:
- name: total
type: long
format: bytes
description: >
Total bytes
- name: peak
type: long
format: bytes
description: >
peak bytes
fearful-symmetry marked this conversation as resolved.
Show resolved Hide resolved
- name: private_working_set.total
type: long
exekias marked this conversation as resolved.
Show resolved Hide resolved
format: bytes
description: >
private working sets on Windows
- name: fail.count
type: scaled_float
description: >
Expand Down Expand Up @@ -34,6 +53,11 @@
description: >
Usage memory stats.
fields:
- name: commit_peak
type: long
format: bytes
description: >
Peak committed bytes on Windows
fearful-symmetry marked this conversation as resolved.
Show resolved Hide resolved
- name: max
type: long
format: bytes
Expand All @@ -49,3 +73,4 @@
format: bytes
description: >
Total memory usage.

43 changes: 29 additions & 14 deletions metricbeat/module/docker/memory/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,20 +29,35 @@ func eventsMapping(r mb.ReporterV2, memoryDataList []MemoryData) {
}

func eventMapping(r mb.ReporterV2, memoryData *MemoryData) {
fields := common.MapStr{
"fail": common.MapStr{
"count": memoryData.Failcnt,
},
"limit": memoryData.Limit,
"rss": common.MapStr{
"total": memoryData.TotalRss,
"pct": memoryData.TotalRssP,
},
"usage": common.MapStr{
"total": memoryData.Usage,
"pct": memoryData.UsageP,
"max": memoryData.MaxUsage,
},

//if we have windows memory data, just report windows stats
var fields common.MapStr
if memoryData.Commit+memoryData.CommitPeak+memoryData.PrivateWorkingSet > 0 {
fields = common.MapStr{
"commit": common.MapStr{
"total": memoryData.Commit,
"peak": memoryData.CommitPeak,
},
"private_working_set": common.MapStr{
"total": memoryData.PrivateWorkingSet,
},
}
} else {
fields = common.MapStr{
"fail": common.MapStr{
"count": memoryData.Failcnt,
},
"limit": memoryData.Limit,
"rss": common.MapStr{
"total": memoryData.TotalRss,
"pct": memoryData.TotalRssP,
},
"usage": common.MapStr{
"total": memoryData.Usage,
"pct": memoryData.UsageP,
"max": memoryData.MaxUsage,
},
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with this approach if we cannot consider these metrics equivalent to others on Linux. The private working set could be similar to the RSS, and commit and commit peak could be similar to usage and max usage, but not sure.
Probably @narph or @andrewkroh can shed more light about this.
If they are equivalent or similar enough we should send them in the same fields.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit - memory space the application has reserved for use, so this will be allocated but not necessarily used. (This is composed of main memory (RAM) and disk (pagefiles).)
CommitPeak - the highest amount that the commit has reached since the operating system was last started or session.
PrivateWorkingSet - the amount of memory used by a process.
Not sure the memory stats are matching here so I would go for separate naming, we might want to look into that as well for the other memory related metricsets

}

r.Event(mb.Event{
Expand Down
8 changes: 8 additions & 0 deletions metricbeat/module/docker/memory/helper.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ type MemoryData struct {
TotalRssP float64
Usage uint64
UsageP float64
//Windows-only memory stats
Commit uint64
CommitPeak uint64
PrivateWorkingSet uint64
}

// MemoryService is placeholder for the the memory stat parsers
Expand Down Expand Up @@ -66,5 +70,9 @@ func (s *MemoryService) getMemoryStats(myRawStat docker.Stat, dedot bool) Memory
TotalRssP: float64(totalRSS) / float64(myRawStat.Stats.MemoryStats.Limit),
Usage: myRawStat.Stats.MemoryStats.Usage,
UsageP: float64(myRawStat.Stats.MemoryStats.Usage) / float64(myRawStat.Stats.MemoryStats.Limit),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if these percentages are NaN because of 0/0 on Windows, we should probably check for these limits not being 0. Though it wouldn't be so important now if we send different fields for Windows.

//Windows memory statistics
Commit: myRawStat.Stats.MemoryStats.Commit,
CommitPeak: myRawStat.Stats.MemoryStats.CommitPeak,
PrivateWorkingSet: myRawStat.Stats.MemoryStats.PrivateWorkingSet,
}
}