Reduce memory usage of elasticsearch.index metricset

[ycombinator]

What does this PR do?

This PR reduces the memory usage of the elasticsearch.index metricset by parsing the JSON API response from Elasticsearch into structs instead of map[string]interface{}.

Why is it important?

To reduce the overall memory footprint of Metricbeat which makes it less susceptible to the OOM Killer and such.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works Benchmarking test added in Benchmark API response parsing code #17046 is used to prove the improvement in memory usage after this PR.

How to test this PR locally

cd $GOPATH/src/github.com/elastic/beats
go test -bench=. -benchmem ./metricbeat/module/elasticsearch/index/

Sample benchmark results

Before this PR

goos: darwin
goarch: amd64
pkg: github.com/elastic/beats/v7/metricbeat/module/elasticsearch/index
BenchmarkParseAPIResponse-8           25          44391131 ns/op        13464952 B/op     207889 allocs/op
PASS
ok      github.com/elastic/beats/v7/metricbeat/module/elasticsearch/index       1.585s

In terms of memory, the API response parsing implementation before this PR consumed 13.46 MB.

After this PR

goos: darwin
goarch: amd64
pkg: github.com/elastic/beats/v7/metricbeat/module/elasticsearch/index
BenchmarkParseAPIResponse-8           33          31846457 ns/op          596808 B/op      12461 allocs/op
PASS
ok      github.com/elastic/beats/v7/metricbeat/module/elasticsearch/index       1.527s

In terms of memory, the API response parsing implementation after this PR consumed 0.59 MB.

That's an improvement of approximately 1:22!

Related issues

• Fixes Reduce memory usage of elasticsearch/index metricset #16503.
• Depends on Benchmark API response parsing code #17046.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[urso]

How big of a problem is it that we have to use a manual parsing approach? We still produce some structured contents. Plus Metricbeat does not acquire these metrics like every 1s. Do we have numbers like before/after?

[ycombinator]

Sorry @urso, please ignore this PR for now as it's in draft state. I'm still in the process of figuring out before/after memory usage numbers to check how much this PR would buy us.

@andresrc Can we make sure draft PRs are not given the [zube]: In Review label? I've removed the one for this PR.

[ycombinator]

Chatted with @urso about this off-PR. We already know that the high memory consumption is coming from parsing the JSON from the Stats API response. He noticed that we were parsing into a map[string]map[string]interface{} here:

beats/metricbeat/module/elasticsearch/index/data.go

Line 34 in 95626b8

Indices map[string]map[string]interface{} `json:"indices"`

He suggested parsing into structs instead, as they have a much lower memory overhead. He also suggested running microbenchmarks to test if there's actually a meaningful gain from such a change.

I did that and here are the results:

go test -bench=. -benchtime=5s -benchmem module/elasticsearch/index/bench_test.go
goos: darwin
goarch: amd64
BenchmarkParseStatsJSON/json_unmarshal_map-8                  67          84769698 ns/op        26763112 B/op     411564 allocs/op
BenchmarkParseStatsJSON/json_unmarshal_struct-8              136          41608684 ns/op         1793964 B/op       9064 allocs/op
PASS
ok      command-line-arguments  16.228s

Parsing a 4.8MB JSON API response into map[string]map[string]interface took up 26.76 MB of memory, whereas parsing the same JSON API response into structs took up 1.79 MB of memory. That's a 1:15 savings!

So I'm going to amend this PR accordingly — we will not switch to a streaming JSON parser, instead we will simply parse into structs instead of the current map[string]map[string]interface{} object.

[ycombinator]

jenkins, test this

[jsoriano]

Great!

[ycombinator]

Travis CI is green and Jenkins CI failures are unrelated. Merging.