[Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16482

mtojek · 2020-02-21T11:39:43Z

This PR introduces new metricsets (node, proxy) for Redis Enterprise Software.

How to test this PR locally

Use docker-compose up redisenterprise to boot up Redis Enterprise module. The port 8070 should be exposed.
Enable redisenterprise metricbeat (x-pack) module.
Configure module similarly to the prox_integration_test.go:

func getConfig(host string) map[string]interface{} {
	return map[string]interface{}{
		"module":                "redisenterprise",
		"metricsets":            []string{"proxy"},
		"hosts":                 []string{host},
		"ssl.verification_mode": "none",
	}
}

metricsets - include all: node, proxy

Start metricbeat with -e -d processors and observe events flow.

jsoriano

LGTM, only a couple of minor details, and please add a changelog entry.

jsoriano · 2020-02-27T19:00:09Z

libbeat/tests/compose/wrapper.go

+	}
+
+	if !found {
+		return "", errors.Wrap(err, "container not found")


No error to wrap here, err will be probably nil here.

Suggested change

return "", errors.Wrap(err, "container not found")

return "", errors.Errorf("container not found for service %s", serviceName)

jsoriano · 2020-02-27T19:01:26Z

metricbeat/docs/modules/redis.asciidoc

@@ -33,8 +33,8 @@ redis://HOST[:PORT][?password=PASSWORD[&db=DATABASE]]
 [float]
 === Compatibility

-The Redis metricsets were tested with Redis 3.2.12, 4.0.11 and 5.0-rc4,  and are expected
-to work with all versions >= 3.0.
+The redis metricsets `info`, `key` and `keyspace` are compatible with the open source distribution of Redis. They were


They should also work with non-OSS redis servers right?

Thanks, fixed.

jsoriano · 2020-02-27T19:04:19Z

x-pack/metricbeat/module/redisenterprise/_meta/supported-versions.yml

@@ -0,0 +1,2 @@
+variants:
+  - REDIS_VERSION: 5.4.10-22


Please use REDISENTERPRISE_VERSION for this variable, in case at some moment we want to have scenarios with different versions of redis and redis enterprise.

jsoriano · 2020-02-28T11:15:31Z

.travis.yml

@@ -235,6 +235,7 @@ before_install:
  - chmod +x docker-compose
  - sudo mv docker-compose /usr/local/bin
  - if [ $TRAVIS_OS_NAME = osx ]; then pip install virtualenv==16.7.9; fi
+  - if [ $TRAVIS_OS_NAME = osx ]; then ulimit -n 1024; fi


Was this added because "too many open files" error in travis? I think this will be solved by #16681

Yes, precisely. I think I can revert it here.

jsoriano

LGTM if it also looks good to Jenkins 🙂 Thanks!

mtojek · 2020-02-28T14:18:33Z

Jenkins errors unrelated. Merging.

Add metricsets: node, proxy

* Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (#16658) * Make memqueue.Broker internal (now memqueue.broker) (#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584) * [Metricbeat] Add vpc metricset for aws module (#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (#16497) * Added NetBeat (#12612) * Close files earlier when generating fields files (#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>

Add metricsets: node, proxy (cherry picked from commit 96bebe4)

…and "proxy" metricsets (#16751) * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy (cherry picked from commit 96bebe4)

* Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (elastic#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (elastic#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (elastic#16658) * Make memqueue.Broker internal (now memqueue.broker) (elastic#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (elastic#16584) * [Metricbeat] Add vpc metricset for aws module (elastic#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (elastic#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (elastic#16497) * Added NetBeat (elastic#12612) * Close files earlier when generating fields files (elastic#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (elastic#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> (cherry picked from commit 07b03a7)

) * Add cloudfoundry module to metricbeat (#16671) * Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (#16658) * Make memqueue.Broker internal (now memqueue.broker) (#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584) * [Metricbeat] Add vpc metricset for aws module (#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (#16497) * Added NetBeat (#12612) * Close files earlier when generating fields files (#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com> (cherry picked from commit 07b03a7) * Fix changelog. * Run mage fmt. * Fix imports to have v7. * Run mage fmt update. * Fix changelog.

Marcin Tojek added 4 commits February 20, 2020 18:59

Prepare redis-enterprise docker image

dd6399a

All existing tests are passing

7091a29

Add metricsets - node and proxy

7411396

Fix: system tests

3ded196

mtojek added Metricbeat Metricbeat Team:Services (Deprecated) Label for the former Integrations-Services team labels Feb 21, 2020

mtojek requested a review from a team February 21, 2020 11:39

mtojek self-assigned this Feb 21, 2020

mtojek mentioned this pull request Feb 21, 2020

[Metricbeat] Prometheus based module should support metrics filtering #15493

Closed

mtojek added [zube]: Inbox Team:Services (Deprecated) Label for the former Integrations-Services team [zube]: In Review and removed Team:Services (Deprecated) Label for the former Integrations-Services team [zube]: Inbox labels Feb 21, 2020

Marcin Tojek added 16 commits February 21, 2020 14:00

Fix: set secondary source

79a4b88

Fix: mage check

aea45fa

Merge branch 'master' into 15269-redis-node-proxy

7944c73

Fix: typo

d0ef810

Try: increase timeout

0d88674

Shorten timeout

08c0b6b

Fix

bb7e861

Fix

1710d20

Fix: use proper port

8999ed9

Fix: use proper port

6fb8360

Fix: dashboard setup issue

792df87

OSX: increase limit of open files

79316ee

Fix: skip integration tests

a8f0673

Merge branch 'master' into 15269-redis-node-proxy

b436634

Use metrics_filters

76bc109

Fix: skip 'up' metric

f96dd2f

Marcin Tojek added 3 commits February 27, 2020 17:15

mage check

80382f4

Merge branch 'master' into 15269-redis-node-proxy

83bf0e3

Exclude 'up'

806a89b

jsoriano reviewed Feb 27, 2020

View reviewed changes

Adjust source after review

514cec0

mtojek requested review from jsoriano and a team February 27, 2020 19:46

jsoriano reviewed Feb 28, 2020

View reviewed changes

Fix

30f7afc

jsoriano approved these changes Feb 28, 2020

View reviewed changes

mtojek merged commit 96bebe4 into elastic:master Feb 28, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Feb 28, 2020

blakerouse pushed a commit to blakerouse/beats that referenced this pull request Feb 28, 2020

[Metricbeat] Add new module for Redis Enterprise (elastic#16482)

06c7585

Add metricsets: node, proxy

mtojek added a commit to mtojek/beats that referenced this pull request Mar 3, 2020

[Metricbeat] Add new module for Redis Enterprise (elastic#16482)

d9d15e7

Add metricsets: node, proxy (cherry picked from commit 96bebe4)

mtojek mentioned this pull request Mar 3, 2020

WIP Cherry-pick #16482 to 7.x: [Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16750

Closed

mtojek added the v7.7.0 label Mar 3, 2020

mtojek added a commit to mtojek/beats that referenced this pull request Mar 3, 2020

[Metricbeat] Add new module for Redis Enterprise (elastic#16482)

48e0eac

Add metricsets: node, proxy (cherry picked from commit 96bebe4)

mtojek mentioned this pull request Mar 3, 2020

Cherry-pick #16482 to 7.x: [Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16751

Merged

andresrc removed the [zube]: Done label Mar 4, 2020

mtojek added the test-plan Add this PR to be manual test plan label Mar 16, 2020

andresrc added the test-plan-added This PR has been added to the test plan label Mar 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16482

[Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16482

mtojek commented Feb 21, 2020 •

edited

Loading

jsoriano left a comment

jsoriano Feb 27, 2020

mtojek Feb 27, 2020

jsoriano Feb 27, 2020

mtojek Feb 27, 2020

jsoriano Feb 27, 2020

mtojek Feb 27, 2020

jsoriano Feb 28, 2020

mtojek Feb 28, 2020

jsoriano left a comment

mtojek commented Feb 28, 2020

	return "", errors.Wrap(err, "container not found")
	return "", errors.Errorf("container not found for service %s", serviceName)

[Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16482

[Metricbeat] Redis Enterprise: add "node" and "proxy" metricsets #16482

Conversation

mtojek commented Feb 21, 2020 • edited Loading

How to test this PR locally

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

mtojek commented Feb 28, 2020

mtojek commented Feb 21, 2020 •

edited

Loading