[Metricbeat] invalid memory address or nil pointer error for rds metricset

[kaiyan-sheng]

@sorantis Thanks for testing RDS metricset and reporting this error:
2019-07-22T07:31:37.449+0200 ERROR runtime/panic.go:82 recovered from panic while fetching 'aws/rds' for host ''. Recovering, but please report this. {"panic": "runtime error: invalid memory address or nil pointer dereference", "stack": "github.com/elastic/beats/libbeat/logp.Recover\n\t/go/src/github.com/elastic/beats/libbeat/logp/global.go:105\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:522\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:82\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:390\ngh.neting.cc/elastic/beats/x-pack/metricbeat/module/aws/rds.getDBInstancesPerRegion\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/rds/rds.go:167\ngh.neting.cc/elastic/beats/x-pack/metricbeat/module/aws/rds.(*MetricSet).Fetch\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/rds/rds.go:89\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).fetch\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:244\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:222\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).run\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:199\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*Wrapper).Start.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:138"}

[kaiyan-sheng]

I can't reproduce it hmm In what condition you saw this error? @sorantis

[sorantis]

This has happened once while I was configuring my setup for the first time. As far as I remember I did this:

1. Enabled AWS module
2. Configured AWS credentials
3. Started metricbeat
4. Created a MariaDb instance in AWS RDS while metricbeat was running.

UPDATE:
Strange, cannot reproduce it either. I suggest we close it for now and reopen if we encounter it later.

[justinwalz]

I've noticed this too. In my case, metricbeat was running correctly for a few days, then, this stack trace emitted and all events ceased to be sent to elasticsearch.

2019-08-04T10:00:13.766Z        ERROR   runtime/panic.go:82     recovered from panic while fetching 'aws/rds' for host ''. Recovering, but please report this.  {"panic": "runtime error: invalid memory address or nil pointer dereference", "stack": "github.com/elastic/beats/libbeat/logp.Recover\n\t/go/src/github.com/elastic/beats/libbeat/logp/global.go:105\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:522\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:82\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:390\ngh.neting.cc/elastic/beats/x-pack/metricbeat/module/aws/rds.getDBInstancesPerRegion\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/rds/rds.go:167\ngh.neting.cc/elastic/beats/x-pack/metricbeat/module/aws/rds.(*MetricSet).Fetch\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/rds/rds.go:89\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).fetch\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:244\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:222\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).run\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:199\ngh.neting.cc/elastic/beats/metricbeat/mb/module.(*Wrapper).Start.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:138"}

One guess -- when RDS is provisioning a new instance, I believe the hostname is not yet available. It's possible metricbeat just needs to check for that if it doesn't already.

[kaiyan-sheng]

@justinwalz This error message is very helpful! I will make some changes in the code to add consideration for this scenario. Thanks!