[Metricbeat] Add dedot to aws ec2 metricset and cloudwatch metricset

[kaiyan-sheng]

What does this PR do?

This PR is to add dedot to ec2 and cloudwatch metricset in aws module.

Why is it important?

If a tag key includes dot, then ec2 metricset will fail with error like:

2020-01-24T14:44:46.013-0700 WARN elasticsearch/client.go:516 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf82f72e084b7960, ext:2131556658, loc:(*time.Location)(0x9bcef40)}, Meta:null, Fields:{"agent":{"ephemeral_id":"6374bf9d-4e60-4862-ada1-6b3165a5a81f","hostname":"KaiyanMacBookPro","id":"23fd6ee9-2255-43f3-9565-78a051337c04","type":"metricbeat","version":"8.0.0"},"aws":{"ec2":{"cpu":{"credit_balance":169.591847,"credit_usage":0.004121,"surplus_credit_balance":0.000000,"surplus_credits_charged":0.000000,"total":{"pct":0.099454}},"diskio":{"read":{"bytes":0.000000,"bytes_per_sec":0.000000,"count":0.000000,"count_per_sec":0.000000},"write":{"bytes":0.000000,"bytes_per_sec":0.000000,"count":0.000000,"count_per_sec":0.000000}},"instance":{"core":{"count":1},"image":{"id":"ami-062f7200baf2fa504"},"monitoring":{"state":"disabled"},"private":{"dns_name":"ip-172-31-19-10.ec2.internal","ip":"172.31.19.10"},"public":{"dns_name":"ec2-3-95-172-1.compute-1.amazonaws.com","ip":"3.95.172.1"},"state":{"code":16,"name":"running"},"threads_per_core":1},"network":{"in":{"bytes":530.400000,"bytes_per_sec":1.768000,"packets":6.800000,"packets_per_sec":0.022667},"out":{"bytes":461.600000,"bytes_per_sec":1.538667,"packets":7.400000,"packets_per_sec":0.024667}},"status":{"check_failed":0,"check_failed_instance":0,"check_failed_system":0}},"tags":{"app":{"kubernetes":{"io/name":"foo"}},"created-by":"ks","kubernetes":"{io/cluster/lakeplacid-eksCluster-0a550bd=owned}"}},"cloud":{"account":{"id":"428152502467","name":"elastic-beats"},"availability_zone":"us-east-1c","instance":{"id":"i-00123db57112ab0b6"},"machine":{"type":"t2.micro"},"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.4.0"},"event":{"dataset":"aws.ec2","duration":20819094576,"module":"aws"},"host":{"architecture":"x86_64","hostname":"KaiyanMacBookPro","id":"9C7FAB7B-29D1-5926-8E84-158A9CA3E25D","name":"KaiyanMacBookPro","os":{"build":"17G10021","family":"darwin","kernel":"17.7.0","name":"Mac OS X","platform":"darwin","version":"10.13.6"}},"metricset":{"name":"ec2","period":300000},"service":{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.tags.app] of type [keyword] in document with id '00uD2W8Bcprh94r7uDqM'. Preview of field's value: '{kubernetes={io/name=foo}}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1042"}}

With this PR, you will be able to see events for EC2 instances that have tags with dot. For example:

{
  "_index": "metricbeat-8.0.0-2020.01.24-000001",
  "_type": "_doc",
  "_id": "4kvx2W8Bcprh94r7CDpf",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-01-24T23:43:49.868Z",
    "cloud": {
      "availability_zone": "us-east-1c",
      "provider": "aws",
      "region": "us-east-1",
      "account": {
        "name": "elastic-beats",
        "id": "428152502467"
      },
      "instance": {
        "id": "i-00123db57112ab0b6"
      },
      "machine": {
        "type": "t2.micro"
      }
    },
    "event": {
      "dataset": "aws.ec2",
      "module": "aws",
      "duration": 18952869978
    },
    "metricset": {
      "name": "ec2",
      "period": 300000
    },
    "aws": {
      "ec2": {
        "network": {
          "out": {
            "packets": 8.4,
            "bytes_per_sec": 1.5946666666666667,
            "packets_per_sec": 0.028,
            "bytes": 478.4
          },
          "in": {
            "bytes": 567.6,
            "packets": 7.6,
            "bytes_per_sec": 1.8920000000000001,
            "packets_per_sec": 0.025333333333333333
          }
        },
        "status": {
          "check_failed": 0,
          "check_failed_instance": 0,
          "check_failed_system": 0
        },
        "cpu": {
          "credit_usage": 0.00306,
          "credit_balance": 169.501774,
          "surplus_credit_balance": 0,
          "surplus_credits_charged": 0,
          "total": {
            "pct": 0.10056497175141618
          }
        },
        "diskio": {
          "write": {
            "bytes_per_sec": 0,
            "count_per_sec": 0,
            "bytes": 0,
            "count": 0
          },
          "read": {
            "count": 0,
            "bytes_per_sec": 0,
            "count_per_sec": 0,
            "bytes": 0
          }
        },
        "instance": {
          "image": {
            "id": "ami-062f7200baf2fa504"
          },
          "state": {
            "name": "running",
            "code": 16
          },
          "monitoring": {
            "state": "disabled"
          },
          "core": {
            "count": 1
          },
          "threads_per_core": 1,
          "public": {
            "dns_name": "ec2-3-95-172-1.compute-1.amazonaws.com",
            "ip": "3.95.172.1"
          },
          "private": {
            "dns_name": "ip-172-31-19-10.ec2.internal",
            "ip": "172.31.19.10"
          }
        }
      },
      "tags": {
        "kubernetes": "{io/cluster/lakeplacid-eksCluster-0a550bd=owned}",
        "app_kubernetes_io/name": "foo",
        "created-by": "ks"
      }
    },
    "service": {
      "type": "aws"
    },
    "ecs": {
      "version": "1.4.0"
    },
    "host": {
      "name": "KaiyanMacBookPro",
      "os": {
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X",
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin"
      },
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64"
    },
    "agent": {
      "version": "8.0.0",
      "type": "metricbeat",
      "ephemeral_id": "d4602b1b-5702-48f6-8895-0d3d4cb34ec1",
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-01-24T23:43:49.868Z"
    ]
  },
  "sort": [
    1579909429868
  ]
}

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

Create an EC2 instance and add a set of tag with dot in there. For example:
app.kubernetes.io/name = foo
Then start Metricbeat aws module with aws.yml:

- module: aws
  period: 5m
  credential_profile_name: elastic-beats
  metricsets:
    - ec2

Then you should see tags collected by aws module in events. Tag name should be aws.tags.app_kubernetes_io/name with value foo.

Related issues

• Closes [Metricbeat] Add dedot to aws module when collecting tags #15843

[mtojek]

LGTM, but wondering what would happen if there is no DeDot option and it's a default behavior, so the end user can't disable it.

[kaiyan-sheng]

LGTM, but wondering what would happen if there is no DeDot option and it's a default behavior, so the end user can't disable it.

Yep, I just had a talk with Carlos and we think it's good to remove the config option and just enable it by default.