Add ability to monitor size of specific directory on server

[eWilliams35]

Use case:

According to support tickets filed with influxdata, they do not recommend enabling internal monitoring on production clusters. The main thing you lose in that scenario is the ability to see the data backing up in the hinted handoff queues / directories, which is the easiest way to tell if one of the nodes has fallen behind.

I'd like to be able to ask telegraf to specifically watch the size of /var/lib/influxdb/hh, and then use kapacitor to generate an alert if data starts backing up.

[danielnelson]

Seems like a good idea, possibly it could become part of the filestat plugin or a similar plugin.

Have you tried the influxdb input plugin? You might be able to get the desired info from here though I don't have a clustered InfluxDB in front of me.

I also did try to come up with a workaround with some success, it requires the nightly build (1.6) for the sum operation in basicstats:

[[inputs.filestat]]
  name_override = "dirstat"
  interval = "10s"
  files = ["/foo/*"]
  tagexclude = ["file"]
  fieldpass = ["size_bytes"]

[[aggregators.basicstats]]
  namepass = "dirstat"
  period = "10s"
  drop_original = true
  stats = ["sum"]
  [[aggregators.basicstats.tags]]
    path = "/foo"

There are a few problems, this counts directories as having 4096 bytes and the first result when you start Telegraf is 2x the actual value.

[eWilliams35]

I have not, but I will. I didn't realize there was a plugin that went the other direction as well. I'll report back once I see how it works. Thank you!

[eWilliams35]

Good news, Telegraf does pull the hinted handoff information like you mentioned above. It scrapes the debug endpoint on the Influx server and spits out all of the info into the target database. With that said, I still think this would be a handy feature.

[gecube]

It will be great to have specific input plugin to count the size and count of file in specific catalogue.
The using of inputs.exec with shell-oneliner or inputs.filestat looks like performance-consuming and non-optimal.

[danielnelson]

@ewillia1983 @gecube We recently added the filecount plugin to handle this issue, it will be included in the 1.8 release. Would be great if you can try it out using one of the nightly builds to ensure it will meet your requirements.

#4363

[Siddhu1096]

And how will we see that in Grafana???

Seems like a good idea, possibly it could become part of the filestat plugin or a similar plugin.

Have you tried the influxdb input plugin? You might be able to get the desired info from here though I don't have a clustered InfluxDB in front of me.

I also did try to come up with a workaround with some success, it requires the nightly build (1.6) for the sum operation in basicstats:

[[inputs.filestat]]
  name_override = "dirstat"
  interval = "10s"
  files = ["/foo/*"]
  tagexclude = ["file"]
  fieldpass = ["size_bytes"]

[[aggregators.basicstats]]
  namepass = "dirstat"
  period = "10s"
  drop_original = true
  stats = ["sum"]
  [[aggregators.basicstats.tags]]
    path = "/foo"

There are a few problems, this counts directories as having 4096 bytes and the first result when you start Telegraf is 2x the actual value.