telegraf 0.1.5 panics talking to 0.9.3-rc1 on Ubuntu 14.04

[srfraser]

Installing 0.1.5 from the provided .deb, telegraf fails to start. It's configured to talk to InfluxDB 0.9.3-rc1

2015/08/19 10:35:35 Starting Telegraf (version 0.1.5)
2015/08/19 10:35:35 Loaded outputs: influxdb
2015/08/19 10:35:35 Loaded plugins: cpu disk docker io mem mongodb mysql net postgresql redis swap system
2015/08/19 10:35:35 Tags enabled: host=myhostname
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x52795f]

goroutine 16 [running]:
github.com/influxdb/telegraf/plugins/system.(_systemPS).DiskUsage(0xc2080360a8, 0x0, 0x0, 0x0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/plugins/system/ps.go:81 +0x35f
github.com/influxdb/telegraf/plugins/system.(_DiskStats).Gather(0xc20808e8a0, 0x7f75669507f8, 0xc20806cbe0, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/plugins/system/disk.go:20 +0x6d
github.com/influxdb/telegraf.func·001(0xc2080fd770)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent.go:182 +0x167
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent.go:185 +0x504

goroutine 1 [semacquire]:
sync.(_WaitGroup).Wait(0xc20801fc40)
/root/.gvm/gos/go1.4.2/src/sync/waitgroup.go:132 +0x169
github.com/influxdb/telegraf.(_Agent).crankParallel(0xc20800c060, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent.go:188 +0x168
github.com/influxdb/telegraf.(*Agent).Run(0xc20800c060, 0xc20800c900, 0x0, 0x0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent.go:346 +0x247
main.main()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:134 +0x1395

goroutine 7 [syscall]:
os/signal.loop()
/root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/root/.gvm/gos/go1.4.2/src/os/signal/signal_unix.go:27 +0x35

main.func·001()
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/cmd/telegraf/telegraf.go:109 +0x48
created by main.main
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/cmd/t
elegraf/telegraf.go:111 +0xa9b

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/root/.gvm/gos/go1.4.2/src/runtime/asm_amd64.s:2232 +0x1

goroutine 13 [runnable]:
net/http.(_persistConn).writeLoop(0xc2080a80b0)
/root/.gvm/gos/go1.4.2/src/net/http/transport.go:945 +0x41d
created by net/http.(_Transport).dialConn
/root/.gvm/gos/go1.4.2/src/net/http/transport.go:661 +0xcbc

goroutine 18 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd7a0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 19 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd7d0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 20 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd800)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 21 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd860)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 22 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd890)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 23 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd8f0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 24 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd950)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 25 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd980)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 26 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd9b0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

goroutine 27 [runnable]:
github.com/influxdb/telegraf.func·001(0xc2080fd9e0)
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:174
created by github.com/influxdb/telegraf.(*Agent).crankParallel
/root/.gvm/pkgsets/go1.4.2/global/src/github.com/influxdb/telegraf/agent
.go:185 +0x504

---

grep -v "^#" /etc/opt/telegraf/telegraf.conf:

[outputs]
[outputs.influxdb]
url = "http://localhost:8086" # required.

database = "telegraf" # required.
username = "telegraf"
password = "telegraf"
[cpu]

no configuration

[disk]

no configuration

[docker]

no configuration

[io]

no configuration

[mem]

no configuration

[mysql]
servers = ["localhost"]

[net]
[postgresql]
[[postgresql.servers]]
address = "sslmode=disable"
[redis]
servers = ["localhost"]
[mongodb]
servers = ["127.0.0.1:27017"]

[swap]

no configuration

[system]

no configuration

[srfraser]

Definitely the disk plugin. Building from HEAD doesn't cause this error, so maybe it's fixed already?

[srfraser]

It appears to be working not because I built it from HEAD, but because I was running as a different user.

When running DiskUsage, there is a syscall.Statfs call. This is trying to run statfs against a fuse filesystem:

gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,user=simon)

[pid 25352] statfs("/run/user/1000/gvfs", 0xc2080dcc40) = -1 EACCES (Permission denied)
[pid 25352] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
[pid 25353] <... select resumed> ) = 0 (Timeout)
[pid 25352] rt_sigreturn( <unfinished ...>
[pid 25353] select(0, NULL, NULL, NULL, {0, 20} <unfinished ...>
[pid 25352] <... rt_sigreturn resumed> ) = 0
[pid 25352] write(2, "panic: ", 7panic: ) = 7
[pid 25352] write(2, "runtime error: invalid memory ad"..., 64runtime error: invalid memory address or nil pointer dereference) = 64

Tag filtering happens after the statfs, so I can't avoid looking at fuse filesystems.

[srfraser]

Opinions on adding a filter to plugins/system/ps.go's DiskUsage so that it doesn't examine fuse.* cgroup tmpfs, etc? Quickest to make it hardcoded, I'm trying to imagine a situation where you'd need those values.

[sparrc]

@srfraser rather than removing looking at those entirely (honestly I can't think of a situation either, but I'm also hesitant to filter it out in case someone would like to see fuse cgroups), couldn't we just capture the error where it's being called and ignore, or raise a debug log message that there is a permission error?

[srfraser]

The standard check for err is there, I can have a look at why it's getting a SEGV instead of being caught

[sparrc]

closed by #124