panic in docker module: interface conversion int64 uint64 #2027

andyceo · 2016-11-10T11:29:36Z

Ubuntu 16.04, telegraf: 1.1.0, installed from ofiicial repo.

If in docker plugin total = true, telegraf fails with mesaage:

ноя 15 14:05:21 newhope telegraf[10797]: panic: interface conversion: interface is int64, not uint64
ноя 15 14:05:21 newhope telegraf[10797]: goroutine 109 [running]:
ноя 15 14:05:21 newhope telegraf[10797]: panic(0xf24700, 0xc420050940)
ноя 15 14:05:21 newhope telegraf[10797]:         /usr/local/go/src/runtime/panic.go:500 +0x1a1
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.gatherBlockIOMetrics(0xc4201d4000, 0x18dfd20, 0xc4202d9650, 0xc420288390, 0xecfbce2f1, 0xc419c899a3, 0x18acdc0, 0xc42010d800, 0x40, 0xd472062c93e10101)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:492 +0x2109
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.gatherContainerStats(0xc4201d4000, 0x18dfd20, 0xc4202d9650, 0xc420288390, 0xc42010d800, 0x40, 0x18d0101)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:385 +0x30b1
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).gatherContainer(0xc420196360, 0xc42010d800, 0x40, 0xc42010d840, 0x1, 0x4, 0xc4205b2d80, 0xc, 0xc4200f9090, 0x47, ...)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:264 +0x68f
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather.func1(0xc4205d2450, 0xc420196360, 0x18dfd20, 0xc4202d9650, 0xc42010d800, 0x40, 0xc42010d840, 0x1, 0x4, 0xc4205b2d80, ...)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:127 +0xc5
ноя 15 14:05:21 newhope telegraf[10797]: created by github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:132 +0x2dd
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Unit entered failed state.
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Failed with result 'exit-code'.
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Service hold-off time over, scheduling restart.
ноя 15 14:05:21 newhope systemd[1]: Stopped The plugin-driven server agent for reporting metrics into InfluxDB.
ноя 15 14:05:21 newhope systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB.

Tuning total = false fix the problem.

The text was updated successfully, but these errors were encountered:

sparrc · 2016-11-14T11:49:46Z

the panic lines appear to be truncated, do you have the full traceback? I can't see the line numbers with this one

sparrc · 2016-11-14T15:24:56Z

also, what is your docker version?

andyceo · 2016-11-15T11:09:45Z

Docker version 1.12.3, build 6b644ec
I updated telegraf to v1.1.1 (git: release-1.1.0 94de9dc), this error still exists (cheked right now).

Full dump is:

ноя 15 14:05:21 newhope telegraf[10797]: panic: interface conversion: interface is int64, not uint64
ноя 15 14:05:21 newhope telegraf[10797]: goroutine 109 [running]:
ноя 15 14:05:21 newhope telegraf[10797]: panic(0xf24700, 0xc420050940)
ноя 15 14:05:21 newhope telegraf[10797]:         /usr/local/go/src/runtime/panic.go:500 +0x1a1
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.gatherBlockIOMetrics(0xc4201d4000, 0x18dfd20, 0xc4202d9650, 0xc420288390, 0xecfbce2f1, 0xc419c899a3, 0x18acdc0, 0xc42010d800, 0x40, 0xd472062c93e10101)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:492 +0x2109
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.gatherContainerStats(0xc4201d4000, 0x18dfd20, 0xc4202d9650, 0xc420288390, 0xc42010d800, 0x40, 0x18d0101)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:385 +0x30b1
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).gatherContainer(0xc420196360, 0xc42010d800, 0x40, 0xc42010d840, 0x1, 0x4, 0xc4205b2d80, 0xc, 0xc4200f9090, 0x47, ...)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:264 +0x68f
ноя 15 14:05:21 newhope telegraf[10797]: github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather.func1(0xc4205d2450, 0xc420196360, 0x18dfd20, 0xc4202d9650, 0xc42010d800, 0x40, 0xc42010d840, 0x1, 0x4, 0xc4205b2d80, ...)
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:127 +0xc5
ноя 15 14:05:21 newhope telegraf[10797]: created by github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather
ноя 15 14:05:21 newhope telegraf[10797]:         /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:132 +0x2dd
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Unit entered failed state.
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Failed with result 'exit-code'.
ноя 15 14:05:21 newhope systemd[1]: telegraf.service: Service hold-off time over, scheduling restart.
ноя 15 14:05:21 newhope systemd[1]: Stopped The plugin-driven server agent for reporting metrics into InfluxDB.
ноя 15 14:05:21 newhope systemd[1]: Started The plugin-driven server agent for reporting metrics into InfluxDB.

I get this dump with sudo journalctl -b command and copy-pasted error messages related to telegraf.

sparrc · 2016-11-15T11:12:59Z

please try to find a way to get untruncated lines from journalctl, some googling around seems to suggest that journalctl -b --no-pager | less might work

andyceo · 2016-11-15T11:15:35Z

@sparrc Thank you for pointing me! I updated comment above, please check.

Upd.: Update error message in first post.

dbienapfl · 2016-12-07T20:59:28Z

I am experiencing the same crash - telegraf running in a ubuntu 14.4 docker container, mounting /var/run into a '--privileged' docker container so i can read both base system metrics as well as '/var/run/docker.sock' - full trace:

panic: interface conversion: interface is int64, not uint64

goroutine 42 [running]:
panic(0xf24700, 0xc420052980)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/influxdata/telegraf/plugins/inputs/docker.gatherBlockIOMetrics(0xc4200f0000, 0x18dfd20, 0xc42000ccc0, 0xc42024da70, 0xecfda6da4, 0xc407be3aee, 0x18acdc0, 0xc42006f0c0, 0x40, 0x9289f8f245e60101)
	/home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:492 +0x2109
github.com/influxdata/telegraf/plugins/inputs/docker.gatherContainerStats(0xc4200f0000, 0x18dfd20, 0xc42000ccc0, 0xc42024da70, 0xc42006f0c0, 0x40, 0x18d0101)
	/home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:385 +0x30b1
github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).gatherContainer(0xc42006c2a0, 0xc42006f0c0, 0x40, 0xc42006f100, 0x1, 0x4, 0xc4204505a0, 0x13, 0xc420013ae0, 0x47, ...)
	/home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:264 +0x68f
github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather.func1(0xc4206ac040, 0xc42006c2a0, 0x18dfd20, 0xc42000ccc0, 0xc42006f0c0, 0x40, 0xc42006f100, 0x1, 0x4, 0xc4204505a0, ...)
	/home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:127 +0xc5
created by github.com/influxdata/telegraf/plugins/inputs/docker.(*Docker).Gather
	/home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/docker/docker.go:132 +0x2dd````

closes #2027

sparrc · 2016-12-21T10:55:02Z

I haven't been able to reproduce this, and looking at the code I don't see how it's possible, but I do have a change that should prevent it from happening. Before merging that I'd like to dig into it a bit more.

Do either of you have steps to reproduce? does the panic happen immediately? how many containers are you running?

closes #2027

closes influxdata#2027

closes #2027

jwilder added bug unexpected problem or unintended behavior panic issue that results in panics from Telegraf labels Nov 11, 2016

sparrc added the Need More Info label Nov 14, 2016

sparrc added this to the 1.2.0 milestone Nov 14, 2016

sparrc removed the Need More Info label Nov 15, 2016

sparrc mentioned this issue Dec 20, 2016

docker engine-api library is deprecated, we should use their official SDK #2071

Closed

sparrc added a commit that referenced this issue Dec 21, 2016

Check type of each blkio metric

8d4092a

closes #2027

sparrc mentioned this issue Dec 21, 2016

Check type of each blkio metric #2190

Merged

3 tasks

121watts assigned sparrc Dec 21, 2016

121watts added the in progress label Dec 21, 2016

sparrc added a commit that referenced this issue Dec 21, 2016

docker: check type when totalling blkio & net metrics

1e12ffb

closes #2027

sparrc added a commit that referenced this issue Dec 21, 2016

docker: check type when totalling blkio & net metrics

b762546

closes #2027

sparrc closed this as completed in #2190 Dec 21, 2016

121watts removed the in progress label Dec 21, 2016

njwhite pushed a commit to njwhite/telegraf that referenced this issue Jan 31, 2017

docker: check type when totalling blkio & net metrics

f3ddf19

closes influxdata#2027

maxunt pushed a commit that referenced this issue Jun 26, 2018

docker: check type when totalling blkio & net metrics

b77dc90

closes #2027

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic in docker module: interface conversion int64 uint64 #2027

panic in docker module: interface conversion int64 uint64 #2027

andyceo commented Nov 10, 2016 •

edited

Loading

sparrc commented Nov 14, 2016 •

edited

Loading

sparrc commented Nov 14, 2016

andyceo commented Nov 15, 2016 •

edited

Loading

sparrc commented Nov 15, 2016

andyceo commented Nov 15, 2016 •

edited

Loading

dbienapfl commented Dec 7, 2016 •

edited

Loading

sparrc commented Dec 21, 2016

panic in docker module: interface conversion int64 uint64 #2027

panic in docker module: interface conversion int64 uint64 #2027

Comments

andyceo commented Nov 10, 2016 • edited Loading

sparrc commented Nov 14, 2016 • edited Loading

sparrc commented Nov 14, 2016

andyceo commented Nov 15, 2016 • edited Loading

sparrc commented Nov 15, 2016

andyceo commented Nov 15, 2016 • edited Loading

dbienapfl commented Dec 7, 2016 • edited Loading

sparrc commented Dec 21, 2016

andyceo commented Nov 10, 2016 •

edited

Loading

sparrc commented Nov 14, 2016 •

edited

Loading

andyceo commented Nov 15, 2016 •

edited

Loading

andyceo commented Nov 15, 2016 •

edited

Loading

dbienapfl commented Dec 7, 2016 •

edited

Loading