Panic on graceful shutdown (close of closed channel)

[balboah]

For fleet version 0.10.2 on CoreOS Stable branch, there seems to be two problems:

• Might close an already closed channel upon graceful shutdown
• The ever growing list of goroutines when having etcd connection issues, almost two million in this example

... snip snip (a few days of 503) ...

Oct 08 07:52:21 fleetd[127879]: INFO client.go:298: Response 503 Service Unavailable from http://127.0.0.1:4001/ unusable
Oct 08 07:52:21 fleetd[127879]: ERROR client.go:214: Unable to get result for {Create /_coreos.com/fleet/machines/0553e988b3e14a6ebd7d3108a2ab4d8e/object}, retrying in 1s
Oct 08 07:52:22 fleetd[127879]: INFO client.go:298: Response 503 Service Unavailable from http://127.0.0.1:4001/ unusable
Oct 08 07:52:22 fleetd[127879]: ERROR client.go:214: Unable to get result for {Create /_coreos.com/fleet/machines/0553e988b3e14a6ebd7d3108a2ab4d8e/object}, retrying in 1s
Oct 08 07:52:23 fleetd[127879]: INFO client.go:298: Response 503 Service Unavailable from http://127.0.0.1:4001/ unusable
Oct 08 07:52:23 fleetd[127879]: ERROR client.go:214: Unable to get result for {Create /_coreos.com/fleet/machines/0553e988b3e14a6ebd7d3108a2ab4d8e/object}, retrying in 1s
Oct 08 07:53:24 systemd[1]: Stopping fleet daemon...
Oct 08 07:53:24 fleetd[127879]: INFO fleet.go:105: Gracefully shutting down
Oct 08 07:53:24 fleetd[127879]: panic: close of closed channel
Oct 08 07:53:24 fleetd[127879]: goroutine 1 [running]:
Oct 08 07:53:24 fleetd[127879]: main.func·002()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/fleetd/fleet.go:106 +0x6b
Oct 08 07:53:24 fleetd[127879]: main.listenForSignals(0xc2080b0ae0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/fleetd/fleet.go:211 +0x1f7
Oct 08 07:53:24 fleetd[127879]: main.main()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/fleetd/fleet.go:137 +0x12c1
Oct 08 07:53:24 fleetd[127879]: goroutine 5 [syscall]:
Oct 08 07:53:24 fleetd[127879]: os/signal.loop()
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/os/signal/signal_unix.go:21 +0x1f
Oct 08 07:53:24 fleetd[127879]: created by os/signal.init·1
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/os/signal/signal_unix.go:27 +0x35
Oct 08 07:53:24 fleetd[127879]: goroutine 6 [IO wait, 2376 minutes]:
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).Wait(0xc208011e90, 0x72, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).WaitRead(0xc208011e90, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Oct 08 07:53:24 fleetd[127879]: net.(*netFD).readMsg(0xc208011e30, 0xc208260d80, 0x10, 0x10, 0xc208338020, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e
Oct 08 07:53:24 fleetd[127879]: net.(*UnixConn).ReadMsgUnix(0xc208034090, 0xc208260d80, 0x10, 0x10, 0xc208338020, 0x1000, 0x1000, 0xa48, 0xc2085caab4, 0x4, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*oobReader).Read(0xc208338000, 0xc208260d80, 0x10, 0x10, 0xc208338000, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
Oct 08 07:53:24 fleetd[127879]: io.ReadAtLeast(0x7f813ca9a510, 0xc208338000, 0xc208260d80, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:298 +0xf1
Oct 08 07:53:24 fleetd[127879]: io.ReadFull(0x7f813ca9a510, 0xc208338000, 0xc208260d80, 0x10, 0x10, 0xa48, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:316 +0x6d
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc20800bc30, 0xc208043560, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).inWorker(0xc208001560)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:241 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
Oct 08 07:53:24 fleetd[127879]: goroutine 7 [chan receive, 2376 minutes]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).outWorker(0xc208001560)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:363 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
Oct 08 07:53:24 fleetd[127879]: goroutine 8 [IO wait]:
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).Wait(0xc208090370, 0x72, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).WaitRead(0xc208090370, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Oct 08 07:53:24 fleetd[127879]: net.(*netFD).readMsg(0xc208090310, 0xc20848b4b0, 0x10, 0x10, 0xc208c99220, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e
Oct 08 07:53:24 fleetd[127879]: net.(*UnixConn).ReadMsgUnix(0xc208034020, 0xc20848b4b0, 0x10, 0x10, 0xc208c99220, 0x1000, 0x1000, 0x79, 0xc20848aed4, 0x4, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*oobReader).Read(0xc208c99200, 0xc20848b4b0, 0x10, 0x10, 0xc208c99200, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
Oct 08 07:53:24 fleetd[127879]: io.ReadAtLeast(0x7f813ca9a510, 0xc208c99200, 0xc20848b4b0, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:298 +0xf1
Oct 08 07:53:24 fleetd[127879]: io.ReadFull(0x7f813ca9a510, 0xc208c99200, 0xc20848b4b0, 0x10, 0x10, 0x79, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:316 +0x6d
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc2080a0660, 0xc20805a540, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).inWorker(0xc2080019e0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:241 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
Oct 08 07:53:24 fleetd[127879]: goroutine 9 [chan receive, 4300 minutes]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).outWorker(0xc2080019e0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:363 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
Oct 08 07:53:24 fleetd[127879]: goroutine 10 [chan receive]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus.func·001()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/subscription.go:6
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus.(*Conn).dispatch
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/subscription.go:9
Oct 08 07:53:24 fleetd[127879]: goroutine 11 [IO wait, 2388 minutes]:
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).Wait(0xc2080901b0, 0x72, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Oct 08 07:53:24 fleetd[127879]: net.(*netFD).accept(0xc208090150, 0x0, 0x7f813ca98bb0, 0xc2081afdb8)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_unix.go:419 +0x40b
Oct 08 07:53:24 fleetd[127879]: net.(*UnixListener).AcceptUnix(0xc20801ea00, 0x437d70, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:282 +0x56
Oct 08 07:53:24 fleetd[127879]: net.(*UnixListener).Accept(0xc20801ea00, 0x0, 0x0, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:293 +0x4c
Oct 08 07:53:24 fleetd[127879]: net/http.(*Server).Serve(0xc20805a4e0, 0x7f813ca9a9d8, 0xc20801ea00, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/http/server.go:1728 +0x92
Oct 08 07:53:24 fleetd[127879]: net/http.Serve(0x7f813ca9a9d8, 0xc20801ea00, 0x7f813ca9ad00, 0xc208032640, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/http/server.go:1606 +0xa1
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/api.func·001()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/api/server.go:49 +0x77
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/api.(*Server).Serve
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/api/server.go:53 +0x14d
Oct 08 07:53:24 fleetd[127879]: goroutine 12 [chan send, 4299 minutes]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/etcd.func·004()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:246 +0x137
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/etcd.(*client).Do
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:247 +0x1ed
Oct 08 07:53:24 fleetd[127879]: goroutine 50 [chan send, 4299 minutes]:


... snip snip ...


Oct 08 07:53:24 fleetd[127879]: goroutine 1784912 [chan send]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/etcd.func·004()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:246 +0x137
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/etcd.(*client).Do
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:247 +0x1ed

[mischief]

i noticed the panic on shutdown in recent versions too (0.11.5 on coreos alpha 815.0.0). however, the etcd goroutine leak issue might be gone, since fleetd moved from go-etcd to etcd/client package.

Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: INFO fleetd.go:113: Gracefully shutting down
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: panic: close of closed channel
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 1 [running]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: main.func·002()
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/fleetd/fleetd.go:114 +0x6b
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: main.listenForSignals(0xc2080b09f0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/fleetd/fleetd.go:219 +0x1f7
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: main.main()
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/fleetd/fleetd.go:145 +0x13d2
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 5 [syscall]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: os/signal.loop()
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/os/signal/signal_unix.go:21 +0x1f
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by os/signal.init·1
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/os/signal/signal_unix.go:27 +0x35
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 6 [IO wait]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).Wait(0xc2080883e0, 0x72, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).WaitRead(0xc2080883e0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*netFD).readMsg(0xc208088380, 0xc20828fb70, 0x10, 0x10, 0xc2087d9220, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*UnixConn).ReadMsgUnix(0xc20802c0b0, 0xc20828fb70, 0x10, 0x10, 0xc2087d9220, 0x1000, 0x1000, 0xc856, 0xc208271004, 0x4, ...)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*oobReader).Read(0xc2087d9200, 0xc20828fb70, 0x10, 0x10, 0xc2087d9200, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: io.ReadAtLeast(0x7f0bbd03d5b8, 0xc2087d9200, 0xc20828fb70, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/io/io.go:298 +0xf1
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: io.ReadFull(0x7f0bbd03d5b8, 0xc2087d9200, 0xc20828fb70, 0x10, 0x10, 0xc856, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/io/io.go:316 +0x6d
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc20800bce0, 0xc20800d560, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).inWorker(0xc2080017a0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:248 +0x58
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 7 [chan receive, 1 minutes]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).outWorker(0xc2080017a0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:370 +0x58
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 8 [IO wait]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).Wait(0xc208088610, 0x72, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).WaitRead(0xc208088610, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*netFD).readMsg(0xc2080885b0, 0xc2087d17d0, 0x10, 0x10, 0xc20851f620, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*UnixConn).ReadMsgUnix(0xc20802c020, 0xc2087d17d0, 0x10, 0x10, 0xc20851f620, 0x1000, 0x1000, 0x4d, 0xc2087d14d4, 0x4, ...)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*oobReader).Read(0xc20851f600, 0xc2087d17d0, 0x10, 0x10, 0xc20851f600, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: io.ReadAtLeast(0x7f0bbd03d5b8, 0xc20851f600, 0xc2087d17d0, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/io/io.go:298 +0xf1
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: io.ReadFull(0x7f0bbd03d5b8, 0xc20851f600, 0xc2087d17d0, 0x10, 0x10, 0x4d, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/io/io.go:316 +0x6d
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc20809c9b0, 0xc2080584e0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).inWorker(0xc208001d40)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:248 +0x58
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 9 [chan receive, 19 minutes]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).outWorker(0xc208001d40)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:370 +0x58
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 10 [chan receive]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus.func·001()
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/subscription.go:66 +0x64
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus.(*Conn).dispatch
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/subscription.go:98 +0x11d
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 11 [IO wait, 5 minutes]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).Wait(0xc2080881b0, 0x72, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).WaitRead(0xc2080881b0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*netFD).accept(0xc208088150, 0x0, 0x7f0bbd03bbb0, 0xc208076490)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_unix.go:419 +0x40b
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*UnixListener).AcceptUnix(0xc208086120, 0x437870, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/unixsock_posix.go:282 +0x56
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*UnixListener).Accept(0xc208086120, 0x0, 0x0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/unixsock_posix.go:293 +0x4c
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net/http.(*Server).Serve(0xc2080585a0, 0x7f0bbd03daa8, 0xc208086120, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/server.go:1728 +0x92
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net/http.Serve(0x7f0bbd03daa8, 0xc208086120, 0x7f0bbd03df20, 0xc20802a300, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/server.go:1606 +0xa1
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/api.func·001()
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/api/server.go:49 +0x77
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/api.(*Server).Serve
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/api/server.go:53 +0x14d
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 30 [sleep]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/server.(*Server).Run(0xc208058180)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/server/server.go:165 +0x2d2
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/server.(*Server).Monitor(0xc208058180)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/server/server.go:192 +0x13e
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/server.(*Server).Run
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/server/server.go:172 +0x14f
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 22407 [IO wait]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).Wait(0xc20865f330, 0x72, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*pollDesc).WaitRead(0xc20865f330, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*netFD).Read(0xc20865f2d0, 0xc2087e9000, 0x1000, 0x1000, 0x0, 0x7f0bbd03bbb0, 0xc2087e63a0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/fd_unix.go:242 +0x40f
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net.(*conn).Read(0xc20802c6a8, 0xc2087e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/net.go:121 +0xdc
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net/http.noteEOFReader.Read(0x7f0bbd03f0a0, 0xc20802c6a8, 0xc2080b74f8, 0xc2087e9000, 0x1000, 0x1000, 0x730780, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/transport.go:1270 +0x6e
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net/http.(*noteEOFReader).Read(0xc2087e2bc0, 0xc2087e9000, 0x1000, 0x1000, 0xc208012000, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: <autogenerated>:125 +0xd4
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: bufio.(*Reader).fill(0xc2087ec000)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/bufio/bufio.go:97 +0x1ce
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: bufio.(*Reader).Peek(0xc2087ec000, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/bufio/bufio.go:132 +0xf0
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net/http.(*persistConn).readLoop(0xc2080b74a0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/transport.go:842 +0xa4
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by net/http.(*Transport).dialConn
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/transport.go:660 +0xc9f
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 36 [chan send]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: github.com/coreos/fleet/unit.(*UnitStateGenerator).Run(0xc2080b0270, 0xc208058ea0, 0xc208058e40)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/unit/generator.go:70 +0x250
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by github.com/coreos/fleet/server.(*Server).Run
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.11.5/work/gopath/src/github.com/coreos/fleet/server/server.go:180 +0x280
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: goroutine 22408 [select]:
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: net/http.(*persistConn).writeLoop(0xc2080b74a0)
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/transport.go:945 +0x41d
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: created by net/http.(*Transport).dialConn
Sep 30 07:25:13 coreos_production_qemu-815-0-0 fleetd[601]: /usr/lib/go/src/net/http/transport.go:661 +0xcbc

[jonboulle]

Fairly sure this is #1067

[jonboulle]

Should be fixed by #1496