Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker daemon crash because of libnetwork issue #1572

Closed
bvis opened this issue Nov 23, 2016 · 4 comments
Closed

Docker daemon crash because of libnetwork issue #1572

bvis opened this issue Nov 23, 2016 · 4 comments

Comments

@bvis
Copy link

bvis commented Nov 23, 2016 •
edited
Loading

Hi,

I don't know if this is the right place to create this issue, but I've seen it seems to be related with libnetwork.

Today I've seen that one of the managers (v1.12.3) of my swarm cluster was not responding. Then I checked the status and it said:

# systemctl status docker
● docker.service
   Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: timeout) since Wed 2016-11-23 17:22:59 UTC; 15min ago
  Process: 29531 ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver aufs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=amazonec2 --log-opt=max-size=100m --log-opt=max-file=5 (code=exited, status=2)
 Main PID: 29531 (code=exited, status=2)

Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork.(*network).(github.com/docker/libnetwork.handleDriverTableEvent)-fm(0x1b653e0, 0xc82167fe50)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:482 +0x34
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork.(*controller).handleTableEvents(0xc8204f02d0, 0xc8210e9800, 0xc82348c0a0)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:523 +0x87
Nov 23 17:21:28 swarm-3 docker[29531]: created by github.com/docker/libnetwork.(*network).addDriverWatches
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:482 +0x3a1
Nov 23 17:21:28 swarm-3 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 23 17:22:59 swarm-3 systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Nov 23 17:22:59 swarm-3 systemd[1]: docker.service: Unit entered failed state.
Nov 23 17:22:59 swarm-3 systemd[1]: docker.service: Failed with result 'timeout'.```

I've not found this reference on any issue but maybe it's something duplicated or fixed on 1.13.

Output of docker info:
After restarting the daemon, of course.

# docker info
Containers: 40
 Running: 4
 Paused: 0
 Stopped: 36
Images: 25
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 282
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: rexray local
 Network: bridge host null overlay
Swarm: active
 NodeID: blgkwwxupbn3ge22549g9dlf6
 Is Manager: true
 ClusterID: ayj83qqfaektzjdvsq0cklskt
 Managers: 3
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.137.146.179
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.2.0-18-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.303 GiB
Name: swarm-3
ID: WGNT:EMUO:ODFS:HXEW:6HE3:NCZV:Z6ZU:EFW2:73XF:QWFV:CL2Z:UCNF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=amazonec2
Insecure Registries:
 127.0.0.0/8

Output of docker version:

# docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:53:11 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:53:11 2016
 OS/Arch:      linux/amd64

** System logs **

Nov 23 17:21:08 swarm-3 systemd-udevd[27328]: Could not generate persistent MAC address for vx-00010d-dc2uo: No such file or directory
Nov 23 17:21:08 swarm-3 kernel: vetha74f2db: renamed from veth2
Nov 23 17:21:08 swarm-3 systemd-udevd[27345]: Could not generate persistent MAC address for vetha74f2db: No such file or directory
Nov 23 17:21:08 swarm-3 kernel: veth05036ee: renamed from eth0
Nov 23 17:21:08 swarm-3 kernel: br0: port 7(veth419) entered disabled state
Nov 23 17:21:08 swarm-3 kernel: vethade4a71: renamed from eth2
Nov 23 17:21:08 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:08 swarm-3 docker[29531]: time="2016-11-23T17:21:08Z" level=info msg="Firewalld running: false"
Nov 23 17:21:08 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09Z" level=info msg="Firewalld running: false"
Nov 23 17:21:09 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09Z" level=info msg="Firewalld running: false"
Nov 23 17:21:09 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09Z" level=info msg="Firewalld running: false"
Nov 23 17:21:09 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09Z" level=info msg="Firewalld running: false"
Nov 23 17:21:09 swarm-3 kernel: docker_gwbridge: port 12(vethb834a20) entered disabled state
Nov 23 17:21:09 swarm-3 kernel: veth71135d0: renamed from eth1
Nov 23 17:21:09 swarm-3 kernel: docker_gwbridge: port 12(vethb834a20) entered disabled state
Nov 23 17:21:09 swarm-3 kernel: device vethb834a20 left promiscuous mode
Nov 23 17:21:09 swarm-3 kernel: docker_gwbridge: port 12(vethb834a20) entered disabled state
Nov 23 17:21:09 swarm-3 kernel: br0: port 7(veth419) entered disabled state
Nov 23 17:21:09 swarm-3 kernel: device veth419 left promiscuous mode
Nov 23 17:21:09 swarm-3 kernel: br0: port 7(veth419) entered disabled state
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09.314960161Z" level=error msg="fatal task error" error="task: non-zero exit (1)" module=taskmanager task.id=c5gilolv
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09.351992343Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:09 swarm-3 docker[29531]: time="2016-11-23T17:21:09.352247709Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:327feacb2
Nov 23 17:21:11 swarm-3 docker[29531]: time="2016-11-23T17:21:11.442968015Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:11 swarm-3 docker[29531]: time="2016-11-23T17:21:11.443108236Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:2a606fa21
Nov 23 17:21:14 swarm-3 docker[29531]: time="2016-11-23T17:21:14.567000659Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:14 swarm-3 docker[29531]: time="2016-11-23T17:21:14.567418193Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:dc234057b
Nov 23 17:21:14 swarm-3 docker[29531]: time="2016-11-23T17:21:14Z" level=info msg="Firewalld running: false"
Nov 23 17:21:14 swarm-3 docker[29531]: time="2016-11-23T17:21:14Z" level=info msg="Firewalld running: false"
Nov 23 17:21:14 swarm-3 docker[29531]: time="2016-11-23T17:21:14Z" level=info msg="Firewalld running: false"
Nov 23 17:21:15 swarm-3 docker[29531]: time="2016-11-23T17:21:15Z" level=info msg="Firewalld running: false"
Nov 23 17:21:15 swarm-3 docker[29531]: time="2016-11-23T17:21:15Z" level=info msg="Firewalld running: false"
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:15 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:16 swarm-3 docker[29531]: time="2016-11-23T17:21:16.654497307Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:16 swarm-3 docker[29531]: time="2016-11-23T17:21:16.654705238Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:eee7bd568
Nov 23 17:21:19 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19Z" level=info msg="Firewalld running: false"
Nov 23 17:21:19 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19Z" level=info msg="Firewalld running: false"
Nov 23 17:21:19 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19Z" level=info msg="Firewalld running: false"
Nov 23 17:21:19 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19Z" level=info msg="Firewalld running: false"
Nov 23 17:21:19 swarm-3 kernel: IPVS: __ip_vs_del_service: enter
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19Z" level=info msg="Firewalld running: false"
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19.578435180Z" level=error msg="Could not find network dc2uozwsl458scj0jtzgca3i5 while handling service table event:
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19.588259024Z" level=warning msg="Error getting v2 registry: Get https://localhost:5000/v2/: http: server gave HTTP r
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19.588309352Z" level=error msg="Attempting next endpoint for pull after error: Get https://localhost:5000/v2/: http:
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19.611027418Z" level=warning msg="Your kernel does not support swap limit capabilities, memory limited without swap."
Nov 23 17:21:19 swarm-3 kernel: aufs au_opts_verify:1597:dockerd[19643]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 23 17:21:19 swarm-3 kernel: aufs au_opts_verify:1597:dockerd[19643]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19.782082777Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:19 swarm-3 docker[29531]: time="2016-11-23T17:21:19.78224225Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:c2ca2c88c1
Nov 23 17:21:22 swarm-3 docker[29531]: time="2016-11-23T17:21:22.910191854Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:22 swarm-3 docker[29531]: time="2016-11-23T17:21:22.910354765Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:4a8e44277
Nov 23 17:21:23 swarm-3 docker[29531]: time="2016-11-23T17:21:23.96153306Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c08
Nov 23 17:21:23 swarm-3 docker[29531]: time="2016-11-23T17:21:23.961673182Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:5c705622d
Nov 23 17:21:24 swarm-3 kernel: aufs au_opts_verify:1597:dockerd[26172]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 23 17:21:24 swarm-3 kernel: aufs au_opts_verify:1597:dockerd[21541]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 23 17:21:24 swarm-3 kernel: IPVS: Creating netns size=2144 id=1269
Nov 23 17:21:24 swarm-3 kernel: br0: renamed from ov-00010d-dc2uo
Nov 23 17:21:24 swarm-3 systemd-udevd[27985]: Could not generate persistent MAC address for vx-00010d-dc2uo: No such file or directory
Nov 23 17:21:24 swarm-3 kernel: vxlan1: renamed from vx-00010d-dc2uo
Nov 23 17:21:24 swarm-3 systemd-udevd[28007]: Could not generate persistent MAC address for vethf86d39d: No such file or directory
Nov 23 17:21:24 swarm-3 kernel: device vxlan1 entered promiscuous mode
Nov 23 17:21:24 swarm-3 kernel: br0: port 1(vxlan1) entered forwarding state
Nov 23 17:21:24 swarm-3 kernel: br0: port 1(vxlan1) entered forwarding state
Nov 23 17:21:24 swarm-3 systemd-udevd[28008]: Could not generate persistent MAC address for vethd89938b: No such file or directory
Nov 23 17:21:24 swarm-3 kernel: veth2: renamed from vethd89938b
Nov 23 17:21:24 swarm-3 kernel: device veth2 entered promiscuous mode
Nov 23 17:21:24 swarm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): veth2: link is not ready
Nov 23 17:21:24 swarm-3 kernel: br0: port 2(veth2) entered forwarding state
Nov 23 17:21:24 swarm-3 kernel: br0: port 2(veth2) entered forwarding state
Nov 23 17:21:24 swarm-3 systemd-udevd[28045]: Could not generate persistent MAC address for veth536ab3f: No such file or directory
Nov 23 17:21:24 swarm-3 systemd-udevd[28048]: Could not generate persistent MAC address for vetha0c2865: No such file or directory
Nov 23 17:21:24 swarm-3 kernel: device vetha0c2865 entered promiscuous mode
Nov 23 17:21:24 swarm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): vetha0c2865: link is not ready
Nov 23 17:21:24 swarm-3 kernel: docker_gwbridge: port 12(vetha0c2865) entered forwarding state
Nov 23 17:21:24 swarm-3 kernel: docker_gwbridge: port 12(vetha0c2865) entered forwarding state
Nov 23 17:21:24 swarm-3 systemd-udevd[28086]: Could not generate persistent MAC address for veth03a38ad: No such file or directory
Nov 23 17:21:24 swarm-3 kernel: veth420: renamed from vethb563767
Nov 23 17:21:24 swarm-3 kernel: device veth420 entered promiscuous mode
Nov 23 17:21:24 swarm-3 kernel: IPv6: ADDRCONF(NETDEV_UP): veth420: link is not ready
Nov 23 17:21:24 swarm-3 kernel: br0: port 7(veth420) entered forwarding state
Nov 23 17:21:24 swarm-3 kernel: br0: port 7(veth420) entered forwarding state
Nov 23 17:21:24 swarm-3 docker[29531]: time="2016-11-23T17:21:24Z" level=info msg="Firewalld running: false"
Nov 23 17:21:24 swarm-3 docker[29531]: time="2016-11-23T17:21:24Z" level=info msg="Firewalld running: false"
Nov 23 17:21:24 swarm-3 docker[29531]: time="2016-11-23T17:21:24Z" level=info msg="Firewalld running: false"
Nov 23 17:21:24 swarm-3 docker[29531]: time="2016-11-23T17:21:24Z" level=info msg="Firewalld running: false"
Nov 23 17:21:24 swarm-3 docker[29531]: time="2016-11-23T17:21:24Z" level=info msg="Firewalld running: false"
Nov 23 17:21:24 swarm-3 kernel: IPVS: Creating netns size=2144 id=1270
Nov 23 17:21:24 swarm-3 docker[29531]: time="2016-11-23T17:21:24Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25.019317476Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25.019469662Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:ba5b97a6f
Nov 23 17:21:25 swarm-3 kernel: eth0: renamed from vethf86d39d
Nov 23 17:21:25 swarm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth2: link becomes ready
Nov 23 17:21:25 swarm-3 kernel: docker_gwbridge: port 12(vetha0c2865) entered disabled state
Nov 23 17:21:25 swarm-3 kernel: br0: port 7(veth420) entered disabled state
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 kernel: eth1: renamed from veth536ab3f
Nov 23 17:21:25 swarm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vetha0c2865: link becomes ready
Nov 23 17:21:25 swarm-3 kernel: docker_gwbridge: port 12(vetha0c2865) entered forwarding state
Nov 23 17:21:25 swarm-3 kernel: docker_gwbridge: port 12(vetha0c2865) entered forwarding state
Nov 23 17:21:25 swarm-3 kernel: eth2: renamed from veth03a38ad
Nov 23 17:21:25 swarm-3 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth420: link becomes ready
Nov 23 17:21:25 swarm-3 kernel: br0: port 7(veth420) entered forwarding state
Nov 23 17:21:25 swarm-3 kernel: br0: port 7(veth420) entered forwarding state
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:25 swarm-3 docker[29531]: time="2016-11-23T17:21:25Z" level=info msg="Firewalld running: false"
Nov 23 17:21:26 swarm-3 docker[29531]: time="2016-11-23T17:21:26.074944721Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:26 swarm-3 docker[29531]: time="2016-11-23T17:21:26.075220169Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:b1217921f
Nov 23 17:21:27 swarm-3 docker[29531]: time="2016-11-23T17:21:27.128399612Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:27 swarm-3 docker[29531]: time="2016-11-23T17:21:27.129243233Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:c0449bc2c
Nov 23 17:21:28 swarm-3 docker[29531]: time="2016-11-23T17:21:28.180742149Z" level=warning msg="containerd: unable to save fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c0
Nov 23 17:21:28 swarm-3 docker[29531]: time="2016-11-23T17:21:28.18105656Z" level=info msg="containerd: fe1f27858b0cbd3cb49687f618c5f014202d5143937d3b267b7534c082eaf622:353635ad54
Nov 23 17:21:28 swarm-3 docker[29531]: panic: runtime error: slice bounds out of range
Nov 23 17:21:28 swarm-3 docker[29531]: goroutine 15619701 [running]:
Nov 23 17:21:28 swarm-3 docker[29531]: panic(0x1a96260, 0xc82000c030)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork/osl.(*networkNamespace).DeleteNeighbor(0xc8250c4a50, 0xc823483c70, 0x10, 0x10, 0xc823483cb0, 0x6, 0x6, 0x1, 0x0
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/osl/neigh_linux.go:82 +0x9b0
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork/drivers/overlay.(*driver).peerDelete(0xc820172380, 0xc82453ce40, 0x19, 0xc8222b2f00, 0x40, 0xc823483c70, 0x10,
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/peerdb.go:346 +0x62c
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork/drivers/overlay.(*driver).EventNotify(0xc820172380, 0xc82453ce03, 0xc82453ce40, 0x19, 0xc822160d00, 0x12, 0xc82
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/joinleave.go:187 +0x77b
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork.(*network).handleDriverTableEvent(0xc825c18c80, 0x1b653e0, 0xc82167fe50)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:560 +0x40c
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork.(*network).(github.com/docker/libnetwork.handleDriverTableEvent)-fm(0x1b653e0, 0xc82167fe50)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:482 +0x34
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork.(*controller).handleTableEvents(0xc8204f02d0, 0xc8210e9800, 0xc82348c0a0)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:523 +0x87
Nov 23 17:21:28 swarm-3 docker[29531]: created by github.com/docker/libnetwork.(*network).addDriverWatches
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/agent.go:482 +0x3a1
Nov 23 17:21:28 swarm-3 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 23 17:21:39 swarm-3 kernel: br0: port 1(vxlan1) entered forwarding state
Nov 23 17:21:39 swarm-3 kernel: br0: port 2(veth2) entered forwarding state
Nov 23 17:21:40 swarm-3 kernel: docker_gwbridge: port 12(vetha0c2865) entered forwarding state
Nov 23 17:21:40 swarm-3 kernel: br0: port 7(veth420) entered forwarding state
Nov 23 17:22:59 swarm-3 systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Nov 23 17:22:59 swarm-3 systemd[1]: docker.service: Unit entered failed state.
Nov 23 17:22:59 swarm-3 systemd[1]: docker.service: Failed with result 'timeout'.
Nov 23 17:22:59 swarm-3 kernel: device veth3 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 3(veth3) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device veth2 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 2(veth2) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device vxlan1 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 1(vxlan1) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device veth5 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 5(veth5) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device vxlan1 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 1(vxlan1) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device veth2 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 2(veth2) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device vxlan1 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 1(vxlan1) entered disabled state
Nov 23 17:22:59 swarm-3 kernel: device veth2 left promiscuous mode
Nov 23 17:22:59 swarm-3 kernel: br0: port 2(veth2) entered disabled state
@aboch
Copy link
Contributor

aboch commented Nov 23, 2016

Thanks @bvis
Do you still have the daemon logs, we should find some more info about the failure in there.

@bvis
Copy link
Author

bvis commented Nov 23, 2016

I'll add a portion of the logs to the task description.

@aboch
Copy link
Contributor

aboch commented Nov 23, 2016 •
edited
Loading

Ahh

Nov 23 17:21:28 swarm-3 docker[29531]: panic: runtime error: slice bounds out of range
Nov 23 17:21:28 swarm-3 docker[29531]: goroutine 15619701 [running]:
Nov 23 17:21:28 swarm-3 docker[29531]: panic(0x1a96260, 0xc82000c030)
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork/osl.(*networkNamespace).DeleteNeighbor(0xc8250c4a50, 0xc823483c70, 0x10, 0x10, 0xc823483cb0, 0x6, 0x6, 0x1, 0x0
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/osl/neigh_linux.go:82 +0x9b0
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork/drivers/overlay.(*driver).peerDelete(0xc820172380, 0xc82453ce40, 0x19, 0xc8222b2f00, 0x40, 0xc823483c70, 0x10,
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/peerdb.go:346 +0x62c
Nov 23 17:21:28 swarm-3 docker[29531]: github.com/docker/libnetwork/drivers/overlay.(*driver).EventNotify(0xc820172380, 0xc82453ce03, 0xc82453ce40, 0x19, 0xc822160d00, 0x12, 0xc82
Nov 23 17:21:28 swarm-3 docker[29531]: /usr/src/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/joinleave.go:187 +0x77b

Thank you @bvis , I recently fixed this in #1555 and the fix is going to be in docker 1.13
(Already present in the 1.13.x branch)

@aboch
Copy link
Contributor

aboch commented Nov 23, 2016

Thanks @bvis this was also reported internally and was fixed by #1555
Next docker release will have the fix

@aboch aboch closed this as completed Nov 23, 2016
@aboch aboch mentioned this issue Nov 28, 2016
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bvis @aboch