-
Notifications
You must be signed in to change notification settings - Fork 670
systemd weave script issues #1607
Comments
I would like to add that i moved the files to /usr/lib/systemd/system and made a symbolic link in /etc/systemd/system as that seems to comply with existing systemd scripts. |
The What exactly isn't working? |
From the command line "weave launch" starts two docker containers; weave and weaveproxy. When i reboot my machine with the original systemd script only the weave container is running and it cannot be stopped by systemctl. systemctl status weave gives the following errors; weave.service operation timed out. exiting With the modification above both containers are started. |
This doesn't make much sense, tbh. Alas I am not a systemd expert. @errordeveloper might be able to figure out what's going on here. |
First of all, @WitchDoc42, thank you for reporting this issue to us. As you said, you are new to systemd, so please let me try to demystify this a little bit. As @rade said, the I am not sure why do use say that it "hangs" when no peers are found... I assume you are implying that when The way you split one unit into two (with You certainly don't need to put unit files in |
Could you please provide the output of |
@WitchDoc42 ping |
@WitchDoc42 I am closing this; if you have more info please re-open. |
I don't know if I'm experiencing the same issue as @WitchDoc42 , but I am having problems getting systemd to start weave automatically on boot. When I use the documented procedure (having systemd run "launch", it hangs). Sometimes I get complaints about the weave plugin or the weave proxy having already started. I split my launch into three commands and start the plugin, proxy, and router on their own.
Weave still fails to start on boot. When I look, there is a docker run command which the weave launch started, and the docker run command simply hangs. When I run the exact same weave launch command that hangs when executed by systemd, weave starts fine. I thought there might be some kind of race condition with startup order. I tried having a 10 second delay before the weave launch is executed. This did not help. |
Does |
@rade Weave report just complains the weave container isn't running:
Using an @reboot cronjob which waits 10 seconds and then launches weave works. Here are some logs of a failed systemd based startup.
I think the issue may have something to do with the weave network interfaces not existing/getting created. |
So this is definitely a different issue to the one reported by the OP since their report predates the incorporation of the Weave docker network plugin into Do you experience any issues when not launching the plugin? (Warning: docker becomes very unhappy if it cannot find a plugin that it thinks is still in use). |
Here's a boot where I reverted to doing a regular weave launch, and before rebooting, I docker rm -f'd all containers - even dead containers - such that the container with a docker restart policy isn't around.
To further narrow things down, I ran a fresh boot with all containers removed prior to boot. On this boot, the only thing systemd is running is It feels to me like the weave router depends on a some part of the boot process having completed which hasn't yet been completed when docker and then it get launched. |
Please post the unit definitions for that most recent of your tests. Are you sure Docker has forgotten about the weave network? It remembers things even across reboots. Make sure there isn't anything unusual in the docker daemon logs (i.e. the kind of stuff you saw in #1607 (comment)). Also, starting weave as |
Thanks for your help here, I really appreciate it. It turns out the issue had nothing to do with Weave and everything to do with a broken Docker installation. When I spun up a fresh VM & ran my CM tools against it to set up a minimal test environment, the problem stopped being reproducible. I was able to identify the difference: My test environment was an instance where I had allowed docker-machine to provision docker on the VM. It installed a systemd file in /etc/systemd/system with a problematic approach to sockets & dependencies (doesn't list any!). This systemd unit file was taking precedence over the systemd unit file in /lib. Which meant that weave's systemd unit was having it's docker dependency satisfied by the broken configuration that docker-machine put in, where niceties like sockets being carefully created and networks being available are ignored. |
@DanielDent Would you please expand on "problematic approach to sockets"? What was the problem and what was the fix? |
I think you'll find the information you are looking for in the related issue docker/machine#2795 - if you have a more specific question, let me know. |
I have some issues with the systemd script. I run weave one a single system, without peers. The "docker attach weave" command is unnecessary in my case and, since no peers are found, hangs. (i'm not sure how long the time-out is, have not waited for it). I resolved this issue by adding a second "service" that executes docker weave attach and that requires weave.service. Since i do not have peers ate the moment this service is not enabled on my system. I cannot test it either.
weave-peers.service;
I removed the docker line from the original scipt and made weave launch the ExecStart target. That was not enough because the ExecStop target is fired directly after ExecStart, killing the just started weave instance. This was fixed by adding "RemainAfterExit=yes".
weave.service;
I am running weave on an up to date CentOs 7 host. I do not know much about systemd and can not say if the above modification is desirable but it works for my environment.
The text was updated successfully, but these errors were encountered: