-
Notifications
You must be signed in to change notification settings - Fork 616
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fabio running using Nomad system scheduler breaks Docker. #192
Comments
Thanks for the detailed description. My suspicion is that this isn't Fabio related but you've done testing around this. I'm currently on vacation and will start picking up Fabio issues as of next week again.
|
@michaelmcguinness Did you make any progress with this? What do the fabio logs say? |
No. Running it as raw_exec to work around the issue. Would I be right in saying that you mean the logs exposed by
That being the case there is nothing interesting there...
and then just repeated messages about Consul. |
If there is nothing in the fabio logs then that supports my suspicion that this is a docker and/or nomad issue. Maybe the way fabio interacts with docker or the way it shuts down triggers this. However, since you're running this with a single listener you could try to simulate this with a simple go program that runs a web server, then a reverse proxy, and then a reverse proxy that makes long polling http requests. Below is a simple reverse proxy for testing. Store it in You can use the following
See how far you get with this.
|
Thanks I'll give this a go and let you know the result. Actually I upgraded to Nomad 0.5.1 just in case but no change. |
This issue remains with this test code so as you suspected it is a more generic issue. I actually also tested it with the Consul binary and got the same thing. Thank you for your time on this. I'll head over to the Nomad issues page with it. As a side note, what should have I been expecting from 'curl localhost:9999'. I am getting a 404. |
You may (or may not :)) be interested to know that this issue is something to do with the kernel version and the LVM storage driver implementation. Haven't quite figured it out but switching to AUFS makes the issue go away. Thanks again for your attention and a great utility. |
I am interested and I'm glad you've figured it out. If you have a reference issue for nomad feel free to link it. Thanks and merry christmas. Enjoy the holidays. |
I raised this up with Kelsey Hightower as it was his demo that made me look at it. Not for a fix but just for some info about his env. I'm not sure I am going to burn time raising it with Nomad as we are way behind here with our kernel revision (one of the many things on my to-do list). It seems to me that LVM may not be the strategic storage option for Docker so I think fixing forward by upgrading is the way to go. |
I realise how unlikely the title to this issue seems but if there is an obvious error in my set up I can't spot it. I want to run Fabio as a Nomad managed service using the Nomad system scheduler (type = "system"). When I do then any subsequent pulls from our private Docker registry fails with the error
failed to register layer: open /dev/mapper/docker-202:32-786433-35e363b33db58a87d6a55b19f3297715b9978052e70edec86f03b51af3e44455: no such file or directory
From that point on I am not able to recover Docker.
Some details about our set up:
Ubuntu 14-04
Kernel = 3.13.0-53-generic
Docker = 1.12.2
Nomad = 0.5.0
Fabio = 1.3.4
I have a 3 x servers with 2 x clients. I am trying to run Fabio using the exec driver and the system scheduler. I am running Nomad as the root user on which I believe is required for the exec driver.
I do not see the issue if I run Fabio using the service scheduler.
I do not see the issue if I run a Docker container using the system scheduler .
I do not see the issue if I run another job (sleep binary) using the system scheduler.
I do not see the issue if I run Fabio using the system scheduler but using the raw_exec driver.
Docker is using the LVM storage option but I see the same issue if I drop back to the devicemapper storage option.
Below is a repeatable test case. After that are copies of the job specs used in the test case.
Go to Nomad user
ubuntu@ip-10-75-70-27:~$ sudo su - nomad
Software versions
From Docker log
Fabio Job Spec
Sleep Job Spec
The text was updated successfully, but these errors were encountered: