Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flood Docker slaves IDLE #54

Closed
Liryna opened this issue Jul 4, 2014 · 5 comments
Closed

Flood Docker slaves IDLE #54

Liryna opened this issue Jul 4, 2014 · 5 comments

Comments

@Liryna
Copy link

Liryna commented Jul 4, 2014

Hi,

I got some flood in the compilation list:
docker
(It is just a part of the screen...there is more)

Here is the jenkins logs:

juil. 04, 2014 2:13:02 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud connect

Building connection to docker host liblogicalaccess-linux-docker-x64 URL http://10.2.3.132:4243

juil. 04, 2014 2:13:02 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud connect

Building connection to docker host liblogicalaccess-linux-docker-x64 URL http://10.2.3.132:4243

juil. 04, 2014 2:13:02 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud connect

-- a lot more --
....

And after some times this appear on the log:

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

Checking DockerComputer{name=59db36ffcb90@liblogicalaccess-linux-docker-x64, slave=DockerSlave{name=59db36ffcb90@liblogicalaccess-linux-docker-x64, containerId=59db36ffcb90a6d507f037a836ec182e84a6c427a88234588359208a8ce754ea, template=DockerTemplate{image=liryna/liblogicalaccess-x64:build, parent=DockerCloud{name=liblogicalaccess-linux-docker-x64, serverUrl=http://10.2.3.132:4243}}}}

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

currentlyChecking: false

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud connect

Building connection to docker host liblogicalaccess-linux-docker-x64 URL http://10.2.3.132:4243

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

Checking DockerComputer{name=7264d5e06408@liblogicalaccess-linux-docker-x64, slave=DockerSlave{name=7264d5e06408@liblogicalaccess-linux-docker-x64, containerId=7264d5e0640828923fda547cb2e8aefc33fdb93eb6a380984b484cba598cf7f8, template=DockerTemplate{image=liryna/liblogicalaccess-x64:build, parent=DockerCloud{name=liblogicalaccess-linux-docker-x64, serverUrl=http://10.2.3.132:4243}}}}

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

currentlyChecking: false

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud connect

Building connection to docker host liblogicalaccess-linux-docker-x64 URL http://10.2.3.132:4243

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

Checking DockerComputer{name=742542d16376@liblogicalaccess-linux-docker-x64, slave=DockerSlave{name=742542d16376@liblogicalaccess-linux-docker-x64, containerId=742542d163761be500d9d0abc0641931b8f5d174631a3e223f8a8fc956ed8e24, template=DockerTemplate{image=liryna/liblogicalaccess-x64:build, parent=DockerCloud{name=liblogicalaccess-linux-docker-x64, serverUrl=http://10.2.3.132:4243}}}}

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

currentlyChecking: false

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud connect

Building connection to docker host liblogicalaccess-linux-docker-x64 URL http://10.2.3.132:4243

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

Checking DockerComputer{name=77f4c7bb539f@liblogicalaccess-linux-docker-x64, slave=DockerSlave{name=77f4c7bb539f@liblogicalaccess-linux-docker-x64, containerId=77f4c7bb539f4e97fc502a2eb8ea36ef4ab6badcbe79a0a18fe191d101b8af93, template=DockerTemplate{image=liryna/liblogicalaccess-x64:build, parent=DockerCloud{name=liblogicalaccess-linux-docker-x64, serverUrl=http://10.2.3.132:4243}}}}

juil. 04, 2014 2:17:06 PM INFO com.nirima.jenkins.plugins.docker.DockerRetentionStrategy check

currentlyChecking: false
-- a lot more --
....

If I look on the slaves informations, they are all created by SYSTEM and attached to none project.

The container is only used by one project that launched every 12 hour.

Every slaves are and stay connected:

4f3dc35bc26a        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49193->22/tcp   desperate_galileo
ff8fada8b2dd        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49192->22/tcp   jovial_babbage
a34dd1ccd0e8        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49190->22/tcp   angry_brown
1d474fb1a996        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49188->22/tcp   loving_goldstine
b0b9690858ba        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49187->22/tcp   ecstatic_turing
09bee119fef4        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49186->22/tcp   focused_babbage
084418970dc0        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49185->22/tcp   compassionate_euclid
800241ca8fb9        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49184->22/tcp   suspicious_lalande
a08d1c776b81        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49183->22/tcp   drunk_sammet
396d9807c095        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49182->22/tcp   pensive_bartik
bddc4f55c6b6        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49181->22/tcp   cocky_kowalevski
1993e3ead112        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49180->22/tcp   insane_albattani
8edc8be601ba        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49179->22/tcp   dreamy_hopper
c3bebb17f3c9        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49178->22/tcp   trusting_heisenberg
2038d85b6677        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49177->22/tcp   pensive_heisenberg6
08c2ef098a3d        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49176->22/tcp   tender_engelbart
aaff07ccf484        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49175->22/tcp   berserk_wright
4c41da41328a        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49174->22/tcp   sleepy_hypatia
1e28a361580f        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49173->22/tcp   mad_torvalds
f74e2722aa9c        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49172->22/tcp   cranky_almeida
81b44c48cbbb        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49171->22/tcp   desperate_brown
b1de8691cc0f        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49170->22/tcp   sick_pare
1560209d7381        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49169->22/tcp   prickly_bardeen
77f4c7bb539f        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49167->22/tcp   cranky_fermat
3f7eae37fbc3        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49166->22/tcp   pensive_hopper
7264d5e06408        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49165->22/tcp   jolly_franklin
521bfad2a5e3        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49164->22/tcp   jolly_lovelace
f6a714609d61        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49163->22/tcp   tender_hopper
ccf7b6786571        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49162->22/tcp   romantic_kowalevski
f119147e663f        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49161->22/tcp   loving_kirch
0b79ef5ff0b2        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49160->22/tcp   suspicious_jones
f93fe1c5ed76        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49159->22/tcp   evil_leakey
496e51bba6c7        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49158->22/tcp   clever_curie
59db36ffcb90        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49157->22/tcp   prickly_euclid
49a4df1257ec        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49156->22/tcp   hopeful_lalande
9916856b367d        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49155->22/tcp   hungry_ardinghelli
742542d16376        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49154->22/tcp   insane_nobel
96e80bd6181a        liryna/liblogicalaccess-x64:build   /usr/sbin/sshd -D   About an hour ago   Up About an hour         0.0.0.0:49153->22/tcp   furious_ardinghelli

I use docker 1.0.0 on debian jessie.
My Jenkins is ver. 1.564 on windows with docker plugin 0.6.2.

I usually restart docker service to make them disappear.
I can not tell when it happen...
@Liryna Liryna changed the title Flood Docker IDLE Flood Docker slaves IDLE Jul 4, 2014
@jocelyndrw
Copy link
Contributor

I also observe the same comportment and haven't been able to track-it down.

In fact, I'd like to be able to enable this as a feature, so Jenkins could "preload" the slaves (within the max container or max instance limit), reducing the cloud plugin overhead when starting a lot of jobs at the same time.

(Jenkins 1.567, Docker 1.0.0 on CentOS 6)

@thomassuckow
Copy link
Contributor

I need to do more investigating, which will probably involve building the plugin myself. But I believe what is happening is that jobs with no label restriction are causing a docker container to be provisioned. The logic in the DockerCloud is to use the first container when a null label is requested, thus they are all the same container getting provisioned. However, for some reason, Jenkins thinks the provisioned slave is not satisfactory and will not send the job to it. Even worse, the Docker retention policy will not terminate an idle slave if it has not done work.

thomassuckow pushed a commit to thomassuckow/docker-plugin that referenced this issue Jul 16, 2014
Add an option to control the slave mode
f89b213 
* Fix provisioning a slave on a job for 'any' slave

Resolves cause of jenkinsci#54, jenkinsci#43
thomassuckow pushed a commit to thomassuckow/docker-plugin that referenced this issue Jul 16, 2014
Add timeout for a slave that gets provisioned but then has no work
9bb571e 
Resolves effect of jenkinsci#54
@thomassuckow thomassuckow mentioned this issue Jul 21, 2014
Closed
@thomassuckow
Copy link
Contributor

Here is a question for you. We are discussing on #59, whether it is worth having the "Utilize node as much as possible"/"Tied jobs only" option on containers. Alternatively should it be hard coded one way or another?

@thomassuckow thomassuckow mentioned this issue Jul 21, 2014
Closed
thomassuckow pushed a commit to thomassuckow/docker-plugin that referenced this issue Jul 21, 2014
Add timeout for a slave that gets provisioned but then has no work
7650a8a 
Resolves effect of jenkinsci#54

Also fix DockerTemplateTest

This was broken before by jenkinsci#46, but also needed to be updated for my changes
@thomassuckow thomassuckow mentioned this issue Jul 21, 2014
Merged
thomassuckow pushed a commit to thomassuckow/docker-plugin that referenced this issue Jul 21, 2014
Fix a discrepancy with untied jobs
37a49b1 
The plugin advertised supporting untied jobs but then all slaves were
EXCLUSIVE

Fixes the root cause of jenkinsci#54
@thomassuckow thomassuckow mentioned this issue Jul 21, 2014
Merged
@Liryna
Copy link
Author

Liryna commented Jul 21, 2014

Thanks alot for these fix ! can't wait to test them :) !!!

@magnayn
Copy link
Contributor

magnayn commented Jul 22, 2014

Fixed in 0.7 release.

@magnayn magnayn closed this as completed Jul 22, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@magnayn @jocelyndrw @thomassuckow @Liryna