[Performance] Check if Jenkins is updating plugins at startup

[chmouel]

There is some report that jenkins is updating its plugins and then cause some slowness,

• Investigate if and how this happening
• Disable this behavior if this is happening.

[chmouel]

@rupalibehera please update this issue with your finding, /cc @pbergene

[joshuawilson]

please add team and type

[pradeepto]

Already discussed with @rupalibehera but pasting her for posterity. The only "official" way, that I am aware of, to stop plugins from upgrading is to pin them.

https://jenkins.io/doc/book/managing/plugins/#pinned-plugins

[chmouel]

I could not see anywhere in the logs where the jenkins plugins get udpated, any ideas where is this @rupalibehera ?

[rupalibehera]

@chmouel, Yes Jenkins plugins don't get updated for sure but they might checking for updates

[chmouel]

@rupalibehera It was reported that it was at boot time but i could not see it in the logs (which is part of the perf issues), is it a periodic tasks? I don't see any documentations anywhere about it,

[chmouel]

@rupalibehera @lordofthejars can you ack on this please if that happen or not and cause slowdown ?

[lordofthejars]

I can check this by just starting and Jenkins and checking or it is something more specific to our Jenkins instance? Can I just run this locally in my Docker instance https://github.com/fabric8-jenkins/jenkins-openshift-base and see?

[chmouel]

I believe we just need a ack

Should we care?
Is it slowing us down ?
Is it only at boot time?
Is it in a scheduled manner.

If it does (as reported) we need to figure a way to make it more robust,
i.e: do we need to have plugins pining, what is a sane way to manage the dependencies pinning.

[rupalibehera]

@lordofthejars , you can run it locally only if it is running inside kubernetes/openshift cluster.
or you can test it on your OSIO account by changing the image in the jenkins tenant dc.

[rupalibehera]

@chmouel , I am not able to assign the issue to @lordofthejars

[chmouel]

@maxandersen just added him to openshiftio org, I think @lordofthejars may need to accept the invite

[lordofthejars]

I have already invited

[jaseemabid]

I can check this by just starting and Jenkins and checking or it is something more specific to our Jenkins instance? Can I just run this locally in my Docker instance https://github.com/fabric8-jenkins/jenkins-openshift-base and see?

@lordofthejars We run https://github.com/fabric8io/openshift-jenkins-s2i-config, derived from the base image you mentioned.

[lordofthejars]

@jfchevrette I have started reviewing this issue. Regarding if plugins are updated during start up, I have not noticed anything related to this in my pod. I have been putting in IDLE jenkins pod and restarting again and I have not seen anything that suggests that the plugins are updated. Of course this means nothing since maybe it happens every X time and I have been barley unlucky to not see that. So if you have a log when this happens, can you send it to me so I can continue the investigation?

Maybe not related to this but also a performance issue I have opened this issue to understand what is this process that takes so much time:
https://stackoverflow.com/questions/49155218/startup-performance-in-jenkins

[lordofthejars]

@jfchevrette news about metadata https://twitter.com/alecharp/status/971684446192709632 so it seems that plugins are not updated by default and requires some kind of script. Other thing to think about is if our jenkins installation bundles this script.

[chmouel]

@lordofthejars it may be good to figure out why this file takes 40s as well https://updates.jenkins-ci.org/current/update-center.json (takes less than a second on my laptop), or is it actually doing something else too ?

[lordofthejars]

This time is for downloading the update-center.json and compare with current plugins. Why it take so much? probably because of network latency. Since user has no permissions to update plugins, we should check how to disable this process.

[chmouel]

I don't think it's the latency for that file, trying this from inside the containers :

$$ oc rsh jenkins-1-kbrnp                                                                               
$ time curl -o/dev/null https://updates.jenkins-ci.org/current/update-center.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1264k  100 1264k    0     0   746k      0  0:00:01  0:00:01 --:--:--  746k

real	0m1.779s
user	0m0.062s
sys	0m0.089s
sh-4.2$

Not saying that the slowness we have seen is not because of the update process (and as you said it's not worth to have it) but i would imagine that there is something going on that takes time after or before the file download.

[lordofthejars]

notice that the time is printed when say Finished Downloaded I can take a look at Jenkins code, but it should not do anything else.

[chmouel]

@lordofthejars would you have an update for us here?

[lordofthejars]

About this no idea, now I am just working on updating plugins for Jenkins since after we removed some of them, then we have some failures at boot up, so checking what's happening there. This is going to be hard to know why it takes so many time on this.

[chmouel]

I guess we can close this issue (since it's an investigation story) then and come back to it if needed. So to recap:

• jenkins is getting the list of updates from this URL
• jenkins doesn't do the actual update of the plugins
• in between the download there is something (networking or others) that takes times which is hard to know what's going on,