Jenkins moves plugins on every startup

[kbsingh]

Every time Jenkins boots, either due to new-init for user, or un-idle or pod relocation for infra work or on user demand, it will move 400+ MB of data to /var/lib/jenkins/plugins - if this is anything other than a new-init, Jenkins first removes each of the plugins, one at a time, and then replaces the exact same content into place.

/var/lib/jenkins/ is mounted as a pvc

We should not be moving this data into the jenkins-home PVC - can we replace the /var/lib/jenkins/plugins/ folder to just be a symlink to the place the content is stored anyway ? we are not expecting this content to be replaced, and in basic testing it looks ok to sit on container storage.

[kbsingh]

adding @jfchevrette and @mmclanerh here to track

[adisakala]

@pradeepto @stevengutz please prioritize this accordingly.

[joshuawilson]

Is this really blocking production or is it a Critical-Dependency?

[rupalibehera]

@kbsingh this is something started happening today itself ? like moving 400+ MB of data to /var/lib/jenkins/plugin everytime ? just curious how did we found this out ?

[chmouel]

@joshuawilson I think this is Critical-Dependency, I think we really need to have a tag called Critical-Performance which make more sense for a lot of issues we have been working on (and can be treated as such for us)

Is this really blocking production or is it a Critical-Dependency?

[lordofthejars]

@kbsingh I am trying to figure out what's happening here, for what I understand at the end everything is related in https://github.com/fabric8-services/fabric8-tenant-jenkins/blob/eeca24d9874760dd4472b394cc8ef33b5b1fb770/apps/jenkins/src/main/fabric8/openshift-deployment.yml#L88

When you say creating a symbolic link, of course this symbolic link should be created in the start script, my question is exactly from where we should create a symbolic link, this is the part that I understand why we would save time but not how to try it.

[kbsingh]

that is right, the /var/lib/jenkins comes from a PVC. What happens is that a plugins dir is created under there and the plugins from the container are copied out to the PVC, repeatedly. What specific info can I help you track down ?

[rupalibehera]

@lordofthejars , I just disabled the behavior of copying the plugins every time , here is the PR fabric8-jenkins/jenkins-openshift-base#18 I need to test it yet.

[rupalibehera]

@lordofthejars @pradeepto @adisakala @stevengutz
I did test this image fabric8/jenkins-openshift:SNAPSHOT-PR-154-1 on both old and new clusters, it does not copy the plugins over and over again. Tested it by un-idling the jenkins image and running a quickstart it was fine.
Once this PR is merged fabric8io/openshift-jenkins-s2i-config#154 it auto deploys the tenant image. I am not merging this one now. I will do this on monday.
@jfchevrette , if we can test the above image on cluster to check the performance as suggested by @kbsingh.

It does solve the current problem so we should go forward with this. But I have one concern with this solution that if tomorrow we add/remove new plugins will that be copied over or not, need to think more on it.

[lordofthejars]

I agree with plugins problem (or for example an update of versions) but for now I will leave as is, let's see if it fixes, and in the near future let's see if it is best option to rewrite for just OpenShift IO purposes.

[adisakala]

What is the ETA on this issue?

[lordofthejars]

We merge the issue on Friday and we agreed with @kbsingh to move to prod tomorrow (Monday) EU morning.

[rupalibehera]

@adisakala , @kbsingh @lordofthejars , we have released the changes. we need to do a tenant wide update ,
I have raised an issue for platform team here #2670

[msrb]

But I have one concern with this solution that if tomorrow we add/remove new plugins will that be copied over or not, need to think more on it.

I think this @rupalibehera's comment is spot on. Jenkins plugins are baked-in inside the image and copied over to the $JENKINS_HOME on the first and all subsequent starts. I don't think (?) OSIO users have permissions to update/install plugins as they wish. They rely on us to give them a well-tested combination of plugins that work together. And we may need to update some of the plugins in future (CVEs, new features).

I think what we could do here is to store information about Jenkins image version alongside plugins in $JENKINS_HOME directory and then check on startup if the image is still the same. And only If it isn't, then wipe out plugins/ directory and copy the new plugins from the image.