Longer CI-BUild and I-Build statistic history

[jukzi]

Currently only 10 builds are kept. For logfile etc that is typically sufficient, but it would be helpfull to have longer statistics like Test Result Trend and buildTimeTrend
for exmple https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/master/buildTimeTrend

Especially after a vacation it's not visible what changed and if there had been silent regression in build time. A buildTimeTrend over the last quarter would be helpfull.
I don't know how to configure that, or if a special plugin is needed. https://plugins.jenkins.io/build-time-blame/ ?

[akurtakov]

@fredg02 Can you help here?

[fredg02]

• Nitpick: I see 15 builds. 🙂

• How can this be configured: see https://github.com/eclipse-jdt/eclipse.jdt.core/blob/master/Jenkinsfile#L4

• We generally advise doing two things:

  • keep the number of builds reasonable
  • don't keep build artifacts around for more than 1 or 2 builds

  So I'd recommend setting this to something like:

  buildDiscarder(logRotator(numToKeepStr: '50', artifactNumToKeepStr: '2'))
  

---

A buildTimeTrend over the last quarter would be helpfull.

Taking https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/master/ as a good example, here are the current stats:

• on the master branch, 15 builds are kept. Each build uses between 80-100 MB. Total size of the master branch job: 1.4 GB.
• between December 4 and today, the builds have been executed every day (except December 7) between 1 and 4 times
• So 15 builds would span a history between almost 4 days (with 4 builds a day) and a maximum of 15 days (with 1 builds per day, not counting days without builds)
• Let's scale this up to a quarter:
  • 90 days of 1 build per day => 90 builds with ~100 MB => 9 GB or
  • 90 days of 4 builds per day => 360 builds with ~100 MB => 36 GB
  • I'd assume that the truth is somewhere in the middle, let's say 180 builds => 18 GB
• https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github currently shows 21 branches and 81 pull requests. In most cases, the branches and especially the PRs use the same Jenkinsfile as the master branch. The JDT JIPP has 17 jobs, 5 of them are multi-branch builds. I assume, you can already tell where I'm going with that.

It does not scale. (without some adjustments)

Proposed adjustments:

1. To keep a longer build history and a longer build time trend, I would use the buildDiscarder settings shown above (keep 50 builds, but only 2 builds with artifacts).
2. Clean up by filtering out old branches in the branch sources settings and merging or closing old PRs (especially if they are several months or years old).
3. Keep a good build config hygiene with ALL builds jobs and clean up old builds/jobs regularly.

[jukzi]

Is it possible to use higher values for master branch only?

[jukzi]

Is it possible to use higher values for master branch only?

@fredg02 would something like env.BRANCH_NAME.equals("master") ? '99' : '3' work on our jenkins? example found in
https://github.com/OpenLMIS/openlmis-referencedata-ui/blob/353f593476605d695eb865ca6804259cad356112/Jenkinsfile#L7

[HannesWell]

Is it possible to use higher values for master branch only?

@fredg02 would something like env.BRANCH_NAME.equals("master") ? '99' : '3' work on our jenkins? example found in https://github.com/OpenLMIS/openlmis-referencedata-ui/blob/353f593476605d695eb865ca6804259cad356112/Jenkinsfile#L7

That's possible, see for example
https://github.com/eclipse-m2e/m2e-core/blob/02d733c939e83248b12fbae40a195f27aa524e33/Jenkinsfile#L4

It could be even a bit simpler:
buildDiscarder(logRotator(numToKeepStr:'5', artifactNumToKeepStr: ('master' == env.BRANCH_NAME) ? '5' : '1' ))

[jukzi]

Since jdt has branches like BETA_JAVA20 which are as important as master, i would like to know a syntax that excludes PR, do you know something for that?

[HannesWell]

Since jdt has branches like BETA_JAVA20 which are as important as master, i would like to know a syntax that excludes PR, do you know something for that?

IIRC PR branch names are PR-<number>, but I can check that.
But usually I prefer to specify for what a special rule is applied, I think that's more robust and clear than specifyng exclusions. In case of JDT, if I understand it right, it means for master and
BETA_JAVA\d+more builds should be retained?

In general you can use almost any groovy expression that produces an int.
I can try to work out an explicit example this evening.

[jukzi]

That would be nice. i think the R*maintenance branches don't need a long history, master and current(?) BETA* should be an exception.

[fredg02]

Is it possible to use higher values for master branch only?

Yep, good point.

buildDiscarder(logRotator(numToKeepStr:'5', artifactNumToKeepStr: ('master' == env.BRANCH_NAME) ? '5' : '1' ))

+1 for that.

[HannesWell]

That would be nice. i think the R*maintenance branches don't need a long history, master and current(?) BETA* should be an exception.

Sounds good. The following line leads to the desired behavior:

buildDiscarder(logRotator(numToKeepStr: (env.BRANCH_NAME == 'master' || env.BRANCH_NAME ==~ 'BETA_JAVA\\d+') ? '99' : '5' , artifactNumToKeepStr: '2'))

For master and all BETA_JAVA\d branches 99 builds are kept, for all others only five. For all branches only the artifacts of the last two (successful IIRC) builds is kept. Of course you can tweak the exact numbers as you like. And if necessary the distinction between master/beta and other branches can also be applied to artifactNumToKeepStr.

Probably mainly for the record: I verified that for PR's the branch name is PR-<number>.