Releases · jenkins-infra/helpdesk

03 Nov 11:34

infra-team-sync-2022-09-13_20

83d450e

infra-team-sync-2022-09-13

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Attendees 👥

@dduportal (Damien Duportal)
@lemeurherve (Hervé Le Meur)
@MarkEWaite (Mark Waite)
@smerle33 (Stéphane Merle)
@gounthar (Bruno Verachten)

Announcements 📢

Weekly build
- 2.368 released, changelog visible, container ready

Upcoming Calendar 📆

Next Weekly: Next week, as usual
Next LTS: ~ October 5th (Kris Stern is release lead)
Next Security Release: Wednesday 21st of September
Next major event:
- DevOps World in 2 weeks= jenkins infra meeting cancelled (27th of September) - @damien to remove the invitation from calendar

Notes 📖

Done:
Work In Progress:
- "Latest" links for some plugins on archives.jenkins.io are outdated
- Publish acceptance-test-harness docker image on release
- Hosting Plugin Health Scoring application on infra
  - CI part for building Docker workflow (stash/unstash) ✅ Need tests and good to merge
  - Adrien working on the helm-chart
  - DNS plugins.health.jenkins.io (currently redirecting to their gh page for now, until service is running in Kubernetes)
  - Upcoming need for a database (see Stephane for Azure managed Postgresql)
- (Re) Introduce an artifact caching proxy for ci.jenkins.io
  - Azure proxy instance ok for health check and storage
  - Digital Ocean incoming
  - Next steps: test it (jenkins-infra-test-plugin + Herve to search a "big plugin" in term of deps.)
- Damien is admin of JFrog now, KB with Daniel
  - Todo @dduportal runbook with the "rules"
  - Todo @dduportal open an issue about a virtual repo in JFrog which has no upstream (e.g. only cached file, that can be lost)
- Missing datadog metrics for AKS clusters
- https://twitter.com/jenkins_release is many weeks behind
- [ci.jenkins.io] collect datadog metrics for ephemeral VMs
  - Azure is ok
  - EC2 working since yesterday
  - Last testing phase and done
- Want containerized Java 17 Windows agents
  - All-in-one container (linux): almost there. Currently battle testing on jenkins-infra-test plugin.
  - Windows Container: provisioning failing, but almost there
- Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
  - GH org to be implemented on helm chart
  - PoC on kubernetes-management
- Publish pipeline-steps-doc-generator and backend-extension-indexer artifacts to some kind of storage
- [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
- Incoming work by @lemeurherve to have an IRC bot "archive" command
  - jenkins-infra/ircbot#111

Contributors

damien, gounthar, and 4 other contributors

Assets 2

03 Nov 11:31

github-actions

infra-team-sync-2022-09-06_19

83d450e

infra-team-sync-2022-09-06

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Attendees 👥

@lemeurherve (Hervé Le Meur)
@MarkEWaite (Mark Waite)
@smerle33 (Stéphane Merle)
@gounthar (Bruno Verachten)

Announcements 📢

Weekly
Announcement

Upcoming Calendar 📆

Next Weekly: 2.367 in progress, GitHub reliability may be issue
- Release checklist not yet complete
Next LTS: 2.261.1 tomorrow
Next Security Release: TBD
Next major event:
Jenkins board and officer elections (usually infra officer runs)
- Sep announce and accept nominations
- Oct register voters
- Nov cast votes
- Dec announce results

Notes 📖

Help Desk Changelog

fix(infra-meeting-release): exclude 'not planned' closed issues from the 'Done' ones by @lemeurherve in #3113

Contributors

gounthar, MarkEWaite, and 2 other contributors

Assets 2

03 Nov 11:26

github-actions

infra-team-sync-2022-08-30_13

83d450e

infra-team-sync-2022-08-30

📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Attendees 👥

@lemeurherve (Hervé Le Meur)
@smerle33 (Stéphane Merle)
@gounthar (Bruno Verachten)

Announcements 📢

Weekly
Announcement
- DigitalOcean sponsorship renewed 🎉 ($18000)
  - Tweets and blog posts to be prepared
- VPN unaccessible this morning: a reboot of the VM fixed it
  - This was due to a failing systemd upgrade by Microsoft on Azure Ubuntu 18.06 image
  - more details: https://azure.status.microsoft/en-us/status

Upcoming Calendar 📆

Next Weekly: 30th August 2022
Next LTS: 7th September 2022

Notes 📖

Contributors

gounthar, lemeurherve, and smerle33

Assets 2

03 Nov 11:23

github-actions

infra-team-sync-2022-08-23_12

83d450e

infra-team-sync-2022-08-23

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Attendees 👥

Bob Miles from salad.com
@dduportal (Damien Duportal)
@lemeurherve (Hervé Le Meur)
@smerle33 (Stéphane Merle)
@gounthar (Bruno Verachten)
@MarkEWaite (Mark Waite)

Salad.com (guest: Bob Miles)

https://salad.com/. Ref. https://community.jenkins.io/t/seeking-advice-we-have-too-much-compute-not-enough-jobs/3292
Proposal: Jenkins project to use the project to get feedbacks for the workload
Network of (personnal) computers running an agent process. Compute is distributed across this network of computers for different workloads: mining since 3 years, ability to run containerized workloads
Average time of a task (given what JenkinsCI build are doing)? Incentive about "finishing" tasks but spectrum is broad
Electric consumption: choice and measure delegated to the compute provider users, so no metrics
Encryption: None, based on trust model (multi layers of trust indicators). Do not put sensitive dataset there.
Workload isolation: Windows Host machines. Relies on WSL2 to tap in the GPU and Linux workload. Containers are run in the WSL isolation engine.

Announcements 📢

Damien in PTO from 27th until the 6th of september (included)
Mark in PTO next Tuesday (25th)
Security Advisory in progress, everyhting looks good
Weekly Core release 2.365, checklist in progress, everything looks good

Upcoming Calendar 📆

Next Weekly: Next Tuesday
Next LTS: 7th of September, dropping JDK8 for this LTS
Next Security Release: today
Next major event: N.A.

Notes 📖

Done:
- UC doesn't offer release of new plugin
- JDK8 images only published for first patch release 2.346.1 of final release cycle
- Migration of jenkins(se) from on prem to google cloud vmware engine
- [accounts.jenkins.io] Fix instructions for spam unblock
- ec2-plugin patch by @lemeurherve to specify custom java bin path
  - jenkinsci/ec2-plugin#766
  - tested with an incremental build, deployed on infra.ci.jenkins.io
  - allow us to specify the agent JDK to use, for every OS (wasn't possible for Windows before. Note: we weren't concerned by this issue as we're using SSH for our Windows agents, using the unix template)
Work In Progress:
- incorrect/missing maven settings file (for plugin builds?)
- Enter too many times CAPCHA code
  - How does the blocking works?
  - Wait for security feedback
- Publish acceptance-test-harness docker image on release
  - Almost there: we are almost there. Trying deploy with tag to be done before closing.
- [ci.jenkins.io] collect datadog metrics for ephemeral VMs
  - Datadog agent enabling validated manually in Azure
  - Currently adding the setup on JCasc for controllers
  - Next step: EC2 VMS
- Want containerized Java 17 Windows agents
  - Merged VM and Container templates into the same one. Linux to be tested (and deployed) if working. @lemeurherve
  - Windows container work in progress. Nitpicking on powershell.
- (Re) Introduce an artifact caching proxy for ci.jenkins.io
  - Goal: limit bandwidth consumed from repo.jenkins (JFrog)
  - Re-introducing the same as years before, but 1 proxy per cloud region (to avoid cross-region bandwitdh)
  - 1st instance installed in Azure, tests in progress to get the correct Maven settings to use it (locate the correct proxy depending of the agent type. Using env. var from agent template)
  - Tested on the jenkins-infra-test-plugin (on real life ci.jenkins.io)
  - Next steps: Add user/password protection (in addition to IP whitelisting) and deploy the 2 other proxies and their configuration
- https://twitter.com/jenkins_release is many weeks behind
  - Account recovered.
  - Next step: understand dlvr and check the actual automation to fix it
  - Note from Hervé: could be a good help to automate tweets for advocacy
  - Damien asks for help. @lemeurherve volunteers
- Access to npm namespace
  - Waiting for the NPM support team. Removing from milestone until they answer.
  - There is a jenkins user with no plugins and no activity: trying to reclaim this account to conert it to an organization (and fix the request from Gavin)
- Weekly release build does not resume
  - Fundamental question: do we want automatic retry? Is it worth it given the frequency.
  - Agent failure? Even this is questionnable for release.
- [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
  - Might be slowed down with Damien PTO
  - Required puppet factorization

Contributors

gounthar, MarkEWaite, and 3 other contributors

Assets 2

03 Nov 11:15

github-actions

infra-team-sync-2022-08-16_11

83d450e

infra-team-sync-2022-08-16

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Attendees 👥

@dduportal (Damien Duportal)
@lemeurherve (Hervé Le Meur)
@gounthar (Bruno Verachten)

Announcements 📢

Weekly: 2.634 WAR published, release checklist to be finished

Upcoming Calendar 📆

Next Weekly: Next Tuesday
Next LTS: N.A.
Next Security Release: N.A.
Next major event: N.A.

Notes 📖

Done:
- Latest Trilead API plugin version 1.71.v9e7860a_67a_df fails when using JDK8 for agent process
  - EC2 plugin contribution to be done for EC2 Windows templates (not blocking the issue)
  - Kubernetes plugin contribution
  - To be production-proffed when we'll changed to JDK17
- Puppet Upgrade Campaign to latest 6.x
  - All agents are using open source Puppet
  - But the puppetmaster uses Puppet Enteprise (<10 nodes)
- All maven-11 agents are unavailable
  - Caused by Trilead. Fixed.
- Pipeline failing from 'maven-windows' nodes offline
  - Caused by Trilead. Fixed.
- checks-api 1.7.5: download plugin checks-api to file: status code: 403, reason phrase: Forbidden
  - Fixed by @daniel-beck , many thanks!
- Granting "Triage" permission to jenkins-cert-app on RPU
- Update Jenkins to 2.346.3
Work In Progress:

Contributors

gounthar, dduportal, and 2 other contributors

Assets 2

16 Aug 12:22

github-actions

infra-team-sync-2022-08-09_10

f5c000c

infra-team-sync-2022-08-09

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Announcements 📢

Weekly 2.363

Complete, checklist complete, Docker images confirmed built
Revision to the changelog is submitted
Git client plugin security fix for host key verification broke the packaging step

Costs

July expenses high on all platforms in July
- May have missed autoscaling on kubernetes, manually scaled to complete a release, forgot to autoscale
- Had machines running but idle
- Fixed last Friday, reduced Azure cost estimate
AWS expenses at $12k per month
- Needs more research to understand costs
DigitalOcean credits are low
- Email notice to the DigitalOcean sponsors to increase
Salad Cloud
- Seeking Advice: We have too much compute, not enough jobs!
Launchable
- Kohsuke Kawaguchi would like to allow the Jenkins project to use Launchable
- Would likely extend buildPlugin() to send test duration and result info to Launchable service

Notes 📖

Assets 2

16 Aug 12:16

github-actions

infra-team-sync-2022-08-02_9

f5c000c

infra-team-sync-2022-08-02

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Generated from the "current" and the "next" milestones.

What's Changed

Enable all comment-ops commands by @timja in #3081

New Contributors

@timja made their first contribution in #3081

Contributors

timja

Assets 2

16 Aug 12:13

github-actions

infra-team-sync-2022-07-26_8

f5c000c

infra-team-sync-2022-07-26

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Generated from the "current" and the "next" milestones.

What's Changed

feat: add a button in the README to prepare infra meeting notes by @lemeurherve in #3063
ci: ignore issue mentioned by @xrkffgg in #3064

New Contributors

@xrkffgg made their first contribution in #3064

Full Changelog: infra-team-sync-2022-07-12_2...infra-team-sync-2022-07-26_8

Contributors

xrkffgg and lemeurherve

Assets 2

16 Aug 12:11

github-actions

infra-team-sync-2022-07-19_4

f5c000c

infra-team-sync-2022-07-19

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Full Changelog: infra-team-sync-2022-07-12_2...infra-team-sync-2022-07-19_4

Assets 2

13 Jul 14:27

github-actions

infra-team-sync-2022-07-12_2

072825c

infra-team-sync-2022-07-12

🎥 Meeting Recording
📆 Shared Calendar
💬 community.jenkins.io & IRC Chat Room #jenkins-infra
📧 Google Group (mailing list) jenkins-infra

Attendees

@dduportal (Damien Duportal)
@hlemeur (Hervé Le Meur)
@MarkEWaite (Mark Waite)
@smerle33 (Stéphane Merle)
@Poddingue (Bruno Verachten)

Announcements 📢

Weekly: 2.359 is published in artifactory + artifactory. Checklist not (yet) complete but no issues detected

Further adoption on the JDK/JS/dependencies that might introduces instabilities. Be warned for infra.ci.jenkins.io
Next weekly will have a Jetty 9 -> 10 upgrade. Major update on a key component.

Start using JDK17 on our weekly instances (infra.ci and weekly.ci)

No known issues
Challenge for agents: we need to update Docker images to have both JDK11 and JDK17 (VM already have)

Notes 📖

Done:
- Update analysis-model and warnings-ng on ci.j
  - ci.jenkins.io is not a "infra as code with plugins.txt" so we want such requests in the helpdesk while it's done manually
- Remove my account
- Remote access API on every non-ci.j.io instance
  - Discovered while working on removal of the embeddable build status plugin
  - No need to block API on privtae instances: request from security team to allow them doing more automation
- Infra meeting notes as helpdesk releases
  - Thanks Herve! We now have a GitHub action to generate the meeting notes
- Docker-compose in JenkinsCI.io
  - A plugin maintainer needed to use testcontainer which requires Docker. It was using VM agents (label linux by default) but it was choosing randomly ARM or Intel agent which caused obvious issues (Intel binary on ARM64...)
  - Hotfix: linux label means "Linux Ubuntu 20.04 AMD64" for now. Labels should be rationalized: but we need to update both pipeline-library, ci.j config AND ci.j documentation.
- Add kmartens27 to jenkins.io triage team
- Weekly release: 2.358 packaging step fails due to AKS CSI Persistent Volume issue (after Kubernetes 1.22 upgrade)
  - Fixed with the help of Herve and Stephane. Same issue as for LDAP/get.jenkins.io during the kubernetes 1.22 upgrade.
  - We check ALL the other azurefile persistent volumes: no more left to fix
- Remove groovy tool configuration from cert.ci JCasC
  - Remove tool config, then the plugin.
- Grant permission to update-center
  - Not done: Tim guided the requesterr to the corect documentation
- 502 proxy error when accessing PR view for jenkinsci/jenkins
  - Tricky issue. Thanks Daniel and Alex for diagnosing
  - Fixed in the culprit plugin by Uli, many thanks!
  - Applied to ci.jenkins once available: immediate fix!
- [ci.jenkins.io] Provides both powershell and pwsh on all agent templates
  - VM agent templates: check
  - Windows Docker image: won't do (no blocker)
  - Real value of this issue: we were able to update the public doc. for Jenkins pipeline
- DockerHub rate limiting
  - We've been granted a team plan on jenkins4eval and jenkinsciinfra: no more rate limit seen on ci.j after enabling authentication again
- [INFRA-1633] Stop building PR merges
  - Old issue. Some "heavy" jobs should not rebuilds PRs when the destination branch is updated, to avoid wasting precious time.
  - Tim set up the aformentioned job. Thanks!
  - We have to write down an issue to manage ci.jenkins.io jobs as code.
- ci.jenkins.io agents are very flaky
  - Initial problem: Issues on container agents for BOM builds (a lot of executors requested but unable to scale up efficiently leading to a build queue slowly emptying)
  - Correlated to a lot of ATH builds at the same time, spawning a lot of EC2 AWS instances of type "highmem" which are the same size as EKS worker nodes
  - Root cause: We hit our limit for EC2 spot instances of this size in this region (us-east-2) leading to a lot of spot reclaims, shutting down agents abruptly during builds
  - Solved by switching from spot to on-demand instances (both AWS and Azure to be safe for ATH). The overcost is low and will be compensated by less builds
Work In Progress:
- Downloads /latest directory out of date
  - Issue on the azurefile bucket storing the "reference" files for the mirror system (get.jenkins.io): it does not support symlink in the current way that we use it (blobxfer with an old version)
  - fixed manually for LTS (initial request of this issue)
  - Need more work with feedbacks from Olivier V., Tim and Daniel if they recall how/why
- Consider removing embeddable-build-status plugin
  - Still to be removed from ci.jenkins.io
  - Batch PR ready to go (prepared and tested by Herve, after a lot of nice tips from Tim, Joseph and Alex)
- enable Development integration in Jira
  - Nothing done, still to do
- [JENKINS-49707] Evaluate retry conditions to improve the stability of the builds
  - Almost closable: kubernetes plugin to be released and deployed (Jesse needs 1 or 2 last minors changes in the code)
  - We are happy with the outcome, really valuable
- Replace s390x Ubuntu 18.04 agent with s390x Ubuntu 20.04 agent
  - Mark did not had time yet: nothing done, still to do
  - Request to share SSH access, to test if the machine can run a puppet agent v6 (or v7)
- Upgrade to Kubernetes 1.22
  - Almost closable: Last steps: merge documentation PR + create issue for Kuberneets 1.23
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  - Huge work by Stephane: Oracle infra is set up with Terraform and the machine has a puppet agent connected to our puppetmaster
  - Next steps: create Puppet role for this machine, make it work and start copying JSON files on it (additionnaly to the actual machine)
New issues:
- Broken taglib docs on reports.jenkins.io => Assigned to Mark, no action required from infra team because it is a javadoc (e.g. dev. scope) thing
- [Hosted javadocs for LTS versions of Jenkins] => No action from the infra team expected: it is a dev. question and should be raised to the contributors usual channels
New/Important:

Contributors

MarkEWaite, gmcdonald, and 5 other contributors

Assets 2

Releases: jenkins-infra/helpdesk

infra-team-sync-2022-09-13

Attendees 👥

Announcements 📢

Upcoming Calendar 📆

Notes 📖

Contributors

infra-team-sync-2022-09-06

Attendees 👥

Announcements 📢

Upcoming Calendar 📆

Notes 📖

Help Desk Changelog

Contributors

infra-team-sync-2022-08-30

Attendees 👥

Announcements 📢

Upcoming Calendar 📆

Notes 📖

Contributors

infra-team-sync-2022-08-23

Attendees 👥

Salad.com (guest: Bob Miles)

Announcements 📢

Upcoming Calendar 📆

Notes 📖

Contributors

infra-team-sync-2022-08-16

Attendees 👥

Announcements 📢

Upcoming Calendar 📆

Notes 📖

Contributors

infra-team-sync-2022-08-09

Announcements 📢

Notes 📖

infra-team-sync-2022-08-02

What's Changed

New Contributors

Contributors

infra-team-sync-2022-07-26

What's Changed

New Contributors

Contributors

infra-team-sync-2022-07-19

infra-team-sync-2022-07-12

Attendees

Announcements 📢

Notes 📖

Contributors