Puppet Upgrade Campaign to latest 6.x #3056

dduportal · 2022-07-14T17:10:08Z

Service(s)

Other

Summary

Today, we are using 2 Puppet versions:

Puppet master and some of the puppet agents are pinned to the 6.32.0 version
All the other agents agents are pinned to the 6.0.4 version
The gem puppet on the jenkins-infra/jenkins-infra repository is using version 6.23.0 but we have a dependanbot update chore(deps): bump puppet from 6.23.0 to 6.25.1 jenkins-infra#2283 pushing to us to "at least 6.25.1"

Let's upgrade to 6.27.y, the latest 6.x known release (for puppet agent) and 6.19.x for Puppetserver

Plan

Step 0: Check changelog
Step 1: Bump the puppetmaster version to 6.27.1
- Stop puppet agents everywhere
- Follow upgrade guide
- Start agents one by one
Step 2: Bump the jenkins-infra/jenkins-infra version
- Gemfile(*)
- spec/spec_helper.rb
- Vagrantfile
Step 3: For each VM, install the Puppet6 repo, remove current puppet-agent (without --purge) and install the pinned 6.27.1 agent

The text was updated successfully, but these errors were encountered:

dduportal · 2022-07-15T08:44:32Z

Changelogs:

A few notes:

Seems like that the latest available Puppet agent is 6.27.0 (ref. tags at https://github.com/puppetlabs/puppet/tags) despite the changelog mentioning a 6.27.1 (ref. https://puppet.com/docs/puppet/6/release_notes_puppet.html)
- Same for the rubygem puppet: https://rubygems.org/gems/puppet/versions
Latest Puppet server is 6.19.0 (associated with 6.27.x puppet line): https://puppet.com/docs/puppet/6/server/release_notes.html

Current versions:

  $ for machine in trusted-ci-bastion vpn.jenkins.io puppet.jenkins.io pkg.origin.jenkins.io ci.jenkins.io cert.ci.jenkins.io archives.jenkins.io edamame.jenkins.io lettuce.jenkins.io census.jenkins.io usage.jenkins.io trusted.ci.jenkins.io trusted-agent-1; do echo "=== Machine: ${machine}"; ssh "${machine}" "puppet --version && facter --version";done
=== Machine: trusted-ci-bastion
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: vpn.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: puppet.jenkins.io
6.23.0
3.14.18 (commit 92e6f64d36589c8a537d4a4d5c574c872c3aa329)
=== Machine: pkg.origin.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: ci.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: cert.ci.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: archives.jenkins.io
6.23.0
3.14.18 (commit 92e6f64d36589c8a537d4a4d5c574c872c3aa329)
=== Machine: edamame.jenkins.io
6.23.0
3.14.18 (commit 92e6f64d36589c8a537d4a4d5c574c872c3aa329)
=== Machine: lettuce.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: census.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: usage.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: trusted.ci.jenkins.io
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)
=== Machine: trusted-agent-1
6.0.4
3.12.1 (commit 5d393c68659509635af18ebe47e551506b20ea80)

Latest Ubuntu package for puppet-agent for the 6.x line (18.04 and 20.04) is 6.27.1-1bionic if you install the "latest" version from their APT repo (and NO 6.27.0 version):

$ curl -sSL -o puppet-repo.deb "http://apt.puppetlabs.com/puppet6-release-bionic.deb" && dpkg -i puppet-repo.deb && apt-get update -q && apt-cache madison puppet-agent
# ...
puppet-agent | 6.27.1-1bionic | http://apt.puppetlabs.com bionic/puppet6 amd64 Packages
puppet-agent | 6.26.0-1bionic | http://apt.puppetlabs.com bionic/puppet6 amd64 Packages
# ...

Installing the "latest" 6.27.1-1 package for puppet-agent DOES:

Includes puppet version 6.27.0
Includes facter version 3.14/23
Install everyting in /opt/puppetlabs:

  root@95d7afa5a587:/# /opt/puppetlabs/puppet/bin/puppet --version
  6.27.0
  root@95d7afa5a587:/# ^C-version
  root@95d7afa5a587:/# /opt/puppetlabs/puppet/bin/facter --version
  3.14.23 (commit 46e6e6ffdecff72b75853af43a11acdf2d36752c)

More informations about the versions hell in Puppet: https://puppet.com/docs/puppet/7/platform_lifecycle.html#about_agent-platform-releases
Side note: ALL of our machine has the puppet-agent content installed in the default path /opt/puppetlabs, BUT we have symlinks created for the puppet command (at least!) /usr/local/bin/puppet -> /opt/puppetlabs/bin/puppet

dduportal · 2022-08-08T16:03:05Z

Ping @olblak it looks like that the radish VM is running an enterprise edition of Puppet (version 2019.8.7):

dpkg -l | grep puppet
ii  pe-puppet-enterprise-release           2019.8.7.0-2bionic                              all          A simple package that lays out the file structure for Puppet Enterprise systems, and provides some basic files necessary for PE.
ii  pe-puppetdb                            6.17.0-1bionic                                  all          Puppet Labs pe-puppetdb
ii  pe-puppetdb-termini                    6.17.0-1bionic                                  all          Termini for pe-puppetdb
ii  pe-puppetserver                        2019.8.3.25-1bionic                             all          Puppet Labs pe-puppetserver
ii  puppet-agent                           6.23.0-1bionic                                  amd64        The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
ii  puppetlabs-release                     1.0-11                                          all          "Package to install Puppet Labs gpg key and apt repo"

=> Is it expected? do we need a licence?

dduportal · 2022-08-08T16:05:13Z

With Enterprise:

https://puppet.com/docs/pe/2019.8/upgrading_pe.html#upgrading_pe
Upgrade path is (for this issue) 2019.8.7 -> 2019.8.11
Backup first: https://puppet.com/docs/pe/2019.8/backing_up_and_restoring_pe.html#back_up_pe_infrastructure
No need for DB migration
Installed should be used to migrate packages

dduportal · 2022-08-09T09:37:27Z

Ping @olblak it looks like that the radish VM is running an enterprise edition of Puppet (version 2019.8.7):

dpkg -l | grep puppet
ii  pe-puppet-enterprise-release           2019.8.7.0-2bionic                              all          A simple package that lays out the file structure for Puppet Enterprise systems, and provides some basic files necessary for PE.
ii  pe-puppetdb                            6.17.0-1bionic                                  all          Puppet Labs pe-puppetdb
ii  pe-puppetdb-termini                    6.17.0-1bionic                                  all          Termini for pe-puppetdb
ii  pe-puppetserver                        2019.8.3.25-1bionic                             all          Puppet Labs pe-puppetserver
ii  puppet-agent                           6.23.0-1bionic                                  amd64        The Puppet Agent package contains all of the elements needed to run puppet, including ruby, facter, and hiera.
ii  puppetlabs-release                     1.0-11                                          all          "Package to install Puppet Labs gpg key and apt repo"

=> Is it expected? do we need a licence?

Confirmed with @olblak : we are using the Puppet Enterprise version with the free licence (less than 10 nodes). Gotta check the details, but it means that the upgrade should be with PE.

dduportal · 2022-08-12T06:17:31Z

dduportal · 2022-08-12T07:54:20Z

Next steps: agents

Update and merge the latest puppet gem support chore: bump Puppet version to latest 6.x jenkins-infra#2288
Upgrade puppet-agent package on the puppetmaster (~~6.28.0~~ 6.27.1)
- Note: the puppet-agent on the puppetmaster should always be managed by the installer itself as per https://support.puppet.com/hc/en-us/articles/360034356933.
- Tried to use the puppet6 apt repo and installed latest puppet-agent packages: it failed the runs with the error message Aug 12 07:52:10 jenkins-radish puppet-agent[12824]: 'curl -f -L -o puppet-agent-ubuntu-16.04-amd64.tar.gz https://pm.puppetlabs.com/puppet-agent/2019.8.11/6.28.0/repos/puppet-agent-ubuntu-16.04-amd64.tar.gz' returned 22 instead of one of [0]
- Note: puppet-enterprise versions: https://puppet.com/docs/pe/2019.8/component_versions_in_recent_pe_releases.html#component_versions_in_recent_pe_releases-pe-agent-server-components
- Fixed by removing the puppet6-release and puppetlabs-release packages + downgrading the puppet-agent package to the version associated to the distribution 2019.8.11 of PE:
```
$ apt-cache policy puppet-agent
puppet-agent:
  Installed: 6.28.0-1bionic
  Candidate: 6.28.0-1bionic
  Version table:
 *** 6.28.0-1bionic 100
        100 /var/lib/dpkg/status
     6.27.1-1bionic 500
        500 file:/opt/puppetlabs/server/data/packages/public/2019.8.11/puppet_enterprise ./ Packages
$ apt-get install puppet-agent=6.27.1-1bionic
# Success downgrade
$ systemctl restart puppet 
# Success run
```
- Holding puppet-agent version on the puppetmaster: apt-mark hold puppet-agent

Update puppet-agent to latest (6.28.0)

Testing on census.jenkins.io as initial target

puppet agent --enable && puppet agent --test --noop && puppet agent --disable "$USER is doing some manual maintenance"
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Applying configuration version '1660292292'
Notice: Applied catalog in 3.60 seconds

Testing on archives.jenkins.io as "not ubuntu-20.04" target

puppet agent --enable && puppet agent --test --noop && puppet agent --disable "$USER is doing some manual maintenance"
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Applying configuration version '1660292273'
Notice: /Stage[main]/Datadog_agent/File[/etc/datadog-agent/install_info]/content: 
--- /etc/datadog-agent/install_info     2022-07-18 08:32:13.373268457 +0000
+++ /tmp/puppet-file20220812-509030-1xel5vw     2022-08-12 08:18:08.393578228 +0000
@@ -1,5 +1,5 @@
 ---
 install_method:
   tool: puppet
-  tool_version: puppet-6.23.0
+  tool_version: puppet-6.28.0
   installer_version: datadog_module-3.17.0

Notice: /Stage[main]/Datadog_agent/File[/etc/datadog-agent/install_info]/content: current_value '{md5}e2aaa62c46631f4496c8b8ff86ef35bd', should be '{md5}a749825fa4afa92bd3bd51467dc8c836' (noop)
Notice: Class[Datadog_agent]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 4.08 seconds

Testing on ci.jenkins.io as "critical" target

# Install latest puppet-agent package with a one liner. No version hold: starting today we want the latest puppet-agent version.
curl -sSLO http://apt.puppet.com/puppet6-release-"$(lsb_release --codename --short)".deb && dpkg -i puppet6-release-"$(lsb_release --codename --short)".deb && apt-get update && apt-get upgrade -y && rm -f puppet*.deb

Generalizing on all other machines:

# Install latest puppet-agent package using the puppet-6 release distribution
for machine in trusted-ci-bastion vpn.jenkins.io pkg.origin.jenkins.io ci.jenkins.io cert.ci.jenkins.io archives.jenkins.io edamame.jenkins.io lettuce.jenkins.io census.jenkins.io usage.jenkins.io trusted.ci.jenkins.io trusted-agent-1; do echo "=== Machine: ${machine}";ssh "${machine}" sudo "bash -c 'apt-mark unhold puppet-agent && curl -sSLO http://apt.puppet.com/puppet6-release-\$(lsb_release --codename --short).deb && dpkg -i puppet6-release-\$(lsb_release --codename --short).deb && apt-get update && apt-get upgrade -y && rm -f puppet*.deb'";done

# Checking version (all 6.28.0 except the puppetmaster in 6.27.1)
for machine in trusted-ci-bastion vpn.jenkins.io puppet.jenkins.io pkg.origin.jenkins.io ci.jenkins.io cert.ci.jenkins.io archives.jenkins.io edamame.jenkins.io lettuce.jenkins.io census.jenkins.io usage.jenkins.io trusted.ci.jenkins.io trusted-agent-1; do echo "=== Machine: ${machine}"; ssh "${machine}" "puppet --version && facter --version";done
=== Machine: trusted-ci-bastion
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: vpn.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: puppet.jenkins.io
6.27.0
3.14.23 (commit 46e6e6ffdecff72b75853af43a11acdf2d36752c)
Connection to puppet.jenkins.io closed.
=== Machine: pkg.origin.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: ci.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: cert.ci.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: archives.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: edamame.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: lettuce.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: census.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: usage.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
=== Machine: trusted.ci.jenkins.io
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)
Connection to 172.31.50.95 closed.
=== Machine: trusted-agent-1
6.28.0
3.14.24 (commit 91ed8a2de5c9d686345859fe12ea2914415758f0)

# Enable and restart agents
for machine in trusted-ci-bastion vpn.jenkins.io pkg.origin.jenkins.io ci.jenkins.io cert.ci.jenkins.io archives.jenkins.io edamame.jenkins.io lettuce.jenkins.io census.jenkins.io usage.jenkins.io trusted.ci.jenkins.io trusted-agent-1; do echo "=== Machine: ${machine}";ssh "${machine}" sudo "sh -c 'puppet agent --enable && systemctl restart puppet'";done

dduportal · 2022-08-12T09:02:28Z

So far so good. Closing issue

dduportal added the triage Incoming issues that need review label Jul 14, 2022

dduportal self-assigned this Jul 14, 2022

dduportal added this to the infra-team-sync-next milestone Jul 14, 2022

dduportal changed the title ~~Puppet Upgrade Campaign to 6.27.1~~ Puppet Upgrade Campaign to latest 6.x Jul 15, 2022

This was referenced Jul 15, 2022

chore: bump Puppet version to latest 6.x jenkins-infra/jenkins-infra#2288

Merged

Puppet Upgrade Campaign to Community Edition (7.x or 8.x) #3058

Open

dduportal added puppet and removed triage Incoming issues that need review labels Jul 15, 2022

dduportal modified the milestones: infra-team-sync-next, infra-team-sync-2022-07-19 Jul 15, 2022

smerle33 modified the milestones: infra-team-sync-2022-07-19, infra-team-sync-2022-07-26 Jul 19, 2022

smerle33 modified the milestones: infra-team-sync-2022-07-26, infra-team-sync-2022-08-02, infra-team-sync-next Jul 27, 2022

dduportal modified the milestones: infra-team-sync-next, infra-team-sync-2022-08-09 Aug 8, 2022

dduportal modified the milestones: infra-team-sync-2022-08-09, infra-team-sync-2022-08-16 Aug 9, 2022

dduportal closed this as completed Aug 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Puppet Upgrade Campaign to latest 6.x #3056

Puppet Upgrade Campaign to latest 6.x #3056

dduportal commented Jul 14, 2022 •

edited

Loading

dduportal commented Jul 15, 2022 •

edited

Loading

dduportal commented Aug 8, 2022

dduportal commented Aug 8, 2022

dduportal commented Aug 9, 2022

dduportal commented Aug 12, 2022 •

edited

Loading

dduportal commented Aug 12, 2022 •

edited

Loading

dduportal commented Aug 12, 2022

Puppet Upgrade Campaign to latest 6.x #3056

Puppet Upgrade Campaign to latest 6.x #3056

Comments

dduportal commented Jul 14, 2022 • edited Loading

Service(s)

Summary

Plan

dduportal commented Jul 15, 2022 • edited Loading

dduportal commented Aug 8, 2022

dduportal commented Aug 8, 2022

dduportal commented Aug 9, 2022

dduportal commented Aug 12, 2022 • edited Loading

dduportal commented Aug 12, 2022 • edited Loading

dduportal commented Aug 12, 2022

dduportal commented Jul 14, 2022 •

edited

Loading

dduportal commented Jul 15, 2022 •

edited

Loading

dduportal commented Aug 12, 2022 •

edited

Loading

dduportal commented Aug 12, 2022 •

edited

Loading