Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to recovery.md to address PE-39730 #526

Merged
merged 2 commits into from
Nov 14, 2024
Merged

Updates to recovery.md to address PE-39730 #526

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef85494
Updates to recovery.md to address PE-39730
mtaggart13 Nov 13, 2024
9ed573e
Post-review updates to recovery.md
mtaggart13 Nov 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 40 additions & 14 deletions documentation/recovery.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,36 +14,62 @@ The new system needs to be provisioned with the same certificate name as the sys
This procedure uses the following placeholder references.

* _\<primary-server-fqdn\>_ - The FQDN and certname of the primary Puppet server
* _\<replacement-replica-fqdn\>_ - The FQDN and certname of the replacement replica Puppet server
* _\<replacement-avail-group-letter\>_ - Either A or B; whichever of the two letter designations is appropriate for the server being replaced. It will be the opposite of the primary server.
* _\<old-replica-fqdn\>_ - The FQDN and certname of the old replica Puppet server that has failed or is missing
* _\<replacement-replica-fqdn\>_ - The FQDN and certname of the new replica Puppet server
* _\<failed-primary-server-fqdn\>_ - The FQDN and certname of the original primary server that the old replica had replaced
Copy link

@taikaa taikaa Nov 13, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest removing failed-primary-server-fqdn from this section altogether because steps about replacing a replica should just be in this section

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Promote the replica (official docs)
  2. purge the failed primary server puppet node purge
  3. Replace missing replica server (same as Replace missing or failed replica Puppet server below)

* _\<replacement-avail-group-letter\>_ - Either A or B; whichever of the two letter designations is appropriate for the replacement server. It will be the opposite of the server that it is replacing.

1. Ensure the old replica server is forgotten.
1. If applicable, purge the failed primary server. (You may need to do this, for example, if the original primary failed and the promoted replica that replaced it has also failed.)

puppet infrastructure forget <replacement-replica-fqdn>
puppet node purge <failed-primary-server-fqdn>

2. Install the Puppet agent on the replacement replica
2. Ensure the old replica server is forgotten.

puppet infrastructure forget <old-replica-fqdn>

3. Install the Puppet agent on the replacement replica.

**Note**: When designating the availability group of the replacement, use the opposite group (A or B) of the server being replaced. This means that, if the old replica server replaced the original primary server, the new replica is assigned the same availability group as the original primary.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest deleting this line. It won't always be the opposite of the server being replaced but it should be opposite of whatever is the primary which is already stated in the former line 18. I suggest leaving that line there <replacement-avail-group-letter>_ - Either A or B; whichever of the two letter designations is appropriate for the server being replaced. It will be the opposite of the primary server.


curl -k https://<primary-server-fqdn>:8140/packages/current/install.bash \
| bash -s -- \
main:certname=<replacement-replica-fqdn> \
extension_requests:1.3.6.1.4.1.34380.1.1.9812=puppet/server \
extension_requests:1.3.6.1.4.1.34380.1.1.9813=<replacement-avail-group-letter>

source /ect/profile.d/puppet-agent.sh

puppet agent -t

3. On the PE-PostgreSQL server in the _\<replacement-avail-group-letter\>_ group
4. Sign the certificate on the new primary server.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to 'Sign the certificate on the primary server.
puppetserver ca sign --certname '
(It is not always a new primary server if they did not promote a replica. They can be doing these steps to just replace a failed replica)


5. On the PE-PostgreSQL server in the _\<replacement-avail-group-letter\>_ group
1. Stop puppet.service
2. Add the following two lines to /opt/puppetlabs/server/data/postgresql/11/data/pg\_ident.conf

puppet resource service puppet ensure=stopped

3. Add the following two lines to /opt/puppetlabs/server/data/postgresql/14/data/pg\_ident.conf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the following two lines to /opt/puppetlabs/server/data/postgresql/<postgres_version>/data/pg_ident.conf
In the latest PE release 2023.8.0 the PostgreSQL version is 14 /opt/puppetlabs/server/data/postgresql/14/data/pg_ident.confhttps://www.puppet.com/docs/pe/latest/component_versions_in_recent_pe_releases.html


pe-puppetdb-pe-puppetdb-map <replacement-replica-fqdn> pe-puppetdb
pe-puppetdb-pe-puppetdb-migrator-map <replacement-replica-fqdn> pe-puppetdb-migrator

3. Restart pe-postgresql.service
3. Provision the new system as a replica
5. Restart pe-postgresql.service

puppet resource service pe-postgresql ensure=stopped
puppet resource service pe-postgresql ensure=running

5. Run Puppet

puppet agent -t

6. Provision the new system as a replica

puppet infrastructure provision replica <replacement-replica-fqdn> --topology mono-with-compile --skip-agent-config --enable

4. On the PE-PostgreSQL server in the _\<replacement-avail-group-letter\>_ group, start puppet.service
7. On the PE-PostgreSQL server in the _\<replacement-avail-group-letter\>_ group, start puppet.service

puppet resource service puppet ensure=running


## Replace failed PE-PostgreSQL server (A or B side)

Expand Down Expand Up @@ -102,11 +128,11 @@ On _\<working-postgres-server-fqdn\>_:

systemctl stop puppet

2. Add this line to /opt/puppetlabs/server/data/postgresql/11/data/pg\_ident.conf
2. Add this line to /opt/puppetlabs/server/data/postgresql/14/data/pg\_ident.conf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this line to /opt/puppetlabs/server/data/postgresql/<postgres_version>/data/pg_ident.conf
In the latest PE release 2023.8.0the PostgreSQL version is 14 /opt/puppetlabs/server/data/postgresql/14/data/pg_ident.confhttps://www.puppet.com/docs/pe/latest/component_versions_in_recent_pe_releases.html


replication-pe-ha-replication-map <replacement-postgres-server-fqdn> pe-ha-replication

3. Add these lines to /opt/puppetlabs/server/data/postgresql/11/data/pg\_hba.conf
3. Add these lines to /opt/puppetlabs/server/data/postgresql/14/data/pg\_hba.conf

# REPLICATION RESTORE PERMISSIONS
hostssl replication pe-ha-replication 0.0.0.0/0 cert map=replication-pe-ha-replication-map clientcert=1
Expand All @@ -123,13 +149,13 @@ Run the following commands.
```
systemctl stop puppet.service pe-postgresql.service

mv /opt/puppetlabs/server/data/postgresql/11/data/certs /opt/puppetlabs/server/data/pg_certs
mv /opt/puppetlabs/server/data/postgresql/14/data/certs /opt/puppetlabs/server/data/pg_certs

rm -rf /opt/puppetlabs/server/data/postgresql/*

runuser -u pe-postgres -- \
/opt/puppetlabs/server/bin/pg_basebackup \
-D /opt/puppetlabs/server/data/postgresql/11/data \
-D /opt/puppetlabs/server/data/postgresql/14/data \
-d "host=<working-postgres-server-fqdn>
user=pe-ha-replication
sslmode=verify-full
Expand Down
Loading