doc: add guide for upgrading to v5 (#407)

* doc: add guide for upgrading to v5

* chore: add changelog fragment

* Apply suggestions from code review

Co-authored-by: Douglas J Hunley <doug.hunley@gmail.com>

* update v5 upgrade doc

* docs: update documentation - version info, broken links and file paths

* docs: fix typo

* Apply suggestions from code review

Co-authored-by: Jim Chanco Jr <69092730+jchancojr@users.noreply.github.com>
Co-authored-by: Douglas J Hunley <doug.hunley@gmail.com>

* docs: remove bloat package from upgrade guide doc

* docs: fix spelling

---------

Co-authored-by: Douglas J Hunley <doug.hunley@gmail.com>
Co-authored-by: Jim Chanco Jr <69092730+jchancojr@users.noreply.github.com>

.gitignore

@@ -1,2 +1,3 @@
 *.swp
 exporter/postgres/queries.yml
+hugo/.hugo_build.lock

changelogs/fragments/407.yml

@@ -0,0 +1,2 @@
+trivial:
+  - docs - Add documentation for upgrading to v5

hugo/content/_index.md

@@ -39,34 +39,30 @@ Running pgMonitor will give you confidence in understanding how well your Postgr
 
 pgMonitor is an open-source monitoring solution for PostgreSQL and the systems that it runs on. pgMonitor came from the need to provide a way to easily create a visual environment to monitor all the metrics a database administrator needs to proactively ensure the health of the system.
 
-pgMonitor combines multiple open-source software packages and necessary configuration to create a robust PostgreSQL monitoring environment.  These include:
+pgMonitor combines multiple open-source software services to create a robust PostgreSQL monitoring environment.  These include:
 
 - [Prometheus](https://prometheus.io/) - an open-source metrics collector that is highly customizable.
 - [Grafana](https://grafana.com/) - an open-source data visualizer that allows you to generate many different kinds of charts and graphs.
-- [PostgreSQL Exporter](https://github.com/wrouesnel/postgres_exporter) - an open-source data export to Prometheus that supports collecting metrics from any PostgreSQL server version 9.1 and above.
+- [SQL Exporter](https://github.com/burningalchemist/sql_exporter) - an open-source exporter for Prometheus that supports collecting metrics from multiple database systems including PostgreSQL.
+- [pgMonitor extension](https://github.com/CrunchyData/pgmonitor-extension) - a PostgreSQL extension that provides a means to collect metrics that can be used by an external collection source.
 
 ![pgMonitor](/images/crunchy-monitoring-arch.png)
 
 ## Supported Platforms
 
 ### Operating Systems
 
-- RHEL 7/8/9 (Build/Run Testing, Setup Instructions)
-- CentOS 7 (Build/Run Testing, Setup Instructions)
+- RHEL 8/9 (Build/Run Testing, Setup Instructions)
 - Ubuntu 20/22 (Build/Run Testing)
 
 ### PostgreSQL
 
 - pgMonitor plans to support all PostgreSQL versions that are actively supported by the PostgreSQL community. Once a major version of PostgreSQL reaches its end-of-life (EOL), pgMonitor will cease supporting that major version soon after. Please see the official PostgreSQL website for [community supported releases](https://www.postgresql.org/support/versioning/).
 
-#### Known issues
-
-- PostgreSQL 10+ SCRAM-SHA-256 encrypted passwords are supported on the Linux version of pgMonitor 4.0 or later only.
-
 ### PgBouncer
 
-- PgBouncer 1.22
-- pgbouncer_fdw 1.1.0
+- PgBouncer 1.21+
+- pgbouncer_fdw 1.1.0 (optional with sql_exporter)
 
 ## Installation
 
@@ -78,6 +74,8 @@ Installation instructions for each package are provided in that packages subfold
 
 ### 3. [Grafana](/grafana)
 
+Notes on upgrading can be found in each relevant section.
+
 ## Version History
 
 For the [full history](/changelog) of pgMonitor, please see the [CHANGELOG](/changelog).

hugo/content/changelog/v5_upgrade/_index.md

@@ -0,0 +1,80 @@
+---
+title: "Upgrading to pgMonitor v5.0.0"
+draft: false
+weight: 5
+---
+
+Version 5 of pgMonitor introduces a new exporter that will be used for collecting PostgreSQL metrics: [https://github.com/burningalchemist/sql_exporter](sql_exporter). Converting to this new exporter will involve cleaning up the old postgres_exporter, updating Prometheus targets, and installing new Grafana dashboards.
+
+## Cleanup
+
+### postgres_exporter
+
+This new exporter for PostgreSQL allows for just a single exporter to connect to all databases within a PostgreSQL instance as well as connecting directly to PgBouncer to collect its metrics.
+There is no longer any need for the postgres_exporter to be running, so its services can be shutdown. Some examples of those service names based on the old documentation are as follows:
+
+```
+sudo systemctl stop crunchy-postgres-exporter@postgres_exporter_pg##
+sudo systemctl disable crunchy-postgres-exporter@postgres_exporter_pg##
+
+sudo systemctl stop crunchy-postgres-exporter@postgres_exporter_pg##_per_db
+sudo systemctl disable crunchy-postgres-exporter@postgres_exporter_pg##_per_db
+```
+
+Note the values after the @ symbol may be different depending on the sysconfig files that have been created for your exporters. There may also be exporters running for multiple clusters and we would recommend replacing all of the existing postgres_exporters with the new sql_exporter.
+
+If you've installed pgMonitor with the packages provided by Crunchy Data, those packages can now be uninstalled as well. 
+
+| Package Name                   | Description                                                               |
+|--------------------------------|---------------------------------------------------------------------------|
+| pgbouncer_fdw                  | Package for the pgbouncer_fdw extension                                   |
+| pgmonitor-pg-common            | Package containing postgres_exporter items common for all versions of PostgreSQL |
+| pgmonitor-pg##-extras          | Crunchy-optimized configurations for postgres_exporter. Note that each major version of PostgreSQL has its own extras package (pgmonitor-pg13-extras, pgmonitor-pg14-extras, etc) |
+| postgres_exporter              | Base package for postgres_exporter                                        |
+
+Note that the pgbouncer_fdw is no longer required to monitor PgBouncer but it can still be used with sql_exporter if desired. Per previous instructions, it was usually only installed on the global database. The extension can be removed as follows if it's not needed.
+```
+DROP EXTENSION pgbouncer_fdw;
+```
+
+If postgres_exporter was not set up with packages, you can now manually remove all the related files. Note the ## is replaced with the major version of PG that was being monitored. It is possible that multiple versions of PG had been monitored and copies of these files could exist for all versions. Also, the sysconfig files listed below are the defaults used in examples; there may be additional postgres_exporter sysconfig files on your system(s).
+
+| System Location |
+|-----------------|
+| /etc/postgres_exporter/  |
+| /usr/lib/systemd/system/crunchy-postgres-exporter@.service  |
+| /etc/sysconfig/postgres_exporter_pg##  |
+| /etc/sysconfig/postgres_exporter_pg##_per_db  |
+| /usr/bin/pgbackrest-info.sh |
+| /etc/pgmonitor.conf |
+
+
+### Prometheus
+All postgres_exporter Prometheus targets can now be removed. The default location for Prometheus targets is `/etc/prometheus/auto.d/`, but please check your Prometheus installation for possible additional target locations. In the identified location(s), remove any targets for the postgres_exporter. The default ports for postgres_exporter were 9187 and 9188, so any targets with these ports should be examined for removal. Note that if alerting had previously been enabled, the previous step likely caused multiple alerts to fire; once this step is done, you can simply reload Prometheus to clear these targets and any related alerts should resolve themselves. 
+
+```bash
+sudo systemctl reload prometheus
+```
+Any alerts related to postgres_exporter can also be removed from the files contained in the default alert files location `/etc/prometheus/alert-rules.d/`. Note the default example alert file had been named `crunchy-alert-rules-pg.yml`
+
+### Grafana
+
+Version 5.x of pgMonitor raises the minimum required version of Grafana to 10.4. It also removes dashboards related to postgres_exporter and adds new ones for sql_exporter. If you are simply using the dashboards provided by pgMonitor, the easiest method to update is to simply remove the old ones and install the new ones.
+
+If you are using Crunchy-provided packages, simply uninstall the old packages. It's recommended to follow the non-package removal process below as well to ensure things are cleaned up properly.
+
+| Package Name              | Description                                                       |
+|---------------------------|-------------------------------------------------------------------|
+| pgmonitor-grafana-extras  | Crunchy configurations for datasource & dashboard provisioning    |
+
+If you didn't use the Crunchy-provided packages, ensure the files in the following folder are removed:
+
+```
+| System Location |
+|-----------------|
+| /etc/grafana/crunchy_dashboards |
+```
+
+## Set up sql_exporter
+
+At this point, you should just be able to follow the [https://access.crunchydata.com/documentation/pgmonitor/latest/](standard setup instructions) for the latest version of pgMonitor. This will setup the new exporter, Prometheus targets, and new Grafana dashboards.

hugo/content/exporter/_index.md

@@ -64,7 +64,7 @@ sudo useradd -m -d /var/lib/ccp_monitoring ccp_monitoring
 
 All executables installed via the above releases are expected to be in the {{< shell >}}/usr/bin{{< /shell >}} directory. A base node_exporter systemd file is expected to be in place already. An example one can be found here:
 
-https://github.com/lest/prometheus-rpm/tree/master/node_exporter
+https://github.com/prometheus/node_exporter/tree/master/examples/systemd
 
 A base blackbox_exporter systemd file is also expected to be in place. No examples are currently available.
 
@@ -97,12 +97,12 @@ The following pgMonitor configuration files should be placed according to the fo
 |------------------------------|-----------------|
 | sql_exporter/common/*.yml | /etc/sql_exporter/*.yml |
 | sql_exporter/common/*.sql | /etc/sql_exporter/*.sql |
-| linux/crunchy-sql-exporter@.service | /usr/lib/systemd/system/crunchy-sql-exporter@.service |
+| sql_exporter/linux/crunchy-sql-exporter@.service | /usr/lib/systemd/system/crunchy-sql-exporter@.service |
 | sql_exporter/linux/sql_exporter.sysconfig | /etc/sysconfig/sql_exporter |
 | sql_exporter/linux/crontab.txt | /etc/sysconfig/crontab.txt |
 | postgres_exporter/linux/pgbackrest-info.sh | /usr/bin/pgbackrest-info.sh |
 | postgres_exporter/linux/pgmonitor.conf | /etc/pgmonitor.conf (multi-backrest-repository/container environment only) |
-| sql_exporter/common/sql_exporter.yml.example | /etc/sql_exporter/sql_exporter.yml.example |
+| sql_exporter/common/sql_exporter.yml.example | /etc/sql_exporter/sql_exporter.yml |
 
 
 ##### blackbox_exporter
@@ -117,6 +117,7 @@ The following pgMonitor configuration files should be placed according to the fo
 
 ## Upgrading {#upgrading}
 
+* If you are upgrading to version 5.0 and transitioning to using the new sql_exporter, please see the documentation in [Upgrading to pgMonitor v5.0.0](/changelog/v5_upgrade/)
 * See the [CHANGELOG ](/changelog) for full details on both major & minor version upgrades.
 
 ## Setup {#setup}
@@ -128,7 +129,7 @@ The following pgMonitor configuration files should be placed according to the fo
 The following files contain defaults that should enable the exporters to run effectively on your system for the purposes of using pgMonitor.  Please take some time to review them.
 
 If you need to modify them, see the notes in the files for more details and recommendations:
-- {{< shell >}}/etc/systemd/system/node_exporter.service.d/crunchy-node-exporter-service-rhel{{< /shell >}}
+- {{< shell >}}/etc/systemd/system/node_exporter.service.d/crunchy-node-exporter-service-rhel.conf{{< /shell >}}
 - {{< shell >}}/etc/sysconfig/node_exporter{{< /shell >}}
 - {{< shell >}}/etc/sysconfig/sql_exporter{{< /shell >}}
 
@@ -598,6 +599,19 @@ https://github.com/CrunchyData/pgbouncer_fdw
 
 Once that is working, you should be able to add the {{< shell >}}queries_pgbouncer.yml{{< /shell >}} file to the {{< yaml >}}QUERY_FILE_LIST{{< /shell >}} for the exporter that is monitoring the database where the FDW was installed.
 
+#### Enable Services
+
+To most easily allow the use of multiple postgres exporters, running multiple major versions of PostgreSQL, and to avoid maintaining many similar service files, a systemd template service file is used. The name of the sysconfig EnvironmentFile to be used by the service is passed as the value after the "@" and before ".service" in the service name. The default exporter's sysconfig file is named "postgres_exporter_pg##" and tied to the major version of postgres that it was installed for. A similar EnvironmentFile exists for the per-db service. Be sure to replace the ## in the below commands first!
+
+```bash
+sudo systemctl enable crunchy-postgres-exporter@postgres_exporter_pg##
+sudo systemctl start crunchy-postgres-exporter@postgres_exporter_pg##
+sudo systemctl status crunchy-postgres-exporter@postgres_exporter_pg##
+
+sudo systemctl enable crunchy-postgres-exporter@postgres_exporter_pg##_per_db
+sudo systemctl start crunchy-postgres-exporter@postgres_exporter_pg##_per_db
+sudo systemctl status crunchy-postgres-exporter@postgres_exporter_pg##_per_db
+```
 #### Monitoring multiple databases and/or running multiple postgres exporters (RHEL)
 
 Certain metrics are not cluster-wide, so multiple exporters must be run to avoid duplication when monitoring multiple databases in a single PostgreSQL instance. To collect these per-database metrics, an additional exporter service is required and pgMonitor provides this using the following query file: ({{< shell >}}queries_per_db.yml{{< /shell >}}). In Prometheus, you can then define the global and per-db exporter targets for a single job. This will place all the metrics that are collected for a single database instance together.

hugo/content/grafana/_index.md

@@ -84,7 +84,9 @@ Please note that due to the change from postgres_exporter to sql_exporter, and i
 
 ## Upgrading {#upgrading}
 
-Please review the ChangeLog for pgMonitor and take note of any changes to metric names and/or dashboards. Note that if you are using the included dashboards that are managed via the provisioning system, they will automatically be updated. If you've made any changes to configuration files and kept their default names, the package will not overwrite them and will instead make a new file with an {{< shell >}}*.rpmnew{{< /shell >}} extension. You can compare your file and the new one and incorporate any changes as needed or desired.
+* If you are upgrading to version 5.0 and transitioning to using the new sql_exporter, please see the documentation in [Upgrading to pgMonitor v5.0.0](/changelog/v5_upgrade/)
+* See the [CHANGELOG ](/changelog) for full details on both major & minor version upgrades.
+* Note that if you are using the included dashboards that are managed via the provisioning system, they will automatically be updated. If you've made any changes to configuration files and kept their default names, the package will not overwrite them and will instead make a new file with an {{< shell >}}*.rpmnew{{< /shell >}} extension. You can compare your file and the new one and incorporate any changes as needed or desired.
 
 ## Setup {#setup}
 

hugo/content/prometheus/_index.md

@@ -92,9 +92,9 @@ The following pgMonitor configuration files should be placed according to the fo
 
 ### Upgrading {#upgrading}
 
-Please review the ChangeLog for any changes that may be relevant to your environment.
-
-Of note, items like the alert rules and configuration files often require user edits. The packages will install newer versions of these files, but if the user has changed their contents but kept the same file name, the package will not overwrite them. Instead it will make a file with an {{< shell >}}*.rpmnew{{< /shell >}} extension that contains the newer version of the file. These new files can be reviewed/compared to he user's file to incorporate any desired changes.
+* If you are upgrading to version 5.0 and transitioning to using the new sql_exporter, please see the documentation in [Upgrading to pgMonitor v5.0.0](/changelog/v5_upgrade/)
+* See the [CHANGELOG ](/changelog) for full details on both major & minor version upgrades.
+* Note, items like the alert rules and configuration files often require user edits. The packages will install newer versions of these files, but if the user has changed their contents but kept the same file name, the package will not overwrite them. Instead it will make a file with an {{< shell >}}*.rpmnew{{< /shell >}} extension that contains the newer version of the file. These new files can be reviewed/compared to he user's file to incorporate any desired changes.
 
 ## Setup {#setup}