Skip to content

Commit

Permalink
Build/Test Tools: Trim down the upgrade testing matrix.
Browse files Browse the repository at this point in the history 
The upgrade testing workflow is currently at ~978 jobs spawned from the strategy matrix. While it's great to test all possible combinations, GitHub's UI cannot keep up with tracking that number of jobs, often taking 30-45 minutes to accurately report the outcome even though the jobs themselves all complete in under 5 minutes.

This is 2x the total number of concurrent jobs allowed for the entire organization (which creates a backlog and slows everything down even more).

This trims down the number of combinations included in the testing matrices to be a bit more thoughtful following the following methodology:

- The last two releases of WordPress are tested against all PHP and MySQL LTS version combinations and the most recent innovation release.
- The next 6 oldest versions of WordPress are tested against both the oldest and newest releases of PHP currently supported for both PHP 7 & 8 along with the oldest and newest MySQL LTS versions currently supported (no innovation releases).
- For the remaining versions of WordPress receiving security updates, they are only included if the database version was different that the previous major release.
- The oldest version of WordPress receiving security updates should always be tested against the same full list of combinations as the last two releases.

When choosing which MySQL versions to test against:

- Only the most recent innovation release should be included in testing.
- Even though MySQL >= 5.5.5 is currently supported, there are no 5.5.x Docker containers available that work on modern architectures.
- 5.6.x Docker containers are available and work, but 5.6 only accounts for ~2.3% of installs as of 12/6/2024.defaults:
- 5.7.x accounts for ~20% of installs, so this is used below instead.

With these changes, the total number of jobs is reduced by ~58%.

Props johnbillion, mukesh27.
See #62221.

git-svn-id: https://develop.svn.wordpress.org/trunk@59508 602fd350-edb4-49c9-b593-d223f7449a82
  • Loading branch information
@desrosj
desrosj committed Dec 13, 2024
1 parent b4969d4 commit ff5850e
Showing 1 changed file with 97 additions and 26 deletions.
123 changes: 97 additions & 26 deletions .github/workflows/upgrade-testing.yml
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
# Confirms that updating WordPress using WP-CLI works successfully.
#
# This workflow is not meant to test wordpress-develop checkouts, but rather tagged versions officially available on WordPress.org.
name: Upgrade Tests

on:
Expand Down Expand Up @@ -31,9 +34,28 @@ concurrency:
# Any needed permissions should be configured at the job level.
permissions: {}

# Because the number of jobs spawned can quickly balloon out of control, the following methodology is applied when
# building out the matrix below:
#
# - The last two releases of WordPress are tested against all PHP/MySQL LTS version combinations and the most recent
# innovation release.
# - The next 6 oldest versions of WordPress are tested against both the oldest and newest releases of PHP currently
# supported for both PHP 7 & 8 along with the oldest and newest MySQL LTS versions currently supported (no innovation
# releases). At the current 3 releases per year pace, this accounts for 2 additional years worth of releases.
# - Of the remaining versions of WordPress still receiving security updates, only test the ones where the database
# version was updated since the previous major release.
# - The oldest version of WordPress receiving security updates should always be tested against the same combinations as
# detailed for the last two releases.

# Notes about chosen MySQL versions:
# - Only the most recent innovation release should be included in testing.
# - Even though MySQL >= 5.5.5 is currently supported, there are no 5.5.x Docker containers available that work on
# modern architectures.
# - 5.6.x Docker containers are available and work, but 5.6 only accounts for ~2.3% of installs as of 12/6/2024.defaults:
# - 5.7.x accounts for ~20% of installs, so this is used below instead.
jobs:
# Spawns upgrade testing from WordPress 6.x versions on PHP 8.x with MySQL.
upgrade-tests-wp-6x-php-8x-mysql:
# Tests the full list of PHP/MySQL combinations for the last two versions of WordPress.
upgrade-tests-last-two-releases:
name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
if: ${{ github.repository == 'WordPress/wordpress-develop' || github.event_name == 'pull_request' }}
Expand All @@ -43,12 +65,22 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '8.0', '8.1', '8.2', '8.3', '8.4' ]
php: [ '7.2', '7.3', '7.4', '8.0', '8.1', '8.2', '8.3', '8.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4' ]
wp: [ '6.0', '6.1', '6.2', '6.3', '6.4', '6.5', '6.6', '6.7' ]
db-version: [ '5.7', '8.0', '8.4', '9.1' ]
wp: [ '6.6', '6.7' ]
multisite: [ false, true ]

exclude:
- php: '7.2'
db-version: '8.4'
- php: '7.3'
db-version: '8.4'
# MySQL 9.0+ will not work on PHP 7.2 & 7.3
- php: '7.2'
db-version: '9.1'
- php: '7.3'
db-version: '9.1'
with:
os: ${{ matrix.os }}
php: ${{ matrix.php }}
Expand All @@ -58,8 +90,8 @@ jobs:
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

# Spawns upgrade testing from WordPress 6.x versions on PHP 7.x with MySQL.
upgrade-tests-wp-6x-php-7x-mysql:
# Tests the remaining 6.x releases on the oldest and newest supported versions of PHP 7 & 8.
upgrade-tests-wp-6x-mysql:
name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
if: ${{ github.repository == 'WordPress/wordpress-develop' || github.event_name == 'pull_request' }}
Expand All @@ -69,10 +101,10 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '7.2', '7.3', '7.4' ]
php: [ '7.2', '7.4', '8.0', '8.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4' ]
wp: [ '6.0', '6.1', '6.2', '6.3', '6.4', '6.5', '6.6', '6.7' ]
db-version: [ '5.7', '8.4' ]
wp: [ '6.0', '6.1', '6.2', '6.3', '6.4', '6.5' ]
multisite: [ false, true ]

exclude:
Expand All @@ -89,7 +121,7 @@ jobs:
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

# Spawns upgrade testing from WordPress 5.x versions on PHP 7.x with MySQL.
# Tests 5.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 7.
upgrade-tests-wp-5x-php-7x-mysql:
name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
Expand All @@ -98,10 +130,10 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '7.2', '7.3', '7.4' ]
php: [ '7.2', '7.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4' ]
wp: [ '5.0', '5.1', '5.2', '5.3', '5.4', '5.5', '5.6', '5.7', '5.8', '5.9' ]
db-version: [ '5.7', '8.4' ]
wp: [ '5.0', '5.1', '5.3', '5.4', '5.5', '5.6', '5.9' ]
multisite: [ false, true ]

exclude:
Expand All @@ -118,7 +150,7 @@ jobs:
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

# Spawns upgrade testing from WordPress 5.x versions on PHP 8.x with MySQL.
# Tests 5.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 8.
#
# WordPress 5.0-5.2 are excluded from PHP 8+ testing because of the following fatal errors:
# - Use of __autoload().
Expand All @@ -131,10 +163,10 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '8.0', '8.1', '8.2', '8.3', '8.4' ]
php: [ '8.0', '8.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4' ]
wp: [ '5.3', '5.4', '5.5', '5.6', '5.7', '5.8', '5.9' ]
db-version: [ '5.7', '8.4' ]
wp: [ '5.3', '5.4', '5.5', '5.6', '5.9' ]
multisite: [ false, true ]
with:
os: ${{ matrix.os }}
Expand All @@ -145,7 +177,9 @@ jobs:
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

# Spawns upgrade testing from WordPress 4.x versions on PHP 7.x with MySQL.
# Tests 4.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 7.
#
# The oldest version of WordPress receiving security updates should always be tested.
upgrade-tests-wp-4x-php-7x-mysql:
name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
Expand All @@ -154,10 +188,10 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '7.2', '7.3', '7.4' ]
php: [ '7.2', '7.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4' ]
wp: [ '4.1', '4.2', '4.3', '4.4', '4.5', '4.6', '4.7', '4.8', '4.9' ]
db-version: [ '5.7', '8.4' ]
wp: [ '4.1', '4.2', '4.3', '4.4', '4.5', '4.6', '4.7' ]
multisite: [ false, true ]

exclude:
Expand All @@ -174,7 +208,9 @@ jobs:
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

# Spawns upgrade testing from WordPress 4.x versions on PHP 8.x with MySQL.
# Tests 4.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 8.
#
# The oldest version of WordPress receiving security updates should always be tested.
#
# WordPress 4.6-4.9 are excluded from PHP 8+ testing because of the following fatal errors:
# - Use of __autoload().
Expand All @@ -187,9 +223,9 @@ jobs:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '8.0', '8.1', '8.2', '8.3', '8.4' ]
php: [ '8.0', '8.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4' ]
db-version: [ '5.7', '8.4' ]
wp: [ '4.1', '4.2', '4.3', '4.4', '4.5' ]
multisite: [ false, true ]
with:
Expand All @@ -201,13 +237,48 @@ jobs:
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

# The oldest version of WordPress receiving security updates should always be tested against
# the full list of PHP/MySQL combinations.
upgrade-tests-oldest-wp-mysql:
name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
if: ${{ github.repository == 'WordPress/wordpress-develop' || github.event_name == 'pull_request' }}
strategy:
fail-fast: false
matrix:
os: [ 'ubuntu-latest' ]
php: [ '7.2', '7.3', '7.4', '8.0', '8.1', '8.2', '8.3', '8.4' ]
db-type: [ 'mysql' ]
db-version: [ '5.7', '8.0', '8.4', '9.1' ]
wp: [ '4.1' ]
multisite: [ false, true ]

exclude:
- php: '7.2'
db-version: '8.4'
- php: '7.3'
db-version: '8.4'
# MySQL 9.0+ will not work on PHP 7.2 & 7.3
- php: '7.2'
db-version: '9.1'
- php: '7.3'
db-version: '9.1'
with:
os: ${{ matrix.os }}
php: ${{ matrix.php }}
db-type: ${{ matrix.db-type }}
db-version: ${{ matrix.db-version }}
wp: ${{ matrix.wp }}
new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
multisite: ${{ matrix.multisite }}

slack-notifications:
name: Slack Notifications
uses: WordPress/wordpress-develop/.github/workflows/slack-notifications.yml@trunk
permissions:
actions: read
contents: read
needs: [ upgrade-tests-wp-6x-php-8x-mysql, upgrade-tests-wp-6x-php-7x-mysql, upgrade-tests-wp-5x-php-7x-mysql, upgrade-tests-wp-5x-php-8x-mysql, upgrade-tests-wp-4x-php-7x-mysql, upgrade-tests-wp-4x-php-8x-mysql ]
needs: [ upgrade-tests-last-two-releases, upgrade-tests-wp-6x-mysql, upgrade-tests-wp-5x-php-7x-mysql, upgrade-tests-wp-5x-php-8x-mysql, upgrade-tests-wp-4x-php-7x-mysql, upgrade-tests-wp-4x-php-8x-mysql, upgrade-tests-oldest-wp-mysql ]
if: ${{ github.repository == 'WordPress/wordpress-develop' && github.event_name != 'pull_request' && always() }}
with:
calling_status: ${{ contains( needs.*.result, 'cancelled' ) && 'cancelled' || contains( needs.*.result, 'failure' ) && 'failure' || 'success' }}
Expand Down

0 comments on commit ff5850e

Please sign in to comment.