Feature Branch: Update the Autoloader

[kbrown9]

This PR introduces major changes to the autoloader package, so it should be versioned 2.0.

---

Why version 2.0?

The previous versions of the Jetpack Autoloader built an in-memory classmap, which mapped each package class to the latest version of the package. It updated this map as each plugin that used the autoloader was loaded. This meant that the autoloader couldn't identify the latest version of a package until all of the plugins have been loaded. To ensure that the latest versions of packages were used, the Autoloader required that plugins never load a package class before all of the plugins had been loaded. This constraint was too limiting, so we decided to try a different approach.

In the new approach introduced in this PR, the first autoloader package that is loaded examines the classmaps for all active plugins and builds the in-memory classmap. This means that after the first plugin that uses the v2.0 autoloader has loaded, the autoloader knows where the latest versions of the packages are located. The plugins no longer need to wait until all the packages have loaded to use the package classes.

Changes proposed in this Pull Request

The plugin build process

1. Note that the Autoloader package's existing composer.json file has an extra value: "class": Automattic\\Jetpack\\Autoloader\\CustomAutoloaderPlugin. This is a custom installer that hooks a method onto the post-autoload-dump event.

2. In CustomAutoloaderPlugin::postAutoloadDump(), we generate an error if the default vendor directory is not used. The autoloader will look for files in the vendor directory, so consuming plugins must use the default directory.

3. When the post-autoload-dump event fires, AutoloadGenerator::dump() is called. This method creates a number of files, including multiple files that are new for v2.0:

   • jetpack_autoload_classmap.php and jetpack_autoload_filemap.php are the filenames of the classmap and filemap generated by the autoloader. We've changed the names of these files to make it easy for the autoloader to distinguish between v2.x and v1.x versions of the autoloader.
   • The other generated files are created with the AutoloadGenerator::getAutoloadPackageFile() method. This method adds a header with a unique namespace to each file. The unique namespace allows the autoloader to load files that are within its own namespace.

The autoloader setup process

1. A consuming plugin loads its autoloader using
   require_once . plugin_dir_path( __FILE__ ) . '/vendor/autoload_packages.php';.

2. The autoload_packages.php file (which was created using the package's autoload.php file) has a unique namespace. It loads the autoload_functions.php file and calls the set_up_autoloader() function.

3. The autoload_functions.php file (which was created using the package's functions.php file) contains a number of functions. Note the global variables in this file, $jetpack_autoloader_latest_version, jetpack_packages_classmap, etc.. These globals are used by the autoloaders to determine what needs to be done.

4. set_up_autoloader() is called first. It does the following:

   • Calls Plugins_Handler::should_autoloader_reset(). This resets the autoloader global variables when an activating plugin is detected. (More info about activating plugins is in step 9).
   • Calls Autoloader_Handler::find_latest_version(). This method traverses through the active plugin folders, looking for autoloaders. It chooses the autoloader with the latest version, and that autoloader does the work.
   • Finally, if the the current autoloader has the latest version, it does the rest of the autoloader setup by calling enqueue_files and Autoloader_Handler::update_autoloader_chain.

5. enqueue_files() sets up the in-memory classmap and filemap for the site. The Classes_Handler and Files_Handler classes traverse through the active plugins and look at the classmap and filemap files for each plugin. They update the locations of the latest version of each package class and file. Each of these classes uses the Plugins_Handler and Version_Selector to help with their work.

6. The Version_Selector class is used to compare package versions.

   • If the JETPACK_AUTOLOAD_DEV constant is set to true, the first development version that is found is used. Development versions are versions that begin with dev- or 9999999-dev.
   • If the JETPACK_AUTOLOAD_DEV is not set to true, development versions are only used if no other version is available.
   • The latest version of a package is chosen using PHP's version_compare function.

7. The Plugins_Handler class is used to collect information about the plugins. The Plugins_Handler::get_all_active_plugins() method is used to retrieve a list of all active and activating plugins.

8. Activating plugins require some special considerations. If a plugin is activated on the wp-admin's Plugins page, the page request will contain an action=activate query. We look for this in the Plugins_Handler::get_plugins_activating_via_request() method.

9. If a plugin is activated using a different method, for example using WP CLI, the autoloader will not be aware of the plugin until it loads its autoload_packages.php file. Then the setup process begins again at step 1, and the Plugins::Handler::should_autoloader_reset() method detects that the current plugin is unknown. This causes a reset of the autoloader global variables. We maintain a list of activating plugins in the $jetpack_autoloader_activating_plugins variable.

10. The autoloader chain is a list of functions that are called in order when a class is requested. The Autoloader_Handler::update_autoloader_chain() method maintains the order of the autoloader functions. It does two things:

    • Ensures that the v2.x autoload function is run before the v1.x autoloader function.
    • Ensures that only one v2.x autoload function is in the autoloader chain.

Version selection when both a v1.x autoloader and a v2.0 autoloader are installed

Version selection is a bit complicated when a v1.x autoloader and a v2.0 autoloader are installed on a site. The v2.0 autoloader cannot access the map files created by the v1.x autoloader, so the v2.0 autoloader doesn’t know anything about packages that have been installed by a plugin using a v1.x autoloader.

• The v2.0 autoloader will select the latest version of a package from the package versions it can access.

• The v2.0 autoloader places its autoloader function before the v1.x autoloader function in the PHP autoloader chain. This means that when the v2.0 autoloader is aware of a package, the v2.0 autoloader will select which version of a package is loaded.

• If a package is used only by plugins with v1.x autoloaders, the v2.0 autoloader won’t know about that package. That package will be loaded by the v1.x autoloader function.

Here is an example site environment with both a v1.7 autoloader and a v2.0 autoloader installed:

Plugin	Autoloader Version	Package A Version	Package B Version	Package C Version
Plugin 1	2.0	2.0	-	1.0
Plugin 2	1.7	1.0	2.0	3.0

• Package A, v2.0, will be loaded with the v2.0 autoloader function.
• Package B, v2.0, will be loaded with the v1.7 autoloader function.
• Package C, v1.0, will be loaded with the 2.0 autoloader function. Note that the latest version of the package (v3.0) is not loaded in this situation because the v2.0 autoloader can’t access Plugin 2's classmap file. This scenario should be rare.

Is this a new feature or does it add/remove features to an existing part of Jetpack?

• This updates the existing Autoloader package. p9dueE-1sf-p2

Testing instructions:

This PR introduces major changes to the autoloader, so thorough testing is required! If you can think of a test environment that isn't covered by the test instructions, please test it.

Build the autoloader files

1. Run composer install and verify that the autoloader files have been generated:
   • vendor/autoload_functions.php
   • vendor/autoload_packages.php
   • vendor/class-autoloader-handler.php
   • vendor/class-classes-handler.php
   • vendor/class-files-handler.php
   • vendor/class-plugins-handler.php
   • vendor/class-version-selector.php
   • vendor/composer/jetpack_autoload_classmap.php
   • vendor/composer/jetpack_autoload_filemap.php

Test in a realistic environment

1. Install multiple plugins on your test site to recreate a realistic environment. Install other plugins that use the autoloader like WooCommerce and VaultPress.
2. Use the site and confirm that everything works as expected and that no errors are generated.
3. Test activating and deactivating plugins to ensure that no errors are generated during activation.

Test with test plugins

1. A12s can access some small test plugins here. These test plugins allow manual testing of plugins with various configurations in various orders. A list of configurations of the example plugins is here.
2. These test plugins display the package version for all the installed packages. Install and activiate/deactive the plugins to test different scenarios. For example:
   • A plugin with a v1.x autoloader is loaded before an autoloader with a v2.0 autoloader.
   • A plugin with a v1.x autoloader is loaded after an autoloader with a v2.0 autoloader.
   • The plugin with the latest autoloader doesn't include all of the packages used by other plugins.
3. Test with the JETPACK_AUTOLOAD_DEV constant set to both true and false and confirm that development versions are selected when the constant is set.

Test with an Updating Plugin

It’s challenging to test a plugin update. Here are instructions for how I test it. Please let me know if have any other ideas!

In these tests, we’ll simulate a file being moved in the updated version of Jetpack.

Cause a fatal error during a plugin update

1. Install the Jetpack beta plugin.
2. Install and activate this branch.
3. Rename the jetpack-dev folder to jetpack.
4. Activate and connect Jetpack.
5. Add this snippet to your site. It does two things:
   • Uses a class from a Jetpack package when the upgrader_process_complete hook fires. Note that the snippet uses the Pre_Connection_JITM class, which is not loaded when Jetpack is connected.
   • Sets the source file for the Jetpack plugin update to wp-content/plugins/jetpack.zip.

add_action( 'upgrader_process_complete', 'load_package_class', 1 );

function load_package_class() {
	$class = new \Automattic\Jetpack\JITMS\Pre_Connection_JITM();
}

add_action( 'upgrader_package_options', 'set_plugin_path' );

function set_plugin_path( $options ) {
	if( 'jetpack/jetpack.php' === $options['hook_extra']['plugin'] ) {
		$options['package'] = WP_PLUGIN_DIR . '/jetpack.zip';
	}
	return $options;
}

6. Create the source zip file for the Jetpack update using a command like zip -r jetpack.zip jetpack. The file name and location must match the source zip file in the upgrader_package_options hook callback. If you use the snippet in step 5, the source file path must be wp-content/plugins/jetpack.zip.

7. In jetpack/vendor/composer/jetpack_autoload_classmap.php, change the file name for the Pre_Connection_JITM class.

   • Note that this file is in the original location in the jetpack.zip file. This change allows us to simulate the file being moved.
   • You can also change the file name of jetpack/vendor/automattic/jetpack-jitm/src/class-pre-connection-jitm.php to match the classmap file name if you'd like. This file won't be requested until after the update, so it doesn't matter.

8. Comment out the add_filter( ‘upgrader_post_install’, …) call here.

9. Navigate to wp-admin -> Plugins. An update should be available for Jetpack. Click update now.

10. The update should fail and a fatal error should be generated in the debug log because the required file for Pre_Connection_JITM couldn't be found. This fatal occurred because the in-memory classmap was not updated.

Test the fix

Using the same snippets as above, test without removing the the add_filter( ‘upgrader_post_install’, …) call.

1. In jetpack/vendor/composer/jetpack_autoload_classmap.php, change the file name for the Pre_Connection_JITM class.

2. Navigate to wp-admin -> Plugins. Reload the page if you're already there. An update should be available for Jetpack. Click update now.

3. The update should be successful and no errors should be generated.

Proposed changelog entry for your changes:

• tbd

[jetpackbot]

	Warnings
⚠️	pre-commit hook was skipped for one or more commits
⚠️	The Privacy section is missing for this PR. Please specify whether this PR includes any changes to data or privacy.

This is an automated check which relies on PULL_REQUEST_TEMPLATE. We encourage you to follow that template as it helps Jetpack maintainers do their job. If you think 'Testing instructions' or 'Proposed changelog entry' are not needed for your PR - please explain why you think so. Thanks for cooperation 🤖

E2E results is available here (for debugging purposes): https://jetpack-e2e-dashboard.herokuapp.com/pr-15106

Generated by 🚫 dangerJS against fce4a97

[kbrown9]

I rebased this branch on master. We had a couple of unit tests that failed for PRs #15375 and #15364, which have been merged into this branch. After the rebase, those unit tests now pass.

[kbrown9]

I rebased on the master branch and force-pushed to resolve some failing unit tests.

[kbrown9]

I rebased on the master branch to resolve a conflict.

[enejb]

This is super exciting! Nice work on this.

I ran into a Fatal error while testing this.

I am not sure if this is due to the woocommerce plugin missing some files or something else but I wasn't able to replicate this issue. :( I wanted to report it here just in case.

What I did was install woo, activate vaultPress, then went to try to activate woocommerce activation failed. I tried again and it worked as expected.

The fatal error that I encountered was:

[19-May-2020 08:45:05 UTC] PHP Warning:  require(/var/www/html/wp-content/plugins/woocommerce/src/Autoloader.php): failed to open stream: No such file or directory in /var/www/html/wp-content/plugins/woocommerce/woocommerce.php on line 31
[19-May-2020 08:45:05 UTC] PHP Stack trace:
[19-May-2020 08:45:05 UTC] PHP   1. {main}() /var/www/html/wp-admin/plugins.php:0
[19-May-2020 08:45:05 UTC] PHP   2. activate_plugin() /var/www/html/wp-admin/plugins.php:44
[19-May-2020 08:45:05 UTC] PHP   3. include_once() /var/www/html/wp-admin/includes/plugin.php:659
[19-May-2020 08:45:05 UTC] PHP Fatal error:  require(): Failed opening required '/var/www/html/wp-content/plugins/woocommerce/src/Autoloader.php' (include_path='.:/usr/share/php') in /var/www/html/wp-content/plugins/woocommerce/woocommerce.php on line 31
[19-May-2020 08:45:05 UTC] PHP Stack trace:
[19-May-2020 08:45:05 UTC] PHP   1. {main}() /var/www/html/wp-admin/plugins.php:0
[19-May-2020 08:45:05 UTC] PHP   2. activate_plugin() /var/www/html/wp-admin/plugins.php:44
[19-May-2020 08:45:05 UTC] PHP   3. include_once() /var/www/html/wp-admin/includes/plugin.php:659
[19-May-2020 08:45:12 UTC] PHP Warning:  require(/var/www/html/wp-content/plugins/woocommerce/src/Autoloader.php): failed to open stream: No such file or directory in /var/www/html/wp-content/plugins/woocommerce/woocommerce.php on line 31
[19-May-2020 08:45:12 UTC] PHP Stack trace:
[19-May-2020 08:45:12 UTC] PHP Warning:  require(/var/www/html/wp-content/plugins/woocommerce/src/Autoloader.php): failed to open stream: No such file or directory in /var/www/html/wp-content/plugins/woocommerce/woocommerce.php on line 31
[19-May-2020 08:45:12 UTC] PHP   1. {main}() /var/www/html/wp-admin/plugins.php:0
[19-May-2020 08:45:12 UTC] PHP Stack trace:
[19-May-2020 08:45:12 UTC] PHP   2. plugin_sandbox_scrape() /var/www/html/wp-admin/plugins.php:177
[19-May-2020 08:45:12 UTC] PHP   1. {main}() /var/www/html/wp-admin/plugins.php:0
[19-May-2020 08:45:12 UTC] PHP   3. include() /var/www/html/wp-admin/includes/plugin.php:2255
[19-May-2020 08:45:12 UTC] PHP   2. plugin_sandbox_scrape() /var/www/html/wp-admin/plugins.php:177
[19-May-2020 08:45:12 UTC] PHP Fatal error:  require(): Failed opening required '/var/www/html/wp-content/plugins/woocommerce/src/Autoloader.php' (include_path='.:/usr/share/php') in /var/www/html/wp-content/plugins/woocommerce/woocommerce.php on line 31
[19-May-2020 08:45:12 UTC] PHP   3. include() /var/www/html/wp-admin/includes/plugin.php:2255
[19-May-2020 08:45:12 UTC] PHP Stack trace:
[19-May-2020 08:45:12 UTC] PHP Fatal error:  require(): Failed opening required '/var/www/html/wp-content/plugins/woocommerce/src/Autoloader.php' (include_path='.:/usr/share/php') in /var/www/html/wp-content/plugins/woocommerce/woocommerce.php on line 31
[19-May-2020 08:45:12 UTC] PHP   1. {main}() /var/www/html/wp-admin/plugins.php:0
[19-May-2020 08:45:12 UTC] PHP Stack trace:
[19-May-2020 08:45:12 UTC] PHP   2. plugin_sandbox_scrape() /var/www/html/wp-admin/plugins.php:177
[19-May-2020 08:45:12 UTC] PHP   1. {main}() /var/www/html/wp-admin/plugins.php:0
[19-May-2020 08:45:12 UTC] PHP   3. include() /var/www/html/wp-admin/includes/plugin.php:2255
[19-May-2020 08:45:12 UTC] PHP   2. plugin_sandbox_scrape() /var/www/html/wp-admin/plugins.php:177
[19-May-2020 08:45:12 UTC] PHP   3. include() /var/www/html/wp-admin/includes/plugin.php:2255

[zinigor]

That's a weird issue with the Autoloader missing, a search yields the same issue someone had on the forum, which went away after reinstalling WooCommerce.

[enejb]

I did more testing here. I tried to look at this from a performance prospective.

I noticed the more active plugin the site has the slower the site performs.
Even if the plugins don't do much such as the autoloader test plugins.

So if you have just 1 plugin installed vs having a number plugin that have auto loader active installed make the page load slower.

I think that the call to should_autoloader_reset() can be quite slow. And it seems to get called for every autoloader. I think it is slow due to the call to get_plugins() which can be quite slow unless its cached in memory.

Another call that is find_latest_autoloader() and it slows down based on the number of active plugins the user is running.

Can we look at improving this performance? Or maybe we can do more proper performance testing to see where we can improve things.

[kbrown9]

I noticed the more active plugin the site has the slower the site performs.
Even if the plugins don't do much such as the autoloader test plugins.

So if you have just 1 plugin installed vs having a number plugin that have auto loader active installed make the page load slower.

I think that the call to should_autoloader_reset() can be quite slow. And it seems to get called for every autoloader. I think it is slow due to the call to get_plugins() which can be quite slow unless its cached in memory.

Another call that is find_latest_autoloader() and it slows down based on the number of active plugins the user is running.

Can we look at improving this performance? Or maybe we can do more proper performance testing to see where we can improve things.

Thanks for looking into this! I haven't seen much of an increase in load times in my tests; maybe I didn't install enough plugins. We should definitely look into improving the performance!

I'll take a look at doing some more comprehensive performance testing and making some changes to improve performance.

[kbrown9]

This PR needs a few updates:

• As @enejb noted in the comment above, the get_plugins() call can be quite slow. I'll investigate removing that call.
• @enejb also noticed that the autoloader should update the maps when a plugin is updated.

[kraftbj]

Rebased. Mostly myself with PHPCS changes.

[dereksmart]

Tested with the suite of example plugins. All seems well here. 🚀