Switch to using the Core local environment

[pento]

Description

Now that WordPress Core has its own Docker environment, it's possible to base the Gutenberg environment on that.

There are several advantages for Gutenberg, when compared to the existing environment:

• It's cross-platform, only requiring Docker to be installed and running. It also works with Docker Toolbox, allowing all Windows users (not just those with Windows 10 Pro) to use it.
• The WordPress environment uses Docker images that we maintain for the purpose of Core development, which allows us to to customise them as we need.
• It ensures that tests for Core and Gutenberg run in the same environment.
• It's a single environment, which helps avoid the naming clashes we've occasionally seen between the Core and Gutenberg environments.

More broadly for WordPress, tying the development processes of Core and Gutenberg a little closer together helps pave the way for the processes to merge over time.

Architectural Aims

This PR removes the Docker config from Gutenberg entirely (barring a minimal template to hook into the Core environment): requiring wordpress-develop to run is intentional.

There are a few reasons for requiring wordpress-develop. It's worth noting that one of my primary guides for making decisions around local-env is to lay the groundwork for a dev environment that works as easily as possible for non-devs. With that in mind:

• Having a single docker-compose.yml file in Core, as opposed to having one in Core, one in Gutenberg, and many more in other feature plugins that will be able to use this, means that there's only set of Docker containers running. Docker can get fairly weird if you're trying to run containers with the same names, which mount volumes with the same names.
• Having multiple docker-compose.yml files doesn't lend itself to testing plugins together: each plugin ends up with its own environment which can't talk to others.
• Managing how plugins can mount older versions of WordPress becomes exponentially more difficult if the docker-compose.yml config needs to work for every WordPress branch (this is a requirement for local-env so that auto updates can be worked on in old branches), rather than being able to tweak the config for each branch.
• Ideally, feature plugins shouldn't need to do much customisation of the Docker environment, beyond mounting their particular volumes. Exposing the entire docker-compose.yml file is overkill.

That said, not every Gutenberg contributor will have a WordPress source clone, or want to manage one. So, there's also a helper which will download and install a copy of the WordPress source repo, when needed.

Much like the WordPress environment, however, I've aimed to not automate everything. As we found with the Gutenberg environment, too much automation tended to make it difficult (particularly for more advanced contributors) to customise or debug their environment. For example, the scripts no longer install NVM, switch your Node version, or run npm install.

How It Works

If you don't have a WordPress repo to work with, just run npm run env install, that'll take care of downloading WordPress, building it, and connecting Gutenberg into it.

Alternatively, setting the WP_DEVELOP_DIR environment variable to your WordPress source directory, and running npm run env connect will write a docker-compose.override.yml file to the WordPress source directory, and restart the Docker containers. The Docker override file contains the appropriate volume settings for mounting Gutenberg within the WordPress environment.

Here's a complete list of the commands added in this PR:

npm run env install: Automatically downloads, builds, and installs a copy of WordPress to work with. This will be installed in the wordpress folder.

npm run env connect: If you have a WordPress respository checked out already, define the WP_DEVELOP_DIR environment variable with the path to your repository, then run this command to mount Gutenberg inside the Core environment.

npm run env start / npm run env stop: After the environment is setup, there's no need to run the install script again. the start and stop scripts will start and stop the Docker containers.

npm run env update: When a managed WP install is being used, the update script will update the WordPress install and rebuild it. This doesn't happen automatically, but the start script will show warnings if it hasn't been run for more than 7 days.

npm run env reinstall: Resets the database and re-configures WordPress again.

npm run env cli: This is a shortcut for WP-CLI, it also ensures that the CLI command is being run against src or build, depending on what you've set in LOCAL_DIR.

npm run env docker-run: This is a shortcut for docker-compose run --rm.

npm run env test-php: This will run the PHPUnit tests.

npm run env lint-php: This will run the composer lint scripts.

Testing

After both of these methods, your WordPress install will be at http://localhost:8889/wp-admin/, username is admin, and password is password.

Method A (download and configure WordPress automagically)

For testing the managed WordPress install, just run npm run env install, and watch the magic happen. 🎉

Method B (connect to an existing wordpress-develop install)

If you want to test with an external WordPress directory, check it out, and run these commands in your WordPress directory:

nvm install
npm install
npm run build:dev
npm run env:start
npm run env:install

Then in your Gutenberg directory, run:

export WP_DEVELOP_DIR=/path/to/wordpress-develop
npm run env connect
npm run env cli plugin activate gutenberg

Reusability

These scripts are intended to be reusable by other Core feature plugins. While they may support third party development in the future, this isn't an immediate goal.

[youknowriad]

The ./bin/setup-local-env.sh script is documented in a lot of places of the repo. Contributors shouldn't have to clone another repository to trigger a local environment to work on Gutenberg.

We should instead have a local environment embedded in this repository instead. Sharing the same config as Core makes sense.

[pento]

Thank you for your initial thoughts, @youknowriad. This PR is still highly experimental, I haven't touched documentation or worked on polishing the UX, since how it all works together is likely to change as I continue to explore it.

Similarly, you're likely to come across fairly hacky bits and pieces which aren't really ready for review. (For example, Travis config is currently hacked together for looking at "what would it take to get Travis passing in this scenario".)

[youknowriad]

Oh sorry @pento for being too fast. Let me know when you need feedback

[pento]

This does have the potential drawback of requiring Gutenberg contributors who may be unfamiliar with WordPress Core development to check out and build a copy of WordPress. I haven't written a script to automate this task, but I'm open to arguments for or against it. See the before_install section of .travis.yml for an example of the steps needed in an automated solution.

After some discussion over in the core-editor channel, there's clearly still a need for this to be automated in some form. I'll look into it tomorrow.

[pierlon]

Instead of cloning wordpress-develop, could it be referenced as a submodule?

[pento]

@pierlon: The Docker config only lives in the wordpress-develop repo, it isn't included in the WordPress zip download.

A shallow clone is very fast, but the downside is that it requires git to be installed, and available in the path. I'll look at grabbing the wordpress-develop zip from GitHub, as an alternative, but that has its own issues.

[shivapoudel]

@pento Not only Gutenberg plugin but if I do have custom plugin development, then how can I take care of downloading WordPress, building it, and connecting custom XYZ plugin into it utilizing @wordpress/scripts package. Is it possible to link the custom XYZ plugin dev environment on core like how Gutenberg does in this branch? Docker override YAML file has hard-coded path can that be fetched according to package.json environment for plugin dir set, so it might open the possibility of creating a template override for custom plugin development.

[pento]

Thank you for reminding me, @shivapoudel! I half-implemented this, but didn't get back to it. 🙂 5e1c8c2 adds the ability to override the default docker-compose.override.yml.template with your own, by adding this section to your package.json file:

	"wp-env": {
		"plugin-dir": "gutenberg",
		"docker-template": "./bin/docker-compose.override.yml.template"
	}

[shivapoudel]

@pento I have some notes for this PR enhancement from my side as I am working on a custom plugin and implementing these scripts wisely :)

Let me go with the flow if the second try of npm run env install is executed. I receive this error means if the folder named wordpress exists inside my plugin then also it is trying to clone in the second run. Maybe there should be existsSync check :)

Some sort of check before downloading WordPress develop source needed

Remember I don't want to set WP_DEVELOP_DIR environment somewhere thus intentionally wordpress directory is generated which I can gitignore inside a plugin, amazing job :)

Override Template is not correctly appended on the second run

This is my docker-compose.override.yml.template:

services:
  wordpress-develop:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%
      - %PLUGIN_MOUNT_DIR%/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins
      - %PLUGIN_MOUNT_DIR%/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier
  php:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%
      - %PLUGIN_MOUNT_DIR%/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins
      - %PLUGIN_MOUNT_DIR%/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier
  cli:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%
  phpunit:
    volumes:
      - %PLUGIN_MOUNT_DIR%:/var/www/${LOCAL_DIR-src}/wp-content/plugins/%PLUGIN_INSTALL_DIR%

Running npm run env install configures docker.override.yml correctly, let's see :)

version: '3.7'
services:
  wordpress-develop:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  php:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  cli:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
  phpunit:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'

Since this is all fine but whenever I think about future addon development and my plugin anatomy, I can add additional add-ons directory inside packages/addons and connect it to our docker environment with npm run env connect command. Now the config is not correctly generated, look the main first line of share volumes link which is not correct :)

version: '3.7'
services:
  wordpress-develop:
    volumes:
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  php:
    volumes:
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
      - 'D:\htdocs\everest-forms/packages/mu-plugins:/var/www/${LOCAL_DIR-src}/wp-content/mu-plugins'
      - 'D:\htdocs\everest-forms/packages/addons/everest-forms-zapier:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms-zapier'
  cli:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'
  phpunit:
    volumes:
      - 'D:\htdocs\everest-forms:/var/www/${LOCAL_DIR-src}/wp-content/plugins/everest-forms'

Updated: If I delete the override file inside directory wordpress and run npm run env connect from plugin directory then this issue vanishes :)

As a custom plugin development can we achieve this?
Is there any possibilities to change the site title, email used from this script. As a custom plugin, I want to set different site title and email address. Finally, how can I set the build rather than src version of wordpress while being in the plugin directory, not in the WP develop core directory and setting up it .env file?

Just like the docker override some sort of config to configure site title, email address, etc. is appreciated from my side :)

Finally, will there be a WP version selection OR development version only as a future concise asking it here? If so will the previous version of WP includes docker-compose.yml or not :)

[shivapoudel]

@pento I am in a windows environment with Git bash and MySQL volume is not synced in my WordPress directory (which is inside the plugin) due to the format of this line :)

If I personally change this format into - ./mysql:/var/lib/mysql then it syncs the MySQL volume perfectly. Maybe the culprit for windows user but honestly if you have much more idea about where it syncs then it will be appreciated.

[pento]

@shivapoudel: Thank you for the testing and feedback!

0b4c7f5 adds a check to see if WordPress is already installed.

Good point on old volumes staying around when you run npm run env connect after removing them from the template. I'll see what can be done about that.

Is there any possibilities to change the site title, email used from this script. As a custom plugin, I want to set different site title and email address.

I think we can explore this, but it's a lower priority at the moment. Right now, the aim is to build something that works for Core and Feature Plugin development. If it works for your own plugin development, that's awesome, but I'd like to keep the focus on adding options that Core/Gutenberg need, at least for now. 🙂

Finally, how can I set the build rather than src version of wordpress while being in the plugin directory, not in the WP develop core directory and setting up it .env file?

It will recognise the LOCAL_DIR environment variable for this. Run export LOCAL_DIR=build before you run npm run env connect to set it all up. this will only effect your current terminal window, though, we can explore making it possible to set this for the entire project.

Finally, will there be a WP version selection OR development version only as a future concise asking it here? If so will the previous version of WP includes docker-compose.yml or not :)

There will be in the future, but WP branches currently don't have the Docker config. I'm planning on backporting it once it's a bit more stable. 🙂

If I personally change this format into - ./mysql:/var/lib/mysql then it syncs the MySQL volume perfectly. Maybe the culprit for windows user but honestly if you have much more idea about where it syncs then it will be appreciated.

It's referring to the mysql further down in the docker-compose.yml file. I put the MySQL files in a Docker volume so that they persist between restarts, but can be easily removed (for example, if you're switching to an older version of MySQL, it's easier to work with a fresh install, rather than downgrading the data file formats).

[shivapoudel]

@pento Thanks you for your reply with such insight... What about the second time run of npm run env connect not storing the old volumes and generating that file correctly? After the install of the environment if I again try to run connect then docker override file is not updated correctly!

TBW storing the old volumes might not be a good decision from my side and there should be an option to store or just update the override file.

[pento]

@shivapoudel: Could you clarify what's going on here? Are you able to reproduce the same problem by modifying the Gutenberg bin/docker-compose.override.yml.template file, and specifying how you would expect it to behave?

[shivapoudel]

@gziolo Good enough to hear that with the release of Gutenberg 6.5, this package will be updated 👍

[aldavigdis]

For anyone, especially Linux users who are arriving here after seeing ERROR: Version in "./docker-compose.yml" is unsupported. and have managed to run the original Gutenberg Docker setup with no issue until now and have the most recent version of the Docker engine installed via the official repository, you may need to update your version of docker-compose separately as well as the official Docker repo does not include it and the one for your Linux distro is very likely outdated.

Go to https://github.com/docker/compose/releases and find the most recent non-RC release and follow the instructions.

I'm sorry for those who are package manager purists and feel like their setup is "dirty" when installing software like this, but it seems like the best way to go.

I'll follow-up on this by updating the docs as soon as I can.

[pento]

Thank you for noting that @aldavigdis!

For reference, docker-compose version 1.22 is the minimum required: Fedora 29 provides this, but every other distro I looked at had older versions.

Unfortunately, it doesn't look like Docker are planning on providing DEB or RPM packages any time soon (docker/compose#2235)

[ellatrix]

I can no longer run e2e tests locally after this merge. npm run env install -- --fast is run when setting up e2e tests and gives the following error:

Warning: The 'gutenberg' plugin could not be found.
Error: No plugins activated.

The plugin is also gone when I load http://localhost:8889/wp-admin/plugins.php.

[shivapoudel]

@pento Out of the context but don't get me wrong. Previously I use PHPMyAdmin image in docker-compose to access the MySQL database. Since I am new to the dockerized system if you do have way out how to access that container database using MySQL workbench or any other way which you prefer in the Windows environment during the development process will be a great help :)

[pento]

@shivapoudel: You can run a PHPMyAdmin Docker container like so:

docker run -d --link wordpress_mysql_1:db --net wordpress_wpdevnet -p 8081:80 phpmyadmin/phpmyadmin

You may need to change wordpress_mysql_1 to match the name of the MySQL Docker container, check docker ps for a list of running containers.

To login to PHPMyAdmin, visit http://localhost:8081, use the username root, with password password.

[shivapoudel]

@pento That works like a charm. Thanks a lot!

[shivapoudel]

@pento Since the env LOCAL_DIR is src, I get an error during new post/page creation. I have to change that to build and execute npm run build inside wordpress directory to resolve that issue. What is the easiest way so that I don't have to change that LOCAL_DIR to successfully add new post/page?

[aduth]

There's many mixed references in code between usage as npm run env:foo and npm run env foo. Using the colon variant always results in an NPM error for me:

npm ERR! missing script: env:start

Should all of these be changed to be space-separated?

gutenberg/.travis.yml

Line 76 in f68a107

npm run env:start

gutenberg/.travis.yml

Line 78 in f68a107

npm run env:install

gutenberg/packages/scripts/scripts/env/cli.js

Line 17 in f68a107

execSync( `npm run env:cli -- --path=/var/www/${ localDir } ` + args.join( ' ' ), { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

gutenberg/packages/scripts/scripts/env/connect.js

Line 76 in f68a107

execSync( 'npm run env:stop', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

gutenberg/packages/scripts/scripts/env/connect.js

Line 77 in f68a107

execSync( 'npm run env:start', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

gutenberg/packages/scripts/scripts/env/start.js

Line 30 in f68a107

execSync( 'npm run env:start', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

gutenberg/packages/scripts/utils/env.js

Line 114 in f68a107

execSync( 'npm run env:start', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

gutenberg/packages/scripts/utils/env.js

Line 123 in f68a107

execSync( 'npm run env:install', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

gutenberg/packages/scripts/scripts/env/stop.js

Line 8 in f68a107

execSync( 'npm run env:stop', { cwd: env.WP_DEVELOP_DIR, stdio: 'inherit' } );

[aduth]

Not sure I understand the purpose or need for these installs.

wait-on seems like something which could have been useful in place of the sleep 10 which follows a few lines down. I suppose it might have been an initial attempt to try to react to the availability of the environment in a more reliable way than a sleep?

[aduth]

#18924

[aduth]

This was the only place we documented our exclusive support commitment to the active LTS. Was it intentional to change this? What versions of Node do we support now?

[aduth]

#18923

[aduth]

This is dead code.

[mcsf]

Make sure you have Docker installed. The Gutenberg/WP environment expects docker-compose to be available.