Automate the testing of popular blocks in current and edge versions of GB #45048

fullofcaffeine · 2020-08-20T02:20:11Z

Changes proposed in this Pull Request

Automates steps 1 to 7 of the Good Mountain testing checklist but does not focus on testing individual blocks (this will come in subsequent PR(s));
Look for blocks on the page and also look for block errors on the page;
Takes screenshots from edge and non-edge editor and preview.
Do that for edge/non-edge sites across the most popular themes
Adds a new test account and several new test sites (the account credentials have been added to the encrypted config file)

Repeat that for the most popular themes in wpcom.

Testing instructions

Setup your env to run Calypso E2E tests (search for "Automated end-to-end Testing") in the FG;
Run with ./node_modules/.bin/mocha specs/wp-calypso-gutenberg-upgrade-spec.js*[0]
Wait for the tests to finish. It might take a good while since it tests everything for each test site
Make sure it didn't err, if it did, it might not be an actual failure, try running that specific step again. If it continues failing, let us know
Look inside the screenshot*[1] folder for the screenshots. There should be a directory per test site name. You can find the list of test sites in the test file itself. They are all part of the same test account called gutenbergUpgradeUser. You can verify it (alongside the credentials) by decrypting the config file in this PR and looking for it in your config/local-decrypted.json file.

*[0] Make sure to cd into test/e2e first.
*[1] test/e2e/screenshots/

TODO

Figure out if setting up blocks in a wpcom site then copying the block markup to another renders the blocks correctly in the new site (it looks like it does! This means we can postpone for now the idea of switching gutenberg or applying stickers using a query param)
Populate tiled-gallery with sample images
Take screenshots of the editor and preview in the non-edge site
Take screenshots of the editor and preview in the edge site, after loading the blocks markup from non-edge
Research how to set it up as a scheduled test and to run it using the "Needs e2e testing Gutenberg edge" tag (we'll just let it run as part of the current e2e pipeline)

matticbot · 2020-08-20T02:20:13Z

Test live: https://calypso.live/?branch=try/gutenberg-upgrade-e2e-tests

matticbot · 2020-08-20T02:23:32Z

This PR does not affect the size of JS and CSS bundles shipped to the user's browser.

Generated by performance advisor bot at iscalypsofastyet.com.

fullofcaffeine · 2020-08-20T02:34:31Z

test/e2e/lib/gutenberg/blocks/contact-info-block-component.js

@@ -0,0 +1,14 @@
+/**
+ * External dependencies


Left those comments for now as I'm not sure if I'll change these files soon, but will remove in a cleanup before taking this PR out of the draft state.

fullofcaffeine · 2020-08-20T05:09:31Z

test/e2e/lib/gutenberg/gutenberg-editor-component.js

+		return codeEditor.sendKeys( Key.CONTROL + 'v' );
+	}
+
+	async setBlocksCode( code ) {


This is not used anymore and will be removed.

fullofcaffeine · 2020-08-20T05:09:36Z

test/e2e/lib/gutenberg/gutenberg-editor-component.js

+		);
+	}
+
+	async getBlocksCode() {


This is not used anymore and will be removed

fullofcaffeine · 2020-08-20T23:48:10Z

test/e2e/specs/wp-calypso-gutenberg-upgrade-spec.js

+
+				// jetpack/layout-grid
+				await gEditorComponent.insertLayoutGrid();
+				const layoutGridDisplayedInEditor = await gEditorComponent.layoutGridDisplayedInEditor();


There's quite a bit of repetition here and I don't like it. I'll see how to dry it up later.

sirreal

Nice start here, this is probably ready for more final reviews? Shall we swap the in progress label?

sirreal · 2020-08-27T16:12:22Z

test/e2e/lib/gutenberg/blocks/blog-posts-block-component.js

+class BlogPostsBlockComponent extends GutenbergBlockComponent {}
+
+BlogPostsBlockComponent.blockTitle = 'Blog Posts';
+BlogPostsBlockComponent.blockName = 'a8c/blog-posts';


This seems more idiomatic:

Suggested change

class BlogPostsBlockComponent extends GutenbergBlockComponent {}

BlogPostsBlockComponent.blockTitle = 'Blog Posts';

BlogPostsBlockComponent.blockName = 'a8c/blog-posts';

class BlogPostsBlockComponent extends GutenbergBlockComponent {

static blockTitle = 'Blog Posts';

static blockName = 'a8c/blog-posts';

}

Oh, that was the first thing I tried. Unfortunately, I got a syntax error. The tests don't go through a transpiler, so it fallbacks to what node supports, and AFAIK, it still doesn't support static class variables yet.

Using node 12.18.0 (the one specified in wp-calypso's .nvmrc):

./node_modules/.bin/mocha specs/wp-calypso-gutenberg-upgrade-spec.js file:///home/fullofcaffeine/workspace/code/wp-calypso/test/e2e/lib/gutenberg/blocks/youtube-block-component.js:7 static blockTitle = 'Foo'; ^ SyntaxError: Invalid or unexpected token at Object.Module._extensions..js (internal/modules/cjs/loader.js:1176:10)

Oh, but it does support static getters:

class YoutubeBlockComponent extends GutenbergBlockComponent { static get blockTitle() { return 'foo '; } }

I think I'll use static getters for now.

I think it's an issue with how this is configured. I believe mocha is using its own parser. In node 12.18.0 it works:

> class A {} undefined > class B { a='a';} undefined > A.a undefined > B.a undefined > new B().a 'a' > A.a='a' 'a' > A.a 'a' > class C { static a = 'a' } undefined > C.a 'a'

Let's just leave it for this PR. It's not really relevant.

Yeah, I tested it with vanilla node and saw it worked. I tried to figure out what as the culprit in mocha but couldn't find and left it aside. I'm using static getters for now.

I'd be interested to know what is the culprit here. I saw the mocha js binary and AFAIK it uses node as well. That's pretty confusing.

Fix for this in #45268. I added static properties in 509fec9.

sirreal · 2020-08-27T16:13:39Z

test/e2e/lib/gutenberg/blocks/index.js

@@ -0,0 +1,18 @@
+// TODO some blocks are exported as default and some not... maybe change all of them to export as default?


Consistency is likely the best approach. Maybe in another PR we change them all to use one or the other.

Slight preference from me for named exports here.

sirreal · 2020-08-27T16:14:30Z

test/e2e/lib/gutenberg/blocks/image-block-component.js

@@ -10,10 +10,6 @@ import * as driverHelper from '../../driver-helper';
 import GutenbergBlockComponent from './gutenberg-block-component';

 export class ImageBlockComponent extends GutenbergBlockComponent {
-	constructor( driver, blockID ) {


This cleanup is nice, but it distracts from the focus on this PR. There's a lot of cleanup like this that we can probably do in a few tiny, focused PRs.

Copy that 👍🏻 , I'll keep that in mind.

sirreal · 2020-08-27T16:15:50Z

test/e2e/lib/gutenberg/blocks/tiled-gallery-block-component.js

+class TiledGalleryBlockComponent extends GutenbergBlockComponent {
+	async uploadImages( filesDetails ) {
+		const fileInput = this.driver.findElement(
+			By.xpath( `//*[@id="${ this.blockID.slice( 1 ) }"]/div/div/div[3]/div[2]/input` )


😅 Oh boy… what's this targeting? Something in the media placeholder? Maybe a dedicated helper would be good?

This selects the hidden file input inside the tile gallery. I've moved this to its own instance field with a more descriptive name. Let me know if you had anything else in mind.

fullofcaffeine · 2020-08-28T01:14:15Z

@Automattic/good-mountain @Automattic/e2e-test-reviewers Please have a look and let me know what you think.

Please do bear in mind that it might seem incomplete - the test data for most of the blocks are not being filled yet (only for two of them) - but that was a deliberate decision taken to allow us to finish the initial groundwork/exploration task earlier instead of having a big/long-running PR (this version already provides some value and we can use it to help us test the next GB version).

Also, the assertion for invalid blocks has been commented out for now due to an issue in the jetpack/tiled-gallery. I'm not sure how to proceed here and could use some input. I see a couple of alternatives: 1) We can ignore this step for the jetpack/tiled-gallery for now; 2) We can leave it commented out until the issue is fixed; 3) We can remove tiled-gallery from the checks?

…f GB

…ed on the frontendn

- Get rid of repeat block factory methods in the GutenbergEditorComponent class; - Let the block class know about its name and title; - Change the usage of name to more closely match the semantics of the block API (name and title); - Added a step to check for errors in the editor in the edge site; - Added a step to check if all expected blocks are in the editor in the edge site;

…omponent classes

…he way of sshots

…-usable helper

Because of because of Automattic/jetpack#16514

…with a name that better describes it

sirreal · 2020-08-28T12:22:06Z

Rebased with #45268 merged.

sirreal

Nice work here! 🚢

sirreal · 2020-08-28T14:47:13Z

I don't see the screenshots artifacts in https://app.circleci.com/pipelines/github/Automattic/wp-e2e-tests-for-branches/36242/workflows/a894971d-692c-4e0e-9951-884404b1774c/jobs/97521/artifacts - is the plan to store them there?

We can introduce that in a follow-up, it should be a small change to the CircleCI configuration.

fullofcaffeine · 2020-08-28T14:54:10Z

I don't see the screenshots artifacts in https://app.circleci.com/pipelines/github/Automattic/wp-e2e-tests-for-branches/36242/workflows/a894971d-692c-4e0e-9951-884404b1774c/jobs/97521/artifacts - is the plan to store them there?

We can introduce that in a follow-up, it should be a small change to the CircleCI configuration.

Hmm, thanks for pointing that out! I just assumed that by saving them to the screenshots folder, everything else would fall into place, but there might be something in CircleCI as you said. I'll take note of that.

…rsions of GB (#45048)" This reverts commit a8bab01.

…rsions of GB (#45048)" (#45277) This reverts commit a8bab01.

… edge versions of GB (#45048)" (#45277)" This reverts commit 6c5580f.

…ons of GB // take 2 (#45278) * Revert "Revert "Automate the testing of popular blocks in current and edge versions of GB (#45048)" (#45277)" This reverts commit 6c5580f. * Update and encrypt fixed settings Make sure it only adds the new Gutenberg upgrade account credentials and nothing else. Co-authored-by: Marcelo Serpa <marcelo.serpa@automattic.com>

fullofcaffeine force-pushed the try/gutenberg-upgrade-e2e-tests branch from 4d1636a to 41f3c11 Compare August 20, 2020 02:32

fullofcaffeine commented Aug 20, 2020

View reviewed changes

sirreal added the [Status] In Progress label Aug 21, 2020

sirreal assigned fullofcaffeine Aug 21, 2020

fullofcaffeine force-pushed the try/gutenberg-upgrade-e2e-tests branch 2 times, most recently from 6f0bdb0 to 5cc6140 Compare August 25, 2020 02:50

fullofcaffeine marked this pull request as ready for review August 25, 2020 03:17

fullofcaffeine mentioned this pull request Aug 26, 2020

[Gutenberg WPCOM e2e tests i0] Automate steps 1 to 7 (Tracking Issue) #45209

Closed

4 tasks

fullofcaffeine linked an issue Aug 26, 2020 that may be closed by this pull request

[Gutenberg WPCOM e2e tests i0] Automate steps 1 to 7 (Tracking Issue) #45209

Closed

4 tasks

fullofcaffeine force-pushed the try/gutenberg-upgrade-e2e-tests branch 2 times, most recently from 856a68f to 790bfa1 Compare August 27, 2020 14:43

sirreal reviewed Aug 27, 2020

View reviewed changes

fullofcaffeine requested a review from a team August 28, 2020 00:56

matticbot added the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Aug 28, 2020

fullofcaffeine removed the [Status] In Progress label Aug 28, 2020

fullofcaffeine force-pushed the try/gutenberg-upgrade-e2e-tests branch from 311e0dd to ae084ab Compare August 28, 2020 01:28

sirreal requested a review from a team August 28, 2020 08:24

sirreal added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. and removed [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. labels Aug 28, 2020

sirreal mentioned this pull request Aug 28, 2020

fix(deps): update dependency esm to ^3.2.25 #45268

Merged

fullofcaffeine added 4 commits August 28, 2020 14:21

Automate the testing of popular blocks in current and edge versions o…

0beed60

…f GB

Take screenshots

565f67d

Improve code a bit and group sshots in a directory named after the site

36e202c

Simplified the assertions by DRYing it in a single re-usable function

13cd9f1

fullofcaffeine and others added 13 commits August 28, 2020 14:21

Cleanup

08d17d7

Take screenshot of editor blocks and preview for edge sites too

b9ea585

Step description improvement

8b3c9ba

Use the published page instead of the preview to check content render…

a999037

…ed on the frontendn

Fix code editor text area not being found in headless mode

d922a96

Use static getters to define block name and title in the test block c…

1c52a2d

…omponent classes

Close the tools and options menu after using to avoid it getting on t…

4ae8d71

…he way of sshots

Added check for invalid blocks and grouped other common steps in a re…

685b7f5

…-usable helper

Comment-out the invalid block assertion temporarily

fe36a72

Because of because of Automattic/jetpack#16514

Move the file selector for the TiledGalleryBlockComponent to a field …

6d1857d

…with a name that better describes it

Simplify a bit more and be explicit about the editor checks

0044d6a

Use static properties for block{Name,Title}

2f206ed

sirreal force-pushed the try/gutenberg-upgrade-e2e-tests branch from 509fec9 to 2f206ed Compare August 28, 2020 12:21

sirreal approved these changes Aug 28, 2020

View reviewed changes

fullofcaffeine merged commit a8bab01 into master Aug 28, 2020

fullofcaffeine deleted the try/gutenberg-upgrade-e2e-tests branch August 28, 2020 15:29

matticbot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Aug 28, 2020

sirreal added a commit that referenced this pull request Aug 28, 2020

Revert "Automate the testing of popular blocks in current and edge ve…

41b7860

…rsions of GB (#45048)" This reverts commit a8bab01.

sirreal mentioned this pull request Aug 28, 2020

Revert "Automate the testing of popular blocks in current and edge versions of GB" #45277

Merged

sirreal added a commit that referenced this pull request Aug 28, 2020

Revert "Automate the testing of popular blocks in current and edge ve…

6c5580f

…rsions of GB (#45048)" (#45277) This reverts commit a8bab01.

fullofcaffeine pushed a commit that referenced this pull request Aug 28, 2020

Revert "Revert "Automate the testing of popular blocks in current and…

4c9bda4

… edge versions of GB (#45048)" (#45277)" This reverts commit 6c5580f.

fullofcaffeine mentioned this pull request Aug 28, 2020

[GB upgrade e2e] Fill all the popular blocks with the necessary minimum data for them to appear in the frontend and make sure each of them is displayed #45284

Closed

2 tasks

fullofcaffeine mentioned this pull request Sep 22, 2020

[GBonWPCOM e2e tests i0] Add assertion for blocks in the frontend (published) page for edge/non-edge, add @parallel tag #45794

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate the testing of popular blocks in current and edge versions of GB #45048

Automate the testing of popular blocks in current and edge versions of GB #45048

fullofcaffeine commented Aug 20, 2020 •

edited

Loading

matticbot commented Aug 20, 2020

matticbot commented Aug 20, 2020

fullofcaffeine Aug 20, 2020

fullofcaffeine Aug 20, 2020

fullofcaffeine Aug 20, 2020

fullofcaffeine Aug 20, 2020

sirreal left a comment

sirreal Aug 27, 2020

fullofcaffeine Aug 27, 2020 •

edited

Loading

fullofcaffeine Aug 27, 2020 •

edited

Loading

fullofcaffeine Aug 27, 2020

fullofcaffeine Aug 27, 2020

sirreal Aug 27, 2020

fullofcaffeine Aug 27, 2020

sirreal Aug 28, 2020 •

edited

Loading

sirreal Aug 27, 2020

sirreal Aug 27, 2020

fullofcaffeine Aug 27, 2020

sirreal Aug 27, 2020

fullofcaffeine Aug 28, 2020 •

edited

Loading

fullofcaffeine commented Aug 28, 2020

sirreal commented Aug 28, 2020

sirreal left a comment

sirreal commented Aug 28, 2020

fullofcaffeine commented Aug 28, 2020 •

edited

Loading

		@@ -0,0 +1,18 @@
		// TODO some blocks are exported as default and some not... maybe change all of them to export as default?

Automate the testing of popular blocks in current and edge versions of GB #45048

Automate the testing of popular blocks in current and edge versions of GB #45048

Conversation

fullofcaffeine commented Aug 20, 2020 • edited Loading

Changes proposed in this Pull Request

Testing instructions

TODO

matticbot commented Aug 20, 2020

matticbot commented Aug 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sirreal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fullofcaffeine Aug 27, 2020 • edited Loading

Choose a reason for hiding this comment

fullofcaffeine Aug 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sirreal Aug 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fullofcaffeine Aug 28, 2020 • edited Loading

Choose a reason for hiding this comment

fullofcaffeine commented Aug 28, 2020

sirreal commented Aug 28, 2020

sirreal left a comment

Choose a reason for hiding this comment

sirreal commented Aug 28, 2020

fullofcaffeine commented Aug 28, 2020 • edited Loading

fullofcaffeine commented Aug 20, 2020 •

edited

Loading

fullofcaffeine Aug 27, 2020 •

edited

Loading

fullofcaffeine Aug 27, 2020 •

edited

Loading

sirreal Aug 28, 2020 •

edited

Loading

fullofcaffeine Aug 28, 2020 •

edited

Loading

fullofcaffeine commented Aug 28, 2020 •

edited

Loading