Agree page to test

[eclarke1]

In order to conduct automated tests on WP Core, we need to agree on which theme or theme(s) to use, which page(s)/template(s) we will test, and what content should be loaded on those pages.

Acceptance

• We have agreed on a theme
• We have agreed on a page to test
• We have agreed on what content to stage on the test page

[adamsilverstein]

I propose we start with two pages:

• The latest core theme home page with it's dummy content (eg https://wordpress.org/themes/twentytwentythree/)
• The last (?) classic core theme home page w/demo content - eg https://wordpress.org/themes/twentytwentyone/

[joemcgill]

I think the two pages @adamsilverstein identified are a good starting point. One question I have is whether we want to hard-pin our tests to a specific version of those themes to avoid variance due to code changes in those themes or if our tests should be inclusive of changes to those themes and always load the latest version?

Alternatives to consider:

1. The new front-end tests added to the Gutenberg repo are using the "Hello World" page instead of the home page. While there is value in coordinating our use cases, I think the "Hello World" page is probably too basic of a use case to measure alone, since it only queries a single post and will not require querying any attachment metadata or rendering many complex blocks.

2. For our server timing analysis, we want to identify a couple of different use cases that cover a range of common templates. For that effort, I was thinking we would target:

• A homepage that includes a basic query loop — we'll need demo content for this)
• A basic page — "Hello World" could work for this use case
• A more complex post, including a featured image, several content images, and a range of other blocks — perhaps there is demo content we can use for this as well?

This might be too many use cases for an initial pass, but wanted to get us thinking about how we might build out these tests over time to cover more use cases.

[eclarke1]

Assigning to @felixarntz and @adamsilverstein to kindly review comments here please.

[felixarntz]

@adamsilverstein @joemcgill I think for now going with the respective home page for 2023 and 2021 like @adamsilverstein is saying is a good start, to have a basic test for both block and classic themes.

Regarding the content though, I think it would be good to align the tests across the two themes to use rather similar content. For this first set of having only 1 test for each theme, I would suggest the following:

• Configure both sites to have the latest posts on the home page.
• For both sites, use the demo posts from the wp-themes.com previews.
• As reference, this would mean our home pages effectively would show what is seen here:
  • https://wp-themes.com/twentytwentythree/
  • https://wp-themes.com/twentytwentyone/?page_id=10002

I think using similar content makes most sense, while of course keeping in mind that, to achieve a similar outcome, minor differences may be needed given how block themes and classic themes work differently.

Making sure that both test cases now display a list of latest posts also sets us up for a future like @joemcgill is saying: This first test could basically be our scenario "home page with posts list". For now it'd be the only one, but we could likely keep it as is later and just add more scenarios.

WDYT?

[joemcgill]

I agree that we can use the same test data for all tests. the wp-theme.com site uses the Theme Unit Test Data maintained by the Themes team (see more: https://developer.wordpress.org/themes/advanced-topics/theme-testing/). We'll need to add a step to our workflows to import the theme data before running the performance tests. I don't see prior art on any of the other repos, but we can probably use WP-CLI like this:

# Activate WP Importer plugin.
wp plugin install wordpress-importer --activate
# Don't know why, but failed importing directly.
wget https://raw.githubusercontent.com/WPTRT/theme-unit-test/master/themeunittestdata.wordpress.xml
# Import
wp import themeunittestdata.wordpress.xml --authors=create

One limitation to be aware of is that the theme test data does not currently get automatically updated as Gutenberg iterates on the way content is authored by the block editor, so there is a chance some markup introduced through a more recent version of the editor could affect our results. I don't think it's a major problem, just something to be aware of and maybe a future place where we can improve the overall tooling landscape for automating WP testing.

See: WordPress/theme-test-data#72

[eclarke1]

@mukeshpanchal27 @joemcgill I see this issue is in progress but with Felix and Adam, could I ask who is actually working on this one please, and ask that assignees are kept accurate and up to date.

[joemcgill]

@eclarke1 Looks to me like we have alignment on the following:

We have agreed on a theme: Twenty Twenty Three (fully block theme) and Twenty Twenty One (classic theme)
We have agreed on a page to test: The homepage
We have agreed on what content to stage on the test page: Theme Unit test data.

There is no other work to be done here, so I'm going to mark this as resolved and reflect these decisions in the requirements for #4.