Performance: Slow loading of tags and categories on sites with large amounts

[tecnogaming]

Crowded blog with 100 categories and thousands of tags (750.000 articles).

Trying to add a category gets a delay so big that the showing up of categories can take up to 30 seconds !!. The same thing is happening on tags.

We are using Yoast SEO and we already tried to disable it and it is not the cause. Even when categories and tags are set, trying to "show" the current settings is painful as opening a post could take 30 seconds or more for current categories and tags show up. This lead to many of my writers incorrectly adding repeated tags as they won't show up on time. The VERY AGGRESIVE autosave feature is also something that affects the delay.

[designsimply]

I will work on a way to generate several thousand tags for testing, and this is exactly the kind of issue that opt-in testing helps uncover. I appreciate that you have reported this!

[tecnogaming]

Thanks so much for your reply. Yes, it does seem like Gutenberg is not designed for big blogs as having tons of categories and tags makes it almost impossible to handle. There must be something wrong on the way Gutenberg ask for that information.

[danielbachhuber]

I will work on a way to generate several thousand tags for testing

WP-CLI is pretty helpful for this:

wp term generate category --count=1000

Trying to add a category gets a delay so big that the showing up of categories can take up to 30 seconds !!. The same thing is happening on tags.

#6694 is potentially related. However, it would be useful to track down whether the source of your slowness is: 1) purely from the amount of data, or 2) caused by some plugin or your theme causing slowness (in addition to the amount of data).

[tecnogaming]

Quick editor is instant, classic editor is ultra fast, clearly gutenberg is doing something wrong El jue., 27 sep. 2018 12:09 p. m., Daniel Bachhuber < notifications@github.com> escribió:

…

I will work on a way to generate several thousand tags for testing WP-CLI is pretty helpful for this: wp term generate category --count=1000 Trying to add a category gets a delay so big that the showing up of categories can take up to 30 seconds !!. The same thing is happening on tags. #6694 <#6694> is potentially related. However, it would be useful to track down whether the source of your slowness is: 1) purely from the amount of data, or 2) caused by some plugin or your theme causing slowness (in addition to the amount of data). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10169 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF8Y6wxvGDM4WlIa-BwAybr01aF5wdqcks5ufOo-gaJpZM4W5Er1> .

[websupporter]

I think the problem is related to the following:
The current implementation does not use the state management system due to #11643. Currently, this results in a lot of API Requests instead which slows down the whole process.

[tecnogaming]

It's impossible for me. It slows down so much that I prefer to do a "quick edit" and add the tags by that method than trying to deal with this slowness and unresponsiveness. Clearly this technique is not working. El vie., 16 nov. 2018 a las 4:17, David Remer (<notifications@github.com>) escribió:

…

[image: ezgif-2-c8ccf08bc746] <https://user-images.githubusercontent.com/6458412/48603667-e9f3fd80-e97f-11e8-8a09-d3334e02284e.gif> I think the problem is related to the following: The current implementation does not use the state management system due to #11643 <#11643>. Currently, this results in a lot of API Requests instead which slows down the whole process. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#10169 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF8Y69ExOZZisA-bEBb-a494b-3tUiuaks5uvmacgaJpZM4W5Er1> .

-- *Alex Vojacek* CEO & Founder - TecnoGaming Gaming News/Reviews - Hardware News/Reviews https://www.tecnogaming.com

[Widhy980]

I also have the same problem, sometimes I type a lot of tags and immediately disappear and re-write again. then I tried to wait for the autocomplete and sometimes it suddenly disappeared. is it possible to use same system on classic editor?.

I have 8.147 Tags on my WordPress.

[ActionCoding]

I am also having the same issue when entering tags. Sometimes it is using so much memory it slows my system to a hault. Is there any way to use the old system for tags where it saves them when you click to update the post instead of when you are typing them in?

[MartinHlavna]

Same issue here, on smaller scale.

Loadnig of 50 categories takes about 3-4 seconds, which is kinda usable.
Hower typing tags (3023 at the time of writing) is real problem. If I type multipe tags at once they dissappear. Selecting tag from search results kinda helps, but loading takes more than 10 seconds.

[SuzuKube]

Same problem :/ ! On the tags ^^'

[designsimply]

I think it helps for new people commenting to mention the total number of tags they have on their site. Thank you to those who have commented so far.

[psokolovs]

I have a few thousand tags, and this is a serious issue for me. Especially because my server is pretty slow, so each API call is about a 1 second response time. It's basically paginating non-stop.

[oxyc]

A quick hack I use for a taxonomy called postal_code so it returns up to 100000 terms per request:

add_filter('rest_postal_code_query', function ($prepared_args, $request) {
  if (strpos($request->get_route(), '/wp/v2/') === 0) {
    if ($prepared_args['number'] === 100) {
      $prepared_args['number'] = 100000;
    }
  }
  return $prepared_args;
}, 10, 2);

source: https://github.com/WordPress/gutenberg/pull/10762/files#diff-d0c65e7ac505d44d8571ddaffc510d98L412

[drawcard]

In my case, my website only has 1 taxonomy, with 1 checkbox to select from, and it still takes a good 5 seconds to load that checkbox in the sidebar.

I've had to disable Gutenberg on that CPT entirely just to avoid this issue. This is quite a nightmare to deal with.

[amrutadotorg]

Hi. I have 845 categories in my custom taxonomy. It takes 90 seconds to load all of them. Do you have any quick solution for me?

Thank you.

[vijayrnathan]

Has anyone found a good solution or workaround for this?

[drawcard]

In my case, switching back to Classic Editor is the only thing I can do to alleviate the problem.

You can disable Gutenberg for a particular custom post or post type: https://digwp.com/2018/04/how-to-disable-gutenberg/ if you prefer to use it in some areas but not in others.

[amrutadotorg]

Hi. After upgrading MariaDB from 10.4.8 to 10.4.13 (Amazon RDS) the load timing improved from 90 seconds to 5 seconds.

[zaidiseed]

Hi,

This issue caused by Gutenberg editor as it using API requests to get taxonomies, So when you have a large number of taxonomies "Tags or Categories" the api request will initiate thousands of requests to get all tags, and this will slow down your site as well as causing a high CPU usage if there is more than admin or author is working at the same time.

What is the solution??

Switch to the classic editor (Download it)

The classic editor using ajax request method instead of API and this will send a single request to get all wp terms instead of sending thousands of requests while using API get method in Gutenberg editor.

Regards.

Zaid Iseed

[MartinHlavna]

@zaidiseed That is more workaround than solution :) Since this is Gutenberg issue tracker, the issue will remain regardless of how many users will not use Gutenberg.

[zaidiseed]

@MartinHlavna I agree it's a workaround, but it solves the issue.

I have more than 120 publishers working at the same time and when I switched back to Classic editor the CPU usage of the server has reduced by 90% and wp-terms queries are being executed within milliseconds.

[gwwar]

Slow tag search was fixed by @mgrenierfarmmedia in #23841. We should see this fix in WP 5.6+ lines. See versions in WP

Here's an example where I have 10000 tags locally:

tags.mp4

We likely need a similar change for the hierarchical-term-selector.js to limit results, instead of requesting them all with per_page: -1

gutenberg/packages/editor/src/components/post-taxonomies/hierarchical-term-selector.js

Line 39 in c993a86

per_page: -1,

[gwwar]

This appears to be broken still for categories 🙈. I have 10000 locally, and the component ends up fetching everything before rendering the panel

categories.mp4

Finally After 262 Requests

[gwwar]

Leaving a summary comment of where I left off while 🔍 the category issue. Folks are welcome to take this issue if it's of interest.

To move this forward we'll want to make sure that the following use cases are addressed:

Search + Tree View

Currently we render categories in a tree view. If I'm correct, categories and other hierarchical terms may be nested to any depth. Adding search means we may need to show a flat list of partial results, or a partial sub-tree. Ideally we can still display the full tree view when the total number of terms is small.

Search for a parent category

This is a little more straightforward, but when adding a category we should be able to search for the parent category.

[megane9988]

@gwwar

I'm having a hard time with this problem.
I have over 1000 categories on my web site.
The current block editor frequently switches between the details panel for each block and the settings panel for posts.
Each time this happens, the database is queried, which can have a negative impact on performance.

I would be very happy if this situation could be fixed.

[Mamaduka]

A few things have changed that could generally improve Post Taxonomies components. We can now make context-specific requests with the data module. It means we can get rid of apiFetch.

I also thought that maybe we could use terms.length for the query per_page instead of using the -1.

[torounit]

I guess we can use getEntityRecords to reuse the terms once they are loaded without destroying them.

[jfdialogs]

We have a project with 45k tags. There's pretty much no way to load that many 100 at a time and have it make sense. The fixes above don't resolve the issue (it confuses the tag downloader and just hits page 3 447 times), and the big sell on this update was Gutenberg, so switching to classic seems off the table. Would like to see the tag search not download everything off the top.

If that's supposed to work, let me know if there's a switch, hook or filter I need to enable to get it to work.

[noesteijver]

Hello,

We are also experiencing big load spikes because of this problem.

It seems that, when the metaboxes for our custom taxonomies are opened all at once (which happens when you had them all opened on the last edit of a post) it starts loading all of the terms at once (100 at a time). But this still results in a load spike which makes the WordPress environment very slow or even result in downtime.

We recently switched to Gutenberg, but this is a very big problem for us.

If I need to provide any extra information to help solve the problem, please let me know!

[bozzmedia]

One of our slowest actions in the backend is displaying/adding categories and tags. 15yr old site, 40,000 posts, hundreds of tags and categories respectively.

WP 6.1.1, PHP 8.0, mysql 5.7.41, redis object cache, nginx page cache, 16GB 4cpu vps

[bozzmedia]

To add to this, using quick edit in the All posts area, simply adding a category to a post through the bulk actions takes up to 30 seconds to process.

wp_term_relationships - 92986 rows (9mb)
wp_terms and wp_term_taxonomy are about the same size ~4,500 rows (650kb)

wp_termmeta is virtually empty?

---

I don't know if it helps but here's one of those long request/response results:

post_title=Test
post_name=test
mm=02
jj=16
aa=2023
hh=12
mn=19
ss=04
post_author=10799
post_password
post_category[]=0
post_category[]=38
post_category[]=1
tax_input[post_tag]=central eastside industrial council, Parking
comment_status=open
_status=publish
_inline_edit=d0b1e276b6
post_view=list
screen=edit-post
action=inline-save
post_type=post
post_ID=370487
edit_date=true
post_status=all

request

POST /wp-admin/admin-ajax.php HTTP/3
Host: redacted.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0
Accept: text/html, /; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://redacted.org/wp-admin/edit.php
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 483
Origin: https://redacted.org
DNT: 1
Alt-Used: redacted.org
Connection: keep-alive
Cookie: wordpress_sec_aa008fe4c7947530ea7a9cc9f17330a5=bozz%7C1676849151%7CzXQpPF4CKeUvKV7sW4qB0UjS6LmkajdjcByZ26pE5cM%7Cd270625212da0faf2954b2e51c95f64def7a1d0c038f761775a2d73da906a9ed; _ga_T74BKBBZGW=GS1.1.1676676245.27.1.1676676354.0.0.0; _ga=GA1.2.1321015878.1670464850; advanced_ads_page_impressions=%7B%22expires%22%3A1985824851%2C%22data%22%3A191%7D; advanced_ads_browser_width=1903; __gads=ID=1540eb385e40008b-22770f039dd70070:T=1670464853:RT=1670464853:S=ALNI_Ma2g4MkGu5tmS_XjK8nOUCO3pggkQ; __gpi=UID=0000090da371fd65:T=1670464853:RT=1676676247:S=ALNI_MbHkmQAl5KXjIXR8ZptsusICu1nxg; advanced_ads_pro_visitor_referrer=%7B%22expires%22%3A1702000882%2C%22data%22%3A%22https%3A%2F%2Fredacted.org%2F%22%7D; wp-settings-443=editor%3Dtinymce%26ampm9%3Do%26ampm8%3Do%26ampm6%3Do%26ampm5%3Do%26ampm10%3Do%26ampm1%3Dc%26ampm4%3Do%26amplibraryContent%3Dbrowse%26ampimgsize%3Dlarge%26ampurlbutton%3Dnone%26amphidetb%3D1%26ampalign%3Dcenter%26ampposts_list_mode%3Dlist%26ampwidgets_access%3Doff%26posts_list_mode%3Dlist%26libraryContent%3Dbrowse%26imgsize%3Dlarge; wp-settings-time-443=1670877147; __stripe_mid=3493215c-ddf4-4036-8f33-fb9aee928d9df9305b; wordpress_test_cookie=WP%20Cookie%20check; _gid=GA1.2.1395441887.1676676246; wordpress_logged_in_aa008fe4c7947530ea7a9cc9f17330a5=bozz%7C1676849151%7CzXQpPF4CKeUvKV7sW4qB0UjS6LmkajdjcByZ26pE5cM%7C5878189fdedc072b6512fe663584c64e5c99bddc04c0b3c09f42dc74f247c022
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
TE: trailers

response

HTTP/3 200 OK
date: Fri, 17 Feb 2023 23:52:23 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding, Origin,Accept-Encoding,User-Agent
cf-edge-cache: no-cache
access-control-allow-origin: https://redacted.org
access-control-allow-credentials: true
x-robots-tag: noindex
x-content-type-options: nosniff
referrer-policy: strict-origin-when-cross-origin
x-frame-options: SAMEORIGIN
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-cache, must-revalidate, max-age=0
x-qm-overview-time_taken: 28.1236
x-qm-overview-time_usage: 46.9% of 60s limit
x-qm-overview-memory: 27,032 kB
x-qm-overview-memory_usage: 2.6% of 1,048,576 kB limit
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=sMNCcMy9In92iv5QwlYrL3CLOsML5Y8fAw3pohLoEJzqsgI%2FAyRIzrYTryUvHhfFws0SYRynw6vlCAgLExFze5Y17j4zx7KLG5xF09I8Hgd9s2q51gOIdp%2BX%2FAtf0gtnawY%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 79b26f0c28c4ebea-SEA
content-encoding: gzip
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

[bozzmedia]

Additional reports/related:
https://wordpress.org/support/topic/post-editor-slow-to-completely-load-categories-tags-etc-slow-to-appear/
https://wordpress.stackexchange.com/questions/327160/number-of-categories-slowing-down-the-site
https://wordpress.stackexchange.com/questions/173358/category-page-and-edit-post-page-with-this-category-very-slow
https://wordpress.stackexchange.com/questions/130501/disable-inline-edit-on-edit-php
https://wordpress.stackexchange.com/questions/405867/speed-up-optimize-admin-edit-post-page-on-large-websites-50-150k-posts-50-cu (I have tried this last solution and it does help the edit pages, but not for quick edit on all posts)

[nickstarkloff]

Unfortunately, this is still a problem .. The editor takes a long time to display terms for custom taxonomies.

[T4ngml]

Same here. Real pain to work with :(

[Blastbeater]

same problem here, WordPress 6.3.1

[cherrygot-personal]

A quick hack I use for a taxonomy called postal_code so it returns up to 100000 terms per request:

add_filter('rest_postal_code_query', function ($prepared_args, $request) {
  if (strpos($request->get_route(), '/wp/v2/') === 0) {
    if ($prepared_args['number'] === 100) {
      $prepared_args['number'] = 100000;
    }
  }
  return $prepared_args;
}, 10, 2);

source: https://github.com/WordPress/gutenberg/pull/10762/files#diff-d0c65e7ac505d44d8571ddaffc510d98L412

This has worked for me. I had to load around 700+ terms for my taxonomy and they were being loaded in a batch of 100 terms each, 7 times. Setting the number to 1000 loaded them in a single request and they load quite fast, compared to previously. I'm not sure why this has not been marked as a solution. Is there a downside to it?

Btw, I've tested it with the gutenberg shipped with wordpress v6.1. So I'm sure it'll work for later versions too.

[simonefontana]

Same problem, I don't have a huge number of terms, but I have a custom post type with 8 taxonomies, and the loading is very slow besides consuming quite a bit of CPU. In my case, it cannot be solved by increasing the 'number' parameter because the issue is not the number of terms but rather the number of taxonomies.

Wp 6.4.3 - php 8.2

[apatwary12]

Same problem. I have about 5000 categories. I'm not understanding why we haven't made a single endpoint to load the all the categories via rest or some kind of pagination. Why is every category a separate fetch call? Saying that you have too many categories is irresponsible and dismissive considering the feature is an O(n) nightmare as it stands.

[bozzmedia]

I'm curious if this issue is on the core performance team's radar, since it's filed under Gutenberg but may be a Core issue, for example we have similar issues with taxonomies using Quick Edit via edit.php. I'll pop into the slack channel and check in.

[annezazu]

@WordPress/performance for awareness as this seems to continue to be an issue. @ellatrix as someone who has driven various performance efforts recently too. I see Bozzmedia pinged in #core-performance with this response:

Thanks for flagging this issue. Currently, the Core Performance team is primarily focused on performance issues that effect end users of the site and not as much performance issues that are encountered in the admin. Even so, it's possible that this demonstrates a potential issue effecting core APIs that needs to be fixed.

I'd love to see us perhaps work together here across the broader view of performance. Noting that there are current designs to evolve this flow too: #61852 I wonder if performance work can be done at the same time.

[bozzmedia]

Thanks @annezazu ! I want to add at least in our case post saving in general is much slower in the block editor (typically 15-45s) than vs classic. I'm not sure if it's connected to this issue or not. I am available to help with testing and further debugging as needed.

[annezazu]

Thank you! Any extra details helps, especially more up to date information with more recent releases. On my end for testing purposes, I tried with a site with 447 categories and didn't find any large amount of lag.

[bozzmedia]

I am encountering 24s load times when doing quick edits or saving posts, it's a little tricky to figure out what is happening that is taking that long since they're ajax requests and Query Monitor doesn't pick them up. I'll have to dig a little deeper on analyzing those load times on ajax requests, I'd welcome any suggestions.

I do note on quick edit of a post's title, the same delay occurs, so it's possible our issue is not limited to taxonomy. When I load edit-tags.php?taxonomy=category or make edits there, no delays.

Current stats: WP 6.5.4,
redis object cache
WordPress Memory Limit: 800M
Software: Apache
PHP Version: 8.0.30
PHP Memory Limit: 2000M
Cloudflare

Categories: 160
Tags: 210
Posts: 28,000
Comments: 550,000

Database Server

Database Management System: MySQL
Version: 5.7.44
Database Character Set: utf8
Database Collation: utf8_general_ci

[jhmonroe]

Seconding all of this and confirming that if I deactivate Rest API for certain post types, the Category/Tag AREAS load instantly in Classic editor. Have noticed similar issues where Featured Image, Category, and Tag areas all loads last in the WP Admin sidebar. Often 10-20 secs after the rest of the admin, content of the post, etc is all loaded. It seems to be something with Rest API and the order in which the admin loads.

In the video I posted here, you can see Categories and Tags loading way after (and also note that before they load, they do not show up as an option in Preferences
#55460 (comment) (< video is comment on this post)

The site overall has 18 categories and 28 tags for normal posts. Even in a CPT with 3 categories, this slow loading occurs. So really not the numbers that are the source of some of the delays mentioned above.

In the attached screenshot, you can see it's one of the last things to load on the page and takes many MS.

Although this is a "backend" issue, it really is front-facing in the sense that our clients see this. Ultimately it affects the entire ecosystem of designers, developers, etc who evangelize custom Wordpress sites to clients as a great product overall but a bug like this makes the client think Wordpress is hard to use when things don't appear sometimes or take a long time to appear.

[swissspidy]

The REST API requests themselves are quite fast on my local site. If they are slow for you, I'd suggest looking into profiling those requests with XHProf.

From what I read in this issue and see in my own testing, the main problem is that the editor only renders the categories panel after paginating through all the categories, which is slow if you have hundreds of those. And there's not even a spinner or anything. #29393 would solve this (and yes, maybe it can be solved as part of #61852). So I'd recommend focusing on that part first and foremost.

Thanks for flagging this issue. Currently, the Core Performance team is primarily focused on performance issues that affect end users of the site and not as much performance issues that are encountered in the admin. Even so, it's possible that this demonstrates a potential issue affecting core APIs that needs to be fixed.

FWIW I second this. The Gutenberg team is fare more experienced with the editor architecture and React performance best practices. It doesn't make sense for the core performance team to "take over", but of course we can help where possible.