[gatsby-source-wordpress] Blogs with a lot of media gets stuck at sourcing

[Jonic]

Description

With the plugin configured to search for media, my gatsby develop command appeared unresponsive at the stage where media is downloaded and indexed. The blog in question has ~6700 images, which all appeared to download (although some incompletely), but my terminal didn't move past the source and transform nodes step. I left it about 90 minutes, and it didn't actually get to the point where I could load the site.

Steps to reproduce

• Have blog with ~6700 images
• Attempt to start Gatsby development server with plugin configured to fetch media

Expected result

• Images download, server starts

Actual result

• Images downloaded (some appeared incomplete), source and transform nodes doesn't appear to complete

Environment

• Gatsby version (npm list gatsby): gatsby@1.9.256
• gatsby-cli version (gatsby --version): n/a - using local version
• Node.js version: 9.4.0
• Operating System: macOS High Sierra - 10.13.4

I've managed to get around this issue with a custom WordPress endpoint, and then excluding any API paths that begin with wp/**, but I thought I'd raise this as an issue anyway. It would be handy to be able to catalogue the images without downloading them, so our production app can reference the images from their original URL.

Anyone else had issues with blogs containing large sets of media?

[m-allanson]

There's a few related issues and PRs which might be useful, have a look at #4293, #4666, #4616 and #4884.

[lightstrike]

@Jonic have you tried setting/adjusting the concurrentRequests option? I have a site that has about ~5,500 media files that I can get working with the built in WP endpoints by setting concurrentRequests to around 10. The source and transform does take a really long time, usually between 9 and 12 minutes. Hopefully will get faster in v2.

[Jonic]

Thanks @m-allanson - I'd taken a look at a couple of those, but still couldn't get my develop server started up! I also didn't check out the concurrentRequests as @lightstrike suggested. I think for now we'll be happy with the custom API endpoint, but I'll definitely be checking this out again when v2 lands :)

I'm going to close this, since the issue numbers previously mentioned in the thread address the same issue I've been having. Thanks for your help, both! :)

[scruffydan]

I have this exact same issue. I tried with concurrentRequests set to 10 and that seems to have no effect. Has anyone found a solution?

Thanks in advance.

[lightstrike]

@scruffydan Another thing that's helped me is using the WP REST API Cache plugin -- I've found it dramatically increases performance on the WP side.

[scruffydan]

Thanks for the suggestion

Unfortunately it didn't solve my issues, I sometimes get the following error when I run gatsby develop

This is what I get:

=END PLUGIN=====================================: 33804.250ms
error Plugin gatsby-source-wordpress returned an error


  TypeError: Cannot read property 'id' of undefined

  - normalize.js:275
    [MindofDan]/[gatsby-source-wordpress]/normalize.js:275:9

  - Array.map

  - normalize.js:270 Object.exports.mapElementsToParent
    [MindofDan]/[gatsby-source-wordpress]/normalize.js:270:19

  - gatsby-node.js:142 _callee$
    [MindofDan]/[gatsby-source-wordpress]/gatsby-node.js:142:34

  - next_tick.js:61 process._tickCallback
    internal/process/next_tick.js:61:11


⠠ source and transform nodes

I am using verboseOutput: true . It then just hangs at source and transform nodes and never completes. I have left it running for 12 hours.

Hoever I don't always get the TypeError: Cannot read property 'id' of undefined error, sometimes it just hangs at source and transform nodes.

I have published my gatsby project to github here: https://github.com/scruffydan/MindofDan

If anyone can point me in the right direction I would appreciate it.

Thanks in advance

[scruffydan]

I just realized this issue has been marked closed, I have opened a new issue: #5371

[aman-developer]

I am getting same issue. i downloaded v2 directory i.e /www from here
https://github.com/gatsbyjs/gatsby

source and transform nodes never end. i waited for 6 hours.

[Polyhistor]

I have the exact same issue, it takes forever.... anyone figured out something ? there is another github issue here, I've sent my repo for a fellow to check out... please do let me know if there are any updates, i'm having a deadline soon.

[shoyebinamdar]

I am also facing the same issue. Has anybody got to any solution yet?