Force a refresh of GraphQL data without restarting gatsby develop

[jeffreznik]

UPDATE

This feature has been available since before this issue was opened; read @pieh's reply below. Here it is again for those who are having a hard time finding it:

You can use ENABLE_GATSBY_REFRESH_ENDPOINT=1 env var to enable http://localhost:8000/__refresh POST webhook ( https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables )

Then it's up to you how you want to POST to http://localhost:8000/__refresh. The simplest way is to: curl -X POST http://localhost:8000/__refresh from a terminal.

Summary

Feature request: in development mode (gatsby develop), provide an ability to make Gatsby rerun source and transform plugins (e.g. gatsby-source-wordpress) to refresh data once it changes in source.

This may only apply to a subset of source plugins as I believe some already support this feature automatically when source changes (e.g. gatsby-source-filesystem).

I'm not sure what the best API for triggering Gatsby to do this would be. It could be an extension to http://localhost/:8000/___graphql that accepts a POST as a trigger to rerun these plugins. This would be a generic enough solution to allow us to write whatever "glue" we need to detect source changes on our own and tell Gatsby to go and retrieve updated source data (and transform it).

Basic example

In development, the http://localhost:8000/___graphql endpoint can be POSTed to, which triggers Gatsby to refresh all data in the graphql layer.

Motivation

Currently, the only way I've found to get new Wordpress data into Gatsby is to kill the development server and rerun gatsby develop. This is slow (~1 min) as it goes through the whole bootstrap and compilation process. Running the source/transform plugins is very quick and only a small part of the entire process, so it would be great to be able to run only this to speed up content-only changes.

[pieh]

You can use ENABLE_GATSBY_REFRESH_ENDPOINT=1 env var to enable http://localhost:8000/__refresh POST webhook ( https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables )

[jeffreznik]

Thank you @pieh!

This should be in more places in the documentation.

[vitordino]

just an idea:

there could be a button on the graphiql interface (at /__graphql) that does this trigger

[pieh]

@vitordino this is excellent idea! Do you mind creating feature request issue for it?

[DotFreelance]

You can use ENABLE_GATSBY_REFRESH_ENDPOINT=1 env var to enable http://localhost:8000/__refresh POST webhook ( https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables )

The Web Hook doesn't seem to work for me

As far as I can tell, the idea of the web hook does not work as documented. Unless I've missed something, using this does not actually refresh any (or all) source plugins.

I've tested this using the Wordpress source plugin. I have a page that is retrieved through: GraphiQL endpoint, a query in a page in Gatsby, and with Postman.

Setting the environment variable does work, and is present in process.env. ENABLE_GATSBY_REFRESH_ENDPOINT called from within Gatsby inside my page (a console log.)

Sending a POST request to http://localhost:8000/__refresh does successfully complete, although there is no response body (don't necessarily expect there to be.)

However, the change made and saved on Wordpress before using the refresh POST webhook does not show, not on the actual page, nor in GraphiQL. It does show in Postman. Stopping and restarting the development server does successfully refresh this data.

I've also tried using curl -X to make the post request, as per the documentation, but it doesn't appear to make any difference.

Should there even be caching on dev?

Frankly, it's a little unnerving that the development server of Gatsby caches data without an option to auto refresh on page request. I realize it's a lot to build and publish the Gatsby output, but it's absolutely necessary to make development not a chore. As it is, we have to build and publish the Gatsby output by restarting the development server now, anyway.

Development servers should never cache the source data. This is a core expectation for a front end framework. It's definitely the type of thing that would turn newer developers away from using Gatsby. Even experienced devs would be hesitant to use a framework that involved such excess effort any time some backend data changed. It almost seems like a slap in the face when the develop server has hot reload.

Anyway, I like a lot of what Gatsby does and think the documentation is really charming and nicely constructed, with well thought out instructions. Hopefully Gatsby can be a rock solid go-to for me in the future and will have the kind of workflow that rivals React, or at least gets close. Given Gatsby's other advantages, it would definitely keep Gatsby at the top of the choice list.

[leepowelldev]

@DotFreelance - I’m experiencing the same issue, don’t suppose you found any solution?

[DotFreelance]

No, unfortunately. I still use Gatsby, the workflow aside from this is incredible, it would just be nice to have the option to refresh the backend data on dev.

For now I’m just restarting the server when data changes.

[leepowelldev]

@DotFreelance If it helps I had some success in getting it to work - documented here #18040

[DmacTorstar]

@jeffreznik can we reopen this issue? This is quite important.

[leepowelldev]

@DmacTorstar - there is a PR happening here that might open up some conversation: #18802

[DmacTorstar]

I agree so strongly with @DotFreelance
The workflow with gatsby having to rebuild manually every time is such a chore.
The very least we could have would be a manual refresh button, and some big disclaimer in the docs so that people are not wondering why their data is not being re-built.
Perhaps in the cli tool, a refresh endpoint hit could be injected into the app.
Something to help the developers make this task easier.

[josephshambrook]

Been hitting this a lot as I've been scaffolding a new app. Not a deal breaker, but it's one of those things that could make the developer experience even better.

[saschb2b]

Ran into the same issue using sourcing from prismic. Only deleting the .cache folder and restarting gatsby develop showed the latest state.

[stonehz]

https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables

ENABLE_GATSBY_REFRESH_ENDPOINT=true yarn develop
and then you can simply run on another teminal
curl -X POST http://localhost:8000/__refresh
every time you want to refresh data without restarting the server 😄

[jeffreznik]

Thanks @stonehz, I've also updated the top comment so hopefully people will see it right away.

[DotFreelance]

https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables

ENABLE_GATSBY_REFRESH_ENDPOINT=true yarn develop
and then you can simply run on another teminal
curl -X POST http://localhost:8000/__refresh
every time you want to refresh data without restarting the server 😄

My comment explicity pointed out that this method doesn’t work.

[stonehz]

if you extend createPages it won't, true!

meaning that custom page creation is not handled cause it only refreshes the data, not the generated html..

It worth adding the above to clarify the limitation of the flag..

[jeffreznik]

@DotFreelance don't know what to say, I've been using it since May 29 and it works fine for me. What source plugins are you using? Is your gatsby version up to date?
Edit: I reread your comment, looks like you are using the Wordpress plugin. Weird, so am I and it works. Try this on a fresh gatsby-starter-wordpress and you'll see it works.

[jeffreznik]

@stonehz It should work even if you use the createPages API. I've tried this on a new gatsby-starter-wordpress copy (against a local Wordpress install); added a new page, hit /__refresh and the page appeared in the navigation.

[kmcaloon]

@jeffreznik thanks. It's actually working beautifully for me right now. I'm sourcing my data in a very custom way and mostly leveraging createPages. Everything is updating as it should extremely quickly.

[sriinnu]

Instead of a complete source-filesystem refresh, is there a way to just refresh only that specific page or menu that has been updated?

[emaildano]

Not sure if this works for every but it was handy for my setup. I created a script in package.json after enabling force refresh. If you run npm run fresh it wil curl / POST the refresh url.

"scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "clean": "rm -rf public && rm -rf .cache",
    "format": "./node_modules/prettier/bin-prettier.js --write \"**/*.{js,jsx,json,md}\"",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing \"",
    "refresh": "curl -X POST http://localhost:8000/__refresh"
  },

[offminded]

I have experienced a similar issue today but in my case I was using gatsby-source-graphql and creating pages programmatically via createPagesStatefully though as it has been explained here Gatsby does not rebuild page-data.json if you use createPagesStatefully.

So I have switched to createPages instead of createPagesStatefully at gatsby-config.js and finally with the /__refresh hook it is re-creating the page-data.json.

[justcodejs]

ENABLE_GATSBY_REFRESH_ENDPOINT=true yarn develop
and then you can simply run on another teminal
curl -X POST http://localhost:8000/__refresh

This work for me! For those who say it is not working you might want to take note the following thing.

1. The http://localhost:8000/__refresh only work on HTTP Post and not HTTP Get. This mean that you can't use any browser to access the URL directly. You need to use curl -X POST or tool like Postman to submit a HTTP Post request to http://localhost:8000/__refresh.

2. If you are using Strapi with MongoDB as your GraphQL source, please take noted that the record you created before the schema change will still have the old schema + new schema field. For example, in Strapi you have a collection call Book and it have the following fields.

{
    id: 'xxxxxxxxx',
    name: 'xxxx',
    price: xxx
}

After the collection is created you had insert a few records in this collection.

Then you modify the collection schema by change the price field name to unitPrice and add a new field called ISBN as shown below.

{
    id: 'xxxxxxxxx',
    name: 'xxxx',
    unitPrice: xxx,
    isbn: 'xxxxxxxxxxx'
}

After the schema changed, you also add in a few records based on the new schema.

Now, even you had successfully call the http://localhost:8000/__refresh your final schema appear in Gatsby will be:

{
    id: 'xxxxxxxxx',
    name: 'xxxx',
    price: xxx
    unitPrice: xxx,
    isbn: 'xxxxxxxxxxx'
}

To solve this issue, you need to edit or delete the old records in the monboDB directly.

One additional thing to take note is that, if there is a field in the collection is optional field, and all record in the collection didn't have the optional field fill in, that particular field will not appear in Gatsby graphql schema.

The second issue had wasted me hours to find out. Hope my post will help someone who face the same issue as me. Good luck!

[DotFreelance]

Hi there, it's been a while but I wanted to come back and apply some things I've discovered after making this work.

1. If you have this setup correctly, when you perform a POST request to http://localhost:8000/__refresh you'll actually see your Gatsby server perform a rebuild. The CLI will show the stages step by step as it did the first time you did yarn develop or gatsby develop.

2. The key problem for me was that the environment variables weren't being accepted correctly. While it did show that the variable was set correctly when I logged from process.env.ENABLE_GATSBY_REFRESH_ENDPOINT, I ended up adding config from the Gatsby documentation to require dotenv. I got that from the updated documents, here: https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables

3. a. You can setup the environment variables to work using the above method ENABLE_GATSBY_REFRESH_ENDPOINT=true yarn develop,
   b. or by adding a env.develop file to your root path in your Gatsby project with ENABLE_GATSBY_REFRESH_ENDPOINT=true.

   c. You could also potentially modify package.json to include ENABLE_GATSBY_REFRESH_ENDPOINT=true for the yarn script develop, making it ENABLE_GATSBY_REFRESH_ENDPOINT=true gatsby develop.

Hopefully this helps people figure out how to get the refresh working.

[machineghost]

It would be helpful if, when ENABLE_GATSBY_REFRESH_ENDPOINT=true, an extra log line in the build appeared.

In other words, if refresh was turned on (successfully), you'd see something like this (with the new part at the bottom):

    You can now view yoursite in the browser.
⠀
      http://localhost:8000/
⠀
    View GraphiQL, an in-browser IDE, to explore your site's data and schema
⠀
      http://localhost:8000/___graphql

    You have enabled the refresh feature.  To use it make a POST request at:

      http://localhost:8000/__refresh

And as a side note, a pox on whoever decided to use a POST for this instead of a GET ;-) REST principles are great, but "do what makes sense for the user" should always trump (and it'd be much easier for Gatsby devs if they could refresh their site using their browser also).

[kyr0]

I've developed a Chrome Extension for this issue to be solved easily. With this installed, you only have to press Ctrl/Cmd + Shift + Y and Gatsby will gracefully reload without restart :) Have fun!

https://chrome.google.com/webstore/detail/gatsby-refresher/npdbjognnchpbocdpddhekiggpklcgpd

If you love this, please add a rating and comment on the Chrome Webstore :)

Thank you

[machineghost]

That's awesome kyr0, I'll check it out. But for the sake of the project maintainers, allow me to say what should already be obvious: it's problematic that a Javascript library all but requires its own browser extension.

Kyr0 could have saved of all the time and effort they put into that extension, if a Gatsby dev had prioritized empathizing with Gatsby user needs over dogmatic adherence to REST, and changed just four simple characters (POST => GET).

[strategore]

I am happy there is a way of doing this, but I have to admit it's counterintuitive and probably unnecessarily complicated. Why is a POST necessary instead of a GET?

[ehubbell]

This doesn't work for me.

I've added ENABLE_GATSBY_REFRESH_ENDPOINT=1 to my .env.development file, cleared cache, restarted the server, and then added an NPM script that calls curl -X POST http://localhost:4200/__refresh and I get the following error:

{"error":"Refresh endpoint is not enabled. Run gatsby with \"ENABLE_GATSBY_REFRESH_ENDPOINT=true\" environment variable set.","isEnabled":false}%