Weird caching issues

[nehalist]

Summary

My blog has a weird caching bug, and I'm running out of ideas on how to fix it. After publishing (or changing/deleting) a post my blog is rebuild via GitHub actions and rsync'd to my server. When the rebuild is done a very weird behavior occurs:

When heading to my blog I can see my new post for a very brief moment, but it disappears within a second afterwards.

Relevant information

• Site url: https://nehalist.io
• The blog is built on GitHub and rsync'd to my server:

// workflow.yml
name: Update blog

on: [push, repository_dispatch]

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1

      - name: Set Node.js
        uses: actions/setup-node@master
        with:
          node-version: 10.x

      - name: Install dependencies
        run: yarn install --prod --pure-lockfile

      - name: Lint
        run: yarn run lint

      - name: Build
        run: yarn run build
        env:
          GHOST_API_URL: "https://ghost.nehalist.io"
          GHOST_CONTENT_API_KEY: ${{ secrets.ghost_content_api_key }}
          GATSBY_ENV: "prod"
          GATSBY_COMMENTO_URL: "https://comments.nehalist.io"
          GA_TRACKING_ID: "UA-40873708-1"
          GATSBY_ALGOLIA_APP_ID: ${{ secrets.algolia_app_id }}
          GATSBY_ALGOLIA_SEARCH_KEY: ${{ secrets.algolia_search_key }}
          ALGOLIA_ADMIN_KEY: ${{ secrets.algolia_admin_key }}

      - name: Deploy
        run: docker-compose run deploy
        if: github.ref == 'refs/heads/master'
        env:
          DEPLOY_KEY: ${{ secrets.deploy_key }}

// docker file
version: '3'

services:
  deploy:
    image: "instrumentisto/rsync-ssh"
    volumes:
      - .:/home/site
    working_dir: /home/site
    environment:
      ref: $GITHUB_REF
      deploy_key: $DEPLOY_KEY
    command: sh scripts/deploy.sh

// deploy script
#!/bin/sh

dir="$(echo "${ref}" | cut -d'/' -f3)"

set -eu

mkdir -p $HOME/.ssh
echo "${deploy_key}" > $HOME/.ssh/deploy_key
chmod 600 $HOME/.ssh/deploy_key

target="~/html"

sh -c "rsync -azh -e 'ssh -i $HOME/.ssh/deploy_key -o StrictHostKeyChecking=no' public/ --rsync-path='mkdir -p ${target} && rsync' <user>@<server>:${target} --delete"

rm $HOME/.ssh/deploy_key

• I do have a custom static/.htaccess file with the following contents:

ErrorDocument 404 /404/index.html

<IfModule mod_headers.c>
    <FilesMatch "\.html$">
        Header set Cache-Control "public, max-age=0, must-revalidate"
    </FilesMatch>
    <FilesMatch "\.(js|css)$">
        Header set Cache-Control "public, max-age=31536000, immutable"
    </FilesMatch>
    <Files sw.js>
        Header set Cache-Control "public, max-age=0, must-revalidate"
    </Files>
</IfModule>

• When I reload with open dev tools (or with ctrl + shift + r - or, in other words, reload the cache) everything works and the new posts appear
• I used https://www.gatsbyjs.org/packages/gatsby-plugin-offline/ some time ago, but removed it and added the gatsby-plugin-remove-serviceworker plugin instead

Environment (if relevant)

  System:
    OS: Linux 4.15 Ubuntu 18.04.3 LTS (Bionic Beaver)
    CPU: (16) x64 AMD Ryzen 7 1700X Eight-Core Processor
    Shell: 5.4.2 - /usr/bin/zsh
  Binaries:
    Node: 10.18.1 - /usr/bin/node
    Yarn: 1.21.1 - /usr/bin/yarn
    npm: 6.13.4 - /usr/bin/npm
  Languages:
    Python: 2.7.17 - /usr/bin/python
  Browsers:
    Chrome: 79.0.3945.130
    Firefox: 73.0
  npmPackages:
    gatsby: ^2.19.1 => 2.19.8
    gatsby-image: ^2.2.39 => 2.2.39
    gatsby-plugin-advanced-sitemap: ^1.4.5 => 1.4.5
    gatsby-plugin-algolia: ^0.5.0 => 0.5.0
    gatsby-plugin-feed: ^2.3.26 => 2.3.26
    gatsby-plugin-google-analytics: ^2.1.33 => 2.1.34
    gatsby-plugin-mailchimp: ^5.1.2 => 5.1.2
    gatsby-plugin-manifest: ^2.2.37 => 2.2.39
    gatsby-plugin-offline: ^3.0.32 => 3.0.32
    gatsby-plugin-react-helmet: ^3.1.21 => 3.1.21
    gatsby-plugin-remove-serviceworker: ^1.0.0 => 1.0.0
    gatsby-plugin-sass: ^2.1.27 => 2.1.27
    gatsby-plugin-sharp: ^2.4.0 => 2.4.3
    gatsby-plugin-twitter: ^2.1.18 => 2.1.18
    gatsby-plugin-typescript: ^2.1.26 => 2.1.26
    gatsby-remark-prismjs: ^3.3.30 => 3.3.30
    gatsby-source-filesystem: ^2.1.46 => 2.1.46
    gatsby-source-ghost: ^3.5.1 => 3.5.1
    gatsby-transformer-remark: ^2.6.48 => 2.6.48
    gatsby-transformer-sharp: ^2.3.13 => 2.3.13
    gatsby-transformer-yaml: ^2.2.23 => 2.2.23
  npmGlobalPackages:
    gatsby-cli: 2.5.4

File contents (if changed)

gatsby-config.js:

const generateFeed = require('./src/utils/Helper.js');
const config = require('./src/utils/Config.js');
const queries = require('./src/utils/algolia');

require("dotenv").config({
  path: `.env`,
});

const plugins = [
  {
    resolve: `gatsby-plugin-algolia`,
    options: {
      appId: process.env.GATSBY_ALGOLIA_APP_ID,
      apiKey: process.env.ALGOLIA_ADMIN_KEY,
      queries,
      chunkSize: 10000
    }
  },
  {
    resolve: `gatsby-plugin-mailchimp`,
    options: {
      endpoint: `<url>`
    }
  },
  `gatsby-plugin-sass`,
  `gatsby-plugin-typescript`,
  `gatsby-plugin-react-helmet`,
  {
    resolve: `gatsby-source-filesystem`,
    options: {
      name: `images`,
      path: `${__dirname}/src/images`
    }
  },
  `gatsby-transformer-sharp`,
  `gatsby-plugin-sharp`,
  {
    resolve: `gatsby-plugin-manifest`,
    options: {
      name: `nehalist.io Blog`,
      short_name: `nehalist.io`,
      start_url: `/`,
      background_color: `#42566a`,
      theme_color: `#42566a`,
      display: `minimal-ui`,
      icon: `src/images/logo.svg` // This path is relative to the root of the site.
    }
  },
  {
    resolve: `gatsby-source-ghost`,
    options: {
      apiUrl: process.env.GHOST_API_URL,
      contentApiKey: process.env.GHOST_CONTENT_API_KEY
    }
  },
  `gatsby-plugin-twitter`,
  `gatsby-plugin-remove-serviceworker`,
  {
    resolve: `gatsby-plugin-advanced-sitemap`,
    options: {
      query: `
          {
            allGhostPost {
              edges {
                node {
                  id
                  slug
                  updated_at
                  created_at
                  feature_image
                }
              }
            }
            allGhostPage {
              edges {
                node {
                  id
                  slug
                  updated_at
                  created_at
                  feature_image
                }
              }
            }
            allGhostTag {
              edges {
                node {
                  id
                  slug
                  feature_image
                }
              }
            }
            allGhostAuthor {
              edges {
                node {
                  id
                  slug
                  profile_image
                }
              }
            }
          }`,
      mapping: {
        allGhostPost: {
          sitemap: `posts`
        },
        allGhostTag: {
          sitemap: `tags`
        },
        allGhostPage: {
          sitemap: `pages`
        }
      },
      exclude: [
        `/dev-404-page`,
        `/404`,
        `/404.html`,
        `/offline-plugin-app-shell-fallback`,
        `/status`
      ]
    }
  },
  {
    resolve: `gatsby-plugin-feed`,
    options: {
      query: `
        {
          allGhostSettings {
            edges {
              node {
                title
                description
              }
            }
          }
        }
      `,
      feeds: [
        generateFeed(config),
      ],
    }
  }
];

if (process.env.GA_TRACKING_ID) {
  plugins.push({
    resolve: `gatsby-plugin-google-analytics`,
    options: {
      trackingId: process.env.GA_TRACKING_ID,
      anonymize: true,
    }
  });
}

module.exports = {
  siteMetadata: {
    siteUrl: 'https://nehalist.io',
    title: 'nehalist.io',
    description: 'A blog about software development, technology and all that kind of geeky stuff.',
    author: `@nehalist`
  },
  plugins
};

package.json:

{
  "name": "nehalist.io",
  "private": true,
  "description": "Theme of nehalist.io",
  "version": "1.2.0",
  "author": {
    "name": "Kevin Hirczy",
    "email": "kevin@nehalist.io",
    "url": "https://nehalist.io"
  },
  "dependencies": {
    "@types/prismjs": "^1.16.0",
    "@types/react-helmet": "^5.0.15",
    "algoliasearch": "^3.35.1",
    "babel-plugin-prismjs": "^1.1.1",
    "cheerio": "^1.0.0-rc.3",
    "gatsby": "^2.19.1",
    "gatsby-image": "^2.2.39",
    "gatsby-plugin-advanced-sitemap": "^1.4.5",
    "gatsby-plugin-algolia": "^0.5.0",
    "gatsby-plugin-feed": "^2.3.26",
    "gatsby-plugin-google-analytics": "^2.1.33",
    "gatsby-plugin-mailchimp": "^5.1.2",
    "gatsby-plugin-manifest": "^2.2.37",
    "gatsby-plugin-offline": "^3.0.32",
    "gatsby-plugin-react-helmet": "^3.1.21",
    "gatsby-plugin-remove-serviceworker": "^1.0.0",
    "gatsby-plugin-sass": "^2.1.27",
    "gatsby-plugin-sharp": "^2.4.0",
    "gatsby-plugin-twitter": "^2.1.18",
    "gatsby-plugin-typescript": "^2.1.26",
    "gatsby-remark-prismjs": "^3.3.30",
    "gatsby-source-filesystem": "^2.1.46",
    "gatsby-source-ghost": "^3.5.1",
    "gatsby-transformer-remark": "^2.6.48",
    "gatsby-transformer-sharp": "^2.3.13",
    "gatsby-transformer-yaml": "^2.2.23",
    "node-sass": "^4.13.1",
    "normalize.css": "^8.0.1",
    "prismjs": "^1.19.0",
    "react": "^16.12.0",
    "react-carbon": "^1.0.6",
    "react-dom": "^16.12.0",
    "react-helmet": "^5.2.0",
    "react-icons": "^3.8.0",
    "react-instantsearch-dom": "^6.2.0",
    "react-typed": "^1.2.0",
    "smooth-scroll": "^16.1.2",
    "styled-normalize": "^8.0.6",
    "tocbot": "^4.10.0",
    "tslint": "^5.20.1",
    "tslint-react": "^4.2.0",
    "typed.js": "^2.0.11"
  },
  "devDependencies": {
    "babel-preset-gatsby": "^0.2.27",
    "prettier": "^1.19.1",
    "typescript": "^3.7.5"
  },
  "license": "SEE LICENSE",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --write src/**/*.{js,jsx}",
    "start": "npm run develop",
    "serve": "gatsby serve",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing\"",
    "type-check": "tsc",
    "lint": "tslint -c tslint.json 'src/**/*.{ts,tsx}'"
  }
}

gatsby-node.js:

const path = require('path');

exports.createPages = ({ graphql, actions }) => {
  const { createPage } = actions;

  /**
   * Posts
   */
  const createPosts = new Promise((resolve, reject) => {
    const postTemplate = path.resolve(`./src/templates/post.tsx`);
    resolve(
      graphql(`
        {
          allGhostPost(
            sort: {order: ASC, fields: published_at}
          ) {
            edges {
              node {
                slug
                primary_tag {
                  slug
                }
              }
            }
          }
        }`
      ).then((result) => {
        if (result.errors) {
          return reject(result.errors)
        }

        if (!result.data.allGhostPost) {
          return resolve()
        }

        const items = result.data.allGhostPost.edges;

        items.forEach(({ node }) => {
          // This part here defines, that our posts will use
          // a `/:slug/` permalink.
          node.url = `/${node.slug}/`;

          createPage({
            path: node.url,
            component: path.resolve(postTemplate),
            context: {
              // Data passed to context is available
              // in page queries as GraphQL variables.
              slug: node.slug,
              primaryTagSlug: node.primary_tag.slug
            }
          })
        });

        return resolve();
      })
    )
  });

  /**
   * Tags
   */
  const createTags = new Promise((resolve, reject) => {
    const tagsTemplate = path.resolve(`./src/templates/tag.tsx`);
    resolve(
      graphql(`
        {
          allGhostTag(
            sort: {order: ASC, fields: name}
          ) {
            edges {
              node {
                slug
                url
              }
            }
          }
        }`
      ).then((result) => {
        if (result.errors) {
          return reject(result.errors)
        }

        if (!result.data.allGhostTag) {
          return resolve()
        }

        const items = result.data.allGhostTag.edges;

        items.forEach(({ node }) => {
          // This part here defines, that our tag pages will use
          // a `/tag/:slug/` permalink.
          node.url = `/tag/${node.slug}/`;

          createPage({
            path: node.url,
            component: path.resolve(tagsTemplate),
            context: {
              // Data passed to context is available
              // in page queries as GraphQL variables.
              slug: node.slug
            }
          });
        });

        return resolve();
      })
    )
  });

  /**
   * Pages
   */
  const createPages = new Promise((resolve, reject) => {
    const pageTemplate = path.resolve(`./src/templates/page.tsx`);
    resolve(
      graphql(`
        {
          allGhostPage(
            sort: {order: ASC, fields: published_at}
          ) {
            edges {
              node {
                slug
                url
              }
            }
          }
        }`
      ).then((result) => {
        if (result.errors) {
          return reject(result.errors)
        }

        if (!result.data.allGhostPage) {
          return resolve()
        }

        const items = result.data.allGhostPage.edges;

        items.forEach(({ node }) => {
          // This part here defines, that our posts will use
          // a `/:slug/` permalink.
          node.url = `/${node.slug}/`;

          createPage({
            path: node.url,
            component: path.resolve(pageTemplate),
            context: {
              // Data passed to context is available
              // in page queries as GraphQL variables.
              slug: node.slug,
            }
          })
        });

        return resolve();
      })
    )
  });

  return Promise.all([createPosts, createTags, createPages]);
};

gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[aychtang]

@nehalist - I think all cache headers have to be max-age=0, must-revalidate or you need to hack around it to enable caching - there is a bunch of discussion on this @ #15080.

[TylerBarnes]

@nehalist have you had a chance to look through the caching docs? https://www.gatsbyjs.org/docs/caching/

[nehalist]

Yes, my .htaccess is configured following the recommendation from the docs.

[LekoArts]

I don't see entries for the mentioned page data and app data?

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[nehalist]

    <Directory "public/page-data/">
        <FilesMatch "\.(json)">
            Header set Cache-Control "public, max-age=0, must-revalidate"
        </FilesMatch>
    </Directory>

Should do the trick in this case, right?

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜