Inconsistent "cache hit"

[iDVB]

What version of Turborepo are you using?

1.4.7

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Mac

Describe the Bug

Even with a very minimal turbo.json one of our workspaces is a cache miss intermittently.
We're running the same command over and over and able to see it cache HIT and MISS intermittently. 🤷‍♂️

We're very much dead in the water on this. We've even tried running -vvv to see if there is anything helpful in the debug logs, but there really isn't anything interesting other then a new hash each time there is a miss, which would be expected.
Just need to know WHY it's a miss.

I would be very happy to get on a zoom and show a core team member whats happening.
https://www.screencast.com/t/8ptO2RVhb4A

Expected Behavior

No matter how many times I run the build command, it should continue to be a 'cache hit` since my inputs never change.

To Reproduce

use this turbo.json

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "outputs": [
        "dist/**"
      ]
    },
    "dev": {
      "cache": false
    }
  }
}

• Run pnpm turbo run build --filter=react-components
• Run the previous command over and over and it will eventually cache miss even though nothing has changed.

[gsoltis]

Is it possible one of the tasks that you're building is writing to another package?

If you can publish a minimal reproduction or otherwise find a way to share the repository, we can definitely dig into it.

Also, this is definitely an area of improvement for turbo, we are working on some tools to help surface the files that go into the hash calculation.

[nathanhammond]

Also, is it intermittent, or is it alternating? The video shows alternating, but I don't know if that's normal.

[iDVB]

Thanks @gsoltis going to be tough to get you a reproduction repo.
but I can zoom or give access to our monorepo to a core team member.
I have VSCode showing me Git changes in the files in the affected workspace and there are no detected changes at any point. I'm also assuming that the default hashing happens on the ALL the files in the workspace minus the ones in .gitignore?

@nathanhammond it seems completely random. You can get the cache to HIT 10-30x times in a row and then suddenly MISS for 4-5 times, then back and forth between HIT and MISS. 🤷

[nathanhammond]

If you're willing to share access that will give us the best opportunity to take a look; we're going to run builds with a debugger attached until it reproduces. (Any of @nathanhammond, @tknickman, @gsoltis can track this down.)

Otherwise we'll need to get a dev environment set up on your box.

[iDVB]

@nathanhammond @tknickman, @gsoltis if any of you want to reach out to me on discord to chat directly... iDVB

I've added you all to our monorepo.
https://github.com/KlickMarketing/brandx-mono/tree/dvb-turbo

Just a thought...
We CAN reproduce even just building react-components > core of our deps.
However, there are two sites/workspaces that depend on react-components that use this....

  "dependenciesMeta": {
    "@klickmarketing/react-components": {
      "injected": true
    }
  }

I wonder if that is causing something.

Also, would Turbo hash break if a files contents are identical but the modified date changes? perhaps there is something like this happening which is causing the hash to break while still leaving git as "no changes"?

Things we've tried: (with each try we delete all npm_modules dirs and rerun pnpm install)

• tried removing dependenciesMeta
• tried adding inputs: ["src/**"] to turbo.json
• tried changing all references to workspaces from workspace:* to *

Still random intermittent cache hit and misses.

[thenbe]

I came across this bug when I upgraded to 1.4.7. For the sake of troubleshooting, @iDVB does the bug still persist if you change your turbo.json to the following? In my case, this turbo.json would still cause random cache misses like you describe them.

• Remove all pipelines except build
• Keep only the build.outputs and build.dependsOn keys as-is, remove the rest build.*
• Add a build.inputs with some random characters
• Run the same command in the same format multiple times pnpm turbo run build --filter=@kickmarketing/react-components
• Example:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": [
        "^build"
      ],
      "inputs": [
        "asdjfsdklfjlkj"
      ],
      "outputs": [
        "build/**",
      ],
    },
  }
}

I've since pinned turbo to 1.4.6 and things work as expected again.

[thenbe]

Tried giving it another go yesterday and while I couldn't really track down the cause (partly because of the randomness and partly because I am not that familiar with turbo/pnpm/etc) I did notice something interesting. After uninstalling eslint (and it's related packages eslint-*, @typescript-eslint/*) in a totally unrelated package residing in the same monorepo, caching was consistent again.

Might this be related to #2029?

[gsoltis]

I've identified a regression (fixed in #2054 ) that can cause cache hits and misses to happen randomly, more likely with more environment variables.

Not 100% sure that accounts for everything here, but lets try again once that's merged and go from there.

[gsoltis]

Can you try with 1.5.3?

[unional]

I'm using 1.5.3 and still have the same issue.

What I have found is, from the console:

pnpm turbo run build --filter @pats/crypto 
• Packages in scope: @pats/crypto
• Running build in 1 packages
@pats/repo-scripts:build: cache hit, replaying output 6baf85263f19cde0
@pats/repo-scripts:build: 
@pats/repo-scripts:build: > @pats/repo-scripts@0.4.1 build /Users/hwong/pan/pats/tools/repo-scripts
@pats/repo-scripts:build: > tsc
@pats/repo-scripts:build: 
@pats/crypto:build: cache miss, executing 0994a82a6ce85f78
@pats/crypto:build: 
@pats/crypto:build: > @pats/crypto@1.0.0 build /Users/hwong/pan/pats/libraries/crypto
@pats/crypto:build: > tsc
@pats/crypto:build: 

 Tasks:    2 successful, 2 total
Cached:    1 cached, 2 total
  Time:    1.278s 

But the .turbo/turbo-build.log shows cache hit:

�[32m@pats/crypto:build�[0m: cache hit, replaying output �[2m0994a82a6ce85f78�[0m
�[32m@pats/crypto:build: �[0m
�[32m@pats/crypto:build: �[0m> @pats/crypto@1.0.0 build /Users/hwong/pan/pats/libraries/crypto
�[32m@pats/crypto:build: �[0m> tsc
�[32m@pats/crypto:build: �[0m

Furthermore, when it gets a cache miss, I notice the .turbo/turbo-build.log was cleared and then updated with cache hit logs.

But each time the hash are different.

[unional]

I can also confirm that it is working correctly with 1.4.6

[Siwon-medi]

I also found a bug in version 1.4.7 and later.
In 1.4.6 it works fine.

[gsoltis]

We had a bug specifically with pnpm lockfile resolution. It's fixed in 1.5.4. Can you give that a try, and if there's still an issue here I can reopen?

[thenbe]

@gsoltis 1.5.4 has fixed the issue for me, thanks!

[Joroze]

Hey @gsoltis, this issue occurs again above v1.5.4. I'm using Vite v3.1.8 + NPM v8.11.0 (Node v16)

EDIT: The issue occurs above v1.5.6, actually.

[nathanhammond]

@Joroze Can you please try 1.6.0? Also, if it still exists after upgrading please file a new issue.

[Joroze]

@Joroze Can you please try 1.6.0? Also, if it still exists after upgrading please file a new issue.

Hi @nathanhammond, unfortunately I do not have the bandwidth to report the issue in full detail, but I tested with 1.6.0 which does not work.

If I have time later I can look into reporting it as a regression

For anyone who comes across this, here are my project details in general:

• Vite (with react plugin) 3.1.8
• NPM v8.11.0 (Node v16)

I'll stay on 1.5.6 for now.

[Joroze]

@nathanhammond FYI that it occurs with Vite Plugin React using TurboRepo v1.5.6/v1.5.7 tags