Local cache not working in CI pipeline

[paulius-valiunas]

Describe the bug
Local cache is not working in my CI pipeline, even though I have skipLocalCache: false in my config.

To Reproduce
Steps to reproduce the behavior:

1. Create a lage.config.js with these options:

{
  pipeline: {
    build: {
      dependsOn: ["^build"],
      inputs: ["./src/**/*.*", "package.json"],
      outputs: ["./lib", "./build"],
    },
    test: ["build"],
    clean: {
      cache: false,
    },
  },
  cacheOptions: {
    // defaults to true in CI systems
    skipLocalCache: false,
  },
}

2. Create a CI pipeline that runs lage build && lage test in the same job/stage
3. Observe how lage runs the build command twice.

Expected behavior
Lage should skip through the build command and only run test in the second invocation since the packages are already built.

Environment/Troubleshooting
Happens in an Azure DevOps pipeline.

envinfo

  System:
    OS: Linux 5.15 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
    CPU: (2) x64 Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
    Memory: 5.70 GB / 6.77 GB
    Container: Yes
    Shell: 5.1.16 - /bin/bash
  Binaries:
    Node: 18.17.1 - /opt/hostedtoolcache/node/18.17.1/x64/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.6.7 - /opt/hostedtoolcache/node/18.17.1/x64/bin/npm
    pnpm: 8.7.0 - /opt/hostedtoolcache/node/18.17.1/x64/bin/pnpm
  Managers:
    Apt: 2.4.10 - /usr/bin/apt
    Cargo: 1.71.1 - ~/.cargo/bin/cargo
    Composer: 2.5.8 - /usr/bin/composer
    Gradle: 8.3 - /usr/bin/gradle
    Maven: 3.8.8 - /usr/bin/mvn
    pip3: 22.0.2 - /usr/bin/pip3
    RubyGems: 3.3.5 - /usr/bin/gem
  Utilities:
    Bazel: 6.3.2 - /usr/local/bin/bazel
    CMake: 3.27.3 - /usr/local/bin/cmake
    Make: 4.3 - /usr/bin/make
    GCC: 11.4.0 - /usr/bin/gcc
    Git: 2.41.0 - /usr/bin/git
    Clang: 14.0.0 - /usr/bin/clang
    Mercurial: 6.1.1 - /usr/bin/hg
    Subversion: 1.14.1 - /usr/bin/svn
    Curl: 7.81.0 - /usr/bin/curl
  Servers:
    Nginx: 1.18.0 - /usr/sbin/nginx
  Virtualization:
    Docker: 24.0.5 - /usr/bin/docker
  SDKs:
    Android SDK:
      API Levels: 27, 28, 29, 30, 31, 32, 33, 33, 33, 34
      Build Tools: 27.0.0, 27.0.1, 27.0.2, 27.0.3, 28.0.0, 28.0.1, 28.0.2, 28.0.3, 29.0.0, 29.0.1, 29.0.2, 29.0.3, 30.0.0, 30.0.1, 30.0.2, 30.0.3, 31.0.0, 32.0.0, 33.0.0, 33.0.1, 33.0.2, 34.0.0
      Android NDK: 25.2.9519653
  IDEs:
    Nano: 6.2 - /usr/bin/nano
    Vim: 8.2 - /usr/bin/vim
  Languages:
    Bash: 5.1.16 - /usr/bin/bash
    Go: 1.20.7 - /usr/bin/go
    Java: 11.0.20 - /usr/bin/javac
    Perl: 5.34.0 - /usr/bin/perl
    PHP: 8.1.2 - /usr/bin/php
    Python: 3.10.12 - /usr/bin/python
    Python3: 3.10.12 - /usr/bin/python3
    R: 4.3.1 - /usr/bin/R
    Ruby: 3.0.2 - /usr/bin/ruby
    Rust: 1.71.1 - /home/vsts/.cargo/bin/rustc
  Databases:
    MySQL: 0.22.04.1 - /usr/bin/mysql
    SQLite: 3.37.2 - /usr/bin/sqlite3
  Browsers:
    Chrome: 116.0.5845.96
    Chromium: 116.0.5845.0

[soumyart]

Facing the same issue. In CI environment node_modules/.cache/lage/cache is not generated.

[soumyart]

In CI environment writeRemoteCache is always set to true

https://github.com/microsoft/lage/blob/master/packages/scheduler/src/cache/createCacheProvider.ts#L37

[soumyart]

@paulius-valiunas - As a workaround, you can try re-setting CI env flag if you can do it.

[paulius-valiunas]

In CI environment writeRemoteCache is always set to true

master/packages/scheduler/src/cache/createCacheProvider.ts#L37

this issue is about the local cache, I don't think writeRemoteCache applies here.

[mvuckovic]

Same issue here.

It seems that runAction only considers CLI argument when determining whether to skipLocalCache. Config value is ignored.

The CLI --skip-local-cache argument doesn't accept false and in combination with isRunningFromCI will also result in skipping the local cache in CI environment.

The only alternative I found for now is to fake NODE_ENV=test when running scripts

[paulius-valiunas]

@kenotron could we get some attention on this?