--mount=type=cache fails silently, debugging is hard

[yebyen]

I spent a while with #1173 and then also found #1414 and #1237, I'd like to suggest there is a UX issue here.

For me, my issue was that ${VARIABLE} substitution in the --mount=type=cache,target=[...] appears to cause a silent failure. It looks like the cache mount does not succeed, but there is no hint about that anywhere in the buildkit output, and image creation progresses as if everything was fine.

Ideally I think you should be able to use the variable substitution there, like you can use it practically anywhere else in the dockerfile (the buildkit progress output actually seems to indicate the substitution was successful, spuriously) – that is a separate issue, and I'm less interested in fixing that, (also since sure, there might be good reasons why it doesn't work.)

But the issue I'd like to report is this silent failure. I suspect this feature is underused because people are running into silent failures and giving up. I spent four hours tracking down this issue, (and for me it was compounded by the fact I guess that between rvm, bundler, ... what else, I wasn't quite sure what location should be saved even to begin with.)

If the mount is not successful, the RUN command should abort with an error indicating that. If the uid,gid needed to be set and so writes in the cache are not successful, similarly the RUN should abort with an error indicating that.

Does anyone who knows that area of the buildkit code well enough to say why this happens have any interest in taking a look? I'd be glad to spend some time testing if there is any chance of a proposed fix to make this better.

[tonistiigi]

Variable substitution is tracked in #815

For the "silent error" you need to post a reproducer. My guess is that it just uses the raw variable name directly and no actual error appeared, just the value you were expecting was not there.

[kingdonb]

Thank you for tracking that down! I will post a working example reproducer below. If #815 is a long-standing known issue and can't be easily fixed right away, maybe we should check if the user has attempted to perform a variable substitution in a place where it is not allowed, and balk the build into failed state rather than failing silently (or not even failing) – this could stop many people from wasting time figuring out what went wrong.

The solution you proposed in #815 thread, of expanding only values, would definitely work for my use case.

My dockerfile that I used as a minimal example is here, (the latest commit on the branch is fixed to not trigger this issue):

https://github.com/kingdonb/docker-rvm/blob/30c7d0a22425b25cc9b754e6aea2ce4a12279a84/docker-rvm-test/Dockerfile

RUN bundle config set app_config .bundle
RUN bundle config set path .cache/bundle
RUN --mount=type=cache,uid=999,target=/home/${RVM_USER}/app/.cache/bundle \
      rvm ${RUBY} do bundle install && \
      mkdir -p vendor && \
      cp -ar .cache/bundle vendor/bundle
RUN bundle config set path vendor/bundle

The commit that resolved it, so you can see what changed: kingdonb/docker-rvm@6c5bab1

I have this ${RVM_USER} variable that I'd like to use here which comes from an upstream image, but unfortunately can't. (So you're suggesting if I inspect the cache-mounted container at runtime I will find a mounted cache volume at /home//app/.cache/bundle ? Or one at /home/${RVM_USER}/app/.cache/bundle? The progress output did not reflect that... would the mount activity silently create a directory, even if recursively? That seems like it could be handy, in some cases.)

The example is borrowed from this site https://qiita.com/k_kind/items/836bc7ba2e33dc2ed3e7 (their version works by itself, as they did not attempt to perform variable substitution).

[tonistiigi]

I will find a mounted cache volume at /home//app/.cache/bundle ? Or one at /home/${RVM_USER}/app/.cache/bundle?

in /home/${RVM_USER}/app/.cache/bundle . You can use mount command inside container to check what directories are mountpoints.

I think we can close this and leave #815 to track variables support.