File system change events (inotify) support for mounted directories

[charleskorn]

Description

From abiosoft/colima#151:

We're using Colima to run Docker in a VM on our macOS laptops. Colima uses Lima under the hood to run this VM.

Are there any plans to support file system change events for mounted directories through inotify?

This is a key feature for us as we use containers in scenarios like auto-reloading dev servers and re-running unit tests on changes.

If there are no plans to support it at the moment, what kind of effort would be involved in adding this? Depending on the effort required, we might be open to putting together a PR to add support for this.

[jandubois]

what kind of effort would be involved in adding this

As I already commented in the colima issue, I'm not sure this is a possibility at all (generating inotify events). Even if you capture them on the host (fsevents), I think you would need to implement a custom notification mechanism in the guest and can't expect to use inotify, which is a kernel mechanism.

[AkihiroSuda]

In far future we may support inotify with virtiofs (not virtio-9p-pci).
https://lwn.net/Articles/874000/

[afbjorklund]

There were some earlier series, about the various limits of inotify.
https://lwn.net/Articles/605128/

Inotify reports only events that a user-space program triggers through the filesystem API. This constitutes a fairly serious limitation of the API. For example, it means that inotify does not inform us of events on monitored objects via a remote filesystem (e.g. NFS) operation.

Previously we "solved" it by keeping the files locally on the VM.

[atipugin]

Looks like virtiofs is already supported (although marked as experimental), but I still can't get FS events working. Tools like entr (which relies on inotify and kqueue) and webpack (in watch mode) don't see file changes. Interesting that FS notifications work fine with Docker Desktop (with same settings - VZ + virtiofs).

[lobshunter]

Looks like Apple's virtio-fs document doesn't mention anything about inotify. Inotify in docker desktop container does work(although it would drop DELETE event, and inject additional WRITE event). docker.log shows messages like level=info msg="injected MODIFY event (ts: 1677314215061760000) on /host_mnt/private/xxx, seems like it's doing some kind of inotify event injection 🤔.

[abiosoft]

docker.log shows messages like level=info msg="injected MODIFY event (ts: 1677314215061760000) on /host_mnt/private/xxx

That shows that Docker is manually handling the notify events i.e. the feature is missing in virtiofs.

[lobshunter]

i.e. the feature is missing in virtiofs.

Looks like so. I just tried qemu-virtiofsd on a Linux host, host to guest inotify doesn't work either(so it's not Apple's issue). There was some work trying to add inotify support for FUSE, but seems like it wasn't merged in mainline.

[lobshunter]

I bet docker desktop watches files in host, and forwards events into guest(perhaps by invoking something like chmod/write).
Start a docker destop container with a volume, then execute lskq(1) in host, it would show docker is watching files.

It might not be a great solution to do the same in lima.

[henrik242]

I'm trying to get node/next.js development working with colima, and next.js doesn't pick up changes I do on the host. However, if I touch the changed file from within the container, next.js picks up the change and my web page refreshes.

So, as a (silly) workaround, maybe one can use a script or daemon in the container to continuously scan the files and touch any changed files?

[lobshunter]

However, if I touch the changed file from within the container, next.js picks up the change and my web page refreshes.

Yes, guest to guest inotify works as expected, but host to guest inotify doesn't(most likely because FUSE protocol doesn't support it).

maybe one can use a script or daemon in the container to continuously scan the files and touch any changed files?

As a personal workaround, that would work(just less performant, performance degrades linearly with the number of files being added).

[henrik242]

As a personal workaround, that would work(just less performant, performance degrades linearly with the number of files being added).

Yeah, it's not ideal at all. This is the easiest I could come up with, in case others need it:

FOLDER=src
CHKPT=/tmp/checkpoint
touch $CHKPT
while true; do
  find "$FOLDER" -newer $CHKPT -type f -print -a -exec touch {} \; | grep -q . && touch $CHKPT
  sleep 1
done

The file system seems to only update about every 10 seconds, so changes aren't immediate. I'm using grep to give find a non-zero exit code, only touching /tmp/checkpoint after a change has been seen.

[henrik242]

Some more testing reveals that running the script on the host is a lot more responsive:

FOLDER=$1
TOUCH="docker exec -ti my-docker-guest touch"
CHKPT=/tmp/checkpoint
touch $CHKPT
while true; do
  find "$FOLDER" -newer $CHKPT -type f -print -a -exec $TOUCH {} \; | grep . && touch $CHKPT
  sleep 1
done

[johan-seesaw]

I figure'd I'd reference another issue from the colima repo for all those who are using lima via colima. The new options in colima are working for my usecase.
abiosoft/colima#261

colima stop
brew unlink colima
brew install --HEAD colima
colima start --mount-inotify

[henrik242]

You don't need to use the HEAD version. The latest regular version already supports --mount-inotify

[linnik]

You don't need to use the HEAD version. The latest regular version already supports --mount-inotify

option is here, but it does not work

[AkihiroSuda]

Linking the latest proposal by @balajiv113

• File system change events support for mounted directories #1853

[aeifn]

the latest regular version already supports --mount-inotify

Is it possible to use something like that with lima only?

[balajiv113]

@aeifn There is a draft PR #1913

[aeifn]

@aeifn There is a draft PR #1913

Thank you! It works for me (vz, virtiofs).