cannot receive inotify on shared directories when using AppleHV

[BlackHole1]

Issue Description

When the host modifies/deletes a file, the guest/container cannot receive inotify, resulting in the file watch inside the container not taking effect. The actual impact is that features like code hot reloading do not work properly.

After extensive testing, I have concluded the following:

1. It is not related to the container, as inotify cannot be received in the machine’s virtual environment either.
2. It is not specific to the Fedora Core Linux distribution, as it also occurs in other distributions.
3. It may not be related to virtiofs. As of 2021, virtiofs has already supported inotify, as seen here: https://lwn.net/Articles/874000/
4. Similar issues have also been observed in the colima and lima virtual machine tools.
5. Docker for Mac and Orbstack support inotify, but I am unsure of their implementation as their code is closed-source 🤷‍♂️

Regarding point 4, relevant discussions and code can be found here:

• lima-vm/lima#615
• lima-vm/lima#1913
• https://github.com/abiosoft/colima/blob/main/daemon/process/inotify/events.go

However, the solutions provided by lima and colima are not perfect as they cannot simulate DELETE events. They can only simulate creation/modification events.

Regarding point 5, I came across a brief introduction in Docker’s blog: https://www.docker.com/blog/deep-dive-into-new-docker-desktop-filesharing-implementation/ (it seems to be for Windows, and I am unsure if the same solution applies to macOS). The approach of Docker seems to be creating an intermediate layer to deceive the virtual machine, in order to successfully simulate the DELETE event using rm -rf.

Steps to reproduce the issue

# Host
podman machine ssh

# Guest
cd ~
curl -L -O https://github.com/watchexec/watchexec/releases/download/v1.25.1/watchexec-1.25.1-x86_64-unknown-linux-musl.tar.xz
tar -xvf watchexec-1.25.1-x86_64-unknown-linux-musl.tar.xz
sudo cp ./watchexec-1.25.1-x86_64-unknown-linux-musl/watchexec /usr/local/bin/
mkdir -p /private/tmp/test
cd /private/tmp/test
watchexec -r --emit-events-to=stdio --only-emit-events

# Host
cd /tmp/test
touch new_file
echo "modify" >> new_file
rm -rf new_file

15-11.19.15.mp4

Describe the results you received

No inotify events received.

Describe the results you expected

I can receive inotify messages for create/modify/delete operations.

podman info output

Version: v5.0.1

Details

host:
  arch: amd64
  buildahVersion: 1.35.3
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.94
    systemPercent: 0.03
    userPercent: 0.03
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2047
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 502
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.7.9-200.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 679960576
  memTotal: 2054959104
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/502/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240326.g4988e2b-1.fc39.x86_64
    version: |
      pasta 0^20240326.g4988e2b-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/502/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 3h 17m 2.00s (Approximately 0.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 3979911168
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/502/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.1
  Built: 1711929600
  BuiltTime: Mon Apr  1 08:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

AppleHV

Additional information

I suspect this is a bug in the Apple virtualization.framework.

---

/cc @baude @cfergeau

[cfergeau]

It may not be related to virtiofs. As of 2021, virtiofs has already supported inotify, as seen here: https://lwn.net/Articles/874000/

Not sure about this, see this comment from about 1 year ago lima-vm/lima#615 (comment)

I just tried qemu-virtiofsd on a Linux host, host to guest inotify doesn't work either(so it's not Apple's issue). There was some work trying to add inotify support for FUSE, but seems like it wasn't merged in mainline.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[BlackHole1]

👀

[rhatdan]

It seems from reading above that this is an issue with virtiofsd, not sure if it is something with the way Podman sets it, or if this is even a podman bug.

[BlackHole1]

It seems from reading above that this is an issue with virtiofsd, not sure if it is something with the way Podman sets it, or if this is even a podman bug.

This issue is not caused by podman, but by an upstream dependency (virtiofs) of podman.

However, it has affected the users of podman, but I am not sure how costly it would be to fix this issue within podman.

[rhatdan]

Can it be fixed by podman? inotify is handled between the kernel and the virtiofsd. Perhaps Podman is not setting a setting correct in the kernel to work with it, but otherwise this is not something that can be fixed by Podman.

[rjeffman]

I'm not sure how I can help with this issue but reporting my scenario, which seems close to this one.

If I'm running a container that requires notifications (Jekyll), if I create or modify a file in a shared folder from macOS, I do not see any notification event, but I can see the modifications in the container (e.g. cat-ing the file). If the modification is done in the shared folder on the VM that runs in the container (e.g. ssh-ing into podman machine), any change on the shared folder are notified in the container (and on macOS).

For now, I'm using this workaround: edit the files I need immediate rebuild directly in the VM, if not, I'll edit on macOS and simply touch it in the VM (so that the container is notified).

If there's anything else I can do to help finding the issue, let me know, as fixing this would be a nice improvement in several of my workflows.

[afh]

Thanks for sharing your workaround, @rjeffman. As I'm familiarising myself with podman I have a few questions about your workaround:

• If the container, not the machine, is run with -v $PWD:/srv/app is that volume available as a shared folder in the machine?
• If yes, how could I find it easily? I see several mounts below pkgs/top-level/darwin-packages.nix, but all use (internal?) hexadecimal identifiers
• If no, what steps are necessary to expose the volume within the podman machine in order to be able to edit it there and trigger the inotify events?

Hopefully I've been able to phrase my questions in a way that makes sense for someone more familiar with podman…

[AshDevFr]

It's been a couple of months since this issue was open, is there any plan on fixing the behavior?

I do understand that this is caused by a dependency (virtiofs) of Podman, but it kind of make Podman unsuitable for local dev usage in some cases