Migrate from d4m-nfs to native Docker NFS Volumes discussion

[mauro-ni]

It doesn't work with Docker for Mac Version 17.12.0-ce-mac46

$ ~/d4m-nfs/d4m-nfs.sh
[d4m-nfs] You will need to provide your Mac password in order to setup NFS.
Password:
[d4m-nfs] Copy the Moby VM APK Cache back.
[d4m-nfs] Create the script for Moby VM.
[d4m-nfs] Start and restop nfsd, for some reason restart is not as kind.
The nfsd service is already running.
[d4m-nfs] Wait until NFS is setup.
[d4m-nfs] Wait until D4M is running.
[d4m-nfs] Setup 'screen' to work properly with the D4M tty, while at it name it 'd4m'.
[d4m-nfs] Run Moby VM d4m-nfs setup script.
[d4m-nfs] Waiting until d4m-nfs setup is done
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
.........................................................................................
......................

[pensiero]

+1

[Plopix]

EDIT to explain better what I understood after more debug.

The new version moved to Linuxkit that is a read-only system.
The screen is still working but it is not possible to install nfs-util and then we cannot mount the host.
I have no clue how to do that on Linuxkit or if it is even possible, that kind of sucks, I am currently looking to docker-sync as I start to think that will be the only way to have acceptable performances.

But you might have a magic solution! I am sure you have one, right? :/

cc @if-kenn

[if-kenn]

The brief time I have had to look into this comes to the same conclusion as @Plopix. With the latest release of Docker for Mac, Docker has again changed the way in which MobyVM sees the Mac's file system.

This would force us to reverse to a previous, more fragile and much harder to maintain way that we injected the script to setup NFS.

With this latest change, I need to revisit if the osxfs volume performance tuning has finally caught up enough to stop the need for d4m-nfs. This whole time I have been honestly hoping to deprecate this software once there was a better solution from Docker. I will report back my findings and the future of d4m-nfs. For now I am going to put a big warning in the README.

[if-kenn]

README has been updated, performance testing and outcome will have to wait until tomorrow.

[Plopix]

thanks @if-kenn, to me the osxfs volume performance tuning does not help as our main issue here is the write performances on a mount and not the read. The tunning is pretty much just a cache system but I might be wrong I did not test extensively as the NFS method was still better ;)

[Plopix]

Also I dig a bit more on docker-sync even if I don't like the duplication and the fact we need a kindof daemon, it seems much more simpler to config than before. This would do the trick (of course using the external volume in the docker-compose-dev).

But the NFS solution was nice, simple, and much more transparent! we have to find a way!

# docker-sync.yml
version: "2"
options:
  compose-file-path: 'docker-compose.yml'
  compose-dev-file-path: 'docker-compose-dev.yml'
syncs:
  project-sync:
    sync_strategy: 'native_osx'
    src: './'
    sync_userid: '501' # to be adapted
    # optional, a list of excludes
    sync_excludes: ['.git', '.gitignore', '.DS_Store', '.idea', 'docker-sync.yml', 'docker-compose.yml', 'docker-compose-dev.yml', 'etc.']

[yeka]

I'll just put it here in case someone already upgrade their docker and want the previous version:
https://download.docker.com/mac/stable/21090/Docker.dmg

[avinson]

@if-kenn my recent testing of osxfs volume tuning (about a month ago) found that rails rake tasks ran about 10-15 times faster under d4m-nfs

[if-kenn]

So some bad news. While you can still screen into the Linux VM, it is a read only root file system. The way that d4m-nfs works is to be able to add the nfs-utils apk package. This means that when running apk add nfs-utils sntpc you get Read-only file system. I have unsuccessfully tried to remount the root files system as read/write. At this point, it looks like Docker for Mac has effectively killed d4m-nfs.

[if-kenn]

I am looking at the possibility to roll a new OS image for Docker for Mac from Linuxkit to understand if it is possible to mount / read/write or roll an image with the nfs package already on it. The problem with this it is not light weight (read: could take a while to make the image and something we cannot support) or there would have to be a trust factor of any OS image that would come from doing it and centrally disseminating it.

[if-kenn]

So an update, I have been playing with Linuxkit and have been able to have a writable root file system, install nfs-utils and run rpcbind. What I have not been able to do is mount the Mac's NFSd, it appears that it is being blocked by how VPNKit is setup. The documentation on all of this is not the easiest to understand what needs to be done. If anyone has knowledge of VPNKit and how it relates to Linuxkit, that would be quite helpful.

[if-kenn]

So, a very small break through. On a customized Linuxkit build, I have been able to successfully mount NFS from Mac host using the IP (192.168.65.2) I found on https://github.com/moby/vpnkit/blob/master/docs/ethernet.md.

That said, I have yet to find a way to programmatically get that IP, and after that there would still be the need to get rpcbind service used in the correct Linuxkit way, and then feed the d4m-nfs info over to it.

[if-kenn]

It looks like according to https://github.com/linuxkit/linuxkit/blob/master/docs/platform-hyperkit.md we can rely on the virtio-net IP to be 192.168.65.2 by default unless it was to be assigned a fixed IP via linuxkit run -ip.

[pensiero]

We could use it as default value, and in case specify it it as a configuration value, what do you think?

[if-kenn]

@pensiero Yes, that is my intention, but let me be very clear, this is not an easy fix, it will not just be a drop in replacement and the solution will not come quickly if at all. In that light please keep the following in mind:

• there is very sparse Linuxkit/VPNkit/etc documentation to help
• it will not be in the immediate future
• it will very well have to a be new github repository

So if you are looking for a fix soon, please downgrade to an older version of Docker for Mac.

I am intending to keep updating on progress here in order to provide transparency or if others have insights that might help.

[justincormack]

See the details here docker/for-mac#2419 (comment)

We moved to a read only rootfs as we now boot off an ISO, as Kubernetes is too large to boot out of a writeable RAM filesystem. It would have been easier if you had always used a privileged container for NFS, rather than assuming that the host would have a working Alpine system, as this was never a guarantee. You should be able to run NFS fine with either a privileged Docker container or a privileged containerd container. The host static IP shouldn't change; there is also a DNS name for it documented. You should not need to respin a whole LinuxKit image, and if you use a privileged Docker container set to always restart it should work reasonably well.

[if-kenn]

@justincormack from what I am understanding from David Scott's comment and what I can make of yours above, I have tried to prototype what is suggested, to no luck. Unless there is some magic that I am missing, it seems like building a new Linuxkit image is still needed.

[Yivan]

@if-kenn
It make several hours i try to mount NFS too (on Docker for Windows, but i got the same problems of you).

What i don't anderstand is there is native nfs volume in docker, why not use them ? (end of this page: https://docs.docker.com/engine/reference/commandline/volume_create/#driver-specific-options)
I tried on windows with no luck... people having the same timeout problem never get answer from the docker team : (( (please see: docker/for-win#1241).
Maybe on mac it can work, you made a try of nfs volume ?
I opened an issue here to have a clear answer on this subject: docker/for-win#1700

From what i found, it is because docker for win (and from what i see maybe docker for mac too) don't have the NFS utils to mount nfs. Did you manage to find a way without relying on a custom VM/ISO linuxkit image ?

[Yivan]

Hum really strange!
It seems with docker-compose we can mount nfs folder... (i don't have the connection timeout problem that i have with docker volume create & docker run)

volumes:
    nfsshare:
        driver_opts:
            type: "nfs"
            o: "addr=10.0.75.1,hard,nolock,rw"
            device: ":/server-dev"

It works fine! I don't undestand how docker manage to do it has there is no nfs tools on the VM....
Maybe it can give you a new route to make d4m working.

[Yivan]

It seems the no-lock option did the trick:
The following work great:

docker volume create --driver local --opt type=nfs --opt o=addr=10.0.75.1,hard,nolock,rw --opt device=:/server-dev mynfs
docker run -it -v mynfs:/server-dev alpine sh

[yvess]

@Yivan so with your one volume creating command, you can work like before? Is the volume persistent, or do you need to recreate it after docker restart.

[Yivan]

@yvess yes you can work as usaually and the volume is like other volume, so it persiste as long you don't delete the VM.
Even if you delete the volume, the data are kept because the volume just mount your host datas (it doesn't conains the data, it just countain the nfs mount point).

[mauro-ni]

@Yivan currently I'm using shared folder for application code in my docker-compose.yml.

version: '2'
services:
  web:
  ...
  volumes:
    - ../www:/home/app/www
  ...

How can I convert this so that NFS is used?

Many thanks.

[mauro-ni]

@Yivan with docker-compose I get a timeout error even if I provide the nolock option

[Yivan]

If you mount it directly from a container does it work ? (you can create an Ubuntu container, install nfs client, and mount the NFS share).
It should work.

• if not, there is pbl with your NFS Server or IP is not good.
• if it work, it should work in docker-compose too, so problem is maybe firewall, try desactivate it

[mauro-ni]

@Yivan could you please create a demo and share it (new git repo)? I'd like to use NFS for the folder containing my rails apps.

[Yivan]

There is no need i set a new repo for just two lines of code...
For instance, if the NFS server has IP 10.0.75.1 and the export name is /server-dev, the code is:

docker volume create --driver local --opt type=nfs --opt o=addr=10.0.75.1,hard,nolock,rw --opt device=:/server-dev mynfs
docker run -it -v mynfs:/server-dev alpine sh

Did you try what i suggest you (mount it manually, disable firewall, etc.) ? What are the results ?
I should add if you try mounting it manually you must add some CAPS to you container, at least SYS_ADMIN and DAC_READ_SEARCH (see here for more information to set them: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities)

[ryansch]

Give this fix a try: docker/for-mac#2419 (comment)

[sponomarev]

Yeah, I was confused with it too.

[myers]

@maurosbu I think they added some support for nfs in the moby vm, like what this project attempts to do, but it was perhaps not so much to address the mac filesystem part, but rather as part of better support for kubernetes.

[Plopix]

Did someone test if now the Linux Kit let us do mount now?

We added rpcbind and rpc.statd at VM startup in order to fix NFS support for persistent volume in Kubernetes.

That would mean we can mount like before and use d4m-nfs like before? Or does that just mean now we can safely use an external volume over nfs?
or both?

[Plopix]

I can answer my own question I have just tested.

=> no we cannot use d4m-nfs as before even with this change

But we can definitively do as mentioned by @myers, just did and it is working great.
It sounds better in term of logic, in term of performance, I checked with a time dd if=/dev/zero of=speedtest bs=1024 count=100000 and I was around 69MB/s (same with d4m-nfs)

One thing though, you need to export explicitly what you want to mount.
You cannot export: /Users/plopix and then mount /Users/plopix/Documents/DEV/ProjectOne
Hope it helps some of you. ++

[myers]

@Plopix

One thing though, you need to export explicitly what you want to mount.
You cannot export: /Users/plopix and then mount /Users/plopix/Documents/DEV/ProjectOne

have you tried the -alldirs option on the mac side's /etc/exports

from mac os's man 5 exports:

The -alldirs flag allows the host(s) to mount at any point within the file system, including regular files if the -R option is used on nfsd.

[Plopix]

hey hey @myers of course! That is working! Awesome! Thank you!

[sponomarev]

@Plopix Can you share a step-by-step guide? 🙏

[Plopix]

@sponomarev, I have just followed what @myers provided actually. If your OSX is correctly set to exports directories you should be good.
Another example

$ cat /etc/exports
/Users/plopix/Documents/PROJECTS -mapall=plopix:staff -alldirs localhost
$ cat /etc/nfs.conf
nfs.server.mount.require_resv_port = 0

Don't forget to restart sudo nfsd restart

Then I have a docker-compose.yml for Linux users (here is an extract regarding the mounts)

    engine:
        volumes:
            - '${PROJECTCOMPOSEPATH}/${PROVISIONINGFOLDERNAME}/dev/engine/php.ini:/usr/local/etc/php/php.ini:ro'
            - '${PROJECTCOMPOSEPATH}:${PROJECTMAPPINGFOLDER}:rw'

don't pay too much attention to the PROJECTCOMPOSEPATH, PROVISIONINGFOLDERNAME, and PROJECTMAPPINGFOLDER it is part of a more complex tooling mechanism

Then OSX users would run docker-compose using an override:
-f docker-compose.yml -f docker-compose-osx.yml

And in this override:

version: '2.1'
services:
    engine:
        volumes:
            - "nfsmount:${PROJECTMAPPINGFOLDER}"
volumes:
  nfsmount:
    driver: local
    driver_opts:
      type: nfs
      o: addr=host.docker.internal,lock
      device: ":${PROJECTCOMPOSEPATH}/"

That is it, @myers thanks again. We don't really need d4m-nfs as before, but thanks as well to IFSight contributors and maintainers, it was awesome to have it.

Note: I did not test extensively, as it is quite new, then I can not confirm if it is 200% working, or if we will be able to work with no issue with that approach. But I don't see we should not.

[if-kenn]

Thrilled to see that d4m-nfs is not needed!

[seanhandley]

I blogged about my solution here https://medium.com/@sean.handley/how-to-set-up-docker-for-mac-with-native-nfs-145151458adc

[patakijv]

@Plopix @seanhandley I just tried the nfs setup mentioned above with latest docker for mac on a ruby on rails app that has been using docker setup for a while (and previously was using d4m-nfs successfully before this issue) - I run into this issue of inability to get file locks (from rack-attack) on first reload of a page

No locks available @ rb_file_flock - /var/www/pnd/tmp/cache/D32/0F0/rack%3A%3Aattack%3A5075617%3Areq%2Fip%3A172.18.0.1

I didn't run into this problem with d4m-nfs prior to the version of docker that broke it.

Any clues as to what needs to be in place to insure file locking works as expected?

[mauro-ni]

@seanhandley many thanks for your post, I commented it with the following two questions:

1. Is it possible to use a more updated version of NFS (version 4.X)?
2. Is there a way to use relative path for ${SOURCE_DIR}?

[seanhandley]

@patakijv You could try setting lock in the o string in the yaml? I set nolock but it may be that you're safer using locking with your setup.

[seanhandley]

I found I couldn't get it working on NFSv4 @maurosbu so I downgraded. You can try it and see what happens.

As for relative paths - I'm not sure. Try it and see?

[mauro-ni]

@seanhandley It doesn't work.

I modified your script in order to force the use of NFSv4 by appending the following line to /etc/nfs.conf

nfs.client.mount.options = vers=4

Then within the container I did a check and I discovered that version 3 is used:

 app@14a33bc2e07c:~/www$ nfsstat -m
{CONTAINER_DIR} from :{LOCAL_DIR}
Flags: rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr={IP},mountvers=3,mountproto=tcp,local_lock=all,addr={IP}

Relative paths are not working.

[patakijv]

@seanhandley I was actually using lock in the options as seen above in @Plopix's post, I hadn't seen your nolock usage

I am using

/etc/exports
"/Users/patakijv/Work/Workspace" -alldirs -mapall=501:20 localhost

docker-compose-dev.yml

 nfsmount:
     driver: local
     driver_opts:
       type: nfs
       o: addr=host.docker.internal,lock
       device: ":${APP_ROOT_MOUNT_PATH_ON_HOST}/"

Where $APP_ROOT_MOUNT_PATH_ON_HOST is /Users/patakijv/Work/Workspace/pnd

[patakijv]

Not sure which of the following changes resolved my locking issue but it is now apparently resolved:

1. I updated to (I think today's) release Version 18.03.0-ce-mac60 (23751)
2. I restarted Docker (after the update of course - previously I had not tried restarting Docker)
3. I changed my nfsmount options like to o: addr=host.docker.internal,rw,lock,hard,nointr,nfsvers=3 (it was previously as my prior post)

For a new docker-compose-dev.yml volume mount entry

volumes:
 nfsmount:
     driver: local
     driver_opts:
       type: nfs
       o: addr=host.docker.internal,rw,lock,hard,nointr,nfsvers=3
       device: ":${APP_ROOT_MOUNT_PATH_ON_HOST}/"

Haven't yet done any file write speedtests or comparisons between using vs not using this setup.

[seanhandley]

Glad it works @patakijv - do please share your speed findings 👍

[birkof]

For me, MacOS 10.13.4 (17E199), works set up device like:
device: ":${PWD}/"

@seanhandley many thanks for your post

[if-kenn]

We are officially deprecating this project. We will leave it in place in case it can help some people migrate to NFS volumes from the command line or compose.

[spaceemotion]

Since this took a bit of time and I wanted to share this with others (I hope this is the right place), here's the solution we've ended up with based on the work of @Plopix, @patakijv and @birkof:

1. An "overwrite" compose file with contents similar to this one:

   version: '3.2'
   
   services:
     webapp:
       volumes:
         - nfsmount:/var/www/html
   
   volumes:
     nfsmount:
        driver: local
        driver_opts:
          type: nfs
          o: addr=host.docker.internal,rw,lock,hard,nointr,nfsvers=3
          device: ":${PWD}/application/"

2. A helper script for starting the setup on OSX that takes care of fixing various permission issues:

   #!/bin/bash
   
   NFS_CONF="/etc/nfs.conf"
   NFS_EXPORTS="/etc/exports"
   WORKING_DIR=$(pwd)
   
   prepare_nfs()
   {
       echo "Checking NFS mounts + permissions"
   
       local do_restart=0
   
       # Update NFS config
       if [[ $(grep -c "require_resv_port" "$NFS_CONF") -eq 0 ]]; then
           echo " - Updating config ($NFS_CONF)"
           echo "nfs.server.mount.require_resv_port = 0" | exec sudo tee -a "$NFS_CONF" > /dev/null
   
           do_restart=1
       fi
   
       # Fix NFS permission errors
       if [[ $(grep -c "$WORKING_DIR" "$NFS_EXPORTS") -eq 0 ]]; then
           echo " - Updating exports ($NFS_EXPORTS)"
           echo "\"$WORKING_DIR\" localhost -alldirs -mapall=501:20" | exec sudo tee -a "$NFS_EXPORTS" > /dev/null
   
           do_restart=1
       fi
   
       if [[ do_restart -eq 1 ]]; then
           echo " - Restarting NFS service"
           exec sudo nfsd restart
       fi
   }
   
   main()
   {
       prepare_nfs
       docker-compose -f docker-compose.yaml -f docker-compose-osx.yaml restart
   }
   
   main

(we have our local application contents in ./application which gets mounted to /var/www/html)