9p in WSL2 is unusable, please expose ext4 vhdx using Samba server instead

[ned14]

My windows build number: 19041.207

My use case is a single source code checkout for a software project which is simultaneously built using both Windows and Linux using WSL. The key productivity gain WSL offers is being able to develop for both Windows and Linux by editing a single source tree from within Visual Studio, and running separate build directories for each platform from that single source tree.

Assuming that now that WSL2 is shipping into Release Preview and it would be production ready, I converted over an existing WSL1 image. I quickly realised that keeping the source tree on Windows and accessing it from Linux via /mnt/c was unviably slow. So I relocated the source tree from Windows into the ext4 vhdx, and tried it the other way round i.e. the single source tree lives within Linux, and Windows accesses it from \wsl$.

Windows accessing Linux via 9p is far faster than Linux accessing Windows via 9p, so that was better. Unfortunately, Visual Studio won't cmake configure from \wsl$ (I mounted \wsl$ as U:\):

U:\home\ned\boostish\llfio\programs\build_vs2019>cmake ..
-- Building for: Visual Studio 16 2019
-- quickcpplib not found, cloning git repository and installing into U:/home/ned/boostish/llfio/programs/build_vs2019/quickcpplib ...
-- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.22.0.windows.1")
-- Selecting Windows SDK version 10.0.17763.0 to target Windows 10.0.19041.
-- The CXX compiler identification is MSVC 19.24.28314.0
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/cl.exe -- broken
CMake Error at C:/Program Files/CMake/share/cmake-3.16/Modules/CMakeTestCXXCompiler.cmake:53 (message):
  The C++ compiler

    "C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/cl.exe"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: U:/home/ned/boostish/llfio/programs/build_vs2019/CMakeFiles/CMakeTmp

    Run Build Command(s):C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin/MSBuild.exe cmTC_90b73.vcxproj /p:Configuration=Debug /p:Platform=x64 /p:VisualStudioVersion=16.0 /v:m && Microsoft (R) Build Engine version 16.4.0+e901037fe for .NET Framework
    Copyright (C) Microsoft Corporation. All rights reserved.

      Microsoft (R) C/C++ Optimizing Compiler Version 19.24.28314 for x64
      Copyright (C) Microsoft Corporation.  All rights reserved.
      testCXXCompiler.cxx
      cl /c /Zi /W3 /WX- /diagnostics:column /Od /Ob0 /D WIN32 /D _WINDOWS /D "CMAKE_INTDIR=\"Debug\"" /D _MBCS /Gm- /EHsc /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /GR /Fo"cmTC_90b73.dir\Debug\\" /Fd"cmTC_90b73.dir\Debug\vc142.pdb" /Gd /TP /errorReport:queue U:\home\ned\boostish\llfio\programs\build_vs2019\CMakeFiles\CMakeTmp\testCXXCompiler.cxx

    LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage.FinalPhase [U:\home\ned\boostish\llfio\programs\build_vs2019\CMakeFiles\CMakeTmp\cmTC_90b73.vcxproj]

        Version 14.24.28314.0

        ExceptionCode            = C0000005
        ExceptionFlags           = 00000000
        ExceptionAddress         = 0000000000000000 (0000000000000000) "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\bin\HostX64\x64\link.exe"
        NumberParameters         = 00000002
        ExceptionInformation[ 0] = 0000000000000008
        ExceptionInformation[ 1] = 0000000000000000

      CONTEXT:
        Rax    = 0000000000000000  R8     = 0000000000000000
        Rbx    = 000001BC4B227C60  R9     = 000001BC4B226F20
        Rcx    = 000001BC4B227C60  R10    = 0000000000000020
        Rdx    = 0000000000000018  R11    = 0000000000000000
        Rsp    = 000000240795D968  R12    = 0000000000003A64
        Rbp    = 000000240795E530  E13    = 0000000000000000
        Rsi    = 0000000000000000  R14    = 0000000000000000
        Rdi    = 000001BC4B1C7900  R15    = 0000000000000002
        Rip    = 0000000000000000  EFlags = 0000000000010246
        SegCs  = 0000000000000033  SegDs  = 000000000000002B
        SegSs  = 000000000000002B  SegEs  = 000000000000002B
        SegFs  = 0000000000000053  SegGs  = 000000000000002B
        Dr0    = 0000000000000000  Dr3    = 0000000000000000
        Dr1    = 0000000000000000  Dr6    = 0000000000000000
        Dr2    = 0000000000000000  Dr7    = 0000000000000000





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:5 (project)


-- Configuring incomplete, errors occurred!
See also "U:/home/ned/boostish/llfio/programs/build_vs2019/CMakeFiles/CMakeOutput.log".
See also "U:/home/ned/boostish/llfio/programs/build_vs2019/CMakeFiles/CMakeError.log".

I noticed that accessing Linux from Windows via 9p is very noticeably slower than accessing a Linux running inside Hyper-V over Samba. So I installed a samba server inside the WSL2 Linux. Unfortunately, Windows does not appear to have a normal virtual network connection with Windows like a Hyper-V VM would have, and it cannot see the Samba server.

Can you therefore:

1. Make it possible for the WSL2 VM to have an ordinary, separate, network which performs standard DHCP IP discovery? I appreciate that this makes WSL2 more like Hyper-V, but in my opinion that is no bad thing.

2. Make it possible to replace 9p with a Samba served from the Linux VM? I'm not sure why you chose 9p, the SMB protocol would seem to me far easier to deeply paravirtualise between a custom Linux kernel and the Windows kernel. After all, the Linux kernel specifically implements oplocks in order to accelerate Samba! You (only!) need to wire together oplocks in the custom Linux kernel with those in the Windows kernel with a shared kernel page cache for filesystem content (which Samba already implements) to provide a superb shared-filesystem experience.

3. Assuming you're going to say no to my number 2 request, can you at least get cmake with Visual Studio working over a \wsl$ mount, and enable splitting of the network so the WSL2 network is completely separate from the Windows network so I can manually replace 9p with Samba?

Finally, I appreciate all of the above is a bit negative. I wish to send my thanks and gratitude to you and all the WSL team for WSL. It has transformed my personal productivity, and thank you so much for enabling that.

[therealkenc]

Its going to be tough to cover the multiple topics on this one. 9p performance hangout is #4197. WSL2 uses a virtual network switch like Hyper-V. WSL uses p9 not SMB/CIFS for reasons. You can run Samba on WSL2, although you'll get better eyeballs in a Samba group forum than here. It isn't how I'd roll personally, but YRMV. For suggestions on best practices with MSVS and WSL you'll probably get better eyeballs where Visual Studio folks hang out, but maybe someone will chime in. Whatever is causing your LINK fatal would need a tighter CLI repro to investigate or turn into an actionable.

[ned14]

I did some more work on this on Sunday.

I configured a new VM in Hyper-V, installed Ubuntu 18.04 Server headless, then installed the Hyper-V Linux integration services using https://onpremisys.nl/2019/06/21/hyper-v-linux-integration-service-for-ubuntu-18-04. I adjusted the network switch configuration to enable RDMA and set Jumbo frames to 9k, and disabled Virtual Machine Queue (VMQ). I forced the Samba configuration on Linux to only ever use SMB3.

This improves Samba performance even more, despite that Samba does not yet implement SMB Direct (RDMA) when Windows is accessing a Linux Samba server (the other way round, apparently, does now work).

Re: the benchmarks on #4197, whilst the bandwidth is important, I think the real problem with 9p is the latency - compiling a large C++ project does a ton of random read i/o of small quantities, and 9p's high access latencies makes it a far worse experience for my use case than Samba, or WSL1.

For my use case, going forth, WSL2 currently offers no benefits whatsoever over WSL1 or a hand configured Hyper-V VM. I'll thus be sticking to a combination of those in the medium term. I do however look forward to the WSL team considerably improving the shared Linux-Windows filesystem experience in the future, that's what I really need to work better than currently. A nice alternative is if Visual Studio ever began working well under wine, then I could run Linux as my main system, and compile and debug the Windows side of things entirely under wine. We'll see how things evolve.

[PavelSosin-320]

Please, take into account the following VERY POSSIBLE and VERY CHEAP laptop configuration:

1. System sad disk medium performance NTFS
2. Extra ssd 2Tb blazing fast system bus disk which can be formatted as the owner wants. Is it possible to use the non-system disk as WSL disk and Docker persistence provisioner disk?

[ned14]

My test system is a recent Dell XPS 13 with Samsung NVMe SSD. Its storage is very quick indeed.

I ended up doing a little bit more work before giving up on WSL2:

• 9p i/o latencies are actually well below those of SMB, but 9p appears to hang if you have any concurrency. I ran a QD4 test (queue depth = 4) and 9p hurt so bad the Windows kernel began to behave funny, so I had to reboot the machine. QD1 testing, on the other hand, was absolutely fine. Though QD1 bandwidth for 9p is a tiny fraction of that for Samba on the same test.

• You the WSL2 guys need to implement some sort of deeply paravirtualised shared filing system between Windows and WSL. Something which shares kernel page cache. Then we could achieve a shared filing system with performance approximating at least ZFS on Linux, which would be tolerable.

• I still think a deep WSL integration of SMB is a lot less work than for 9p. Turn on SMB Direct, bypass the network stack, directly map Windows kernel memory into the Linux kernel to avoid memory copies during the RDMA emulation, and I think you'd have a winner there.