[libc++] use copy_file_range for fs::copy

[Jannik2099]

Hi,

this optimizes std::filesystem::copy_file to use the copy_file_range syscall (Linux and FreeBSD) when available. It allows for reflinks on filesystems such as btrfs, zfs and xfs, and server-side copy for network filesystems such as NFS.

This behaviour is in line with Boost.Filesystem and libstdc++ (where I implemented this some time ago). It's also the default behaviour of the standard file copy operations in .NET Core, Java, and (hopefully) soon python, and in the cp program from GNU coreutils.

I don't have the opportunity to test this on FreeBSD, but I think the FreeBSD folks will want a look at this. Sadly I don't know any by name, perhaps someone could CC the relevant person(s).

As for the LIBCXX_USE_COPY_FILE_RANGE cmake check, I was unsure where to put it, and this is probably not the desired location.

Lastly I am not sure if the largefile handling (see usage of off_t) is correct.

[github-actions]

Thank you for submitting a Pull Request (PR) to the LLVM Project!

This PR will be automatically labeled and the relevant teams will be notified.

If you wish to, you can add reviewers by using the "Reviewers" section on this page.

If this is not working for you, it is probably because you do not have write permissions for the repository. In which case you can instead tag reviewers by name in a comment by using @ followed by their GitHub username.

If you have received no comments on your PR for a week, you can request a review by "ping"ing the PR by adding a comment “Ping”. The common courtesy "ping" rate is once a week. Please remember that you are asking for valuable time from other developers.

If you have further questions, they may be answered by the LLVM GitHub User Guide.

You can also ask questions in a comment on this PR, on the LLVM Discord or on the forums.

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: Jannik Glückert (Jannik2099)

Changes

Hi,

this optimizes std::filesystem::copy_file to use the copy_file_range syscall (Linux and FreeBSD) when available. It allows for reflinks on filesystems such as btrfs, zfs and xfs, and server-side copy for network filesystems such as NFS.

This behaviour is in line with Boost.Filesystem and libstdc++ (where I implemented this some time ago). It's also the default behaviour of the standard file copy operations in .NET Core, Java, and (hopefully) soon python, and in the cp program from GNU coreutils.

I don't have the opportunity to test this on FreeBSD, but I think the FreeBSD folks will want a look at this. Sadly I don't know any by name, perhaps someone could CC the relevant person(s).

As for the LIBCXX_USE_COPY_FILE_RANGE cmake check, I was unsure where to put it, and this is probably not the desired location.

Lastly I am not sure if the largefile handling (see usage of off_t) is correct.

---

Full diff: https://github.com/llvm/llvm-project/pull/109211.diff

2 Files Affected:

• (modified) libcxx/src/CMakeLists.txt (+6)
• (modified) libcxx/src/filesystem/operations.cpp (+59-1)

diff --git a/libcxx/src/CMakeLists.txt b/libcxx/src/CMakeLists.txt
index 48c5111a0acbf6..3f97d3e730a42c 100644
--- a/libcxx/src/CMakeLists.txt
+++ b/libcxx/src/CMakeLists.txt
@@ -173,6 +173,12 @@ if (APPLE AND LLVM_USE_SANITIZER)
   endif()
 endif()
 
+include(CheckCXXSymbolExists)
+check_cxx_symbol_exists("copy_file_range" "unistd.h" LIBCXX_USE_COPY_FILE_RANGE)
+if(LIBCXX_USE_COPY_FILE_RANGE)
+  list(APPEND LIBCXX_COMPILE_FLAGS "-D_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE")
+endif()
+
 split_list(LIBCXX_COMPILE_FLAGS)
 split_list(LIBCXX_LINK_FLAGS)
 
diff --git a/libcxx/src/filesystem/operations.cpp b/libcxx/src/filesystem/operations.cpp
index d771f200973528..42d16c0546ae10 100644
--- a/libcxx/src/filesystem/operations.cpp
+++ b/libcxx/src/filesystem/operations.cpp
@@ -32,6 +32,7 @@
 #  include <dirent.h>
 #  include <sys/stat.h>
 #  include <sys/statvfs.h>
+#  include <sys/types.h>
 #  include <unistd.h>
 #endif
 #include <fcntl.h> /* values for fchmodat */
@@ -178,8 +179,35 @@ void __copy(const path& from, const path& to, copy_options options, error_code*
 namespace detail {
 namespace {
 
+#if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE)
+bool copy_file_impl_copy_file_range(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
+  size_t count = read_fd.get_stat().st_size;
+  // a zero-length file is either empty, or not copyable by this syscall
+  // return early to avoid the syscall cost
+  if (count == 0) {
+    ec = {EINVAL, generic_category()};
+    return false;
+  }
+  do {
+    ssize_t res;
+    // do not modify the fd positions as copy_file_impl_sendfile may be called after a partial copy
+    off_t off_in  = 0;
+    off_t off_out = 0;
+
+    if ((res = ::copy_file_range(read_fd.fd, &off_in, write_fd.fd, &off_out, count, 0)) == -1) {
+      ec = capture_errno();
+      return false;
+    }
+    count -= res;
+  } while (count > 0);
+
+  ec.clear();
+
+  return true;
+}
+#endif
 #if defined(_LIBCPP_FILESYSTEM_USE_SENDFILE)
-bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
+bool copy_file_impl_sendfile(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
   size_t count = read_fd.get_stat().st_size;
   do {
     ssize_t res;
@@ -194,6 +222,36 @@ bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_cod
 
   return true;
 }
+#endif
+
+#if defined(_LIBCPP_FILESYSTEM_USE_SENDFILE)
+bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
+  bool has_copied = false;
+#  if defined(_LIBCPP_FILESYSTEM_USE_COPY_FILE_RANGE)
+  has_copied = copy_file_impl_copy_file_range(read_fd, write_fd, ec);
+  if (has_copied) {
+    return true;
+  }
+  // EINVAL: src and dst are the same file (this is not cheaply
+  // detectable from userspace)
+  // EINVAL: copy_file_range is unsupported for this file type by the
+  // underlying filesystem
+  // ENOTSUP: undocumented, can arise with old kernels and NFS
+  // EOPNOTSUPP: filesystem does not implement copy_file_range
+  // ETXTBSY: src or dst is an active swapfile (nonsensical, but allowed
+  // with normal copying)
+  // EXDEV: src and dst are on different filesystems that do not support
+  // cross-fs copy_file_range
+  // ENOENT: undocumented, can arise with CIFS
+  // ENOSYS: unsupported by kernel or blocked by seccomp
+  if (ec.value() != EINVAL && ec.value() != ENOTSUP && ec.value() != EOPNOTSUPP && ec.value() != ETXTBSY &&
+      ec.value() != EXDEV && ec.value() != ENOENT && ec.value() != ENOSYS) {
+    return false;
+  }
+  ec.clear();
+#  endif
+  return copy_file_impl_sendfile(read_fd, write_fd, ec);
+}
 #elif defined(_LIBCPP_FILESYSTEM_USE_COPYFILE)
 bool copy_file_impl(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
   struct CopyFileState {

[mstorsjo]

I don't have the opportunity to test this on FreeBSD, but I think the FreeBSD folks will want a look at this. Sadly I don't know any by name, perhaps someone could CC the relevant person(s).

CC @emaste

@Jannik2099 Note that the CI environment does contain a FreeBSD setup, which shows a failure (a somewhat boring one):

/home/buildkite/.buildkite-agent/builds/freebsd-test-1/llvm-project/libcxx-ci/libcxx/src/filesystem/operations.cpp:183:6: error: unused function 'copy_file_impl_copy_file_range' [-Werror,-Wunused-function]
  183 | bool copy_file_impl_copy_file_range(FileDescriptor& read_fd, FileDescriptor& write_fd, error_code& ec) {
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

(There are also a couple other similar failing cases in the same CI run on buildkite.)

[Jannik2099]

Thanks!

Yeah, I forgot to slap on -Werror yesterday, will fix later.

Another note, how TOCTOU-safe should this be? Right now I fstat both in the copy_file_range and in the sendfile impl. If the file size changes inbetween and copy_file_range fails halfway through, this may lead to inconsistent output.

[Jannik2099]

So I looked at the FreeBSD failure again...

FreeBSD does have sendfile, but it does NOT have sys/sendfile.h, which means libc++ has been doing userspace copies on FreeBSD all this time!

I'll think of a more robust check.

[Jannik2099]

FreeBSD does have sendfile, but it does NOT have sys/sendfile.h, which means libc++ has been doing userspace copies on FreeBSD all this time!

so turns out that was on purpose, because FreeBSD sendfile cannot send to files - I had assumed it was one of those "ancient unix syscalls that made it's way into every OS".

I'll have to restructure this a bit then.

Unrelated, I also took the opportunity to support copying special zero-length files on linux (such as those in /proc). Funnily enough, we discovered the same deficiency in libstdc++ when I was adding copy_file_range support there.