Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Untar'ing archive to VirtioFS volume fails to create symlinks #6277

Open
3 tasks done
Chuxel opened this issue Apr 12, 2022 · 17 comments
Open
3 tasks done

Untar'ing archive to VirtioFS volume fails to create symlinks #6277

Chuxel opened this issue Apr 12, 2022 · 17 comments

Comments

@Chuxel
Copy link

Chuxel commented Apr 12, 2022

  • I have tried with the latest version of Docker Desktop
  • I have tried disabling enabled experimental features
  • I have uploaded Diagnostics
  • Diagnostics ID: EAE618C2-8AA2-4308-9BD0-9FD561232A2A/20220412151827

Expected behavior

Un-tar'ing an archive with symlinks in it to a VirtioFS file share location results in them being created.

Actual behavior

tar reports permissions errors and the files end up getting created as zero byte files with no allowed permissions.

Example:

root ➜ /test $ tar -xf /home/node/.devcontainer-cache/cache.tar
tar: ./extensions/markdown-math/node_modules/@iktakahiro/markdown-it-katex/node_modules/.bin/katex: Cannot open: Permission denied
tar: ./extensions/markdown-math/node_modules/.bin/katex: Cannot open: Permission denied
tar: ./extensions/ipynb/node_modules/.bin/uuid: Cannot open: Permission denied
...

Then

root ➜ /test $ ls -la extensions/markdown-math/node_modules/@iktakahiro/markdown-it-katex/node_modules/.bin
ls: extensions/markdown-math/node_modules/@iktakahiro/markdown-it-katex/node_modules/.bin/katex: No such file or directory
total 0
drwxr-xr-x 3 root root 96 Apr 12 15:04 .
drwxr-xr-x 3 root root 96 Apr 12 15:04 ..
---------- 1 root root  0 Apr 12 15:04 katex

In this example, katex is a symlink in the archive.

Information

  • macOS Version: 12.3.1 (Monterey)
  • Intel chip or Apple chip: Intel
  • Docker Desktop Version: 4.7.0

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0031: does the Docker API work?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0017: can a VM be started?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0007: is the backend responding?
[PASS] DD0014: are the backend processes running?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0012: is the VM networking working?
[PASS] DD0032: do Docker networks overlap with host IPs?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0019: is the com.docker.vmnetd process responding?
[PASS] DD0033: does the host have Internet access?
No fatal errors detected.

Steps to reproduce the behavior

Here's the repro:

  1. Enable "Enable VirtioFS accelerated directory sharing"
  2. docker run -it --rm -v $(pwd):/test mcr.microsoft.com/vscode/devcontainers/repos/microsoft/vscode:branch-main bash
  3. In the container:
    cd /test
    tar -xf /home/node/.devcontainer-cache/cache.tar
    ls -la extensions/markdown-math/node_modules/@iktakahiro/markdown-it-katex/node_modules/.bin
    

This does not repro when VirtioFS is disabled, so this is specifically feedback for this experiment.

@guodong000
Copy link

sed -i in VirtioFs volume also report permission error: sed: couldn't open temporary file ./sedC49Su9: Permission denied

sed -i create an temporary file like sedxxxxxx, but the file permission is incorrect.

---------- 1 root root    0 May 13 16:15 sedC49Su9

And I try strace sed -i 's/h/H/g' ./hello

Output:

...
umask(0700)                             = 022
getpid()                                = 493
openat(AT_FDCWD, "./sed6Y6dds", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EACCES (Permission denied)
umask(022)                              = 0700
write(2, "sed: ", 5sed: )                    = 5
write(2, "couldn't open temporary file ./s"..., 59couldn't open temporary file ./sed6Y6dds: Permission denied) = 59
write(2, "\n", 1
)                       = 1
exit_group(4)                           = ?
+++ exited with 4 +++

And I try to disable [Experimental Features] -> [Enable VirtioFs accelerated directory sharing], everything works fine.

So weird.

  • macOS Version: 12.3.1 (21E258)
  • Intel chip or Apple chip: Intel chip
  • Docker Desktop Version: 4.8.1 (78998)

@docker-robott
Copy link
Collaborator

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@llamasoft
Copy link

For tar, this only appears to happen when the symlink target lives in a different directory. Here's a shell script that provides a minimal reproducible example. Note that this only works when the directory is a volume mounted from the host.

# Create the directory structure that triggers the bug.
mkdir demo
touch demo/src
mkdir demo/subdir
( cd demo/subdir && ln -s ../src ./dst; )

# Create the archive and remove the source files.
tar -czvf demo.tgz ./demo
rm -rf ./demo

# Attempt to unpack the archive.
tar -xvf demo.tgz

The output will look something like this:

root@f1b80959d15c:~/volume# tar -tvf demo.tgz 
drwxr-xr-x root/root         0 2022-08-17 23:42 ./demo/
drwxr-xr-x root/root         0 2022-08-17 23:42 ./demo/subdir/
lrwxr-xr-x root/root         0 2022-08-17 23:42 ./demo/subdir/dst -> ../src
-rw-r--r-- root/root         0 2022-08-17 23:42 ./demo/src

root@f1b80959d15c:~/volume# tar -xvf demo.tgz
./demo/
./demo/subdir/
./demo/subdir/dst
tar: ./demo/subdir/dst: Cannot open: Permission denied
./demo/src
tar: Exiting with failure status due to previous errors

The extracted result is so broken that ls will throw missing file errors:

root@f1b80959d15c:~/volume# ls -al demo/subdir
ls: demo/subdir/dst: No such file or directory
total 0
drwxr-xr-x 3 root root  96 Aug 17 23:42 .
drwxr-xr-x 4 root root 128 Aug 17 23:42 ..
---------- 1 root root   0 Aug 17 23:47 dst

@llamasoft
Copy link

Ah! The root cause appears to be openat() with strange permission.

When unpacking a symlink, tar first creates a symlink placeholder, creates the target, then replaces the placeholder with a real symlink.
That normally looks like this:

openat(AT_FDCWD, "./demo/subdir/dst", O_WRONLY|O_CREAT|O_EXCL, 000) = 4
...
openat(AT_FDCWD, "./demo/src", O_WRONLY|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK|O_CLOEXEC, 0600) = 4
...
unlinkat(AT_FDCWD, "./demo/subdir/dst", 0) = 0
symlinkat("../src", AT_FDCWD, "./demo/subdir/dst") = 0

However, on VirtioFS, calling openat() with all-zero permissions fails with "Permission denied" and creates a bugged file.
I've created a minimal C example that will allow you to easily replicate the issue:

#include <stdio.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>

int main(int argc, char **argv) {
    char *file_name = (argc > 1 ? argv[1] : "demo");
    int rtn = openat(AT_FDCWD, file_name, O_RDWR|O_CREAT|O_EXCL, 000);
    if ( rtn == -1 ) {
        printf("rtn = %d; errno = %d, %s\n", rtn, errno, strerror(errno));
        return errno;
    } else {
        printf("OK\n");
    }
    return 0;
}

When you create a file on VirtioFS, it fails. Anywhere else it works as expected.

root@f1b80959d15c:~# gcc  -o openat ./openat.c

root@f1b80959d15c:~# ./openat ./mounted/demo
rtn = -1; errno = 13, Permission denied

root@f1b80959d15c:~# ./openat ./volume/demo
OK

root@f1b80959d15c:~# ./openat /tmp/demo
OK

@dac-humi
Copy link

I saw a similar issue, and workaround, with sed in the Docker forum. Allegedly, a patch was created for sed:

Could this be reported upstream to untar?

@Chuxel
Copy link
Author

Chuxel commented Sep 7, 2022

/remove-lifecycle stale

This still appears to repro as described above as of Docker Desktop 4.12.0

@docker-robott
Copy link
Collaborator

There hasn't been any activity on this issue for a long time.
If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment.
If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

@dserodio
Copy link

dserodio commented Dec 6, 2022

I can still reproduce it on 4.14.1 like in the original report.

/remove-lifecycle stale

@dac-humi
Copy link

dac-humi commented Dec 6, 2022

@dserodio, have you seen?

(Edit: Experimental MacOS builds related to VirtioFS not handling permissions)

#6243 (comment)

@docker-robott
Copy link
Collaborator

There hasn't been any activity on this issue for a long time.
If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment.
If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

@groner
Copy link

groner commented Mar 6, 2023

I've run into this with abiosoft/colima (which uses Code-Hex/vz) as well as with Docker Desktop.

From looking at the APIs vz uses to set up virtiofs shares, it doesn't seem like this behavior is controllable.

https://developer.apple.com/documentation/virtualization/shared_directories
https://developer.apple.com/documentation/virtualization/vzvirtiofilesystemdevice

With VZVirtioFileSystemDevice, the framework enforces several permissions policies for shared directories:

  • The framework reads and writes files using the user ID (UID) of the effective user, which is the UID of the current user, rather than the UID of the system process.
  • The framework doesn’t allow reading or overwriting of files with permissions where the file is inaccessible to the current user.
  • The framework ignores requests from guest operating systems to change the UID or group ID (GID) of files on the host.

Unfortunately these policies (specifically the second one) break existing widely deployed software, like GNU tar.

@evs-xsarus
Copy link

/lifecycle frozen

@lukas-hofstaetter
Copy link

Possibly the same issue than recorded here: #6614

This is really a showstopper. I am developing a buildroot system and cannot use virtiofs at all because it uses these kind of symlinks during the build process.

@manarth
Copy link

manarth commented Nov 23, 2023

Still reproducible with 4.25.1

@jperrin-dojofive
Copy link

jperrin-dojofive commented Dec 5, 2023

I just tried 4.26.0 after running into this issue with 4.24.0 and I was able to successfully untar an archive containing symlinks to a VirtioFS volume using docker version 4.26.0!

@qiushixiang
Copy link

Still reproducible with 4.25.2, untar or sed was successfully when upgrade to 4.26.0!

@zyberspace
Copy link

Can confirm. Updating from docker desktop 4.25.2 to 4.26.0 fixed this issue for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests