sh: write error: Invalid argument - Centos 7 #100

jc21 · 2019-12-16T05:43:24Z

/kind bug

Description:

Similar error to #38, but different circumstances.

Operating system info:

$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

$ uname -a
Linux saturn 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ ll /proc/sys/fs/binfmt_misc
.rw-r--r-- 0 root root 16 Dec 15:06 jexec
.-w------- 0 root root 16 Dec 15:28 register
.rw-r--r-- 0 root root 16 Dec 15:05 status

$ rpm -qa | grep qemu
# nothing, there are no pre-existing qemu packages installed

Then when I run this as root:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

I get:

Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
sh: write error: Invalid argument
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
sh: write error: Invalid argument
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
sh: write error: Invalid argument
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
sh: write error: Invalid argument
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
sh: write error: Invalid argument

What am I missing? This command works on my other linux mint distro, but my CI nodes are running Centos7.

The text was updated successfully, but these errors were encountered:

junaruga · 2019-12-16T11:55:17Z

One possibility is your kernel 3.10.0-1062.9.1.el7.x86_64 (= uname -r) is too old to use the command or the option -p yes.

Does following command without -p yes work on your environment?

docker run --rm --privileged multiarch/qemu-user-static --reset

Because flags: F used for -p yes is relatively newer feature in binfmt_misc itself. I saw the option worked on Travis CI Ubuntu xenial, but did not work on Travis CI Ubuntu trusty.

Here is the document on kernel 4.10.
https://www.kernel.org/doc/html/v4.10/admin-guide/binfmt-misc.html
flags - F - fix binary is the used function for the -p yes.

junaruga · 2019-12-16T12:07:32Z

Here is the binfmt_misc document for kernel 3.10. There is no F option of flags there.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Documentation/binfmt_misc.txt?h=v3.10#n37

junaruga · 2019-12-16T14:26:02Z

Here is the alternative workflow for your environment possibly.

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

or

$ docker run --rm --privileged multiarch/qemu-user-static --reset

Above commands are just to add binfmt_misc files to your host environment.

Then for example if you want to run arm64v8/ubuntu Ubuntu ARM 64-bit (= aarch64) container, it is like this with your container "my-aarch64-ubuntu".

$ docker build --rm -t my-aarch64-ubuntu -<<EOF
FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM arm64v8/ubuntu
COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
EOF

$ docker run --rm -t my-aarch64-ubuntu uname -m
aarch64

jc21 · 2019-12-16T23:24:34Z

Thanks those steps worked. But the situation sucks.

I'd prefer not to have to include the binary in the docker images and it looks like my only option is to update to Centos 8 which is currently kernel 4.18.0 and does in fact work as expected.

jc21 · 2019-12-17T01:23:52Z

Update for anyone who really wants this to work on Centos 7:

junaruga · 2019-12-17T11:47:31Z

I'd prefer not to have to include the binary in the docker images and it looks like my only option is to update to Centos 8 which is currently kernel 4.18.0 and does in fact work as expected.

There is an alternative way not to include the binary in the docker images for your case.

That is to copy the binary qemu-*-static from the container to the host and use docker run -v host_dir:container_dir.
I referred this document to copy the binary from the container to the host.

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: 
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

$ docker create -it --name dummy multiarch/qemu-user-static:x86_64-aarch64 bash

$ docker container ls -a
CONTAINER ID        IMAGE                                       COMMAND             CREATED             STATUS              PORTS               NAMES
6ab622a76dfa        multiarch/qemu-user-static:x86_64-aarch64   "bash"              3 minutes ago       Created                                 dummy

$ docker cp dummy:/usr/bin/qemu-aarch64-static qemu-aarch64-static

$ ls qemu-aarch64-static
qemu-aarch64-static*

$ docker rm -f dummy

$ docker container ls -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ docker run --rm -t -v $(pwd)/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/ubuntu uname -m
aarch64

wanganhong · 2020-11-06T01:58:37Z

@junaruga

Hi,junaruga! I repeat your steps, but got an error at the last step.

Error detail:

$ docker run --rm -t -v $(pwd)/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/ubuntu uname -m
: error while loading shared libraries: cannot apply additional memory protection after relocation: Permission denied

My device is a virtual machine，operating system info:

$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

$ uname -a
Linux localhost.localdomain 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

zyystudio · 2020-12-14T12:51:12Z

Update for anyone who really wants this to work on Centos 7:

Install the latest unofficial mainline kernel

Select the newly installed kernel to be default in grub

Profit
+1
This is the right one.

junaruga · 2021-04-19T08:38:50Z

@wanganhong Sorry when I was checking past notification emails which I was mentioned in, I found this ticket's notification email.

: error while loading shared libraries: cannot apply additional memory protection after relocation: Permission denied

Sorry I have no idea about the message.

~~Can you check the binary file type of the qemu-aarch64-static binary file you copied to the host by the command $ file ./qemu-aarch64-static? You can see if you copied the correct aarch64 binary.~~

Sorry my mistake. The qemu-aarch64-static is not aarch64 binary but x86_64 binary. I checked the steps with podman on my Fedora 33 , where the kernel version is 5.11.7-200.fc33.x86_64 now.

$ uname -r
5.11.7-200.fc33.x86_64

$ podman --version
podman version 3.0.1

$ sudo podman run --rm --privileged multiarch/qemu-user-static:register --reset

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags:
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

$ podman create -it --name dummy multiarch/qemu-user-static:x86_64-aarch64 bash

$ podman container ls -a
CONTAINER ID  IMAGE                                                COMMAND  CREATED         STATUS   PORTS   NAMES
03c6c59f1678  docker.io/multiarch/qemu-user-static:x86_64-aarch64  bash     21 seconds ago  Created          dummy

$ podman cp dummy:/usr/bin/qemu-aarch64-static qemu-aarch64-static

$ ls qemu-aarch64-static
qemu-aarch64-static*

$ file ./qemu-aarch64-static
./qemu-aarch64-static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=30efd3930fb9519b21470b113679376f2ffbb41a, for GNU/Linux 3.2.0, stripped

$ podman rm -f dummy

$ podman container ls -a
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

$ podman run --rm -t -v $(pwd)/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/ubuntu uname -m
aarch64

junaruga · 2021-04-19T08:43:50Z

@wanganhong Or you might be able to try the way @zyystudio shared. I have not checked the links. But thanks for the info!

constanine · 2021-06-21T12:16:45Z

@junaruga Excuse me,I want to ask how to use command docker build with the same issue.
I had try use your command and at last use command like this

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: 
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

$ docker create -it --name dummy multiarch/qemu-user-static:x86_64-aarch64 bash

$ docker container ls -a
CONTAINER ID        IMAGE                                       COMMAND             CREATED             STATUS              PORTS               NAMES
6ab622a76dfa        multiarch/qemu-user-static:x86_64-aarch64   "bash"              3 minutes ago       Created                                 dummy

$ docker cp dummy:/usr/bin/qemu-aarch64-static qemu-aarch64-static

$ ls qemu-aarch64-static
qemu-aarch64-static*

$ docker rm -f dummy

$ docker container ls -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ docker build --force-rm -v $(pwd)/qemu-aarch64-static:/usr/bin/qemu-aarch64-static -t basic-centos-arm-env:1.0 .

and Dockerfile is like

FROM arm64v8/centos:7
MAINTAINER bokesoft.com

RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7; \
    yum update -y;
RUN \
    yum install -y wget; \
    yum install -y telnet; \
    yum install -y nano; \
    yum install -y vim;
	
ENV  TIME_ZONE Asia/Shanghai
RUN ln -sf /usr/share/zoneinfo/${TIME_ZONE} /etc/localtime;

RUN yum install -y kde-l10n-Chinese; \
    yum install -y glibc-common; \
    yum clean all; \
    localedef -c -f UTF-8 -i zh_CN zh_CN.utf8; \
    echo 'LANG="zh_CN.UTF-8"' > /etc/locale.conf && echo 'LC_ALL="zh_CN.UTF-8"' >> /etc/locale.conf && source /etc/locale.conf; \
    localedef -c -f UTF-8 -i zh_CN zh_CN.utf8;
ENV LANG zh_CN.UTF-8
	
RUN yum install -y vixie-cron; \
    yum -y install crontabs;

...

jc21 closed this as completed Dec 16, 2019

junaruga mentioned this issue Feb 1, 2020

"sh: write error: Invalid argument" errors when register #38

Closed

Silex mentioned this issue Mar 15, 2020

failed to load LLB: runtime execution on platform linux/arm64 not supported docker/buildx#138

Closed

junaruga mentioned this issue Apr 8, 2020

CentOS8 info multiarch/centos#11

Closed

junaruga mentioned this issue Jul 23, 2021

Unknown Host QEMU IFLA Type #59

Open

fisherxu mentioned this issue Feb 25, 2022

buildx on x86 kubeedge/kubeedge#3625

Merged

marcelstoer mentioned this issue Jul 12, 2022

Installing arm64 emulator on CentOS 7: invalid argument moby/buildkit#2957

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sh: write error: Invalid argument - Centos 7 #100

sh: write error: Invalid argument - Centos 7 #100

jc21 commented Dec 16, 2019

junaruga commented Dec 16, 2019 •

edited

Loading

junaruga commented Dec 16, 2019

junaruga commented Dec 16, 2019 •

edited

Loading

jc21 commented Dec 16, 2019

jc21 commented Dec 17, 2019

junaruga commented Dec 17, 2019

wanganhong commented Nov 6, 2020

zyystudio commented Dec 14, 2020

junaruga commented Apr 19, 2021 •

edited

Loading

junaruga commented Apr 19, 2021

constanine commented Jun 21, 2021 •

edited

Loading

sh: write error: Invalid argument - Centos 7 #100

sh: write error: Invalid argument - Centos 7 #100

Comments

jc21 commented Dec 16, 2019

junaruga commented Dec 16, 2019 • edited Loading

junaruga commented Dec 16, 2019

junaruga commented Dec 16, 2019 • edited Loading

jc21 commented Dec 16, 2019

jc21 commented Dec 17, 2019

junaruga commented Dec 17, 2019

wanganhong commented Nov 6, 2020

zyystudio commented Dec 14, 2020

junaruga commented Apr 19, 2021 • edited Loading

junaruga commented Apr 19, 2021

constanine commented Jun 21, 2021 • edited Loading

junaruga commented Dec 16, 2019 •

edited

Loading

junaruga commented Dec 16, 2019 •

edited

Loading

junaruga commented Apr 19, 2021 •

edited

Loading

constanine commented Jun 21, 2021 •

edited

Loading