Add support to use qemu-system to run linux aarch64 binaries #153

malbarbo · 2017-11-13T16:28:09Z

Fixes #146

malbarbo · 2017-11-13T16:29:14Z

@japaric If you consider this approach viable, we can write some docs add support for other archs in another PR.

malbarbo · 2017-11-13T16:32:57Z

docker/aarch64-unknown-linux-gnu/Dockerfile

+
+COPY linux-runner /
+
+ENV CROSS_RUNNER="qemu-user" \


The default runner is qemu-user, but the user can set CROSS_RUNNER=qemu-system to change the runner.

malbarbo · 2017-11-13T17:12:36Z

docker/linux-runner

+    -kernel /qemu/vmlinuz \
+    -initrd /qemu/initrd.gz \
+    -serial tcp:127.0.0.1:4444,server,nowait \
+    -virtfs local,path=/target,security_model=mapped,mount_tag=target \


We use virtfs to share /target with the guest running in qemu. The files do not need to be explicit copied!

In stdsimd we use objdump from a procedural macro to check the assembly generated by SIMD intrinsics. objdump isn't typically installed by default in the distros that are usually loaded into qemu-sytem.

I think we could hack around this by copying whichever objdump we want to use into target/ and passing its path in an environment variable to qemu system, but maybe there would be a better way of sharing resources and binaries with the code running in qemu-system ?

qemu-system is used to run the target binaries, the build happens in the docker container, which has objdump. Does this makes sense?

I think so. So what happens if my target binary tries to start a new process that runs objdump ? Does the one from the docker container get picked up?

In this case there would be no objdump... The qemu-system image is really small, it is basically a busybox rootfs... If this is necessary (sorry if I misunderstood you), I think you could put extra binaries in /target and use passthrough to pass env variable to inform where the binaries are.

I see, that's exactly what I meant in my first comment. Once this is merged I'll try to get it to work.

malbarbo · 2017-11-13T17:12:45Z

docker/linux-runner

+begin=no
+
+# execute the program and then print the exit code
+echo "echo $begin_tag; $prog; echo $end_tag \$?" | nc -q -1 127.0.0.1 4444 | while read LINE; do


We can run $prog directly on the guest!

malbarbo · 2017-11-14T14:05:05Z

Using CROSS_RUNNER is simpler, but we can consider using CROSS_TARGET_..._RUNNER, like cargo does.

japaric

Really impressive, @malbarbo. 👏

One of the concerns I had about using qemu-system was the wait time required for the VM boot process. I was expecting that a new VM would have to be booted per executed binary. But with your approach the VM is booted once per Cross invocation even if several binaries are going to be executed (e.g. if there are several integration tests in the tests folder) during that invoctaion. Very good.

Regarding the UI, I'm wondering if it would be better to have a per target setting in Cross.toml instead of a single CROSS_RUNNER variable. Something like this:

# Cross.toml
[target.aarch64-unknown-linux-gnu]
runner = "qemu-system"

Eventually we want to support changing the settings through environment variables (cf. #147) so something like CROSS_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUNNER=qemu-system would also work.

An advantage of doing this is that we can error earlier when an invalid runner is selected (before starting a Docker container), rather than erroring on each binary invocation -- which I think is what happens with the current implementation.

I'm also wondering if runner is the right term for this. That sounds like we are directly executing the binary using $runner (because that's how Cargo runners work and they existed first) but there's more logic involved when, for example, qemu-system is used. Perhaps emulation would be better? At least it won't cause confusion with the Cargo runner feature, I think.

Before rubber stamping this I'd like to see how it would integrate with #131 which I have yet to review.

Finally, before landing this we should document the downsides and advantages of using qemu-system instead of qemu-user. For example with qemu-system you can't modify the host filesystem (that's probably a good thing though) or invoke binaries in the host $PATH (what @gnzlbg mentioned about objdump) but threading support is better.

P.S. Any idea of what other architectures could get qemu-system support? I think the main requirement is getting our hands on a compiled kernel and root filesystem for the target so it's up to what we can find and what's provided by distributions.

japaric · 2017-11-18T16:06:56Z

docker/linux-runner

+# execute the program and then print the exit code
+echo "echo $begin_tag; $prog; echo $end_tag \$?" | nc -q -1 127.0.0.1 4444 | while read LINE; do
+  LINE=$(echo "$LINE" | tr -d '\r')
+  if [[ "$LINE" == "oach8aobiePheew1kahl-start" ]]; then


nit: could this be $begin_tag instead of the whole string?

japaric · 2017-11-18T16:35:59Z

docker/qemu.sh

    nice make -j$(nproc)
    make install

    # HACK the binfmt_misc interpreter we'll use expects the QEMU binary to be
    # in /usr/bin. Create an appropriate symlink
    ln -s /usr/local/bin/qemu-$arch /usr/bin/qemu-$arch-static
+    ln -s /usr/local/bin/qemu-system-$arch /usr/bin/qemu-system-$arch || true


Q: is this necessary? /usr/local/bin should be in $PATH.

It is not necessary. Thanks!

japaric · 2017-11-18T16:37:58Z

docker/linux-runner

+    -virtfs local,path=/target,security_model=mapped,mount_tag=target \
+    -append init=/init-alt > /dev/null &
+
+  sleep 5


could you add a rationale for the sleep here (probably "waiting for OS to boot")?

do you think this could be improved in the future (not necessarily in this PR) to wait until the OS finishes booting instead of waiting for a fixed amount of time?

also I think it would helpful to at least print "booting QEMU virtual machine..." before sleep to let the user know what's going on. Right now Cross kind of silently "hangs" for like 10 seconds before the binary is executed, which is a bit unsettling.

At some point it would be useful to add an option to Cross (-vv?) that lets you see the QEMU boot process on the console, for debugging purposes.

I added a mechanism to wait until the OS finishes to boot. Also add boot messages including the time to boot.

japaric · 2017-11-18T16:40:28Z

docker/qemu.sh

+         true
+         ;;
+      *)
+         echo "Invalid option $softmmu=$softmmu"


shouldn't this branch contain an exit 1?

also the message will probably be rendered as Invalid option foo=foo. It should probably look like Invalid softmmu option: foo.

malbarbo · 2017-11-22T14:05:30Z

I tried to address all pointed issues. For now I continue to use the runner term. I made each target have at least one runner, witch simplified the code but also brings the possibility of adding new generic runners (like a ssh remote runner, for example). I do not have preference for the runner term, but I think using node, native, etc as a kind of emulator is strange.

Finally, before landing this we should document the downsides and advantages of using qemu-system instead of qemu-user. For example with qemu-system you can't modify the host filesystem (that's probably a good thing though) or invoke binaries in the host $PATH (what @gnzlbg mentioned about objdump) but threading support is better.

I will need help with this. I have some difficulties writing in english, so it will take a lot of my time.

P.S. Any idea of what other architectures could get qemu-system support? I think the main requirement is getting our hands on a compiled kernel and root filesystem for the target so it's up to what we can find and what's provided by distributions.

I've tried s390x, sparc64 and arm. I think Debian provides kernel and initrd for all rust supported archs, so we can offers qemu-system for many targets!

gnzlbg · 2017-11-22T14:22:46Z

I will need help with this. I have some difficulties writing in english, so it will take a lot of my time.

If you just write "the contents" in schematic way I can go over it; feel free to use words in your mother tongue if you don't know the english ones.

I've tried s390x, sparc64 and arm.

This is more like a wishlist for stdsimd but it would be cool if we could have:

some arm build bots:
- arm-unknown-linux-gnueabihf
- armv7-unknown-linux-gnueabihf
- aarch64-unknown-linux-gnu
x86:
- i586-unknown-linux-gnu
- i686-unknown-linux-gnu
- x86_64-unknown-linux-gnu with:
  - a TBM enabled AMD CPU to test TBM
  - an AVX-512 CPU to test AVX/AVX2/AVX-512..
powerpc64le-unknown-linux-gnu to test AltiVec SIMD
mips64el-unknown-linux-gnuabi64 to test MSA SIMD
sparc64-unknown-linux-gnu to test Sparc V SIMD

malbarbo · 2017-11-25T13:56:21Z

I rewrote linux-image.sh and linux-runner to be more robust and make it easier to add other archs (I did preliminary test with 10 archs!). Instead of using debian installer kernel and initrd, we are using the normal debian packages and creating an initrd with busybox and some libs (this initrd is smaller and the boot is faster). Color is missing (adding ncurses-term package in the qemu image did not help).

@gnzlbg Thanks for your help. When the PR is ready to merge I will try to write some docs. I will add the other targets after that.

@japaric After this rewrite I think a new review from zero is need... but it is for better!

malbarbo · 2017-12-03T21:34:20Z

See the reason for this change in the last commit message.

This is a prerequisite for #166

japaric · 2018-03-10T00:12:53Z

Thanks for the patience @malbarbo @gnzlbg and sorry for the overdue review.

This looks great to me. Let's land it now and punt on documenting the internals.

bors r+

153: Add support to use qemu-system to run linux aarch64 binaries r=japaric a=malbarbo Fixes #146

bors · 2018-03-10T01:42:54Z

Timed out

japaric · 2018-03-10T01:59:18Z

bors r+

153: Add support to use qemu-system to run linux aarch64 binaries r=japaric a=malbarbo Fixes #146

japaric · 2018-03-10T02:26:23Z

@malbarbo The aarch64 build got stuck at:

Booting QEMU virtual machine with 2 cpus...

It's hard to tell what the problem was but if I had to guess I'd say that it could've been an OOM problem -- the Travis builders don't have much memory available.

bors · 2018-03-10T02:33:16Z

Canceled

gnzlbg · 2018-03-10T09:32:20Z

@japaric thank you!

Dylan-DPC-zz · 2018-10-13T17:44:37Z

@malbarbo resolve the conflict so we can move ahead with this 👍

…e starting docker

This avoids hacks to get the exit code of the executed programs and also allows the correct color for the host terminal. It increases the time to start the qemu-system emulator and to start the execution of a program on it. But the overall time to execute ripgrep tests does not changes. We use dropbear because the openssh client refuses to execute in docker with an unknown user id.

The linux-runner hangs on travis, instead of using tailf to wait for qemu, we grep the log directly. This stopped the hang.

malbarbo · 2018-10-13T20:12:37Z

Rebased and adapted to #217. We are ready to go.

Dylan-DPC-zz · 2018-10-14T22:13:20Z

bors: try

bors · 2018-10-15T00:38:25Z

try

Build succeeded

continuous-integration/travis-ci/push

Dylan-DPC-zz · 2018-10-15T19:45:28Z

bors: r+

153: Add support to use qemu-system to run linux aarch64 binaries r=Dylan-DPC a=malbarbo Fixes #146 Co-authored-by: Marco A L Barbosa <malbarbo@gmail.com>

malbarbo · 2018-10-15T20:42:16Z

Network issue https://travis-ci.org/rust-embedded/cross/jobs/441838459#L2929

Dylan-DPC-zz · 2018-10-15T21:55:03Z

bors: r-

bors · 2018-10-15T21:55:04Z

Canceled

Dylan-DPC-zz · 2018-10-15T21:55:14Z

bors: r+

153: Add support to use qemu-system to run linux aarch64 binaries r=Dylan-DPC a=malbarbo Fixes #146 Co-authored-by: Marco A L Barbosa <malbarbo@gmail.com>

bors · 2018-10-15T23:36:33Z

Build succeeded

continuous-integration/travis-ci/push

malbarbo commented Nov 13, 2017

View reviewed changes

malbarbo mentioned this pull request Nov 13, 2017

Make the ci infrastructure reusable from other crates rust-lang/libc#834

Closed

malbarbo commented Nov 13, 2017

View reviewed changes

gnzlbg mentioned this pull request Nov 16, 2017

Implement run-time feature detection for arm rust-lang/stdarch#120

Closed

japaric reviewed Nov 18, 2017

View reviewed changes

malbarbo force-pushed the qemu-system branch from d83b621 to 6ede842 Compare November 22, 2017 13:41

malbarbo force-pushed the qemu-system branch 2 times, most recently from 3772cf1 to 24f6239 Compare November 25, 2017 13:43

malbarbo mentioned this pull request Dec 3, 2017

Add qemu-system support for more targets #166

Merged

bors bot added a commit that referenced this pull request Mar 10, 2018

Merge #153

fa95101

153: Add support to use qemu-system to run linux aarch64 binaries r=japaric a=malbarbo Fixes #146

bors bot added a commit that referenced this pull request Mar 10, 2018

Merge #153

6cb23c8

153: Add support to use qemu-system to run linux aarch64 binaries r=japaric a=malbarbo Fixes #146

japaric mentioned this pull request Mar 14, 2018

Looking for maintainers for the Cross and Trust projects #183

Closed

malbarbo force-pushed the qemu-system branch 5 times, most recently from bd2a46e to 0291f42 Compare March 16, 2018 21:46

Dylan-DPC-zz added the blocked label Oct 9, 2018

Dylan-DPC-zz removed the blocked label Oct 13, 2018

malbarbo added 9 commits October 13, 2018 16:09

Add support to use qemu-system to run linux aarch64 binaries

61decda

Use target.*.runner instead of CROSS_RUNNER and validate runner befor…

4a859a8

…e starting docker

Make linux-image.sh and linux-runner more robust

88fe45a

Update linux kernel version

67ba0d6

Add CROSS_DEBUG env variable to debug qemu-system

02da030

Do not use tailf to wait for qemu

fc30f04

The linux-runner hangs on travis, instead of using tailf to wait for qemu, we grep the log directly. This stopped the hang.

Remove specific runners from source code

b826d58

Bump aarch64 qemu-system kernel to 4.9.0-8

c7e04f6

malbarbo force-pushed the qemu-system branch from 657aee7 to c7e04f6 Compare October 13, 2018 20:10

bors bot added a commit that referenced this pull request Oct 14, 2018

Try #153:

85c7814

Dylan-DPC-zz approved these changes Oct 15, 2018

View reviewed changes

bors bot added a commit that referenced this pull request Oct 15, 2018

Merge #153

68b9464

153: Add support to use qemu-system to run linux aarch64 binaries r=Dylan-DPC a=malbarbo Fixes #146 Co-authored-by: Marco A L Barbosa <malbarbo@gmail.com>

bors bot added a commit that referenced this pull request Oct 15, 2018

Merge #153

200f06c

153: Add support to use qemu-system to run linux aarch64 binaries r=Dylan-DPC a=malbarbo Fixes #146 Co-authored-by: Marco A L Barbosa <malbarbo@gmail.com>

bors bot merged commit c7e04f6 into cross-rs:master Oct 15, 2018

malbarbo deleted the qemu-system branch October 16, 2018 13:54

malbarbo mentioned this pull request Sep 4, 2019

Release 0.1.15 #302

Merged

Alexhuszagh mentioned this pull request Jul 2, 2022

Consider adding an option to run under full-system emulation #242

Closed


		COPY linux-runner /

		ENV CROSS_RUNNER="qemu-user" \

Add support to use qemu-system to run linux aarch64 binaries #153

Add support to use qemu-system to run linux aarch64 binaries #153

Conversation

malbarbo commented Nov 13, 2017

malbarbo commented Nov 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnzlbg Nov 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malbarbo commented Nov 14, 2017

japaric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

malbarbo commented Nov 22, 2017

gnzlbg commented Nov 22, 2017

malbarbo commented Nov 25, 2017

malbarbo commented Dec 3, 2017

japaric commented Mar 10, 2018

bors bot commented Mar 10, 2018

Timed out

japaric commented Mar 10, 2018

japaric commented Mar 10, 2018

bors bot commented Mar 10, 2018

Canceled

gnzlbg commented Mar 10, 2018

Dylan-DPC-zz commented Oct 13, 2018

malbarbo commented Oct 13, 2018

Dylan-DPC-zz commented Oct 14, 2018

bors bot commented Oct 15, 2018

try

Build succeeded

Dylan-DPC-zz commented Oct 15, 2018

malbarbo commented Oct 15, 2018

Dylan-DPC-zz commented Oct 15, 2018

bors bot commented Oct 15, 2018

Canceled

Dylan-DPC-zz commented Oct 15, 2018

bors bot commented Oct 15, 2018

Build succeeded

gnzlbg Nov 14, 2017 •

edited

Loading