Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Program cross compiled for mips-unknown-linux-musl crashes with SIGILL #37507

Open
japaric opened this issue Nov 1, 2016 · 10 comments
Open

Program cross compiled for mips-unknown-linux-musl crashes with SIGILL #37507

japaric opened this issue Nov 1, 2016 · 10 comments
Labels
C-bug Category: This is a bug. O-MIPS Target: MIPS processors

Comments

@japaric
Copy link
Member

japaric commented Nov 1, 2016

cc @Bigomby @alexcrichton

Originally reported as japaric/rust-cross#27

Original report below:


Hi, I'm trying to cross compile a Rust app for LEDE (fork of OpenWRT) but I'm stuck.

The application is just an hello_world example:

fn main() {
    println!("Hello, world");
}

These are the steps I performed:

  1. In first place I downloaded the LEDE SDK and compiled successfully the image and the toolchain for my device (mips-openwrt-linux-musl-gcc).

  2. Install the Rust std using rustup add mips-unknown-linux-musl.

  3. Create a .cargo/config file in my project with the following content:

    [target.mips-unknown-linux-musl]
    linker = "mips-openwrt-linux-musl-gcc"
    
  4. Build the app using cargo build --release --target=mips-unknown-linux-musl.

  5. Deploy the executable to the device using scp.

  6. Got Illegal instruction

On target

$ ldd hello_world
/lib/ld-musl-mips-sf.so.1 (0x558c2000)                                                 
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77cc4000)                                       
libc.so => /lib/ld-musl-mips-sf.so.1 (0x558c2000)  

$ ldd --version
musl libc (mips-sf)
Version 1.1.15
Dynamic Program Loader
Usage: ldd [options] [--] pathname

On host

$ mips-openwrt-linux-musl-gcc -v
Reading specs from /opt/lede/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/bin/../lib/gcc/mips-openwrt-linux-musl/5.4.0/specs
COLLECT_GCC=mips-openwrt-linux-musl-gcc
COLLECT_LTO_WRAPPER=/opt/lede/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/bin/../libexec/gcc/mips-openwrt-linux-musl/5.4.0/lto-wrapper
Target: mips-openwrt-linux-musl
Configured with: /home/diego/source/build_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/gcc-5.4.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 5.4.0 r2032' --prefix=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mips-openwrt-linux-musl --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-float=soft --with-gmp=/home/diego/source/staging_dir/host --with-mpfr=/home/diego/source/staging_dir/host --with-mpc=/home/diego/source/staging_dir/host --disable-decimal-float --with-mips-plt --with-diagnostics-color=auto-if-env --disable-libssp --enable-__cxa_atexit --with-headers=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/home/diego/source/staging_dir/toolchain-mips_mips32_gcc-5.4.0_musl-1.1.15/lib --enable-lto --with-libelf=/home/diego/source/staging_dir/host
Thread model: posix
gcc version 5.4.0 (LEDE GCC 5.4.0 r2032)

Disabling jemalloc

If I use rust nightly and disable jemalloc as following:

#![feature(alloc_system)]
extern crate alloc_system;

fn main() {
    println!("Hello, world");
}

Then it works! But as I keep adding code the application eventually crashes with another Illegal instruction. For example:

Works

TcpStream::connect("130.206.193.115:80").unwrap();

Does not work

TcpStream::connect("google.com:80").unwrap();'

I don't know what I'm doing wrong. Any help will be appreciated.

@japaric
Copy link
Member Author

japaric commented Nov 1, 2016

@Bigomby

I don't know what I'm doing wrong.

The only thing that I think you could be doing wrong is compiling the binary for
MIPS (big endian) but running it on a MIPSel (little endian) device but then not
even the "Hello, world!" program that uses system_alloc would work so I'm
going to discard that possibility.

I'd like to check a few things:

  • What device are you running the binaries on? I'd like to know known the exact
    architecture and CPU model but if you can at least tell me the brand and model
    of the router/board/device you are using, that would be a great start.
  • Could you point me to the LEDE SDK, its URL, that you are using?
  • Do you have access to GDB on the target device? Any chance you could get a
    backtrace/disassemble of a program at the moment that it crashes?

My guess, since C code is involved in all the crashes, is that we are building
our binary releases of std with C compiler flags that generate instructions
that are not available on your device. We'll hopefully only need to tweak those
and then a new nightly release of std should work for you.

If it's not the C side, then we may need to tweak our LLVM codegen options.

Or it could be a bad libc definition.

@Bigomby
Copy link

Bigomby commented Nov 1, 2016

  • The device is a Huawei EchoLife HG556a.

  • I got the SDK from this repo https://github.com/lede-project/source (commit 76847c7)

  • I got this when running with gdb

    #![feature(alloc_system)]
    extern crate alloc_system;
    
    use std::net::TcpStream;
    
    fn main() {
      TcpStream::connect("google.com:80").unwrap();
    }
    (gdb) r
    Starting program: /root/kinton-yeelight 
    warning: Unable to find dynamic linker breakpoint function.
    GDB will be unable to debug shared library initializers
    and track explicitly loaded dynamic code.
    
    Program received signal SIGILL, Illegal instruction.
    0x5556269c in ?? ()
    (gdb) bt
    #0  0x5556269c in ?? ()
    #1  0x55565fdc in _$LT$str$u20$as$u20$std..net..addr..ToSocketAddrs$GT$::to_socket_addrs::h7d1fe6bb64ce9c00 ()
    #2  0x55560cec in _ftext ()
    

@Bigomby
Copy link

Bigomby commented Nov 1, 2016

This is what happends when using jemalloc

fn main() {
    println!("Hello world");
}
(gdb) r
Starting program: /root/hello_world 
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGILL, Illegal instruction.
je_arena_mapbits_binind_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:801
801     /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h: No such file or directory.
(gdb) bt
#0  je_arena_mapbits_binind_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:801
#1  je_arena_salloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1317
#2  je_isalloc () at include/jemalloc/internal/jemalloc_internal.h:924
#3  je_iallocztm () at include/jemalloc/internal/jemalloc_internal.h:937
#4  a0ialloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:321
#5  je_a0malloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:336
#6  je_arena_tdata_get_hard () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:519
#7  0x5558f7dc in je_arena_tdata_get () at include/jemalloc/internal/jemalloc_internal.h:818
#8  je_decay_ticker_get () at include/jemalloc/internal/jemalloc_internal.h:842
#9  je_arena_decay_ticks () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1253
#10 je_arena_decay_tick () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1264
#11 je_arena_malloc_large () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:2476
#12 0x55590ad4 in je_arena_malloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/arena.h:1287
#13 je_arena_palloc () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:2601
#14 0x555b2420 in je_ipallocztm () at include/jemalloc/internal/jemalloc_internal.h:982
#15 je_tcache_create () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/tcache.c:328
#16 0x555849b0 in je_tcache_get () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/include/jemalloc/internal/tcache.h:236
#17 je_imalloc () at include/jemalloc/internal/jemalloc_internal.h:954
#18 imallocx_no_prof () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:2135
#19 mallocx () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:2164
#20 0x5557bb30 in alloc_jemalloc::imp::__rust_allocate () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc_jemalloc/lib.rs:106
#21 0x5557aaa4 in alloc::heap::allocate () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc/heap.rs:59
#22 alloc::raw_vec::{{impl}}::with_capacity<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/liballoc/raw_vec.rs:96
#23 collections::vec::{{impl}}::with_capacity<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/vec.rs:356
#24 collections::slice::hack::to_vec<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:160
#25 collections::slice::{{impl}}::to_vec<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:1161
#26 collections::slice::{{impl}}::to_owned<u8> () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/slice.rs:1281
#27 collections::str::{{impl}}::to_owned () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libcollections/str.rs:167
#28 0x5556c934 in std::rt::lang_start () at /buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/obj/../src/libstd/rt.rs:50
#29 0x55561cd8 in main ()
warning: GDB can't find the start of the function at 0x77f78acf.

    GDB is unable to find the start of the function at 0x77f78acf
and thus can't determine the size of that function's stack frame.
This means that GDB may be unable to access that stack frame, or
the frames below it.
    This problem is most likely caused by an invalid program counter or
stack pointer.
    However, if you think GDB should simply search farther back
from 0x77f78acf for code which looks like the beginning of a
function, you can increase the range of the search using the `set
heuristic-fence-post' command.

@alexcrichton
Copy link
Member

A SIGILL probably means that we're miscompiling jemalloc somehow (maybe some weird codegen option) but if it still faults with the system allocator then it's likely not exclusively a jemalloc problem.

@Bigomby
Copy link

Bigomby commented Nov 4, 2016

I tried to run the program on an OM2P (MIPS device) and it's working fine, even with jemalloc. OM2P uses an Atheros AR7240 so the issue seems to be something more specific to Broadcom 6358.

UPDATE: It crashes also when using glibc instead musl on the Huawei EchoLife HG556a.

mips-openwrt-linux-gnu-gcc -v

Reading specs from /opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/../lib/gcc/mips-openwrt-linux-gnu/6.2.0/specs
COLLECT_GCC=/opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/mips-openwrt-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/opt/l/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/bin/../libexec/gcc/mips-openwrt-linux-gnu/6.2.0/lto-wrapper
Target: mips-openwrt-linux-gnu
Configured with: /home/diego/lede/build_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/gcc-6.2.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 6.2.0 r2065' --prefix=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24 --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mips-openwrt-linux-gnu --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-float=soft --with-gmp=/home/diego/lede/staging_dir/host --with-mpfr=/home/diego/lede/staging_dir/host --with-mpc=/home/diego/lede/staging_dir/host --disable-decimal-float --with-mips-plt --with-diagnostics-color=auto-if-env --enable-libssp --enable-__cxa_atexit --with-headers=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/home/diego/lede/staging_dir/toolchain-mips_mips32_gcc-6.2.0_glibc-2.24/lib --enable-lto --with-libelf=/home/diego/lede/staging_dir/host
Thread model: posix
gcc version 6.2.0 (LEDE GCC 6.2.0 r2065) 

@japaric
Copy link
Member Author

japaric commented Nov 7, 2016

Interesting. Then this could be due to codegen options related to "architecture level" (-march) or CPU optimizations (-mcpu) in the sense that the current rustc options generate instructions that are not available on some of these devices. Unfortunately, I don't know the MIPS ISA well enough to confirm this.

@sanxiyn sanxiyn added the O-MIPS Target: MIPS processors label Jan 10, 2017
@jcowgill
Copy link
Contributor

This is probably due to the ISA level. The HG556a seems to use MIPS32r1 and the OM2P uses MIPS32r2. I think LLVM defaults to MIPS32r2 unless told otherwise. It would be interesting to get a disassembly at the SIGILL, then I could tell you for certain that this was the problem.

@Mark-Simulacrum Mark-Simulacrum added the C-bug Category: This is a bug. label Jul 26, 2017
@matan129
Copy link

matan129 commented Sep 17, 2017

I'm experiencing this issue too (on a Broadcom chip, though).
The chip is BCM63168. I have a working OpenWRT toolchain for it. Using mips-unknown-linux-musl as a target with rustc\Cargo works for a simple hello world application, but for anything more complicated I get an illegal instruction.

@Jedzia
Copy link

Jedzia commented Mar 25, 2019

I have suffered similar losses in the battle for Illegal Instructions on mips. My research so far and i hope this helps:

Minimal build for VU+ DUO2 mipsel-unknown-linux-gnu with Rust on Windows

To save you some hours i describe the process how i build a hello world without
ending up with Illegal instruction errors on the BCM7425B2 STB platform.

Target Specs

root@vuduo2:~# cat /proc/cpuinfo
system type             : BCM7425B2 STB platform
machine                 : Unknown
processor               : 0
cpu model               : Broadcom BMIPS5000 V1.1  FPU V0.1
BogoMIPS                : 864.25
cpu MHz                 : 1305.097
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : no
isa                     : mips1 mips2 mips32r1
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

processor               : 1
cpu model               : Broadcom BMIPS5000 V1.1  FPU V0.1
BogoMIPS                : 655.36
cpu MHz                 : 1305.097
wait instruction        : yes
microsecond timers      : yes
tlb_entries             : 64
extra interrupt vector  : yes
hardware watchpoint     : no
isa                     : mips1 mips2 mips32r1
ASEs implemented        :
shadow register sets    : 1
kscratch registers      : 0
core                    : 0
VCED exceptions         : not available
VCEI exceptions         : not available

What you need

To compile for the target i used Codescape GNU Tools 2018.09-03 Binaries
https://codescape.mips.com/components/toolchain/2018.09-03/downloads.html
installed in C:\Toolchain\MIPS\mips-2016.05.

Rust components used:

E:\Projects\Rust\vuduo2\hello-vuduo>rustup show
Default host: x86_64-pc-windows-gnu

installed targets for active tool chain
--------------------------------------

mipsel-unknown-linux-gnu
x86_64-pc-windows-gnu

active tool chain
----------------

nightly-x86_64-pc-windows-gnu (default)
rustc 1.35.0-nightly (719b0d984 2019-03-13)

Install "The sysroot manager that lets you build and customize std"
xargo.

cargo install xargo

Setup

Environment

Paths are specific to my setup and locations. This can vary in your case and you
have to adjust them.

Add your cross-compiler to the search path

set PATH=C:\Toolchain\MIPS\mips-2016.05\bin;%PATH%

Specify the compiler setup to use:

set CC=mips-linux-gnu-gcc
set CFLAGS_mipsel-unknown-linux-gnu=-EL
set AR_mipsel-unknown-linux-gnu=mips-linux-gnu-ar
  • where CC specifies the compiler you use to build the target and std crate binaries
  • CFLAGS_mipsel-unknown-linux-gnu the specific compiler switches to use. In this case
    -EL tells the compiler to build code for little endian processors (the el in mipsel).
    Other compilers may come for a specific architecture and are named mipsel-linux-gnu-gcc,
    but in my case the compiler is mips-linux-gnu-gcc and can output in both endianess. You
    have to consider this when using a different tool chain.
    To learn more about the capabilities of the compiler use mips-linux-gnu-gcc --target-help.
  • The AR_mipsel-unknown-linux-gnu binary util is used to create, modify, and extract from
    archives (libraries).

Build files

Any minimal Cargo.toml. Nothing special:

[package]
name = "hello-vuduo"
version = "0.1.0"
authors = ["Jedzia <jed69@gmx.de>"]
edition = "2018"

Build file for Xargo Xargo.toml:

[dependencies]
std = {features = ["panic_unwind"]}

[build]
target = "mipsel-linux-gnu-gcc"
  • std: The std dependency is what you want to build on your own.
    This is the cause of Illegal instruction errors on mips32r1 targets as the
    upstream target from the Rust repository is built for mips32r2 with fpxx and
    nooddspreg enabled. This change was done on Mar 15, 2018:
    bump mipsel isa leval and enable fpxx #48874

  • The [build] target specifies the default architecture to build.
    Can also be specified at the command-line with

      cargo build --target=mipsel-unknown-linux-gnu
      xargo build --target=my-unknown-system-dos
    

Cargo configuration

Local cargo configuration .cargo/config:

[build]
target = "mipsel-unknown-linux-gnu"

[target.mipsel-unknown-linux-gnu]
rustflags = [
  "-C", "link-arg=-EL",
  "-C", "target-cpu=mips32",
  "-C", "target-feature=+mips32,-mips32r2,-fpxx,-nooddspreg",
]

ar = "mips-linux-gnu-ar"
linker = "mips-linux-gnu-gcc"
runner = "run.bat"  
  • The [build] target specifies the default architecture to build (see above).
  • Under [target.mipsel-unknown-linux-gnu] we specify the flags used for creating the binary
    and linking it. -C is forwarded to the rustc-compiler and indicates a code-generation option.
    rustc -C help will list all options available. The default target specs for the current rustc
    compiler can be listed with:
    rustc -Z unstable-options --print target-spec-json --target mipsel-unknown-linux-gnu.
    The used options in detail:
    • link-arg=-EL is daisy-chained to the linker(mips-linux-gnu-gcc): little endian output is generated.
    • target-cpu=mips32 for use with our good old mips32r1. A list can be retrieved with
      rustc --print target-cpus. We are building for mips so with our target:
      rustc --target mipsel-unknown-linux-gnu --print target-cpus.
    • target-feature: For a list of available options use rustc --print target-features or in our case:
      rustc --target mipsel-unknown-linux-gnu -C target-cpu=mips32 -C target-feature=help.
      • +mips32: Enable Mips32 ISA Support.
      • -mips32r2: Disable Mips32r2 ISA Support ... illegal instruction.
      • -fpxx: Disable support for FPXX. The FPXX extension mandates that all code must execute correctly
        when run using 32-bit or 64-bit registers. The code can be interlinked with either FP32 or FP64,
        but not both.
      • -nooddspreg: Disable odd numbered single-precision registers.
  • ar: binary util used to create, modify, and extract from archives (libraries). Use our target
    specific one.
  • linker: mips-linux-gnu-gcc is used as linker in conjunction with the specified
    link-arg=-EL flag to produce little endian binaries.
  • runner: What (x/c)argo uses to execute the run command. The first command-line parameter is
    the relative path of your binary. I use a combination of scp to the target-box or copy to a network
    share and ssh to run it remotely.
    Something like:
    run.bat:
    cp %1 \VUDUO2/Harddisk/jedzia/rust
    ssh jedzia@vuduo2 /media/hdd/jedzia/rust/hello-vuduo

I was able to successfully compile and run with the fpxx and nooddspreg options enabled
("-C", "target-feature=+mips32,-mips32r2,+fpxx,+nooddspreg").
Maybe they are not necessary for
the VU+ Duo2. But i can't prove this as i only compile a little hello world with floating point
arithmetic in it. When it fails you have to tweak it for your target. I have disabled these options
as they are part of the #48874 change set (see above).

Compiling

run

xargo build

or if you have set up a runner in .cargo/config for remote execution

xargo run 

to see it work.

and

cargo build

to see it fail again with illegal instruction as it is compiled against the Rust
mipsel-unknown-linux-gnu target and not our custom one built by xargo.

@Jedzia
Copy link

Jedzia commented Mar 26, 2019

I cant withstand not to add an example: https://github.com/Jedzia/RokerDepp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-bug Category: This is a bug. O-MIPS Target: MIPS processors
Projects
None yet
Development

No branches or pull requests

8 participants