make environment fails on M1 MacBook Pro

[MadsRC]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

make environment from master branch fails with error 1 on an M1 MacBook Pro running latest version of MacOS.

mrh@Mrh's-MacBook-Pro vector % make environment
Building the environment. (ENVIRONMENT_AUTOBUILD=true) This may take a few minutes...
docker build --quiet --tag docker.io/timberio/vector-dev:sha-3eadc96742a33754a5859203b58249f6a806972a --file scripts/environment/Dockerfile .
[+] Building 0.0s (12/13)                                                                                                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                                                                                                0.0s
 => => transferring dockerfile: 1.01kB                                                                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                                                                   0.0s
 => => transferring context: 34B                                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                     0.8s
 => [1/9] FROM docker.io/library/ubuntu:20.04@sha256:db8bf6f4fb351aa7a26e27ba2686cf35a6a409f65603e59d4c203e58387dc6b3                                                                                               0.0s
 => [internal] load build context                                                                                                                                                                                   0.0s
 => => transferring context: 576B                                                                                                                                                                                   0.0s
 => CACHED [2/9] RUN echo America/New York > /etc/timezone                                                                                                                                                          0.0s
 => CACHED [3/9] COPY scripts/environment/*.sh /git/vectordotdev/vector/scripts/environment/                                                                                                                        0.0s
 => CACHED [4/9] RUN cd git/vectordotdev/vector && ./scripts/environment/bootstrap-ubuntu-20.04.sh                                                                                                                  0.0s
 => CACHED [5/9] WORKDIR /git/vectordotdev/vector                                                                                                                                                                   0.0s
 => CACHED [6/9] COPY scripts/Gemfile scripts/Gemfile.lock     /git/vectordotdev/vector/scripts/                                                                                                                    0.0s
 => CACHED [7/9] COPY rust-toolchain.toml     /git/vectordotdev/vector/                                                                                                                                             0.0s
 => ERROR [8/9] RUN ./scripts/environment/prepare.sh && ./scripts/environment/setup-helm.sh                                                                                                                        50.4s
------
 > [8/9] RUN ./scripts/environment/prepare.sh && ./scripts/environment/setup-helm.sh:
#12 0.169 
#12 0.169 git config --global --add safe.directory /git/vectordotdev/vector
#12 0.183 
#12 0.183 rustup show # causes installation of version from rust-toolchain.toml
#12 0.205 Default host: aarch64-unknown-linux-gnu
#12 0.205 rustup home:  /root/.rustup
#12 0.205 
#12 0.206 info: syncing channel updates for '1.66.1-aarch64-unknown-linux-gnu'
#12 0.517 info: latest update on 2023-01-10, rust version 1.66.1 (90743e729 2023-01-10)
#12 0.517 info: downloading component 'cargo'
#12 0.799 info: downloading component 'clippy'
#12 2.695 info: downloading component 'rust-docs'
#12 3.241 info: downloading component 'rust-std'
#12 4.265 info: downloading component 'rustc'
#12 6.247 info: downloading component 'rustfmt'
#12 6.421 info: installing component 'cargo'
#12 6.888 info: installing component 'clippy'
#12 7.162 info: installing component 'rust-docs'
#12 8.744 info: installing component 'rust-std'
#12 11.36 info: installing component 'rustc'
#12 15.74 info: installing component 'rustfmt'
#12 16.14 1.66.1-aarch64-unknown-linux-gnu (overridden by '/git/vectordotdev/vector/rust-toolchain.toml')
#12 16.16 rustc 1.66.1 (90743e729 2023-01-10)
#12 16.17 rustup default "$(rustup show active-toolchain | awk '{print $1;}')"
#12 16.21 info: using existing install for '1.66.1-aarch64-unknown-linux-gnu'
#12 16.21 
#12 16.21 info: default toolchain set to '1.66.1-aarch64-unknown-linux-gnu'
#12 16.23   1.66.1-aarch64-unknown-linux-gnu unchanged - rustc 1.66.1 (90743e729 2023-01-10)
#12 16.23 
#12 16.23 info: note that the toolchain '1.66.1-aarch64-unknown-linux-gnu' is currently in use (overridden by '/git/vectordotdev/vector/rust-toolchain.toml')
#12 16.23 if [[ "$(cargo-deb --version)" != "1.41.3" ]] ; then
#12 16.23   rustup run stable cargo install cargo-deb --version 1.41.3 --force --locked
#12 16.23 fi
#12 16.23 ./scripts/environment/prepare.sh: line 8: cargo-deb: command not found
#12 16.27     Updating crates.io index
#12 49.43  Downloading crates ...
#12 49.87   Downloaded cargo-deb v1.41.3
#12 50.07   Installing cargo-deb v1.41.3
#12 50.16 warning: package `crossbeam-channel v0.5.6` in Cargo.lock is yanked in registry `crates-io`, consider running without --locked
#12 50.21 error: failed to compile `cargo-deb v1.41.3`, intermediate artifacts can be found at `/tmp/cargo-installHpwpH0`
#12 50.21 
#12 50.21 Caused by:
#12 50.21   failed to run `rustc` to learn about target-specific information
#12 50.21 
#12 50.21 Caused by:
#12 50.21   process didn't exit successfully: `/root/.cargo/bin/wrap-rustc rustc - --crate-name ___ --print=file-names --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 255)
#12 50.21   --- stderr
#12 50.21   qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
------
executor failed running [/bin/sh -c ./scripts/environment/prepare.sh && ./scripts/environment/setup-helm.sh]: exit code: 101
make: *** [environment] Error 1
mrh@Mrh's-MacBook-Pro vector % git branch
* master
mrh@Mrh's-MacBook-Pro vector % git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
mrh@Mrh's-MacBook-Pro vector % git rev-parse HEAD
68b54a9bc0ae07d916ec48e997a03f7681e54ccc

Seems like Qemu is hardcoded for x86_64, potentially because the docker base Docker image is based on x86?

Configuration

No response

Version

master

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[bruceg]

We have confirmed this bug. The problem is that the dev image is hard coded to a tag that is x86_64-only, which won't work on anything else.

[MadsRC]

We have confirmed this bug. The problem is that the dev image is hard coded to a tag that is x86_64-only, which won't work on anything else.

Thank you for verifying that for me @bruceg - I'm still not entirely comfortable with the codebase.

As I'm experimenting with contributing to Vector, I'd like to take a stab at fixing this minor thing and will try and get it working and make a PR.

The dev image you are referring to must be the ubuntu:20.04 image in the scripts/environment/Dockerfile file, if I'm not mistaken?

[bruceg]

Hmm, I think now the tag name is actually a red herring. The issue is coming at some other point. As this is running in Docker, is docker itself running in qemu emulating x86_64? It might be we're downloading the wrong Rust binaries because of confusion over that emulation.

[spencergilbert]

I'm not sure if it is emulating, it looks like it's ARM64 to me locally?