Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bazel on Alpine Linux #1492

Closed
pspeter3 opened this issue Jul 8, 2016 · 28 comments
Closed

Bazel on Alpine Linux #1492

pspeter3 opened this issue Jul 8, 2016 · 28 comments
Labels
P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) platform: other

Comments

@pspeter3
Copy link

pspeter3 commented Jul 8, 2016

I've been thinking about developing with Bazel inside of docker containers so that way I can have sandboxing on Mac. Is it possible to use Bazel with Alpine Linux or is Ubuntu required?

@kchodorow
Copy link
Contributor

I don't think anyone's tried it, but we'd love to hear how it goes.

@kchodorow kchodorow added P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) platform: other labels Jul 11, 2016
@pspeter3
Copy link
Author

So I tried running it with this Dockerfile and get the below error. My best guess is that unzip does not work with the self extracting archive.

FROM alpine:3.4
RUN apk add --no-cache \
  bash=4.3.42-r3 \
  bash-completion=2.1-r2 \
  ca-certificates=20160104-r4 \
  g++=5.3.0-r0 \
  openjdk8=8.92.14-r1 \
  pkgconf=0.9.12-r0 \
  unzip=6.0-r1 \
  wget=1.17.1-r1 \
  zip=3.0-r4 \
  zlib-dev=1.2.8-r2
ENV JAVA_HOME /usr/lib/jvm/default-jvm
RUN wget -O bazel-installer.sh -q \
  https://github.com/bazelbuild/bazel/releases/download/0.3.0/bazel-0.3.0-installer-linux-x86_64.sh \
  && bash bazel-installer.sh \
  && rm bazel-installer.sh
bash-4.3$ bazel
/usr/local/bin/bazel: line 86: /usr/local/lib/bazel/bin/bazel-real: No such file or directory

@AmandaCameron
Copy link

I did some digging into this today, specifically to see why bazel can't build under alpine, and I believe it's failing in the protoc step, because the grpc protoc plugin isn't statically compiled, so it's failing to find a shared object it needs.

However I can't be sure, because it just dies instantly after printing "Building bazel from scratch."

@kchodorow
Copy link
Contributor

You can add set -x to compile.sh and it'll print each line as it executes.

@AmandaCameron
Copy link

Yup, added that to compile.sh and can confirm it'e dying in the protoc step. Full log can be uploaded if wanted.

@pspeter3
Copy link
Author

Is there a way to ensure the shared object is in the Alpine container?

bsilver8192 added a commit to bsilver8192/grpc that referenced this issue Aug 15, 2016
On my glibc (Debian Jessie amd64 if anybody cares) system,
<linux/unistd.h> is a strict subset of <sys/sycall.h>, which the file is
already including. musl libc doesn't provide this file, and with this
change all the C++ tests pass with musl libc.

This came up in bazelbuild/bazel#1492.
@bsilver8192
Copy link
Contributor

I have no experience with Alpine Linux or Docker, but I thought this looked interesting and got it working.

I think using the installer built for a different libc is pretty hopeless, but I did get building from scratch working with just a few tweaks.

I assume you tried to compile an older Bazel version before 168031d, because I didn't see any issues with protoc shared libraries.

third_party/grpc/protoc-gen-grpc-java-0.15.0-linux-x86_64.exe is dynamically linked (but not to libstdc++), so I followed the directions in the README.md of sgerrand/alpine-pkg-glibc to install glibc.

I've sent out a few fixes, and there's something else going on that I think is busybox env handling a -- in the arguments list differently. For now, removing the commandLineArgs.add("--"); line from src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxRunner.java makes it work on Alpine but breaks it on my Debian system.

I submitted grpc/grpc#7730 and I2bd5ec198522ea1ea19df0aaa1eb7c6977217eb3 to fix a few musl libc incompatibilities.

@perezd
Copy link
Contributor

perezd commented Jan 29, 2017

Noticing that https://bazel-review.googlesource.com/#/q/I2bd5ec198522ea1ea19df0aaa1eb7c6977217eb3 never got merged, would love to see that land, support musl/libc would be great.

@r351574nc3
Copy link

@bsilver8192 How did you get ./compile.sh to use the glibc-compat from sgerrand/alpine-pkg-glibc? I added it to my docker image, but it didn't seem to solve anything for me. FYI, I added: /usr/glibc-compat/bin:/usr/glibc-compat/sbin to the front of my PATH as well as added /usr/glibc-compat/lib to LD_LIBRARY_PATH

@r351574nc3
Copy link

I am trying to build bazel-4.3 on alpine-3.5. I am running into the following compile error I believe is connected to libc compatibility:

  /tmp/bazel_XXGIjcLb/out/execroot/bazel_build/_bin/process-wrapper -1 5 - - /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-canonical-system-headers -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fpermissive -fdata-sections '-std=c++0x' -MD -MF bazel-out/local-opt/bin/third_party/ijar/_objs/platform_utils/third_party/ijar/platform_utils.d '-frandom-seed=bazel-out/local-opt/bin/third_party/ijar/_objs/platform_utils/third_party/ijar/platform_utils.o' -iquote . -iquote bazel-out/local-opt/genfiles -iquote external/bazel_tools -iquote bazel-out/local-opt/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/ijar/platform_utils.cc -o bazel-out/local-opt/bin/third_party/ijar/_objs/platform_utils/third_party/ijar/platform_utils.o).
In file included from third_party/ijar/platform_utils.cc:15:0:
./third_party/ijar/platform_utils.h:31:3: error: 'mode_t' does not name a type
   mode_t file_mode;
   ^~~~~~
./third_party/ijar/platform_utils.h: In function 'devtools_ijar::u4 devtools_ijar::stat_to_zipattr(const devtools_ijar::Stat&)':
./third_party/ijar/platform_utils.h:38:26: error: 'const struct devtools_ijar::Stat' has no member named 'file_mode'
   return (((u4)file_stat.file_mode) << 16) |
                          ^~~~~~~~~
./third_party/ijar/platform_utils.h: At global scope:
./third_party/ijar/platform_utils.h:51:35: error: 'mode_t' has not been declared
 bool write_file(const char* path, mode_t perm, const void* data, size_t size);
                                   ^~~~~~
./third_party/ijar/platform_utils.h:68:34: error: 'mode_t' has not been declared
 bool make_dirs(const char* path, mode_t perm);
                                  ^~~~~~
third_party/ijar/platform_utils.cc: In function 'bool devtools_ijar::stat_file(const char*, devtools_ijar::Stat*)':
third_party/ijar/platform_utils.cc:47:11: error: 'struct devtools_ijar::Stat' has no member named 'file_mode'
   result->file_mode = statst.st_mode;
           ^~~~~~~~~
third_party/ijar/platform_utils.cc: In function 'bool devtools_ijar::write_file(const char*, mode_t, const void*, size_t)':
third_party/ijar/platform_utils.cc:67:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   if (write(fd, data, size) != size) {
       ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
Target //src:bazel failed to build

Any ideas?

achimnol added a commit to lablup/backend.ai-kernels that referenced this issue Mar 6, 2017
 * JDK 8, Python 3.6-dev packages are readily available, but
   Google's Bazel seems not working on Alpine yet.

   ref) bazelbuild/bazel#1492

   Let's come back after some time here.
@kchodorow
Copy link
Contributor

It looks like mode_t is defined in a different header than on other systems. Maybe try grepping to find where mode_t is defined in /usr/include. third_party/ijar/platform_utils.cc is already including all of the places I'd expect to find it. Maybe sys/file.h?

@dimpavloff
Copy link

dimpavloff commented Apr 6, 2017

@r351574nc3 , you can see a workaround in this Dockerfile which builds Bazel and TF with musl instead of glibc:
https://github.com/tatsushid/docker-alpine-py3-tensorflow-jupyter/blob/master/Dockerfile#L46

@davido
Copy link
Contributor

davido commented Nov 8, 2017

It seems, that those workarounds are not needed any more. Here is the working Dockerfile that is building Bazel on Alpine Linux.

Moreover, I'm looking in creating native bazel package for Alpine Linux, so that something like that:

apk add bazel

just work.

@pspeter3
Copy link
Author

pspeter3 commented Nov 8, 2017

Thanks! This should mean lighter docker containers for all

@davido
Copy link
Contributor

davido commented Nov 8, 2017

Moreover, I'm looking in creating native bazel package for Alpine Linux, so that something like that:

Native Bazel package for Alpine Linux is here.

@perezd
Copy link
Contributor

perezd commented Nov 9, 2017 via email

@akofman
Copy link

akofman commented Dec 6, 2017

Hello all,
I think that as it is explained here, the issue is actually not from Bazel but from Buzybox. So imho the best and simplest solution is to add the coreutils dependency in your Dockerfiles.
As an example I did this Docker image and it works perfectly: https://hub.docker.com/r/alexiskofman/bazel-alpine/

@davido
Copy link
Contributor

davido commented Dec 6, 2017

So imho the best and simplest solution is to add the coreutils dependency in your Dockerfiles.

Sure, but why are you worried about this diff line:
https://github.com/davido/bazel-alpine-package/blob/master/APKBUILD#L22

In the end users should be able to just say:

apk add bazel

BTW, the Bazel 0.8.0 native package for Alpine Linux is here: https://github.com/davido/bazel-alpine-package/releases/tag/0.8.0.

@hlopko
Copy link
Member

hlopko commented Mar 21, 2018

Closing since this is not really an issue anymore?

@grzesuav
Copy link

@davido @mhlopko abovementioned builds are not up to date (0.15.2 is latest there, yet we have 0.18.0). Is there any alternative way to get those binaries on alpine?

@KEINOS
Copy link

KEINOS commented Sep 4, 2019

[F.Y.I.] For those who're looking for the latest Bazel Alpine package (apk file like I do), check:

https://github.com/davido/bazel-alpine-package/releases

  • Currently (2019/09/04) available Bazel versions:
    • PreRelease: Bazel v0.26.1
    • Latest: Bazel v0.22.0
[LOG] Alpine:latest(3.10.2) @ macOS

$ # Host Env Infos
$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G7024
$ 
$ docker version
Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     false
$
$ docker run --rm -it alpine:latest /bin/sh
/ # 
/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.10.2
PRETTY_NAME="Alpine Linux v3.10"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/ #
/ # # Install Bazel
/ # #   See: https://github.com/davido/bazel-alpine-package
/ # apk --no-cache add ca-certificates wget
...
/ # wget -q -O /etc/apk/keys/david@ostrovsky.org-5a0369d6.rsa.pub \
>  https://raw.githubusercontent.com/davido/bazel-alpine-package/master/david@ostrovsky.org-5a0369d6.rsa.pub
/ #
/ # wget https://github.com/davido/bazel-alpine-package/releases/download/0.26.1/bazel-0.26.1-r0.apk
...
/ #
/ # apk add bazel-0.26.1-r0.apk && rm bazel-0.26.1-r0.apk
...
/ #
/ # bazel version
Extracting Bazel installation...
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Build label: 0.26.1- (@non-git)
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Sat Jun 8 20:32:26 2019 (1560025946)
Build timestamp: 1560025946
Build timestamp as int: 1560025946
/ #
/ # # Build C++ sample
/ # #   See: https://docs.bazel.build/versions/master/tutorial/cpp.html
/ # 
/ # apk --no-cache add git g++
/ # git clone https://github.com/bazelbuild/examples/
...
/ # cd examples/cpp-tutorial/stage1
/ # bazel build //main:hello-world
INFO: Analyzed target //main:hello-world (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //main:hello-world up-to-date:
  bazel-bin/main/hello-world
INFO: Elapsed time: 0.450s, Critical Path: 0.32s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed successfully, 2 total actions
/ # 
/ # ./bazel-bin/main/hello-world
Hello world
Wed Sep  4 02:40:06 2019
/ # 
/ # ldd ./bazel-bin/main/hello-world
	/lib/ld-musl-x86_64.so.1 (0x7fc58ae7d000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7fc58ad23000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7fc58ad0f000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7fc58ae7d000)

@kevinsimper
Copy link

apk add bazel

Just to anyone landing here from Google Search, you can't add bazel like that anymore.

@findmyname666
Copy link

findmyname666 commented Jan 12, 2023

The last month I was using bazel binary downloaded from releases and it worked in Alpine container.
For some reason it doesn't work now :(
Did anybody figure-out some process for it ?

Now I'm getting weird errors:

  • ash shell
/ # bazel
/bin/sh: bazel: not found
  • bash
8a7b77737ea5:/# bazel
bash: /usr/bin/bazel: cannot execute: required file not found

@yesudeep
Copy link
Contributor

yesudeep commented Jan 13, 2023

@findmyname666 FYI:

I'm new to docker, but I've got a RBE container image working with this Dockerfile (remove what you don't use). It builds bazel from source using a release distribution and plants it at /usr/local/bin/bazel. It currently doesn't verify sha256sums or .sig, but I'm working on adding that.

FROM alpine:edge

RUN \
    sed -i -e 's/^http:/#http:/' -e 's/^#http:\(.*edge.*\)/https:\1/' /etc/apk/repositories \
    && apk update \
    && apk add --no-cache \
    autoconf \
    autoconf-archive \
    automake \
    bash \
    bash-completion \
    binutils-gold \
    clang \
    cmake \
    coreutils \
    curl \
    g++ \
    gcc \
    gcompat \
    gettext \
    git \
    gmp-dev \
    go \
    gpg \
    libc-dev \
    libc6-compat \
    libffi-dev \
    libtool \
    linux-headers \
    m4 \
    make \
    musl-dev \
    ncurses-dev \
    openjdk11-jdk \
    openjdk17-jdk \
    patch \
    perl \
    pv \
    py3-pip \
    python3-dev \
    shadow \
    tar \
    texinfo \
    tig \
    tree \
    unzip \
    vim \
    wget \
    xz \
    zip

RUN \
    mkdir -p /tmp/bazel-release \
    && chmod a+rwx /tmp/bazel-release \
    && cd /tmp/bazel-release \
    && wget -O bazel-release-dist.zip \
       https://github.com/bazelbuild/bazel/releases/download/6.0.0/bazel-6.0.0-dist.zip \
    && unzip bazel-release-dist.zip \
    && env JAVA_HOME=/usr/lib/jvm/java-11-openjdk \
       EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" \
       bash ./compile.sh \
    && cp output/bazel /usr/local/bin/ \
    && cd

RUN printf 'export JAVA_HOME=$(dirname $(dirname $(readlink -f $(type -P javac))))' > /etc/profile.d/java_home.sh

RUN printf 'startup --output_user_root=/tmp/bazel\n\
run --color=yes\n\
build --color=yes\n\
build --verbose_failures\n\
build --enable_runfiles\n\
test --test_output=errors\n\
test --test_verbose_timeout_warnings\n\
test --test_summary=terse\n'\
> /etc/bazel.bazelrc

ENV PATH="/usr/local/bin:$PATH"

WORKDIR /src

@strophy
Copy link

strophy commented Jan 30, 2023

Thanks @yesudeep! I was able to use this to set up Bazel 6.0.0 on Alpine 3.17 for both amd64 and arm64 with a slightly more minimal Dockerfile as follows:

FROM alpine:3.17
RUN apk add --no-cache \
    bash \
    build-base \
    curl \
    linux-headers \
    openjdk11-jdk \
    python3 \
    unzip \
    zip

ARG BAZEL_VERSION=6.0.0
RUN mkdir -p /tmp/bazel-release
WORKDIR /tmp/bazel-release
RUN curl -sSLO https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip && unzip -q bazel-${BAZEL_VERSION}-dist.zip
RUN env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
RUN install -D output/bazel /usr/local/bin/bazel

I noticed there are packages for Alpine edge testing, but 4.2.2 suffers from this bug which probably won't be fixed, and 5.3.2 is not available for aarch64, so building bazel from source using the above steps is the only way I know to be able to use it in QEMU.

@mowoe
Copy link

mowoe commented May 9, 2023

Hi @strophy, thank you for the instructions! I am trying to build bazel in alpine on arm64 and was wondering how long you build roughly takes. While i can get the container to build, it takes ages, while building bazel on the host machine (M2 Air) takes 3-4 minutes.

@strophy
Copy link

strophy commented May 9, 2023

Hi @mowoe actually I'm unable to reliably build Bazel 6 for ARM64 on Alpine either on native AWS Graviton 2 CPUs or an emulator running on my x86_64 laptop. The build hangs forever as described here: #17220

I ended up packaging bazel6 for alpine but was unable to release an ARM binary for this reason.

You might be having more success due to using a different filesystem in the base OS? It's a complicated bug, beyond my understanding unfortunately.

@mowoe
Copy link

mowoe commented May 9, 2023

Thank you @strophy,
i must have somehow overlooked the mentioned issue.
Actually i only needed bazel to compile some tensorflow stuff, but as there are no bazel builds working with alpine, i tried building it in an alpine container.
This is until i realized that bazel does supply an arm64 build, but unfourtunately it is dynamically linked, which breaks libc stuff, as alpine uses musl libc and bazel uses glibc.

I came across these glibc builds build for alpine, which unfourtunately do not include arm64 ones, however there is a fork which does (being behind 1 minor version).

While certainly not optimal (you definetely notice the libc issues and i would not use it for production stuff), i got it to work in the end and for what im trying to do, it is working fine.

Here is the Dockerfile:

FROM python:3.10-alpine

RUN wget https://github.com/SatoshiPortal/alpine-pkg-glibc/releases/download/2.31-r0/glibc-bin-2.31-r0-aarch64.apk
RUN wget https://github.com/SatoshiPortal/alpine-pkg-glibc/releases/download/2.31-r0/glibc-2.31-r0-aarch64.apk
RUN apk add --force-overwrite --allow-untrusted glibc-2.31-r0-aarch64.apk
RUN apk add --force-overwrite --allow-untrusted glibc-bin-2.31-r0-aarch64.apk
RUN wget https://github.com/bazelbuild/bazel/releases/download/4.2.4/bazel-4.2.4-linux-arm64
RUN chmod +x ./bazel-4.2.4-linux-arm64
RUN apk add libstdc++6 libstdc++-dev libstdc++
CMD ["./bazel-4.2.4-linux-arm64","--version"]
$ docker build . -t bazel-arm
$ docker run --rm bazel-arm
./bazel-4.2.4-linux-arm64: /usr/lib/libstdc++.so.6: no version information available (required by ./bazel-4.2.4-linux-arm64)
bazel 4.2.4
$

The error about libstdc++ is described here, i havent had time to really look into it, but as far as i can see, everything seems to work fine anyway.
I know all of this has nothing to do with the actual issue, but i thought that it may help someone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) platform: other
Projects
None yet
Development

No branches or pull requests