Add a Docker container for doing automated builds for CloudABI. #47427

EdSchouten · 2018-01-14T17:43:35Z

Setting up a cross compilation toolchain for CloudABI is relatively
easy. It's just a matter of installing a somewhat recent version of
Clang (5.0 preferred) and installing the corresponding
${target}-cxx-runtime package, containing a set of core C/C++ libraries
(libc, libc++, libunwind, etc).

Eventually it would be nice if we could also run x.py test. That,
however still requires some more work. Both libtest and compiletest
would need to be adjusted to deal with CloudABI's requirement of having
all of an application's dependencies injected. Let's settle for just
doing x.py dist for now.

kennytm

Given that we are not going to add a new CI job in the short term, could you move the dist-cloudabi folder as src/ci/docker/disabled/dist-cloudabi?

kennytm · 2018-01-14T18:10:52Z

.travis.yml

@@ -126,6 +126,8 @@ matrix:
      if: branch = auto
    - env: IMAGE=dist-armv7-linux DEPLOY=1
      if: branch = auto
+    - env: IMAGE=dist-cloudabi DEPLOY=1
+      if: branch = auto


I don't think we want to put this as a separate job on the CI for now due to resource limitation.

EdSchouten · 2018-01-14T18:49:29Z

Aww... That's too bad. Be sure to get in touch with me if CPU capacity opens up.

EdSchouten · 2018-01-14T21:00:50Z

Sorry if I'm sounding a bit negative, but not having automated builds for CloudABI is going to be more annoying than I had anticipated. If no automated builds can be provided, my idea would be to set up an automated build myself. That way, people can install Rust nightly on their operating system of choice and extend that with CloudABI support by extracting a rust-std-nightly-x86_64-unknown-cloudabi.tar.xz tarball hosted by me.

It turns out this won't work in practice, because rustc is quite picky:

error[E0514]: found crate `core` compiled by an incompatible version of rustc
  |
  = help: please recompile that crate using this compiler (rustc 1.25.0-nightly (e6072a7b3 2018-01-13))
  = note: the following crate versions were found:
          crate `core` compiled by rustc 1.25.0-dev (3f92e8d89 2018-01-14): /home/ed/.rustup/toolchains/nightly-x86_64-unknown-freebsd/lib/rustlib/x86_64-unknown-cloudabi/lib/libcore-14da3e8a93d4bc72.rlib

Do you by any chance know what would be the most elegant way to deal with this?

Mark-Simulacrum · 2018-01-15T02:20:21Z

A potential solution is https://github.com/japaric/xargo. However, what you suggest would work, you'd just have to ship all of the "std" component that we normally do, including core. In theory, I could see rustup being extended to support using "components" from sources other than static.rust-lang.org, but I don't think there's a current maintainer of rustup to talk to about that.

malbarbo · 2018-01-16T15:25:00Z

@EdSchouten You can try adding this to dist-various-1 or dist-various-2 builder.

EdSchouten · 2018-01-16T15:41:37Z

@kennytm Integrating it in dist-various-* would be fine for me. Would that be all right for you?

kennytm · 2018-01-16T15:47:09Z

@EdSchouten dist-various-* uses Ubuntu 16.04 LTS. Does this work for CloudABI? I'm not sure if we want to upgrade them to 17.10. cc @alexcrichton

alexcrichton · 2018-01-16T15:56:48Z

@EdSchouten ah yes @Mark-Simulacrum and @malbarbo are both right in that if this is a separate builder we don't currently have the capacity to add more so it'll need to be disabled for now. If, however, it can be folded into an existing builder (which as it's just libstd I believe it should be totally feasible to do this) then our existing builders should have enough capacity to cross compile a new platform. We typically just pick whichever is fastest and throw it in there. Looks like the last successful run had dist-various-1 clocking in at 1h20m and dist-various-2 clocking in at 1h9m, so adding to dist-various-2 should do the trick!

Most of our containers use 16.04 as that was the latest when they were created, but I'm all for updating them as eagerly as possible (pending we're able to fix accidental regressions that creates of course). The dist-various-* containers should be able to get updated any time, the build will fail if they fail to build! Otherwise we in general don't want to have a dependency on a specific version, so if we are accidentally relying on something 16.04-specific we can fix that later.

In other words, let's see if we can fold this into dist-various-2? And you can feel free to update dist-various-2 if you'd like. You'll want to run locally though to make sure it all still works via ./src/ci/docker/run.sh dist-various-2

Setting up a cross compilation toolchain for CloudABI is relatively easy. It's just a matter of installing a somewhat recent version of Clang (5.0 preferred) and installing the corresponding ${target}-cxx-runtime package, containing a set of core C/C++ libraries (libc, libc++, libunwind, etc). Eventually it would be nice if we could also run 'x.py test'. That, however still requires some more work. Both libtest and compiletest would need to be adjusted to deal with CloudABI's requirement of having all of an application's dependencies injected. Let's settle for just doing 'x.py dist' for now.

There is not enough capacity to do automated builds for CloudABI at this time.

As discussed in rust-lang#47427, let's not have a separate container for doing CloudABI builds. It's a lot faster if we integrate it into an existing container, so there's less duplication of what's being built. Upgrade the existing container to Ubuntu 17.10, which is required for CloudABI builds. The version of Clang shipped with 16.04 is not recent enough to support CloudABI properly.

EdSchouten · 2018-01-16T21:28:23Z

Hi @alexcrichton. Thanks for clarifying!

I just integrated CloudABI support into dist-various-2. It worked all right, but I did have to set DEPLOY=1. Apparently that's needed to work around #46781 / #43982. Let me know what you think!

alexcrichton · 2018-01-16T22:13:26Z

Thanks @EdSchouten! I wonder if we could perhaps start with 1 or 2 targets just to make sure things work? We can always expand if we've still got capacity afterwards!

We'll turn on other architectures if it turns out we have enough capacity.

EdSchouten · 2018-01-16T22:22:34Z

Done! \o/

alexcrichton · 2018-01-16T22:23:24Z

@bors: r+

bors · 2018-01-16T22:23:24Z

📌 Commit dcf0cd0 has been approved by alexcrichton

EdSchouten · 2018-01-17T07:16:07Z

@bors retry

Hmm... odd. The Travis page mentioned this build got cancelled? Not sure whether I have rights to force a retry.

bors · 2018-01-17T07:16:07Z

@EdSchouten: 🔑 Insufficient privileges: and not in try users

kennytm · 2018-01-17T07:56:36Z

@EdSchouten The build is canceled because Travis ded https://www.traviscistatus.com/incidents/c1s4dlyxd4lf

The build hasn't actually started, so retry does nothing (besides insufficient privilege).

Add a Docker container for doing automated builds for CloudABI. Setting up a cross compilation toolchain for CloudABI is relatively easy. It's just a matter of installing a somewhat recent version of Clang (5.0 preferred) and installing the corresponding `${target}-cxx-runtime` package, containing a set of core C/C++ libraries (libc, libc++, libunwind, etc). Eventually it would be nice if we could also run `x.py test`. That, however still requires some more work. Both libtest and compiletest would need to be adjusted to deal with CloudABI's requirement of having all of an application's dependencies injected. Let's settle for just doing `x.py dist` for now.

Rollup of 21 pull requests - Successful merges: #47302, #47333, #47387, #47404, #47407, #47426, #47427, #47436, #47444, #47456, #47458, #47467, #47479, #47481, #47483, #47487, #47497, #47498, #47505, #47509, #47514 - Failed merges:

kennytm reviewed Jan 14, 2018

View reviewed changes

kennytm added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jan 14, 2018

EdSchouten added 3 commits January 16, 2018 20:31

Move dist-cloudabi/ into disabled/.

5801f95

There is not enough capacity to do automated builds for CloudABI at this time.

EdSchouten force-pushed the cloudabi-ci branch from 7b2e25a to ae4288f Compare January 16, 2018 21:25

Only enable CloudABI builds for x86-64 for now.

dcf0cd0

We'll turn on other architectures if it turns out we have enough capacity.

kennytm added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 16, 2018

kennytm mentioned this pull request Jan 17, 2018

Rollup of 19 pull requests #47513

Closed

kennytm mentioned this pull request Jan 17, 2018

Rollup of 21 pull requests #47522

Merged

bors merged commit dcf0cd0 into rust-lang:master Jan 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a Docker container for doing automated builds for CloudABI. #47427

Add a Docker container for doing automated builds for CloudABI. #47427

EdSchouten commented Jan 14, 2018

kennytm left a comment

kennytm Jan 14, 2018

EdSchouten commented Jan 14, 2018

EdSchouten commented Jan 14, 2018

Mark-Simulacrum commented Jan 15, 2018

malbarbo commented Jan 16, 2018

EdSchouten commented Jan 16, 2018

kennytm commented Jan 16, 2018 •

edited

Loading

alexcrichton commented Jan 16, 2018

EdSchouten commented Jan 16, 2018

alexcrichton commented Jan 16, 2018

EdSchouten commented Jan 16, 2018

alexcrichton commented Jan 16, 2018

bors commented Jan 16, 2018

EdSchouten commented Jan 17, 2018

bors commented Jan 17, 2018

kennytm commented Jan 17, 2018

Add a Docker container for doing automated builds for CloudABI. #47427

Add a Docker container for doing automated builds for CloudABI. #47427

Conversation

EdSchouten commented Jan 14, 2018

kennytm left a comment

Choose a reason for hiding this comment

kennytm Jan 14, 2018

Choose a reason for hiding this comment

EdSchouten commented Jan 14, 2018

EdSchouten commented Jan 14, 2018

Mark-Simulacrum commented Jan 15, 2018

malbarbo commented Jan 16, 2018

EdSchouten commented Jan 16, 2018

kennytm commented Jan 16, 2018 • edited Loading

alexcrichton commented Jan 16, 2018

EdSchouten commented Jan 16, 2018

alexcrichton commented Jan 16, 2018

EdSchouten commented Jan 16, 2018

alexcrichton commented Jan 16, 2018

bors commented Jan 16, 2018

EdSchouten commented Jan 17, 2018

bors commented Jan 17, 2018

kennytm commented Jan 17, 2018

kennytm commented Jan 16, 2018 •

edited

Loading