Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build dist for x86_64-unknown-illumos #71272

Merged
merged 1 commit into from
Jul 15, 2020
Merged

Conversation

jclulow
Copy link
Contributor

@jclulow jclulow commented Apr 18, 2020

This change creates a new Docker image, "dist-x86_64-illumos", and sets
things up to build the full set of "dist" packages for illumos hosts, so
that illumos users can use "rustup" to install packages. It also
adjusts the manifest builder to expect complete toolchains for this
platform.

@rust-highfive
Copy link
Collaborator

r? @pietroalbini

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 18, 2020
@jclulow
Copy link
Contributor Author

jclulow commented Apr 18, 2020

Note that I am not super familiar with the CI/infra stuff at play here, and have mostly just tried to make everything look as much like it does for FreeBSD as possible, on the basis that it is a similarly cross-compiled target. If there is anybody in the infra team to whom I should speak about this as part of getting review, please let me know! Thanks.

I have tested the Docker CI image a bunch of times locally, and @pfmooney and I are using illumos toolchain bits built this way as we work on platform support in various ecosystem crates. Things have been going well thus far!

@jclulow
Copy link
Contributor Author

jclulow commented Apr 18, 2020

A further note: this makes use of an archive of core OS artefacts from an illumos system, which is described in some more detail over at: https://github.com/illumos/sysroot

@rust-highfive
Copy link
Collaborator

The job mingw-check of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2020-04-18T06:56:45.4367914Z ========================== Starting Command Output ===========================
2020-04-18T06:56:45.4372176Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/39c18008-e123-4505-b4a0-f32cdf4783a5.sh
2020-04-18T06:56:45.4372608Z 
2020-04-18T06:56:45.4379315Z ##[section]Finishing: Disable git automatic line ending conversion
2020-04-18T06:56:45.4410142Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/71272/merge to s
2020-04-18T06:56:45.4415299Z Task         : Get sources
2020-04-18T06:56:45.4415821Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-04-18T06:56:45.4416311Z Version      : 1.0.0
2020-04-18T06:56:45.4416652Z Author       : Microsoft
---
2020-04-18T06:56:46.4479943Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-04-18T06:56:46.4484154Z ##[command]git config gc.auto 0
2020-04-18T06:56:46.4486932Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-04-18T06:56:46.4489483Z ##[command]git config --get-all http.proxy
2020-04-18T06:56:46.4494805Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/71272/merge:refs/remotes/pull/71272/merge
---
2020-04-18T06:59:23.0984905Z  ---> 78ad2f4d4aca
2020-04-18T06:59:23.0985122Z Step 6/7 : ENV RUN_CHECK_WITH_PARALLEL_QUERIES 1
2020-04-18T06:59:23.0985470Z  ---> Using cache
2020-04-18T06:59:23.0985791Z  ---> 4d2dc61c4d00
2020-04-18T06:59:23.0987018Z Step 7/7 : ENV SCRIPT python3 ../x.py test src/tools/expand-yaml-anchors &&            python3 ../x.py check --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu &&            python3 ../x.py build --stage 0 src/tools/build-manifest &&            python3 ../x.py test --stage 0 src/tools/compiletest &&            python3 ../x.py test src/tools/tidy &&            /scripts/validate-toolstate.sh
2020-04-18T06:59:23.0988214Z  ---> 776b6266a8b7
2020-04-18T06:59:23.0988404Z Successfully built 776b6266a8b7
2020-04-18T06:59:23.1033727Z Successfully tagged rust-ci:latest
2020-04-18T06:59:23.1327714Z Built container sha256:776b6266a8b7d63e2d3c2b5a784dbf521184a904fb10bf818c6b5c7e1ab74d4a
2020-04-18T06:59:23.1327714Z Built container sha256:776b6266a8b7d63e2d3c2b5a784dbf521184a904fb10bf818c6b5c7e1ab74d4a
2020-04-18T06:59:23.1342067Z Looks like docker image is the same as before, not uploading
2020-04-18T06:59:23.8278081Z [CI_JOB_NAME=mingw-check]
2020-04-18T06:59:23.8471540Z [CI_JOB_NAME=mingw-check]
2020-04-18T06:59:23.8510171Z == clock drift check ==
2020-04-18T06:59:23.8519390Z   local time: Sat Apr 18 06:59:23 UTC 2020
2020-04-18T06:59:24.0299077Z   network time: Sat, 18 Apr 2020 06:59:24 GMT
2020-04-18T06:59:24.0326120Z Starting sccache server...
2020-04-18T06:59:24.1339072Z configure: processing command line
2020-04-18T06:59:24.1340436Z configure: 
2020-04-18T06:59:24.1341354Z configure: rust.parallel-compiler := True
---
2020-04-18T07:02:36.5725732Z     Checking rustc_feature v0.0.0 (/checkout/src/librustc_feature)
2020-04-18T07:02:36.6041449Z     Checking fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
2020-04-18T07:02:36.7624008Z     Checking rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
2020-04-18T07:02:36.9094498Z     Checking rustc_hir v0.0.0 (/checkout/src/librustc_hir)
2020-04-18T07:02:37.3463785Z     Checking rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
2020-04-18T07:02:39.3017365Z     Checking rustc_attr v0.0.0 (/checkout/src/librustc_attr)
2020-04-18T07:02:39.6833707Z     Checking rustc_parse v0.0.0 (/checkout/src/librustc_parse)
2020-04-18T07:02:41.3752156Z     Checking rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
2020-04-18T07:02:41.7372924Z     Checking rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
2020-04-18T07:04:10.6104043Z configure: rust.debug-assertions := True
2020-04-18T07:04:10.6104415Z configure: build.cargo-native-static := True
2020-04-18T07:04:10.6104639Z configure: rust.channel         := nightly
2020-04-18T07:04:10.6104853Z configure: build.submodules     := False
2020-04-18T07:04:10.6105305Z configure: build.configure-args := ['--enable-sccache', '--disable-manage-submodu ...
2020-04-18T07:04:10.6105704Z configure: writing `config.toml` in current directory
2020-04-18T07:04:10.6105893Z configure: 
2020-04-18T07:04:10.6106195Z configure: run `python /checkout/x.py --help`
2020-04-18T07:04:10.6106360Z configure: 
---
2020-04-18T07:05:34.0608077Z Hugepagesize:       2048 kB
2020-04-18T07:05:34.0608239Z DirectMap4k:      131008 kB
2020-04-18T07:05:34.0608413Z DirectMap2M:     4063232 kB
2020-04-18T07:05:34.0608574Z DirectMap1G:     5242880 kB
2020-04-18T07:05:34.0650519Z + python3 ../x.py test src/tools/expand-yaml-anchors
2020-04-18T07:05:35.2103354Z Ensuring the YAML anchors in the GitHub Actions config were expanded
2020-04-18T07:05:35.2103354Z Ensuring the YAML anchors in the GitHub Actions config were expanded
2020-04-18T07:05:35.2139472Z Building stage0 tool expand-yaml-anchors (x86_64-unknown-linux-gnu)
2020-04-18T07:05:35.4127403Z    Compiling unicode-xid v0.2.0
2020-04-18T07:05:35.5302873Z    Compiling syn v1.0.11
2020-04-18T07:05:36.2752363Z    Compiling linked-hash-map v0.5.2
2020-04-18T07:05:36.2971884Z    Compiling lazy_static v1.4.0
2020-04-18T07:05:36.2971884Z    Compiling lazy_static v1.4.0
2020-04-18T07:05:36.4739564Z    Compiling yaml-rust v0.4.3
2020-04-18T07:05:40.0220548Z    Compiling quote v1.0.2
2020-04-18T07:05:51.8362975Z    Compiling thiserror-impl v1.0.5
2020-04-18T07:05:55.6781724Z    Compiling thiserror v1.0.5
2020-04-18T07:05:55.7352002Z    Compiling yaml-merge-keys v0.4.0
2020-04-18T07:05:56.7021699Z    Compiling expand-yaml-anchors v0.1.0 (/checkout/src/tools/expand-yaml-anchors)
2020-04-18T07:05:58.0755267Z     Finished release [optimized] target(s) in 22.86s
2020-04-18T07:05:58.0900984Z error: .github/workflows/ci.yml is not up to date
2020-04-18T07:05:58.0902486Z caused by: src/ci/github-actions/ci.yml and .github/workflows/ci.yml are different
2020-04-18T07:05:58.0903061Z 
2020-04-18T07:05:58.0903061Z 
2020-04-18T07:05:58.0904161Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/expand-yaml-anchors" "check" "/checkout"
2020-04-18T07:05:58.0905222Z 
2020-04-18T07:05:58.0905386Z 
2020-04-18T07:05:58.0905386Z 
2020-04-18T07:05:58.0911568Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/expand-yaml-anchors
2020-04-18T07:05:58.1010381Z == clock drift check ==
2020-04-18T07:05:58.1022959Z   local time: Sat Apr 18 07:05:58 UTC 2020
2020-04-18T07:05:58.1022959Z   local time: Sat Apr 18 07:05:58 UTC 2020
2020-04-18T07:05:58.4081758Z   network time: Sat, 18 Apr 2020 07:05:58 GMT
2020-04-18T07:06:03.9067103Z 
2020-04-18T07:06:03.9067103Z 
2020-04-18T07:06:03.9129276Z ##[error]Bash exited with code '1'.
2020-04-18T07:06:03.9143794Z ##[section]Finishing: Run build
2020-04-18T07:06:03.9189428Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/71272/merge to s
2020-04-18T07:06:03.9193699Z Task         : Get sources
2020-04-18T07:06:03.9193988Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-04-18T07:06:03.9194252Z Version      : 1.0.0
2020-04-18T07:06:03.9194454Z Author       : Microsoft
2020-04-18T07:06:03.9194454Z Author       : Microsoft
2020-04-18T07:06:03.9194752Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-04-18T07:06:03.9195088Z ==============================================================================
2020-04-18T07:06:04.2190874Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-04-18T07:06:04.2235832Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/71272/merge to s
2020-04-18T07:06:04.2319020Z Cleaning up task key
2020-04-18T07:06:04.2320158Z Start cleaning up orphan processes.
2020-04-18T07:06:04.2473834Z Terminate orphan process: pid (4448) (python)
2020-04-18T07:06:04.2745514Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

@jclulow jclulow force-pushed the illumos-x86-ci branch 2 times, most recently from e7dac73 to 092af47 Compare April 18, 2020 21:14
@pietroalbini
Copy link
Member

Thanks for the PR! Unfortunately we don't have the CI capacity for an extra builder right now: could you add this target to the dist-various-2 builder?

@jclulow jclulow force-pushed the illumos-x86-ci branch 2 times, most recently from 6825d95 to 320b996 Compare April 21, 2020 05:08
@jclulow
Copy link
Contributor Author

jclulow commented Apr 21, 2020

Hi, @pietroalbini! As requested, I have moved the build into the existing dist-various-2 builder. I have confirmed that I am able to use the new image locally on my workstation to build a full toolchain for illumos, and the existing standard library archive that is built for Solaris on x86 and SPARC. Please let me know if there is anything else I should look at! Thanks.

@pietroalbini
Copy link
Member

Pushed a temporary commit to test the builder in a try build.

@bors try

@bors
Copy link
Contributor

bors commented Apr 21, 2020

⌛ Trying commit d3c0912c8090af3a2bbf4dd1ab12a8aba4c0aadd with merge c18e4bc243f34aa6702a147a3d961974ed2434c8...

@pietroalbini
Copy link
Member

Why was some Solaris code changed as part of this PR?

Also, cc @rust-lang/compiler, this PR practically adds x86_64-unknown-illumos as a Tier 2 target (building a full host compiler). Do you have some requirements on your end for that?

@pfmooney
Copy link
Contributor

Why was some Solaris code changed as part of this PR?

This is to disambiguate illumos and Solaris. Both require a Solaris-y toolchain, but given the divergence in the platforms (see also: #71145 in the effort to split them), they require separate compilers using their respective sysroots.

Also, cc @rust-lang/compiler, this PR practically adds x86_64-unknown-illumos as a Tier 2 target (building a full host compiler). Do you have some requirements on your end for that?

We're interested in making rustup a reality on this platform.

@bors
Copy link
Contributor

bors commented Apr 21, 2020

💥 Test timed out

@jclulow
Copy link
Contributor Author

jclulow commented Apr 21, 2020

Looking into the failure at https://github.com/rust-lang-ci/rust/runs/605427771 it seems like the --enable-lld flag to Rust's configure may be enabled in error for this image. This is the first time we're actually shipping a toolchain out of dist-various-2, rather than just cross-built library artefacts. Looking at, say, dist-x86_64-freebsd, where a cross built toolchain is delivered, that flag is not present. Indeed, the only other images which include it are dist-x86_64-musl and x86_64-gnu-debug which are probably not (perhaps merely not completely or hygienically) cross built.

I'll investigate what's happening in some more detail, but I think the right thing to do is drop the flag here, because I don't think we need lld and it seems like the cross build for lld is not quite right anyhow.

@nikomatsakis
Copy link
Contributor

Something we've been discussing a lot is that, when we add a new target, we ought to have a list of folks that we can ping for advice, or people that we can request to review PRs that are proposing changes specific to that tier. Another common thing is to have someone who can reproduce and help to diagnose problems specific to that target.

I also think it would make sense to file a Major Change Proposal -- that process is still "underway" but it mostly just means creating an issue with this template.

@jclulow, would you be willing to file a MCP as described above, and include in there the names of any folks who would be willing to help review and maintain the target?

@jclulow
Copy link
Contributor Author

jclulow commented Apr 23, 2020

@jclulow, would you be willing to file a MCP as described above, and include in there the names of any folks who would be willing to help review and maintain the target?

Definitely! @pfmooney and myself in particular are keen to get this across the line in a sustainable way for illumos, and to stick around to assist with maintenance in the future. I'm still working on technical issues with this PR (e.g., #71457), but in the meantime I will write up an MCP! Thanks for pointing us in the right direction.

@bors
Copy link
Contributor

bors commented Apr 26, 2020

☔ The latest upstream changes (presumably #71434) made this pull request unmergeable. Please resolve the merge conflicts.

@Dylan-DPC-zz
Copy link

@jclulow can you rebase? thanks

@Dylan-DPC-zz Dylan-DPC-zz added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 12, 2020
@jclulow
Copy link
Contributor Author

jclulow commented May 12, 2020

@jclulow can you rebase? thanks

Definitely! What should I do with the extra commit that was added on top that seems to make it possible to try this out? Just leave it in place?

@Manishearth
Copy link
Member

@bors retry yield

@bors
Copy link
Contributor

bors commented Jul 11, 2020

⌛ Testing commit 8368a35 with merge 1a3f4791a589d554233d4baf7d94226252d4d15e...

@Manishearth
Copy link
Member

@bors retry yield

@Dylan-DPC-zz
Copy link

@bors treeclosed-

(will close the tree again if issue persists)

@bors
Copy link
Contributor

bors commented Jul 12, 2020

⌛ Testing commit 8368a35 with merge 6cc06ce7c6e05afdeb1d3cd3d531547d714adc27...

@Dylan-DPC-zz
Copy link

Dylan-DPC-zz commented Jul 12, 2020

@bors retry
(oops)

@bors
Copy link
Contributor

bors commented Jul 12, 2020

⌛ Testing commit 8368a35 with merge 89111c88a3537c25d577fa304fbebbc23237fd1e...

@bors
Copy link
Contributor

bors commented Jul 12, 2020

⌛ Testing commit 8368a35 with merge 66fb914346eb33a90b2945d2dbfbbed978b6c4b3...

@bors
Copy link
Contributor

bors commented Jul 12, 2020

💥 Test timed out

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jul 12, 2020
@Manishearth
Copy link
Member

@bors retry

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 15, 2020
@bors
Copy link
Contributor

bors commented Jul 15, 2020

⌛ Testing commit 8368a35 with merge c714eae...

@bors
Copy link
Contributor

bors commented Jul 15, 2020

☀️ Test successful - checks-actions, checks-azure
Approved by: pietroalbini
Pushing c714eae to master...

@liulitchi
Copy link

Thank you all, I've just installed Rust on my OpenIndiana OS, and it runs pretty well.
This command should be very helpful for other illumos users:
curl --proto '=https' --tlsv1.2 -sSf https://illumos.org/downloads/rustup/init.sh | bash

@jclulow
Copy link
Contributor Author

jclulow commented Dec 27, 2020

@liulitchi Note that as of rustup 1.23.1 (3df2264a9 2020-11-30) (see also: rust-lang/rustup#2432 and rust-lang/rustup#2580), you should be able to use the stock rustup instead of the preliminary illumos-specific build. The instructions up on https://rustup.rs/ should be sufficient -- if not, open an issue on the rustup repository and tag me in it! Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.