[buildkite] Implement secrets encryption and sandboxing

This adds a proof-of-concept demonstration of two new buildkite plugins: * `cryptic` adds secrets management to privileged pipelines. These pipelines cannot be freely modified; their integrity is verified against a signature maintained by committers with a secret key. This allows certain portions of the CI configuration (which are privileged and can decrypt encrypted files/environment variables) to remain public, but read-only to the general populace. * `sandbox` adds a generic sandboxing mechanism that allows CI steps to be run within user-provided rootfs images. We're using these here to provide compiler toolchains for the `llvm-passes` CI steps, and the plan is to eventually provide _all_ compiler toolchains through such rootfs images.
JuliaLang · Jun 17, 2021 · a3b97dc · a3b97dc
1 parent 6dfa690
commit a3b97dc
Show file tree

Hide file tree

Showing 11 changed files with 367 additions and 26 deletions.
diff --git a/.buildkite/0_webui.yml b/.buildkite/0_webui.yml
@@ -0,0 +1,24 @@
+# This file represents what is put into the webUI.
+# It is purely for keeping track of the changes we make to the webUI configuration; modifying this file has no effect.
+# We use the `cryptic` buildkite plugin to provide secrets management, which requires some integration into the WebUI's steps.
+agents:
+  queue: "julia"
+  sandbox.jl: "true"
+
+steps:
+  - label: ":unlock: Unlock secrets, launch pipelines"
+    plugins:
+      - staticfloat/cryptic:
+          # Our list of pipelines that should be launched (but don't require a signature)
+          # These pipelines can be modified by any contributor and CI will still run.
+          # Build secrets will not be available in these pipelines (or their children)
+          # but some of our signed pipelines can wait upon the completion of these unsigned
+          # pipelines.
+          unsigned_pipelines:
+            - .buildkite/pipeline.yml
+
+          # Our signed pipelines must have a `signature` or `signature_file` parameter that
+          # verifies the treehash of the pipeline itself and the inputs listed in `inputs`
+          signed_pipelines:
+            - pipeline: .buildkite/signed_pipeline_test.yml
+              signature: "U2FsdGVkX18aZgryp6AJTArgD2uOnVWyFFGVOP5qsY4WbGQ/LVAcYiMEp9cweV+2iht+vmEF949CuuGTeQPA1fKlhPwkG3nZ688752DUB6en9oM2nuL31NoDKWHhpygZ"
diff --git a/.buildkite/cryptic_repo_keys/README.md b/.buildkite/cryptic_repo_keys/README.md
@@ -0,0 +1,6 @@
+## Cryptic repository keys
+
+This folder contains RSA-encrypted symmetric AES keys.
+These are used by buildkite agents to decrypt the secrets embedded within this repository.
+Each buildkite agent contains an RSA secret key that is used to unlock the symmetric AES key that was used to encrypt the secrets within this repository.
+For more information, see the [`cryptic` buildkite plugin repository](https://github.com/staticfloat/cryptic-buildkite-plugin).
diff --git a/.buildkite/cryptic_repo_keys/repo_key.2297e5e7 b/.buildkite/cryptic_repo_keys/repo_key.2297e5e7
diff --git a/.buildkite/llvm_passes.yml b/.buildkite/llvm_passes.yml
@@ -0,0 +1,40 @@
+# These steps should only run on `sandbox.jl` machines, not `docker`-isolated ones
+# since we need nestable sandboxing.  The rootfs images being used here are built from
+# the `.buildkite/rootfs_images/llvm-passes.jl` file.
+agents:
+  queue: "julia"
+  # Only run on `sandbox.jl` machines (not `docker`-isolated ones) since we need nestable sandboxing
+  sandbox.jl: "true"
+  os: "linux"
+
+steps:
+  - label: "analyzegc"
+    plugins:
+      - JuliaCI/julia#v1:
+          version: 1.6
+      - staticfloat/sandbox#v1:
+          rootfs_url: https://github.com/JuliaCI/rootfs-images/releases/download/v1/llvm-passes.tar.gz
+          rootfs_treehash: "f3ed53f159e8f13edfba8b20ebdb8ece73c1b8a8"
+    commands: |
+      echo "--- Install in-tree LLVM dependencies"
+      make -j 6 -C deps install-llvm install-clang install-llvm-tools install-libuv install-utf8proc install-unwind
+      echo "+++ run clangsa/analyzegc"
+      make -j 6 -C test/clangsa
+      make -j 6 -C src analyzegc
+    timeout_in_minutes: 60
+
+  - label: "llvmpasses"
+    plugins:
+      - JuliaCI/julia#v1:
+          version: 1.6
+      - staticfloat/sandbox#v1:
+          rootfs_url: https://github.com/JuliaCI/rootfs-images/releases/download/v1/llvm-passes.tar.gz
+          rootfs_treehash: "f3ed53f159e8f13edfba8b20ebdb8ece73c1b8a8"
+          uid: 1000
+          gid: 1000
+    commands: |
+      echo "+++ run llvmpasses"
+      make -j 6 release
+      make -j 6 -C src install-analysis-deps
+      make -j 6 -C test/llvmpasses
+    timeout_in_minutes: 60
diff --git a/.buildkite/pipeline.yml b/.buildkite/pipeline.yml
@@ -1,28 +1,19 @@
+# This file launches all the build jobs that _don't_ require secrets access.
+# These jobs can pass their output off to jobs that do require secrets access,
+# but those privileged steps require signing before they can be run.
+#
+# Yes, this is creating another layer of indirection; the flow now looks like:
+#
+#   [webui] -> pipeline.yml -> llvm_passes.yml
+#
+# when we could theoretically just have the `webui` launch `llvm_passes.yml`,
+# however this raises the bar for contributors to add new (unsigned) steps to
+# our CI configuration, so I'd rather live with an extra layer of indirection
+# and only need to touch the webui configuration when we need to alter
+# something about the privileged steps.
 steps:
-  - label: "analyzegc"
-    commands:
-      - echo "--- Install apt-get pre-reqs"
-      - apt-get update
-      - apt-get install -y build-essential libatomic1 python python3 gfortran perl wget m4 cmake pkg-config curl
-      - echo "--- Install in-tree LLVM dependencies"
-      - make -j 6 -C deps install-llvm install-clang install-llvm-tools install-libuv install-utf8proc install-unwind
-      - echo "+++ run clangsa/analyzegc"
-      - make -j 6 -C test/clangsa
-      - make -j 6 -C src analyzegc
+  - label: ":buildkite: Launch unsigned pipelines"
+    commands: |
+      buildkite-agent pipeline upload .buildkite/llvm_passes.yml
     agents:
-      queue: "juliacpu" # this should be julia -- also in pipeline settings
-      # os: linux # tag missing for juliacpu queue
-    timeout_in_minutes: 60
-  - label: "llvmpasses"
-    commands:
-      - echo "--- Install apt-get pre-reqs"
-      - apt-get update
-      - apt-get install -y build-essential libatomic1 python python3 gfortran perl wget m4 cmake pkg-config curl
-      - echo "+++ run llvmpasses"
-      - make -j 6 release
-      - make -j 6 -C src install-analysis-deps
-      - make -j 6 -C test/llvmpasses
-    agents:
-      queue: "juliacpu" # this should be julia -- also in pipeline settings
-      # os: linux # tag missing for juliacpu queue
-    timeout_in_minutes: 60
+      queue: julia
diff --git a/.buildkite/rootfs_images/Manifest.toml b/.buildkite/rootfs_images/Manifest.toml
@@ -0,0 +1,134 @@
+# This file is machine-generated - editing it directly is not advised
+
+[[ArgTools]]
+uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f"
+
+[[Artifacts]]
+uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33"
+
+[[Base64]]
+uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
+
+[[Dates]]
+deps = ["Printf"]
+uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
+
+[[Downloads]]
+deps = ["ArgTools", "LibCURL", "NetworkOptions"]
+uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
+
+[[InteractiveUtils]]
+deps = ["Markdown"]
+uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
+
+[[JLLWrappers]]
+deps = ["Preferences"]
+git-tree-sha1 = "642a199af8b68253517b80bd3bfd17eb4e84df6e"
+uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210"
+version = "1.3.0"
+
+[[LibCURL]]
+deps = ["LibCURL_jll", "MozillaCACerts_jll"]
+uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"
+
+[[LibCURL_jll]]
+deps = ["Artifacts", "LibSSH2_jll", "Libdl", "MbedTLS_jll", "Zlib_jll", "nghttp2_jll"]
+uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0"
+
+[[LibGit2]]
+deps = ["Base64", "NetworkOptions", "Printf", "SHA"]
+uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"
+
+[[LibSSH2_jll]]
+deps = ["Artifacts", "Libdl", "MbedTLS_jll"]
+uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8"
+
+[[Libdl]]
+uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
+
+[[Logging]]
+uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"
+
+[[Markdown]]
+deps = ["Base64"]
+uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"
+
+[[MbedTLS_jll]]
+deps = ["Artifacts", "Libdl"]
+uuid = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
+
+[[MozillaCACerts_jll]]
+uuid = "14a3606d-f60d-562e-9121-12d972cd8159"
+
+[[NetworkOptions]]
+uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908"
+
+[[Pkg]]
+deps = ["Artifacts", "Dates", "Downloads", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "REPL", "Random", "SHA", "Serialization", "TOML", "Tar", "UUIDs", "p7zip_jll"]
+uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
+
+[[Preferences]]
+deps = ["TOML"]
+git-tree-sha1 = "00cfd92944ca9c760982747e9a1d0d5d86ab1e5a"
+uuid = "21216c6a-2e73-6563-6e65-726566657250"
+version = "1.2.2"
+
+[[Printf]]
+deps = ["Unicode"]
+uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
+
+[[REPL]]
+deps = ["InteractiveUtils", "Markdown", "Sockets", "Unicode"]
+uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
+
+[[Random]]
+deps = ["Serialization"]
+uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+
+[[SHA]]
+uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
+
+[[Scratch]]
+deps = ["Dates"]
+git-tree-sha1 = "0b4b7f1393cff97c33891da2a0bf69c6ed241fda"
+uuid = "6c6a2e73-6563-6170-7368-637461726353"
+version = "1.1.0"
+
+[[Serialization]]
+uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
+
+[[Sockets]]
+uuid = "6462fe0b-24de-5631-8697-dd941f90decc"
+
+[[TOML]]
+deps = ["Dates"]
+uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
+
+[[Tar]]
+deps = ["ArgTools", "SHA"]
+uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
+
+[[UUIDs]]
+deps = ["Random", "SHA"]
+uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
+
+[[Unicode]]
+uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
+
+[[Zlib_jll]]
+deps = ["Libdl"]
+uuid = "83775a58-1f1d-513f-b197-d71354ab007a"
+
+[[ghr_jll]]
+deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"]
+git-tree-sha1 = "f5c8cb306d4fe2d1fff90443a088fc5ba536c134"
+uuid = "07c12ed4-43bc-5495-8a2a-d5838ef8d533"
+version = "0.13.0+1"
+
+[[nghttp2_jll]]
+deps = ["Artifacts", "Libdl"]
+uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d"
+
+[[p7zip_jll]]
+deps = ["Artifacts", "Libdl"]
+uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0"
diff --git a/.buildkite/rootfs_images/Project.toml b/.buildkite/rootfs_images/Project.toml
@@ -0,0 +1,5 @@
+[deps]
+Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
+SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce"
+Scratch = "6c6a2e73-6563-6170-7368-637461726353"
+ghr_jll = "07c12ed4-43bc-5495-8a2a-d5838ef8d533"
diff --git a/.buildkite/rootfs_images/README.md b/.buildkite/rootfs_images/README.md
@@ -0,0 +1,5 @@
+## Rootfs images
+
+Our CI setup makes use of rootfs images that contain our build tools.
+These rootfs images are built using the fairly simple scripts held within this directory.
+Most images are based on Debian, making use of `debootstrap` to provide a quick and easy rootfs with packages installed through an initial `apt` invocation.
diff --git a/.buildkite/rootfs_images/llvm-passes.jl b/.buildkite/rootfs_images/llvm-passes.jl
@@ -0,0 +1,27 @@
+#!/usr/bin/env julia
+
+## This rootfs includes enough of a host toolchain to build the LLVM passes.
+## Eventually, this image will probably be replaced with the actual builder image,
+## as that will have the necessary toolchains as well, but that image is not built yet.
+
+include("rootfs_utils.jl")
+
+# Build debian-based image with the following extra packages:
+packages = [
+    "build-essential",
+    "libatomic1",
+    "python",
+    "python3",
+    "gfortran",
+    "perl",
+    "wget",
+    "m4",
+    "cmake",
+    "pkg-config",
+    "curl",
+    "git",
+]
+tarball_path = debootstrap("llvm-passes"; packages)
+
+# Upload it
+upload_rootfs_image(tarball_path)
diff --git a/.buildkite/rootfs_images/rootfs_utils.jl b/.buildkite/rootfs_images/rootfs_utils.jl
@@ -0,0 +1,92 @@
+#!/usr/bin/env julia
+
+# This is an example invocation of `debootstrap` to generate a Debian/Ubuntu-based rootfs
+using Scratch, Pkg, Pkg.Artifacts, ghr_jll, SHA, Dates
+
+# Utility functions
+getuid() = ccall(:getuid, Cint, ())
+getgid() = ccall(:getgid, Cint, ())
+
+function debootstrap(name::String; release::String="buster", variant::String="minbase",
+                     packages::Vector{String}=String[], force::Bool=false)
+    if Sys.which("debootstrap") === nothing
+        error("Must install `debootstrap`!")
+    end
+
+    tarball_path = joinpath(@get_scratch!("rootfs-images"), "$(name).tar.gz")
+    if !force && isfile(tarball_path)
+        @error("Refusing to overwrite tarball without `force` set", tarball_path)
+        error()
+    end
+
+    artifact_hash = create_artifact() do rootfs
+        packages_string = join(push!(packages, "locales"), ",")
+        @info("Running debootstrap", release, variant, packages)
+        run(`sudo debootstrap --variant=$(variant) --include=$(packages_string) $(release) "$(rootfs)"`)
+
+        # Remove special `dev` files
+        @info("Cleaning up `/dev`")
+        for f in readdir(joinpath(rootfs, "dev"); join=true)
+            # Keep the symlinks around (such as `/dev/fd`), as they're useful
+            if !islink(f)
+                run(`sudo rm -rf "$(f)"`)
+            end
+        end
+
+        # take ownership of the entire rootfs
+        @info("Chown'ing rootfs")
+        run(`sudo chown $(getuid()):$(getgid()) -R "$(rootfs)"`)
+
+        # Write out rootfs-info to contain a minimally-identifying string
+        open(joinpath(rootfs, "etc", "rootfs-info"), write=true) do io
+            write(io, """
+            rootfs_type=debootstrap
+            release=$(release)
+            variant=$(variant)
+            packages=$(packages_string)
+            build_date=$(Dates.now())
+            """)
+        end
+
+        # Write out a reasonable default resolv.conf
+        open(joinpath(rootfs, "etc", "resolv.conf"), write=true) do io
+            write(io, """
+            nameserver 1.1.1.1
+            nameserver 8.8.8.8
+            """)
+        end
+
+        # Remove `_apt` user so that `apt` doesn't try to `setgroups()`
+        @info("Removing `_apt` user")
+        open(joinpath(rootfs, "etc", "passwd"), write=true, read=true) do io
+            filtered_lines = filter(l -> !startswith(l, "_apt:"), readlines(io))
+            truncate(io, 0)
+            seek(io, 0)
+            for l in filtered_lines
+                println(io, l)
+            end
+        end
+
+        # Set up the one true locale
+        @info("Setting up UTF-8 locale")
+        open(joinpath(rootfs, "etc", "locale.gen"), "a") do io
+            println(io, "en_US.UTF-8 UTF-8")
+        end
+        run(`sudo chroot --userspec=$(getuid()):$(getgid()) $(rootfs) locale-gen`)
+    end
+
+    # Archive it into a `.tar.gz` file
+    @info("Archiving", tarball_path, artifact_hash)
+    archive_artifact(artifact_hash, tarball_path)
+
+    return tarball_path
+end
+
+function upload_rootfs_image(tarball_path::String; github_repo::String="JuliaCI/rootfs-images")
+    # Upload it to `github_repo`
+    tag_name = "v1"
+    tarball_url = "https://github.com/$(github_repo)/releases/download/$(tag_name)/$(basename(tarball_path))"
+    @info("Uploading to $(github_repo)@$(tag_name)", tarball_url)
+    run(`$(ghr_jll.ghr()) -u $(dirname(github_repo)) -r $(basename(github_repo)) -replace $(tag_name) $(tarball_path)`)
+    return tarball_url
+end