[buildkite] Implement secrets encryption and sandboxing

[staticfloat]

This adds a proof-of-concept demonstration of two new buildkite plugins:

• cryptic adds secrets management to privileged pipelines. These
  pipelines cannot be freely modified; their integrity is verified against
  a signature maintained by committers with a secret key. This allows
  certain portions of the CI configuration (which are privileged and can
  decrypt encrypted files/environment variables) to remain public, but
  read-only to the general populace.

• sandbox adds a generic sandboxing mechanism that allows CI steps to
  be run within user-provided rootfs images. We're using these here to
  provide compiler toolchains for the llvm-passes CI steps, and the plan
  is to eventually provide all compiler toolchains through such rootfs
  images.

[staticfloat]

🎉

[staticfloat]

@maleadt you're probably interested in this as well.

The workers are running via the configurations curated here: https://github.com/JuliaCI/sandboxed-buildkite-agent/blob/main/linux-sandbox.jl/build_systemd_config.jl

I intend to add windows/macos/freebsd setups there as well in the future (these workers will only be for base CI, I don't intend to run CI for the ecosystem on these). I am using the sandbox as the isolation mechanism for these workers as well because it's nestable. This allows us to run sandboxes inside of sandboxes, which is how we're able to do this rootfs nonsense for the LLVM pipelines.

Eventually, our buildbot configuration is just going to be a rootfs image that gets downloaded onto a machine and unpacked as an artifact. And yes, this is basically just us reinventing Docker, but since they can't nest, it's just impossible to use it securely without opening the host up for trivial exploitation.

Also, this showcases some simple secrets management with cryptic. I'll expand the example here after this gets merged and I expand Dilum's codecov work.

[DilumAluthge]

I disabled Buildkite commit statuses since they were failing on some PRs (e.g. #41264).

So we should remember to re-enable Buildkite commit statuses right before we merge this PR.

[staticfloat]

I have re-enabled buildkite commit statuses

[DilumAluthge]

Does this need to be backported to 1.6 and 1.7? Otherwise, Buildkite statuses will always be failing on the release-1.6 and release-1.7 branches.

[staticfloat]

Sure, it can be backported.

…

On Fri, Jun 18, 2021 at 05:32 Dilum Aluthge ***@***.***> wrote: Does this need to be backported to 1.6 and 1.7? Otherwise, Buildkite statuses will always be failing on the release-1.6 and release-1.7 branches. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#41256 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA762DIW62JUB3NLZ37QODTTM4FNANCNFSM464OZHOQ> .