PVF worker: separate worker binaries and build with musl

[mrcnski]

ISSUE

Overview

Per the security/sandboxing work in #882, we want to separate the PVF worker binaries and build them with musl. There are several advantages in doing this:

• Forcing the use of musl instead of whatever the system C lib is, means we can better control and sandbox the expected syscalls.
• It is also a win for more deterministic execution, i.e. more homogeneity in execution times and memory used, as opposed to when validators can use different libc's.
• [nvm: we already use LTO in production builds.] With a separate binary we can always apply LTO optimizations to this binary alone. LTO may make compile times prohibitively long for all of polkadot, but the impact on a small binary is much smaller and there may be performance wins. (Related: Optimize validator binaries with LTO polkadot#4311.)
• With smaller binaries we minimize the risk of ROP gadgets.

Todo

• move PVF workers into separate binaries
• adapt wasm-builder for worker binaries
• include the binary into polkadot executable at build-time (similar to this)
• ... some other stuff
• 1. Update the CI jobs, docker scripts, etc. to build the worker binaries with musl.
• 2. Update the validator guide with new instructions for building the worker binaries.
• 3. Announce to validators that the instructions have changed, and non-musl binaries will become a hard error in the next release.
• 4. Make musl workers a requirement for running secure mode.
• 5. Finally, enable full seccomp rules (logging violations for a while).

[mrcnski]

Despite my comment at #652, it seems that something like wasm-builder is the only way to embed executables. I had this hack which seemed to work, until I did cargo clean:

const PREPARE_EXE: &'static [u8] =
	include_bytes!(concat!(env!("OUT_DIR"), "/../../../prepare_worker"));
const EXECUTE_EXE: &'static [u8] =
	include_bytes!(concat!(env!("OUT_DIR"), "/../../../execute_worker"));

And env!("CARGO_BIN_EXE_prepare_worker") only works in integration tests :(. So I will go ahead and implement musl-builder by copy/pasting wasm-builder for now.

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/ux-of-distributing-multiple-binaries-take-2/2854/1

[mrcnski]

With #2009 merged, I believe this is the only task stopping us from enabling the full seccomp rules (only logging violations for now).

And, with #1663 we have everything we need to build with musl right in the repo.

What's left:

• 1. Update the CI jobs, docker scripts, etc. to build the worker binaries with musl.
• 2. Update the validator guide with new instructions for building the worker binaries.
• 3. Announce to validators that the instructions have changed, and non-musl binaries will become a hard error in the next release.
• 4. Make musl workers a requirement for running secure mode.
• 5. Finally, enable full seccomp rules (logging violations for a while).

[mrcnski]

If possible we should recommend a specific, set version of the musl toolchain for better determinism.

[mrcnski]

Anouncement / Release Schedule

The musl requirement should be announced at some point. It's not a requirement for on-demand, so a bit lower priority. I'm thinking we give secure-mode some time to settle in before adding the musl requirement, as it will affect all validators.

So if we release secure-mode in 1.5 then maybe we can announce the musl requirement in 1.7, adding a runtime warning. Then enforce it in 1.9 or something like this. Reason being, it might make sense to give more time than for secure-mode, because secure-mode will affect hardly anyone. On the other hand, the musl requirement simply involves doing a different build instruction, and not migrating to a different machine or upgrading the kernel, so while it affects everyone the impact is low.