PVF workers: consider zeroing all process memory

[mrcnski]

ISSUE

Overview

As part of our security work, we are concerned about attackers executing arbitrary code and getting access to sources of randomness. This might allow them to e.g. vote against a candidate with 50% chance, preventing consensus and stalling the chain.

@eskimor brought up zeroing out memory to eliminate a source of randomness (uninitialized memory), so I looked into it more. Unfortunately there are still other sources of randomness that we would need virtualization for: #652. It might still be good to do this where feasible.

Zeroing the heap

The modern OS already provides a zeroed-out, COW shared page on first allocation, but on subsequent allocations it reuses any pages that were released by the same process without zeroing them out. This mechanism is mainly intended to prevent data leaking from other processes. So an attacker could still abuse subtle indeterminacies in the allocator's memory reuse strategy from the same process to get some pseudo-randomness.¹ If it's not too expensive, maybe we could have the allocator wrapper do a similar strategy to provide "zeroed" memory whenever it provides it to the process.

Zeroing the stack

We could zero-out up to the native stack limit. To prevent the operation from being optimized away we can use something like explicit_bzero which is supported by Rust libc, but it apparently does not provide full protection. Might be better than nothing. We should also make sure that stack protection is turned on (see below).

Alternative: binary hardening checks

Modern OS/architectures do provide some limited protections against exploits of this kind, as long as the binary was built properly. I raised this issue to add some checks for that.

Footnotes

1. On the other hand though, subtle indeterminacies will always exist and it's not feasible to eliminate them all. ↩

[s0me0ne-unkn0wn]

We have alloc and alloc_zeroed in the global allocator interface (generally, alloc_zeroed is just alloc with the following bzero, but some allocators, like Jemalloc, implement their own alloc_zeroed logic).

What if we just forward all the allocs to alloc_zeroed in the wrapper allocator? Would it be enough?

[mrcnski]

In this case we have to consider performance. I'm not sure if forwarding alloc to alloc_zeroed can be guaranteed to be inlined or if it might be additional overhead. Would be good to have three benchmarks (with the production profile): without any changes, with forwarding, and with bzero in alloc.

[burdges]

Is there some trick to tell libc and/or linux that we should no longer have access to the old pages? What if you give the PVF worker a new pid?

pid_t process = fork();
if (process < 0) {
  panic();
}
// Die while our child lives
if (process != 0) {
  exit(0);
}
// Child continues here,

[mrcnski]

@burdges Interesting idea, we could fake a new process in-between PVF jobs, as right now all jobs run on the same process and previous jobs would provide randomness on the heap. OTOH if we benchmark zeroing all alloc'ed pages and the performance is okay, then this wouldn't be needed, otherwise we should explore it.

[koute]

Is there some trick to tell libc and/or linux that we should no longer have access to the old pages?

Well, there's madvise(MADV_DONTNEED) which one can use on specific pages to zero them out.