Info regarding std and no_std incomplete or missing

[maltekliemann]

Content request

There doesn't seem to be a thorough explanation how exactly std and no_std relate to the Wasm binary,
neither in the tutorials nor the docs, and some of the hints on this topic seem very confusing to me. I apologize for the long issue.

In Add a Pallet, we get the following:

It is important to note that the Substrate runtime compiles to both a native Rust std binary and a WebAssembly (Wasm) binary. For more information about compiling std and no_std features, see XXX.

It's unfortunate that the link is missing (link to line). Where was this supposed to point?

That's all the info we get from Add a Pallet. The Build a proof of existence dApp tutorial has the following to say:

1. Add the macro required to build both the native Rust binary (std) and the WebAssembly (no_std) binary.

#![cfg_attr(not(feature = "std"), no_std)]

All of the pallets used in a runtime must be set to compile with the no_std features.

As a minor complaint, "no_std features" sounds wrong: std is a feature; no_std is an attribute used in Rust and doesn't really add any functionality. Other than that, the association of Wasm with no_std sounds ok, until you take a look into the runtime code:

#![cfg_attr(not(feature = "std"), no_std)]
// `construct_runtime!` does a lot of recursion and requires us to increase the limit to 256.
#![recursion_limit = "256"]

// Make the WASM binary available.
#[cfg(feature = "std")]
include!(concat!(env!("OUT_DIR"), "/wasm_binary.rs"));

// ...

The Wasm binary was just associated with no_std, why is it now exclusively included if std is enabled? From a stackexchange question, I gather the following: When cargo build is called for the runtime, first the build.rs builds a Wasm binary of the runtime (with no_std), then the runtime is built with std enabled and includes the Wasm binary.

But this doesn't really answer all of my questions. So I suggest that the following questions be answered/details pointed out:

• What happens exactly when we run cargo build?
• (Provided that my guess from above is correct) If I want a native (std) build, why is the Wasm binary (no_std) built and included? Why not just do a native build?
• How can I build only the no_std Wasm binary? Is there any purpose to doing that?

It's quite confusing to me that the Wasm binary, one of the defining features of substrate, is not explained in detail anywhere in the tutorials or the docs. I basically got stuck in the second tutorial trying to understand this.

If you could help me with answers to these questions, I would be perfectly happy to work them into the docs myself.

Are you willing to help with this request?

Maybe (please elaborate above)

[lisa-parity]

I got stuck trying to sort out exactly the same question. I'm working on some draft documentation for this topic, but I'm going to need some technical assistance because the current doc (which I took a stab at fixing and failed) seems contradictory. @shawntabrizi can you provide some detail around how this works or suggest someone I should reach out to?

[maltekliemann]

@lisa-parity Which doc exactly do you mean by "the current doc"?

[lisa-parity]

What I was looking at revising was in the Add a Nicks pallet tutorial (most likely, I would make this its own topic instead of embedding the information inside of the tutorial--that was the "plan" behind the XXX placeholder--oops, my bad!).
I see that you also mention the proof of existence tutorial (I think the content is similar). In both cases, older versions of the documentation weren't any clearer. Is there another bit of documentation on this topic that I should take a look at?

[maltekliemann]

@lisa-parity Making this its own topic seems like a good idea. There seems to be some duplicated content between these tutorials, but many of the more technical details are missing.

I found this:

Runtime: the logic that defines how blocks are processed, including state transition logic. In Substrate, runtime code is compiled to Wasm and becomes part of the blockchain's storage state. This enables one of the defining features of a Substrate-based blockchain: forkless runtime upgrades. Substrate clients may also include a "native runtime" that is compiled for the same platform as the client itself (as opposed to Wasm). The component of the client that dispatches calls to the runtime is known as the executor, whose role is to select between the native code and interpreted Wasm. Although the native runtime may offer a performance advantage, the executor will select to interpret the Wasm runtime if it implements a newer version.

For a while, I thought that this answered all my question. But the more I think about this, the more it confuses me. So in addition to all questions above, I now have this problem: If the Wasm binary is part of the on-chain storage, then it must be loaded during runtime, not during compile time (otherwise, how would forkless upgrades work?). But then why the Wasm binary also included in the native binary at compile-time. And what happens to the included Wasm binary when it becomes outdated due to a forkless upgrade?

Funny enough, the dedicated Key Concept page doesn't mention much of this: https://docs.substrate.io/v3/concepts/runtime/

I'm really looking forward to your update :)

[lisa-parity]

So this is very raw and not technically vetted but this is my best guess so far (and doesn't address your additional questions -- all good points that I don't have answers for):

The Substrate runtime compiles to both a native Rust binary that includes standard library (std) functions and a WebAssembly (Wasm) binary that is compiled without any standard library functions (no_std) but is (can be?) embedded (?) in the native Rust binary.
Because Rust programs rely on a Cargo.toml file to define the configuration settings and dependencies that determine what gets compiled in each binary, the Cargo.toml file controls two important pieces of information for compiling the Substrate runtime:

• The pallets to be imported as dependencies for the runtime, including the location and version of the pallets.
• The standard library features in each pallet that should be enabled when compiling the native Rust binary. By enabling the standard (std) feature set from each pallet, you can compile the runtime to include functions, types, and primitives that would otherwise be missing when you build the WebAssembly binary.

When you run the cargo build command, the compiler interprets the following macro to build the WebAssembly (no_std) binary for the runtime:

#![cfg_attr(not(feature = "std"), no_std)]

The compiler then builds the native Rust (std) binary—with all of the pallet std features that you enabled in the Cargo.toml file for the runtime—and embeds the WebAssembly in the resulting binary.
The macro in the runtime source code that accomplishes this taks looks like this:

// Make the WASM binary available.
#[cfg(feature = "std")]
include!(concat!(env!("OUT_DIR"), "/wasm_binary.rs"));

DISCLAIMER: This might not make any sense at all. I'm trying to piece together the facts from the clues 😉.

[maltekliemann]

@lisa-parity Thank you! That's mostly compatible with my understanding. Except (maybe I misunderstood)...

When you run the cargo build command, the compiler interprets the following macro to build the WebAssembly (no_std) binary for the runtime: [...]

The build.rs file is responsible for building the Wasm (see https://doc.rust-lang.org/cargo/reference/build-scripts.html and https://docs.rs/substrate-wasm-builder/latest/substrate_wasm_builder/). The "std" feature/no_std attribute is only for explicitly including/excluding lines of code from the two binaries.

[shawntabrizi]

What happens exactly when we run cargo build?

I think this question needs a bit more scope. What you have stated above is an accurate answer for this question.

We compile a Wasm runtime first, then use that Wasm to embed into the native client binary. This happens with the susbtrate-wasm-builder crate your linked to above.

If I want a native (std) build, why is the Wasm binary (no_std) built and included? Why not just do a native build?

One of the fundamental architecture decisions of Substrate/Polkadot is the use of a Wasm runtime. This Wasm runtime is embedded into the genesis state of your blockchain, and thus must exist to be embeded. This is not optional, but a requirement of the framework. You cannot build the Substrate client without some Wasm file being embeded into the binary. This file can be a dummy file (I think there is a flag for that), but then your node would break/panic if not executing everything in Native. Here is a note about that in Substrate:

/// Wasm binary unwrapped. If built with `SKIP_WASM_BUILD`, the function panics.
#[cfg(feature = "std")]
pub fn wasm_binary_unwrap() -> &'static [u8] {
	WASM_BINARY.expect(
		"Development wasm binary is not available. This means the client is built with \
		 `SKIP_WASM_BUILD` flag and it is only usable for production chains. Please rebuild with \
		 the flag disabled.",
	)
}

For local testing purposes you can build only the native client, skipping a new Wasm build like so:

SKIP_WASM_BUILD=1 cargo build --release

Here is the note on the environment variable:

- `SKIP_WASM_BUILD` - Skips building any Wasm binary. This is useful when only native should be recompiled.
                      If this is the first run and there doesn't exist a Wasm binary, this will set both
                      variables to `None`.

In this case,

How can I build only the no_std Wasm binary? Is there any purpose to doing that?

You can do that with the following script: https://github.com/paritytech/substrate/blob/master/.maintain/build-only-wasm.sh

build-only-wasm.sh node-runtime ./output.wasm

Yes, it can makes sense to compile only the Wasm binary, if for example you are just trying to provide the upgrade Wasm to perform a forkless upgrade. Practically speaking, usually when you do a runtime upgrade, you want to provide both a native and Wasm binary, so doing only one of those halves usually is an incomplete release process.

[maltekliemann]

@shawntabrizi Thanks for the answer! Unfortunately, I remain confused. Particularly by this:

This Wasm runtime is embedded into the genesis state of your blockchain, and thus must exist to be embeded. This is not optional, but a requirement of the framework. You cannot build the Substrate client without some Wasm file being embeded into the binary.

First of all, by embedded into the binary you mean that the Wasm binary is compiled into the native binary, like a string literal, right? And by embedded into the genesis state, you mean the Wasm binary is copied into on-chain storage, right? See Architecture.

And if so, what's the purpose of having the Wasm binary embedded into the native binary? After the first forkless update (which doesn't change the native binary... right?), the native binary would contain an outdated version of the Wasm binary. That doesn't sound right to me.

And in addition to that, what is the purpose of all this? A Wasm binary in on-chain storage for the purpose of forkless upgrades makes sense. But why build the native binary with this strange indirection?

[shawntabrizi]

The native binary at one point ran over 100x faster than the wasm binary, which for performance purposes was significant. People running the latest native client would reap those performance gains when running a node.

As of recent, improvements to the wasm execution has made it so the difference is only 10x or less. (still quite large. we hope for this to be equal one day)

We even have an issue to remove the native binary from our build process all together: paritytech/substrate#7288

So you are asking the right questions, just something that is still a work in progress.

And if so, what's the purpose of having the Wasm binary embedded into the native binary?

When starting a new chain you need that initial Wasm binary. The one for Polkadot and Kusama are included in the chain specification found here, (in the genesis state): https://github.com/paritytech/polkadot/tree/master/node/service/res

However, what do you think happens when you run a simple dev node starting on block 0? (it uses this embeded wasm)

Again, it is not strictly required, as you can skip the wasm build, and configure your node to never look at the wasm binary, however, this is not the greenfield scenario.

[maltekliemann]

@shawntabrizi Sorry, I'm still not sure I understand.

When starting a new chain you need that initial Wasm binary. The one for Polkadot and Kusama are included in the chain specification found here, (in the genesis state): https://github.com/paritytech/polkadot/tree/master/node/service/res

However, what do you think happens when you run a simple dev node starting on block 0? (it uses this embeded wasm)

So for production code, the "default" move is to place the original Wasm binary into on-chain storage at genesis, but a dev node will just use the Wasm embedded into the native binary (no copy)?

But this seems strange to me. If production code always refers to on-chain storage for the Wasm, why embed it into the native binary. Why not just build the native binary without embedding the Wasm.

I don't know, maybe I'm just being dumb, but this just doesn't check out for me.

[shawntabrizi]

The Wasm is needed to run a dev chain.

Like:

substrate --dev

In this case, genesis state is generated on the fly, and the Wasm needs to come from somewhere: https://github.com/paritytech/substrate/blob/master/bin/node/cli/src/chain_spec.rs#L298

GenesisConfig {
	system: SystemConfig { code: wasm_binary_unwrap().to_vec() },
	balances: BalancesConfig {
		balances: endowed_accounts.iter().cloned().map(|x| (x, ENDOWMENT)).collect(),
	},
	...
}

[maltekliemann]

@shawntabrizi Okay, that makes sense. I have two last questions:

• Is there any other purpose for integrating the Wasm into the native binary?
• Why can't/shouldn't we load the Wasm from the filesystem?

Oh, and thanks for bearing with me!

[shawntabrizi]

Is there any other purpose for integrating the Wasm into the native binary?

Not that I know of.

Why can't/shouldn't we load the Wasm from the filesystem?

You can, using wasm-runtime-overrides CLI flag:

	/// Get the path where WASM overrides live.
	///
	/// By default this is `None`.
	fn wasm_runtime_overrides(&self) -> Option<PathBuf> {
		self.import_params().map(|x| x.wasm_runtime_overrides()).unwrap_or_default()
	}

[maltekliemann]

@shawntabrizi Great! I think that clears everything up.

Thank you very much for the all answers! 👍

[sacha-l]

Did some digging and found some helpful context here here and here. Leaving this issue open as we write new docs that capture the information shared here.

[lisa-parity]

It is better documented now than before.