Concept of Diagnostic Runtimes

[pepyakin]

This issue #1790 made me to come up with an idea of a diagnostic runtime.

To recap: In contracts we have the lack of debuggability problem. In the best case, if the top-level contract fails you get the execution failure error. In the worst case, because contract failures don't cascade by default (i.e. if a leaf contract call fails it doesn't propagate to the root call) you don't get anything. There are even no way to log a message.

The simplest way of solving it is by adding some little features such as posting an event every time a contract generates an error, or e.g. add a special function that logs message that is available only for a testnet.

The problem with a testnet only debugging capabilities that they are no use for the production nets. Mixing debugging code into the runtime is also not the best idea either, because it adds overhead, possibly pessimizes the normal path, increases binary size and enlarges security surface.

It seems to me that a better solution would be to provide this functionality off chain.

Enter diagnostic runtime.

Imagine we had:

1. A special cfg feature in a runtime, which enables some off-chain debugging capabilities.
2. An RPC method that is the same as state_call, but that takes one extra parameter: a file path to a wasm runtime to use to execute.
3. Potentially, a way for a runtime to output a large chunk of data that can be returned by the aforementioned RPC method. Possibly in a file.

Having this, we could add a special path (behind a cfg feature) in the contract module logic which output a trace information.

This, for example, can be done exclusively on the runtime level without touching the substrate host. Here are a few ideas:

1. We can log traps in the trace.
2. We can record calls/instantiations, their parameters such as transferred value, consumed gas, input/output buffers.
3. At the present, the contract module performs instrumentation before executing a contract (e.g. to inject gas metering statements). We could alter the instrumentation so as to add extra statements and tracing. Here are few examples:
   1. We can insert a call to an ext_begin_fn function in the prologue and a call to an ext_end_fn in the epilogue of each function. With this we can construct stack traces inside of contracts in the case of errors.
   2. Or could can instrument every execution path to
   3. We could log every parameter
   4. In the extreme, we could instrument every instruction and get every operand trace.
   5. or as an another extreme option, we could alter wasmi and compile it in wasm and use it instead of using sandboxing.
4. we can add a function ext_print to the contract runtime. In the normal path it just doesn't do anything (i.e. don't even charge for gas for reading the code), but if the diagnostic feature is enabled the function reads the given buffer and records it in the trace. I think @Robbepop could be interested in this one.

After the execution we can assemble this as a trace and spew it out somehow, which could be decoded by tools (or alternatively, we could use human-readable format) or block explorers.

This setup gives us sheer number of possibilities for diagnostics without burdening the substrate host APIs. There is a caveat though: diagnostic runtimes likely would want to be state-compatible (i.e. produce the same state root) with the on-chain runtime just for the sake of being exchangable (i.e. for doing execute_block) and that might be tricky in one cases and limiting in other. However, this is not a hard requirement.

I think it might be possible that this mechanism could be used outside of the contracts module. For instance, we could use this use mechanism for running quick experiments with on-chain data, quickly testing new versions of runtimes at the development time or before an upgrade.

[xlc]

I would like to have ability to run this diagnostic runtime natively so it is possible to use debugger to debug it. It will be even better if there is a way to run the contracts natively as well.

[kianenigma]

An RPC method that is the same as state_call, but that takes one extra parameter: a file path to a wasm runtime to use to execute.
Potentially, a way for a runtime to output a large chunk of data that can be returned by the aforementioned RPC method. Possibly in a file.

@emostov once we

1. implement arbitrary runtime api calls
2. proper integration into wasm-override instead of the current hack of writing into :code

we basically achieve the same thing through CLI in try-runtime.

[liamaharon]

@kianenigma @xlc @pepyakin I wonder if we can condense this issue down into some concrete feature requests for try-runtime and try-runtime-cli?

I see potentially:

1. Ability to execute the runtime natively so it can hook into debugging tools
2. Patterns for adding verbose diagnostic logs behind a try-runtime feature flag (that actually works for all types - I've experienced weird issues with existing logging where many variables are logged as empty strings)
3. Perhaps some patterns for debugging SCs with try-runtime/-cli?

[xlc]

(1) will be useful.
(2) is most likely you need sp-debug-derive/force-debug enabled wasm build

[liamaharon]

(2) is most likely you need sp-debug-derive/force-debug enabled wasm build

is there any reason not to implicitly enable this when building with --features try-runtime?