Addressing crate local state for procedural macros problem

[mikevoronov]

Hi! Thank you for this cool work!

I think that this approach could also address this issue. More precisely, I mean persistent data storage for all macros. This could be obtained by Wasm multimodules approach and sharing the same memory for one modules. Moreover, SQLite or Redis could be used to save state. All of them compiled in such a way to work like in-memory db without any imports. With our proc macro they could be used in the following way:

#[invocation_handler(side_modules = redis)]
fn run(arg: String) -> Vec<u8> {
    redis::call("SET A 1")
}

#[invocation_handler(side_modules = sqlite)]
fn run(arg: String) -> Vec<u8> {
    sqlite::call("SELECT * FROM Table")
}

And I can adapt them to any convention.

What do you think about that? I haven't look at the internals of watt, so can't estimate how many should be add to support this idea.

[bjorn3]

This doesn't solve part two of rust-lang/rust#44034 (comment):

There is no guarantee of ordering -- if do_it! needs data from all config! invocations, that's a problem.

You also have the problem that there is no place to store the database. watt can't just think of a place to store it without co-operation with cargo. For build scripts, cargo passes the OUT_DIR env var, but for rustc invocations it only does it when there was a build script. For rustc invocations, cargo assumes that nobody will write to the OUT_DIR. Any persistence of state without co-operation with cargo is a problem, as cargo doesn't know if and when it needs to invalidate the state. It is also completely valid for the rustc process to have read-only access to all files except for the /tmp dir and the output artifacts. In fact I wouldn't be surprised if for example bazel already does this. In any case bazel doesn't persist the state across rustc invocations. Every rustc invocation gets it's own container.

[mikevoronov]

@bjorn3 thank you for the answer

There is no guarantee of ordering -- if do_it! needs data from all config! invocations, that's a problem.

What about the next answer on that topic with depends_on feature of proc macro?

Regarding the other - all of proposed DBs are only in-memory, so access to filesystem isn't needed, only thing that need is using the same memory in macro interpreting (sorry, I've added this later while modifing the issue text). To be honest, I don't know is it possible with watt to make memory persistent.

[bjorn3]

What about the next answer on that topic with depends_on feature of proc macro?

That would need to be added to rustc first.

[mikevoronov]

Yes, you are right. Let's consider another approach.

I think that significant part of the people, whom this issue is addressed to, want to just two-stage scheme when some information is collected from applying proc macros and on the second stage this infomation is used by some other code that need to be run automatically on compile time.

In my case I am developing a SDK for Wasm with support of interface-types generation. Smth similar to wasm-bindgen, but mostly for server-side platform without integration with JS. Moreover, wasm-bindgen works in two forms - at first, it embeds some additional info to wasm binary (such as describe functions and some static arrays) by applying proc macro and then it uses all of this info in CLI form by cargo wasi/wasm-pack. With watt I believe this two form could be reduces just to one.

It could be done by registering a closure in dtor of some static object inside wa-like crate with our proc macros declaration. This dtor's closure needs only an access to Wasm memory that should previously be filled with neccessary info from proc macros.

[bjorn3]

You would still need a seperate step for wasm-bindgen. First multiple crates may independently use it, in which case the proc-macro can't ever read the info from the other crate. (think sibling crates) Two there is no way to generate js using the proc-macro. There is no place to write it to.

[mikevoronov]

First multiple crates may independently use it

Haven't thought about it, and don’t know whether it is possible to solve it at all...

Two there is no way to generate js using the proc-macro.

I meant that this will be done by dtor of some global object inside crate with proc-macro definition. Based on the wa example it could be written smth like that:

struct Test {
  wasm_macro: WasmMacro,
  final_state_generator: Box<dyn FnOnce <...>>,
}

// assuming that this is possible in Rust
static mut TEST: Test = Test::new(WasmMacro::new(WASM), Box::new(wasm_state_generation));

static WASM: &[u8] = include_bytes! {
    "../../impl/target/wasm32-unknown-unknown/release/watt_demo.wasm"
};

impl Drop for Test {
  fn drop(&mut self) {
    self.final_state_generator(self.wasm_macro.get_state());
  }
}

fn wasm_state_generation(wasm_state: &WasmState) {
   // generating js, interface-types and other stuff based on the Wasm state
}

#[proc_macro_derive(Demo)]
pub fn demo(input: TokenStream) -> TokenStream {
    TEST.proc_macro_derive("demo", input)
}