-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow creating and writing directly to buffers in wasm memory from JS #1079
Comments
Thanks for the report! I'm not actually sure how we'd best support this! This seems like it may actually be best done in user code rather than in wasm-bindgen itself though? There's some degree of memory unsafety happening here due to the fact that it's uninitialized memory being exposed to the outside world. Do you have an idea though about how we could expose something like this to JS? FWIW I'd definitely avoid the |
I think this is now best done with functions like |
@alexcrichton So I think Rust: #[wasm_bindgen]
pub struct WasmMemBuffer {
buffer: Vec<u8>,
}
#[wasm_bindgen]
impl WasmMemBuffer {
#[wasm_bindgen(constructor)]
pub fn new(byte_length: u32, f: &js_sys::Function) -> Self {
let mut buffer = vec![0; byte_length as usize];
unsafe {
let array = js_sys::Uint8Array::view(&mut buffer);
f.call1(&JsValue::NULL, &JsValue::from(array))
.expect("The callback function should not throw");
}
Self { buffer }
}
}
fn compute_buffer_hash_impl(data: &[u8]) -> u32 { ... }
#[wasm_bindgen]
pub fn compute_buffer_hash(data: &WasmMemBuffer) -> u32 {
compute_buffer_hash_impl(&data.buffer)
} JavaScript: let buffer = new WasmMemBuffer(100000, array => {
// "array" wraps a piece of wasm memory. Fill it with some values.
for (let i = 0; i < array.length; i++) {
array[i] = Math.floor(Math.random() * 256);
}
});
let hash = compute_buffer_hash(buffer); // No buffer copy when passing this argument. Yay!
buffer.free();
console.log(hash); This is great, but I think it's lying to the rust compiler: |
In the past, we were relying on wasm-bindgen implementation details. See rustwasm/wasm-bindgen#1079 for more information.
Yes this is a case where we're sort of lying to the compiler a bit, but afaik there's no practical ramifications today and we can add bindings if necessary in the future. |
I have a use case where my JS code needs to load a 2GB file and then pass its contents to a wasm-compiled rust function, which expects the data as a
&[u8]
parameter. If I do this the straightforward way, I end up with 4GB of memory usage and an extra 2GB copy:Uint8Array
and reads the data into it.wasm-bindgen
-generated binding function and passes it that typed array.I would like to avoid the extra 2GB of memory usage and the copy. Ideally, wasm-bindgen would generate an API to allocate a buffer, expose that buffer to JS as a typed array that directly wraps the wasm memory buffer, then let me write my data into that typed array from JS (while I promise not to call any other wasm functions as long as I'm using that typed array, because doing so might recreate the wasm memory's underlying array buffer), and then also let me pass that buffer handle to any generated binding functions that accept
&[u8]
parameters.At the moment, I'm working around this problem by bypassing the
wasm-bindgen
-generated binding function and calling the underlying wasm function directly, as well as__wbindgen_malloc
and__wbindgen_free
. It would be nice if this workaround wasn't necessary.The text was updated successfully, but these errors were encountered: