[wasm] Webcil-in-WebAssembly #85932

lambdageek · 2023-05-08T19:17:51Z

Define a WebAssembly module wrapper for Webcil assemblies.
Contributes to #80807

Why

In some settings serving application/octet-stream data, or files with weird extensions will trigger firewalls or AV tools. But let's assume that if you're interested in deploying a .NET WebAssembly app, you're in an environment that can at least serve WebAssembly modules.

How

Essentially we serve this WebAssembly module:

(module
  (data "\0f\00\00\00") ;; data segment 0: payload size
  (data "webcil Payload\cc")  ;; data segment 1: webcil payload
  (memory (import "webcil" "memory") 1)
  (global (export "webcilVersion") i32 (i32.const 0))
  (func (export "getWebcilSize") (param $destPtr i32) (result)
    local.get $destPtr
    i32.const 0
    i32.const 4
    memory.init 0)
  (func (export "getWebcilPayload") (param $d i32) (param $n i32) (result)
    local.get $d
    i32.const 0
    local.get $n
    memory.init 1))

The module exports two WebAssembly functions getWebcilSize and getWebcilPayload that write some bytes (being the size or payload of the webcil assembly) to the linear memory at a given offset. The module also exports the constant webcilVersion to version the wrapper format.

So a runtime or tool that wants to consume the webcil module can do something like:

const wasmModule = new WebAssembly.Module (...);
const wasmMemory = new WebAssembly.Memory ({initial: 1});
const wasmInstance =
      new WebAssembly.Instance(wasmModule, {webcil: {memory: wasmMemory}});
const { getWebcilPayload, webcilVersion, getWebcilSize } = wasmInstance.exports;
console.log (`Version ${webcilVersion.value}`);
getWebcilSize(0);
const size = new Int32Array (wasmMemory.buffer)[0]
console.log (`Size ${size}`);
console.log (new Uint8Array(wasmMemory.buffer).subarray(0, 20));
getWebcilPayload(4, size);
console.log (new Uint8Array(wasmMemory.buffer).subarray(0, 20));

How (Part 2)

But actually, we will define the wrapper to consist of exactly 2 data segments in the WebAssembly data section: segment 0 is 4 bytes and encodes the webcil payload size; and segment 1 is of variable size and contains the webcil payload.

So to load a webcil-in-wasm module, the runtime gets the raw bytes of the WebAssembly module (ie: without instantiating it), and parses it to find the data section, assert that there are 2 segments, ensure they're both passive, and get the data directly from segment 1.

Remaining work

Add an option to the Webcil converter to emit a wasm module containing the webcil assembly as a passive data segment
Add code to Mono's webcil loader to parse WASM modules and find the data segment
Update the webcil spec
Change the extension from ".webcil" to ".wasm" and update the wasm SDK assets logic and the runtime's extension probing logic
WebcilReader (used by DebugStore) needs ot understand webcil-in-wasm

ghost · 2023-05-08T19:18:21Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Define a WebAssembly module wrapper for Webcil assemblies.

Why

In some settings serving application/octet-stream data, or files with weird extensions will trigger firewalls or AV tools. But let's assume that if you're interested in deploying a .NET WebAssembly app, you're in an environment that can at least serve WebAssembly modules.

How

Essentially we serve this WebAssembly module:

(module
  (data "\0f\00\00\00") ;; data segment 0: payload size
  (data "webcil Payload\cc")  ;; data segment 1: webcil payload
  (memory (import "webcil" "memory") 1)
  (global (export "webcilVersion") i32 (i32.const 0))
  (func (export "getWebcilSize") (param $destPtr i32) (result)
    local.get $destPtr
    i32.const 0
    i32.const 4
    memory.init 0)
  (func (export "getWebcilPayload") (param $d i32) (param $n i32) (result)
    local.get $d
    i32.const 0
    local.get $n
    memory.init 1))

The module exports two WebAssembly functions getWebcilSize and getWebcilPayload that write some bytes (being the size or payload of the webcil assembly) to the linear memory at a given offset. The module also exports the constant webcilVersion to version the wrapper format.

So a runtime or tool that wants to consume the webcil module can do something like:

const wasmModule = new WebAssembly.Module (...);
const wasmMemory = new WebAssembly.Memory ({initial: 1});
const wasmInstance =
      new WebAssembly.Instance(wasmModule, {webcil: {memory: wasmMemory}});
const { getWebcilPayload, webcilVersion, getWebcilSize } = wasmInstance.exports;
console.log (`Version ${webcilVersion.value}`);
getWebcilSize(0);
const size = new Int32Array (wasmMemory.buffer)[0]
console.log (`Size ${size}`);
console.log (new Uint8Array(wasmMemory.buffer).subarray(0, 20));
getWebcilPayload(4, size);
console.log (new Uint8Array(wasmMemory.buffer).subarray(0, 20));

How (Part 2)

But actually, we will define the wrapper to consist of exactly 2 data segments in the WebAssembly data section: segment 0 is 4 bytes and encodes the webcil payload size; and segment 1 is of variable size and contains the webcil payload.

So to load a webcil-in-wasm module, the runtime gets the raw bytes of the WebAssembly module (ie: without instantiating it), and parses it to find the data section, assert that there are 2 segments, ensure they're both passive, and get the data directly from segment 1.

Remaining work

Add an option to the Webcil converter to emit a wasm module containing the webcil assembly as a passive data segment
Add code to Mono's webcil loader to parse WASM modules and find the data segment
Update the webcil spec
Change the extension from ".webcil" to ".wasm" and update the wasm SDK assets logic and the runtime's extension probing logic

Author:	lambdageek
Assignees:	lambdageek
Labels:	`arch-wasm`, `area-AssemblyLoader-mono`, `area-Build-mono`, `area-VM-meta-mono`
Milestone:	-

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilConverter.cs

kg · 2023-05-08T19:23:20Z

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilWasmWrapper.cs

+// length of the webcil payload.  segment 1 is of a variable size and contains the webcil payload.
+//
+// the unchanging parts are stored as a "prefix" and "suffix" which contain the bytes for the following
+// WAT program, split into the parts that come before the data section, and the bytes that come after:


nit: i think in this case it's a WAT module; it's not a program since it isn't actually executable in a meaningful sense

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilWasmWrapper.cs

kg · 2023-05-08T19:32:53Z

src/mono/mono/utils/wasm-module-reader.c

+
+	gboolean stop = FALSE;
+	while (success && !stop && ptr < boundp) {
+		success = visit_section (ptr, boundp, &ptr, visitor, user_data, &stop);


this seems to implicitly rely on visit_section to advance to the end of the section, and visit_section seems to implicitly rely on section_visitor to do that. am I misunderstanding it? I think it would be ideal if visit_section always ensured that endp ended up in the right place

oooh. good catch. actually I explicitly wanted visit_section to advance ptr and not to rely on the visitor (which is "user code" - and I don't actually want ot trust it to do the traversal). There's a missing line in visit_section to bump ptr before returning

This was a copy/paste mistake from my toy standalone prototype

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilWasmWrapper.cs

lambdageek · 2023-05-09T17:28:55Z

WBT tests should work now...

lambdageek · 2023-05-09T18:50:44Z

~~(The .webcil->.wasm commits are not... super-duper clean yet, but it seems like at least for simple cases everything is working...)~~

Also works with Blazor (at least as far as network loading is concerned. actually running tests is blocked on dotnet/installer#16318 - which includes dotnet/aspnetcore#48067)

lambdageek · 2023-05-09T20:23:53Z

Ok this is ready for review

lambdageek · 2023-05-11T21:10:11Z

debugger tests are working in manual testing.

next steps:

figure out why the icu sharding WBT tests aren't working on CI
add a CI leg that runs debugger tests with webcil

lambdageek · 2023-05-12T02:50:23Z

I think I somehow managed to make the runtime load both "System.Private.CoreLib.dll" and "System.Private.CoreLib.wasm"... otherwise I don't understand how despite a catch (CultureNotFoundException), I'm getting an uncaught CultureNotFoundException...

pavelsavara · 2023-05-12T09:37:16Z

fr-FR is an invalid culture identifier.

I wonder is this is related to ICU data files or to some satellite assemblies ?

ilonatommy · 2023-05-12T10:04:09Z

fr-FR is an invalid culture identifier.
I wonder is this is related to ICU data files or to some satellite assemblies ?

ICU data files are loaded fine in the test. We are loading custom ICU file that does not contain fr-FR. We are also setting PredefinedCulturesOnly to true - these two things should result in CultureNotFoundException when we try to create fr-FR cultureand they do, we get fr-FR is an invalid culture identifier., but it seems we are not getting into the catch block, so exception type must have changed. I'll edit the tests to see where it comes from.

lambdageek · 2023-05-12T13:40:07Z

but it seems we are not getting into the catch block, so exception type must have changed

As far as I can tell it's the same exception as before. I think what is happening is that the webcil packaging is somehow allowing the runtime to load the same assembly (CoreLib) twice. So somehow the CultureNotFoundException from the throw, and the CultureNotFoundException from the catch are from two different copies of CoreLib. And so the runtime can't match them up.

I think I see how it is happening. I'm going to try to repro locally

lambdageek · 2023-05-12T20:24:41Z

I think I see how it is happening. I'm going to try to repro locally

I can repro locally, but I think I was wrong about what is going wrong - or at least i'm getting a different-seeming failure locally.

I think getting resource streams from assemblies isn't working right when webcil-in-wasm (it does appear to work right with standalone .webcil. So it's something about offsets into a MonoImage:raw_data - or else wrapping the image in .wasm is being done incorrectly and i'm truncating the image somehow...

lambdageek · 2023-05-15T19:02:21Z

I think I understand what the issue was. It wasn't resources (although there was an issue with resources) and it wasn't somehow loading the same assembly multiple times.

The issue is alignment.

When we load webcil-in-wasm by reading the wasm module directly, the webcil payload wasn't aligning on a good bounadry. The binary WASM format doesn't really take care to align data segments on multi-byte boundaries - and the uleb128 encoding of all the internal data sizes would make that hard to achieve, anyway. On the other hand, ECMA-335 does care about the internal boundaries:

II.25.4.5 Method data section:

At the next 4-byte boundary following the method body can be extra method data sections. These method data sections start with a two byte header (1 byte for flags, 1 byte for the length of the actual data) or a 4-byte header (1 byte for flags, and 3 bytes for length of the actual data). The first byte determines the kind of the header, and what data is in the actual section:

As a result, the code for parsing a the exception clause information for an ECMA-335 method header was doing a call to align the input pointer to a 4-byte boundary and ending up at an unexpected offset. Which resulted in mis-parsing the header and not setting up a correct catch clause for the CultureNotFoundException when webcil-in-wasm was turned on.

runtime/src/mono/mono/metadata/metadata.c

Lines 4157 to 4158 in 63e6777

    
           /* align on 32-bit boundary */ 
        
           ptr = dword_align (ptr);

The solution is to change the spec slightly and use data segment 0 not only to carry the size of the webcil payload, but also to add extra padding bits so that the content of data segment 1 (not segment 1's header!) ends up on a 16-byte boundary (smaller is probably ok, but 16 is probably a safer choice).

(We don't want to somehow pad out segment 1's data because in the case where we do use module instantiation then we wouldn't want the segment 1 data to be extracted by getWebcilPayload to an address with a proper alignment.)

Adding padding to segment 0 is ok because we only want the first 4 bytes of it anyway.

Add padding to data segment 0 to ensure that data segment 1's payload (ie the webcil content itself) is 4-byte aligned

lambdageek · 2023-05-15T20:28:05Z

pushed a commit to fix the alignment problem. also pushed some debugging junk, so that will need to be backed out before we merge. but I want to see how CI does.

Also there's a fix here to ensure the debugger sees the debugging sections of the webcil image.

The upshot is that for in-memory MonoImageStorage, the webcil loader will bump MonoImage:raw_data to the start of the payload - otherwise the PDB data's embedded offset pointers (which count from the beginning of the "PE" file are wrong). To de-allocate the image when we close, we use the existing MonoImage:raw_data_handle field (which was previously only used for mmap handles and such) to point to the beginning of the malloc'd memory and pass it to free.

instead just keep track of the webcil offset in the MonoImageStorage. This introduces a situation where MonoImage:raw_data is different from MonoImageStorage:raw_data. The one to use for accessing IL and metadata is MonoImage:raw_data. The storage pointer is just used by the image loading machinery

lambdageek · 2023-05-16T17:04:01Z

WBT is passing 🎉

but over in the other PR #86330 some debugger tests are failing, I want to get those fixes in here before merging this PR . (I couldn't figure out how to add a new variant of the debugger runs yet - so I'm testing webcil-in-wasm debugger support by changing the default packaging and running the "normal" debugger lane).

lambdageek · 2023-05-16T20:10:22Z

Stepping tests were failing with plain .webcil also. it's a problem in the debugger's webcil parser where it is calling BlobReader.ReadUtf8NullTerminated using reflection and failing to pick up the updated state (BlobReader is a struct type). I'll fix it in the "on by default" PR.

This one is

dotnet-issue-labeler bot added the area-Build-mono label May 8, 2023

ghost assigned lambdageek May 8, 2023

lambdageek added arch-wasm WebAssembly architecture area-AssemblyLoader-mono area-VM-meta-mono labels May 8, 2023

lambdageek removed area-Build-mono area-VM-meta-mono labels May 8, 2023

lambdageek requested review from pavelsavara, radekdoulik and kg May 8, 2023 19:18

kg reviewed May 8, 2023

View reviewed changes

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilConverter.cs Show resolved Hide resolved

kg reviewed May 8, 2023

View reviewed changes

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilWasmWrapper.cs Show resolved Hide resolved

lambdageek force-pushed the webcil-wasm-wrapper branch from 798112c to 2aff523 Compare May 8, 2023 19:25

kg reviewed May 8, 2023

View reviewed changes

This was referenced May 8, 2023

IOException running NuGet-Migrations during tests in dotnet CLI first run #80619

Closed

[mono][wasm] System.Numerics.Vectors.Tests assert equal failure #84775

Closed

pavelsavara reviewed May 9, 2023

View reviewed changes

src/libraries/Microsoft.NET.WebAssembly.Webcil/src/Webcil/WebcilWasmWrapper.cs Show resolved Hide resolved

lambdageek mentioned this pull request May 9, 2023

[tracking] Publish .NET assemblies in Webcil files #80807

Closed

16 tasks

lambdageek marked this pull request as ready for review May 9, 2023 20:10

lambdageek requested review from radical, lewing, thaystg, vargaz and SamMonoRT as code owners May 9, 2023 20:10

lambdageek force-pushed the webcil-wasm-wrapper branch from f9936b6 to f879fcd Compare May 9, 2023 20:15

why fail?

61afae7

did we load the same asm twice??

8a42b70

lambdageek force-pushed the webcil-wasm-wrapper branch from cfd08f5 to 8a42b70 Compare May 12, 2023 02:58

lambdageek added 2 commits May 15, 2023 11:06

Merge remote-tracking branch 'origin/main' into webcil-wasm-wrapper

ccd074e

checkpoint. things are broken. but I adjusted MonoImage:raw_data

6166a67

align webcil payload to a 4-byte boundary within the wasm module

eedf447

Add padding to data segment 0 to ensure that data segment 1's payload (ie the webcil content itself) is 4-byte aligned

lambdageek added 7 commits May 15, 2023 22:06

remove WIP tracing

d9b396e

assert that webcil raw data is 4-byte aligned

89ef589

revert unrelated build change

d9a3fea

revert unrelated change

6de7dee

revert whitespace

88f956e

revert WBT debugging output changes

0542c51

add 4-byte alignment requirement to the webcil spec

7c0643d

build-analysis bot mentioned this pull request May 16, 2023

Timeout in System.Net.Quic.Functional.Tests #86019

Closed

lambdageek force-pushed the webcil-wasm-wrapper branch from d1b27c0 to 15f6666 Compare May 16, 2023 15:03

lambdageek mentioned this pull request May 16, 2023

[wasm] Set WasmEnableWebcil to true by default #86330

Merged

lambdageek force-pushed the webcil-wasm-wrapper branch from b1a9934 to e187b00 Compare May 16, 2023 15:18

lambdageek merged commit 55c4e8c into dotnet:main May 16, 2023

ghost locked as resolved and limited conversation to collaborators Jun 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wasm] Webcil-in-WebAssembly #85932

[wasm] Webcil-in-WebAssembly #85932

lambdageek commented May 8, 2023 •

edited

Loading

ghost commented May 8, 2023

Why

How

How (Part 2)

Remaining work

kg May 8, 2023

kg May 8, 2023

lambdageek May 8, 2023 •

edited

Loading

lambdageek commented May 9, 2023

lambdageek commented May 9, 2023 •

edited

Loading

lambdageek commented May 9, 2023

lambdageek commented May 11, 2023 •

edited

Loading

lambdageek commented May 12, 2023

pavelsavara commented May 12, 2023

ilonatommy commented May 12, 2023 •

edited

Loading

lambdageek commented May 12, 2023

lambdageek commented May 12, 2023

lambdageek commented May 15, 2023 •

edited

Loading

lambdageek commented May 15, 2023

lambdageek commented May 16, 2023

lambdageek commented May 16, 2023

[wasm] Webcil-in-WebAssembly #85932

[wasm] Webcil-in-WebAssembly #85932

Conversation

lambdageek commented May 8, 2023 • edited Loading

Why

How

How (Part 2)

Remaining work

ghost commented May 8, 2023

Why

How

How (Part 2)

Remaining work

kg May 8, 2023

Choose a reason for hiding this comment

kg May 8, 2023

Choose a reason for hiding this comment

lambdageek May 8, 2023 • edited Loading

Choose a reason for hiding this comment

lambdageek commented May 9, 2023

lambdageek commented May 9, 2023 • edited Loading

lambdageek commented May 9, 2023

lambdageek commented May 11, 2023 • edited Loading

lambdageek commented May 12, 2023

pavelsavara commented May 12, 2023

ilonatommy commented May 12, 2023 • edited Loading

lambdageek commented May 12, 2023

lambdageek commented May 12, 2023

lambdageek commented May 15, 2023 • edited Loading

lambdageek commented May 15, 2023

lambdageek commented May 16, 2023

lambdageek commented May 16, 2023

lambdageek commented May 8, 2023 •

edited

Loading

lambdageek May 8, 2023 •

edited

Loading

lambdageek commented May 9, 2023 •

edited

Loading

lambdageek commented May 11, 2023 •

edited

Loading

ilonatommy commented May 12, 2023 •

edited

Loading

lambdageek commented May 15, 2023 •

edited

Loading