System.Reflection.Emit.AssemblyBuilder.Save

[jonorossi]

AssemblyBuilder.Save and AssemblyBuilderAccess.RunAndSave isn't available in .NET Core, however coreclr seems to have the code to implement it but I looks conditionally compiled out.

https://github.com/dotnet/coreclr/blob/bc146608854d1db9cdbcc0b08029a87754e12b49/src/mscorlib/src/System/Reflection/Emit/AssemblyBuilder.cs#L1690-L1711

Our use case in Castle DynamicProxy is to write out dynamically created assemblies to disk so we can run peverify over the assembly in unit tests. It also greatly helps writing out the assembly and opening it in ildasm to manually verify IL.

Is there any chance we can get this functionality that .NET Framework has introduced into .NET Core, or an alternative way of doing the same thing. This might not sound like a big deal if you aren't familiar with Castle DynamicProxy's internals, but it really is as it heavily reduces our confidence in both our code and the .NET runtime as well as reducing our ability to track down defects.

[joshfree]

Related issue https://github.com/dotnet/coreclr/issues/1709

[bartdesmet]

Another interesting omission is the lack of LambdaExpression.Compile accepting a MethodBuilder which tends to be very useful when combined with AssemblyBuilder.Save support.

[jkotas]

Large chunks of Reflection.Emit code in the runtime are Windows-specific. It is not easy to just enable them for CoreCLR because of it would not work on Unix. In particular, the support for emitting debug information depends on unmanaged PDB writer that is very complex Windows-specific component that we have no plans to open source and bring x-plat.

We are also not happy about Reflection.Emit being integrated deeply into the runtime. Reflection.Emit should be a independent higher level component that generates PE file that the runtime loads as any other assembly (note that runtime supports loading assemblies from byte array, so the PE file does not have to be persisted on disk). IKVM.Reflection.Emit done by Jeroen Frijters proved that it is pretty doable and can actually work much better than the current Reflection.Emit built into the runtime.

The plan to address the Reflection.Emit is:

1. (Done) Create fast standalone managed metadata, IL and portable PDB writer that can be used by Roslyn, and anybody else to produce managed assemblies. @tmat has been working on it: Metadata writer refactoring roslyn#7683.
2. Build Reflection.Emit object model as a layer on top of the standalone managed metadata writer to make it easier for existing projects that use Reflection.Emit to migrate to .NET Core.

[jonorossi]

@jkotas many thanks for the explanation here, it makes complete sense. I'm glad you guys are planning to provide a more general purpose object model around the heart of .NET, IL. I'd seen the portable PDB reader/writer and some metadata object model appear in Roslyn, they look great. In the past I've used Cecil for the exact reason the .NET Framework API is so heavily tied into the runtime, glad to see more stuff being pulled out of the runtime.

[tmat]

FYI @ManishJayaswal @jaredpar

[karelz]

Next step: We need API proposal with deep analysis how implementable it is. The old Desktop API might be an option, likely not a good one.
This is quite hard to do and requires lots of experience.

[KevinRansom]

@karelz @jkotas --- What is the status of this feature? When can we expect to see dynamic codegen with persistence on the coreclr?

[karelz]

There is currently no plan (milestone Future + my comment above) - area owners @atsushikan @DnlHarvey @joshfree can provide more details.

[stakx]

@atsushikan @DnlHarvey @joshfree, one question about the plan that @jkotas laid out above:

We are also not happy about Reflection.Emit being integrated deeply into the runtime. Reflection.Emit should be a independent higher level component that generates PE file that the runtime loads as any other assembly [...].

The plan to address the Reflection.Emit is:

1. [...]
2. Build Reflection.Emit object model as a layer on top of the standalone managed metadata writer to make it easier for existing projects that use Reflection.Emit to migrate to .NET Core.

I do not fully understand the plan, in particular with regard to dynamic assemblies (AssemblyBuilder.DefineDynamicAssembly) and their ability to be incrementally updated / augmented with new dynamic types. How would this still be possible if assemblies must first be fully written out as a byte stream or file, then be loaded as a whole?

I imagine one would have to write each dynamically generated type to a single assembly... but wouldn't this in turn interfere with internal type visibility and type member accessibility between generated types that should all be in the same assembly, and with stuff like [assembly: InternalsVisibleTo("DynamicallyGeneratedAssembly") (on which e. g. DynamicProxy currently depends to be able to proxy internal user types)?

Would it be possible to load more than one assembly having the exact same name (whether weak or strong), so that each such single-type assembly can have a [assembly: InternalsVisibleTo] attribute for its own name, such that types in the separate assemblies can reference each other's internals?

[jkotas]

their ability to be incrementally updated / augmented with new dynamic types

Yes, there would need to be a runtime hook for this.

Conceptually, each type addition in Reflection.Emit produces a new assembly that contains all existing metadata (with all existing tokens preserved) plus the new type. Then the runtime is asked to update the metadata of the already loaded assembly to the new version.

Sending the whole assembly over for each edit is not efficient for large assemblies with small edits. It would be better to just send the metadata delta with the new stuff over. Fortunately, this exists in the runtime for edit and continue already. Look for https://github.com/dotnet/coreclr/search?q=ApplyEditAndContinue.

So exposing the ApplyDelta metadata API for the standalone Reflection.Emit implementation should do the trick here.

[masonwheeler]

Large chunks of Reflection.Emit code in the runtime are Windows-specific. It is not easy to just enable them for CoreCLR because of it would not work on Unix. In particular, the support for emitting debug information depends on unmanaged PDB writer that is very complex Windows-specific component that we have no plans to open source and bring x-plat.

The existence of cross-platform Roslyn suggests that this isn't a particularly intractable problem, at a high level at least.

Could this please be made a high priority? The inability to use Reflection.Emit is currently my dotnet/corefx#1 blocker preventing me from migrating existing work to Core, and as such it's far more urgent, from my perspective at least, than the development of new features.

[zgramana]

Here's a good starting point for a non-Windows implementation (and it's rather mature): https://github.com/mono/mono/tree/master/mcs/class/corlib/System.Reflection.Emit

[stakx]

@atsushikan: Out of curiosity, is there any documentation / spec at all about the data format used for EnC metadata deltas, other than the actual CoreCLR source code?

[ghost]

Out of curiosity, is there any documentation / spec at all about the data format used for EnC metadata deltas, other than the actual CoreCLR source code?

Not that I've heard of but EnC is not an area where I have any knowledge.

[stakx]

@atsushikan - Thanks for the quick reply. 👍 I almost expected this to be a really arcane area, but I thought I'd give it a try and ask. I'll see what I can gather together on my own.

[karelz]

@stakx EnC MetaData format was never thoroughly documented (neither publicly nor internally). It was always defined just in the implementation.

[tmat]

@stakx Curious, what do you need it for?

[stakx]

@karelz - I almost suspected so... Thanks for this info! 👍

@tmat - Mostly out of curiousity.

I have spent a lot of time with System.Reflection in the past few months, and noticed its various limitations or bugs. There are sometimes situations where you'd rather just see plain, uninterpreted metadata instead of the semi-high-level view that Reflection provides—which is neither original metadata, nor as high-level as e.g. C#, but something of its own sitting somewhere in-between.

I have also discovered System.Reflection.Metadata (SRM). I have been thinking about how nice it would be if the (Core)CLR had the necessary runtime hooks and extension points so that both Reflection and Reflection.Emit could be rebuilt on top of SRM, outside of mscorlib / System.Private.CoreLib. (At the same time, this would also enable folks to build their own custom reflection / dynamic code emission libraries.) Two things appear to be missing as of today to make this possible:

• Neither the CLR nor the CoreCLR runtimes allow user code to get at the metadata stream & IL of loaded assemblies/modules, so it could be inspected using e.g. SRM. (This assumes that the metadata as per ECMA-335 is still available unaltered after a module has been loaded by the CLR, which likely isn't the case.)

• Neither the CLR nor the CoreCLR runtime allows altering/updating an assembly that's been loaded via Assembly.Load(byte[]) (which is what you can use today to load an assembly generated with SRM). Perhaps if EnC was exposed in some way, updating a loaded assembly would become possible.

I realise that having such runtime hooks closely allied with System.Reflection.Metadata is probably a pipe dream at this stage. But it might allow some super-interesting scenarios. So I'm trying to learn how this part of the runtime works to better gauge how big of a project this would be.

If there are any plans or efforts in this direction, I'd love to learn about it & be directed to the appropriate places. Otherwise... I don't intend to hijack this issue, which is just about AssemblyBuilder.Save. 😉

[jkotas]

Neither the CLR nor the CoreCLR runtimes allow user code to get at the metadata stream & IL of loaded assemblies/modules, so it could be inspected using e.g. SRM

Have you seen https://github.com/dotnet/coreclr/blob/85374ceaed177f71472cc4c23c69daf7402e5048/src/System.Private.CoreLib/src/System/Reflection/Metadata/AssemblyExtensions.cs#L25 ?

[stakx]

@jkotas - Yes, I have, and that's a really nice extension point! But it comes with some limitations: It's a Try... method and so may fail, e.g. for dynamically generated assemblies (AssemblyBuilder); something that today's Reflection (IIRC) can deal with just fine. So it seems Reflection couldn't be built on top of that to work as reliably as it does today. But I was definitely thinking of something like this. 👍

[vermorel]

We have finally rolled our own open-source version of AssemblyBuilder.Save based on System.Reflection.Metadata (and no further dependencies). It targets .NET Standard and works under .NET Core. It does support dynamically generated assemblies.

Check-out https://github.com/Lokad/ILPack Feedback will be most welcome.

[masonwheeler]

@vermorel Looks interesting. Does it generate PDBs too?

[vermorel]

@masonwheeler ILPack does not generate PDB.

[arlm]

We have finally rolled our own open-source version of AssemblyBuilder.Save based on System.Reflection.Metadata (and no further dependencies). It targets .NET Standard and works under .NET Core. It does support dynamically generated assemblies.

Check-out https://github.com/Lokad/ILPack Feedback will be most welcome.

I tried it on a Sigil branch of mine ( https://github.com/arlm/Sigil-vNext ) nad could not make it work because it does not install on .NET Standard, only on .NET Core solutions.

[ForNeVeR]

While this is, of course, some good news, the implemented surface is still very far from what we would ideally like to see.

In most use cases I've seen so far, people wanted to save assemblies with actual executable code / method bodies, and as far as I understand, this is yet not implemented.

[MichaelKetting]

@ForNeVeR I got the definitive answer here in #62956 (comment)

Summary: No, right now we should not expect support executable code / method bodies in .NET 8

[buyaa-n]

I had modified the milestone into .NET 9, even though seems it was not clear about what happens with the remaining work.
We will continue the effort in .NET 9.

[buyaa-n]

UPDATE: we are making a good progress:

• Added ILGenerator implementation
• Added support for remaining members
  • ConstructorBuilderImpl
  • PropertyBuilderImpl
  • EventBuilderImpl
• Added implementations for other unimplemented APIs in AssemblyBuilderImpl, TypeBuilderImp, MethodBuilderImpl
  • Still need to add a few APIs like CreateGlobalFuntions, DefineGlobalMethod etc. The work is in progress

Planning to finish the main functionality (excluding entry point and PDB) and make related APIs public in preview 1. Here is the tracking issue for details: #92975.

Now it would be great to test the implementation with a real-life scenario. I would really appreciate if you could:

• Prepare a simple app with your use case and share with me (in this thread or email me)
• Provide a sample assembly similar what your app generates, we can use for testing.
• Dogfood the daily build and try out the new AssemblyBuilder implementation yourself and give feedback. (The APIs are still internal so need to use reflection for now as mentioned in the sample)

Thank you in advance!

[kg]

Now it would be great to test the implementation with a real-life scenario. I would really appreciate if you could:

* Prepare a simple app with your use case and share with me (in this thread or email me)

* Provide a sample assembly similar what your app generates, we can use for testing.

I have a large application currently targeting net4x that uses AssemblyBuilder. Would having generated assemblies from it be useful at all for this? I could also provide the code for the implementation, but I don't have the spare time to do a test port to NET8/NET9 yet. It's a fairly complex script compiler with a small automated test suite, and then there are around a hundred scripts or so that are loaded by a host .NET application. It leans heavily on LINQ Expression.CompileToMethod to generate most of the methods (and does classic SRE for things that LINQ can't generate, like instance methods or pointers), so I don't know if that is relevant here and making that scenario work is considered in scope.

[buyaa-n]

I have a large application currently targeting net4x that uses AssemblyBuilder. Would having generated assemblies from it be useful at all for this?

Yes, the generated assembly with your app would be useful, thanks!

It leans heavily on LINQ Expression.CompileToMethod to generate most of the methods (and does classic SRE for things that LINQ can't generate, like instance methods or pointers), so I don't know if that is relevant here and making that scenario work is considered in scope.

LINQ Expression.CompileToMethod is not relevant here, though the assembly created with combination of LINQ Expression.CompileToMethod and SRE could be useful for testing. I have a testing tool that would try to regenerate the assembly with .NET Core implementation

[acaly]

Will XmlSerializer.GenerateSerializer be added back, or we will only have the Microsoft.XmlSerializer.Generator package?

[jkotas]

Will XmlSerializer.GenerateSerializer be added back, or we will only have the Microsoft.XmlSerializer.Generator package?

I do not expect that XmlSerializer.GenerateSerializer is going to be added back. See #1413 and related issues.

[acaly]

Will XmlSerializer.GenerateSerializer be added back, or we will only have the Microsoft.XmlSerializer.Generator package?

I do not expect that XmlSerializer.GenerateSerializer is going to be added back. See #1413 and related issues.

Thanks for the reply.

I know there is the new package, but it's different from the .NET Framework GenerateSerializer method, which allows me to generate a serializer assembly of a specified assembly, with specific types and mappings.

In one of the projects that I am working on, we have an obfuscator that extracts types that need to be serialized and generate the corresponding serializer assembly, which gets merged into the main assembly and obfuscated together. The current package based method is very hard to be used in this process. Our workaround is to use a .NET Framework app to generate the serializer assembly, which is rewritten to target .NET.

This workaround currently works fine, but obviously it's risky. I think a source-generator based method or a public generator method should be more helpful. I'm not sure whether there are undocumented methods or better solutions for this though.

This is probably not a good place to continue this topic. If you think it's necessary I can open a new issue.

[buyaa-n]

I have a large application currently targeting net4x that uses AssemblyBuilder. Would having generated assemblies from it be useful at all for this?

Yes, the generated assembly with your app would be useful, thanks!
I have a testing tool that would try to regenerate the assembly with .NET Core implementation

Thank you @kg for sending assemblies, I have regenerated the assembly with .NET Core implementation and found many issues that now fixed and will be ported with .NET 9 preview 2.

[buyaa-n]

Planning to finish the main functionality (excluding entry point and PDB) and make related APIs public in preview 1.

The feature is now available in https://dotnet.microsoft.com/en-us/download/dotnet/9.0, please give it a try and give us a feedback. We also adding bunch of fixes in preview 2 mostly related to complex generic types/methods

Here is related doc: https://learn.microsoft.com/dotnet/fundamentals/runtime-libraries/system-reflection-emit-assemblybuilder#persistable-dynamic-assemblies-in-net-core

[mjsabby]

Will Portable PDB support be there as well in this preview?

[buyaa-n]

Entry point and PDB not included in preview 1. Entry point will be added by preview 3. PDB support timeline not clear yet.

[jgh07]

There are a few methods still missing unrelated to PDB generation:

• AssemblyBuilder.DefineUnmanagedResource
• AssemblyBuilder.AddResourceFile

There might be alternatives that I'm unaware of. If not, will these methods also be supported in the future?

[masonwheeler]

PDB support timeline not clear yet.

I really hope it makes it in by release. At its core, this is a tool for compiler developers, and a compiler that can't produce debug builds is severely disadvantaged.

[Martin1994]

this is a tool for compiler developers, and a compiler that can't produce debug builds is severely disadvantaged.

Portable PDB is not blocked. System.Reflection.Metadata.Ecma335 is for this use case (not documented too well though IMHO). See https://github.com/Martin1994/ikvm/blob/18b1fce8265a3f128f4742b506f8e10e0cae0f51/src/IKVM.Reflection/Impl/PortablePdbWriter.cs

System.Reflection.Emit is more for dynamic emitting use cases, such as IronPython I believe.

[masonwheeler]

@Martin1994 That link looks kind of useful, but it looks like it only works if you already have an existing System.Reflection.Metadata emitter.

System.Reflection.Emit is more for dynamic emitting use cases, such as IronPython I believe.

IIRC IronPython style dynamic emitting was covered by the DLR, which was released with .NET Framework 4.0. System.Reflection.Emit, and the ability to for compiler writers to use it to save new assemblies, has been around since 1.0.

[buyaa-n]

FYI: We took completely different approach for adding EntryPoint support and setting other options, we are providing all metadata information produced with Reflection.Emit APIs (MetadataBuilder and ILStreams) so that user could embed them into the corresponding section of PEBuidler. This will let the customers handle their assembly building process by themselves using the PEHeaderBuilder and ManagedPEBuilder options. Because the MetadataBuilder and BlobBuilder types are not accessible from within CoreLib we decided to make the PersistedAssemblyBuilder type public and reintroduce the related new APIs there, remove the new APIs from the base AssemblyBuilder type.

namespace System.Reflection.Emit;
public abstract partial class AssemblyBuilder
{
    // Previously approved new APIs
-   public static AssemblyBuilder DefinePersistedAssembly(AssemblyName name, Assembly coreAssembly, IEnumerable<CustomAttributeBuilder>? assemblyAttributes = null);

-   public void Save(Stream stream);
-   public void Save(string assemblyFileName);
-   protected abstract void SaveCore(Stream stream);
}

+public sealed class PersistedAssemblyBuilder : AssemblyBuilder
{
+   public PersistedAssemblyBuilder(AssemblyName name, Assembly coreAssembly, IEnumerable<CustomAttributeBuilder>? assemblyAttributes = null);

+   public void Save(Stream stream);
+   public void Save(string assemblyFileName);
+   public MetadataBuilder GenerateMetadata(out BlobBuilder ilStream, out BlobBuilder mappedFieldData);
}

See more details and usage example from the API proposal. This change will be available in preview 3.

[buyaa-n]

Further the PDB support will be handled similar way as one would set EntryPoint in PEBuidler, here is the initial API proposal and usage example, please give your feedbacks.

With that I am closing this issue as alternate for the AssemblyBuilder.Save is implementation is done