Add generation of structured object file dumps #73913
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The compiler can currently emit an XML file with everything that was emitted into the output object file. This works and it's easily human readable. But the captured information is not very structured and cannot be used to generate aggregate information like 'this assembly contributed X bytes". This adds another dumper format similar to what we do for MIBC and others: it's ECMA-335 based and uses ldtoken/ldstr/ldc to encode the data. This can be used to build better tools. There's currently none but ILDASM, but it would be good to have it for 7.0. Good hackathon project.
ghost
assigned 
|
I've played around with this a bit, for the smallest possible Hello World (967kb), i could get following numbers out (which is sums up to only half of the executable size): Code: using Mono.Cecil;
using Mono.Cecil.Rocks;
namespace NativeAOTSizeAnalyzer;
internal class Program
{
static void Main(string[] args)
{
var asm = AssemblyDefinition.ReadAssembly(@"something.mstat");
var globalType = (TypeDefinition)asm.MainModule.LookupToken(0x02000001);
var types = globalType.Methods.First(x => x.Name == "Types");
var typeStats = GetTypes(types).ToList();
var typeSize = typeStats.Sum(x => x.Size);
var typesByModules = typeStats.GroupBy(x => x.Type.Scope).Select(x => new { x.Key.Name, Sum = x.Sum(x => x.Size) }).ToList();
Console.WriteLine($"// ********** Types Total Size {typeSize:n0}");
foreach (var m in typesByModules.OrderByDescending(x => x.Sum))
{
Console.WriteLine($"{m.Name,-40} {m.Sum,7:n0}");
}
Console.WriteLine($"// **********");
Console.WriteLine();
var methods = globalType.Methods.First(x => x.Name == "Methods");
var methodStats = GetMethods(methods).ToList();
var methodSize = methodStats.Sum(x => x.Size + x.GcInfoSize + x.EhInfoSize);
var methodsByModules = methodStats.GroupBy(x => x.Method.DeclaringType.Scope).Select(x => new { x.Key.Name, Sum = x.Sum(x => x.Size + x.GcInfoSize + x.EhInfoSize) }).ToList();
Console.WriteLine($"// ********** Methods Total Size {methodSize:n0}");
foreach (var m in methodsByModules.OrderByDescending(x => x.Sum))
{
Console.WriteLine($"{m.Name,-40} {m.Sum,7:n0}");
}
Console.WriteLine($"// **********");
Console.WriteLine();
string FindNamespace(TypeReference type)
{
var current = type;
while (true)
{
if (!String.IsNullOrEmpty(current.Namespace))
{
return current.Namespace;
}
if (current.DeclaringType == null)
{
return current.Name;
}
current = current.DeclaringType;
}
}
var methodsByNamespace = methodStats.Select(x => new TypeStats { Type = x.Method.DeclaringType, Size = x.Size + x.GcInfoSize + x.EhInfoSize }).Concat(typeStats).GroupBy(x => FindNamespace(x.Type)).Select(x => new { x.Key, Sum = x.Sum(x => x.Size) }).ToList();
Console.WriteLine($"// ********** Size By Namespace");
foreach (var m in methodsByNamespace.OrderByDescending(x => x.Sum))
{
Console.WriteLine($"{m.Key,-40} {m.Sum,7:n0}");
}
Console.WriteLine($"// **********");
Console.WriteLine();
var blobs = globalType.Methods.First(x => x.Name == "Blobs");
var blobStats = GetBlobs(blobs).ToList();
var blobSize = blobStats.Sum(x => x.Size);
Console.WriteLine($"// ********** Blobs Total Size {blobSize:n0}");
foreach (var m in blobStats.OrderByDescending(x => x.Size))
{
Console.WriteLine($"{m.Name,-40} {m.Size,7:n0}");
}
Console.WriteLine($"// **********");
}
public static IEnumerable<TypeStats> GetTypes(MethodDefinition types)
{
types.Body.SimplifyMacros();
var il = types.Body.Instructions;
for (int i = 0; i + 2 < il.Count; i += 2)
{
var type = (TypeReference)il[i + 0].Operand;
var size = (int)il[i + 1].Operand;
yield return new TypeStats
{
Type = type,
Size = size
};
}
}
public static IEnumerable<MethodStats> GetMethods(MethodDefinition methods)
{
methods.Body.SimplifyMacros();
var il = methods.Body.Instructions;
for (int i = 0; i + 4 < il.Count; i += 4)
{
var method = (MethodReference)il[i + 0].Operand;
var size = (int)il[i + 1].Operand;
var gcInfoSize = (int)il[i + 2].Operand;
var ehInfoSize = (int)il[i + 3].Operand;
yield return new MethodStats
{
Method = method,
Size = size,
GcInfoSize = gcInfoSize,
EhInfoSize = ehInfoSize
};
}
}
public static IEnumerable<BlobStats> GetBlobs(MethodDefinition blobs)
{
blobs.Body.SimplifyMacros();
var il = blobs.Body.Instructions;
for (int i = 0; i + 2 < il.Count; i += 2)
{
var name = (string)il[i + 0].Operand;
var size = (int)il[i + 1].Operand;
yield return new BlobStats
{
Name = name,
Size = size
};
}
}
}
public class TypeStats
{
public TypeReference Type { get; set; }
public int Size { get; set; }
}
public class MethodStats
{
public MethodReference Method { get; set; }
public int Size { get; set; }
public int GcInfoSize { get; set; }
public int EhInfoSize { get; set; }
}
public class BlobStats
{
public string Name { get; set; }
public int Size { get; set; }
}Edit (2023-01-23 @MichalStrehovsky): updated the parser to account for the file format change in #75328. |
|
Btw. dunno if you're aware, i've just discovered https://devblogs.microsoft.com/performance-diagnostics/sizebench-a-new-tool-for-analyzing-windows-binary-size/ which seems quite neat! Looks like 109KB of these 967KB are just bitonic sort / vxsort in the GC... |
Nice tool!!!
Yeah, this is more meaningful for bigger apps than the absolute minimal app. The absolute minimal app is dominated by the native parts of the binary (GC, vxsort tables, etc.) One thing .mstat doesn't capture is relocations - the NativeAOT data structures are pointer-heavy, so there's a lot of data in the
Oh nice, I didn't know they made it public!
Yeah, I disable all that in bflat. It's one of the reasons why it's smaller than stock NativeAOT. We build two flavors of the GC for NativeAOT - one that only has the workstation GC, and one that has server. I thought about disabling vxsort for the workstation one, but I don't have any data to base the decision on (besides size). We want to have comparable perf between CoreCLR-JIT and CoreCLR-NativeAOT. We could build yet another flavor of the GC, but we already have 3 (wks, server, server+guardcf) and we need to stop somewhere. |
A potential solution for this is to add an option for building GC from sources as part of application publish, similar to what's done in #72896 for ICU shims. It would allow you to build the GC with any custom settings you wish. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The compiler can currently emit an XML file with everything that was emitted into the output object file. This works and it's easily human readable. But the captured information is not very structured and cannot be used to generate aggregate information like 'this assembly contributed X bytes".
This adds another dumper format similar to what we do for MIBC and others: it's ECMA-335 based and uses ldtoken/ldstr/ldc to encode the data.
This can be used to build better tools. There's currently none but ILDASM, but it would be good to have it for 7.0. Good hackathon project.
Cc @dotnet/ilc-contrib