Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linker dependency framework #101277

Merged
merged 42 commits into from
May 6, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
afe8990
MarkStep runs inside dependency framework
vitek-karas Sep 13, 2023
4a92aaf
Merge branch 'main' of https://github.com/dotnet/runtime into LinkerD…
jtschuster Apr 19, 2024
76fe923
Put MarkType into DependencyFramework
jtschuster Apr 19, 2024
b3ccd66
Leave MarkType in MarkStep, use captured MarkStep to call it
jtschuster Apr 19, 2024
eb5e333
Format
jtschuster Apr 19, 2024
e1270bb
Use DF for TypeDefinition and MethodDefinition
jtschuster Apr 19, 2024
2d12ade
Rename MarkTypeImpl to ProcessType and clean up code
jtschuster Apr 20, 2024
1dc80db
Clean up code a bit more
jtschuster Apr 20, 2024
991f7a0
Use ProjectReference for DAF, and undo method moves
jtschuster Apr 22, 2024
069f47b
Merge branch 'main' of https://github.com/dotnet/runtime into LinkerD…
jtschuster Apr 22, 2024
c1670eb
Don't have additional virtualmethods processing
jtschuster Apr 22, 2024
8382e68
Merge branch 'main' of https://github.com/dotnet/runtime into LinkerD…
jtschuster Apr 22, 2024
5bf5ece
Remove old header comment copied from MarkStep
jtschuster Apr 22, 2024
e6edbce
Remove nullable directives from ilc files
jtschuster Apr 22, 2024
10a5c57
Sign DAF instead of nowarn on trimmer
jtschuster Apr 22, 2024
166d2d5
Re-add doc comment; Make types internal
jtschuster Apr 22, 2024
01f3a79
Undo Queue changes, fix visibility issues
jtschuster Apr 23, 2024
098f7bc
Rename TypeDefinitionDependencyNode file to match class
jtschuster Apr 23, 2024
25ec8ce
PR feedback:
jtschuster Apr 23, 2024
9ef1343
PR Feedback:
jtschuster Apr 25, 2024
d50d70c
Use static lambda for TypeDefinitionNode creation
jtschuster Apr 25, 2024
dfe51f2
Only warn on duplicate members within a single descriptors file
jtschuster Apr 25, 2024
afedfb7
Only warn on duplicate members within a single descriptors file
jtschuster Apr 25, 2024
a600348
Make duplicate preserve an info message
jtschuster Apr 26, 2024
8d642a5
Merge branch 'main' of https://github.com/dotnet/runtime into LinkerD…
jtschuster Apr 26, 2024
a0ea640
Merge branch 'main' of https://github.com/dotnet/runtime into ILLinkD…
jtschuster Apr 26, 2024
2bb99d7
Merge branch 'ILLinkDescriptionDuplicatePreserve' into LinkerDependen…
jtschuster Apr 26, 2024
b27bc04
Update expectations in xml warnings tests
jtschuster Apr 26, 2024
3394758
Merge branch 'ILLinkDescriptionDuplicatePreserve' into LinkerDependen…
jtschuster Apr 26, 2024
1abe4b3
Make method internal for API compat
jtschuster Apr 27, 2024
8cbedd1
Merge branch 'ILLinkDescriptionDuplicatePreserve' into LinkerDependen…
jtschuster Apr 27, 2024
df7a522
Merge branch 'main' into LinkerDependencyFramework
jtschuster Apr 30, 2024
28bf0be
Merge branch 'main' into LinkerDependencyFramework
jtschuster May 1, 2024
3ca9f9a
PR Feedback:
jtschuster May 2, 2024
830f23e
Merge branch 'LinkerDependencyFramework' of https://github.com/jtschu…
jtschuster May 2, 2024
3001124
Add space before method call parentheses
jtschuster May 2, 2024
f003f74
Don't pass TargetOS and TargetArch to ILLink.Tasks when using live il…
jtschuster May 3, 2024
a0f6861
Merge branch 'main' into LinkerDependencyFramework
jtschuster May 3, 2024
d358b42
Update comment in liveILLink.targets
jtschuster May 3, 2024
0fab4e3
Merge branch 'LinkerDependencyFramework' of https://github.com/jtschu…
jtschuster May 3, 2024
5e124b5
Don't append platform to output dirs
jtschuster May 3, 2024
855b109
Merge branch 'main' into LinkerDependencyFramework
jtschuster May 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/coreclr/tools/Common/Sorting/ArrayAccessor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System;

#nullable disable

namespace ILCompiler.Sorting.Implementation
{
internal struct ArrayAccessor<T> : ISortableDataStructureAccessor<T, T[]>
Expand Down
2 changes: 2 additions & 0 deletions src/coreclr/tools/Common/Sorting/ICompareAsEqualAction.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System.Diagnostics;

#nullable disable

namespace ILCompiler
{
internal interface ICompareAsEqualAction
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

#nullable disable

namespace ILCompiler
{
internal interface ISortableDataStructureAccessor<T, TDataStructure>
Expand Down
2 changes: 2 additions & 0 deletions src/coreclr/tools/Common/Sorting/ListAccessor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System.Collections.Generic;

#nullable disable

namespace ILCompiler.Sorting.Implementation
{
internal struct ListAccessor<T> : ISortableDataStructureAccessor<T, List<T>>
Expand Down
2 changes: 2 additions & 0 deletions src/coreclr/tools/Common/Sorting/MergeSort.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
using System.Collections.Generic;
using ILCompiler.Sorting.Implementation;

#nullable disable

namespace ILCompiler
{
public static class MergeSortApi
Expand Down
2 changes: 2 additions & 0 deletions src/coreclr/tools/Common/Sorting/MergeSortCore.cs
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
using System.Collections.Generic;
using System.Threading.Tasks;

#nullable disable

namespace ILCompiler.Sorting.Implementation
{
internal static class MergeSortCore<T, TDataStructure, TDataStructureAccessor, TComparer, TCompareAsEqualAction>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
using System.Collections.Generic;
using System.Diagnostics;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public abstract class ComputedStaticDependencyNode<DependencyContextType> : DependencyNodeCore<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
using System.Collections.Immutable;
using System.Diagnostics;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
using System.Collections.Generic;
using System.Collections.Immutable;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System.Diagnostics;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public abstract class DependencyNode : IDependencyNode
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
using System;
using System.Collections.Generic;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public abstract class DependencyNodeCore<DependencyContextType> : DependencyNode, IDependencyNode<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
using System.IO;
using System.Diagnostics;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public class DgmlWriter
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
using System.Diagnostics.Tracing;
using System.Runtime.InteropServices;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
[EventSource(Name = "Microsoft-ILCompiler-DependencyGraph")]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
using System.Collections.Generic;
using System.Diagnostics;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public struct FirstMarkLogStrategy<DependencyContextType> : IDependencyAnalysisMarkStrategy<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
using System.Collections.Generic;
using System.Diagnostics;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public struct FullGraphLogStrategy<DependencyContextType> : IDependencyAnalysisMarkStrategy<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System.Collections.Generic;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public interface IDependencyAnalysisMarkStrategy<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public interface IDependencyAnalyzerLogEdgeVisitor<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public interface IDependencyAnalyzerLogNodeVisitor<DependencyContextType>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System.Collections.Generic;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
public interface IDependencyNode
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

using System.Collections.Generic;

#nullable disable

namespace ILCompiler.DependencyAnalysisFramework
{
/// <summary>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
using System;
using System.Diagnostics.Tracing;

#nullable disable

/// <summary>
/// Performance events specific to the dependency graph.
/// </summary>
Expand Down
106 changes: 98 additions & 8 deletions src/tools/illink/src/linker/Linker.Steps/MarkStep.cs
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
using System.Reflection.Metadata.Ecma335;
using System.Reflection.Runtime.TypeParsing;
using System.Text.RegularExpressions;
using ILCompiler.DependencyAnalysisFramework;
using ILLink.Shared;
using ILLink.Shared.TrimAnalysis;
using ILLink.Shared.TypeSystemProxy;
Expand Down Expand Up @@ -229,6 +230,7 @@ public MarkStep ()
_pending_isinst_instr = new List<(TypeDefinition, MethodBody, Instruction)> ();
_entireTypesMarked = new HashSet<TypeDefinition> ();
_compilerGeneratedMethodRequiresScanner = new Dictionary<MethodBody, bool> ();
_analyzer = new DependencyAnalyzer<NoLogStrategy<MarkStepNodeFactory>, MarkStepNodeFactory> (new MarkStepNodeFactory (this), null);
}

public AnnotationStore Annotations => Context.Annotations;
Expand Down Expand Up @@ -372,15 +374,103 @@ internal void MarkEntireType (TypeDefinition type, in DependencyInfo reason)
}
}

DependencyAnalyzer<NoLogStrategy<MarkStepNodeFactory>, MarkStepNodeFactory> _analyzer;
void Process ()
{
while (ProcessPrimaryQueue () ||
_analyzer.ComputeDependencyRoutine += (List<DependencyNodeCore<MarkStepNodeFactory>> nodes) => {
foreach (DependencyNodeCore<MarkStepNodeFactory> node in nodes) {
if (node is ProcessCallbackDependencyNode processNode)
processNode.Process ();
}
};
_analyzer.AddRoot (new ProcessCallbackDependencyNode (ProcessAllPendingItems), "start");
_analyzer.ComputeMarkedNodes ();

ProcessPendingTypeChecks ();

bool ProcessAllPendingItems ()
=> ProcessPrimaryQueue () ||
ProcessMarkedPending () ||
ProcessLazyAttributes () ||
ProcessLateMarkedAttributes () ||
MarkFullyPreservedAssemblies ()) ;
MarkFullyPreservedAssemblies ();
}

ProcessPendingTypeChecks ();
public class MarkStepNodeFactory (MarkStep markStep)
{
public MarkStep MarkStep { get; } = markStep;
}

sealed class ProcessCallbackDependencyNode : DependencyNodeCore<MarkStepNodeFactory>
{
Func<bool> _processAction;
DependencyList? _dependencies;

public ProcessCallbackDependencyNode (Func<bool> action) => _processAction = action;

public void Process ()
{
_dependencies = new DependencyList ();
if (_processAction ()) {
_dependencies.Add (new ProcessCallbackDependencyNode (_processAction), "Some processing was done, continuation required");
}
}

public override bool InterestingForDynamicDependencyAnalysis => false;

public override bool HasDynamicDependencies => false;

public override bool HasConditionalStaticDependencies => false;

public override bool StaticDependenciesAreComputed => _dependencies != null;

public override IEnumerable<DependencyListEntry>? GetStaticDependencies (MarkStepNodeFactory context) => _dependencies;

public override IEnumerable<CombinedDependencyListEntry>? GetConditionalStaticDependencies (MarkStepNodeFactory context) => null;
public override IEnumerable<CombinedDependencyListEntry>? SearchDynamicDependencies (List<DependencyNodeCore<MarkStepNodeFactory>> markedNodes, int firstNode, MarkStepNodeFactory context) => null;
protected override string GetName (MarkStepNodeFactory context) => "Process";
}

public class TypeDependencyNode : DependencyNodeCore<MarkStepNodeFactory>
jtschuster marked this conversation as resolved.
Show resolved Hide resolved
{
readonly TypeReference reference;
readonly DependencyInfo reason;
readonly MessageOrigin? origin;
readonly MarkStep markStep;

public TypeDependencyNode (TypeReference reference, DependencyInfo reason, MessageOrigin? origin, MarkStep markStep)
{
this.reference = reference;
this.reason = reason;
this.origin = origin;
this.markStep = markStep;
}
public override bool InterestingForDynamicDependencyAnalysis => false;

public override bool HasDynamicDependencies => false;

public override bool HasConditionalStaticDependencies => false;

public override bool StaticDependenciesAreComputed => true;

public override IEnumerable<DependencyListEntry>? GetStaticDependencies (MarkStepNodeFactory context)
{
// Add other types that are marked in MarkType
yield break;
}

public override IEnumerable<CombinedDependencyListEntry>? GetConditionalStaticDependencies (MarkStepNodeFactory context) => null;
public override IEnumerable<CombinedDependencyListEntry>? SearchDynamicDependencies (List<DependencyNodeCore<MarkStepNodeFactory>> markedNodes, int firstNode, MarkStepNodeFactory context) => null;
protected override string GetName (MarkStepNodeFactory context) => "TypeNode";
protected override void OnMarked (MarkStepNodeFactory context)
{
markStep.MarkTypeImpl (reference, reason, origin);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know enough of the DF to tell if this is the right place to do the heavy processing. But I guess for now it is.
I assume eventually all of this would move to GetDependencies and we would remove OnMarked, right?

Future thought: Try to figure out how the DF will play with Annotations and the notion of "marked item" which is used outside of the mark step in the trimmer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think ideally most of this moves to GetDependencies and this is minimal or removed.

My thought is that eventually the Annotations markings will forward to the analyzer markings, but that could get complicated if marking (not just IsMarked checks) happen outside of MarkStep. In the meantime, I think OnMarked will call Annotations.MarkX.

}

protected internal virtual void MarkType (TypeReference reference, DependencyInfo reason, MessageOrigin? origin = null)
{
_analyzer.AddRoot (new TypeDependencyNode (reference, reason, origin, this), "MarkedType");
}

static bool IsFullyPreservedAction (AssemblyAction action) => action == AssemblyAction.Copy || action == AssemblyAction.Save;
Expand Down Expand Up @@ -666,7 +756,7 @@ void ProcessVirtualMethod (MethodDefinition method)
MarkMethod (dimInfo.Override, new DependencyInfo (DependencyKind.Override, dimInfo.Base), ScopeStack.CurrentScope.Origin);
}
}
List<OverrideInformation>? overridingMethods = (List<OverrideInformation>?)Annotations.GetOverrides (method);
List<OverrideInformation>? overridingMethods = (List<OverrideInformation>?) Annotations.GetOverrides (method);
if (overridingMethods is not null) {
for (int i = 0; i < overridingMethods.Count; i++) {
OverrideInformation ov = overridingMethods[i];
Expand Down Expand Up @@ -1896,7 +1986,7 @@ protected virtual void MarkSerializable (TypeDefinition type)
MarkMethodsIf (type.Methods, HasOnSerializeOrDeserializeAttribute, new DependencyInfo (DependencyKind.SerializationMethodForType, type), ScopeStack.CurrentScope.Origin);
}

protected internal virtual TypeDefinition? MarkTypeVisibleToReflection (TypeReference type, TypeDefinition definition, in DependencyInfo reason, in MessageOrigin origin)
protected internal virtual void MarkTypeVisibleToReflection (TypeReference type, TypeDefinition definition, in DependencyInfo reason, in MessageOrigin origin)
{
// If a type is visible to reflection, we need to stop doing optimization that could cause observable difference
// in reflection APIs. This includes APIs like MakeGenericType (where variant castability of the produced type
Expand All @@ -1907,7 +1997,7 @@ protected virtual void MarkSerializable (TypeDefinition type)

MarkImplicitlyUsedFields (definition);

return MarkType (type, reason, origin);
MarkType (type, reason, origin);
}

internal void MarkMethodVisibleToReflection (MethodReference method, in DependencyInfo reason, in MessageOrigin origin)
Expand Down Expand Up @@ -1958,7 +2048,7 @@ internal void MarkStaticConstructorVisibleToReflection (TypeDefinition type, in
/// <param name="reference">The type reference to mark.</param>
/// <param name="reason">The reason why the marking is occuring</param>
/// <returns>The resolved type definition if the reference can be resolved</returns>
protected internal virtual TypeDefinition? MarkType (TypeReference reference, DependencyInfo reason, MessageOrigin? origin = null)
protected internal virtual TypeDefinition? MarkTypeImpl (TypeReference reference, DependencyInfo reason, MessageOrigin? origin = null)
{
#if DEBUG
if (!_typeReasons.Contains (reason.Kind))
Expand Down Expand Up @@ -2759,7 +2849,7 @@ void MarkGenericArguments (IGenericInstance instance)
var argument = arguments[i];
var parameter = parameters[i];

TypeDefinition? argumentTypeDef = MarkType (argument, new DependencyInfo (DependencyKind.GenericArgumentType, instance));
var argumentTypeDef = MarkTypeImpl (argument, new DependencyInfo (DependencyKind.GenericArgumentType, instance));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the place where it gets interesting for TypeRef versus TypeDef :-)


if (Annotations.FlowAnnotations.RequiresGenericArgumentDataFlowAnalysis (parameter)) {
// The only two implementations of IGenericInstance both derive from MemberReference
Expand Down
Loading
Loading