Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add step to run linker on entire runtime pack during libraries build #40172

Merged
merged 8 commits into from
Aug 4, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 30 additions & 21 deletions eng/illink.targets
Original file line number Diff line number Diff line change
Expand Up @@ -184,50 +184,59 @@
</PropertyGroup>
</Target>

<!-- ILLink.Tasks arguments common to runs for both individual libraries and for the entire runtime pack -->
<Target Name="SetCommonILLinkArgs">
<PropertyGroup>
<!-- keep type-forward assemblies (facades) -->
<ILLinkArgs>$(ILLinkArgs) -t</ILLinkArgs>
<!-- don't remove the embedded link attributes xml resource since ILLink may run again on the assembly -->
<!-- and ignore the link attributes xml file during the library build, since we need the attributes preserved until the final app is linked -->
<ILLinkArgs>$(ILLinkArgs) --strip-link-attributes false --ignore-link-attributes true</ILLinkArgs>
<!-- ignore unresolved references -->
<ILLinkArgs>$(ILLinkArgs) --skip-unresolved true</ILLinkArgs>
<!-- keep interface implementations -->
<ILLinkArgs>$(ILLinkArgs) --disable-opt unusedinterfaces</ILLinkArgs>
</PropertyGroup>
</Target>

<!-- ILLinkTrimAssembly
Examines the "input assembly" for IL that is unreachable from public API and trims that,
rewriting the assembly to an "output assembly"
-->
<UsingTask TaskName="ILLink" AssemblyFile="$(ILLinkTasksPath)" />
<Target Name="ILLinkTrimAssembly" Condition="'$(ILLinkTrimAssembly)' == 'true'">
<Target Name="ILLinkTrimAssembly"
Condition="'$(ILLinkTrimAssembly)' == 'true'"
DependsOnTargets="SetCommonILLinkArgs">
<PropertyGroup>
<ILLinkArgs>$(ILLinkArgs)-r $(TargetName)</ILLinkArgs>
<ILLinkArgs>$(ILLinkArgs) -r $(TargetName)</ILLinkArgs>
<!-- default action for core assemblies -->
<ILLinkArgs>$(ILLinkArgs) -c skip</ILLinkArgs>
<!-- default action for non-core assemblies -->
<ILLinkArgs>$(ILLinkArgs) -u skip</ILLinkArgs>
<!-- trim the target assembly -->
<ILLinkArgs>$(ILLinkArgs) -p link $(TargetName)</ILLinkArgs>
<!-- keep type-forward assemblies (facades) -->
<ILLinkArgs>$(ILLinkArgs) -t</ILLinkArgs>
<ILLinkArgs Condition="'$(ILLinkRewritePDBs)' == 'true' and Exists('$(ILLinkTrimAssemblySymbols)')">$(ILLinkArgs) -b true</ILLinkArgs>
<!-- don't remove the embedded root xml resource since ILLink may run again on the assembly -->
<ILLinkArgs Condition="'$(ILLinkTrimXml)' != ''">$(ILLinkArgs) --strip-descriptors false</ILLinkArgs>
<!-- pass the non-embedded root xml file on the command line -->
<ILLinkArgs Condition="'$(ILLinkTrimXmlLibraryBuild)' != ''">$(ILLinkArgs) -x "$(ILLinkTrimXmlLibraryBuild)"</ILLinkArgs>
<!-- don't remove the embedded substitutions xml resource since ILLink may run again on the assembly -->
<ILLinkArgs Condition="'$(ILLinkSubstitutionsXml)' != ''">$(ILLinkArgs) --strip-substitutions false</ILLinkArgs>
<!-- don't remove the embedded link attributes xml resource since ILLink may run again on the assembly -->
<!-- and ignore the link attributes xml file during the library build, since we need the attributes preserved until the final app is linked -->
<ILLinkArgs>$(ILLinkArgs) --strip-link-attributes false --ignore-link-attributes true</ILLinkArgs>
<!-- ignore unresolved references -->
<ILLinkArgs>$(ILLinkArgs) --skip-unresolved true</ILLinkArgs>
<!-- keep interface implementations -->
<ILLinkArgs>$(ILLinkArgs) --disable-opt unusedinterfaces</ILLinkArgs>
<!-- keep DynamicDependencyAttribute unless a project explicitly disables it -->
<ILLinkArgs Condition="'$(ILLinkKeepDepAttributes)' != 'false'">$(ILLinkArgs) --keep-dep-attributes true</ILLinkArgs>
<!-- suppress warnings with the following codes:
IL2006: The generic parameter 'T' from A with dynamically accessed member kinds B is passed into the generic parameter
'T' from 'System.Lazy<T>' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'
IL2008: Could not find type A specified in resource B
IL2009: Could not find method A in type B specified in resource C
IL2012: Could not find field A in type B specified in resource C
IL2025: Duplicate preserve of A in B
IL2026: Calling A which has B can break functionality when trimming application code. The target method might be removed.
IL2035: Unresolved assembly A in DynamicDependencyAttribute on B
IL2041: The DynamicallyAccessedMembersAttribute is only allowed on method parameters, return value or generic parameters.
IL2006: The generic parameter 'T' from A with dynamically accessed member kinds B is passed into the generic parameter
'T' from 'System.Lazy<T>' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'
IL2008: Could not find type A specified in resource B
IL2009: Could not find method A in type B specified in resource C
IL2012: Could not find field A in type B specified in resource C
IL2025: Duplicate preserve of A in B
IL2026: Calling A which has B can break functionality when trimming application code. The target method might be removed.
IL2035: Unresolved assembly A in DynamicDependencyAttribute on B
IL2050: P/invoke method A declares a parameter with COM marshalling. Correctness of COM interop
cannot be guaranteed after trimming. Interfaces and interface members might be removed.
-->
<ILLinkArgs>$(ILLinkArgs) --nowarn IL2006;IL2008;IL2009;IL2012;IL2025;IL2026;IL2035;IL2041</ILLinkArgs>
<ILLinkArgs>$(ILLinkArgs) --nowarn IL2006;IL2008;IL2009;IL2012;IL2025;IL2026;IL2035;IL2050</ILLinkArgs>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the change from IL2041 to IL2050? Nothing in this change should have changed what warnings are produced for the individual library build. So I think both 2041 and 2050 can be removed here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started getting 2050 after syncing with master at the beginning of this PR. Some new code since the initial suppressions check in must have been added which causes these errors:

ILLink : error IL2050: System.Runtime.InteropServices.Marshal.CreateBindCtx(UInt32,IBindCtx&): P/invoke method 'System.Runtime.InteropServices.Marshal.CreateBindCtx(UInt32,IBindCtx&)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
ILLink : error IL2050: System.Runtime.InteropServices.Marshal.CreateBindCtx(UInt32,IBindCtx&): P/invoke method 'System.Runtime.InteropServices.Marshal.CreateBindCtx(UInt32,IBindCtx&)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
ILLink : error IL2050: System.Runtime.InteropServices.Marshal.MkParseDisplayName(IBindCtx,String,UInt32&,IMoniker&): P/invoke method 'System.Runtime.InteropServices.Marshal.MkParseDisplayName(IBindCtx,String,UInt32&,IMoniker&)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
ILLink : error IL2050: System.Runtime.InteropServices.Marshal.MkParseDisplayName(IBindCtx,String,UInt32&,IMoniker&): P/invoke method 'System.Runtime.InteropServices.Marshal.MkParseDisplayName(IBindCtx,String,UInt32&,IMoniker&)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
ILLink : error IL2050: System.Runtime.InteropServices.Marshal.BindMoniker(IMoniker,UInt32,Guid&,Object&): P/invoke method 'System.Runtime.InteropServices.Marshal.BindMoniker(IMoniker,UInt32,Guid&,Object&)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
ILLink : error IL2050: System.Runtime.InteropServices.Marshal.BindMoniker(IMoniker,UInt32,Guid&,Object&): P/invoke method 'System.Runtime.InteropServices.Marshal.BindMoniker(IMoniker,UInt32,Guid&,Object&)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\coreclr\src\System.Private.CoreLib\System.Private.CoreLib.csproj]
ILLink : error IL2050: Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VarNumFromParseNum(Byte[],Byte[],Int32): P/invoke method 'Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VarNumFromParseNum(Byte[],Byte[],Int32)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\libraries\Microsoft.VisualBasic.Core\src\Microsoft.VisualBasic.Core.vbproj]
ILLink : error IL2050: Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VarNumFromParseNum(Byte[],Byte[],Int32): P/invoke method 'Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VarNumFromParseNum(Byte[],Byte[],Int32)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\libraries\Microsoft.VisualBasic.Core\src\Microsoft.VisualBasic.Core.vbproj]
ILLink : error IL2050: Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VariantChangeType(Object&,Object&,Int16,Int16): P/invoke method 'Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VariantChangeType(Object&,Object&,Int16,Int16)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\libraries\Microsoft.VisualBasic.Core\src\Microsoft.VisualBasic.Core.vbproj]
ILLink : error IL2050: Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VariantChangeType(Object&,Object&,Int16,Int16): P/invoke method 'Microsoft.VisualBasic.CompilerServices.UnsafeNativeMethods.VariantChangeType(Object&,Object&,Int16,Int16)' declares a parameter with COM marshalling. Correctness of COM interop cannot be guaranteed after trimming. Interfaces and interface members might be removed. [D:\repos\dotnet_runtime_2\src\libraries\Microsoft.VisualBasic.Core\src\Microsoft.VisualBasic.Core.vbproj]
    0 Warning(s)
    10 Error(s)

2041 doesn't show up anymore following the sync hence removed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like that came in last week with dotnet/linker#1382.

cc @MichalStrehovsky

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we now detect COM usage and mark it as unsafe. Apps that end up referencing these methods after trimming will always warn.

</PropertyGroup>

<MakeDir Directories="$(ILLinkTrimInputPath)" />
Expand Down
1 change: 1 addition & 0 deletions src/libraries/Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -328,4 +328,5 @@
<CLSCompliant Condition="'$(CLSCompliant)' == '' and ('$(IsTestProject)' == 'true' or '$(IsTestSupportProject)' == 'true')">false</CLSCompliant>
<CLSCompliant Condition="'$(CLSCompliant)' == ''">true</CLSCompliant>
</PropertyGroup>

</Project>
76 changes: 76 additions & 0 deletions src/libraries/illink-sharedframework.targets
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<Project>

<Target Name="ILLinkTrimSharedFramework"
AfterTargets="Build"
DependsOnTargets="SetCommonILLinkArgs">

<PropertyGroup>
<LibrariesTrimmedArtifactsPath>$([MSBuild]::NormalizePath('$(ArtifactsBinDir)', 'ILLinkTrimAssembly', '$(BuildSettings)', 'trimmed-runtimepack'))</LibrariesTrimmedArtifactsPath>
</PropertyGroup>

<PropertyGroup>
<!-- default action for core assemblies -->
<ILLinkArgs>$(ILLinkArgs) -c link</ILLinkArgs>
<!-- update debug symbols -->
<ILLinkArgs>$(ILLinkArgs) -b true</ILLinkArgs>
<!-- suppress warnings with the following codes:
IL2006: The generic parameter 'T' from A with dynamically accessed member kinds B is passed into the generic parameter
'T' from 'System.Lazy<T>' which requires dynamically accessed member kinds 'PublicParameterlessConstructor'
IL2009: Could not find method A in type B specified in resource C
IL2025: Duplicate preserve of A in B
IL2026: Calling A which has B can break functionality when trimming application code. The target method might be removed.
IL2035: Unresolved assembly A in DynamicDependencyAttribute on B
IL2050: P/invoke method A declares a parameter with COM marshalling. Correctness of COM interop
cannot be guaranteed after trimming. Interfaces and interface members might be removed.
-->
<ILLinkArgs>$(ILLinkArgs) --nowarn IL2006;IL2009;IL2025;IL2026;IL2035;IL2050</ILLinkArgs>
</PropertyGroup>

<!-- Retrieve CoreLib's targetpath via GetTargetPath as it isn't binplaced yet. -->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand this. Why can't we use $(MicrosoftNetCoreAppRuntimePackNativeDir) instead?

Copy link
Contributor Author

@layomia layomia Aug 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(MicrosoftNetCoreAppRuntimePackNativeDir) does not have S.P.CoreLib present when the new target is running. @safern's thought on it was that the directory gets populated in pretest.proj as a preparation for being able to run tests.

fwiw, wasm.targets takes a similar approach -

<!-- Retrieve CoreLib's targetpath via GetTargetPath as it isn't binplaced yet. -->
<MSBuild Projects="$(CoreLibProject)"
Targets="GetTargetPath">
<Output TaskParameter="TargetOutputs" ItemName="WasmPInvokeAssembly" />
</MSBuild>
.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My naive thinking is that there should be somewhere in the build (that doesn't rely on tests) that we can hook in where the entire runtimepack is built. But I'm not seeing where that place is. Maybe @safern @ViktorHofer @joperezr or @ericstj might know.

But for now, what you have works so let's go with that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SPC and the runtime gets binplaced into libraries runtime pack and testhost as part of runtime.depproj.

Runtime.depproj is built as part of pretest.proj which is the last step of the libs subset. Which happens after libs.src that is the subset that builds src.proj. That is why wasm.targets also follows that approach.

FWIW we could introduce a common target to get SPC target path instead of duplicating the logic to call MSBuild on it with GetTargetPath target.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW we could introduce a common target to get SPC target path instead of duplicating the logic to call MSBuild on it with GetTargetPath target.

It feels to me that the correct fix here would be to fully populate the runtimepack earlier, and not make it dependent on testing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels to me that the correct fix here would be to fully populate the runtimepack earlier, and not make it dependent on testing.

I agree with @eerhardt, we have talked about this couple times already. I believe we should binplace the runtime artifacts incrementally after they are built (CoreLib, coreclr.dll, etc). For CI we just need a target that re-binplaces the runtime artifacts (to preserve the existing behavior which allows to build coreclr and libraries in parallel).

The reason why having the runtime binplace at pretest is handy, is for inner loop development. For example if I update code in SPC after a vertical build, in order to run tests against the new SPC, I don't have to do a full libraries vertical, I can just do: build libs.pretest -rc

I don't understand why binplacing the runtime artifacts in pretest is handy for inner loop development instead of binplacing the runtime assets at the time they are (re-)built, which doesn't require a libraries rebuild.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why binplacing the runtime artifacts in pretest is handy for inner loop development instead of binplacing the runtime assets at the time they are (re-)built, which doesn't require a libraries rebuild.

We've also talked about this a couple of times. If you build for example System.Runtime which has a P2P to SPC, and don't specify RuntimeConfiguration as an argument you would potentially be binplacing a Debug build of SPC when probably you built the runtime for release, this could cause a mismatch in between the native bits and the managed bits of coreclr which is not recommended:
#38885 (comment)

So we should be careful on how we do that.

Copy link
Member

@safern safern Aug 4, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, I agree we should change this if possible, I'm just saying we should be really careful how and where we put this binplacing to happen.

Also, I wouldn't like removing the build.pretest binplacing the runtime as it is handy for CI as well as I explained above... if we remove that, then provide another hook to binplace the runtime as an individual step.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've also talked about this a couple of times. If you build for example System.Runtime which has a P2P to SPC, and don't specify RuntimeConfiguration as an argument you would potentially be binplacing a Debug build of SPC when probably you built the runtime for release, this could cause a mismatch in between the native bits and the managed bits of coreclr which is not recommended:
#38885 (comment)
Btw, I agree we should change this if possible, I'm just saying we should be really careful how and where we put this binplacing to happen.

I think we all agree that we should change when we binplace the runtime artifacts and I agree that it should be done with the highest diligence. We should do this as part of #38034.

Also, I wouldn't like removing the build.pretest binplacing the runtime as it is handy for CI as well as I explained above... if we remove that, then provide another hook to binplace the runtime as an individual step.

That's what I meant with:

For CI we just need a target that re-binplaces the runtime artifacts (to preserve the existing behavior which allows to build coreclr and libraries in parallel).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I meant with:

I somehow missed that part 🤦

We should do this as part of #38034.

Sounds good.

<MSBuild Projects="$(CoreLibProject)"
Targets="GetTargetPath">
<Output TaskParameter="TargetOutputs" PropertyName="SystemPrivateCoreLibPath" />
</MSBuild>

<PropertyGroup>
<_AssemblyPaths>$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir);$(SystemPrivateCoreLibPath)</_AssemblyPaths>
</PropertyGroup>

<ItemGroup>
<!-- add references from the libraries directory -->
<_DependencyDirectories Include="$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir.TrimEnd('\'))" />
</ItemGroup>

<PropertyGroup>
<ILLinkArgs>$(ILLinkArgs) -d @(_DependencyDirectories->'"%(Identity)"', ' -d ')</ILLinkArgs>
</PropertyGroup>

<ItemGroup>
<_AssembliesToLink Include="System.Private.CoreLib" />

<_LibrariesToLink Include="$(MicrosoftNetCoreAppRuntimePackRidLibTfmDir)*.dll" />
<_AssembliesToLink Include="@(_LibrariesToLink->'%(FileName)')" />
</ItemGroup>

<PropertyGroup>
<ILLinkArgs>$(ILLinkArgs) -r @(_AssembliesToLink->'%(Identity)', ' -r ')</ILLinkArgs>
</PropertyGroup>

<!-- When running from Desktop MSBuild, DOTNET_HOST_PATH is not set.
In this case, explicitly specify the path to the dotnet host. -->
<PropertyGroup Condition=" '$(DOTNET_HOST_PATH)' == '' ">
<_DotNetHostDirectory>$(RepoRoot).dotnet</_DotNetHostDirectory>
layomia marked this conversation as resolved.
Show resolved Hide resolved
<_DotNetHostFileName>dotnet</_DotNetHostFileName>
<_DotNetHostFileName Condition=" '$(OS)' == 'Windows_NT' ">dotnet.exe</_DotNetHostFileName>
layomia marked this conversation as resolved.
Show resolved Hide resolved
</PropertyGroup>

<ILLink AssemblyPaths="$(_AssemblyPaths)"
RootAssemblyNames=""
OutputDirectory="$(LibrariesTrimmedArtifactsPath)"
ExtraArgs="$(ILLinkArgs)"
ToolExe="$(_DotNetHostFileName)"
ToolPath="$(_DotNetHostDirectory)" />
</Target>

<Import Project="$([MSBuild]::NormalizePath('$(RepositoryEngineeringDir)', 'illink.targets'))" />
layomia marked this conversation as resolved.
Show resolved Hide resolved
</Project>
3 changes: 3 additions & 0 deletions src/libraries/src.proj
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,9 @@
Properties="$(TraversalGlobalProperties)" />
</Target>

<Import Condition="'$(BuildTargetFramework)' == '$(NetCoreAppCurrent)'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "$(BuildNetCoreAppVertical) == true"

Copy link
Contributor Author

@layomia layomia Aug 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The BuildNetCoreAppVertical property does not appear to be set at this point in the build.

Project="$(MSBuildThisFileDirectory)\illink-sharedframework.targets" />

<Target Name="RunApiCompat"
Condition="'@(ApiCompatProject)' != ''"
AfterTargets="BuildManualShims">
Expand Down