Switch to new Microsoft.DotNet.SharedFramework.Sdk and refactor Host/…

…Installer build subsets (#38457)

* Start creating runtime, apphost, and targeting packs using the new SDK in the NewSFX subset.

* Fix generation of runtime package, runtime symbols package, and host symbols package.

* Correcly include hosting files and DIA in the shared framework archives and packages

* Include PackageOverrides.txt in the ref pack.

* Fix product brand prefix.

* Add the shared host installer.

* Add the hostfxr build and clean up MacOS building.

* Rename host and hostfxr projects to ensure we don't collide with the nuget packaging projects.

* Add bundle installer.

* Update platform manifest entries to include linux/mac file variations from the runtime.

* Update apphost pack to match master.

* Fix typo

* Include manpages and fix linux package properties.

* Fix linux package properties.

* Match short-description from packages in master.

* Update to uploaded package.

* Update metadata and SDK to generate an exact match of the productbuild xml file.

* First draft moving crossgen2 to new SDK.

* Get crossgen2 pack working with updated SDK.

* Build archives from the installer and bundle projects.

* Move the dotnet-runtime-deps installers to use the new SDK.

* Rough attempt to convert the build to use the new sfx sdk.

* Remove now empty subsets.

* Update installer subsets for testing.

* Remove non-existent packaging project from subsets.

* Add missing Windows api shim dll to platform manifest.

* Update sharedfx sdk.

* Add both versions of the long-name dac to the platform manifest.

* Skip builds for non-relevant components on Mono.

* Remove installer tests by default temporarily to validate product build.

* Remove pkgprojs subset from mono installer build since it doesn't build the hosts.

* Skip the apphost pack on mono.

* Update host test infra to use the new sfx projects.

* Delete an out of date test (the alternative case this test exercises is now the only case in this repo)

* Fix packaging tests and remove NETStandard packaging tests since we aren't producing that ref pack any more.

* Fixes for test official build failures. Re-eneable installer tests by default.

* Fix condition for building linux installers.

* Exclude Mono header files from the deps.json. They don't need to be in it.

* Don't include the host files on the sfxproj.

* Add mono-specific files to the platform manifest.

* Remove extra copy step in installer job.

* Include libmonosgen-2.0.a in platform manifest.

* Fix change in linux-x64 test steps.

* Fix display name of linux package steps.

* Build the installers based on the portable build when applicable so they can find their artifacts.

* The installer build no longer consumes Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets, try removing the dependency on the AllConfigurations leg from the installer build.

* Set PackageBrandNameSuffix for runtime-deps packages.

* Updated sharedfx sdk

* Fix debian runtime-deps dependency elements.

* Add new platform manifest entries.

* Add icudt.dat to the platform manifest.

* Invert conditions to correctly skip generating installers on non-applicable platforms.

* Create extra installer subset so we can build just the installers (and avoid double-building nuget packages). Rename the newsfx subset and folder to sfx.

* Fix Crossgen2 package name.

* Clean up some dead MSBuild goo and update the sharedfx package so we can override the crossgen2 package name without breaking the default package id logic.

* Update SharedFX SDK.

* Fix discovery of the built dotnet layout for installer tests. Clean up some MSBuild along this path.

* Update to use the new refactored packages.

* Fix macOS build failures.

* Fix linux installer build.

* Upgrade to newest version of the SDK and centrally manage versions.

* Update SDK to fix build skipping for the shared framework sdk.

* Update SDK with full fix for skipbuild with packaging.

* Another try for fixing the build-skip infra.

* Upgrade SDK.

* Fix package ids for mono packages.

* Fix installer tests after update.

* Add System.Private.Runtime.InteropServices.Javascript.dll to platform manifest

* Fix Crossgen2 package to place all files in the tools folder.

* Update to SDK with validation targets.

* Add new ICU files to manifest.

* Crossgen2 doesn't need to go through framework resolution so we can permit non-file-versioned dlls. (jitinterface.dll is currently unversioned).

Signed-off-by: Jeremy Koritzinsky <jekoritz@microsoft.com>

* Put the property in a property group.

* Ensure we build corehost before sfx since we no longer directly build the project from the sfx projects.

* Add missing entry for additional ICU data file.

* Remove step downloading all artifacts and remove MSBuild glue to hook up looking up artifacts in the full download.

Signed-off-by: Jeremy Koritzinsky <jekoritz@microsoft.com>

* Update SDK

Signed-off-by: Jeremy Koritzinsky <jekoritz@microsoft.com>

* Update SDK to get MSI fix.

Signed-off-by: Jeremy Koritzinsky <jekoritz@microsoft.com>

* Don't even try building the hosting, crossgen2, or bundle installers when RuntimeFlavor is Mono.

* Remove out of date comment.

Signed-off-by: Jeremy Koritzinsky <jekoritz@microsoft.com>

* Rename subsets based on PR discussion.

* Update platform manifest and fix path to 3rd party notices for installer.

* Remove test feed.

* Update subset references I missed.

* Fixed the subset arguments in other yaml files.

* Add NuGet metadata to packs.

* Add diagnostics-specific incremental signing in CoreCLR when the files are produced so the installer portion doesn't need to sign them.

* Pass sign type to job.

* Add new WASM files.

* Enable static graph restore for the whole repo

* Fix nits

* Add back the runtime-deps packages. They were lost when we moved away from globs.

* Don't disable NuGet static graph restore.

* Make sure we sign the cross-arch same-os DAC

* Update installers package.

* Update Installers package from arcade

* Update installers package.

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

Build.proj

@@ -12,16 +12,6 @@
     <ProjectReference Include="@(ProjectToBuild)" />
   </ItemGroup>
 
-  <!--
-    Exclude installer depproj and pkgproj from static graph restore. We restore them below.
-    Remove when https://github.com/NuGet/Home/issues/9398 is fixed.
-  -->
-  <ItemGroup Condition="'$(MSBuildRestoreSessionId)' != ''">
-    <ProjectReference Remove="@(DepprojProjectToBuild)"  />
-    <ProjectReference Remove="@(PkgprojProjectToBuild)" />
-    <ProjectReference Remove="@(BundleProjectToBuild)" />
-  </ItemGroup>
-
   <!-- Custom arcade target which isn't available in Microsoft.Build.Traversal. -->
   <Target Name="Rebuild" DependsOnTargets="Clean;Build" />
 
@@ -36,12 +26,4 @@
     <MSBuild Projects="$(RepoTasksDir)tasks.proj"
              Targets="BuildAndRestoreIncrementally"/>
   </Target>
-
-  <Target Name="RestoreWithoutStaticGraph"
-          BeforeTargets="Restore">
-    <MSBuild Projects="@(DepprojProjectToBuild);@(PkgprojProjectToBuild);@(BundleProjectToBuild)"
-             Properties="MSBuildRestoreSessionId=$([System.Guid]::NewGuid());RestoreUseStaticGraphEvaluation=false"
-             Targets="Restore" />
-  </Target>
-
 </Project>

eng/Signing.props

@@ -1,4 +1,4 @@
-<Project>
+<Project InitialTargets="SetupFilesToSign">
 
   <!-- If this file was pulled in via prepare-artifacts.proj (a non-SDK project, these files are already
        imported. -->
@@ -12,7 +12,7 @@
 
       During post build signing, there are no packages to sign during SignFinalPackages.
     -->
-    <AllowEmptySignList Condition="'$(SignFinalPackages)' != 'true' or '$(PostBuildSign)' == 'true'">true</AllowEmptySignList>
+    <AllowEmptySignList>true</AllowEmptySignList>
   </PropertyGroup>
 
   <ItemGroup>
@@ -22,10 +22,6 @@
     -->
     <ItemsToSign Remove="@(ItemsToSign)" />
 
-    <!-- Find bundle artifacts, which need multiple stages to fully sign. -->
-    <BundleInstallerEngineArtifact Include="$(ArtifactsPackagesDir)**/*engine.exe" />
-    <BundleInstallerExeArtifact Include="$(ArtifactsPackagesDir)**/*.exe" />
-
     <!-- apphost and comhost template files are not signed, by design. -->
     <FileSignInfo Include="apphost.exe;singlefilehost.exe;comhost.dll" CertificateName="None" />
 
@@ -49,126 +45,32 @@
     <FileExtensionSignInfo Include=".deb;.rpm" CertificateName="LinuxSign" />
   </ItemGroup>
 
-  <!-- When doing post build signing, the file containers (e.g. nupkg, msi, etc.) are
-       processed for signing (opened up, individually signed, etc.) and these individual ItemsToSign
-       elements are unnecessary. When signing within the build, we need to individually process
-       dll's, exes, etc. that go into msi's because these containers are not able to be processed
-       by SignTool after they are packed up. What makes this possible for post build signing
-       is that the build will produce a zip file containing the inputs to the Wix light linker
-       which can be used to create the installer later, after the inputs have been signed and replaced. -->
-  <Choose>
-    <When Condition="'$(PostBuildSign)' != 'true'">
-      <ItemGroup Condition="'$(SignBinaries)' == 'true'">
-        <!-- Sign CoreCLR. -->
-        <ItemsToSign Include="$(CoreCLRSharedFrameworkDir)*.dll" />
-        <ItemsToSign Include="$(CoreCLRSharedFrameworkDir)*.exe" />
-
-        <ItemsToSign Include="$(CoreCLRArtifactsPath)System.Private.CoreLib.dll" />
-
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)crossgen2.exe" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)crossgen2.dll" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)ILCompiler.DependencyAnalysisFramework.dll" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)ILCompiler.ReadyToRun.dll" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)ILCompiler.TypeSystem.ReadyToRun.dll" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)jitinterface_$(TargetArchitecture).dll" />
-
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)clrjit_win_x86_$(TargetArchitecture).dll" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)clrjit_win_arm_$(TargetArchitecture).dll" />
-        <ItemsToSign Include="$(CoreCLRCrossgen2Dir)clrjit_unix_arm_$(TargetArchitecture).dll" />
-        <ItemsToSign Condition="'$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'x64'" Include="$(CoreCLRCrossgen2Dir)clrjit_win_x64_$(TargetArchitecture).dll" />
-        <ItemsToSign Condition="'$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'x64'" Include="$(CoreCLRCrossgen2Dir)clrjit_win_arm64_$(TargetArchitecture).dll" />
-        <ItemsToSign Condition="'$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'x64'" Include="$(CoreCLRCrossgen2Dir)clrjit_unix_x64_$(TargetArchitecture).dll" />
-        <ItemsToSign Condition="'$(TargetArchitecture)' == 'arm64' or '$(TargetArchitecture)' == 'x64'" Include="$(CoreCLRCrossgen2Dir)clrjit_unix_arm64_$(TargetArchitecture).dll" />
-
-        <ItemsToSign Include="$(CoreCLRArtifactsPath)$(CoreCLRCrossTargetComponentDirName)/sharedFramework/*.dll" Condition="'$(CoreCLRCrossTargetComponentDirName)' != ''" />
-        <ItemsToSign Include="$(CoreCLRArtifactsPath)$(CoreCLRCrossTargetComponentDirName)/sharedFramework/*.exe" Condition="'$(CoreCLRCrossTargetComponentDirName)' != ''" />
-
-        <!-- Sign api-ms-win-core-xstate-l2-1-0 binary as it is only catalog signed in the current SDK. -->
-        <ItemsToSign
-          Condition="'$(Configuration)' == 'Release' and '$(TargetArchitecture)' == 'x86'"
-          Include="$(CoreCLRArtifactsPath)Redist\ucrt\DLLs\$(TargetArchitecture)\api-ms-win-core-xstate-l2-1-0.dll" />
-
-        <!-- Sign libraries. -->
-        <ItemsToSign Include="$(LibrariesNativeArtifactsPath)*.dll" />
-        <ItemsToSign Include="$(LibrariesSharedFrameworkRefArtifactsPath)*.dll" />
-        <!-- Most runtime artifacts will be crossgenned, so sign them post-crossgen. mscorlib isn't. -->
-        <ItemsToSign Include="$(LibrariesSharedFrameworkBinArtifactsPath)mscorlib.dll" />
-
-        <!-- Sign the host. -->
-        <ItemsToSign Include="$(BaseOutputRootPath)corehost/**/hostfxr.dll" />
-        <ItemsToSign Include="$(BaseOutputRootPath)corehost/**/hostpolicy.dll" />
-        <ItemsToSign Include="$(BaseOutputRootPath)corehost/**/dotnet.exe" />
-        <ItemsToSign Include="$(BaseOutputRootPath)corehost/**/ijwhost.dll" />
-        <ItemsToSign Include="$(BaseOutputRootPath)corehost/**/nethost.dll" />
-
-        <!-- Sign managed libraries in installer subset. -->
-        <ItemsToSign Include="$(ArtifactsBinDir)Microsoft.NET.HostModel/**/*.dll" />
-      </ItemGroup>
-
-      <!-- Sign ready-to-run binaries after crossgen is applied. -->
-      <ItemGroup Condition="'$(SignR2RBinaries)' == 'true'">
-        <ItemsToSign Include="$(CrossGenRootPath)**/*.dll" />
-      </ItemGroup>
-
-      <ItemGroup Condition="'$(SignMsiFiles)' == 'true'">
-        <ItemsToSign Include="$(ArtifactsPackagesDir)**/*.msi" />
-        <ItemsToSign Include="$(ArtifactsPackagesDir)**/*.cab" />
-      </ItemGroup>
-
-      <ItemGroup Condition="'$(SignBurnEngineFiles)' == 'true'">
-        <ItemsToSign Include="@(BundleInstallerEngineArtifact)" />
-      </ItemGroup>
-
-      <ItemGroup Condition="'$(SignBurnBundleFiles)' == 'true'">
-        <!-- Sign the bundles, now that the engine is reattached. Avoid re-signing the engine. -->
-        <ItemsToSign
-          Include="@(BundleInstallerExeArtifact)"
-          Exclude="@(BundleInstallerEngineArtifact)" />
-        <!-- Note: wixstdba is internal to the engine bundle and does not get signed. -->
-      </ItemGroup>
-
-      <ItemGroup Condition="'$(SignFinalPackages)' == 'true'">
-        <DownloadedSymbolPackages Include="$(DownloadDirectory)**\*.symbols.nupkg" />
-        <ItemsToSign Include="$(DownloadDirectory)**\*.nupkg" Exclude="@(DownloadedSymbolPackages)" />
-
-        <!-- The cross OS diagnostics symbol packages need to be signed as they are the only packages
-        that have a specific version of assets that are only meant to be indexed in symbol servers.
-        Since only *symbols.nupkg get indexed, and installer doesn't produce these, we need to glob them for signing. -->
-        <ItemsToSign Include="$(DownloadDirectory)**\*CrossOsDiag*.symbols.nupkg" />
-
-        <ItemsToSign Include="$(DownloadDirectory)**\*.deb" />
-        <ItemsToSign Include="$(DownloadDirectory)**\*.rpm" />
-      </ItemGroup>
-    </When>
-
-    <!-- When doing post build signing, we sign all artifacts we would push.
-         Symbol packages are included too. -->
-    <When Condition="'$(PostBuildSign)' == 'true'">
-      <ItemGroup>
-        <ItemsToSignWithPaths Include="$(DownloadDirectory)**/*.msi" Condition="'$(PrepareArtifacts)' == 'true'" />
-        <ItemsToSignWithPaths Include="$(DownloadDirectory)**/*.exe" Condition="'$(PrepareArtifacts)' == 'true'" />
-        <ItemsToSignWithPaths Include="$(DownloadDirectory)**/*.nupkg" Condition="'$(PrepareArtifacts)' == 'true'" />
-        <ItemsToSignWithPaths Include="$(DownloadDirectory)**/*.zip" Condition="'$(PrepareArtifacts)' == 'true'" />
-
-        <ItemsToSignWithoutPaths Include="@(ItemsToSignWithPaths->'%(Filename)%(Extension)')" />
-        <ItemsToSignPostBuild Include="@(ItemsToSignWithoutPaths->Distinct())" />
-      </ItemGroup>
-
-      <!-- Even when doing post build signing, sign mscordaccore*.dll and mscordbi.dll -->
-      <ItemGroup Condition="'$(SignBinaries)' == 'true'">
-        <ItemsToSign Include="$(CoreCLRSharedFrameworkDir)mscordaccore*.dll" />
-        <ItemsToSign Include="$(CoreCLRSharedFrameworkDir)mscordbi.dll" />
-        <ItemsToSign Include="$(CoreCLRArtifactsPath)$(CoreCLRCrossTargetComponentDirName)/sharedFramework/mscordaccore*.dll" Condition="'$(CoreCLRCrossTargetComponentDirName)' != ''" />
-        <ItemsToSign Include="$(CoreCLRArtifactsPath)$(CoreCLRCrossTargetComponentDirName)/sharedFramework/mscordbi.dll" Condition="'$(CoreCLRCrossTargetComponentDirName)' != ''" />
-      </ItemGroup>
-
-      <ItemGroup Condition="'$(SignFinalPackages)' == 'true'">
-        <!-- The cross OS diagnostics symbol packages need to be signed as they are the only packages
-        that have a specific version of assets that are only meant to be indexed in symbol servers.
-        Since only *symbols.nupkg get indexed, and installer doesn't produce these, we need to glob them for signing. -->
-        <ItemsToSign Include="$(DownloadDirectory)**\*CrossOsDiag*.nupkg" />
-      </ItemGroup>
-    </When>
-  </Choose>
+  <Target Name="SetupFilesToSign">
+    <!-- Ensure that we don't miss the DAC or DBI with the globbing below -->
+    <PropertyGroup Condition="'$(SignDiagnostics)' == 'true'">
+      <AllowEmptySignList>false</AllowEmptySignList>
+    </PropertyGroup>
+
+    <ItemGroup Condition="'$(SignDiagnostics)' == 'true'">
+      <ItemsToSign Include="$(DiagnosticsFilesRoot)/**/mscordaccore*.dll" />
+      <ItemsToSign Include="$(DiagnosticsFilesRoot)/**/mscordbi.dll" />
+      <!--
+        The DAC should be signed with the SHA2 cert (both long and short name).
+        We already add the short-name DAC above, so add the long-name DAC here.
+      -->
+      <DacFileSignInfo Include="@(ItemsToSign->'%(FileName)%(Extension)')"
+                    Condition="$([System.String]::new('%(FileName)').StartsWith('mscordaccore'))" />
+      <FileSignInfo Include="@(DacFileSignInfo->ClearMetadata()->Distinct())"
+                    Exclude="mscordaccore.dll"
+                    CertificateName="MicrosoftSHA2" />
+    </ItemGroup>
+
+    <ItemGroup Condition="'$(SignFinalPackages)' == 'true'">
+      <!-- The cross OS diagnostics symbol packages need to be signed as they are the only packages
+      that have a specific version of assets that are only meant to be indexed in symbol servers.
+      Since only *symbols.nupkg get indexed, and installer doesn't produce these, we need to glob them for signing. -->
+      <ItemsToSign Include="$(DownloadDirectory)**\*CrossOsDiag*.nupkg" />
+    </ItemGroup>
+  </Target>
 
 </Project>