Use Roslyn to create ref/ assemblies (#23403)

* Remove all ref/ projects

* Remove GenAPI infrastructure

* Remove notion of a reference assembly project
  - remove `$(IsReferenceAssemblyProject)`, `$(ReferenceReferenceAssemblies)` and `$(ReferenceImplementationAssemblies)`
    - remove unnecessary `$(NoWarn)` settings

nits:
- remove a few misleading comments
- wrap some long lines

* Move .0 package version workaround into Versions.props
  - touch up SharedFramework.External.props

* Expose `%(LatestPackageReference.RTMVersion)` metadata
  - automate use of properties in the `@(LatestPackageReference)` item group to make this maintainable
    - add a couple of special cases at the bottom of eng/Dependencies.props
    - add one more `$(...PackageVersion)` property to avoid yet-another special case

* Enable Roslyn reference assemblies
  - exclude ref/ assembly from packages other than targeting pack
  - update Microsoft.AspNetCore.App.Ref.csproj
    - `%(IsReferenceAssembly)` and `%(ReferenceGrouping)` metadata no longer relevant
    - only ref/ assemblies are in `@(ReferencePathWithRefAssemblies)` item group

nits:
  - remove now-unnecessary workaround
    - issues with TFM transition are behind us
  - clean up Microsoft.AspNetCore.App.Runtime.csproj slightly
    - use `GeneratePathProperty="true"`
        - reorder item / property settings for meta-expansion
    - correct spelling errors and phrasing in comments

* Update documentation to reflect recent changes
  - remove CrossRepoBreakingChanges.md; was tied to old TeamCity infrastructure
    - also much less relevant given repo merges
  - adjust details and examples in ReferenceResolution.md
    - reflect repo merges, Dependencies.props changes, and current Maestro++ channels
    - add a few more details e.g. specific files where Version.Details.xml versions are used

* !fixup! Remove another irrelevant doc file

* !fixup! Address PR review suggestions
  - convert a couple of warnings to errors
  - use consistent casing for Microsoft.NETCore.App.Runtime.* packages
  - reduce `%(LatestPackageReference.Version)` metadata special cases
  - add and improve comments e.g.
    - improve comments about `$(*V0PackageVersion)` properties
    - improve placement of comments about item removal in ResolveReferences.targets
    - confirmed `$(*V0PackageVersion)` property list is complete

nits:
- fix solution example in ReferenceResolution.md
- remove item group definition for `@(LatestPackageReference)`
- remove `%(LatestPackageReference.VersionName)` metadata after use; large item group
    - similarly, remove `%(LatestPackageReference.RTMVersion)` when not needed; just complicates `Condition`s

When I squash, I must remember this fixes
- #14801
- dotnet/aspnetcore-internal#2693

* Actually use `%(LatestPackageReference.RTMVersion)` metadata
  - gather RTM package references in a new project
    - a (very) separate project to work around package conflict resolution
    - empty `Test` target works around Arcade's testing approach
  - new target in ResolveReferences.targets updates relevant assembly paths to use the RTM packages
    - done as soon as possible after `ResolvePackageAssets` determines the paths
    - done for all compilation inputs, not just ref/ assemblies

Directory.Build.props

@@ -11,10 +11,6 @@
   <PropertyGroup>
     <RepoRelativeProjectDir>$([MSBuild]::MakeRelative($(RepoRoot), $(MSBuildProjectDirectory)))</RepoRelativeProjectDir>
 
-    <IsReferenceAssemblyProject>false</IsReferenceAssemblyProject>
-    <IsReferenceAssemblyProject Condition="$(MSBuildProjectDirectory.EndsWith('ref'))">true</IsReferenceAssemblyProject>
-    <OutDirName Condition="'$(IsReferenceAssemblyProject)' == 'true'">$(MSBuildProjectName)-ref</OutDirName>
-
     <IsBenchmarkProject Condition="$(MSBuildProjectName.EndsWith('.Performance')) OR $(RepoRelativeProjectDir.Contains('perf')) OR $(RepoRelativeProjectDir.Contains('benchmarkapps'))">true</IsBenchmarkProject>
     <IsSpecificationTestProject Condition="$(MSBuildProjectName.EndsWith('.Specification.Tests'))">true</IsSpecificationTestProject>
     <IsUnitTestProject>false</IsUnitTestProject>
@@ -130,7 +126,7 @@
     <RuntimeInstallerBaseName>aspnetcore-runtime</RuntimeInstallerBaseName>
     <TargetingPackInstallerBaseName>aspnetcore-targeting-pack</TargetingPackInstallerBaseName>
 
-    <!-- Produce targeting pack installers/packages once per major.minor. -->
+    <!-- This is used to produce targeting pack installers/packages once per major.minor. -->
     <IsTargetingPackBuilding Condition=" '$(DotNetBuildFromSource)' == 'true' ">false</IsTargetingPackBuilding>
     <IsTargetingPackBuilding
         Condition=" '$(IsTargetingPackBuilding)' == '' AND '$(AspNetCorePatchVersion)' != '0' ">false</IsTargetingPackBuilding>

Directory.Build.targets

@@ -1,10 +1,10 @@
-<Project>
+<Project>
 
   <PropertyGroup>
     <!-- Only build Microsoft.AspNetCore.App and ref/ assemblies in source build. -->
     <!-- Analyzer package are needed in source build for WebSDK -->
     <ExcludeFromSourceBuild
-        Condition="'$(ExcludeFromSourceBuild)' == '' and '$(DotNetBuildFromSource)' == 'true' and '$(IsAspNetCoreApp)' != 'true' and '$(IsReferenceAssemblyProject)' != 'true' and '$(IsAnalyzersProject)' != 'true'">true</ExcludeFromSourceBuild>
+        Condition="'$(ExcludeFromSourceBuild)' == '' and '$(DotNetBuildFromSource)' == 'true' and '$(IsAspNetCoreApp)' != 'true' and '$(IsAnalyzersProject)' != 'true'">true</ExcludeFromSourceBuild>
 
     <!-- If the user has specified that they want to skip building any test related projects with SkipTestBuild,
      suppress all targets for TestProjects using ExcludeFromBuild. -->
@@ -57,7 +57,8 @@
 
   <PropertyGroup Condition=" '$(MSBuildProjectExtension)' == '.csproj' ">
     <PackageId Condition=" '$(PackageId)' == '' ">$(AssemblyName)</PackageId>
-    <IsPackable Condition="'$(IsPackable)' == '' AND ( '$(IsTestProject)' == 'true' OR '$(IsTestAssetProject)' == 'true' OR '$(IsBenchmarkProject)' == 'true' OR '$(IsSampleProject)' == 'true' OR '$(IsReferenceAssemblyProject)' == 'true' ) ">false</IsPackable>
+    <IsPackable
+        Condition="'$(IsPackable)' == '' AND ( '$(IsTestProject)' == 'true' OR '$(IsTestAssetProject)' == 'true' OR '$(IsBenchmarkProject)' == 'true' OR '$(IsSampleProject)' == 'true' ) ">false</IsPackable>
   </PropertyGroup>
 
   <Import Project="eng\Baseline.Designer.props" />
@@ -100,14 +101,19 @@
 
   <PropertyGroup>
     <!-- Implementation projects are the projects which produce nuget packages or shipping assemblies. -->
-    <IsImplementationProject Condition=" '$(IsImplementationProject)' == '' AND '$(IsAnalyzersProject)' != 'true' AND '$(IsTestAssetProject)' != 'true' AND '$(IsTestProject)' != 'true' AND '$(IsBenchmarkProject)' != 'true' AND '$(IsSampleProject)' != 'true' AND '$(IsReferenceAssemblyProject)' != 'true' ">true</IsImplementationProject>
+    <IsImplementationProject Condition=" '$(IsImplementationProject)' == '' AND '$(IsAnalyzersProject)' != 'true' AND '$(IsTestAssetProject)' != 'true' AND '$(IsTestProject)' != 'true' AND '$(IsBenchmarkProject)' != 'true' AND '$(IsSampleProject)' != 'true' ">true</IsImplementationProject>
 
     <!-- This determines whether a project is available as a <Reference> to other projects in this repo. -->
     <IsProjectReferenceProvider Condition=" '$(IsProjectReferenceProvider)' == '' AND '$(IsImplementationProject)' == 'true' AND '$(PackAsTool)' != 'true' ">true</IsProjectReferenceProvider>
 
     <HasReferenceAssembly
-        Condition=" '$(HasReferenceAssembly)' == '' AND '$(IsProjectReferenceProvider)' == 'true' AND '$(IsAspNetCoreApp)' == 'true' ">true</HasReferenceAssembly>
+        Condition=" '$(TargetFramework)' == '$(DefaultNetCoreTargetFramework)' AND '$(IsAspNetCoreApp)' == 'true' ">true</HasReferenceAssembly>
     <HasReferenceAssembly Condition="'$(HasReferenceAssembly)' == ''">false</HasReferenceAssembly>
+    <ProduceReferenceAssembly>$(HasReferenceAssembly)</ProduceReferenceAssembly>
+
+    <!-- Duplicate setting from Microsoft.Common.CurrentVersion.targets because this is imported after that file. -->
+    <TargetRefPath
+        Condition=" '$(TargetRefPath)' == '' and $(ProduceReferenceAssembly) ">$([MSBuild]::NormalizePath($(TargetDir), 'ref', $(TargetFileName)))</TargetRefPath>
 
     <IsPackable Condition="'$(IsPackable)' == '' AND ('$(IsImplementationProject)' == 'true' OR '$(IsAnalyzersProject)' == 'true') ">true</IsPackable>
     <IsPackable Condition="'$(IsPackable)' == '' ">false</IsPackable>
@@ -124,13 +130,7 @@
     <None Include="$(PackageThirdPartyNoticesFile)" Pack="true" PackagePath="." />
   </ItemGroup>
 
-  <ItemGroup Condition="'$(Language)' == 'C#' AND '$(IsReferenceAssemblyProject)' == 'true'">
-    <Compile Include="$(SharedSourceRoot)ReferenceAssemblyInfo.cs" LinkBase="Properties" />
-  </ItemGroup>
-
   <PropertyGroup Condition="'$(Language)' == 'C#'">
-    <!-- Reference assemblies should always use Major.Minor.0.0 for assembly versions even during servicing. Only the package version should be updated. -->
-    <!-- Pinning the implementation assemblies at Major.Minor.0.0 as we figure out compiling against ref assemblies. -->
     <AssemblyVersion>$(AspNetCoreMajorVersion).$(AspNetCoreMinorVersion).0.0</AssemblyVersion>
   </PropertyGroup>
 
@@ -180,7 +180,6 @@
   <Import Project="eng\targets\FSharp.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.fsproj'" />
   <Import Project="eng\targets\Wix.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.wixproj'" />
   <Import Project="eng\targets\Npm.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.npmproj'" />
-  <Import Project="eng\targets\ReferenceAssembly.targets" Condition=" $(HasReferenceAssembly) " />
   <Import Project="eng\targets\Helix.targets" Condition="'$(IsTestProject)' == 'true'" />
   <Import Project="eng\targets\FunctionalTestAsset.targets" Condition="'$(IsTestAssetProject)' == 'true'" />
   <Import Project="eng\targets\FunctionalTestWithAssets.targets" Condition="'$(ContainsFunctionalTestAssets)' == 'true'" />

docs/ReferenceResolution.md

@@ -10,10 +10,12 @@ See [ResolveReferences.targets](/eng/targets/ResolveReferences.targets) for the
 
 The requirements that led to this system are:
 
-* Versions of external dependencies should be consistent.
-* Servicing updates of ASP.NET Core should minimize the number of assemblies which need to re-build and re-ship.
+* Versions of external dependencies should be consistent and easily discovered.
 * Newer versions of packages should not have lower dependency versions than previous releases.
 * Minimize the cascading effect of servicing updates where possible by keeping a consistent baseline of dependencies.
+* Servicing releases should not add or remove dependencies in existing packages.
+
+As a minor point, the current system also makes our project files somewhat less verbose.
 
 ## Recommendations for writing a .csproj
 
@@ -29,30 +31,31 @@ The requirements that led to this system are:
 
 ## Important files
 
-* [eng/Baseline.xml](/eng/Baseline.xml) - this contains the 'baseline' of the latest servicing release for this branch. It should be modified and used to update the generated file, Baseline.Designer.props.
+* [eng/Baseline.xml](/eng/Baseline.xml) - this contains the 'baseline' of the latest servicing release for this branch.
+  It should be modified and used to update the generated file, [eng/Baseline.Designer.props](eng/Baseline.Designer.props).
 * [eng/Dependencies.props](/eng/Dependencies.props) - contains a list of all package references that might be used in the repo.
 * [eng/ProjectReferences.props](/eng/ProjectReferences.props) - lists which assemblies or packages might be available to be referenced as a local project.
 * [eng/Versions.props](/eng/Versions.props) - contains a list of versions which may be updated by automation. This is used by MSBuild to restore and build.
-* [eng/Version.Details.xml](/eng/Version.Details.xml) - used by automation to update dependencies variables in other files.
+* [eng/Version.Details.xml](/eng/Version.Details.xml) - used by automation to update dependency variables in
+  [eng/Versions.props](/eng/Versions.props) and, for SDKs and `msbuild` toolsets, [global.json](global.json).
 
 ## Example: adding a new project
 
 Steps for adding a new project to this repo.
 
 1. Create the .csproj
 2. Run `eng/scripts/GenerateProjectList.ps1`
-3. Add it to Extensions.sln
+3. Add new project to AspNetCore.sln and any relevant `*.slnf` files
 
 ## Example: adding a new dependency
 
 Steps for adding a new package dependency to an existing project. Let's say I'm adding a dependency on System.Banana.
 
 1. Add the package to the .csproj file using `<Reference Include="System.Banana" />`
-2. Add an entry to [eng/Dependencies.props](/eng/Dependencies.props), `<LatestPackageReference Include="System.Banana" Version="0.0.1-beta-1" />`
-3. If this package comes from another dotnet team and should be updated automatically by our bot...
-    1. Change the LatestPackageReference entry to `Version="$(SystemBananaPackageVersion)"`.
-    2. Add an entry to [eng/Versions.props](/eng/Versions.props) like this `<SystemBananaPackageVersion>0.0.1-beta-1</SystemBananaPackageVersion>`.
-    3. Add an entry to [eng/Version.Details.xml](/eng/Version.Details.xml) like this:
+2. Add an entry to [eng/Dependencies.props](/eng/Dependencies.props) e.g. `<LatestPackageReference Include="System.Banana" />`
+3. If this package comes from another dotnet team and should be updated automatically by our bot&hellip;
+    1. Add an entry to [eng/Versions.props](/eng/Versions.props) like this `<SystemBananaPackageVersion>0.0.1-beta-1</SystemBananaPackageVersion>`.
+    2. Add an entry to [eng/Version.Details.xml](/eng/Version.Details.xml) like this:
 
         ```xml
         <ProductDependencies>
@@ -65,27 +68,28 @@ Steps for adding a new package dependency to an existing project. Let's say I'm
         </ProductDependencies>
         ```
 
-       If you don't know the commit hash of the source code used to produce "0.0.1-beta-1", you can use `000000` as a placeholder for `Sha`
-       as its value is unimportant and will be updated the next time the bot runs.
+        If you don't know the commit hash of the source code used to produce "0.0.1-beta-1", you can use `000000` as a
+        placeholder for `Sha` as its value will be updated the next time the bot runs.
 
-        If the new dependency comes from dotnet/CoreFx, dotnet/code-setup or dotnet/extensions, add a
+        If the new dependency comes from dotnet/runtime and you are updating dotnet/aspnetcore-tooling, add a
         `CoherentParentDependency` attribute to the `<Dependency>` element as shown below. This example indicates the
-        dotnet/CoreFx dependency version should be determined based on the build that produced the chosen
-        Microsoft.NETCore.App. That is, the dotnet/CoreFx dependency and Microsoft.NETCore.App should be
-        coherent.
+        dotnet/runtime dependency version for System.Banana should be determined based on the dotnet/aspnetcore build
+        that produced the chosen Microsoft.CodeAnalysis.Razor. That is, the dotnet/runtime and dotnet/aspnetcore
+        dependencies should be coherent.
 
         ```xml
-        <Dependency Name="System.Banana" Version="0.0.1-beta-1" CoherentParentDependency="Microsoft.NETCore.App">
+        <Dependency Name="System.Banana" Version="0.0.1-beta-1" CoherentParentDependency="Microsoft.CodeAnalysis.Razor">
           <!-- ... -->
         </Dependency>
         ```
 
-        The attribute value should be `"Microsoft.Extensions.Logging"` for dotnet/core-setup dependencies and
-        `"Microsoft.CodeAnalysis.Razor"` for dotnet/extensions dependencies.
+        The attribute value should be `"Microsoft.CodeAnalysis.Razor"` for dotnet/runtime dependencies in
+        dotnet/aspnetcore-tooling.
 
 ## Example: make a breaking change to references
 
-If Microsoft.AspNetCore.Banana in 2.1 had a reference to `Microsoft.AspNetCore.Orange`, but in 3.0 this reference is changing to `Microsoft.AspNetCore.BetterThanOrange`, you would need to make these changes to the .csproj file
+If Microsoft.AspNetCore.Banana in 2.1 had a reference to `Microsoft.AspNetCore.Orange`, but in 3.1 or 5.0 this reference
+is changing to `Microsoft.AspNetCore.BetterThanOrange`, you would need to make these changes to the .csproj file
 
 ```diff
 <!-- in Microsoft.AspNetCore.Banana.csproj -->
@@ -100,8 +104,12 @@ If Microsoft.AspNetCore.Banana in 2.1 had a reference to `Microsoft.AspNetCore.O
 
 If the `dotnet-maestro` bot has not correctly updated the dependencies, it may be worthwhile running `darc` manually:
 
-1. Install `darc` as described in https://github.com/dotnet/arcade/blob/master/Documentation/Darc.md#setting-up-your-darc-client
-2. Run `darc update-dependencies --channel '.NET Core 3 Dev'`
-   * Use `'.NET Core 3 Release'` in a `release/3.0-*` branch
+1. Install `darc` as described in <https://github.com/dotnet/arcade/blob/master/Documentation/Darc.md#setting-up-your-darc-client>
+2. Run `darc update-dependencies --channel '.NET Core 3.1 Release'`
+   * Use `'trigger-subscriptions'` to prod the bot to create a PR if you do not want to make local changes
+   * Use `'.NET 3 Eng''` to update dependencies from dotnet/arcade
+   * Use `'.NET Eng - Latest'` to update dependencies from dotnet/arcade in the `master` branch
+   * Use `'VS Master'` to update dependencies from dotnet/roslyn in the `master` branch
+   * Use `'.NET 5 Dev'` to update dependencies from dotnet/efcore or dotnet/runtime in the `master` branch
 3. `git diff` to confirm the tool's changes
 4. Proceed with usual commit and PR