VS2019 build tools on CI server fails to restore NuGet packages

[mrabey]

When building our solution using a CI server and VS 2019 build tools, a previously working msbuild command no longer works. VS2017 build tools successfully restores NuGet package references within the solution and then continues on with the build process. All attempts to build the project locally have succeeded, even when using VS2019 build tools. This issue only pops up when using our CI server using the VS2019 build tools specifically.

Our current strategy to handle NuGet packages is to upload to the version control repository and then copy all packages to another directory to be used during building as if they were always a part of the project. However, this relied on packages.config files using HintPaths. I am attempting to find a solution that avoids packages.config files as you guys have mentioned many times that PackageReferences are the future for NuGet package management. You can see my attempt at pointing NuGet to a local directory for a package source in the nuget.config file provided below. From what I've read, it doesn't sound like there is much needed for a local NuGet source feed other than the directory follows the hierarchy structure. While the "Solution Packages" feed would show up in the Visual Studio Solution Package Manager window and, when selected, will display all packages located in that directory, I'm unsure whether it has any affect on the restoration steps of the MSBuild command below.

As a side note, our CI server's Build Agents do not have network access so all package information needs to be loaded into memory before the build process can commence. That is why the old strategy was to avoid NuGet restore all together and use HintPaths that point to copied directory locations.

Steps to reproduce

Have a .NET Framework executable reference a .NET Standard library that depends on the NuGet package Microsoft.AspNetCore.SignalR.Client. Place all NuGet packages into a directory at the solution level.

Project file

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <Platforms>AnyCPU;x64</Platforms>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
    <PlatformTarget>x64</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.SignalR.Client" Version="1.1.0" />
  </ItemGroup>

</Project>

Directory structure:

- Extension/
   - Communication/
      - ServerCommunication/
         - AdminProxy/
         - Messages/
         - TerminalProxy/
         - BtmCommunication.csproj
   - packages/
      - { all of the NuGet packages for everything in Extension directory }
   - BuildAll.proj
   - nuget.config
   - <SolutionName>.sln

NuGet Config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <!-- This points all NuGet packages referenced in PackageReferences (in *.csproj files)
         to the current working directory's 'packages' directory -->
    <add key="globalPackagesFolder" value=".\packages" />
    <!-- This points all NuGet packages referenced in package.config files 
         to the current working directory's 'packages' directory -->
    <add key="repositoryPath" value=".\packages" />
  </config>
   <!--
      I attempted to use this to point NuGet to the local 'packages' directory rather than a "global-packages" directory, but leaving it in or out results in the same error on the CI server. 
      <packageSources>
         <add key="Solution Packages" value="packages" />
      </packageSources>
   -->
</configuration>

Command line

msbuild /m /restore /t:Rebuild /p:OutputDir="%teamcity.build.workingDir%\Assemblies" /p:Configuration=Debug /p:WarningLevel=0

The "Rebuild" target in BuildAll.proj runs a clean and then build on the solution.

Expected behavior

When NuGet restores the packages located within the 'packages' directory, the restoration fully generates the dependency graph and continues on to the build process.

Actual behavior

When building with VS2017 on TeamCity, the restoration works fine. The project continues to build and the output is generated as expected.

However, when building with VS2019 on TeamCity, the restoration fails with error:

[Step 1/2] <SolutionName>\BuildAll.proj.teamcity: Build targets: Restore;Restore
   [<SolutionName>\BuildAll.proj.teamcity] Restore
      [Restore] MSBuild
         [MSBuild] <SolutionName>\Extension\<SolutionName>.sln: Build target: Restore
            [<SolutionName>\Extension\<SolutionName>.sln] ValidateSolutionConfiguration
            [<SolutionName>\Extension\<SolutionName>.sln] _FilterRestoreGraphProjectInputItems
               [_FilterRestoreGraphProjectInputItems] MSBuild
                  [MSBuild] <SolutionName>\Extension\Communication\BtmCommunication\BtmCommunication.csproj: Build target: _IsProjectRestoreSupported
                     [<SolutionName>\Extension\Communication\BtmCommunication\BtmCommunication.csproj] Project <SolutionName>\Extension\Communication\BtmCommunication\BtmCommunication.csproj failed.

There is no other error output that follows this "failed" on NuGet restoration. Googling anything about "_IsProjectRestoreSupported" provides me nothing other than an issue where that target does not exist. From everything I've been able to see, this is not the issue. Everything within the solution utilizes the PackageReferences as we had one .NET Framework project referencing a .NET Standard project (BtmCommunication). I'm unaware of any other settings that would pass/fail the "_IsProjectRestoreSupported" target.

We have also tried running a separate MSBuild command to restore the BuildAll.proj contents before ever running the "rebuild" target.

Update:

When running the MSBuild command above with the following argument:
/flp1:logfile=output.log;verbosity=diagnostic;
The output is the same:

15:52:08.493    17>Building with tools version "Current".
15:52:08.494    17>Done Building Project "D:\BuildAgent\work\20dadb24aca6d1b\<SolutionName>\Extension\Communication\BtmCommunication\BtmCommunication.csproj" (_IsProjectRestoreSupported target(s)) -- FAILED.

I've been looking at the NuGet.targets file and it claims _IsProjectRestoreSupported is defined as:

<Target Name="_IsProjectRestoreSupported"
          Returns="@(_ValidProjectsForRestore)">
  <ItemGroup>
    <_ValidProjectsForRestore Include="$(MSBuildProjectFullPath)" />
  </ItemGroup>
</Target>

That to me states that as long as the project provides a "MSBuildProjectFullPath" value, then it will return true. Looking through the diagnostic build output, every project within the solution passes the restoration except the BtmCommunication project which is the .NET Standard project. However, I am at a loss as to why it would fail on that project. It was the first project I had using the PackageReferences and worked to update the remainder of the solution to match this project. Yet, everything else is passing the _IsProjectRestoreSupported target.

An excerpt from the BuildAll.proj file:

<Target Name="Restore">
    <MSBuild Projects="@(SolutionsToRestore)" Targets="Restore" BuildInParallel="True" StopOnFirstFailure="True"/>
</Target>

"SolutionsToRestore" is an entry in an ItemGroup that targets the solution that contains ServerCommunication:

<SolutionsToRestore Include="$(MSBuildProjectDirectory)\Extension\<SolutionName>.sln">
    <Properties>Configuration=$(Configuration)</Properties>
    <Properties>PlatformTarget=$(PlatformTarget)</Properties>
</SolutionsToRestore>

Environment data

msbuild /version output:
When building with VS2017 on TeamCity:

• version: 15.9.21.664

When building with VS2019 on TeamCity:

• version: 16.1.76.45076

OS info:
Windows 10.0.18362

This is a VM running on our CI server (TeamCity).

[livarcocc]

@nkolev92 @rrelyea any ideas what might be going on here?

[nkolev92]

  <config>
    <!-- This points all NuGet packages referenced in PackageReferences (in *.csproj files)
         to the current working directory's 'packages' directory -->
    <add key="globalPackagesFolder" value=".\packages" />
    <!-- This points all NuGet packages referenced in package.config files 
         to the current working directory's 'packages' directory -->
    <add key="repositoryPath" value=".\packages" />
  </config>

The repository path and the global packages folder cannot point to the same folder.
The structure of the global packages folder is different from the structure of the packages folder used by packages.config.
Specifically:
global packages folder

packages
  packageId
    packageVersion

packages folder:

packages
  packageId.packageVersion

Did you mean to have the packages be source from the packages folder in the packageReference scenario?

That to me states that as long as the project provides a "MSBuildProjectFullPath" value, then it will return true. Looking through the diagnostic build output, every project within the solution passes the restoration except the ServerCommunication project which is the .NET Standard project. However, I am at a loss as to why it would fail on that project. It was the first project I had using the PackageReferences and worked to update the remainder of the solution to match this project. Yet, everything else is passing the _IsProjectRestoreSupported target.

The purpose of that target is to figure out if a project imports the NuGet targets.
If it does, then the project will be able to return itself. Can you upload the value of preprocess command.
msbuild /pp:pp.txt <projectName.csproj>
Specifically comparing the preprocess outputs of both toolsets would help get to the bottom of this.

[mrabey]

I will update the nuget.config file to have the repositoryPath point to a different location temporarily seeing as though every project uses PackageReferences for this solution.

Did you mean to have the packages be source from the packages folder in the packageReference scenario?

Yes, I am attempting to make the packages directory act as the package source feed for NuGet when using PackageReferences. That way we can totally avoid packages.config files in the future, and slowly migrate our current process of copied NuGet package directories with manually updated HintPaths in the csproj files.

Can you upload the value of preprocess command.
msbuild /pp:pp.txt <projectName.csproj>
Specifically comparing the preprocess outputs of both toolsets would help get to the bottom of this.

Certainly.
When running that commend using VS2019 Developer Command Prompt, it produces an error:

C:\SBK\Extension\Communication\BtmCommunication\BtmCommunication.csproj : error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not be found.

When running that command using VS2017 Developer Command Prompt, it successfully generates a pp.txt file. If you want to see the output, what would the best way to get you the file be? All of my company SharePoints are password protected.

In the meantime, I'll look into why the VS2019 Developer Command Prompt cannot find the .NET SDK while the VS2017 Developer Command Prompt has no issue.

[nkolev92]

Yes, I am attempting to make the packages directory act as the package source feed for NuGet when using PackageReferences. That way we can totally avoid packages.config files in the future, and slowly migrate our current process of copied NuGet package directories with manually updated HintPaths in the csproj files.

You need to that location explitly as a feed, the same you'd add nuget.org or any other feed.

C:\SBK\Extension\Communication\BtmCommunication\BtmCommunication.csproj : error MSB4236: The SDK 'Microsoft.NET.Sdk' specified could not be found.

I think this is our root cause. The SDK cannot be found, so NuGet cannot evaluate the project at restore time.

@livarcocc would have better context here, but I am surprised you didn't see this error in addition to the NuGet error when restoring.

if you run msbuild /t:restore on the same project, what's the full output? Does it contain MSB4236 at all?

[mrabey]

if you run msbuild /t:restore on the same project, what's the full output? Does it contain MSB4236 at all?

Strangely, the build finishes successfully using VS2019 when I run msbuild /t:restore BtmCommunication.csproj. However, running the msbuild command in the original post still fails with the same error.

[mrabey]

I have been reading through #2532 to glean some information as to why VS2019 Developer Command Prompt wouldn't be able to find the SDK while the VS2017 Developer Command Prompt can.
I tried both resolutions:

• Ensure the Build Tools has installed ".NET Core Build Tools"
• Add a System PATH env variable pointing to C:\Program Files\dotnet

Both are done on our CI Build Agent. However, the error still occurs trying to discover the SDK when attempting to run that preprocess target on the project in question.

I have also created a VM that mirrors our Build Agent environment and turned off network capabilities. When I run the msbuild command in the original post, the build fails with the output:

(_FilterRestoreGraphProjectInputItems target) -> C:\<SolutionName>\Extension\Communication\BtmCommunication\BtmCommunication.csproj: warning NU1503: Skipping restore for project 'C:\<SolutionName>\Extension\Communication\BtmCommunication\BtmCommunication.csproj'. The project file may be invalid or missing targets required for restore. [C:\<SolutionName>\Extension\<SolutionName>.sln]

which seems to confirm that the underlying issue could be related to the failure of the VS2019 Developer Command Prompt from locating the .NET SDK.

[mrabey]

@nkolev92
So, we've found a temporary solution, but it feels rather hackish that solves a symptom of the problem rather than solving the underlying issue: the fact that the VS2019 Developer Command Prompt cannot locate the SDK at C:\Program Files\dotnet\sdk\.

Our solution:
Create a new environment variable for our TeamCity project MSBuildSDKsPath and set it equal to C:\Program Files\sdk\2.2.204\Sdks. This allows the SDK to be properly discovered and the restore succeeds without warnings or failures. However, this would require us to update the environment variable any time we update to a newer SDK version. I'd like to avoid tribal knowledge at all costs.

I'm still curious why the VS2019 Developer Command Prompt cannot locate the SDK properly. From all that I've read, you don't need to create an MSBuildSDKsPath environment variable as the build tools should be able to infer the directory path based on the provided information in the csproj file. Any ideas why it couldn't locate this SDK file?

[livarcocc]

Adding @nguerrera to help investigate why msbuild is failing to find the SDK when being used from the VS 2019 command prompt.

A couple of question do come to mind:

1. Do you have a global.json anywhere in the folder structure of your solution/projects?
2. What is your VS version and what versions of the SDK do you have installed on the machine?

[mrabey]

Do you have a global.json anywhere in the folder structure of your solution/projects?

No where in the folder structure exists a global.json file from my findings.

What is your VS version...?

We've only installed the Build Tools for both VS2017 and VS2019. I don't exactly know where to find the version numbers for them. The only place I've been able to find any form of version number is when running the Developer Command Prompts for both 2017 and 2019. I'll give you those, but let me know if those are not what you're looking for.

• VS2017: v15.9.12
• VS2019: v16.1.3

Following information from #2532, I ensured that the .NET Core Build Tools option was selected in the VS2019 Build Tools installer.

...what versions of the SDK do you have installed on the machine?

The versions of the SDK we have installed at C:\Program Files\dotnet\sdk\ are as follows:

• 2.1.504
• 2.1.507
• 2.2.204

There is also a NuGetFallbackFolder containing a plethora of DOTNETSENTINEL files that seem to mimic version numbers.

[livarcocc]

Ok, I see what is going on now. For 16.1, which is your version of VS 2019, you need to have at least 2.1.700 (for .NET Core 2.1) and 2.2.300 (for .NE Core 2.2).

This is needed because those are the versions of the .NET Core SDK that carry matching VS components like NuGet, MSBuild, Roslyn etc. As the SDK can require features from those components, we need to enforce a certain minimum version.

Are you manually uninstalling an SDK? If you are selecting the build tools sku with the .NET Core components selected, I would expect the right SDKs to be installed for you by VS.

[mrabey]

That's good to know. I'll give that a go and see if the problem is resolved.

Are you manually uninstalling an SDK?

I wasn't involved with the initial setup of our virtual machines running on our Build Agents so I'm unsure whether they were installed with the Build Tools installer or manually installed separately. I'll pass this information along to the person that usually deals with TeamCity maintenance.

Thank you.

[mrabey]

Closing the issue as it has been resolved. Thank you for the information regarding the version mismatch of the .NET Core SDK. That was the solution needed to get the NuGet restoration functioning on the CI server.

[mrabey]

As a side note for those that may look at this in the future, manually installing the .NET Core SDK 2.2.300 did not solve the issue. It seems that the Build Tools installer, choosing the ".NET Core Build Tools" workload option, does some other setup that points to the installation directory (C:\Program Files\dotnet\sdk\2.2.300) that the .NET Core SDK 2.2.300 installer does not.