Unnecessary dlls copied to output folder

[jaredpar]

Consider the following solution where all projects use the new SDK (VS 15.6.7):

• Util: a DLL project targeting netstandard1.3
• MyApp: an EXE project targeting net46 and referencing Util

This projects have no other than the project template (hello world). I would expect building such a solution would put the minimum set of DLLs into the output folder of MyApp. Yet when I build I get the following set:

• Microsoft.Win32.Primitives.dll
• System.AppContext.dll
• System.Console.dll
• System.Diagnostics.DiagnosticSource.dll
• System.Globalization.Calendars.dll
• System.IO.Compression.dll
• System.IO.Compression.ZipFile.dll
• System.IO.FileSystem.dll
• System.IO.FileSystem.Primitives.dll
• System.Net.Http.dll
• System.Net.Sockets.dll
• System.Runtime.InteropServices.RuntimeInformation.dll
• System.Security.Cryptography.Algorithms.dll
• System.Security.Cryptography.Encoding.dll
• System.Security.Cryptography.Primitives.dll
• System.Security.Cryptography.X509Certificates.dll
• System.Xml.ReaderWriter.dll

Almost none of these are necessary to run my application yet they are included in my project output. How am I supposed to know what DLLs are and aren't necessary for execution? This is important because I need to construct a minimal NuGet package.

Adding System.Net.Http to a "hello world" app is fairly suspicious.

[nguerrera]

cc @ericstj @AlexGhiondea @joperezr

[dsplaisted]

These are facades coming from NuGet packages that are newer than the ones included in-box in the framework, so they are deployed locally with your app. For example, this is what's happening with System.Net.Http.dll:

Encountered conflict between 'Reference:C:\Users\daplaist.nuget\packages\system.net.http\4.3.0\ref\net46\System.Net.Http.dll' and 'Platform:System.Net.Http.dll'. Choosing 'Reference:C:\Users\daplaist.nuget\packages\system.net.http\4.3.0\ref\net46\System.Net.Http.dll' because AssemblyVersion '4.1.1.0' is greater than '4.0.0.0'.

The folks Nick mentioned can comment, but I think the basic issue is that these facades are needed to support the APIs of .NET Standard 1.3. Determining whether they are needed for a specific project would involve inspecting the references of the built assembly and seeing what contracts are actually referenced.

[AlexGhiondea]

You might be running into this known issue.

[ericstj]

That's just NETStandard.Library package flowing across project reference. If you really want to reduce those references you can try the trimmer: https://github.com/dotnet/standard/blob/master/Microsoft.Packaging.Tools.Trimming/docs/trimming.md

@AlexGhiondea not that issue, this is net46 / netstandard1.x. As @dsplaisted mentioned those DLLs are necessary to make up all of netstandard1.x on net46.

[jaredpar]

These are facades coming from NuGet packages that are newer than the ones included in-box in the framework, so they are deployed locally with your app

These are not all facades though. System.Net.Http is a full implementation assembly. The tools are now suggestion I need to package this with my tool even though it makes no HTTP requests whatsoever.

That's just NETStandard.Library package flowing across project reference.

This is pretty unfortunate. It means that I need to either:

1. Ship a number of implementation assemblies, that my app in no way uses.
2. Attempt to be smarter than the SDK and trim out references that I believe aren't used.

We've tried 2 a number of times and it pretty much always blows up in our face. But simply accepting the output of the tooling here as is significantly increases the size of our package (for no reason).

[nguerrera]

What changed for this to impact Roslyn from one day to the next? Was it that you started using the implicit framework references to NETStandard.Library? If so, maybe it is worth going back to granular package refs.

[jaredpar]

What changed for this to impact Roslyn from one day to the next?

This inability to understand what is needed in order for Roslyn to deploy correctly has been an ongoing problem since we moved to NetStandard. Recently there have been two changes that caused us headaches:

1. Moving to implicit framework references
2. Taking a dependency on System.Threading.Tasks.Extensions

Each of these broke our infrastructure in various ways. I can go into the details of how but I don't think that's particularly interesting here. The core problem for us is: how can I create a NuGet package that has the minimal correct dependency set for our compilers? This issue was meant at helping with the "minimal" part of that problem but it seems like the answer is there is no way to do that. At this point I'm focusing on correct which itself is a huge task.

If so, maybe it is worth going back to granular package refs.

That won't really help us here. Even with package references we still have the fundamental problem of what DLLs do we need to deploy into our NuPkg?

Let me elaborate on that a bit. For our NuGet package we essentially do the following:

1. Build three to four exes for desktop
2. Create a NuGet package that has a single directory that contains the combined output of these exes.

Building a NuGet package, which is correct by construction, for this scenario is a vexing problem. I've tried a number of approaches and they've all failed.

[dsplaisted]

@jaredpar Did you take a look at whether the trimmer Eric linked to could help?

[nguerrera]

Even with package references we still have the fundamental problem of what DLLs do we need to deploy into our NuPkg?

There are some annoying cases with netstandard >= 1.5 and netfx < 4.7.2, but netfx any referencing netstandard 1.3, I don't see why you still have the problem if you go back to individual package references. Let the tooling pick, and it will not pick System.Net.Http if there is no System.Net.Http in the package graph.

[jaredpar]

@dsplaisted not yet. At the moment I'm still working on the correctness issues here. Those are more pressing and there just doesn't seem to be an easy solution here. Once that's done I may come back and try that.

Also I may just leave it as is. Constructing a correct NuGet package has consumed an amount of my time, and others, that is quite frankly unreasonable at this point. Once we get to a working solution I'm going to be loath to change it again for fear of creating yet another problem for us.

[jaredpar]

@nguerrera

I don't see why you still have the problem if you go back to individual package references.

Imagine you have the following directories in your build output:

• csc\net46
• vbc\net46

How would you construct a NuGet package that properly merges these two directories into a single one?

This is now the core problem I'm facing. PackageReference or implicit framework references don't make this problem any easier or harder.

Sorry, I know this has drifted a bit from the original issue. Devil is always in the detail.

[ericstj]

How would you construct a NuGet package that properly merges these two directories into a single one?

If both target the same TFM and use the same version of packages you can just overlay them. If you want something better, packagereference actually enables that. You create a project that references the two projects and publish it. That project would have the job of reconciling the dependencies of the two, and with package-references being transitive it now has all the info it needs to do so.

[jaredpar]

If both target the same TFM and use the same version of packages you can just overlay them.

How to overlay them though? My first attempt was the following

<file src="csc\net46\System.*" dest="tools" ?>
<file src="vbc\net46\System.*" dest="tools" ?>

That results in every single System file being included twice in the NuPkg file. The package is overall completely invalid and crashes on almost every zip tool

The only solution we could come up with was to write a tool that looks at the output of all our tools, takes the union of the DLLs and then ensures our NuPkg file has that collection. It's a good chunk of work and makes me feel like we're just missing a scenario here. Possibly we're the only team that does this.

[ericstj]

The tool to do a union could just be a project and projectreferences. It depends on how you're building those projects but it can work. Here's a sample: ericstj/scratch@03e12a0.

If this doesn't work for you, just create a target in your build that copies things into the right layout then have your nuspec use a wildcard on that: effectively lifting the wildcard out of the nuspec (which isn't handling duplicates correctly) into msbuild (which can handle the duplicates correctly).

That's definitely a bug that nuget produces a package from a nuspec with duplicates. It should error out. I've filed an issue on that: NuGet/Home#6941.

[jaredpar]

@ericstj

The tool to do a union could just be a project and projectreferences.

That only works when targeting desktop frameworks. When targeting netcoreapp the same behavior is not maintained when combined with publish. This was brought up in #1675 and seems to be "by design-ish".

Thanks for filing the NuGet bug.

[ericstj]

I see. I guess you could manually copy the missing deps/runtimeconfig. The desktop case actually has a similar problem with app.config. If you had different versions merged together you might have the wrong binding redirects. It feels to me like this is an aspect of your deployment that you might just want to add tests to validate.

[jaredpar]

It turns out this layout is incorrect by default. Consider that:

• System.Net.Http references System.Diagnostics.DiagnosticSource at 4.0.0.0
• System.Diagnostics.DiagnosticSource actually has version 4.0.1.0

Unsure why this is the case.

[nguerrera]

Unsure why this is the case

Not sure if you mean:

1. Why did the sdk (since that's where this bug is filed) copy these dlls, or
2. Why are the inputs to the SDK defined such that this is the layout.

For (1), that's what the package references imply: NETStandard.Library pulls S.N.H and S.D.DS, both at package version 4.3.0, and the DLLs in those packages have this versioning.

For (2), it's a question for corefx as to why the DLLs in these 4.3.0 packages have this versioning. I vaguely recall the answer from prior discussions, but I will let @ericstj or @weshaggard answer authoritatively.

[jaredpar]

I think we've discussed (1) a bit and I understand why that is happening. I'm more interested in (2) with the most recent comment. The outputs here don't agree on their own versions.

[ericstj]

There are many reasons for this sort of mismatch. One common reason is this: https://github.com/dotnet/corefx/issues/30106#issuecomment-395204921. Though there are many others.

Back in 1.1 we were actually building most components against the last stable dependenices unless they required to consume the live bits. This was actually desireable at that time because we were hoping to have more granular servicing. Priorities changed and folks wanted to reduce combinatorix. We then listed the dependencies as latest (due to package unification requests to reduce the size of the NuGet graph). That's the specific reason for the mismatch you pointed out. This all changed in 2.0 to make this particular scenario less likely.

That said, the mismatch isn't incorrect; a newer implementation will satisfy the lower reference, with bindingRedirects on desktop. You'll see this a lot across the packages, even in the latest builds for reasons like the one I linked.

[nguerrera]

One issue I'm seeing, though, is that no binding redirect is getting generated. I haven't dug in to the logs yet to be sure, but it appears that S.N.Http -> S.D.DiagnosticSource dependency has Exclude=Compile so RAR doesn't see a conflict. This is a design problem with RAR being the one to generate binding redirects: it doesn't see the runtime graph, which is what binding redirects should really be based on. IIRC, this was worked around in the extensions case with netfx.forceconflicts.dll, but no such mechanism hits here. So it does seem that at present, out of the box, a console app would have to add a manual binding redirect to go down certain code paths through netstandard.library from netfx. If I have that right, I think that part is definitely incorrect.

[ericstj]

Indeed if that is the case it is a problem. For desktop we go out of our way to make sure that assembly versions seen by RAR are the same as runtime, but we may have missed the exclude=compile case. /cc @joperezr

[minecraftchest1]

Dotnet core 3.1.201 when running dotnet publish -o ~/dotnet/projects/bin/helloworld --self-contained=true -r linux-x64 on the following code, after running dotnet build gives me the entire .net core runtime in the output directory. This is annoying me because it is causing the application to take up way more space on my drive then it should. May I get a suggestion on what I can do to get a minimal runtime? Forgive me if I am hijacking this thread.

Program.cs
`using System;

namespace helloworld
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hello World!");
}
}
}`

Directory contents
createdump helloworld helloworld.deps.json helloworld.dll helloworld.pdb helloworld.runtimeconfig.json libclrjit.so libcoreclr.so libcoreclrtraceptprovider.so libdbgshim.so libhostfxr.so libhostpolicy.so libmscordaccore.so libmscordbi.so Microsoft.CSharp.dll Microsoft.VisualBasic.Core.dll Microsoft.VisualBasic.dll Microsoft.Win32.Primitives.dll Microsoft.Win32.Registry.dll mscorlib.dll netstandard.dll SOS_README.md System.AppContext.dll System.Buffers.dll System.Collections.Concurrent.dll System.Collections.dll System.Collections.Immutable.dll System.Collections.NonGeneric.dll System.Collections.Specialized.dll System.ComponentModel.Annotations.dll System.ComponentModel.DataAnnotations.dll System.ComponentModel.dll System.ComponentModel.EventBasedAsync.dll System.ComponentModel.Primitives.dll System.ComponentModel.TypeConverter.dll System.Configuration.dll System.Console.dll System.Core.dll System.Data.Common.dll System.Data.DataSetExtensions.dll System.Data.dll System.Diagnostics.Contracts.dll System.Diagnostics.Debug.dll System.Diagnostics.DiagnosticSource.dll System.Diagnostics.FileVersionInfo.dll System.Diagnostics.Process.dll System.Diagnostics.StackTrace.dll System.Diagnostics.TextWriterTraceListener.dll System.Diagnostics.Tools.dll System.Diagnostics.TraceSource.dll System.Diagnostics.Tracing.dll System.dll System.Drawing.dll System.Drawing.Primitives.dll System.Dynamic.Runtime.dll System.Globalization.Calendars.dll System.Globalization.dll System.Globalization.Extensions.dll System.Globalization.Native.so System.IO.Compression.Brotli.dll System.IO.Compression.dll System.IO.Compression.FileSystem.dll System.IO.Compression.Native.a System.IO.Compression.Native.so System.IO.Compression.ZipFile.dll System.IO.dll System.IO.FileSystem.AccessControl.dll System.IO.FileSystem.dll System.IO.FileSystem.DriveInfo.dll System.IO.FileSystem.Primitives.dll System.IO.FileSystem.Watcher.dll System.IO.IsolatedStorage.dll System.IO.MemoryMappedFiles.dll System.IO.Pipes.AccessControl.dll System.IO.Pipes.dll System.IO.UnmanagedMemoryStream.dll System.Linq.dll System.Linq.Expressions.dll System.Linq.Parallel.dll System.Linq.Queryable.dll System.Memory.dll System.Native.a System.Native.so System.Net.dll System.Net.Http.dll System.Net.HttpListener.dll System.Net.Http.Native.a System.Net.Http.Native.so System.Net.Mail.dll System.Net.NameResolution.dll System.Net.NetworkInformation.dll System.Net.Ping.dll System.Net.Primitives.dll System.Net.Requests.dll System.Net.Security.dll System.Net.Security.Native.a System.Net.Security.Native.so System.Net.ServicePoint.dll System.Net.Sockets.dll System.Net.WebClient.dll System.Net.WebHeaderCollection.dll System.Net.WebProxy.dll System.Net.WebSockets.Client.dll System.Net.WebSockets.dll System.Numerics.dll System.Numerics.Vectors.dll System.ObjectModel.dll System.Private.CoreLib.dll System.Private.DataContractSerialization.dll System.Private.Uri.dll System.Private.Xml.dll System.Private.Xml.Linq.dll System.Reflection.DispatchProxy.dll System.Reflection.dll System.Reflection.Emit.dll System.Reflection.Emit.ILGeneration.dll System.Reflection.Emit.Lightweight.dll System.Reflection.Extensions.dll System.Reflection.Metadata.dll System.Reflection.Primitives.dll System.Reflection.TypeExtensions.dll System.Resources.Reader.dll System.Resources.ResourceManager.dll System.Resources.Writer.dll System.Runtime.CompilerServices.Unsafe.dll System.Runtime.CompilerServices.VisualC.dll System.Runtime.dll System.Runtime.Extensions.dll System.Runtime.Handles.dll System.Runtime.InteropServices.dll System.Runtime.InteropServices.RuntimeInformation.dll System.Runtime.InteropServices.WindowsRuntime.dll System.Runtime.Intrinsics.dll System.Runtime.Loader.dll System.Runtime.Numerics.dll System.Runtime.Serialization.dll System.Runtime.Serialization.Formatters.dll System.Runtime.Serialization.Json.dll System.Runtime.Serialization.Primitives.dll System.Runtime.Serialization.Xml.dll System.Runtime.WindowsRuntime.dll System.Runtime.WindowsRuntime.UI.Xaml.dll System.Security.AccessControl.dll System.Security.Claims.dll System.Security.Cryptography.Algorithms.dll System.Security.Cryptography.Cng.dll System.Security.Cryptography.Csp.dll System.Security.Cryptography.Encoding.dll System.Security.Cryptography.Native.OpenSsl.a System.Security.Cryptography.Native.OpenSsl.so System.Security.Cryptography.OpenSsl.dll System.Security.Cryptography.Primitives.dll System.Security.Cryptography.X509Certificates.dll System.Security.dll System.Security.Principal.dll System.Security.Principal.Windows.dll System.Security.SecureString.dll System.ServiceModel.Web.dll System.ServiceProcess.dll System.Text.Encoding.CodePages.dll System.Text.Encoding.dll System.Text.Encoding.Extensions.dll System.Text.Encodings.Web.dll System.Text.Json.dll System.Text.RegularExpressions.dll System.Threading.Channels.dll System.Threading.dll System.Threading.Overlapped.dll System.Threading.Tasks.Dataflow.dll System.Threading.Tasks.dll System.Threading.Tasks.Extensions.dll System.Threading.Tasks.Parallel.dll System.Threading.Thread.dll System.Threading.ThreadPool.dll System.Threading.Timer.dll System.Transactions.dll System.Transactions.Local.dll System.ValueTuple.dll System.Web.dll System.Web.HttpUtility.dll System.Windows.dll System.Xml.dll System.Xml.Linq.dll System.Xml.ReaderWriter.dll System.Xml.Serialization.dll System.Xml.XDocument.dll System.Xml.XmlDocument.dll System.Xml.XmlSerializer.dll System.Xml.XPath.dll System.Xml.XPath.XDocument.dll WindowsBase.dll

[joperezr]

Not sure what you are trying to do, but that is what self-contained means. It means that everything you need in order to run your app in a linux-x64 Machine will be placed on your output folder, which includes the whole .net runtime. If what you want instead is just your app code to be published, try running dotnet publish Which will produce a framework dependent app instead, meaning it will depend on you installing dotnet core on the target machine before trying to run your app.

[minecraftchest1]

I want it to be self contained. I just don't want it to include unnecessary assemblies. I can manually remove the unnecessary assemblies, but that takes a while and is error prone. Why should a console app require System.Drawing and System.Web? Just an example.

[joperezr]

It probably doesn’t but they are part of the shared framework and self contained apps by default get all the shared framework. Sounds like what you want is to trim your output which we do support, you can get all info from here https://docs.microsoft.com/en-us/dotnet/core/deploying/trim-self-contained

But basically just add a -p:PublishedTrimmed=True which will do a tree shaking in your app to remove what you don’t need. Also, If I remember correctly, it will do it not only at an assembly-level, but go one step further and do it at a method-level instead.

[minecraftchest1]

Deleted previous post.

[minecraftchest1]

Running dotnet publish -o ~/dotnet/projects/bin/helloworld --self-contained=true -r linux-x64 p:PublishTrimmed=true on the same project above, gives the the following error:

`Microsoft (R) Build Engine version 16.5.0+d4cbfca49 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: p:PublishTrimmed=true`

Edit:
Running it without p:PublishTrimmed=true works without errors.

[joperezr]

looks like you missed a dash before the p:. The full command you have to run is:

dotnet publish -o ~/dotnet/projects/bin/helloworld -r linux-x64 -p:PublishTrimmed=true

NIT: you don't need to specify --self-contained because by passing in a -r linux-x64 we will by default assume you wanted it self-contained.

[minecraftchest1]

@joperezr, It works just fine now. Thanks.

[minecraftchest1]

WOW! The output is much smaller. Thanks.