Cannot use file stream for [*.deps.json]: No such file or directory

[mikkeljohnsen]

Description

Why is a published app with

<GenerateDependencyFile>false</GenerateDependencyFile>
<GenerateRuntimeConfigurationFiles>false</GenerateRuntimeConfigurationFiles>

Still trying to locate the "*.deps.json" ?

How do I turn this warning off ?

Reproduction Steps

dotnet publish -c Release -r linux-x64 --self-contained
a exe file with:

<GenerateDependencyFile>false</GenerateDependencyFile>
<GenerateRuntimeConfigurationFiles>false</GenerateRuntimeConfigurationFiles>

Expected behavior

Running with out warning about "*.deps.json"

Actual behavior

Cannot use file stream for [*.deps.json]: No such file or directory

Regression?

No

Known Workarounds

Create the "*.deps.json"

Configuration

No response

Other information

No response

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Why is a published app with

<GenerateDependencyFile>false</GenerateDependencyFile>
<GenerateRuntimeConfigurationFiles>false</GenerateRuntimeConfigurationFiles>

Still trying to locate the "*.deps.json" ?

Who do I turn this warning off ?

Reproduction Steps

dotnet publish -c Release -r linux-x64 --self-contained
a exe file with:

<GenerateDependencyFile>false</GenerateDependencyFile>
<GenerateRuntimeConfigurationFiles>false</GenerateRuntimeConfigurationFiles>

Expected behavior

Running with out warning about "*.deps.json"

Actual behavior

Cannot use file stream for [*.deps.json]: No such file or directory

Regression?

No

Known Workarounds

Create the "*.deps.json"

Configuration

No response

Other information

No response

Author:	mikkeljohnsen
Assignees:	-
Labels:	area-Host, untriaged
Milestone:	-

[vitek-karas]

I don't think there's a way to turn it off. It's a simple bug, the host should not write this out.

Just curious: Why do you remove the configuration files. Especially the .runtimeconfig.json, that is not a supported scenario to run without one (it will do weird things at times).
Also curious - how did you found out about these settings? (I want to make sure that whichever doc has them clearly states that these should NOT be turned off for applications).

[mikkeljohnsen]

I do not understand the reason for these files. It is all encoded in the platform exe file, what dependencies is needed. And the DLL files also contains references to needed files.

On Unix/Linux all files have a place, like "/usr/bin" and "/usr/lib", they are not just placed in a "C:\Program Files\name" folder. So want to keep the disk clean and not have unnecessary files around.

Is it really the meaning that for all "exe" files in "/usr/bin/" there should be a ".runtimeconfig.json" and a ".deps.json". That makes no sense.

I'm not sure where I found the configs.

Could the bug be fixed ?

[mikkeljohnsen]

I know that most apps is in containers or otherwise contained, so it really do not matter if these files are they.

But I do not understand why it is not just encoded in the binary files. Which I believe it is when using "dotnet publish"

[vitek-karas]

You can create a single executable which "bundles" everything: https://docs.microsoft.com/en-us/dotnet/core/deploying/single-file
If your concern is "random files on disk", then I think the single-file feature is the right solution for you.

And yes - dotnet build doesn't do this, but it's kind of by design - "build" is for inner dev loop, so tries to be fast. "publish" is for producing the bits which will be deployed.

[mikkeljohnsen]

Yes, but I have 20+ server jobs, and don't want to embed all dependencies in all server jobs.

[vitek-karas]

Do I understand it correctly that you:

• Have multiple executables which are all located in the same directory
• They use similar set of dependent assemblies
• You want to share the dependencies
• The way you do that is to build each separately and then ... copy them over to the same directory?
• How do you resolve potential conflicts in dependencies (version conflicts)?

You could selectively exclude assemblies from the bundle: https://docs.microsoft.com/en-us/dotnet/core/deploying/single-file#exclude-files-from-being-embedded

[mikkeljohnsen]

I have made a "Server.sln" including the server jobs. Then I just do

dotnet publish --nologo -c Linux Server.sln -p:NoWarn='"0649;0612;0168;0472"' -p:PublishProfile=Linux -p:ShouldUnsetParentConfigurationAndPlatform=false -r linux-x64 --self-contained -o app

I solve conflicts in dependencies, by not using NuGet and have made all dependencies as "RPM" packages, so I'm certain, that a new dependency is not pulled in. And I control the versions of the packages. (I'm not a fan of NuGet for large projects)

I should properly use "--no-self-contained" and then just install the "dotnet" in the container image. But then "*.runtimeconfig.json" is required, and I find that stupid. (But then again when it is inside a container it do not matter much).

[vitek-karas]

Thanks for the description of the scenario.

It's technically possible to publish a single-file where all assemblies are excluded, which would basically produce the exe with the config files bundled and nothing else. But it would have to be published as --no-self-contained as otherwise the exe would include the runtime itself.

We've been thinking about using basically the above technique to "hide" the config files, so it's good to know that there's interest in it.

[mikkeljohnsen]

It would be great to have a option to create a single "exe" (on Unix without .exe) file where the corresponding ".dll", ".runtimeconfig.json", "*.deps.json" was combined (excluding all other assemblies). That would make i more suitable on Unix, so it could be placed in "/usr/bin".

Not sure if now you can put the "exe" file in "/usr/bin" and the "dll" file in "/usr/lib" and make it work.

What is the idea behind having both an exe and dll file, and not a combined file ?

[vitek-karas]

The .dll is a managed assembly - the compiled C# code into IL. The .exe is just a small "shim" which finds and loads the .NET runtime and runs the .dll.

[richlander]

We do have an option for creating single file apps as well. It just isn't a default behavior.

[vitek-karas]

@richlander The feature ask here is basically "Framework dependent single-file with shared libraries". It's sort of possible with a little bit of hacking, but it's definitely not easy.

[wzchua]

From https://docs.microsoft.com/en-us/dotnet/core/deploying/single-file
it says single file works with framework dependent
it gives dotnet publish -r linux-x64 -p:PublishSingleFile=true --self-contained false as the example command

[vitek-karas]

@wzchua yes, that already works, but it will bundle all application assemblies. My understanding of the above is that the ask is to have a way to share some assemblies between the multiple apps. That is currently doable, but only through sort of hacks ->
"shared libraries" (that's what we've been calling this feature recently, ability to share assemblies between apps outside of the framework, in some easy to do way)

[mikkeljohnsen]

Yes I need away to use "ExcludeFromSingleFile" for all "ProjectReference" and "Reference", is there a way to do that ?

[iq2luc]

[...] That is currently doable, but only through sort of hacks -> "shared libraries" (that's what we've been calling this feature recently, ability to share assemblies between apps outside of the framework, in some easy to do way)

@vitek-karas Yes, saying that it would be super great without hacks would be an understatement from me.

I'm also in a similar situation where I need to deploy multiple executables (lots of small console based tools) that all depend on a set of shared libraries.

Regarding *.runtimeconfig.json, in my opinion, ideally the apphost should try to load it first from the assembly itself (maybe an option at the project level to automatically embedded it, or just use it if the user embedded it as a resource) and then - if not found in the assembly - try from the filesystem. Right now it just goes for the filesystem and that is not optimal (for every little executable another runtimeconfig.json, which in @mikkeljohnsen's case for example, it means polluting /usr/bin).
Still, it looks like there is a workaround for this problem: -p:PublishSingleFile=true --no-self-contained.

But the above workaround hits another problem: NuGet package references.
As @mikkeljohnsen pointed out already, I also consider NuGet inappropriate for all my use cases. In no way my statement is a denigration of NuGet, I understand it is great for lots of people and their use cases, it just doesn't work for me. One small step forward, in tackling our current problem, would be for ExcludeFromSingleFile to work on PackageReference similar to how it works on ProjectReference. Ideally ExcludeFromSingleFile would affect all implicit dependencies.
Or maybe some sort of one option to rule them all: combine apphost, *.runtimeconfig.json (and whatever any other jsons are required) and the main assembly into a single executable (with everything else similar to what happens when not publishing as a single file).

There are situations where I cannot create a framework-dependent deployment because I just cannot enforce the client to do a system wide install of the .NET framework. In this situation, the message Cannot use file stream for [...]: No such file or directory is especially annoying for console programs. In these particular use case, where I need a framework-independent deploy (with all required native and managed libraries in the same directory as the executables), I had to do an ultra-ugly dirty little hack by patching libhostfxr.so to remove that message (i.e. zeroing a byte; it still prints a new-line, but that is tolerable).
So even if only this little thing with the above message could be fixed, it will be greatly appreciated.
Probably a much better way in this case would be something like a custom apphost as mentioned in "Hosting managed apps" scenario (https://github.com/dotnet/runtime/blob/main/docs/design/features/native-hosting.md). I'll look more into it.

[vitek-karas]

The idea of embedding the config files (.runtimeconfig.json and .deps.json) into the exe "always" has been floating around for a while. I think we now have the tech to do this relatively cheaply and maybe even do that by default (with an opt-out). I can't promise when it will happen, but it's likely going to happen at some point. This is basically same ask as dotnet/sdk#18597.

The idea to be able to exclude entire packages from single-file makes sense. Please create an issue just for that in the dotnet/sdk repo.

Re excluding everything - it should be possible (didn't try this, but I know we did something similar for trimming for example) to add a custom target into your build which will automatically apply the ExcludeFromSingleFile to all assemblies, so you could do that and then just selectively allow it for the app assembly (and potentially others per your decision).

#69314 should fix the writing to stderr.

[iq2luc]

Thank you @vitek-karas, for the fix, for the hints and for the hope.
I created dotnet/sdk#25411.