Add support for banana linux

[tmds]

.NET can't source-build on a distro it doesn't know (that is: not yet in the rid graph).

This is an impediment when building .NET on new versions of an existing distro, or when building it on a new distro.

The build should handle the unknown rid appropriately.
Package maintainers shouldn't have to become rid graph maintainers.

Tasks:

• runtime: support building an unknown rid (Pass TargetRid and SourceBuildNonPortable to the native scripts runtime#74504)
• runtime: make the built runtime know about the rid (Ensure shared framework consumes source-built RID graph runtime#53550)
• update source-build/installer
• add test

@dotnet/source-build-contrib

[tmds]

@crummel @dagood we are often running into issues with newer versions of Fedora due to the rid being unknown.
The suggestion here is to change /etc/os-release as part of one of the ci builds to ensure source-build works on an unknown glibc linux-x64.
Can we do this?

cc @omajid

[adaggarwal]

@crummel Chris Rummel FTE @dagood Davis Goodin FTE we are often running into issues with newer versions of Fedora due to the rid being unknown.
The suggestion here is to change /etc/os-release as part of one of the ci builds to ensure source-build works on an unknown glibc linux-x64.
Can we do this?

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

[omajid]

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

I think dotnet-install.sh falls back to linux-x64 rid correctly.

For example, arch-x64 RID doesn't exist in the RID graph, but we get past dotnet-install.sh on Arch Linux just fine: #1310.

[dagood]

I think that's how dotnet-install.sh determines the os. Incase we replace it, we might need a file/placeholder that would contain the RID?

It seems better to me to look at it from a dev's point of view: what do I expect source-build to do when it can't recognize my OS? My understanding is that the point of adding banana (which I think is a really good idea) is to find situations like that.

For the prebuilt SDK, I imagine I'll normally need linux-x64 (glibc). Then I'd probably want a non-portable build with my new RID in the right place in the RID graph that allows me to restore Microsoft assets if necessary for cross-RID publish, etc. Specifying those each time I want to build seems reasonable, because my OS is unknown (not mentioned in the sources). (Edit: Sounds like I may only need to specify the RID I want, or not even that.)

[crummel]

I like this idea, it'll keep us on top of new versions and unknown distros. It does seem like we'll probably have to soft-patch the RID graph though, unless something in the runtime changes to allow arbitrary unknown RIDs. We can try this out and see what all breaks.

[dagood]

we'll probably have to soft-patch the RID graph

I thought we were already doing that with:

source-build/repos/corefx.proj

Lines 97 to 107 in be813df

	<Target Name="UpdateRuntimeGraph"
	BeforeTargets="Build"
	Condition="'$(_IsBootstrapping)' == 'true'">
	<PropertyGroup>
	<RuntimeJsonFile>$(ProjectDirectory)pkg/Microsoft.NETCore.Platforms/runtime.json</RuntimeJsonFile>
	</PropertyGroup>
	<Message Importance="High" Text="Adding rid, $(TargetRid), to $(RuntimeJsonFile)" />
	<AddRidToRuntimeJson RuntimeJson="$(RuntimeJsonFile)"
	Rid="$(TargetRid)-$(Platform)" />
	</Target>

[crummel]

Oh, I think IsBootstrapping went away when we switched from BuildTools to Arcade. That should be an easy fix though.

[tmds]

(Edit: Sounds like I may only need to specify the RID I want, or not even that.)

Yes. Source-build should figure out the target rid from /etc/os-release and assume a parent of linux-{arch} (on glibc Linux).
As optional parameters, the user may want to name the target rid (e.g. rhel.8 instead of rhel.8.1)
And specify a list of parent rids.

For this issue, I think we can focus on patching /etc/os-release as a pre-build CI step. And verify we get an sdk for the patched os-release rid.

[tmds]

I think we can focus on patching /etc/os-release as a pre-build CI step. And verify we get an sdk for the patched os-release rid.

@dagood @crummel can we do this?

[dagood]

The approach sounds good to me, if that's the question. I don't know where it sits in the backlog... seems weird that @leecow is assigned and it's in a closed ZenHub epic. @crummel do you know the background, should it be unassigned to retriage or something like that?

[crummel]

Yeah, we'll assign this to someone in triage. Thanks for the discussion around how to do this everyone!

[tmds]

@crummel @dagood having this CI build will catch rid issues that are now hitting us post-release.

[tmds]

And considering #1428, we should build the source-tarball on banana Linux, and verify it builds on pineapple Linux.

[dagood]

Issues have come up with versions--stuff like banana-x64/banana.2-x64 vs. banana.2.9-x64 failing on this regex:

source-build/tools-local/tasks/Microsoft.DotNet.SourceBuild.Tasks.XPlat/PublishCoreSetupBinaries.cs

Lines 26 to 31 in 1eba1a6

	// This regular expression is crafted to extract the semver component from the artifacts
	// that Core-Setup produces. They have filenames like this (note the two formats!)
	// dotnet-runtime-rhel.7-x64.2.0.0-preview2-25401-9.tar.gz
	// dotnet-runtime-2.0.0-preview2-25401-9-rhel.7-x64.tar.gz
	// the "semver" capture would be 2.0.0-preview2-25401-9 in this case.
	protected virtual string VersionMatchRegex => @"(\.|-)(?'semver'[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9]+(-|\.)[0-9]+(-|\.)[0-9]+)?)";

Concretely, @lupin-de-mid hit this issue with astra.1.6-x64. Also, the SDK seemed to parse os-release as astra.16-x64 at runtime, not sure if this is related (or a problem).(edit: see below clarification) Context starting here: https://gitter.im/dotnet/source-build?at=5f21801125cae90e6d8afb48

I think this problem will repro on Ubuntu too (e.g. ubuntu.20.04-x64).

[lupin-de-mid]

Concretely, @lupin-de-mid hit this issue with astra.1.6-x64. Also, the SDK seemed to parse os-release as astra.16-x64 at runtime,

I workaround problem by changing /etc/os-release from VERSION=1.6 to VERSION=16. So RID in dotnet --info output is defined at compile time

[dagood]

I think this problem will repro on Ubuntu too (e.g. ubuntu.20.04-x64).

We actually run Ubuntu 18.04 CI, which doesn't hit this, so I'm wrong here. Will need to look deeper.

[dagood]

Something to consider for this issue is patching up the RID graph in the prebuilt/tooling SDK that we download in the tarball production build. It isn't necessary in all cases (the catch-all linux-x64 may work) but it's probably necessary for cases like alpine-x64 / linux-musl-x64. See #1868. If we don't do this as part of resolving this issue, we should probably open a new issue to track support for that.

[tmds]

RHEL 9 is Banana Linux: we can't build .NET Core on it until the rids have rippled through :/

[omajid]

Also starting to see issues on the recently-forked Fedora 35 :(

[tmds]

Now that dotnet/runtime#50818 is in.

Can we add a CI job to source-build that verifies we can build a source-tarball on banana Linux, and use it to build .NET on pineapple Linux?

The CI job can write to /etc/os-release to change the distro id.

@MichaelSimons @crummel @dseefeld can you take a look into this?

[tmds]

cc @ericstj

[tmds]

@MichaelSimons @crummel @dseefeld friendly reminder.

It would be great if we can close the circle by adding a CI job to verify source-build is no longer blocked by unknown rids.

[crummel]

I'm looking into this now, I'll hopefully have some x64 RHEL machines soon so I'm planning on making CI changes for that anyway.

[tmds]

I've updated the top-level comment with a task list.

cc @MichaelSimons @crummel @omajid @ericstj @richlander

[tmds]

This is fixed as part of dotnet/installer#14549.

We should add a CI test for it. Preferably it does two builds where the second build consumes the output of the first. For the two builds we should use two different unknown rids (e.g. pinapple.14 and banana.18).

To control the rids, we can make /etc/os-release writable in the container image we intend to use for the test.
We don't need to apply it to all images, as this is a functional test for the unknown rid scenario.
It would be nice if we have both a glibc-based and a musl-based image to test against.

We can update @omajid's PR (dotnet/dotnet-buildtools-prereqs-docker#679) to enable the test scenario.

[tmds]

This is working. Closing old issue.