Owner Matt Thalman
There are a set of .NET-related customer workflows that require the use of platform libraries contained within the operating system. These are divided into two categories:
- Execution of a .NET process: Runtime components defer some of their logic to platform libraries. Examples include ICU libraries for globalization support, GNU C Library, and others.
- Building .NET from source code: The toolchain required to build the .NET source consists of platform dependency prerequisites on the build machine. Examples include CMake, curl, Python, and others.
The problem is that these dependencies vary based on usage (i.e. a dependency may only be needed in a particular environment or scenario), they change as the product and the dependent ecosystem evolve, and the documentation is often out-of-date with the product. This leaves the community with few options other than an iterative and error-prone approach of attempting to execute their workflow and finding what breaks due to missing dependencies (e.g. dotnet/runtime#36888 (comment), dotnet/dotnet-docker#1767). Even worse is when there are cases where the community has discovered incompatible dependencies in environments that are supported (e.g. dotnet/runtime#38755).
We need to provide a better user experience for those developers that continually encounter issues of missing platform dependencies in their systems. By providing .NET developers access to self-service tools that consume an up-to-date and comprehensive model of platform dependency information, we can help these developers to accurately configure their systems.
This design provides a uniform description of all platform dependencies for .NET. It collects the dependency data into a machine-readable format that serves as the complete model for reasoning about .NET's platform dependencies.
In this context, platform dependency refers to a pre-existing artifact in the operating environment that satisfies the usage of a .NET component. For example, in order to run a .NET application in a Linux environment, the GNU C Library must exist.
By providing a means for .NET contributors to describe the product's platform dependencies -- both runtime and source build toolchain -- in a common model format, it can be used to satisfy other downstream workflows that require the data. Contributors also need to understand what their responsibilities are in order to maintain the model and avoid having its contents go stale. This design provides a developer workflow that describes what actions need to be taken by contributors in a variety of scenarios.
- Runtime: In this context, runtime refers to the execution of any of the .NET shared frameworks and libraries. The term is meant to be broad and not be limited to the ".NET runtime" but also include the execution of .NET SDK logic. In other words, it represents the runtime lifecycle phase of .NET functionality, not the .NET runtime system.
- Toolchain: The set of tools and libraries that are needed in order to build .NET from source.
A .NET product area owner has files that need to list out the Linux packages that .NET depends on. Examples of such product areas include the following:
- Docker: A Dockerfile that installs dependent packages
- Linux packages: A deb package that lists the packages it depends on.
- Docs: User documentation that lists the Linux packages that are necessary to have installed for .NET.
They want to ensure this list is up-to-date with .NET's requirements. They author a tool which reads from .NET's platform dependency model and compares it against their package list to ensure it aligns for a given platform and version.
A security incident has been disclosed in a Linux package that could potentially affect a platform dependency that .NET has. The response team needs to have an understanding of what components of .NET have a dependency on the Linux package so that the appropriate mitigation can be implemented. They search through .NET's platform dependency model to determine which .NET components are affected and on which platforms.
The .NET team wants to provide users with self-service tools so that they can know, a priori, what platform dependencies their specific application has. This allows them to configure the systems that will host their application with confidence. The developers creating these tools need a single source of truth from which to derive .NET's platform dependencies. They author the tools to read from the platform dependency model to get this information.
A .NET contributor makes a change to the NativeMethods.cs file in their project. They create a PR with their changes. A bot alerts the contributor to the potential need to update .NET's platform dependency model in response to the change in the NativeMethods.cs file. The bot alerts them by adding a special label to the PR and a comment that provides instructions on next steps they should follow.
Debian 9 goes EOL and a maintainer of the platform dependency model needs to clean up the model to remove references to Debian 9 since those references are now obsolete. They run a CLI command to update the model automatically:
dotnet-deps platform remove debian.9- Common schema capable of describing both runtime and toolchain dependencies.
- Model that limits repetition to make maintenance easier.
- File format that is machine-readable to allow for automated transformation into other output formats.
- Ability to describe:
- Dependencies for multiple platform types (Linux, Windows, MacOS)
- Dependencies from various granularities of .NET components such as shared frameworks and NuGet packages.
- Dependencies that exist outside of canonical scenarios, such as opting into specific features like diagnostics.
- Workflow that .NET contributors can easily follow for maintaining the model.
- Any dependency that is included in the publishing of a project is outside the scope of this design. Native files that are contained within NuGet packages are an example of this. A .NET application's dependency on a .NET package, and any assets contained in those packages (managed, native, or otherwise), is explicitly included in the deployment of the application itself. Therefore, the operating environment is not required to be pre-configured with those specific assets; it'll get them naturally through the deployment of the application. Keep in mind that a NuGet package may have its own platform dependency (e.g. a Linux package) that is not physically contained in the NuGet package; such a platform dependency would be in scope with this design. The concern addressed here is solely focused on what the operating environment must be pre-configured to contain in order to operate on .NET scenarios.
- ASP.NET Core
- CoreCLR
- Acquisition & Deployment
- Docs
- Libraries
- Mono
- Release
- SDK
using System.Collections.Generic;
namespace Microsoft.DotNet.Dependencies.Platform
{
// Root of the model
public class PlatformDependencyModel
{
// Major.Minor.Patch version of .NET release the dependencies are associated with
public string DotnetReleaseVersion { get; set; }
// Set of available dependency usages that can be referenced by a platform dependency.
// A dependency usage is a descriptor of how a dependency is used by a component (e.g. default, diagnostics, localization).
// This maps the name of the name of the usage to its description.
public IDictionary<string, string> DependencyUsages { get; }
// Set of top-level platforms whose dependencies are described
public IList<Platform> Platforms { get; }
}
// A supported platform
public class Platform
{
// Rid identifying the platform
public string Rid { get; set; }
// Child platforms (e.g. version-specific, arch-specific)
// Child platforms inherit the characteristics of their parents but can override them
public IList<Platform> Platforms { get; }
// Set of components and their dependencies specific to this platform
public IList<Component> Components { get; }
}
// A logical component within .NET that encapsulates a set of platform dependencies
public class Component
{
// Name of the component
public string Name { get; set; }
// Type of the component
public ComponentType Type { get; set; }
// Set of platform dependencies this component has
public IList<PlatformDependency> Dependencies { get; }
}
// Types of .NET components
public enum ComponentType
{
// A git repository (e.g. https://github.com/dotnet/runtime.git). This is used when needing to describe that a .NET repository has platform dependencies
// in order to build it.
GitRepository,
// A shared framework (e.g. Microsoft.NETCore.App, Microsoft.AspNetCore.App)
SharedFramework,
// A NuGet package (e.g. System.Drawing.Common)
NuGetPackage
}
// Description of a platform dependency
public class PlatformDependency
{
// ID of the dependency
public string Id { get; set; }
// The name expression of the dependency artifact
public string Name { get; set; }
// Type of the dependency
public DependencyType Type { get; set; }
// A value indicating in what manner the dependency is used
public string Usage { get; set; }
// Reference to the dependency from the nearest parent platform that this dependency overrides
public DependencyRef? Overrides { get; set; }
}
// Types of dependencies
public enum DependencyType
{
// An executable file
Executable,
// A library file (e.g. Windows DLL, Linux shared object library)
Library,
// A Linux package contained in a repository
LinuxPackage
}
// A reference to a dependency
public class DependencyRef
{
// ID of the referenced dependency
public string Id { get; set; }
// Type of the referenced dependency
public DependencyType DependencyType { get; set; }
}
}Each platform has its own way of identifying assets that .NET is dependent upon. For example, the Open SSL library is labeled as openssl-libs in Fedora and libssl in Debian. For that reason, platform dependencies can't be described without identifying which platform they are for. The model handles this by describing platforms as a top-level concept to organize dependencies. Since many dependencies apply to multiple versions of a given platform, the ability to describe a versionless platform, as well as more specific versioned platforms if necessary, is possible. This is done by associating a platform with a rid. Rids are hierarchical allowing more specific platforms to be targeted which provides an ideal mechanism to be able to flexibly describe the scope of dependencies. In the same way that rids are hierarchical, so too are the platforms in this model. This allows for less repetition in the actual model content and enables platforms with more specificity to append or amend dependencies from their parent platform. Some examples of this:
- A dependency is taken on a new asset that is only available in a newer version of the platform.
- A dependency name is different in a newer version of the platform which often happens when the version of the asset is contained in its name (e.g. libicu).
Within each platform can be described a set of components. These components represent logical parts of .NET that are delivered as a unit. Examples of these are shared frameworks (Microsoft.NETCore.App, Microsoft.AspNetCore.App, etc) and NuGet packages.
Each component describes the dependencies it has for the platform it is contained within. A dependency is identified by its ID, which defaults to the value of the name, and type (Linux distro package, DLL).
The name of a dependency is an expression that can be used to describe a variety of conditions. The syntax allows for version ranges to be specified using interval notation. Examples:
libgcc1: Any version of thelibgcc1package.libgcc1:4.9.2: Version 4.9.2 or higher of thelibgcc1package.libgcc1:[4.9.2,5.0): A version of thelibgcc1package that lies within the range 4.9.2 >= x < 5.0.libssl1.0.0 || libssl1.1: Any version of thelibssl1.0.0orlibssl1.1packages.
Expression syntax (BNF):
<name-expr> ::= <dependency> | <name-expr> <logical-op> <dependency>
<dependency> ::= <dependency-name> | <dependency-name> ":" <version-range>
<version-range> ::= <version-num> | <start-range> <range-version-num> "," <range-version-num> <end-range>
<range-version-num> ::= <version-num> | ""
<start-range> ::= "[" | "("
<end-range> ::= "]" | ")"
<logical-op> ::= "||"
Supported operators:
- Logical OR (
||): Indicates that the component can use either of the specified dependency names. Operands should be listed in the order with the most preferred usage listed last; this allows consumers of the model to resolve the expression to a single value in cases where a value needs to be outputted.
The logical OR operator is specifically useful in cases where a platform provides multiple dependency artifacts with distinct names and the component is compatible with any of them. As an example, the .NET runtime is compatible with either version 1.0.0 or 1.1 of libssl. If the platform happens to provide both of these packages, whose names are distinct, it can be described in the model as the following:
{
"id": "libssl",
"name": "libssl1.0.0:1.0.1t-1 || libssl1.1:1.1.1d-0",
"dependencyType": "LinuxPackage",
"usage": "default"
},Notes: The
idfield is set here to provide a valid addressable ID for the platform dependency. The minimum version forlibssl1.0.0is1.0.1t-1while the minimum version forlibssl1.1is1.1.1d-0.
The ID of the dependency can be explicitly set; otherwise, its value defaults to the name portion of the dependency's name expression. For example, a dependency with a name expression of libgcc1:4.9.2 will have its ID defaulted to libgcc1. This allows the dependency to easily be referenced for overriding without unnecessarily specifying the version portion of the expression.
A key piece of metadata that gives context to the dependency is the "usage" field. This field is set to one of the model-defined values that describes the scenario in which this dependency applies. Here are some examples of usage values:
- default: Indicates that the dependency applies to a canonical app scenario (i.e. Hello World). Note that this is specifically about a canonical app and not intended to be a description of required dependencies in the absolute sense. An example of this is libicu. While libicu is not required to run an app if it has been set to use invariant globalization, the default/canonical setting of a .NET app is that invariant globalization is set to false in which case libicu is necessary. This is why the term "default" is used rather than something like "required".
- diagnostics: Indicates the dependency should be used in scenarios where diagnostic tools are being used such as with LTTng-UST.
- httpsys: Indicates the dependency should be used for ASP.NET Core apps that are configured to use the HTTP.sys web server.
- localization: Indicates the dependency should be used for localization/globalization scenarios such as with tzdata.
Dependencies are able to override the content of another dependency that is inherited from its platform hierarchy. The "overrides" field identifies the name and type of the target dependency to be overridden. Any other fields that are set in the dependency will override the corresponding values from the referenced dependency.
{
"dotnetReleaseVersion": "6.0.0",
"dependencyUsages": {
"default": "Used by default or for canonical scenarios",
"diagnostics": "Used for diagnostic scenarios such as tracing or debugging",
"httpSys": "Used for ASP.NET Core apps that are configured to use the HTTP.sys web server",
"localization": "Used for locale and culture scenarios"
},
"platforms": [
{
"rid": "debian",
"components": [
{
"name": "Microsoft.NETCore.App",
"type": "Framework",
"platformDependencies": [
{
"name": "libc6",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "libgcc1",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "libgssapi-krb5-2",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "libicu57",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "liblttng-ust0",
"dependencyType": "LinuxPackage",
"usage": "diagnostics"
},
{
"name": "libssl1.1",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "libstdc++6",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "tzdata",
"dependencyType": "LinuxPackage",
"usage": "localization"
},
{
"name": "zlib1g",
"dependencyType": "LinuxPackage",
"usage": "default"
}
]
},
{
"name": "System.DirectoryServices.Protocols",
"type": "NuGetPackage",
"platformDependencies": [
{
"name": "libldap-2.4-2",
"dependencyType": "LinuxPackage",
"usage": "default"
}
]
}
],
"platforms": [
{
"rid": "debian.10",
"components": [
{
"name": "Microsoft.NETCore.App",
"type": "Framework",
"platformDependencies": [
{
"name": "libicu63",
"overrides": {
"name": "libicu57",
"dependencyType": "LinuxPackage"
}
}
]
}
]
}
]
}
]
}{
"dependencyUsages": {
"default": "Used by default or for canonical scenarios",
"numa": "Used to enable NUMA support"
},
"platforms": [
{
"rid": "debian",
"components": [
{
"name": "https://github.com/dotnet/runtime.git",
"type": "GitRepository",
"platformDependencies": [
// Just a snippet of the full list is shown
{
"name": "clang-9",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "cmake",
"dependencyType": "LinuxPackage",
"usage": "default"
},
{
"name": "libnuma-dev",
"dependencyType": "LinuxPackage",
"usage": "numa"
},
{
"name": "llvm-9",
"dependencyType": "LinuxPackage",
"usage": "default"
}
]
}
]
}
]
}The model above uses the Debian Linux distro as an example. It shows the Microsoft.NETCore.App shared framework as having a base set of dependencies but for Debian 10, the libicu57 package dependency is overriden to be libicu63 since that is the version available in Debian 10. It also shows that the System.DirectoryServices.Protocols NuGet package has a dependency on libldap-2.4-2.
Dependencies will change over time from release to release. For that reason, it is reasonable to consider the platform dependency model to be a release artifact, tied to a particular release. Precedent already exists for this kind of artifact with the releases.json file. In order to provide a consistent experience for consuming release artifacts, this design roughly follows that pattern used for the releases.json file.
The runtime dependency model will be represented as a JSON file and be stored in the release-notes folder for each release. For example, the 6.0.0 release would have the runtime dependency model located at https://github.com/dotnet/core/blob/master/release-notes/6.0/6.0.0/6.0.0-runtime-deps.json; a preview release would be located at https://github.com/dotnet/core/blob/master/release-notes/6.0/preview/6.0.0-preview.1-runtime-deps.json. This design deviates from the releases.json file which is stored at major/minor folder level instead of the patch folder for a couple reasons:
- There is no dependency data that is common to all patch releases. While there could end up being commonality in the data, it is not an inherent feature.
- From a human-readability standpoint, the size of the json file would be quite large and difficult to navigate as patch releases accumulate.
The toolchain dependency model will also be represented as a JSON file but be stored in each .NET git repository at a known location: eng/toolchain-dependencies.json. It's not treated as a release artifact in the same way as the runtime dependency model. Its purpose is more relevant to the repo so it is co-located there instead of consolidating them into the dotnet/core repo.
It is necessary for all .NET contributors to participate in the maintenance of the platform dependency model. The scope of .NET is too large to have a single person or small set of people actively monitor for changes to the dependencies. It needs to be a distributed effort where the maintainers that are familiar with that area of .NET can keep an eye on things.
There will also be a small group charged with maintaining the platform dependency tracking system as a whole. This group will be responsible for approving updates to the model and acting as points-of-contact for any questions from contributors.
Luckily, the burden of maintenance is expected to be low since dependencies don't often change. However, there are a few scenarios that do cause change which are described further below.
In all of the scenarios below, the person introducing or discovering the dependency change should file an issue in the dotnet/core repo describing the change. Such issues should be labeled as area-dependency. This is necessary in order to notify stakeholder teams (e.g. installers, container images, documentation) of such changes.
New platform dependencies can be introduced by making a code change that has the dependency or including a new component into .NET (e.g. shared framework, NuGet package). When such a dependency is added, the contributor should also update the appropriate platform dependency model file with the necessary information that describes the dependency across all supported platforms.
In order to avoid the reliance upon contributors to recognize when they've changed a dependency, a more automated solution would be preferable. This can be done by defining a GitHub bot that checks for files in PRs containing NativeMethods or Interop in their name. If such a file is detected, a label is added to the PR alerting the submitter that they should evaluate their changes for changes to the dependencies. While not a fool-proof solution, it should provide coverage for the vast majority of dependencies. Work is still required by the submitter to make the appropriate changes to the platform dependency model but the GitHub bot helps alert them when there is potentially action that is needed.
The platform dependency model describes dependencies by name and those names are subject to change from upstream sources.
A classic example of this is the ICU library. In many Linux distros, the ICU library's package contains the version number in the name (e.g. libicu57). When a Linux distro updates to use a new version of the ICU library, they typically remove the previous version from the package repository. Thus, any dependency described for the ICU library needs to account for the name change.
We will rely on manual discovery to determine when the name of a package has changed since the typical scenario in which a name change is detected happens when a build breaks. When a dependency name change is discovered, the person discovering the change should file an issue in dotnet/core as described above.
When support for a new platform is added to the product, the platform dependency model of future releases needs to be updated include this platform and all its supported versions.
When support for a platform is removed from the product, the platform dependency model of future releases needs to be updated remove this platform.
The first thing a maintainer needs to do in order to edit the dependency model is to know which model file to update.
In cases where a new dependency is added or support is added for a new platform, the maintainer should select the dependency model associated with the upcoming release in the https://github.com/dotnet/core/tree/master/release-notes location. In all likelihood, there will not yet be a folder for the upcoming release so one can be added as the placeholder for the future release notes and define the dependency model file there.
In all other cases, it is the past release dependency files that need to be updated. For example, if support for Debian 10 ends, all dependency model files for all past releases that contain the debian.10 rid should be updated to have that platform removed.
Because there can be numerous dependency model files that need to be updated for certain scenarios -- and to reduce human error -- a CLI tool will be created in order to perform certain operations across multiple dependency model files. This will be intended for scenarios that would otherwise be repetitive busywork and can easily be automated. Examples of such operations include dependency name changes and platform removal.
dependency override [--path <path>] <dependency-type> <source-rid> <source-dependency-name> <target-rid> <target-dependency-name>Options:
path: Root path location where dependency model files will be searched and updated. Default is the current working directory.
Arguments:
dependency-type: type of the dependency to overridesource-rid: rid of the dependency to overridesource-dependency-name: name of the dependency to overridetarget-rid: rid of the platform to contain the dependency overridetarget-dependency-name: new name of the dependency
Example:
Overrides the libicu package from the default of libicu57 for all Debian platforms to libicu63 for Debian 10:
dotnet-deps dependency override LinuxPackage debian libicu57 debian.10 libicu63platform remove [--path <path>] [--force] <rid>Options:
path: Root path location where dependency model files will be searched and updated. Default is the current working directory.force: Forces the platform to be removed even if it has child platforms.
Arguments:
rid: rid of the platform to remove
Example:
Removes the debian.10 platform:
dotnet-deps platform remove debian.10