diff --git a/keps/prod-readiness/sig-node/4009.yaml b/keps/prod-readiness/sig-node/4009.yaml new file mode 100644 index 000000000000..37629f261f60 --- /dev/null +++ b/keps/prod-readiness/sig-node/4009.yaml @@ -0,0 +1,3 @@ +kep-number: 4009 +alpha: + approver: "@johnbelamaric" diff --git a/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api/README.md b/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api/README.md new file mode 100644 index 000000000000..5d4e7738808c --- /dev/null +++ b/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api/README.md @@ -0,0 +1,851 @@ + +# KEP-4009: Add CDI devices to device plugin API + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +This KEP proposes extending the Device Plugin API, adding a field to specify +Container Device Interface (CDI) device IDs in the `AllocateResponse`. This +supplements the existing fields such as annotations and allows device plugin +implementations to uniquely specify devices using their fully-qualified CDI +devices names. + +The recent addition of CDI device IDs to the CRI structures in [#3731](https://github.com/kubernetes/enhancements/pull/3731) allow these IDs to be forwarded to the CRI runtimes in a secure manner. Although +these changes were motivated by [KEP-3063](https://github.com/kubernetes/enhancements/issues/3063), adding support for these fields to the +existing device plugin API allows this mechanism to also be used for devices +supported by these plugins. + +## Motivation + + + +The Container Device Inteface (CDI) provides a standard mechanism for device +vendors to describe what is required to provide access to a specific resource +such as a GPU. These resources can be uniquely identified using a +fully-qualified CDI device name. + +The changes proposed in [#3731]((https://github.com/kubernetes/enhancements/pull/3731)) extend the CRI to provide a well-defined mechanism for forwarding such +requests to CRI runtimes such as Containerd and Cri-o. These have already +been extended to accept CDI device requests, and to use the associated CDI +specifications to ensure that the required +modifications are made to the OCI runtime specification for a container being +launched. + +The addition of an explicit field for specifying CDI device names to the Device +Plugin API allows this CRI field to be used to indicate which devices should be +injected. This removes the need to use workarounds such as container annotations +to pass this information to the runtimes and allows Device Plugin authors to +adopt CDI to inject devices without requiring that users move to a Dynamic +Resource Allocation (DRA) based implementation. + +### Goals + + +* Allow Device Plugin authors to forward device requests to CRI runtimes as a CRI field. +* Allow Device Plugin authors to use CDI to define the modifications required for containerised environments. + +## Design Details + + + +This adds a repeated `CDIDevice` field to the exiting `ContainerAllocateResponse` returned as part of the +`AllocateResponse` in the Device Plugin API. This matches the modifications made to the Dynamic Resource Allocation API in [#3731](https://github.com/kubernetes/enhancements/pull/3731). + +The values contained in this field are then used to populate the corresponding fields in the CRI +which is passed to the container runtimes. + +```protobuf +// CDIDevice specifies a CDI device information. +message CDIDevice { + // Fully qualified CDI device name + // for example: vendor.com/gpu=gpudevice1 + // see more details in the CDI specification: + // https://github.com/container-orchestrated-devices/container-device-interface/blob/main/SPEC.md + string name = 1; +} + +message ContainerAllocateResponse { + // List of environment variable to be set in the container to access one of more devices. + map envs = 1; + // Mounts for the container. + repeated Mount mounts = 2; + // Devices for the container. + repeated DeviceSpec devices = 3; + // Container annotations to pass to the container runtime + map annotations = 4; + // CDI devices for the container. + repeated CDIDevice cdi_devices = 5; +} +``` + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- `devicemanager`: `2023-06-06` - `85.1%` + + + + + + + + + +##### e2e tests + + + +These cases will be added in the existing e2e tests: + - Device Plugin works with CDI devices + +### Graduation Criteria + + + +### Upgrade / Downgrade Strategy + + + +With gRPC the version is part of the service name. +Old versions and new versions should always be served and listened by the kubelet. + +To a cluster admin upgrading to the newest API version, means upgrading Kubernetes to a newer version as well as upgrading the monitoring component. + +To a vendor changes in the API should always be backwards compatible. + +### Version Skew Strategy + +Kubelet will always be backwards compatible, so going forward existing plugins are not expected to break. + + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +This change is not behind a feature gate. It is adding an optional +field to a stable API. This mirrors what was done in []() when +adding a `cpu_ids` field to the existing PodResources API. + +Furthermore, the only code that would be featured-gated in the Device Manager would be the code to propagate this field from the `AllocateResponse` to the +corresponding field in the CRI. + +###### Does enabling the feature change any default behavior? + +No. Device Plugins need to be updated to make use of the new field. + + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + +No. + + +###### What happens if we reenable the feature if it was previously rolled back? + +Not applicable. The feature cannot be toggled. + +###### Are there any tests for feature enablement/disablement? + + + +End to end tests will be added that exercise this feature, but since the feature +cannot be toggled, this will not be added. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +The failure of the kubelet would mean that fields from new device allocations +will not be processed. This will not affect workloads that are already running +since these should already have access to their required devices. + +###### What specific metrics should inform a rollback? + + + +Not applicable. + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + +Not applicable. + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +No. + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +This depends on Device Plugin vendor implementations making use of the required field and cannot be +directly determined. + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + +Not applicable. + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + + + +Note applicable. +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +- The container runtime must support CDI. +- A Device Plugin must be implemented to use the field. + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + +No. + +###### Will enabling / using this feature result in introducing new API types? + + +No. + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + +No. + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +No. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + +No. + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +No. The additional field will replace existing usages where used. + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +No. + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +Not applicable. + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +Not applicable. + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api/kep.yaml b/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api/kep.yaml new file mode 100644 index 000000000000..cb3a742e2284 --- /dev/null +++ b/keps/sig-node/4009-add-cdi-devices-to-device-plugin-api/kep.yaml @@ -0,0 +1,40 @@ +title: Add CDI devices to device plugin API +kep-number: 4009 +authors: + - "@elezar" +owning-sig: sig-node +participating-sigs: [] +status: implementable +creation-date: 2023-05-16 +reviewers: + - swatisehgal +approvers: + - mrunalp + +see-also: + - "/keps/sig-node/3063-dynamic-resource-allocation" + - "/keps/sig-node/3573-device-plugin" +replaces: [] + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.28" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.28" + beta: "v1.29" + stable: "v1.30" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: [] + +disable-supported: false + +# The following PRR answers are required at beta release +metrics: []