storage: GenericEphemeralVolume #22438

pohly · 2020-07-09T10:21:17Z

This is the initial documentation for one new feature:

generic ephemeral inline volumes enhancements#1698

The feature should get included in 1.19, its implementation is approved and just needs to be merged.

k8sio-netlify-preview-bot · 2020-07-09T10:21:22Z

Deploy preview for kubernetes-io-vnext-staging processing.

Building with commit 4401fd8

https://app.netlify.com/sites/kubernetes-io-vnext-staging/deploys/5f0f2eb584370e000742a261

savitharaghunathan · 2020-07-09T15:49:56Z

/milestone 1.19
/assign

pohly · 2020-07-10T15:50:41Z

Pinging:

@msau42, @jsafrane: technical review
@divya-mohan0209, @savitharaghunathan: docs review

pohly · 2020-07-10T15:52:35Z

The URL for "Storage Capacity Tracking" assumes that PR #21634 gets merged.

The URLs for the 1.19 API are tentative: I believe they will be correct once the API gets published, but currently it isn't.

sftim

Thanks for this PR @pohly

Here's some feedback. Any comment I've labelled as a nit is nice-to-fix, but it's OK to merge this without addressing it.

content/en/docs/concepts/storage/ephemeral-volumes.md

sftim · 2020-07-11T18:06:00Z

content/en/docs/concepts/storage/volumes.md

@@ -1355,37 +1358,9 @@ as usual, without any CSI specific changes.

 {{< feature-state for_k8s_version="v1.16" state="beta" >}}

-This feature allows CSI volumes to be directly embedded in the Pod specification instead of a PersistentVolume. Volumes specified in this way are ephemeral and do not persist across Pod restarts.
+This feature allows CSI volumes to be directly embedded in the Pod specification instead of a PersistentVolume. Volumes specified in this way are ephemeral and do not persist across Pod restarts. See [the ephemeral volume page](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) for more information.


Suggested change

This feature allows CSI volumes to be directly embedded in the Pod specification instead of a PersistentVolume. Volumes specified in this way are ephemeral and do not persist across Pod restarts. See [the ephemeral volume page](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) for more information.

You can directly configure CSI volume claims within the Pod specification. Volumes specified in this way are ephemeral and do not persist across Pod restarts. See [Ephemeral Volumes](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) for more information.

I'm saying volume claim because “ephemeral PersistentVolumeClaim” doesn't really make sense.

In this case "CSI volumes" makes more sense than "CSI volume claims" because there is nothing resembling a "claim" in the API. The rest makes sense, changed.

content/en/docs/concepts/storage/ephemeral-volumes.md

sftim · 2020-07-11T18:14:02Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+Kubernetes supports several different kinds of ephemeral volumes for
+different purposes:
+- [emptyDir]((/docs/concepts/volumes/#emptydir): a directory on the root disk or


Does the backing storage emptyDir have to live on the node's root filesystem? (I don't believe so).

I'd explain it as “an amount of node-local ephemeral storage, empty at Pod startup”

Actual storage is limited to the root disk. The only alternatives are RAM. I was trying to point this out because it's the difference compared to CSI.

Let's change it into "empty at Pod startup, with storage provided locally from the root disk or RAM"

The kubelet has a base directory for other locally stored data (/var/lib/kubelet by default).

Typically, /var/lib/kubelet is on the system root filesystem, and the kubelet is designed with that layout in mind.

(adapted from https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#local-ephemeral-storage)

I don't think “root disk” is correct. First because the root filesystem isn't always backed by durable storage, and second because /var/lib/kubelet doesn't have to live on that filesystem.

"root disk" is a simplification. How about "with storage provided locally from the kubelet base directory (usually the root disk) or from memory"?

I'm also not sure about "provided locally from" - perhaps "coming locally from"?

I've switched to "with storage coming locally from the kubelet base directory (usually the root disk) or RAM"

sftim · 2020-07-11T18:17:13Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+- pohly
+title: Ephemeral Volumes
+content_type: concept
+weight: 20


This weight interposes Ephemeral Volumes between the Volumes concept page and the Persistent Volumes concept page. I don't think that's what you want here.

Cut-and-paste without thinking about what this field means... I'm torn between placing it directly after "persistent volumes" and adding it at the end, after all the concepts linked to in the document itself have been introduced.

Let me add it at the end.

pohly · 2020-07-13T12:14:55Z

@sftim thanks a lot for the detailed and constructive review. It's very appreciated!

Perhaps you can also work your review magic on #21634?

I just pushed a version which fixes some of the things I learned about in this review here (URL format, weight).

divya-mohan0209 · 2020-07-13T13:17:02Z

@pohly : Just checking in to see whether you've received a signoff from a docs & a technical (if applicable) perspective? If you haven't yet, please could you get them done at the earliest for a successful merge?

msau42 · 2020-07-13T19:04:20Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+Other applications expect some read-only input data to be present in
+files, like configuration data or secret keys.
+
+_Ephemeral volumes_ are designed for these use cases. Because volumes


I think it's important to mention somehwere in the intro that ephemeral volumes follow the pod's lifetime and get created and deleted along with the pod.

Replaced "created anew for each pod" with "follow the pod's lifetime and get created and deleted along with the pod".

Or rather, Pod (capital case)... I also fixed that elsewhere.

msau42 · 2020-07-13T19:06:23Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+They are managed by kubelet on each node.
+
+CSI ephemeral volumes *must* be provided by third-party CSI storage
+drivers. Generic ephemeral volumes *can* be provided by third-party


Start "Generic ephemeral volumes on a separate line" since we're talking about a different feature.

msau42 · 2020-07-13T19:07:44Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+CSI ephemeral volumes *must* be provided by third-party CSI storage
+drivers. Generic ephemeral volumes *can* be provided by third-party
+CSI storage drivers, but also by any other storage driver that


The same storage drivers that support CSI ephemeral volumes may not be necessarily able to support generic ephemeral volumes.

Added: "Some CSI drivers are written specifically for CSI
ephemeral volumes and do not support dynamic provisioning: those then
cannot be used for generic ephemeral volumes."

msau42 · 2020-07-13T19:09:49Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+- Volumes can have a fixed size that pods are not able to exceed.
+- Volumes may have some initial data, depending on the driver and
+  parameters.
+- Typical operations on volumes are supported, including


assuming that the driver supports them

msau42 · 2020-07-13T19:11:49Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+are allowed inside a volume source of the pod. Labels, annotations and
+the whole set of fields for a PersistentVolumeClaim are supported. When such a Pod gets
+created, the ephemeral volume controller then creates an actual PersistentVolumeClaim
+object in the same namespace as the pod.


And gets deleted when the Pod is deleted.

I've added "... and ensures that the PersistentVolumeClaim gets deleted when the Pod gets deleted." How that works specifically is then explained later - I didn't want to repeat that explanation.

IMHO explaining the steps in chronological order makes sense.

msau42 · 2020-07-13T19:13:04Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+### PersistentVolumeClaim naming
+
+Naming of the additional PVCs is deterministic: the name is


additional -> ephemeral? although "ephemeral PVC" is a bit of an oxymoron

Yes, that's why I avoided it 😢

Replaced with "automatically created".

msau42 · 2020-07-13T19:14:15Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+Enabling the GenericEphemeralVolume feature allows users to create
+PVCs indirectly if they can create pods, even if they do not have
+permission to create them directly. Cluster administrators must be


them -> PVCs

msau42 · 2020-07-13T19:15:49Z

content/en/docs/concepts/storage/volumes.md

-referenced in a Pod via a `PersistentVolumeClaim` object.
+A `csi` volume can be used in a pod in three different ways:
+- through a reference to a [`persistentVolumeClaim`](#persistentvolumeclaim)
+- with a [generic ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volume)


should we note that these are alpha/beta respsectively?

msau42 · 2020-07-13T19:17:22Z

content/en/docs/concepts/storage/volumes.md

@@ -1291,8 +1291,11 @@ Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users
 may use the `csi` volume type to attach, mount, etc. the volumes exposed by the


Should we link to the new ephemeral-volumes page somewhere at the top of this page? (and link to the persistent volumes page in the same area)

The "background" section may need a more thorough rewrite. For example, it still says:

A Kubernetes volume, on the other hand, has an explicit lifetime - the same as
the Pod that encloses it.

That became obsolete when persistent volumes were introduced, which was a while ago...

Let me punt on that for now for the sake of getting this PR merged before the docs deadline. I've created #22505 to track it.

This is the initial documentation for one new feature: - kubernetes/enhancements#1698 A new page gets created for different ephemeral volumes because the relationship between them needs to be explained. Co-authored-by: Tim Bannister <tim@scalefactory.com>

pohly

Pushed an updated version.

pohly · 2020-07-14T08:05:31Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+Other applications expect some read-only input data to be present in
+files, like configuration data or secret keys.
+
+_Ephemeral volumes_ are designed for these use cases. Because volumes


Replaced "created anew for each pod" with "follow the pod's lifetime and get created and deleted along with the pod".

Or rather, Pod (capital case)... I also fixed that elsewhere.

pohly · 2020-07-14T08:07:52Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+They are managed by kubelet on each node.
+
+CSI ephemeral volumes *must* be provided by third-party CSI storage
+drivers. Generic ephemeral volumes *can* be provided by third-party


pohly · 2020-07-14T08:11:01Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+CSI ephemeral volumes *must* be provided by third-party CSI storage
+drivers. Generic ephemeral volumes *can* be provided by third-party
+CSI storage drivers, but also by any other storage driver that


Added: "Some CSI drivers are written specifically for CSI
ephemeral volumes and do not support dynamic provisioning: those then
cannot be used for generic ephemeral volumes."

pohly · 2020-07-14T08:12:44Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+CSI ephemeral volumes *must* be provided by third-party CSI storage
+drivers. Generic ephemeral volumes *can* be provided by third-party
+CSI storage drivers, but also by any other storage driver that
+supports dynamic provisioning. These drivers can offer functionality


"These drivers" became a bit ambiguous. I replaced it with "The advantage of using third-party drivers is that they can..."

pohly · 2020-07-14T08:14:04Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+- Volumes can have a fixed size that pods are not able to exceed.
+- Volumes may have some initial data, depending on the driver and
+  parameters.
+- Typical operations on volumes are supported, including


pohly · 2020-07-14T08:20:39Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+using a StorageClass with a reclaim policy of `retain`: the storage outlives the Pod, 
+and in this case you need to ensure that volume clean up happens separately.
+
+Once these PVCs exist, they can be used like any other PVC. In


Replaced "Once" with "While" - it's more accurate and hints that this can change...

pohly · 2020-07-14T08:21:37Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+### PersistentVolumeClaim naming
+
+Naming of the additional PVCs is deterministic: the name is


Yes, that's why I avoided it 😢

Replaced with "automatically created".

pohly · 2020-07-14T08:22:15Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+
+Enabling the GenericEphemeralVolume feature allows users to create
+PVCs indirectly if they can create pods, even if they do not have
+permission to create them directly. Cluster administrators must be


pohly · 2020-07-14T08:31:12Z

content/en/docs/concepts/storage/volumes.md

@@ -1291,8 +1291,11 @@ Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users
 may use the `csi` volume type to attach, mount, etc. the volumes exposed by the


The "background" section may need a more thorough rewrite. For example, it still says:

A Kubernetes volume, on the other hand, has an explicit lifetime - the same as
the Pod that encloses it.

That became obsolete when persistent volumes were introduced, which was a while ago...

Let me punt on that for now for the sake of getting this PR merged before the docs deadline. I've created #22505 to track it.

pohly · 2020-07-14T08:32:03Z

content/en/docs/concepts/storage/volumes.md

-referenced in a Pod via a `PersistentVolumeClaim` object.
+A `csi` volume can be used in a pod in three different ways:
+- through a reference to a [`persistentVolumeClaim`](#persistentvolumeclaim)
+- with a [generic ephemeral volume](/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volume)


sftim

I'd actually be happy to see this merge, we can fix the one bit of dodgy Markdown in a follow-up good first issue PR.

Preview at https://deploy-preview-22438--kubernetes-io-vnext-staging.netlify.app/docs/concepts/storage/ephemeral-volumes/ otherwise looks OK.

content/en/docs/concepts/storage/ephemeral-volumes.md

sftim · 2020-07-14T18:00:21Z

/sig storage
@kubernetes/sig-storage-pr-reviews is this OK from a technical accuracy point of view?

Co-authored-by: Tim Bannister <tim@scalefactory.com>

pohly · 2020-07-14T18:49:31Z

I'd actually be happy to see this merge, we can fix the one bit of dodgy Markdown in a follow-up good first issue PR.

I took your commit, thanks for the fix.

@msau42 wants to check the technical side, but it might still take a few more hours before she has time for it.

Same with #21634

msau42 · 2020-07-14T22:11:18Z

Tech LGTM.

There were some comments on other PRs about not linking to keps or enhancement issues.

sftim · 2020-07-14T22:32:34Z

content/en/docs/concepts/storage/ephemeral-volumes.md

+### Ephemeral volumes managed by kubelet
+
+See [local ephemeral storage](/docs/concepts/configuration/manage-resources-containers/#local-ephemeral-storage).
+
+### CSI ephemeral volumes
+
+- For more information on the design, see the [Ephemeral Inline CSI
+  volumes KEP](https://github.com/kubernetes/enhancements/blob/ad6021b3d61a49040a3f835e12c8bb5424db2bbb/keps/sig-storage/20190122-csi-inline-volumes.md).
+- For more information on further development of this feature, see the [enhancement tracking issue #596](https://github.com/kubernetes/enhancements/issues/596).
+
+### Generic ephemeral volumes
+
+- For more information on the design, see the
+[Generic ephemeral inline volumes KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1698-generic-ephemeral-volumes/README.md).
+- For more information on further development of this feature, see the [enhancement tracking issue #1698](https://github.com/kubernetes/enhancements/issues/1698).


This could be a little tidier (for example, we usually and intentionally don't link to KEPs other than for historical context), but I'm happy to log a separate issue to track tidying that up after this merges. It's a minor detail for sure.

I think that for alpha and beta features, easy access to both the KEP and the enhancement issue are important: the KEP explains "why" the feature is what it is and in particular, what other alternatives were considered. The enhancement issue answers the questions "when will this graduate" and "what is still changing with this feature".

I can remove it in a follow-up PR if this explanation is not convincing enough. But for now, let's merge.

sftim

/lgtm

content/en/docs/concepts/storage/ephemeral-volumes.md

sftim · 2020-07-14T22:34:32Z

(not sure what GitHub is playing at there, but it does LGTM)

Co-authored-by: Tim Bannister <tim@scalefactory.com>

pohly · 2020-07-15T16:31:06Z

New changes are detected. LGTM label has been removed.

The only change was the formatting fix from @sftim, so his and @msau42's LGTM still apply.

savitharaghunathan · 2020-07-15T16:37:59Z

@sftim's comment on formatting has been addressed.

/lgtm
/approve

k8s-ci-robot · 2020-07-15T16:38:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: savitharaghunathan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [savitharaghunathan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 9, 2020

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 9, 2020

k8s-ci-robot requested review from jimangel and thockin July 9, 2020 10:21

k8s-ci-robot added language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Jul 9, 2020

pohly mentioned this pull request Jul 9, 2020

generic ephemeral inline volumes kubernetes/enhancements#1698

Closed

k8s-ci-robot assigned savitharaghunathan Jul 9, 2020

k8s-ci-robot added this to the 1.19 milestone Jul 9, 2020

pohly force-pushed the generic-ephemeral-volumes branch from ee31153 to 1f1fa07 Compare July 10, 2020 15:47

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 10, 2020

pohly changed the title ~~WIP: storage: GenericEphemeralVolume~~ storage: GenericEphemeralVolume Jul 10, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 10, 2020

sftim reviewed Jul 11, 2020

View reviewed changes

pohly force-pushed the generic-ephemeral-volumes branch 2 times, most recently from 934c42f to f5f21d7 Compare July 13, 2020 09:38

pohly mentioned this pull request Jul 13, 2020

storage: CSIStorageCapacity #21634

Merged

msau42 reviewed Jul 13, 2020

View reviewed changes

pohly mentioned this pull request Jul 14, 2020

content/en/docs/concepts/storage/volumes.md: out-dated introduction #22505

Closed

pohly force-pushed the generic-ephemeral-volumes branch from f5f21d7 to 5981f4f Compare July 14, 2020 08:33

pohly commented Jul 14, 2020

View reviewed changes

sftim reviewed Jul 14, 2020

View reviewed changes

content/en/docs/concepts/storage/ephemeral-volumes.md Outdated Show resolved Hide resolved

k8s-ci-robot added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Jul 14, 2020

Update content/en/docs/concepts/storage/ephemeral-volumes.md

25d50c1

Co-authored-by: Tim Bannister <tim@scalefactory.com>

sftim reviewed Jul 14, 2020

View reviewed changes

content/en/docs/concepts/storage/ephemeral-volumes.md Outdated Show resolved Hide resolved

k8s-ci-robot assigned sftim Jul 14, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 14, 2020

Update content/en/docs/concepts/storage/ephemeral-volumes.md

4401fd8

Co-authored-by: Tim Bannister <tim@scalefactory.com>

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 15, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 15, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 15, 2020

k8s-ci-robot merged commit 3f7af5d into kubernetes:dev-1.19 Jul 15, 2020

	This feature allows CSI volumes to be directly embedded in the Pod specification instead of a PersistentVolume. Volumes specified in this way are ephemeral and do not persist across Pod restarts. See [the ephemeral volume page](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) for more information.
	You can directly configure CSI volume claims within the Pod specification. Volumes specified in this way are ephemeral and do not persist across Pod restarts. See [Ephemeral Volumes](/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volume) for more information.


		### PersistentVolumeClaim naming

		Naming of the additional PVCs is deterministic: the name is

		@@ -1291,8 +1291,11 @@ Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users
		may use the `csi` volume type to attach, mount, etc. the volumes exposed by the

storage: GenericEphemeralVolume #22438

storage: GenericEphemeralVolume #22438

Conversation

pohly commented Jul 9, 2020 • edited Loading

k8sio-netlify-preview-bot commented Jul 9, 2020 • edited Loading

savitharaghunathan commented Jul 9, 2020

pohly commented Jul 10, 2020

pohly commented Jul 10, 2020

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Jul 13, 2020

divya-mohan0209 commented Jul 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim left a comment

Choose a reason for hiding this comment

sftim commented Jul 14, 2020

pohly commented Jul 14, 2020 • edited Loading

msau42 commented Jul 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim left a comment

Choose a reason for hiding this comment

sftim commented Jul 14, 2020

pohly commented Jul 15, 2020

savitharaghunathan commented Jul 15, 2020

k8s-ci-robot commented Jul 15, 2020

pohly commented Jul 9, 2020 •

edited

Loading

k8sio-netlify-preview-bot commented Jul 9, 2020 •

edited

Loading

pohly commented Jul 14, 2020 •

edited

Loading