Correctly handle named pipe host mounts for Windows #69484

ddebroy · 2018-10-05T23:55:10Z

What this PR does / why we need it:
Windows named pipes in host mounts were not being correctly handled and passed through to the container runtime. This PR adds support to detect named pipes in Windows and skip the processing associated with regular file system based host mount paths in Windows.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes # #67147

Special notes for your reviewer:
Tested the following spec spawns up the pod container on Windows with the mount paths to \\.\pipe\docker_pipe correctly configured:

apiVersion: v1
kind: Pod
metadata:
  name: docker-pipe-windows
spec:
  nodeSelector:
    beta.kubernetes.io/os: windows
  containers:
    - name: main
      image: microsoft/windowsservercore:1803
      command:
        - powershell
        - Start-Sleep
        - "-s 3600"
      volumeMounts:
        - mountPath: \\.\pipe\docker-engine
          name: docker-pipe
  volumes:
    - name: docker-pipe
      hostPath: 
        path: \\.\pipe\docker-engine
        type: null
  restartPolicy: Never

docker inspect output for above container:

...
        "Mounts": [
            {
                "Type": "npipe",
                "Source": "\\\\.\\pipe\\docker-engine",
                "Destination": "\\\\.\\pipe\\docker-engine",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "c:\\var\\lib\\kubelet\\pods\\1e664368-c8f7-11e8-97bc-000d3a35a802\\volumes\\kubernetes.io~secret\\default-token-5kttv",
                "Destination": "c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount",
                "Mode": "",
                "RW": false,
                "Propagation": ""
            }
        ],
...

Release note:

Handle Windows named pipes in host mounts.

Signed-off-by: Deep Debroy <ddebroy@docker.com>

ddebroy · 2018-10-05T23:55:36Z

/sig storage

ddebroy · 2018-10-05T23:56:21Z

/assign @dchen1107

ddebroy · 2018-10-05T23:56:48Z

cc @feiskyer @andyzhangx

ddebroy · 2018-10-06T00:31:07Z

/kind bug

ddebroy · 2018-10-06T02:45:23Z

/test pull-kubernetes-verify

ddebroy · 2018-10-06T02:45:37Z

/test pull-kubernetes-local-e2e-containerized

feiskyer

LGTM

@PatrickLang Would you like also take a look?

feiskyer · 2018-10-06T05:04:27Z

/retest

andyzhangx · 2018-10-08T02:41:46Z

pkg/volume/util/util.go

@@ -945,6 +945,13 @@ func CheckPersistentVolumeClaimModeBlock(pvc *v1.PersistentVolumeClaim) bool {
 	return utilfeature.DefaultFeatureGate.Enabled(features.BlockVolume) && pvc.Spec.VolumeMode != nil && *pvc.Spec.VolumeMode == v1.PersistentVolumeBlock
 }

+func IsWindowsNamedPipe(goos, path string) bool {
+	if goos == "windows" && strings.HasPrefix(path, `\\.\pipe\`) {


@PatrickLang @atomaras Does named pipe also starts with \\.\pipe\? what about upper case like \\.\Pipe\ ?

It also supports like this: \.\pipe\ ...

docker run -it --rm -v \.\pipe\docker_engine:\.\pipe\docker_engine microsoft:windowsservercore:1803 powershell

I think the logic here needs to be aligned with logic in the runtime. Here are some references:
https://github.com/moby/moby/blob/master/volume/mounts/windows_parser.go#L44
and
https://github.com/moby/moby/blob/master/libcontainerd/client_local_windows.go#L303
Both consider \\.\pipe as the standard named pipe prefix for local pipes on Windows.

@andyzhangx, \.\pipe\... is rejected by Docker:

C:\k>docker run -ti -v \.\pipe\docker_engine:\.\pipe\docker_engine microsoft/windowsservercore:1803 docker: Error response from daemon: invalid volume specification: '\.\pipe\docker_engine:\.\pipe\docker_engine'.

The underlying CreateFileW() is case insensitive by default, so \\.\pipe\foo == \\.\Pipe\foo
https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-createfilew

If it's constrained in the lower layer (moby / containerd), then the same check should be used here. I think it's ok as-is

bertinatto · 2018-10-08T14:57:12Z

I seems like pull-kubernetes-local-e2e-containerized is failing because of #69465 .

ddebroy · 2018-10-09T21:17:19Z

/test pull-kubernetes-local-e2e-containerized

ddebroy · 2018-10-10T04:19:28Z

/retest

PatrickLang · 2018-10-10T17:09:27Z

I talked to @ddebroy on Slack, and this requires Docker EE-basic 18.03 or later. If you're testing on an Azure VM, you need to be sure to upgrade Docker first. This is tracked at Azure/acs-engine#3852

PatrickLang · 2018-10-10T17:22:58Z

pkg/kubelet/kubelet_pods.go

 		containerPath := mount.MountPath
 		if runtime.GOOS == "windows" {
-			if (strings.HasPrefix(hostPath, "/") || strings.HasPrefix(hostPath, "\\")) && !strings.Contains(hostPath, ":") {
+			if !volumeutil.IsWindowsNamedPipe(runtime.GOOS, hostPath) && (strings.HasPrefix(hostPath, "/") || strings.HasPrefix(hostPath, "\\")) && !strings.Contains(hostPath, ":") {


nit - can you preserve the parenthesis so that the grouping is still easy to see?

ddebroy · 2018-10-17T16:56:37Z

@Random-Liu @saad-ali PTAL if this looks good to be approved. Not sure if @dchen1107 (who was originally assigned) is online/offline.

saad-ali

Couple of minor comments.

/approve

saad-ali · 2018-10-19T17:57:12Z

pkg/volume/util/util.go

+// IsWindowsUNCPath checks if path is prefixed with \\
+// This can be used to skip any processing of paths
+// that point to SMB shares, local named pipes and local UNC path
+func IsWindowsUNCPath(goos, path string) bool {


nit, why is goos a parameter and not generated inside this method?

goos as a parameter allows for an easy way to unit-test the function to make sure in other OS environments, there are no unexpected results.

saad-ali · 2018-10-19T18:08:01Z

pkg/kubelet/kubelet_pods.go

@@ -218,11 +218,13 @@ func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, h
 		// Docker Volume Mounts fail on Windows if it is not of the form C:/
 		containerPath := mount.MountPath
 		if runtime.GOOS == "windows" {
-			if (strings.HasPrefix(hostPath, "/") || strings.HasPrefix(hostPath, "\\")) && !strings.Contains(hostPath, ":") {
+			// Append C: only if it looks like a local path. Do not process UNC path/SMB shares/named pipes
+			if (strings.HasPrefix(hostPath, "/") || strings.HasPrefix(hostPath, "\\")) && !strings.Contains(hostPath, ":") && !volumeutil.IsWindowsUNCPath(runtime.GOOS, hostPath) {


IsWindowsUNCPath is checking for prefix \ e.g. \foo\bar which is different from what strings.HasPrefix(hostPath, "\") is checking for: \ vs \

That is very subtle. Add a comment explaining this, so folks are less likely to "fix" this in the future by mistake. Also is there any unit test that can be added to kubelet_pods_tests.go for this to catch regressions?

PatrickLang · 2018-10-23T21:54:32Z

For other reviewers - this change allows Windows mounts that are equivalent to /var/run/docker.sock on Linux. The usual caveats around allowing HostPath mounts should apply to this too.

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#volumes-and-file-systems

PatrickLang · 2018-10-23T21:57:10Z

/lgtm
/approve

PatrickLang · 2018-10-23T21:58:13Z

/assign @yujuhong
can you approve or assign to Dawn if you can't?

Signed-off-by: Deep Debroy <ddebroy@docker.com>

yujuhong · 2018-10-25T23:02:58Z

/assign @yujuhong
can you approve or assign to Dawn if you can't?

Reviewed and left comments.

Signed-off-by: Deep Debroy <ddebroy@docker.com>

ddebroy · 2018-10-26T16:12:16Z

@yujuhong @PatrickLang @saad-ali thanks for reviewing so far. I have addressed the review comments ... PTAL.

yujuhong · 2018-10-26T17:44:39Z

pkg/volume/util/util.go

+
+// IsWindowsLocalPath checks if path is a local path
+// prefixed with "/" or "\" like "/foo/bar" or "\foo\bar"
+func IsWindowsLocalPath(goos, path string) bool {


Should MakeAbsolutePath call this function instead?

@yujuhong I would prefer to keep MakeAbsolutePath as is for now in the context of this PR. The scope of this PR is to enable named pipe mounting support in Windows and that code path skips MakeAbsolutePath. MakeAbsolutePath is called in a couple of contexts (not involving named pipes) and maybe refactored in a separate PR later?

yujuhong · 2018-10-26T17:45:40Z

pkg/kubelet/kubelet_pods.go

-				hostPath = "c:" + hostPath
-			}
+		if volumeutil.IsWindowsLocalPath(runtime.GOOS, hostPath) {
+			hostPath = "c:" + hostPath


Should this call MakeAbsolutePath instead?

@yujuhong yes, done.

Signed-off-by: Deep Debroy <ddebroy@docker.com>

ddebroy · 2018-10-27T16:31:39Z

/retest

ddebroy · 2018-10-30T07:16:31Z

@yujuhong PTAL

yujuhong · 2018-10-30T20:14:19Z

/lgtm
/approve

k8s-ci-robot · 2018-10-30T20:14:31Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ddebroy, PatrickLang, saad-ali, yujuhong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/OWNERS~~ [yujuhong]
~~pkg/volume/util/OWNERS~~ [saad-ali]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Correctly handle named pipe host mounts for Windows

b4bb5dd

Signed-off-by: Deep Debroy <ddebroy@docker.com>

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 5, 2018

k8s-ci-robot requested review from dashpole and rootfs October 5, 2018 23:55

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. labels Oct 5, 2018

k8s-ci-robot assigned dchen1107 Oct 5, 2018

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Oct 6, 2018

feiskyer reviewed Oct 6, 2018

View reviewed changes

andyzhangx reviewed Oct 8, 2018

View reviewed changes

feiskyer mentioned this pull request Oct 10, 2018

Switch windows runtime endpoints to npipe #69516

Merged

PatrickLang reviewed Oct 10, 2018

View reviewed changes

k8s-ci-robot assigned saad-ali Oct 17, 2018

saad-ali reviewed Oct 19, 2018

View reviewed changes

k8s-ci-robot assigned PatrickLang Oct 23, 2018

k8s-ci-robot assigned yujuhong Oct 23, 2018

Improve comments for when hostPath in Windows needs to be transformed

2e19f70

Signed-off-by: Deep Debroy <ddebroy@docker.com>

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 25, 2018

Address CR comments and add more tests

119e2a1

Signed-off-by: Deep Debroy <ddebroy@docker.com>

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 26, 2018

yujuhong reviewed Oct 26, 2018

View reviewed changes

Address code review comments

5da66fd

Signed-off-by: Deep Debroy <ddebroy@docker.com>

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 30, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 30, 2018

k8s-ci-robot merged commit 63a7e06 into kubernetes:master Oct 30, 2018

PatrickLang mentioned this pull request Jan 29, 2019

Unable to volume mount named pipe using HostPath on windows container #67147

Closed

JasonChinsen mentioned this pull request Apr 16, 2019

[EKS] [bug]: ARGO jobs on EKS Windows Nodes aws/containers-roadmap#245

Closed

ddebroy mentioned this pull request Jul 30, 2019

Update "Windows Storage Support" KEP status to implementable and address prior feedback kubernetes/enhancements#1184

Merged

This was referenced Apr 19, 2020

Running Argo on hybrid Kubernetes cluster argoproj/argo-workflows#1383

Closed

feat: Windows Container Support. Fixes #1507 and #1383 argoproj/argo-workflows#2747

Merged

lippertmarkus mentioned this pull request Jun 25, 2020

feat(controller): Add support for Docker workflow executor for Windows nodes argoproj/argo-workflows#3301

Merged

6 tasks

Correctly handle named pipe host mounts for Windows #69484

Correctly handle named pipe host mounts for Windows #69484

Conversation

ddebroy commented Oct 5, 2018 • edited Loading

ddebroy commented Oct 5, 2018

ddebroy commented Oct 5, 2018

ddebroy commented Oct 5, 2018

ddebroy commented Oct 6, 2018

ddebroy commented Oct 6, 2018

ddebroy commented Oct 6, 2018

feiskyer left a comment

Choose a reason for hiding this comment

feiskyer commented Oct 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang Oct 10, 2018 • edited Loading

Choose a reason for hiding this comment

bertinatto commented Oct 8, 2018

ddebroy commented Oct 9, 2018

ddebroy commented Oct 10, 2018

PatrickLang commented Oct 10, 2018

Choose a reason for hiding this comment

ddebroy commented Oct 17, 2018 • edited Loading

saad-ali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddebroy Oct 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PatrickLang commented Oct 23, 2018

PatrickLang commented Oct 23, 2018

PatrickLang commented Oct 23, 2018

yujuhong commented Oct 25, 2018

ddebroy commented Oct 26, 2018 • edited Loading

Choose a reason for hiding this comment

ddebroy Oct 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddebroy Oct 27, 2018 • edited Loading

Choose a reason for hiding this comment

ddebroy commented Oct 27, 2018

ddebroy commented Oct 30, 2018

yujuhong commented Oct 30, 2018

k8s-ci-robot commented Oct 30, 2018

ddebroy commented Oct 5, 2018 •

edited

Loading

PatrickLang Oct 10, 2018 •

edited

Loading

ddebroy commented Oct 17, 2018 •

edited

Loading

ddebroy Oct 25, 2018 •

edited

Loading

ddebroy commented Oct 26, 2018 •

edited

Loading

ddebroy Oct 27, 2018 •

edited

Loading

ddebroy Oct 27, 2018 •

edited

Loading