feat/windows slaves #148

rbutcher · 2018-08-02T14:23:08Z

There are two existing pull requests trying to fix the problem of support for Windows slaves. This issue is described in JENKINS-36776, JENKINS-48518, JENKINS-51443, and JENKINS-52120.

PR-98 is more that a year old and is mostly the same changes as are in this PR.
PR-117 is a slimmer change that does not completely address Windows slave support.

While this is a duplicate of some of the other work already done I am opening this PR primarily to bring attention to this deficiency and the fact that this change is highly desired by the Windows developer community.

rbutcher · 2018-08-02T14:24:17Z

src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java

-import hudson.LauncherDecorator;
-import hudson.Proc;
-import hudson.Util;
+import hudson.*;


Sorry for the aggressive import changes. Alt+Enter on intellij 😄
I can change this back if necessary.

rbutcher · 2018-08-20T16:01:36Z

It's been a little over 2 weeks. Is there something I can do to get this reviewed and moving along?

lglijo · 2018-09-02T01:31:46Z

I so much want this fix.Really waiting to get this to work with windows images on windows boxes.
I approved the PR sometime ago, what is the process to approve and get this merged?

lglijo · 2018-09-02T01:36:36Z

@rbutcher I marked the issue as in progress here JENKINS-36776

rbutcher · 2018-09-17T18:01:20Z

It's been about a month and a half since this was opened. What needs done to get this reviewed and possibly merged?

henryborchers · 2018-09-18T15:49:20Z

Same question as @rbutcher . I want to use Windows containers.

jglick · 2018-09-25T00:40:36Z

I do not think this plugin has a maintainer, and given that every change causes a critical regression, IMO it is best left untouched. (Particularly in this area, since we have no meaningful CI on Windows.) If you want to use Windows server containers in build steps, it is advisable to run Docker commands directly. You do not need any special plugin to do so.

Snapstromegon · 2018-09-25T01:38:21Z

I do not think this plugin has a maintainer, and given that every change causes a critical regression, IMO it is best left untouched.

I think that (as you have a CI and testing process on Linux to ensure nothing gets broken on Linux) its not the right way to break the CI usabilaty or such a large Audience. We were waiting for this for month and missing out this feature is preventing us from using Jenkins, because Docker (or Containers in general) is the only accaptable way for us to have a bundled, recreateable Environment in which we can repeat our CI Process. And since we are required to use Windows only tools, switching to Linux is not an option.

(Particularly in this area, since we have no meaningful CI on Windows.)

IMHO I think, that it's correct to say, that we shouldn't break Linux features to introduce Windows ones. But if I'm correct, it should be possible to test the Linux environment to be unchanged and as Windows currently doesn't have this feature, it's impossible to break it. It might create some new Bugreports, but IMO it's worse to just "ignore" the existing ones to preserve the stability of a feature for a platform it's not available for.

If you want to use Windows server containers in build steps, it is advisable to run Docker commands directly. You do not need any special plugin to do so.

Is there a way to run a Jenkins Plugin inside the container of which I don't know? Because transforming each container to a Jenkins Node creates way to much overhead and running the commands as "docker exec" / "docker run" isn't really usable.

I know, that my answer might seem a little harsh, but I think that theres nothing to break for the Windows CI world and that the gain would be massive, while the risk for Linux would be miniscule.

I hope someone will find the time and gives Jenkins and Docker for Windows some love and merges this (or a compareable) pull and maybe even to become a maintainer for this plugin.

jetersen · 2018-10-09T13:45:37Z

@rbutcher we tested your branch and found that the method for whoAmI did not work because it is calling id function on Windows

rbutcher · 2018-10-09T15:02:39Z

@rbutcher we tested your branch and found that the method for whoAmI did not work because it is calling id function on Windows

You are totally right. When I setup my build machines I put the git /usr/bin on path so that folks can use things like sed.

I am going to push an override to the whoAmI function that will do the same thing, but without relying on id. The result of that function is not super important because it is not used to set the user inside the container because that works funky on Windows. The bind mount still results in correctly ACL'ed files that the worker process can still read.

Another thing to note, if your Jenkins workspace is on a drive other than C:\ you will run into container start issues too. For example, we have a C:\ and an M:\ drive. Jenkins workspace is at M:\j\workspace. The plugin will create the bind mount of M:\j\workspace\<build number>:M:\j\workspace\<build number>. This fails because the container does not know of any M:\ drive.

I am thinking to resolve this I would just replace M:\ in this case when constructing bind mounts. Any thoughts?

Finally, I am planning on maintaining the fork since this does not seem likely to progress. I will leave it open in case @jglick changes his mind.

jetersen · 2018-10-09T16:52:28Z

Tbh on windows you can skip the user parameter so I would rather have it as a noop.
Then skip adding -u to docker run 😜

Snapstromegon · 2018-10-09T18:06:37Z

@Casz Thanks for being active on this PR.
IMHO Skipping the user parameter might be helpful for now, but maybe at some other point it's going to cause some trouble.
It's not recommended to run as su or admin inside the container. For most people this won't be a problem, because they are doing it anyways (and in a controlled environment like a CI system it's less of a problem), but someone will certenly run into unexpected errors when forcing a user.

For now I think skipping will be an okay solution to get Docker on Windows rolling, but someone should look into this in the future if this doesn't get solved in this PR.

jetersen · 2018-10-10T01:44:48Z

If you had to use it you go: docker run -u ContainerUser or docker run -u ContainerAdministrator

-u, --user string                    Username or UID (format: <name|uid>[:<group|gid>])

I rarely know of any windows image that create users beside using the existing.

In this scenario I think the option should be given inside the agent { docker { options } } as user = 'ContainerUser'

rbutcher · 2018-10-10T03:27:52Z

It's not recommended to run as su or admin inside the container. For most people this won't be a problem, because they are doing it anyways (and in a controlled environment like a CI system it's less of a problem), but someone will certainly run into unexpected errors when forcing a user.

I am aware of this. For the Windows containers we are getting ready to run in production we do create non-ContainerAdministrator users and what not. In this case it does not affect the users on the ACLs for the files written while executing in the container so I am less worried about problems. I did add an implementation that does provide a proper username on Windows. For example, on my work system it returns onbase\rbutcher. I would have to create the user in the container beforehand for that to be useful though, so it gets thrown away.

I am however looking for insight into how to handle the path mappings from any other drive to C:. I just don't know how common it is for non-C:\ drives on CI systems. Its common at my job because of antivirus scanning exceptions.

jetersen · 2018-10-10T04:01:27Z

@rbutcher what I would do is just replace M: to C: and then when launcher is inside a docker container use ws() function to switch workspace.

We used this to shorten path because of compiler limitation and hardware limited us to one executor anyway.

So E:/jenkins/workspace/build/multibranch<branch-feature>-<hash> became J:/ with subst.

I don't have access to the source code any longer

But from memory

node ('bluetooth') {
  checkout()
  def workdir = pwd()
  bat "subst J: ${workdir}"
  ws('j:/') {
     bat 'make'
  }
}

Work flawlessly I guess the same would work for bind mount in docker.

solvingj · 2018-11-08T17:17:42Z

@Casz would you be willing to comment on whether there is any chance you will help get this PR through the pipeline and merged?

jetersen · 2018-11-08T21:46:16Z

I have plenty of work to do and we are waiting for 1809 and new dotnet SDK docker image to do any serious work on windows 😅

jglick · 2018-11-14T21:11:15Z

running the commands as "docker exec" / "docker run" isn't really usable

Subjective I guess, but: #105

The ideal outcome from my PoV:

The withDockerRegistry and withDockerServer steps are moved to another plugin, maybe just docker-commons. (They could even be SimpleBuildWrappers were it not for the compatibility code in [JENKINS-51395] Syntax problems in withDockerRegistry #138.)
Someone other than myself or regular Pipeline developers maintains (the rest of) this plugin.
It is removed from the deps of workflow-aggregator and pipeline-model-definition, removed from Evergreen, and all mentions in jenkins.io documentation removed other than to say “do not use this”.
Pending something nicer, [JENKINS-48050] Declarative Pipeline support for dockerNode docker-plugin#681 is listed as the only supported integration of (non-Kubernetes) Docker containers in (Declarative) Pipeline.

But for that I need some commitment from @abayer, @ndeloof, and probably @bitwiseman etc.

basdl · 2019-01-02T07:29:23Z

Is there any chance this PR will actually be merged?

dpn982 · 2019-04-18T16:12:19Z

What is the current status of this pull request?

jetersen · 2019-04-19T16:53:37Z

I have a working branch, and we are using it with great success.
Have yet to finalize the branch there is some work to be done for them to be equal on Windows and Linux I'll see if I can get up next week.

Though for the work to be truly done it would require Windows agent with Docker on Jenkins CI.
I guess I could try and get some appveyor tests going.

bitwiseman · 2019-04-19T16:58:15Z

@dpn982
Are you interested in contributing to this PR?

bitwiseman · 2019-04-19T17:01:33Z

@Casz
We have windows agents on Jenkins CI now right? I assume they lack Docker though.
In any case, would it make sense to push what you have to this PR for folks to look at?

jetersen · 2019-04-19T17:18:36Z

🎯 focused on JCasC atm 😓

jglick

#148 (comment): I think it is unwise to make any but the most trivial changes to this plugin until the scary and misconceived parts are officially marked experimental and removed from standard guidelines.

jglick · 2019-04-22T13:08:20Z

src/main/resources/org/jenkinsci/plugins/docker/workflow/Docker.groovy

@@ -75,6 +75,12 @@ class Docker implements Serializable {
        new Image(this, id)
    }

+    String shell() {
+        node {


Can be deleted.

jglick · 2019-04-22T13:09:33Z

src/main/resources/org/jenkinsci/plugins/docker/workflow/Docker.groovy

@@ -93,7 +99,7 @@ class Docker implements Serializable {
            def commandLine = "docker build -t ${image} ${args}"
            def buildArgs = DockerUtils.parseBuildArgs(commandLine)

-            script.sh commandLine
+            script."${shell()}" commandLine


More simply:

Suggested change

script."${shell()}" commandLine

script[shell()] commandLine

bitwiseman · 2019-04-22T16:36:35Z

@jglick
You're talking about JENKINS-48050, right? Is there plan to land that?

If the general plan is to not make further changes to this plugin that needs to be communicated clearly at the top level of this project and there needs to be a person or group driving the effort to migrate to the new plugin.

In the absence of that, not changing this plugin doesn't seem like an option. People will continue to submit PRs for this plugin and there needs to be a path forward for them. What if anything can be done to make changes less risky?

jglick · 2019-04-22T16:39:20Z

You're talking about JENKINS-48050, right?

Well, not so much, since that seems to have been abandoned. See jenkinsci/docker-plugin#681 (comment) for discussion.

If the general plan is to not make further changes to this plugin that needs to be communicated clearly at the top level of this project and there needs to be a person or group driving the effort to migrate to the new plugin.

Yes and yes!

sboehmann · 2019-05-03T10:51:57Z

Sorry, but this is so frustrating! We worked two weeks to migrate from our old CI to Jenkins and now we've stumbled across this issue. First, we started running Linux builds (with Docker) and Windows builds (without Docker). Then we updated Windows (to Windows Server 2019) to use Docker. And now this! Why is there no warning anywhere that this plugin is dead?

Is there any easy-to-use and maintenable workaround or alternative?

Snapstromegon · 2019-05-03T11:34:46Z

@sboehmann even though I did not test this yet:
As far as I know there is no easy clean way to run pure docker on windows with jenkins.
but
you could try using a kubernetes cluster and add a windows node. This might solve your problem as kubernetes doesn't use that type of volume definition, which is problematic in docker for windows.

Adding a kubernetes Windows Node

Using kubernetes with jenkins works fine and even better when you deploy your jenkins in the cluster itself. But be aware that setting up a kubernetes cluster is harder than a docker swarm.

sboehmann · 2019-05-03T12:26:38Z

you could try using a kubernetes cluster and add a windows node. This might solve your problem as kubernetes doesn't use that type of volume definition, which is problematic in docker for windows.

@Snapstromegon That might work. But it's way too much for us. We build C/C++ software and just wanted to use Jenkins/Docker to have a clean build environment and a simpler setup of the build machines. For Linux this works very well, but Windows is just as important. Therefore Jenkins doesn't seem to be a good choice for us. It was an experiment and unfortunately it failed. We will investigate another CI.

jjathman · 2019-05-03T12:34:23Z

@sboehmann what my company did was just wrote some powershell scripts to emulate what Jenkins does for Linux. This plugin makes working with containers very easy, but for windows you can still execute the same Docker commands. You just need to write it yourself. As long as you use some shared Jenkins scripts, it isn’t too bad. Just a handful of commands.

sboehmann · 2019-05-03T13:00:18Z

@jjathman Yes, we already have a shared library. We already build 50+ repositories - practically always in the same way. So our entire pipeline script is in a shared library.

I'll try your suggestion. But would you be so kind to share your shell script? Or could you give some more hints on how to write such a script? Sorry, but I'm really new to the Windows/Docker universe (I'm more the Unix guy). And it would probably also help others who run into this issue.

ghost · 2019-05-03T13:11:07Z

@sboehmann :

I understand your frustration, we had to solve it as well. That said, we're sticking with Jenkins for now.

Here's how to run Docker builds on Windows 2016/2019.

Edit: I replaced the code with a link to Stackoverflow where I posted the answer.

rgl · 2019-05-03T17:08:46Z

@gyorgynadaban looks very interesting! thanks for sharing!

Can you please explain what is the SCRIPTS environment variable, the initializeWorkspace, publishAllArtifacts and updateJira functions?

Is this kind of Jenkinsfile considered a Declarative Pipeline or Scripted Pipeline?

sboehmann · 2019-05-03T22:37:11Z

@gyorgynadaban Thank you very much. I was really frustrated, sorry for that.
But with your example I almost made it work now. Thanks!

However, I still have one (last?) problem. How can I pull the image from my private registry. Or especially, how can I get the already configured registry (configured for docker-workflow-plugin)?

ghost · 2019-05-03T23:44:26Z

@rgl : I think this is considered a scripted pipeline.

SCRIPTS is just an environment variable, you'd have similar ones based on your actual implementation, that you can pass on to the Docker container. You'd have to implement it as it's not in my example code, but it's definitely not hard.

Function explanations:
initializeWorkspace: You would do pre-build initialization here, e.g. create artifact directories, copy helper scripts to the workspace, etc.
publishAllArtifacts: The way I opted to implement support for various output types is handled here, basically this function looks at a predefined path to see if there's anything to publish.

Eg. if you create directories like this in the workspace:
__artifacts\chocolatey
__artifacts\nuget
__artifacts\npm
__artifacts\tests\nunit
__artifacts\tests\junit
__artifacts\tests\mstest

Then in your publishAllArtifacts function you'd simply scoop up output types and publish to your choice of artifact destination. It helps manage multi-output projects eg. which produce more than one type of output, e.g. mix of Chocolatey packages, NUnit and JUnit test results.

updateJira: If you use Jira issue tracking, you might want to add a comment to the corresponding case with the build results. This is completely optional as well, but the idea is that your branch name can be e.g. your jira case ID.

ghost · 2019-05-03T23:55:18Z

@sboehmann : No worries, I was as frustrated as well before I had the solution worked out, I'm glad I could help. :)

As for image registry, you'd need to do 2 things:

Add your registry to the Docker settings file
If necessary, log into your private registry with the user that's running the build. Alternatively, you can do a Docker login command as part of your init phase, which is what I did.

Your registry would be configured in C:\programdata\Docker\config\daemon.json on your Jenkins node:

{
    "registry-mirrors": [ "https://docker-dev.myserver.net" ], 
    "hosts": [ "tcp://0.0.0.0:2376", "npipe://" ]
}

The authentication step depends on whether your registry requires authentication to get or publish images, and it depends on your individual setup.

rgl · 2019-05-04T00:00:43Z

@gyorgynadaban I see... so those are your custom functions! are you putting them in a shared library like described at https://jenkins.io/doc/book/pipeline/shared-libraries/? or can they also go somewhere in the workspace?

ghost · 2019-05-04T02:40:44Z

@rgl : Yes, those are shared libraries. I found it most practical to have a repository with the shared libraries that are loaded implicitly in the Jenkins job.

solvingj · 2019-06-06T18:51:06Z

@Casz can you share your working branch, or even a built version, so that some of us who are somewhat desperate could deploy it locally to our dev environments and provide feedback sooner rather than later?

jetersen · 2019-06-06T19:04:13Z

I'll try and get around to it tomorrow.

recampbell · 2019-06-19T17:30:14Z

Hi @Casz -- I just wanted to check in here. This is a really exciting feature!

- Fixing haphazard changes to Docker.groovy - Fixing FindBugs issue with annotations on WindowsDockerClient.launch()

clement-cln · 2019-07-30T12:57:19Z

Hi guys, is there any update about the potential merge of this feature ?

jetersen · 2019-08-23T11:05:53Z

here is my updated branch: #184

rbutcher commented Aug 2, 2018

View reviewed changes

rbutcher mentioned this pull request Aug 2, 2018

dont append user and selinux flags on windows slaves #117

Open

lglijo approved these changes Sep 2, 2018

View reviewed changes

jglick reviewed Apr 22, 2019

View reviewed changes

jglick mentioned this pull request May 28, 2019

[JENKINS-48050] introduce "container" agent type jenkinsci/pipeline-model-definition-plugin#255

Closed

rbutcher added 9 commits July 2, 2019 14:39

Added support to run Docker stages on Windows slaves

75e190f

Updated changes to support docker.inside in reference to PR-98

599057f

Cleaning up imports

96d6a58

Fixing build issues

c91f4f8

- Fixing haphazard changes to Docker.groovy - Fixing FindBugs issue with annotations on WindowsDockerClient.launch()

Removing annotations

45eee83

Fixing DockerWindowsClient test_run()

6da111f

Updated whoAmI to properly get user on Windows

a960218

Fixing findBugs issues with default charset

b797434

Fixed imports in WithContainerStep.java

6d22762

jetersen mentioned this pull request Aug 23, 2019

Features/windows agents with pathing issues fixed #184

Merged

oleg-nenashev closed this in #184 Oct 9, 2019

feat/windows slaves #148

feat/windows slaves #148

Conversation

rbutcher commented Aug 2, 2018 • edited Loading

rbutcher Aug 2, 2018

Choose a reason for hiding this comment

rbutcher commented Aug 20, 2018

lglijo commented Sep 2, 2018

lglijo commented Sep 2, 2018

rbutcher commented Sep 17, 2018

henryborchers commented Sep 18, 2018

jglick commented Sep 25, 2018

Snapstromegon commented Sep 25, 2018

jetersen commented Oct 9, 2018

rbutcher commented Oct 9, 2018

jetersen commented Oct 9, 2018 • edited Loading

Snapstromegon commented Oct 9, 2018

jetersen commented Oct 10, 2018

rbutcher commented Oct 10, 2018

jetersen commented Oct 10, 2018 • edited Loading

solvingj commented Nov 8, 2018

jetersen commented Nov 8, 2018

jglick commented Nov 14, 2018

basdl commented Jan 2, 2019

dpn982 commented Apr 18, 2019

jetersen commented Apr 19, 2019 • edited Loading

bitwiseman commented Apr 19, 2019

bitwiseman commented Apr 19, 2019

jetersen commented Apr 19, 2019

jglick left a comment

Choose a reason for hiding this comment

jglick Apr 22, 2019

Choose a reason for hiding this comment

jglick Apr 22, 2019

Choose a reason for hiding this comment

bitwiseman commented Apr 22, 2019

jglick commented Apr 22, 2019

sboehmann commented May 3, 2019

Snapstromegon commented May 3, 2019 • edited Loading

sboehmann commented May 3, 2019

jjathman commented May 3, 2019

sboehmann commented May 3, 2019

ghost commented May 3, 2019 • edited by ghost Loading

rgl commented May 3, 2019

sboehmann commented May 3, 2019

ghost commented May 3, 2019 • edited by ghost Loading

ghost commented May 3, 2019 • edited by ghost Loading

rgl commented May 4, 2019

ghost commented May 4, 2019

solvingj commented Jun 6, 2019

jetersen commented Jun 6, 2019

recampbell commented Jun 19, 2019

clement-cln commented Jul 30, 2019

jetersen commented Aug 23, 2019

rbutcher commented Aug 2, 2018 •

edited

Loading

jetersen commented Oct 9, 2018 •

edited

Loading

jetersen commented Oct 10, 2018 •

edited

Loading

jetersen commented Apr 19, 2019 •

edited

Loading

Snapstromegon commented May 3, 2019 •

edited

Loading

ghost commented May 3, 2019 •

edited by ghost

Loading

ghost commented May 3, 2019 •

edited by ghost

Loading

ghost commented May 3, 2019 •

edited by ghost

Loading