Version 0.8 - incompatibility with Docker 1.4.0 - 'Failed to mkdirs'

[hcguersoy]

Hi,

I tested today the plugin version 0.8 against docker 1.4.0.

Looks at the first glance nice e.g. containers starting very fast.

But trying to build a simple java project fails immediately while trying to install Maven:

FATAL: Failed to install http://archive.apache.org/dist/maven/binaries/apache-maven-3.2.1-bin.zip to /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/3.2.1
java.io.IOException: Failed to install http://archive.apache.org/dist/maven/binaries/apache-maven-3.2.1-bin.zip to /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/3.2.1
    at hudson.FilePath.installIfNecessaryFrom(FilePath.java:822)
    at hudson.tools.DownloadFromUrlInstaller.performInstallation(DownloadFromUrlInstaller.java:70)
    at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:68)
    at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:107)
    at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:205)
    at hudson.tasks.Maven$MavenInstallation.forNode(Maven.java:609)
    at hudson.tasks.Maven.perform(Maven.java:288)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:770)
    at hudson.model.Build$BuildExecution.build(Build.java:199)
    at hudson.model.Build$BuildExecution.doRun(Build.java:160)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:533)
    at hudson.model.Run.execute(Run.java:1745)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:89)
    at hudson.model.Executor.run(Executor.java:240)
Caused by: java.io.IOException: Failed to mkdirs: /home/jenkins/tools/hudson.tasks.Maven_MavenInstallation/3.2.1
    at hudson.FilePath.mkdirs(FilePath.java:1153)
    at hudson.FilePath.installIfNecessaryFrom(FilePath.java:788)
    ... 15 more

Downgrading Docker on my slaves to 1.3.2 helps and the job runs fine again.

[magnayn]

I doubt that this is a plugin problem (NB: the stacktrace is entirely docker plugin free at this point, it's purely jenkins).

[hcguersoy]

I know but strange.
I used the same slave images under 1.4.0 and 1.3.2 on the same machine with the same mounts and parameters.
The path /home/jenkins/tools/ is mounted to a volume.
I sshd'ed into a started slaved container under 1.4.0 and tried if I can create files and dirs as user jenkins and this was without any problems.

Has anyone else observed same behavior?

[magnayn]

If you're feeling brave, you can try a -SNAPSHOT version here:
http://repo.jenkins-ci.org/snapshots/com/nirima/docker-plugin/0.9-SNAPSHOT/

I am successfully running on a fleet of dockers, some are 1.4.1.

[hcguersoy]

Thanks for the hint, I'll test it next week in my test environment and report.

[hcguersoy]

Hi, tested with the 0.9-SNAPSHOT, together with Docker 1.4.1.
Same result.

I'll investigate further - if no one else observes the same maybe this is a issue with the image.

[hcguersoy]

Hi,

I've done intensive testing today (built new images etc., tested with different docker version and so on).

I've still this issue. Maybe it's related to mounting volumes to the containers started by the plugin as the directory's which are provided by the volume in a running build container don't exists but then I start the container manually supplying -v parameter

Doing a inspect shows me this for the slave container:

        "Volumes": {
            "/data/jenkins": {}
        },
        "WorkingDir": ""
    },
    "Created": "2015-01-05T16:12:51.87879333Z",
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": null,
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": null,

And this is from the manually started container:

        "Volumes": {
            "/data/jenkins": {}
        },
        "WorkingDir": ""
    },
    "Created": "2015-01-05T16:30:07.100792311Z",
    "Driver": "aufs",
    "ExecDriver": "native-0.2",
    "HostConfig": {
        "Binds": [
            "/pro/data/jenkins:/data/jenkins:rw"
        ],
        "CapAdd": null,
        "CapDrop": null,
        "ContainerIDFile": "",
        "Devices": [],

The main difference is that in the container started by the plugin the Binds part is empty.

And this difference at the end of inspect:

Generated by plugin:

    "Volumes": {
        "/data/jenkins": "/pro/data/docker/vfs/dir/e7bc8e82f95a3ddd9fc326e74cc80961b22eeca6f0390fabae2570061a2bd70e"
    },
    "VolumesRW": {
        "/data/jenkins": true
    }

manually:

    "Volumes": {
        "/data/jenkins": "/pro/data/jenkins"
    },
    "VolumesRW": {
        "/data/jenkins": true
    }

In this output the volume references to a path in the docker directory.

An example for a jenkins log file:

{"AppArmorProfile":"","Args":["-D"],"Config":{"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["/usr/sbin/sshd",
"-D"],"CpuShares":0,"Cpuset":"","Domainname":"","Entrypoint":null,
"Env":["LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/"],
"ExposedPorts":{"22/tcp":{}},"Hostname":"2b3e38415c1f","Image":"722543a10272",
"MacAddress":"","Memory":0,"MemorySwap":0,"NetworkDisabled":false,
"OnBuild":null,"OpenStdin":false,"PortSpecs":null,"StdinOnce":false,
"Tty":false,"User":"","Volumes":{"/data/jenkins":{}},"WorkingDir":""},
"Created":"2015-01-05T15:55:41.400979521Z","Driver":"aufs","ExecDriver":"native-0.2",
"HostConfig":{"Binds":null,"CapAdd":null,"CapDrop":null,"ContainerIDFile":"",
"Devices":null,"Dns":["10.10.1.1"],"DnsSearch":null,"ExtraHosts":null,"IpcMode":"","Links":null,
"LxcConf":null,"NetworkMode":"","PortBindings":{"22/tcp":[{"HostIp":"0.0.0.0","HostPort":""}]},
"Privileged":true,"PublishAllPorts":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":""},
"SecurityOpt":null,"VolumesFrom":null},
"HostnamePath":"/pro/data/docker/containers/2b3e38415c1f447ce7ef055f552c6a43046fe8dbf9c4e33d59bc37b8a877621f/hostname",
"HostsPath":"/pro/data/docker/containers/2b3e38415c1f447ce7ef055f552c6a43046fe8dbf9c4e33d59bc37b8a877621f/hosts",
"Id":"2b3e38415c1f447ce7ef055f552c6a43046fe8dbf9c4e33d59bc37b8a877621f",
"Image":"722543a102723c3672edec3aa9f28a40cc00735cd31cdf623169cd68551142b6",
"MountLabel":"","Name":"/gloomy_bell","NetworkSettings":{"Bridge":"docker0","Gateway":"172.17.42.1","IPAddress":"172.17.0.3",
"IPPrefixLen":16,"MacAddress":"02:42:ac:11:00:03","PortMapping":null,"Ports":{"22/tcp":[{"HostIp":"0.0.0.0","HostPort":"49154"}]}},"Path":"/usr/sbin/sshd","ProcessLabel":"",
"ResolvConfPath":"/pro/data/docker/containers/2b3e38415c1f447ce7ef055f552c6a43046fe8dbf9c4e33d59bc37b8a877621f/resolv.conf","State":{"Error":"","ExitCode":0,
"FinishedAt":"0001-01-01T00:00:00Z","OOMKilled":false,"Paused":false,"Pid":3202,"Restarting":false,
"Running":true,"StartedAt":"2015-01-05T15:55:41.513761551Z"},
"Volumes":{"/data/jenkins":"/pro/data/docker/vfs/dir/46eb1e381756efa4e8d2927c3e9cb9207b54104ac822a5dc6237160db3cbf9a6"},
"VolumesRW":{"/data/jenkins":true}}

(I've used here the jenkins log for another container creation, not exact the same as the inspected above).

Due to this, my jenkins is unable to create directorys in /data/jenkins/ :-(

Is this maybe a problem with the used java docker client?

[hcguersoy]

BTW, using a data container instead of defining the volume in the cloud definition works.

[hcguersoy]

Is #131 related to this?

[evilbwil]

@hcguersoy: yeah. #131 fixed the mounting for me. without it docker didn't properly mount host-directories into the containers.

#131 needs an update of jDocker as well.

[KostyaSha]

According to description it was fixed, but volumes handling will be also reviewed as part of volumes support PR.

[hcguersoy]

Hi,

just wanted to mention that this issue is fixed in 0.9-rc - tested intensively with Docker 1.6.