Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"bad_certificate" on TLS connection to remote dockerd #825

Open
Osipion opened this issue Jan 27, 2021 · 18 comments
Open

"bad_certificate" on TLS connection to remote dockerd #825

Osipion opened this issue Jan 27, 2021 · 18 comments

Comments

@Osipion
Copy link

Osipion commented Jan 27, 2021

Summary:

I am unable to connect to a remote dockerd instance due to TLS issues.

Versions & Error Messages:

  • docker-plugin version: 1.2.2
  • jenkins version: 2.263.3
  • docker engine version: 20.10.2
  • java version: OpenJDK 1.11.0 (Java 11)
  • OS: ubuntu-sever-20.04
  • issue occurs during cloud setup, before any containers are in use
  • stack trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
	at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
	at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
	at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
	at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
	at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:294)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
Caused: io.netty.handler.codec.DecoderException
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
  • TLS signature algorithms: All certificates use ecdsa-with-SHA256

Details

The remote host (remote.host) with an exposed docker socket is reachable from the Jenkins master (local.host) using:

$ docker -H tcp://remote.host:2376 --tlsverify --tlscacert=root_ca.crt --tlscert=fullchain.pem --tlskey=privkey.pem info
# ... info printed successfully ...

The root_ca.crt has also been imported into local.host's system Java truststore for good measure.

When I attempt to create a new Docker cloud in https://local.host/configureClouds/, I specify the following settings:

name: remote-1
docker host uri: tcp://remote.host:2376
server credentials: X.509 Client Certificate with the privkey.pem, fullchain.pem and root_ca.crt copied into the respective fields. These are all standard X.509 base64 encoded certificates (e.g. not der).

No other fields are changed from their defaults.

I then run 'Test Connection', which results in the stacktrace above.

Notes

  1. Thank you for putting in the effort to develop this plugin. It's much appreciated.
  2. I also tried Yet Another Docker, which also didn't work (same error using the default client)
@pjdarton
Copy link
Member

I've not seen this error first-hand myself - my docker hosts aren't set to use this - so I can't easily reproduce the issue.
If you can provide idiot-proof instructions for setting up the docker side of things to reproduce this issue then there's more chance I'll be able to spare the time to investigate.

...also, if you get the same problem with the yet-another-docker plugin then there's a high chance that the problem doesn't lie within either of them, but in code that's common to both (e.g. the docker-common plugin, or the docker-java library).

...and it's also quite possible that this problem has been triggered by other changes, e.g. to other plugins/libraries/Jenkins-core changes that were done at the same time.
i.e. if this used to work, please describe the setup in which it used to work, as the key to finding the bug is likely to be hidden within the differences.

@dnwe
Copy link

dnwe commented Jan 27, 2021

@Osipion you mentioned fullchain.pem which suggests your client certificate has more than one cert contained within it? I think I'm hitting the same issue as you and I think it might be caused by docker-java/docker-java#1171 — which wasn't fixed until docker-java 3.2.2 under docker-java/docker-java#1371 — and Jenkins is only exposing docker-java 3.1.5 (via the org.jenkins-ci.plugins:docker-java-api wrapper plugin)

@Osipion
Copy link
Author

Osipion commented Jan 27, 2021

@dnwe you're absolutely right, the pki I'm using is provided by smallstep ca, and fullchain.pem is an ACME provisioned certificate that has an intermediary so needs a chain. I guess you're saying this is really an issue with Jenkins not providing an up-to-date docker-java?

@dnwe
Copy link

dnwe commented Jan 28, 2021

@Osipion by coincidence I'm also using smallstep. I did reconfigure my cert to be issued directly from the root.pem, but that still didn't seem to be working for me with the released version. Please can you test that out yourself just to confirm if that does or does not fix it?

I subsequently built my own docker-java hpi at the 3.2.2 level and everything sprang to life (though I'm still using the single leaf cert rather than intermediate+leaf client cert). I can probably push up my forked branch if you're keen to test it (and your Jenkins isn't a production instance etc.) and show you how to compile a .hpi from it

@Osipion
Copy link
Author

Osipion commented Jan 28, 2021

@dnwe - thanks for the kind offer! I'd love to give it a go if you could push your branch. I just tried pulling docker-java-api-plugin and upgrading the docker-java dependency to 3.2.7 but I'm not familiar with building Jenkins plugins and had some issues with slf4j dependency alignment ("Failed while enforcing RequireUpperBoundDeps."). I've "fixed" them, but based on the other pom.xmls I've looked at, probably not in the correct way (I've added the matching versions as top-level dependencies in the pom.xml, but I don't see other plugin poms doing this). So I now have docker-java-api.hpi but not really sure how to test it - guess I'd need to rebuild docker-plugin with it as a dependency?

In any-case, it seems like a key problem with the Jenkins docker plugin ecosystem if docker-java-api isn't actively maintained (I see "This plugin is up for adoption! We are looking for new maintainers." when I visit the plugin page), whilst docker-plugin depends on it. It's quite a simple project as far as I can see, so perhaps this project (docker-plugin) should take it under it's wing? Otherwise the future seems a little uncertain, unless I'm missing something?

@pjdarton
Copy link
Member

  1. If you've got a .hpi file, you can install it into your Jenkins instance via Manage Plugins -> Advanced and upload it there. There's no need to rebuild the docker-plugin - it uses the docker-java library that's provided by the docker-java-api-plugin.

  2. Don't panic
    The Jenkins OSS project has a process to allow someone who wants changes made to obtain access to a plugin that no longer has an active maintainer. That's how Eric got access to the docker-java-api-plugin so that Docker java api 3.1.5 + Add Extra Groups field #777 could be merged last year; if we're lucky, he'll be willing to help bump the version again.
    ...and FYI while I'm officially the maintainer of this plugin, I'm only here because "I touched it last" and that situation is not uncommon - folks get involved, drive things forwards for a while, achieve what they wanted, and then reduce their involvement; eventually someone else picks things up again.

  3. The docker-java-api-plugin is a shared resource; it's used by a few plugins, not just this one; that's why it lives independently and why it isn't "under the wing" of this one. ...and it's why anyone wanting to bump the version of docker-java will need to coordinate with those other plugins in order to ensure that they can test things before it's released. Last time we bumped the docker-java version, it broke the world and required some rapid fix-ups (fortunately simple ones, but urgent) which wasn't fun - there's no guarantee of binary compatibility between versions.

@Osipion
Copy link
Author

Osipion commented Jan 28, 2021

@pjdarton - thanks for the clairification, much appreciated. Yes, I can see how upgrades can break stuff - upgrading to docker-java 3.2.7 led to binary compatibility issues and a NoSuchMethodError exception. This commit removed com.github.dockerjava.netty.NettyDockerCmdExecFactory.withReadTimeout from NettyDockerCmdExecFactory and put it on the abstract base class AbstractDockerCmdExecFactory (it used to be required as part of an interface). This is called here. I assume this has broken the binary expectations? Odd that @dnwe's rebuild with 3.2.2 worked, as the patch went into master in March, but the pull request fixing certificate chains appears to have been merged in April.

I'd appreciate any thoughts on what's the right way to proceed here? I'm limited by having to use our corporate PKI (with intermediaries), and we have a requirement to encrypt and authenticate connections to the remote dockerd. I guess in the short term my primary option is to rebuild the docker-plugin using docker-java 3.2.7 as @dnwe has?

@dnwe
Copy link

dnwe commented Jan 28, 2021

@Osipion can you test with manually uploading the .hpi if you grab it from here?

@Osipion
Copy link
Author

Osipion commented Jan 29, 2021

@dnwe thanks for this - I actually took it from here, just because I could see that it was a result of your PR. I assume they are the same though?

Anyway, I've uploaded the docker-java-api hpi from that PR and I hit the binary compatability issues described above:

WARNING h.i.i.InstallUncaughtExceptionHandler#handleException: Caught unhandled exception with ID 4c2a5cb5-72b8-49bf-9e91-07ba81a3076b
java.lang.NoSuchMethodError: 'com.github.dockerjava.netty.NettyDockerCmdExecFactory com.github.dockerjava.netty.NettyDockerCmdExecFactory.withReadTimeout(java.lang.Integer)'
        at io.jenkins.docker.client.DockerAPI.makeClient(DockerAPI.java:249)
        at io.jenkins.docker.client.DockerAPI.getOrMakeClient(DockerAPI.java:199)
        at io.jenkins.docker.client.DockerAPI.getClient(DockerAPI.java:168)
        at io.jenkins.docker.client.DockerAPI.getClient(DockerAPI.java:151)
        at io.jenkins.docker.client.DockerAPI$DescriptorImpl.doTestConnection(DockerAPI.java:397)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
        at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
        at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
        at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
        at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
Caused: javax.servlet.ServletException
 ...

I think we'll need to rebuild docker-plugin (this repo) with this as a dependency to get it working (if the only thing that's changed is this, it should just re-compile AFAIK - I guess this is a binary level thing as moving from a direct implementation required by an interface to something in a super class wouldn't normally break stuff at the code level IIRC). I was starting to look into this yesterday but found building with a local dependency on the custom docker-java-api.jar (using docker-java 3.2.7) to be difficult. I've not done a huge amount of java and usually use gradle when I do so.

I tried basically just copying the docker-java-api.jar into /lib/docker-java-api.jar, and modifying the pom to look like:

        <dependency>
            <groupId>org.jenkins-ci.plugins</groupId>
            <artifactId>docker-java-api</artifactId>
            <version>3.2.7</version>
            <scope>system</scope>
            <systemPath>${basedir}/lib/docker-java-api.jar</systemPath>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.httpcomponents</groupId>
                    <artifactId>httpcore</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

I then ran mvn clean verify but get loads of errors like:

<docker-plugin-repo>/src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplate.java:[3,33] package com.github.dockerjava.api does not exist
<docker-plugin-repo>/src/main/java/com/nirima/jenkins/plugins/docker/DockerTemplate.java:[4,41] package com.github.dockerjava.api.command does not exist

I think my inexperience with maven and the technical details of the underlying Jenkins plugin format is showing here TBH. Any tips much appreciated.

EDIT: fixed this by installing the jar into my local maven repository rather than using a system file reference. Guess this pulled in the right depencies as they weren't in the jar. Looks like there are some issues that genuinely require refactoring - the DockerClient interface has changed. I'll give it a go.

@Osipion
Copy link
Author

Osipion commented Jan 29, 2021

Right, after some dependency juggling and a handful of code changes, I've built a docker-plugin that works with my PKI. I've made quite a few dependecy changes and, as I say, I'm not 100% they are the best way of doing things. I also can't seem to run all the tests - the ssh based ones appear to fail. Here are some diffs:

docker-java-api-plugin

diff --git a/pom.xml b/pom.xml
index f0a41b0..92a515f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -9,7 +9,7 @@
     </parent>
 
     <artifactId>docker-java-api</artifactId>
-    <version>${revision}.3${changelist}</version>
+    <version>3.2.7</version>
     <packaging>hpi</packaging>
 
     <name>Docker API Plugin</name>
@@ -20,13 +20,13 @@
         <connection>scm:git:ssh://github.com/jenkinsci/docker-java-api-plugin.git</connection>
         <developerConnection>scm:git:ssh://git@github.com/jenkinsci/docker-java-api-plugin.git</developerConnection>
         <url>https://github.com/jenkinsci/docker-java-api-plugin</url>
-        <tag>${scmTag}</tag>
+        <tag>docker-java-api-3.2.7</tag>
     </scm>
 
     <properties>
-        <revision>3.1.5</revision>
+        <revision>3.2.7</revision>
         <changelist>-SNAPSHOT</changelist>
-        <jenkins.version>1.609.1</jenkins.version>
+        <jenkins.version>2.263.3</jenkins.version>
         <java.level>8</java.level>
         <hpi.compatibleSinceVersion>3.1</hpi.compatibleSinceVersion>
     </properties>
@@ -68,10 +68,10 @@
                 </exclusion>
 
                 <!-- use the version supplied by bouncycastle-api -->
-                <exclusion>
-                    <groupId>org.bouncycastle</groupId>
-                    <artifactId>bcpkix-jdk15on</artifactId>
-                </exclusion>
+<!--                <exclusion>-->
+<!--                    <groupId>org.bouncycastle</groupId>-->
+<!--                    <artifactId>bcpkix-jdk15on</artifactId>-->
+<!--                </exclusion>-->
 
                 <!-- use versions supplied with Jenkins -->
                 <exclusion>
@@ -124,7 +124,27 @@
         <dependency>
             <groupId>org.jenkins-ci.plugins</groupId>
             <artifactId>jackson2-api</artifactId>
-            <version>2.6.4</version>
+            <version>2.10.3</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>jcl-over-slf4j</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>log4j-over-slf4j</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-api</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-jdk14</artifactId>
+            <version>1.7.30</version>
         </dependency>
     </dependencies>
 </project>

docker-plugin:

diff --git a/pom.xml b/pom.xml
index 2811c7e..5ae2cad 100644
--- a/pom.xml
+++ b/pom.xml
@@ -55,20 +55,20 @@
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <java.level>8</java.level>
         <groovy.version>2.4.7</groovy.version>
-        <jenkins.version>2.73.3</jenkins.version>
+        <jenkins.version>2.263.3</jenkins.version>
     </properties>
 
     <dependencies>
         <dependency>
             <groupId>org.jenkins-ci.plugins</groupId>
             <artifactId>bouncycastle-api</artifactId>
-            <version>2.16.2</version>
+            <version>2.18</version>
         </dependency>
 
         <dependency>
             <groupId>org.jenkins-ci.plugins</groupId>
             <artifactId>docker-java-api</artifactId>
-            <version>3.1.5</version>
+            <version>3.2.7</version>
             <exclusions>
                 <exclusion>
                     <groupId>org.apache.httpcomponents</groupId>
@@ -108,15 +108,20 @@
         <dependency>
             <groupId>org.jenkins-ci.plugins.workflow</groupId>
             <artifactId>workflow-api</artifactId>
-            <version>2.23.1</version>
+            <version>2.40</version>
             <optional>true</optional>
         </dependency>
         <dependency>
             <groupId>org.jenkins-ci.plugins.workflow</groupId>
             <artifactId>workflow-step-api</artifactId>
-            <version>2.14</version>
+            <version>2.16</version>
             <optional>true</optional>
         </dependency>
+        <dependency>
+            <groupId>org.jenkins-ci.plugins</groupId>
+            <artifactId>mailer</artifactId>
+            <version>1.20</version>
+        </dependency>
         <dependency>
             <groupId>org.jenkins-ci.plugins.workflow</groupId>
             <artifactId>workflow-support</artifactId>
@@ -176,7 +181,7 @@
         <dependency>
             <groupId>org.jenkins-ci.modules</groupId>
             <artifactId>instance-identity</artifactId>
-            <version>2.1</version>
+            <version>2.2</version>
         </dependency>
         <dependency>
             <groupId>com.google.code.findbugs</groupId>
@@ -204,6 +209,36 @@
             <version>2.6</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.jenkins-ci.plugins</groupId>
+            <artifactId>jackson2-api</artifactId>
+            <version>2.10.3</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>jcl-over-slf4j</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>log4j-over-slf4j</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-api</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.slf4j</groupId>
+            <artifactId>slf4j-jdk14</artifactId>
+            <version>1.7.30</version>
+        </dependency>
+        <dependency>
+            <groupId>org.jenkins-ci</groupId>
+            <artifactId>trilead-ssh2</artifactId>
+            <version>build-217-jenkins-27</version>
+        </dependency>
     </dependencies>
 
     <dependencyManagement>
@@ -211,7 +246,7 @@
             <dependency>
                 <groupId>org.jenkins-ci.plugins</groupId>
                 <artifactId>structs</artifactId>
-                <version>1.9</version>
+                <version>1.20</version>
             </dependency>
         </dependencies>
     </dependencyManagement>
diff --git a/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java b/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java
index 1b56a9c..ecfe160 100644
--- a/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java
+++ b/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java
@@ -17,6 +17,7 @@ import com.github.dockerjava.api.command.CopyFileFromContainerCmd;
 import com.github.dockerjava.api.command.CreateContainerCmd;
 import com.github.dockerjava.api.command.CreateImageCmd;
 import com.github.dockerjava.api.command.CreateNetworkCmd;
+import com.github.dockerjava.api.command.CreateSecretCmd;
 import com.github.dockerjava.api.command.CreateServiceCmd;
 import com.github.dockerjava.api.command.CreateVolumeCmd;
 import com.github.dockerjava.api.command.DisconnectFromNetworkCmd;
@@ -38,6 +39,7 @@ import com.github.dockerjava.api.command.LeaveSwarmCmd;
 import com.github.dockerjava.api.command.ListContainersCmd;
 import com.github.dockerjava.api.command.ListImagesCmd;
 import com.github.dockerjava.api.command.ListNetworksCmd;
+import com.github.dockerjava.api.command.ListSecretsCmd;
 import com.github.dockerjava.api.command.ListServicesCmd;
 import com.github.dockerjava.api.command.ListSwarmNodesCmd;
 import com.github.dockerjava.api.command.ListTasksCmd;
@@ -53,11 +55,15 @@ import com.github.dockerjava.api.command.PushImageCmd;
 import com.github.dockerjava.api.command.RemoveContainerCmd;
 import com.github.dockerjava.api.command.RemoveImageCmd;
 import com.github.dockerjava.api.command.RemoveNetworkCmd;
+import com.github.dockerjava.api.command.RemoveSecretCmd;
 import com.github.dockerjava.api.command.RemoveServiceCmd;
 import com.github.dockerjava.api.command.RemoveVolumeCmd;
 import com.github.dockerjava.api.command.RenameContainerCmd;
+import com.github.dockerjava.api.command.ResizeContainerCmd;
+import com.github.dockerjava.api.command.ResizeExecCmd;
 import com.github.dockerjava.api.command.RestartContainerCmd;
 import com.github.dockerjava.api.command.SaveImageCmd;
+import com.github.dockerjava.api.command.SaveImagesCmd;
 import com.github.dockerjava.api.command.SearchImagesCmd;
 import com.github.dockerjava.api.command.StartContainerCmd;
 import com.github.dockerjava.api.command.StatsCmd;
@@ -75,6 +81,7 @@ import com.github.dockerjava.api.exception.DockerException;
 import com.github.dockerjava.api.model.AuthConfig;
 import com.github.dockerjava.api.model.Identifier;
 import com.github.dockerjava.api.model.PruneType;
+import com.github.dockerjava.api.model.SecretSpec;
 import com.github.dockerjava.api.model.ServiceSpec;
 import com.github.dockerjava.api.model.SwarmSpec;
 
@@ -195,6 +202,11 @@ public class DelegatingDockerClient implements DockerClient {
         return getDelegate().execCreateCmd(arg0);
     }
 
+    @Override
+    public ResizeExecCmd resizeExecCmd(String execId) {
+        return getDelegate().resizeExecCmd(execId);
+    }
+
     @Override
     public ExecStartCmd execStartCmd(String arg0) {
         return getDelegate().execStartCmd(arg0);
@@ -320,11 +332,21 @@ public class DelegatingDockerClient implements DockerClient {
         return getDelegate().restartContainerCmd(arg0);
     }
 
+    @Override
+    public ResizeContainerCmd resizeContainerCmd(String containerId) {
+        return getDelegate().resizeContainerCmd(containerId);
+    }
+
     @Override
     public SaveImageCmd saveImageCmd(String arg0) {
         return getDelegate().saveImageCmd(arg0);
     }
 
+    @Override
+    public SaveImagesCmd saveImagesCmd() {
+        return getDelegate().saveImagesCmd();
+    }
+
     @Override
     public SearchImagesCmd searchImagesCmd(String arg0) {
         return getDelegate().searchImagesCmd(arg0);
@@ -454,4 +476,19 @@ public class DelegatingDockerClient implements DockerClient {
     public PruneCmd pruneCmd(PruneType pruneType) {
         return getDelegate().pruneCmd(pruneType);
     }
+
+    @Override
+    public ListSecretsCmd listSecretsCmd() {
+        return getDelegate().listSecretsCmd();
+    }
+
+    @Override
+    public CreateSecretCmd createSecretCmd(SecretSpec secretSpec) {
+        return getDelegate().createSecretCmd(secretSpec);
+    }
+
+    @Override
+    public RemoveSecretCmd removeSecretCmd(String secretId) {
+        return getDelegate().removeSecretCmd(secretId);
+    }
 }
diff --git a/src/main/java/io/jenkins/docker/client/DockerAPI.java b/src/main/java/io/jenkins/docker/client/DockerAPI.java
index 9fbf85f..d2771a7 100644
--- a/src/main/java/io/jenkins/docker/client/DockerAPI.java
+++ b/src/main/java/io/jenkins/docker/client/DockerAPI.java
@@ -245,7 +245,7 @@ public class DockerAPI extends AbstractDescribableImpl<DockerAPI> {
         NettyDockerCmdExecFactory cmdExecFactory = null;
         DockerClient actualClient = null;
         try {
-            cmdExecFactory = new NettyDockerCmdExecFactory()
+            cmdExecFactory = (NettyDockerCmdExecFactory) new NettyDockerCmdExecFactory()
                     .withReadTimeout(readTimeoutInMillisecondsOrNull)
                     .withConnectTimeout(connectTimeoutInMillisecondsOrNull);
             final DefaultDockerClientConfig.Builder configBuilder = new DefaultDockerClientConfig.Builder()

"Test Connection" now works for me. Not sure how you'd like me to proceed with upstreaming these changes?

@pjdarton
Copy link
Member

@Osipion I took at look at the docker-java commit you referenced - yup, that's an API-breaking change.
However, if that's all there is to it, it should be easy to adapt to it.
In DockerClient.java, between line 4 & 5, add:
import com.github.dockerjava.api.command.DockerCmdExecFactory;
and on line 245 (which is now line 246), change NettyDockerCmdExecFactory cmdExecFactory = null; to be DockerCmdExecFactory cmdExecFactory = null;
That ought to be enough to cope with the API change.

One thing I would point out though is that this discussion is alternating between talking about docker-java version 3.2.2 and 3.2.7.
Bumping to 3.2.2 may be sufficient to fix this issue, but going further and bumping to 3.2.7 may introduce more issues than it solves.
I would suggest that it'd be best to keep the scope of this issue as narrow as possible and "just fix the issue at hand" ... and then maybe raise follow-on issues/PRs to bring both plugins fully up to date with the most recent stable docker-java release.

@pjdarton
Copy link
Member

Edit: just seen your later msg - I'm currently struggling with IT issues woth now (laptop has decided it doesn't have an external kb or mouse anymore). Will re-read once I've turned it all off and on again.

@Osipion
Copy link
Author

Osipion commented Jan 29, 2021

@pjdarton - thanks. I'm happy to stick with 3.2.2 - I've just rebuilt both docker-java-api and docker-plugin to use docker-java 3.2.2 and does seem to fix the issue. I also had to update much fewer dependencies in docker-plugin to get it there, so that's a win I think - diff below. Two things to note:

  1. I can't seem to get DockerComputerSSHConnectorTest tests to pass on either 3.2.2/3.2.7. Always get a timeout. This may have something to do with the trilead-ssh2 dependency?
  2. I also seem to have to remove the bouncycastle exclusion in docker-java-api otherwise I get java.lang.NoClassDefFoundError: org/bouncycastle/jce/provider/BouncyCastleProvider.

diffs

docker-java-api

diff --git a/pom.xml b/pom.xml
index f0a41b0..00de5bd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -12,6 +12,38 @@
     <version>${revision}.3${changelist}</version>
     <packaging>hpi</packaging>
 
+    <dependencyManagement>
+        <dependencies>
+            <dependency>
+                <groupId>io.jenkins.tools.bom</groupId>
+                <artifactId>bom-2.235.x</artifactId>
+                <version>21</version>
+                <scope>import</scope>
+                <type>pom</type>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>jcl-over-slf4j</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>log4j-over-slf4j</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-api</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-jdk14</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
     <name>Docker API Plugin</name>
     <description>Plugin providing Docker API for other plugins. Wrap docker-java and required dependency for Netty transport only. JAX-RS default transport is not usable</description>
     <url>https://github.com/jenkinsci/docker-java-api-plugin</url>
@@ -24,9 +56,9 @@
     </scm>
 
     <properties>
-        <revision>3.1.5</revision>
+        <revision>3.2.2</revision>
         <changelist>-SNAPSHOT</changelist>
-        <jenkins.version>1.609.1</jenkins.version>
+        <jenkins.version>2.235.1</jenkins.version>
         <java.level>8</java.level>
         <hpi.compatibleSinceVersion>3.1</hpi.compatibleSinceVersion>
     </properties>
@@ -68,10 +100,10 @@
                 </exclusion>
 
                 <!-- use the version supplied by bouncycastle-api -->
-                <exclusion>
-                    <groupId>org.bouncycastle</groupId>
-                    <artifactId>bcpkix-jdk15on</artifactId>
-                </exclusion>
+<!--                <exclusion>-->
+<!--                    <groupId>org.bouncycastle</groupId>-->
+<!--                    <artifactId>bcpkix-jdk15on</artifactId>-->
+<!--                </exclusion>-->
 
                 <!-- use versions supplied with Jenkins -->
                 <exclusion>
@@ -124,7 +156,6 @@
         <dependency>
             <groupId>org.jenkins-ci.plugins</groupId>
             <artifactId>jackson2-api</artifactId>
-            <version>2.6.4</version>
         </dependency>
     </dependencies>
 </project>

docker-plugin

diff --git a/pom.xml b/pom.xml
index 2811c7e..05e4f65 100644
--- a/pom.xml
+++ b/pom.xml
@@ -55,20 +55,20 @@
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <java.level>8</java.level>
         <groovy.version>2.4.7</groovy.version>
-        <jenkins.version>2.73.3</jenkins.version>
+        <jenkins.version>2.235.1</jenkins.version>
     </properties>
 
     <dependencies>
         <dependency>
             <groupId>org.jenkins-ci.plugins</groupId>
             <artifactId>bouncycastle-api</artifactId>
-            <version>2.16.2</version>
+            <version>2.18</version>
         </dependency>
 
         <dependency>
             <groupId>org.jenkins-ci.plugins</groupId>
             <artifactId>docker-java-api</artifactId>
-            <version>3.1.5</version>
+            <version>3.2.2</version>
             <exclusions>
                 <exclusion>
                     <groupId>org.apache.httpcomponents</groupId>
@@ -176,7 +176,7 @@
         <dependency>
             <groupId>org.jenkins-ci.modules</groupId>
             <artifactId>instance-identity</artifactId>
-            <version>2.1</version>
+            <version>2.2</version>
         </dependency>
         <dependency>
             <groupId>com.google.code.findbugs</groupId>
@@ -204,6 +204,16 @@
             <version>2.6</version>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.jenkins-ci</groupId>
+            <artifactId>trilead-ssh2</artifactId>
+            <version>build-217-jenkins-27</version>
+        </dependency>
+        <dependency>
+            <groupId>org.json</groupId>
+            <artifactId>json</artifactId>
+            <version>20190722</version>
+        </dependency>
     </dependencies>
 
     <dependencyManagement>
@@ -213,6 +223,26 @@
                 <artifactId>structs</artifactId>
                 <version>1.9</version>
             </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>jcl-over-slf4j</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>log4j-over-slf4j</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-api</artifactId>
+                <version>1.7.30</version>
+            </dependency>
+            <dependency>
+                <groupId>org.slf4j</groupId>
+                <artifactId>slf4j-jdk14</artifactId>
+                <version>1.7.30</version>
+            </dependency>
         </dependencies>
     </dependencyManagement>
 
diff --git a/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java b/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java
index 1b56a9c..757f472 100644
--- a/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java
+++ b/src/main/java/io/jenkins/docker/client/DelegatingDockerClient.java
@@ -17,6 +17,7 @@ import com.github.dockerjava.api.command.CopyFileFromContainerCmd;
 import com.github.dockerjava.api.command.CreateContainerCmd;
 import com.github.dockerjava.api.command.CreateImageCmd;
 import com.github.dockerjava.api.command.CreateNetworkCmd;
+import com.github.dockerjava.api.command.CreateSecretCmd;
 import com.github.dockerjava.api.command.CreateServiceCmd;
 import com.github.dockerjava.api.command.CreateVolumeCmd;
 import com.github.dockerjava.api.command.DisconnectFromNetworkCmd;
@@ -29,6 +30,7 @@ import com.github.dockerjava.api.command.InspectContainerCmd;
 import com.github.dockerjava.api.command.InspectExecCmd;
 import com.github.dockerjava.api.command.InspectImageCmd;
 import com.github.dockerjava.api.command.InspectNetworkCmd;
+import com.github.dockerjava.api.command.ListSecretsCmd;
 import com.github.dockerjava.api.command.InspectServiceCmd;
 import com.github.dockerjava.api.command.InspectSwarmCmd;
 import com.github.dockerjava.api.command.InspectVolumeCmd;
@@ -53,11 +55,15 @@ import com.github.dockerjava.api.command.PushImageCmd;
 import com.github.dockerjava.api.command.RemoveContainerCmd;
 import com.github.dockerjava.api.command.RemoveImageCmd;
 import com.github.dockerjava.api.command.RemoveNetworkCmd;
+import com.github.dockerjava.api.command.RemoveSecretCmd;
 import com.github.dockerjava.api.command.RemoveServiceCmd;
 import com.github.dockerjava.api.command.RemoveVolumeCmd;
 import com.github.dockerjava.api.command.RenameContainerCmd;
+import com.github.dockerjava.api.command.ResizeContainerCmd;
+import com.github.dockerjava.api.command.ResizeExecCmd;
 import com.github.dockerjava.api.command.RestartContainerCmd;
 import com.github.dockerjava.api.command.SaveImageCmd;
+import com.github.dockerjava.api.command.SaveImagesCmd;
 import com.github.dockerjava.api.command.SearchImagesCmd;
 import com.github.dockerjava.api.command.StartContainerCmd;
 import com.github.dockerjava.api.command.StatsCmd;
@@ -75,6 +81,7 @@ import com.github.dockerjava.api.exception.DockerException;
 import com.github.dockerjava.api.model.AuthConfig;
 import com.github.dockerjava.api.model.Identifier;
 import com.github.dockerjava.api.model.PruneType;
+import com.github.dockerjava.api.model.SecretSpec;
 import com.github.dockerjava.api.model.ServiceSpec;
 import com.github.dockerjava.api.model.SwarmSpec;
 
@@ -315,6 +322,16 @@ public class DelegatingDockerClient implements DockerClient {
         return getDelegate().renameContainerCmd(arg0);
     }
 
+    @Override
+    public ResizeContainerCmd resizeContainerCmd(String containerId) {
+        return getDelegate().resizeContainerCmd(containerId);
+    }
+
+    @Override
+    public ResizeExecCmd resizeExecCmd(String execId) {
+        return getDelegate().resizeExecCmd(execId);
+    }
+
     @Override
     public RestartContainerCmd restartContainerCmd(String arg0) {
         return getDelegate().restartContainerCmd(arg0);
@@ -325,6 +342,11 @@ public class DelegatingDockerClient implements DockerClient {
         return getDelegate().saveImageCmd(arg0);
     }
 
+    @Override
+    public SaveImagesCmd saveImagesCmd() {
+        return getDelegate().saveImagesCmd();
+    }
+
     @Override
     public SearchImagesCmd searchImagesCmd(String arg0) {
         return getDelegate().searchImagesCmd(arg0);
@@ -454,4 +476,19 @@ public class DelegatingDockerClient implements DockerClient {
     public PruneCmd pruneCmd(PruneType pruneType) {
         return getDelegate().pruneCmd(pruneType);
     }
+
+    @Override
+    public ListSecretsCmd listSecretsCmd() {
+        return getDelegate().listSecretsCmd();
+    }
+
+    @Override
+    public CreateSecretCmd createSecretCmd(SecretSpec secretSpec) {
+        return getDelegate().createSecretCmd(secretSpec);
+    }
+
+    @Override
+    public RemoveSecretCmd removeSecretCmd(String secretId) {
+        return getDelegate().removeSecretCmd(secretId);
+      }
 }
diff --git a/src/main/java/io/jenkins/docker/client/DockerAPI.java b/src/main/java/io/jenkins/docker/client/DockerAPI.java
index 9fbf85f..2c923a0 100644
--- a/src/main/java/io/jenkins/docker/client/DockerAPI.java
+++ b/src/main/java/io/jenkins/docker/client/DockerAPI.java
@@ -2,6 +2,7 @@ package io.jenkins.docker.client;
 
 import com.cloudbees.plugins.credentials.domains.DomainRequirement;
 import com.github.dockerjava.api.DockerClient;
+import com.github.dockerjava.api.command.DockerCmdExecFactory;
 import com.github.dockerjava.api.command.VersionCmd;
 import com.github.dockerjava.api.model.Version;
 import com.github.dockerjava.core.DefaultDockerClientConfig;
@@ -242,7 +243,7 @@ public class DockerAPI extends AbstractDescribableImpl<DockerAPI> {
     @SuppressWarnings("resource")
     private static SharableDockerClient makeClient(final String dockerUri, final String credentialsId,
             final Integer readTimeoutInMillisecondsOrNull, final Integer connectTimeoutInMillisecondsOrNull) {
-        NettyDockerCmdExecFactory cmdExecFactory = null;
+        DockerCmdExecFactory cmdExecFactory = null;
         DockerClient actualClient = null;
         try {
             cmdExecFactory = new NettyDockerCmdExecFactory()

@pjdarton
Copy link
Member

Looks like I'm playing catch-up here and you're a way ahead of me right now (I've just done a "mvn install" of the docker-java-api-plugin version 3.2.2, bumped the dependency in docker-plugin and then found that "mvn compile" on the docker-plugin reports loads of slf4j dependency issues).

What I'd suggest/ask is that you submit a pull-request for this plugin that contains all your changes above.
Sure, the PR won't build on jenkinsci until docker-java-api-plugin version 3.2.2 is out, but I'll be able to pull your changes and build & test it locally (as I have docker-java-api version 3.2.2 in my mvn cache locally).

I can test that on my local PC, and then on a "mostly unimportant" Jenkins server at work ... then more important ones if things work OK.
We can also pressure/advise other plugin maintainers who are dependent on docker-java-api to do the same.
Once everyone's happy they have all the code changes they need in a PR "ready" then we can do a coordinated release of a new docker-java-api-plugin and any affected plugins.

@pjdarton
Copy link
Member

pjdarton commented Jan 29, 2021

Note: If anyone knows how we could generate DelegatingDockerClient at runtime then that would be way better than having to keep adjusting the code to match docker-java...

@Osipion
Copy link
Author

Osipion commented Jan 29, 2021

@pjdarton - thanks again 🙂 . I've created a PR so you can pull my diffs - just remember that in addition to @dnwe 's changes to docker-java-api, I've also commented out the bouncycastle exclusion in the pom.xml. This definitely feels wrong as I think it should come from bouncycastle-api, but it failed when I ran it on my Jenkins server without this change.

@pshirshov
Copy link

Got into the same problem, is there any news on this?

@pjdarton
Copy link
Member

I suspect that this issue will've been fixed by the docker-java upgrade that was done recently.
Can anyone here confirm?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants