[Support] Fabric8 IO client creation is failing with below stacktrace

[srinivasev]

Describe the bug

Hi Team,

I am facing below error during spring boot startup. Any idea what could be the issue ?.
Below dependency is used in my spring boot project.

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-kubernetes-fabric8-config</artifactId>
    </dependency>

Stack trace -

14:36:37.314 ERROR SpringApplication 835 Application run failed io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:103) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:97) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:253) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:85) at io.fabric8.kubernetes.client.BaseClient.<init>(BaseClient.java:53) at io.fabric8.kubernetes.client.BaseClient.<init>(BaseClient.java:45) at io.fabric8.kubernetes.client.BaseKubernetesClient.<init>(BaseKubernetesClient.java:151) at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:32) at org.springframework.cloud.kubernetes.fabric8.profile.Fabric8ProfileEnvironmentPostProcessor.isInsideKubernetes(Fabric8ProfileEnvironmentPostProcessor.java:29) at org.springframework.cloud.kubernetes.commons.profile.AbstractKubernetesProfileEnvironmentPostProcessor.addKubernetesProfileIfMissing(AbstractKubernetesProfileEnvironmentPostProcessor.java:81) at org.springframework.cloud.kubernetes.commons.profile.AbstractKubernetesProfileEnvironmentPostProcessor.postProcessEnvironment(AbstractKubernetesProfileEnvironmentPostProcessor.java:66) at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:102) at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:87) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85) at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120) at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114) at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65) at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:343) at org.springframework.boot.SpringApplication.run(SpringApplication.java:301) at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164) at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:195) at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:114) at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:77) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85) at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120) at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114) at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65) at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:343) at org.springframework.boot.SpringApplication.run(SpringApplication.java:301) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1317) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) at com.ericsson.oss.common.service.ns.NotificationServiceApplication.main(NotificationServiceApplication.java:35) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:115) at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355) at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:98) at io.fabric8.kubernetes.client.internal.CertUtils.createTrustStore(CertUtils.java:74) at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:115) at io.fabric8.kubernetes.client.internal.SSLUtils.trustManagers(SSLUtils.java:91) at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:147) ... 49 common frames omitted Caused by: java.io.IOException: Empty input at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:111) ... 55 common frames omitted

Fabric8 Kubernetes Client version

6.0.0

Steps to reproduce

Need to understand more on the stacktrace debugging.

Expected behavior

Microservice pod should be up and running.

Runtime

other (please specify in additional context)

Kubernetes API Server version

1.23

Environment

other (please specify in additional context)

Fabric8 Kubernetes Client Logs

No response

Additional context

kubernetes in openstack environment

[srinivasev]

Hi Team,
I checked the kubernetes cluster and the Fabric8 IO compatibility matrix - [https://github.com/fabric8io/kubernetes-client#compatibility-matrix]

Below are the findings -

K8s version = 1.24
Fabric8 IO =

<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
<version>5.10.2</version>

Is this incompatibility is causing above failure while creating the client ?

[shawkins]

Closing to avoid cross-posting with #4653

[srinivasev]

Hi @shawkins / @manusa ,

The issue was because the ca.crt in path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt having comments with it which was resulting in empty input exception.

Exception in thread "main" java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input

Below is the certificate details.

-----BEGIN CERTIFICATE-----
MIIDAzCCAeugAwIBAgIJAMtSC6dXApfDMA0GCSqGSIb3DQEBCwUAMBgxFjAUBgNV
BAMMDUt1YmVybmV0ZXMtY2EwHhcNMjEwMzEwMTAzNDM5WhcNNDgwNzI2MTAzNDM5
WjAYMRYwFAYDVQQDDA1LdWJlcm5ldGVzLWNhMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1ox+fKoEbLI+My5DGgV97zm6hseCNS/WsCOuoPFPaJR5sHnH
P+4xoX/Z72J0pCw6q6JfVJEjwfdioNvafBCNwpoyy2zzP3KS0miWGpyfjdgxUs+J
nrNihYeOibuWtvig10352BsMNGf6AsFMe3svBjBv8Ae3c6aKW4j2nZdiiqm0EqRO
mw3x1gAAUitb6eVrmr/wqNkXDQoGgmyII+XZkj24ZrSE92JyK2c0tOZkhN2AO37I
l6uC9AP7+VOgY3FLNKgKyxdYbEoJm7gTTf8VC3+v8Xec2ba/+S0HurHxBEG4tJIQ
/Az18X2pYXRlrs5Ntng89hmqfxbX1WimYDC3+wIDAQABo1AwTjAdBgNVHQ4EFgQU
SGC8p9FPwF6gqUyuqJCvNPlc33owHwYDVR0jBBgwFoAUSGC8p9FPwF6gqUyuqJCv
NPlc33owDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAhVlnqJTmw1gA
wYkPZC7XoFp7RYAX6JLtfrqUPFMjQMm9znH3BOsJsKkaihC5GjiBM0flN2Vg1LJ+
Ne0EhFclGTaXZ9ftZMr70hQ3BfrrXQVd6eOHZqw3K2ixguJ9HRRIpiIGJl0wYcQf
FGZw+DWnh6NMG833yz+xi+/FxMvYOsGSR8k5bNjlDwzBugkEqZhHUtfzKsoLA5rO
NgbHTyGME3zCzbmyVTBX9kOANp45G/RueJRrY3bm6MT79sSck1QyJlF54zGJ3ahA
L0FmGwf3/Gegbs3dn2h0ChpHkZLtNfu98tr0TcZYCCFf31CuD9O4W4CdnEXOqxU6
J1SeZCLVlQ==
-----END CERTIFICATE-----

#ca_cert: <fill>
#ca_cert: |
#-----BEGIN CERTIFICATE-----
#MIJJEE387484838475asjdNHFJRUUUGGF2497932759834793hgjrttifkf92298
#.
#.
#.
#-----END CERTIFICATE-----

Could you please let me know why the fabric8 io is not ignoring the comments present in the end of the certificate and reading everything while creating the x509 certificate instance in its code..

The below is block of code throwing the exception when pemInputStream is having commented lines.. CertUtils.java is the class.

while (pemInputStream.available() > 0) {
CertificateFactory certFactory = CertificateFactory.getInstance("X509");
X509Certificate cert = (X509Certificate) certFactory.generateCertificate(pemInputStream);
String alias = cert.getSubjectX500Principal().getName() + "_" + cert.getSerialNumber().toString(16);
trustStore.setCertificateEntry(alias, cert);
}
return trustStore;

Please let us know on this.
Also let me know how I can reopen this issue as this is currently in CLOSED state.

[shawkins]

Could you please let me know why the fabric8 io is not ignoring the comments present in the end of the certificate and reading everything while creating the x509 certificate instance in its code..

This seems to be a problem only for the trailing entry in the file - if the comment appears in any other place, no exception will be thrown.

[srinivasev]

Hi @shawkins ,
Thanks for the fix. Hope as per your response, it will work fine now when the comments are present in the trailing / end of the file. Which version of maven library should I use to adopt this change ?.

[shawkins]

Which version of maven library should I use to adopt this change ?.

See #4716 - it has been targeted for 6.4, but not committed yet so there isn't a snapshot with it available. Feel free to compile from that branch if you want the fix sooner.