fix(tls): update existing certificates #360

ebaron · 2022-03-31T21:00:17Z

Uses controllerutil.CreateOrUpdate on certificates/issuers/secrets created in certmanager.go. I've also added some additional tests for this code.

To test:

make deploy IMG=quay.io/ebaron/cryostat-operator:before-352
make create_cryostat_cr
<wait for cryostat-sample-grafana-tls secret to be created>
kubectl patch deploy cryostat-operator-controller-manager -p \
'{"spec":{ "template": {"spec": {"containers": [{"name": "manager", "image": "quay.io/ebaron/cryostat-operator:after-352-update"}]}}}}'
<redeployed Cryostat container logs should show no CertificateException>

quay.io/ebaron/cryostat-operator:before-352 is a build before #352
quay.io/ebaron/cryostat-operator:after-352-update is a build of this PR, after #352

Fixes: #359

andrewazores · 2022-04-01T20:21:26Z

I followed the given test steps and I see exceptions like the following in the main Cryostat container logs when I try to open the web-client:

Apr 01, 2022 8:19:49 PM io.cryostat.core.log.Logger info
INFO: (10.217.0.1:33976): GET /health 200 41ms
Apr 01, 2022 8:19:52 PM io.cryostat.core.log.Logger info
INFO: (10.217.0.1:33754): GET /api/v1/notifications_url 200 0ms
Apr 01, 2022 8:19:52 PM io.cryostat.core.log.Logger error
SEVERE: HTTP 500: Internal Server Error
io.cryostat.net.web.http.api.v2.ApiException: Internal Server Error
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at io.cryostat.net.OpenShiftAuthManager.getLoginRedirectUrl(OpenShiftAuthManager.java:155)
	at io.cryostat.net.web.http.api.v2.AuthPostHandler.handle(AuthPostHandler.java:104)
	at io.cryostat.net.web.http.api.v2.AbstractV2RequestHandler.handle(AbstractV2RequestHandler.java:117)
	at io.cryostat.net.web.http.api.v2.AbstractV2RequestHandler.handle(AbstractV2RequestHandler.java:69)
	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at io.cryostat.net.OpenShiftAuthManager.lambda$computeAuthorizationEndpoint$4(OpenShiftAuthManager.java:382)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at io.cryostat.net.OpenShiftAuthManager.computeAuthorizationEndpoint(OpenShiftAuthManager.java:372)
	... 11 more
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at io.cryostat.net.OpenShiftAuthManager.lambda$computeOauthMetadata$7(OpenShiftAuthManager.java:434)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at io.cryostat.net.OpenShiftAuthManager.computeOauthMetadata(OpenShiftAuthManager.java:418)
	at io.cryostat.net.OpenShiftAuthManager.lambda$computeAuthorizationEndpoint$4(OpenShiftAuthManager.java:379)
	... 13 more
Caused by: javax.net.ssl.SSLHandshakeException: Failed to create SSL connection
	at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:115)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
	at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1260)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1241)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	... 2 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
	... 20 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
	... 31 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 37 more

ebaron · 2022-04-01T20:27:08Z

@andrewazores this looks a lot like #357. Can you double check this pod has GRAFANA_DASHBOARD_URL set to https://cryostat-health.local:3000?

andrewazores · 2022-04-01T20:57:04Z

Huh.

Containers:
  cryostat-sample:
    Container ID:   cri-o://f5114bb6d7bb14825993f90da148a469cb606ed23044109349e6e855a7f1ccb5
    Image:          quay.io/cryostat/cryostat:2.1.0-SNAPSHOT
    Image ID:       quay.io/cryostat/cryostat@sha256:c4ccc9e84d96a6f7c827a2ecec371e6c4d3323c2971a60c52589bb9c488c4fc9
    Ports:          8181/TCP, 9091/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 01 Apr 2022 16:44:29 -0400
    Ready:          True
    Restart Count:  1
    Liveness:       http-get https://:8181/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Startup:        http-get https://:8181/health delay=0s timeout=1s period=10s #success=1 #failure=18
    Environment Variables from:
      cryostat-sample-jmx-auth  Secret  Optional: false
      cryostat-sample-keystore  Secret  Optional: false
    Environment:
      CRYOSTAT_WEB_PORT:                    8181
      CRYOSTAT_CONFIG_PATH:                 /opt/cryostat.d/conf.d
      CRYOSTAT_ARCHIVE_PATH:                /opt/cryostat.d/recordings.d
      CRYOSTAT_TEMPLATE_PATH:               /opt/cryostat.d/templates.d
      CRYOSTAT_CLIENTLIB_PATH:              /opt/cryostat.d/clientlib.d
      CRYOSTAT_PROBE_TEMPLATE_PATH:         /opt/cryostat.d/probes.d
      CRYOSTAT_ENABLE_JDP_BROADCAST:        false
      CRYOSTAT_EXT_WEB_PORT:                443
      CRYOSTAT_WEB_HOST:                    cryostat-sample-myproject.apps-crc.testing
      CRYOSTAT_REPORT_GENERATION_MAX_HEAP:  200
      CRYOSTAT_TARGET_CACHE_SIZE:           -1
      CRYOSTAT_TARGET_CACHE_TTL:            10
      CRYOSTAT_PLATFORM:                    io.cryostat.platform.internal.OpenShiftPlatformStrategy
      CRYOSTAT_AUTH_MANAGER:                io.cryostat.net.OpenShiftAuthManager
      CRYOSTAT_OAUTH_CLIENT_ID:             cryostat-sample
      CRYOSTAT_OAUTH_ROLE:                  cryostat-operator-oauth-client
      GRAFANA_DATASOURCE_URL:               http://127.0.0.1:8080
      GRAFANA_DASHBOARD_EXT_URL:            https://cryostat-sample-grafana-myproject.apps-crc.testing
      GRAFANA_DASHBOARD_URL:                https://cryostat-health.local:3000
      KEYSTORE_PATH:                        /var/run/secrets/operator.cryostat.io/cryostat-sample-tls/keystore.p12

ebaron · 2022-04-04T16:14:19Z

@andrewazores this is strange, I'm not able to reproduce what you're seeing. I tried the steps in a fresh cluster to try to rule out unexpected side effects.

andrewazores · 2022-04-04T16:15:31Z

I'll try it out again tomorrow. I'm in airports/on planes all day today.

jan-law · 2022-04-04T19:06:53Z

I followed the given test steps and I see exceptions like the following in the main Cryostat container logs when I try to open the web-client

FWIW, I'm also getting the same error as Andrew with my Quicklab cluster. My env vars look similar too:
GRAFANA_DASHBOARD_URL=https://cryostat-health.local:3000 and GRAFANA_DASHBOARD_EXT_URL=https://cryostat-sample-grafana-cryostat-operator-system.apps.jalaw0.lab.upshift.rdu2.redhat.com

hareetd · 2022-04-04T21:56:54Z

Works as expected for me on CRC.

Edit: Oops, didn't realize Andrew's error is occurring when opening the web-client. I get the same error if I do the same.

andrewazores · 2022-04-05T14:21:10Z

I just tried it again with a fresh CRC instance and I'm still seeing the same failure when opening the Cryostat web-client.

andrewazores · 2022-04-05T14:49:46Z

Those exceptions aren't unique to this PR though - I'm seeing them on main now as well. I'll try to go before #352 next and see what happens. Seems like I may have made some mistake in the testing process when reviewing #352.

ebaron · 2022-04-05T14:57:58Z

I think I overlooked the opening the web-client step as well. I'll try again shortly.

ebaron · 2022-04-05T18:38:23Z

I think I see the problem now. It has nothing to do with Grafana or this PR. It's another side effect of removing CRYOSTAT_ALLOW_UNTRUSTED_SSL. Cryostat sends a GET request to OpenShift's OAuth server to discover its endpoints. Cryostat is not configured to trust the OAuth server's certificate. The fix should be something like adding /var/run/secrets/kubernetes.io/serviceaccount/ca.crt to its truststore.

ebaron · 2022-04-08T15:52:56Z

@andrewazores @jan-law @hareetd With https://github.com/cryostatio/cryostat/pull/882 merged, this should be easier to test now using the instructions in the PR description.

hareetd

Works well now.

fix(tls): update existing certificates

0f3bdd4

ebaron added the fix label Mar 31, 2022

ebaron requested review from andrewazores, hareetd and jan-law March 31, 2022 21:00

ebaron mentioned this pull request Apr 7, 2022

OpenShift OAuth server is not trusted cryostatio/cryostat-legacy#881

Closed

Merge branch 'main' into update-certs

b8efc23

hareetd approved these changes Apr 8, 2022

View reviewed changes

jan-law approved these changes Apr 8, 2022

View reviewed changes

andrewazores approved these changes Apr 8, 2022

View reviewed changes

ebaron merged commit 92aac6d into cryostatio:main Apr 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(tls): update existing certificates #360

fix(tls): update existing certificates #360

ebaron commented Mar 31, 2022

andrewazores commented Apr 1, 2022

ebaron commented Apr 1, 2022

andrewazores commented Apr 1, 2022

ebaron commented Apr 4, 2022

andrewazores commented Apr 4, 2022

jan-law commented Apr 4, 2022

hareetd commented Apr 4, 2022 •

edited

Loading

andrewazores commented Apr 5, 2022

andrewazores commented Apr 5, 2022

ebaron commented Apr 5, 2022

ebaron commented Apr 5, 2022

ebaron commented Apr 8, 2022

hareetd left a comment

fix(tls): update existing certificates #360

fix(tls): update existing certificates #360

Conversation

ebaron commented Mar 31, 2022

andrewazores commented Apr 1, 2022

ebaron commented Apr 1, 2022

andrewazores commented Apr 1, 2022

ebaron commented Apr 4, 2022

andrewazores commented Apr 4, 2022

jan-law commented Apr 4, 2022

hareetd commented Apr 4, 2022 • edited Loading

andrewazores commented Apr 5, 2022

andrewazores commented Apr 5, 2022

ebaron commented Apr 5, 2022

ebaron commented Apr 5, 2022

ebaron commented Apr 8, 2022

hareetd left a comment

Choose a reason for hiding this comment

hareetd commented Apr 4, 2022 •

edited

Loading