Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Webhook support custom SSL on the target #76

Open
LHozzan opened this issue Jul 14, 2023 · 4 comments
Open

[FEATURE] Webhook support custom SSL on the target #76

LHozzan opened this issue Jul 14, 2023 · 4 comments
Labels
enhancement New feature or request

Comments

@LHozzan
Copy link

LHozzan commented Jul 14, 2023

Is your feature request related to a problem?
My current goal is provide our developers ability to watch over time some conditions and when reached matching criteria, they will be noticed via Prometheus AlertManager. Unfortunately, we would like to have more secure infrastructure, so internal AlertManager endpoint is secured via SSL certificate, which is signed by our internal CA. And here is a problem with supporting that scenario.

What solution would you like?
Have some settings (in the OS Dashboard GUI preferable, but settings in config files are fine too) to specify custom CA, which is used to signed different endpoints for webhook targets.

What alternatives have you considered?
A some way, how to set or specify custom CA certificate for webhook endpoint.

Do you have any additional context?
If the endpoint isnt secured, everything working fine, but we need rise security.
I asked on the forum and I was pointed to the old closed issue with workaround. Unfortunately, with OpenSearch version 2.7.0 the workaround not working. If I made trustore with keytool utility from the Docker image, JKS trustore looks good for the utility, but OpenSearch refuse start properly with bunch of warnings and errors. Looks like:

{"type": "logging", "timestamp": "2023-07-14T12:06:53,170Z", "level": "ERROR", "component": "o.o.b.OpenSearchUncaughtExceptionHandler", "cluster.name": "logging", "node.name": "ofd-client-77f4d94d7c-84nsr", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.opensearch.bootstrap.StartupException: org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.7.0.jar:2.7.0]",
"Caused by: org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:83) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.security.KeyStoreException: problem accessing trust store",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
uncaught exception in thread [main]
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.io.IOException: keystore password was incorrect",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]",
"at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]",
"at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more"] }
org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242)
        at java.base/java.security.KeyStore.load(KeyStore.java:1473)
        at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390)
        at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336)
        at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57)
        at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49)
        at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282)
        at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168)
        at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81)
        at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155)
        at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160)
        at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130)
        at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129)
        at org.opensearch.node.Node.lambda$new$16(Node.java:788)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at org.opensearch.node.Node.<init>(Node.java:802)
        at org.opensearch.node.Node.<init>(Node.java:375)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        <<<truncated>>>
For complete error details, refer to the log at /usr/share/opensearch/logs/logging.log

For generating truststore I used the command:

/usr/share/opensearch/jdk/bin/keytool \
  -importcert -trustcacerts -noprompt -file /tmp/devel-ca.pem \
  -alias company-ca -keystore /shared/truststore.jks \
  -storepass password123

Java settings for OpenSearch Client node looks like:

-Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore.jks
-Djavax.net.ssl.trustStoreType=JKS
-Djavax.net.ssl.trustStorePassword="password123"

If I check truststore with keytool, everything looks good:

/usr/share/opensearch/jdk/bin/keytool -list -v -keystore /shared/truststore.jks -storepass password123

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: company-ca
Creation date: Jul 14, 2023
Entry type: trustedCertEntry

Owner: CN=cluster:devel
Issuer: CN=fake-ca
Serial number: 5a70d3abd4521488e4f702589377772cdcae0c2d
Valid from: Fri Jun 23 07:18:00 UTC 2023 until: Sat Jun 21 07:18:00 UTC 2031
Certificate fingerprints:
         SHA1: 9E:BA:EA:44:A5:D0:2D:B4:7C:3A:95:AD:D7:A1:9E:26:AA:CB:F3:87
         SHA256: D3:C8:E9:8B:3D:1E:53:1D:15:55:3D:26:A1:7D:F1:CE:01:F3:E2:01:55:2C:2E:94:D4:2C:EA:3E:EA:FC:BC:12
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 C4 D4 ED 99 07 BF D0   A3 18 11 1F 43 A8 18 BA  x...........C...
0010: 5F 9C 11 B8                                        _...
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
  Key_CertSign
  Crl_Sign
]

#5: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 70 91 5C 95 80 1F 77 14   DE 7C 19 3A 48 0B D6 FA  p.\...w....:H...
0010: 1E 67 69 DC                                        .gi.
]
]



*******************************************
*******************************************

If you need some additional information, please, let me know.
@LHozzan LHozzan added enhancement New feature or request untriaged labels Jul 14, 2023
@scubbx
Copy link

scubbx commented Dec 7, 2023

I have the same problem. Providing a custom keystore did not solve the situation.

@Hailong-am
Copy link
Collaborator

@LHozzan
can you try import the cert into java default cacerts file which under $JAVA_HOME/lib/security folder

/usr/share/opensearch/jdk/bin/keytool \
  -importcert -trustcacerts -noprompt -file /tmp/devel-ca.pem \
  -alias company-ca -keystore $JAVA_HOME/lib/security/cacerts \
  -storepass changeit

@LHozzan
Copy link
Author

LHozzan commented Mar 31, 2024

Hi @Hailong-am .

Thank you for your effort and interest. Meantime we tested another opportunity to get notifications from OS Dash (via M$ Teams) and this solution looks usable for us.

I will let the feature request opened for anybody else, who required this feature due to some limitations on his side.

BR, Lu

@cwperks
Copy link
Member

cwperks commented Dec 9, 2024

FYI There are other instances in OpenSearch where connecting with an external system with separate CA is supported. See the LDAP docs where you can configure TLS for the LDAP authc backend: https://opensearch.org/docs/latest/security/authentication-backends/ldap/#tls-settings

Perhaps, that can be used as a model to fulfill this request?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@scubbx @cwperks @LHozzan @Hailong-am