Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting Unknown authType: GENERIC when connecting to postgres #1983

Closed
tamirhad opened this issue May 15, 2024 · 5 comments · Fixed by #1993 or #2017
Closed

Getting Unknown authType: GENERIC when connecting to postgres #1983

tamirhad opened this issue May 15, 2024 · 5 comments · Fixed by #1993 or #2017
Assignees
@hessjcg
Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@tamirhad
Copy link

Bug Description

When trying to connect to postgresql running on cloudSQL I am getting the following error:

javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC

I came across this issue when looking at the stacktrace: google/conscrypt#1033 but I couldnt find any workaround.

The code is running perfectly when executed on my Mac, but when running on DataProc I am getting this error.

Example code (or command)

No response

Stacktrace

2024-05-15 17:00:30,263 DEBUG hikari.HikariConfig: HikariPool-1 - configuration:
2024-05-15 17:00:30,267 DEBUG hikari.HikariConfig: allowPoolSuspension.............false
2024-05-15 17:00:30,267 DEBUG hikari.HikariConfig: autoCommit......................true
2024-05-15 17:00:30,267 DEBUG hikari.HikariConfig: catalog.........................none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: connectionInitSql...............none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: connectionTestQuery.............none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: connectionTimeout...............30000
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: dataSource......................none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: dataSourceClassName.............none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: dataSourceJNDI..................none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: dataSourceProperties............{password=<masked>, cloudSqlInstance=CloudSqlConnectionString, socketFactory=com.google.cloud.sql.postgres.SocketFactory}
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: driverClassName.................none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: healthCheckProperties...........{}
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: healthCheckRegistry.............none
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: idleTimeout.....................600000
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: initializationFailFast..........true
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: initializationFailTimeout.......1
2024-05-15 17:00:30,268 DEBUG hikari.HikariConfig: isolateInternalQueries..........false
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: jdbc4ConnectionTest.............false
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: jdbcUrl........................."jdbc:postgresql://someip:5432/db"
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: leakDetectionThreshold..........0
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: maxLifetime.....................1800000
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: maximumPoolSize.................1
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: metricRegistry..................none
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: metricsTrackerFactory...........none
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: minimumIdle.....................1
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: password........................<masked>
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: poolName........................"HikariPool-1"
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: readOnly........................false
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: registerMbeans..................false
2024-05-15 17:00:30,269 DEBUG hikari.HikariConfig: scheduledExecutor...............none
2024-05-15 17:00:30,270 DEBUG hikari.HikariConfig: scheduledExecutorService........internal
2024-05-15 17:00:30,270 DEBUG hikari.HikariConfig: threadFactory...................internal
2024-05-15 17:00:30,270 DEBUG hikari.HikariConfig: transactionIsolation............default
2024-05-15 17:00:30,270 DEBUG hikari.HikariConfig: username........................"masked"
2024-05-15 17:00:30,270 DEBUG hikari.HikariConfig: validationTimeout...............5000
2024-05-15 17:00:30,272 INFO hikari.HikariDataSource: HikariPool-1 - Starting...
2024-05-15 17:00:30,388 DEBUG core.InternalConnectorRegistry: First Cloud SQL connection, generating RSA key pair.
2024-05-15 17:00:30,460 DEBUG core.Connector: [CloudSqlConnectionString] Connection info added to cache.
2024-05-15 17:00:30,467 DEBUG core.Refresher: [CloudSqlConnectionString] Force Refresh: the next refresh operation was cancelled. Scheduling new refresh operation immediately.
2024-05-15 17:00:30,467 DEBUG core.Refresher: [CloudSqlConnectionString] Refresh Operation: Acquiring rate limiter permit.
2024-05-15 17:00:30,478 DEBUG core.Refresher: [CloudSqlConnectionString] Refresh Operation: Rate limiter permit acquired.
2024-05-15 17:00:30,687 DEBUG core.DefaultConnectionInfoRepository: [CloudSqlConnectionString] METADATA DONE
2024-05-15 17:00:30,712 DEBUG core.DefaultConnectionInfoRepository: [CloudSqlConnectionString 18] CERT DONE
2024-05-15 17:00:30,879 DEBUG core.DefaultConnectionInfoRepository: [CloudSqlConnectionString 21] SSL CONTEXT
2024-05-15 17:00:30,879 DEBUG core.DefaultConnectionInfoRepository: [CloudSqlConnectionString] INSTANCE DATA DONE
2024-05-15 17:00:30,880 DEBUG core.DefaultConnectionInfoRepository: [CloudSqlConnectionString] ALL FUTURES DONE
2024-05-15 17:00:30,881 DEBUG core.Refresher: [CloudSqlConnectionString] Refresh Operation: Completed refresh with new certificate expiration at 2024-05-15T18:00:30Z.
2024-05-15 17:00:30,882 DEBUG core.Refresher: [CloudSqlConnectionString] Refresh Operation: Next operation scheduled at 2024-05-15T17:56:29Z.
2024-05-15 17:00:30,887 DEBUG core.Refresher: [CloudSqlConnectionString] Now = 2024-05-15T17:00:30.886740Z, Current client certificate expiration = 2024-05-15T18:00:30Z
2024-05-15 17:00:30,887 DEBUG core.Connector: [192.168.1.1] Connecting to instance.
2024-05-15 17:00:30,901 DEBUG core.Connector: TLS handshake failed!
2024-05-15 17:00:30,902 DEBUG core.Connector: [CloudSqlConnectionString] Socket connection failed! Trigger a refresh.
2024-05-15 17:00:30,902 DEBUG core.Refresher: [CloudSqlConnectionString] Force Refresh: the next refresh operation was cancelled. Scheduling new refresh operation immediately.
2024-05-15 17:00:30,902 DEBUG core.Refresher: [CloudSqlConnectionString] Refresh Operation: Acquiring rate limiter permit.
2024-05-15 17:00:30,909 DEBUG pool.PoolBase: HikariPool-1 - Failed to create/setup connection: The connection attempt failed.
2024-05-15 17:00:30,910 DEBUG pool.HikariPool: HikariPool-1 - Cannot acquire connection from data source
org.postgresql.util.PSQLException: The connection attempt failed.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:358)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:273)
	at org.postgresql.Driver.makeConnection(Driver.java:446)
	at org.postgresql.Driver.connect(Driver.java:298)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:112)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:118)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:428)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:499)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.google.cloud.hadoop.services.agent.job.shim.HadoopRunClassShim.main(HadoopRunClassShim.java:19)
Caused by: javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC
	at org.conscrypt.SSLUtils.toSSLHandshakeException(SSLUtils.java:361)
	at org.conscrypt.ConscryptEngine.convertException(ConscryptEngine.java:1138)
	at org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1093)
	at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:880)
	at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:751)
	at org.conscrypt.ConscryptEngine.unwrap(ConscryptEngine.java:716)
	at org.conscrypt.ConscryptEngineSocket$SSLInputStream.processDataFromSocket(ConscryptEngineSocket.java:833)
	at org.conscrypt.ConscryptEngineSocket$SSLInputStream.access$100(ConscryptEngineSocket.java:706)
	at org.conscrypt.ConscryptEngineSocket.doHandshake(ConscryptEngineSocket.java:230)
	at org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:209)
	at com.google.cloud.sql.core.Connector.connect(Connector.java:122)
	at com.google.cloud.sql.core.InternalConnectorRegistry.connect(InternalConnectorRegistry.java:179)
	at com.google.cloud.sql.postgres.SocketFactory.createSocket(SocketFactory.java:81)
	at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
	at org.postgresql.core.PGStream.<init>(PGStream.java:98)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:136)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:262)
	... 30 more
Caused by: java.security.cert.CertificateException: Unknown authType: GENERIC
	at java.base/sun.security.validator.EndEntityChecker.checkTLSServer(EndEntityChecker.java:289)
	at java.base/sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:144)
	at java.base/sun.security.validator.Validator.validate(Validator.java:277)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:222)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
	at org.conscrypt.ConscryptEngineSocket$2.checkServerTrusted(ConscryptEngineSocket.java:156)
	at org.conscrypt.Platform.checkServerTrusted(Platform.java:330)
	at org.conscrypt.ConscryptEngine.verifyCertificateChain(ConscryptEngine.java:1643)
	at org.conscrypt.NativeCrypto.ENGINE_SSL_read_direct(Native Method)
	at org.conscrypt.NativeSsl.readDirectByteBuffer(NativeSsl.java:567)
	at org.conscrypt.ConscryptEngine.readPlaintextDataDirect(ConscryptEngine.java:1099)
	at org.conscrypt.ConscryptEngine.readPlaintextData(ConscryptEngine.java:1083)
	... 44 more

Steps to reproduce?

  1. ?
  2. ?
  3. ?
    ...

Environment

  1. OS type and version: DataProc running image 2.2.16-ubuntu22
  2. Java SDK version: 11
  3. Cloud SQL Java Socket Factory version: latest

Additional Details

No response
@tamirhad tamirhad added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label May 15, 2024
@hessjcg
Copy link
Collaborator

hessjcg commented May 20, 2024

Hi @tamirhad,

It looks like the DataProc runtime has changed the default crypto library used in Java from whatever is built into the VM to Conscrypt.

As a work-around, you might look for a way to instruct your DataProc app to use default JVM crypto libraries, or the BouncyCastle instead of Conscript.

I will try to reproduce this problem outside of DataProc by running the connector with the Conscrypt crypto library.

-Jonathan

@hessjcg
Copy link
Collaborator

hessjcg commented May 20, 2024

@tamirhad I was able to write an integration test that connected successfully to a real Cloud SQL instance using the Google Conscript crypto library for Java. This evidence suggests that the problem is not with the crypto library.

There is something curious I noticed in the log you posted:

2024-05-15 17:00:30,887 DEBUG core.Connector: [192.168.1.1] Connecting to instance.
2024-05-15 17:00:30,901 DEBUG core.Connector: TLS handshake failed!

This IP address is in the metadata for the Cloud SQL instance. Is this the private IP address that you expect, or is possible that there is something funny with that IP Address?

@tamirhad
Copy link
Author

@hessjcg Thanks for looking into it so quickly!

I masked the ip address due to privacy reasons. The real ip is also private ip but from the class A range (10.x.x.x).

@hessjcg
Copy link
Collaborator

hessjcg commented May 21, 2024

@tamirhad Ok, that makes a lot more sense, thanks for clarifying.

I think you are right that this is an incompatibility between OpenJDK and Conscript's use of GENERIC vs UNKNOWN.

I'm going to retry my test case using Conscrypt crypto with IAM Authentication enabled, and then implement the workaround described in Conscrypt #1033

hessjcg added a commit that referenced this issue May 21, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
00ac31a
@hessjcg
Copy link
Collaborator

hessjcg commented May 21, 2024

I was able to reproduce the exception by modifying my local JDK to use Conscrypt as the primary crypto provider. I edited the JDK file $JAVA_HOME/conf/security/java.security from

security.provider.1=SUN

to

security.provider.1=org.conscrypt.OpenSSLProvider
security.provider.13=SUN

Then I ran PostgresIamAuthIntegrationTests and got the exception. Then I implemented a wrapper around TrustManagerFactory to work around the issue and the tests passed.

Unfortunately, due to the fact that the JVM needs to be manually configured to prioritize the Conscrypt JCE implementation, I don't think there is a good way to automate the tests for this fix.

hessjcg added a commit that referenced this issue May 21, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
0adc46c
@hessjcg hessjcg mentioned this issue May 21, 2024
Merged
@hessjcg hessjcg added the priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. label May 21, 2024
hessjcg added a commit that referenced this issue May 21, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
c452850
hessjcg added a commit that referenced this issue May 22, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
c2019a4
hessjcg added a commit that referenced this issue May 22, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
4a4ca0a
hessjcg added a commit that referenced this issue May 22, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
eff3963
hessjcg added a commit that referenced this issue May 22, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
8e48061
hessjcg added a commit that referenced this issue May 22, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
6b4dfde
hessjcg added a commit that referenced this issue May 22, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
31a72fa
hessjcg added a commit that referenced this issue May 24, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.
20eefa3
hessjcg added a commit that referenced this issue May 24, 2024
@hessjcg
fix: Add TrustManagerFactory workaround for Conscrypt bug, Fixes #1983.…
0735a91 
… (#1993)

This is a workaround for an underlying bug in the Google Conscrypt crypto library google/conscrypt #1033.

The root cause is that the Conscrypt and OpenJDK X509 certificate libraries sometimes interpret the AuthType
field differently: Conscrypt finds 'GENERIC' auth type when OpenJDK finds 'UNKNOWN' auth type. This causes certificate validation to fail.

The workaround implemented here is to add a delegate TrustManager that replaces 'GENERIC' auth type with 'UNKNOWN' auth type so that the Conscrypt crypto plays nice with the JDK crypto. See comment on #1033.

I manually tested this on a modified JVM that used Conscrypt as it's primary crypto library. The integration tests passed. I have not found a good way to make this test part of the test suite.

Fixes #1983
@release-please release-please bot mentioned this issue Jun 11, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hessjcg hessjcg

Labels
priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
2 participants
@tamirhad @hessjcg