Skip to content

Commit

Permalink
[KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
Browse files Browse the repository at this point in the history
# 🔍 Description
## Issue References 🔗

This pull request fixes #6338

## Describe Your Solution 🔧

Support `kyuubi.server.thrift.resultset.default.fetchsize` conf to respect `hive.server2.thrift.resultset.default.fetch.size` hive conf.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests
KyuubiTBinaryFrontendServiceSuite.test("test kyuubi.server.thrift.resultset.default.fetch.size")

---

# Checklist 📝

- [X] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6340 from wForget/KYUUBI-6338.

Closes #6338

acd73a1 [wforget] fix
5e64ed3 [wforget] comments
48b8ec6 [wforget] fix style and address comments
ca65349 [wforget] [KYUUBI #6338] Support `kyuubi.server.thrift.resultset.default.fetch.size` conf

Authored-by: wforget <643348094@qq.com>
Signed-off-by: wforget <643348094@qq.com>
  • Loading branch information
wForget committed Apr 29, 2024
1 parent 5cbbdc3 commit ef28a61
Show file tree
Hide file tree
Showing 4 changed files with 41 additions and 0 deletions.
1 change: 1 addition & 0 deletions docs/configuration/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,7 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
| kyuubi.server.name | &lt;undefined&gt; | The name of Kyuubi Server. | string | 1.5.0 |
| kyuubi.server.periodicGC.interval | PT30M | How often to trigger a garbage collection. | duration | 1.7.0 |
| kyuubi.server.redaction.regex | &lt;undefined&gt; | Regex to decide which Kyuubi contain sensitive information. When this regex matches a property key or value, the value is redacted from the various logs. || 1.6.0 |
| kyuubi.server.thrift.resultset.default.fetch.size | 1000 | The number of rows sent in one Fetch RPC call by the server to the client, if not specified by the client. Respect `hive.server2.thrift.resultset.default.fetch.size` hive conf. | int | 1.9.1 |

### Session

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3620,4 +3620,23 @@ object KyuubiConf {
.version("1.8.1")
.booleanConf
.createWithDefault(false)

private val HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE: ConfigEntry[Int] =
buildConf("hive.server2.thrift.resultset.default.fetch.size")
.doc("This is a hive server configuration used as a fallback conf" +
s" for `kyuubi.server.thrift.resultset.default.fetch.size`.")
.version("1.9.1")
.internal
.serverOnly
.intConf
.createWithDefault(1000)

val KYUUBI_SERVER_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE: ConfigEntry[Int] =
buildConf("kyuubi.server.thrift.resultset.default.fetch.size")
.doc("The number of rows sent in one Fetch RPC call by the server to the client, if not" +
" specified by the client. Respect `hive.server2.thrift.resultset.default.fetch.size`" +
" hive conf.")
.version("1.9.1")
.serverOnly
.fallbackConf(HIVE_SERVER2_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE)
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.kyuubi.KyuubiSQLException
import org.apache.kyuubi.cli.Handle
import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.config.KyuubiConf.KYUUBI_SERVER_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE
import org.apache.kyuubi.config.KyuubiReservedKeys._
import org.apache.kyuubi.ha.client.{KyuubiServiceDiscovery, ServiceDiscovery}
import org.apache.kyuubi.metrics.MetricsConstants._
Expand All @@ -49,6 +50,8 @@ final class KyuubiTBinaryFrontendService(
}
}

private lazy val defaultFetchSize = conf.get(KYUUBI_SERVER_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE)

override def initialize(conf: KyuubiConf): Unit = synchronized {
super.initialize(conf)

Expand Down Expand Up @@ -94,6 +97,11 @@ final class KyuubiTBinaryFrontendService(

respConfiguration.put(KYUUBI_SESSION_ENGINE_LAUNCH_SUPPORT_RESULT, true.toString)

// HIVE-23005(4.0.0), Hive JDBC driver supposes that server always returns this conf
respConfiguration.put(
"hive.server2.thrift.resultset.default.fetch.size",
defaultFetchSize.toString)

resp.setSessionHandle(sessionHandle.toTSessionHandle)
resp.setConfiguration(respConfiguration)
resp.setStatus(OK_STATUS)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import org.apache.kyuubi.shaded.hive.service.rpc.thrift.{TOpenSessionReq, TSessi
class KyuubiTBinaryFrontendServiceSuite extends WithKyuubiServer with KyuubiFunSuite {

override protected val conf: KyuubiConf = KyuubiConf()
.set(KyuubiConf.KYUUBI_SERVER_THRIFT_RESULTSET_DEFAULT_FETCH_SIZE, 500)

test("connection metrics") {
val totalConnections =
Expand Down Expand Up @@ -116,4 +117,16 @@ class KyuubiTBinaryFrontendServiceSuite extends WithKyuubiServer with KyuubiFunS
Thread.sleep(3000L)
assert(server.backendService.sessionManager.allSessions().size == sessionCount)
}

test("test kyuubi.server.thrift.resultset.default.fetch.size") {
TClientTestUtils.withThriftClient(server.frontendServices.head) {
client =>
val req = new TOpenSessionReq()
req.setUsername(Utils.currentUser)
req.setPassword("anonymous")
val resp = client.OpenSession(req)
assertResult("500")(
resp.getConfiguration.get("hive.server2.thrift.resultset.default.fetch.size"))
}
}
}

0 comments on commit ef28a61

Please sign in to comment.