Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-49673][CONNECT] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE #48122

Closed
wants to merge 8 commits into from
Closed

[SPARK-49673][CONNECT] Increase CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to 0.7 * CONNECT_GRPC_MAX_MESSAGE_SIZE #48122

Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a68f280
increase maxBatchSize to 0.7 * grpc message size limit for sqlCommand…
dillitz Sep 16, 2024
18a330f
remove unused import
dillitz Sep 16, 2024
c04795c
Revert "remove unused import"
dillitz Sep 17, 2024
cf006e8
Revert "increase maxBatchSize to 0.7 * grpc message size limit for sq…
dillitz Sep 17, 2024
01eddd3
change CONNECT_GRPC_ARROW_MAX_BATCH_SIZE to CONNECT_GRPC_MAX_MESSAGE_…
dillitz Sep 17, 2024
4eb6711
add E2E test
dillitz Sep 17, 2024
ca10ef9
adjust client's maxMessageSize
dillitz Sep 17, 2024
b0bfb27
linter
dillitz Sep 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ object Connect {
"conservatively use 70% of it because the size is not accurate but estimated.")
.version("3.4.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefault(4 * 1024 * 1024)
.createWithDefault(ConnectCommon.CONNECT_GRPC_MAX_MESSAGE_SIZE)

val CONNECT_GRPC_MAX_INBOUND_MESSAGE_SIZE =
buildStaticConf("spark.connect.grpc.maxInboundMessageSize")
Expand Down