Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kafka import additional consumer properties #41

Closed
morazow opened this issue Oct 8, 2019 · 1 comment
Closed

Add Kafka import additional consumer properties #41

morazow opened this issue Oct 8, 2019 · 1 comment

Comments

@morazow
Copy link
Contributor

morazow commented Oct 8, 2019

An additional useful properties for the Kafka consumer application:

  • MAX_POLL_RECORDS: The maximum number of records returned in a single call to poll(). The default value is 500. Since we perform a single poll() each time the udf is run, users can increase this value to import more records from partitions on each udf run.

  • FETCH_MIN_BYTES: The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

  • SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: The endpoint identification algorithm to validate server hostname using server certificate. Default value is https. User should be able to set this to empty string ("") if they want to disable host identification while using secure connection to Kafka clusters.

morazow added a commit that referenced this issue Oct 10, 2019
Thanks for your feedback and contributions!

Fixes #41.
@morazow
Copy link
Contributor Author

morazow commented Oct 17, 2019

Fixed in #43.

@morazow morazow closed this as completed Oct 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant