Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose BatchingMaxSize from ProducerOptions #280

Merged
merged 1 commit into from
Jun 13, 2020

Conversation

dferstay
Copy link
Contributor

@dferstay dferstay commented Jun 12, 2020

Previously, the producer maximum batch size was hard-coded to 128 KB.

Now, the produdcer maximum batch size is exposed via ProducerOptions
and defaults to 128 KB

Motivation

Different applications producing to Pulsar require different batching policies.
Previously, users had control over the maximum batching delay and the maximum number of messages in a batch, but could not control the maximum number of bytes in a batch. This change enables the latter.

Modifications

Expose a new ProducerOption.BatchingMaxSize (default: 128KB) that allows the user to configure the maximum batch size. This option has been plumbed through partition_producer module to the internal/batch_builder module where it is used.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as producer_test.TestFlushInPartitionedProducer and producer_test.TestFlushInProducer.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • The public API: The pulsar.ProducerOptions structure gains a new field: BatchingMaxSize

Documentation

  • Does this pull request introduce a new feature? Yes
  • If yes, how is the feature documented? GoDocs

@dferstay dferstay force-pushed the producer-opts-max-batch-size branch from 4ff1b9c to 8b68fc1 Compare June 12, 2020 23:02
Previously, the producer maximum batch size was hard-coded to 128 KB.

Now, the produdcer maximum batch size is exposed via ProducerOptions
and defaults to 128 KB
@dferstay dferstay force-pushed the producer-opts-max-batch-size branch from 8b68fc1 to 0c7fffe Compare June 12, 2020 23:07
@merlimat merlimat added this to the 0.2.0 milestone Jun 13, 2020
@merlimat merlimat merged commit 62203d7 into apache:master Jun 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants