Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added DbLedgerStorage configuration keys with default values in bookkeeper.conf #123

Merged
merged 1 commit into from
Nov 28, 2016

Conversation

merlimat
Copy link
Contributor

Motivation

Added missing configuration keys for BookKeeper DbLedgerStorage

@yahoocla
Copy link

CLA is valid!

@merlimat merlimat added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Nov 21, 2016
@merlimat merlimat added this to the 1.16 milestone Nov 21, 2016
@merlimat merlimat self-assigned this Nov 21, 2016
@sschepens
Copy link
Contributor

@merlimat do you have any recommendations for rocksdb tuning? I believe if we set writeCacheMaxSizeMb very high, a single flush would write a lot of level0 files, which would probably be above rocksDB_numFilesInLevel0 and it will begin to stall writes. Am I correct?

Is there a good ratio between writeCacheMaxSizeMb and rocksDB_writeBufferSizeMB you find useful?

@sschepens
Copy link
Contributor

@merlimat please read my recent comments on #110

@merlimat
Copy link
Contributor Author

@sschepens In DbLedgerStorage, RocksDB is used to just store the indexes, not the actual entries payloads. All writes to RocksDB are done in a background flush thread, so the stall are not a concern here.

To recap:

  • dbStorage_writeCacheMaxSizeMb : Memory portion where payloads are accumulated before flushing. Ideally this should big enough to accumulate ~1min worth of entries
  • dbStorage_rocksDB_writeBufferSizeMB : Size of RocksDB memtable. I don't think this need to be increased from the default 64MB
  • dbStorage_rocksDB_blockCacheSize : This is the RocksDB cache, used when reading. It should be big enough to hold the "active" portion of database to avoid disk access to read the indexes.

Copy link
Contributor

@rdhabalia rdhabalia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@merlimat merlimat merged commit 5af7d3f into apache:master Nov 28, 2016
sijie pushed a commit to sijie/pulsar that referenced this pull request Mar 4, 2018
* Added publish interface to the api

* Address comments
hrsakai pushed a commit to hrsakai/pulsar that referenced this pull request Dec 10, 2020
apache#124)

* [Issue apache#123] Ensure message is sent if no room in current batch.

* Add issue to test comment.

* Increase test wait timeout.

* Add timeout log for test.

* Add log when a single message fails to be added to batch.
massakam pushed a commit to massakam/pulsar that referenced this pull request Feb 9, 2021
Updates Message.cc to include support for delayed delivery from the C++ client. (pulsar_message_set_deliver_after & pulsar_message_set_deliver_at)

Co-authored-by: savearray2 <savearray2>
hangc0276 pushed a commit to hangc0276/pulsar that referenced this pull request May 26, 2021
* reuse ordering key to avoid String and bytes converting for message key

* update to pulsar 2.5.0-ad0224407, this verison is closer to 2.5.1, and include some fix like ratelimit in pulsar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc Your PR contains doc changes, no matter whether the changes are in markdown or code files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants