Skip to content
This repository has been archived by the owner on Apr 1, 2024. It is now read-only.

Cherry pick/limit the memory #5909

Closed
wants to merge 5 commits into from
Closed

Cherry pick/limit the memory #5909

wants to merge 5 commits into from

Commits on Sep 25, 2023

  1. InflightReadsLimiter - limit the memory used by reads end-to-end (fro…

    …m storage/cache to the write to the consumer channel) (apache#18245)
    
    * InflightReadsLimiter - limit the memory used by reads end-to-end (from storage/cache to the write to the consumer channel)
    
    Motivation:
    
    Broker can go out of memory due to many reads enqueued on the PersistentDispatcherMultipleConsumers dispatchMessagesThread (that is used in case of dispatcherDispatchMessagesInSubscriptionThread set to true, that is the default value)
    The limit of the amount of memory retained due to reads MUST take into account also the entries coming from the Cache.
    
    When dispatcherDispatchMessagesInSubscriptionThread is false (the behaviour of Pulsar 2.10) there is some kind of natural (but still unpredictable!!) back pressure mechanism because the thread that receives the entries from BK of the cache dispatches immediately and synchronously the entries to the consumer and releases them
    
    Modifications:
    
    - Add a new component (InflightReadsLimiter) that keeps track of the overall amount of memory retained due to inflight reads.
    - Add a new configuration entry managedLedgerMaxReadsInFlightSizeInMB
    - The feature is disabled by default
    - Add new metrics to track the values
    
    * Change error message
    
    * checkstyle
    
    * Fix license
    
    * remove duplicate method after cherry-pick
    
    * Rename onDeallocate
    
    (cherry picked from commit 6fec66b)
    (cherry picked from commit 47c98e5)
    eolivelli authored and liangyepianzhou committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    30fc91c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d315d18 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. remove gpg plugin

    liangyepianzhou committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    8054d59 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    64627fd View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. remove release plugin

    liangyepianzhou committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    20522ea View commit details
    Browse the repository at this point in the history