[DocDB] Tserver hits soft limit and not reclaiming back memory | Rejecting Write request: Soft memory limit exceeded

[shantanugupta-yb]

Jira Link: DB-1085

Description

While testing for yb cloud free tier(RF1, 2GB memory), tserver was hitting soft limit and Writes were getting rejected with error "Rejecting Write request: Soft memory limit exceeded (at 92.86% of capacity), score: 0.00"

Ideally when tserver has already hit memory soft limit with no active workload running, the background thread should reclaim back memory.

I also wanted to check the current allocation held in memory for which I took difference between to memory snapshots. Below are the top 5 allocation. The 1st snapshot had allocations of 270MB and the 2nd snapshot had allocation 610M. So out of the 310M below are the top 5 allocation:

1. 148M - rocksdb::Arena::AllocateNewBlock
2. 61.2M - std::__cxx11::basic_string::reserve  
3. 42M. - rocksdb::UncompressBlockContents  
4. 23.8M - std::__cxx11::basic_string::_M_construct  
5. 13.2M. - yb::rpc::CircularReadBuffer::CircularReadBuffer  

The rocksdb version used by us seems to of v4.5 and I seen a bug "rocksdb::Arena::AllocateNewBlock allocate memory grows without limit #8371" recently got fixed which is missing from the rocksdb version we are using. facebook/rocksdb#8371

[bmatican]

This is not a bug, this is just the memory overhead that we have, per tablet, from the rocksdb arena and metric histograms