Fix slowTickIfNecessary with infrequently used EWMA #3929
Conversation
|
It seems like versions prior to More context on the issue and how it was found CASSANDRA-19332 |
aweisberg
force-pushed
the
fix_slow_ewma_ticking
branch
from
40dc56a
to
bfbf60e
Compare
|
@aweisberg Thanks for your contribution!
Yes, that's correct. Any version before Dropwizard Metrics 4.2.x is unmaintained. Which version are you using exactly in Cassandra? |
|
It's a mix, this is a list of active versions with 3.0/3.11 soon to be unsupported. 4.x will probably be supported for 5+ years. Not a big deal as the work around is to read all the Meters periodically. |
aweisberg
force-pushed
the
fix_slow_ewma_ticking
branch
2 times, most recently
from
3102202
to
7642773
Compare
|
@joschi what is the next step? I reworked the fix slightly to reset the It doesn't set it to 0.0 because that was not the existing behavior which is after the |
aweisberg
force-pushed
the
fix_slow_ewma_ticking
branch
from
7642773
to
e4d9561
Compare
|
@joschi would you mind to take a look at this please? Cassandra project would be very happy if this was included :) |
aweisberg
force-pushed
the
fix_slow_ewma_ticking
branch
from
e4d9561
to
64652b4
Compare
aweisberg
force-pushed
the
fix_slow_ewma_ticking
branch
from
64652b4
to
171865a
Compare
EWMA.tickIfNecessary does an amount of work that is linear to the amount of time that has passed since the last time the EWMA was ticked. For infrequently used EWMA this can lead to pauses observed in the 700-800 millisecond range after a few hundred days. It's not really necessary to perform every tick as all that is doing is slowly approaching the smallest representable positive number in a double. Instead pick a number close to zero and if the number of ticks required is greater then that don't do the ticks just set the value to close to zero immediately. Actually approaching the smallest representable number is still measurably slow and not particularly useful. To avoid changing the observed output of the EWMA (which previous was only 0.0 if never used) set it close to Double.MIN_NORMAL rather then to 0.0
joschi
force-pushed
the
fix_slow_ewma_ticking
branch
from
171865a
to
84e7464
Compare
Refs #3929 Co-authored-by: aweisberg <ariel@weisberg.ws>
EWMA.tickIfNecessary does an amount of work that is linear to the amount of time that has passed since the last time the EWMA was ticked. For infrequently used EWMA this can lead to pauses observed in the 700-800 millisecond range after a few hundred days.
It's not really necessary to perform every tick as all that is doing is slowly approaching the smallest representable positive number in a double. Instead pick a number close to zero and then bound the number of ticks to allow that to be reachable from the largest value representable by the EWMA. Actually approaching the smallest representable number is still measurably slow and not particularly useful.