Update wtema

[Mengerian]

Remove global variable
Re-arrange math
Add expanded comments

[dgenr8]

Nice, ACK.

[kyuupichan]

Beautiful; I like the arithmetic too. Doesn't lose precision unnecessarily and doesn't overflow.

[zawy12]

It's not a wtema, it's a standard solvetime EMA. [edit: correction, it's a EMA of the "needed target"]

Can the two lines be combined into one for clarity and simplified to what I posted yesterday?
wtema_target = wtema_target // alpha_recip * (block_time//IDEAL_BLOCK_TIME + alpha_recip - 1 )

I would modify the comment on the alpha_recip to be
N =~ floor(1/(1 - pow(2.71828, -1/N)))
To make the mathematical source of N=alpha_recip clear. The "72" makes it less comparable to other algorithms and prevented us from initially seeing that this works indistinguishably from Jacob's hashrate EMA, other than negative sol

Devs often do not want to allow a negative block time and if they use
block_time = 0 if block_time < 0
it will leave them open to a disastrous exploit, so I would place comment:
# negative block_times must be allowed or an exploit is possible

[jacob-eliosoff]

The comments are very helpful, thanks.

wtema and my ema both use similar exp moving avg logic: as discussed on the other thread, the big difference is that wtema is "block-based" (blocks are weighted just by their block #, ignoring time), whereas ema is "time-based" (longer blocks, and blocks more recent in clock time, get more weight). As I argued there, I believe weighting by time yields better behavior, especially in the case where miners leave and blocks suddenly get very slow. But overall the two are similar and in most scenarios either should do.

[jacob-eliosoff]

(For reference, most of https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average discusses the "block-based" EMA, and that's the one I've mostly seen in finance. https://en.wikipedia.org/wiki/Moving_average#Application_to_measuring_computer_performance describes a "time-based" EMA, which is the type I used extensively in my algo market making bot.)

[zawy12]

Using e^x=1+x in yours shows they are equal when t=T, which is far from being able to say they are equal. But as avg t = T, then on average they maybe equal. All the plots indicate "same". So, if I make the assumption the approximation is equal and extend your accuracy to e^x=1+x+x^2/2, then your accuracy is improved by:
Jacob's EMA = Tom's EMA *(1 + t /T/2/N^3 * (t/T-1))
This is higher D for longer solvetimes, so it should respond less than Tom's by this very small factor, which implies Tom's over-reacts some since his avg ST is not as accurate. Even with N=20 which might be best for T=600 seconds, this is pretty small. A high t=10xT with this small N gives 1.006. ( Note there's no correction when t=T which helps to validate the math. )

But the EMA is pretty sensitive to differences. Surprisingly, the absence of the factor causes a 2.5% error in avg solvetime at N=20. So for my purposes, if the WT and WHM are not better due to rising faster, the EMA with this approximation (for integer math) is the best, assuming I don't want to take advantage of the negative solvetimes Tom's provides.

[zawy12]

The t=T could be saying that he has a T where you have a t, which would justify your statement that it is time-based and his is block based. But I can't believe that because the approximation of yours and his have only one t and they are in their "reciprocal" places.

[jacob-eliosoff]

The simplest case that illustrates the difference is the one I gave: very slow (many-hour) blocks. My understanding is block-based EMAs like wtema adjust difficulty no more sharply in response to 10 very slow blocks than in response to 10 very fast blocks. Time-based EMAs adjust difficulty much more sharply in response to 10 very slow blocks (or even 1 very slow block) than to 10 very fast ones.

[zawy12]

No, they're the same. Here's hashrate drop to 1/50th with N=20 (I've decided N=20 for T=600 seconds is best)

[dgenr8]

Bitcoin Cash had a negative experience with weighting corrections in one direction more heavily than the other. EDA miners would withhold hash rate to drive the difficulty down and then feast on resulting correction for an extended time.

[zawy12]

What are comparing that situation to? They didn't do anything except follow simple profit motive based on seeking the best Price/Difficulty ratio between BTC and BCH. I would call what they did limits rather than weightings. There was a limit on the ability to rise quickly. If the ability to rise logic was the inverse of the ability to fall logic then it would have been OK. When the ability to fall is greater than the ability to rise, then avg solvetime decreases. We're seeing it in BTG now, about 5% too fast. Conversely, less of a limit on rising verses falling would cause longer avg solvetime. But yeah, giving stronger weight to help D fall faster than rise will also cause shorter avg solvetime. Your EMA has a little bit too high avg solvetime which implies D is being allowed to rise faster than fall. I made ideal block time slightly smaller to counteract the effect. The equation above shows why your D is allowed to rise ever-so-slightly faster than it falls.

[jacob-eliosoff]

Huh! In a quick test, @zawy12 appears to be right: wtema-100 and ema-1d increase and decrease difficulty ~exactly the same! Guess I should have followed all your equations.

I'll have a closer look later but maybe wtema is just a more clever, integer-math implementation of the same EMA. In which case, great.

If so, the only reason to maybe look at ema2 would be the (timestamp - max_previous_timestamp) logic for handling extreme timestamps.

[zawy12]

wtema (which I'm going to call EMA-TH and yours EMA-JE) is best because it allows negative timestamps. There are always exploits when we place an asymmetrical limit on the solvetimes. The exploits are not bad at N>100 but N=100 is way too high. I've linked elsewhere to my article that goes into excruciating depth on the exploits.

For lower N values, you have to use this to prevent a 20% miner from lowering D by 30%:

limit=6
maxT=timestamp[height-limit_ST-2] 
for ( i = height - limit -1 to i <= height-2 )  { maxT = max(maxT, timestamp[i]) }
ST = timestamp[height-1] - maxT 
ST = max(T/200,min(T*limit, ST))

But the above raises avg solvetime by 0.5% and reduces ability to respond, and still leaves a 20% miner the ability to lower difficulty by 20%. The way it uses limit in the last line makes it symmetrical to prevent problems.

The following keeps avg solvetime, and does not allow a 20% miner to get anything by manipulating the timestamps. The block after bad timestamp can go low or high, but a correct timestamp after it immediately corrects it. This is what people fail to realise as the enormous advantage of allowing negative solvetimes.

    if ST > (7+1)*TT then ST=(7+1)*TT
    if ST < -(7-1)*TT then ST= -(7-1)*TT

[jacob-eliosoff]

See my latest thoughts at #30 (comment) (trying to avoid thread fragmentation...)