ESS Fix

[cpfiffer]

Fixes #90 by actually doing the math (mostly) right. ESS now also reports Rhat, as below:

ess(chn)

ESS

│ Row │ parameters │ ess     │ r_hat    │
│     │ Symbol     │ Any     │ Any      │
├─────┼────────────┼─────────┼──────────┤
│ 1   │ theta      │ 1558.33 │ 0.999004 │

Rhat will also show up in the summary stats block.

[goedman]

Just out of interest, in addition to the ess plot, a plot for r_hat. Below both plots.

ess__estimates_plot.pdf
rhat__estimates_plot.pdf

For ess the values for 100 simulations (run) came from stansummary, MCMCChains and MCMCDiagnostics.

For r-hat the plotted values came from stansummary and MCMCChains.

The test program is part of StatisticalRethinking ess-rhat_tests.jl.

[itsdfish]

Rob-

I have a quick question about the r-hat plot. If I understand correctly, it seems like Turing produces larger and more variable r-hat than CmdStan for the same model, data and r-hat function. However, it appears that the delta parameter for Turing is .95 and .80 for CmdStan. Is this causing the difference and is that intentional?

[goedman]

Yes, both the ess and r-hat show much greater variability, which surprised me as well. Early on I did use runs with the accept target at .65 and .85, and saw no difference for ess. Will run the test with .80, just to make sure.

[itsdfish]

Equating the target acceptance rate seems like a good idea. If the problem persists, it suggests there might be a problem with Turing somewhere.

[goedman]

These are the plots with the accept target at .80:

ess__estimates_plot.pdf
rhat__estimates_plot.pdf

[itsdfish]

I've had a suspicion for a while that Turing is not producing the right results, especially for autocorrelation/ESS. It might be worth filing an issue unless you can think of some other explanation for the large divergence.

[goedman]

And with delta = 0.65:

ess__estimates_plot.pdf
rhat__estimates_plot.pdf

[goedman]

Will do that when back in DC an check on Slack first. Of course I’m the world’s most limited Turing user, just Nuts.

[itsdfish]

That sounds good. Let me know if you want me to file an issue to help out. I just wanted to make sure you couldn't think of any other explanations for the behavior.