Quadrature (and Monte Carlo) for computing variational expectations

[rossviljoen]

For many non-Gaussian likelihoods, the expectation in the first term in the ELBO E_q(f) [ \log p(y|f) ] from [1] has no closed form solution, so it needs to approximated somehow - typically by Gauss-Hermite quadrature, but possibly also via Monte Carlo. So, the planned approach for implementing this is:

• An initial end-to-end working version of quadrature using FastGaussQuadrature.jl with AD.
• Compile a list of which non-Gaussian likelihoods have closed form solutions (probably by looking through GPFlow etc.)
• Add support for MC methods.

[1] Hensman, James, Alexander Matthews, and Zoubin Ghahramani. "Scalable variational Gaussian process classification." Artificial Intelligence and Statistics. PMLR, 2015.

[theogf]

Compile a list of which non-Gaussian likelihoods have closed form solutions (probably by looking through GPFlow etc.)

I am afraid that only works for the Gaussian likelihood :(

[rossviljoen]

Isn't it possible for some - e.g. for Gamma here?

[st--]

I am afraid that only works for the Gaussian likelihood :(

Closed-form posterior only for Gaussian likelihood, but closed-form E_q(f)[ \log p(y|f) ] exists for some other p(y|f) as well, as in the example Ross found.

[rossviljoen]

I have a working end-to-end example of classification in https://github.com/rossviljoen/SparseGPs.jl/blob/master/examples/classification.jl

[rossviljoen]

Implemented in #9