-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding a Link object #14
Conversation
An alternative would be to not use different link functions but define additional likelihood functions (i.e., one for Poisson with the |
I thought that it would be cooler to be able to input really any kind of link! One could even think about passing a constrained NN (paper idea here 😆 ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to keep links separated from the likelihoods? That is, not have it part of every likelihood? We could think of this as an optional step between the GP outputs and likelihood function.
We could also define LinkedLikelihood
similar to transforms in KernelFunctions.jl
such that it is just another likelihood.
struct LinkedLikelihood{T}
link::Link
lik::T
end
(ll::LinkedLikelihood)(f::Real) = ll.lik(ll.link(f))
(ll::LinkedLikelihood)(fs::AbstractVector{<:Real}) = ll.lik(ll.link.(fs))
or something like
struct LinkedLikelihood{T1,T2}
link::T1
lik::T2
end
(ll::LinkedLikelihood)(f::Real) = ll.lik(ll.link(f))
(ll::LinkedLikelihood)(fs::AbstractVector{<:Real}) = ll.lik(ll.link.(fs))
I am supportive of the second approach as both links and likelihoods are essentially functions. And the second approach provides the maximum amount of flexibility.
I don't think it is possible to separate links and likelihoods. Unlike in |
Okay that makes sense. 🙂 But we should always have a sensible default link in such cases. |
Actually one change that could be need is the naming. Should it be |
As an alternative (I feel I am rewriting existing code) we could use |
IMO we shouldn't use Bijectors.jl, mainly because it's a very heavy dependency and, more importantly, since my impression is that its design is very much focused on Turing and AdvancedHMC. In particular, all inputs and outputs have to be arrays and have to be of the same dimension and size. IMO this is quite limiting in more general applications where you want to map a submanifold of lower dimension (e.g. matrices with a special structure or the simplex - there are some open issues but I don't think the design will be changed in the near future since it would be very breaking for Turing). However, I also think it would be good to use existing functionality in other packages, if possible. Maybe TransformVariables could be helpful? IIRC it supports more flexible mappings and input and output types and would be a lighter dependency. |
I had a look at TransformVariables and it looks great indeed, only I think we would need to use the low level API the |
Depends on which way around you define the bijector. In the statistics community, the link function is well defined to be the object that satisfies link(y) = f, where y is your observations/likelihood parameters, and f is your latent (generally assumed linear) model. So y = invlink(f). In implementing likelihoods for our GP models we generally care about the f -> y direction only, so we define the invlink explicitly. |
@st-- Thanks for the explanation. I was indeed not familiar with the statistics nomenclature. The names have been changed accordingly already. One can switch from one representation to another (when possible) with |
Can I get a fresh review on this @devmotion @st-- @willtebbutt ? Happy to get it back from the deads |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just done a quick pass.
Codecov Report
@@ Coverage Diff @@
## master #14 +/- ##
===========================================
+ Coverage 69.56% 82.53% +12.97%
===========================================
Files 6 7 +1
Lines 23 63 +40
===========================================
+ Hits 16 52 +36
- Misses 7 11 +4
Continue to review full report at Codecov.
|
@st-- @willtebbutt |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Broadly looks really nice -- a number of style things / comments on docstrings, and a couple of small design things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm happy with this now.
We need to support different type of mapping when a constraint is applied on the GP, e.g. for Poisson the given standard was
exp(f)
but there are more existing options :f^2
, ors * logistic(f)
The convention is to call the connection between the parameter(s)
m
needed for the likelihood and the input(s)f
the inverse link. Cause traditionally the link would be the function to go fromm
tof
. See for example : http://web.pdx.edu/~newsomj/mvclass/ho_link.pdfThis PR aims at introducing this with
AbstractLink
type.Some likelihoods need an inverse link, for example
BernoulliLikelihood
orPoissonLikelihood
. So it makes sense to have these likelihoods contain the inverse linkinvlink
and parametrize it. This allows furthermore to differentiate different overloads later and for people to use their own transformations. For example my augmentation stuff only works for theLogitLink
but not theProbitLink
.For a small collection of
AbstractLink
s I defined justapply(link, x)
(which is called by(link::LinkType)(x)
. and alsoBase.inv
which returns the existing inverse link if it exists (similarly to Bijectors).I already did a basic implementation but I realised a lot of it is overlapping with
Bijectors.jl
and we could consider using them. However this is also a heavier dependency ? I thinkNNlib
is pretty heavy right?