Weighted KDE #26

axsk · 2016-06-27T10:53:12Z

In a project of mine I need a weighted KDE.
I achieved this by weighting the increase in the tabulate method.

Unfortunately in the current version I need to allocate an array for the weights, which is not necessary with uniform weights. Any suggestions on how to tackle this?

ararslan · 2016-07-07T07:13:29Z

src/univariate.jl

        end
    end

    # returns an un-convolved KDE
    UnivariateKDE(midpoints, grid)
 end

+function tabulate(data::RealVector, midpoints::Range)
+    weights = ones(data)
+    tabulate(data, weights, midpoints)


You could do something like tabulate(data, midpoints; weights=ones(data)), which makes the weighting more apparent (since it's a keyword argument) and removes the need for two separate methods like this.

As you say below, better use the WeightsVec type than a keyword argument for consistency.

ararslan · 2016-07-07T07:14:06Z

Since this package uses StatsBase, I'd recommend setting up the weighting using WeightVecs from StatsBase, or at least providing another method that uses them. Also, if you're adding weighted methods for univariate, I'd recommend adding similar functionality to bivariate.

nalimilan · 2016-07-07T09:09:37Z

To avoid allocating a vector of ones for unit weights, we could use a Ones/UnitWeights type as described at JuliaStats/StatsBase.jl#135.

EDIT: you could also add UniformWeights since you appear to need them.

nalimilan · 2016-07-07T09:12:58Z

src/univariate.jl

    npoints = length(midpoints)
    s = step(midpoints)

    # Set up a grid for discretized data
    grid = zeros(Float64, npoints)
-    ainc = 1.0 / (ndata*s*s)
+    ainc = 1.0 / (sum(weights)*s*s)


When you switch to WeightsVec, you can use its sum field to avoid recomputing its sum.

nalimilan · 2016-07-07T09:14:27Z

Could you also add tests?

lstagner · 2016-10-12T09:28:23Z

@axsk Are you planning to finish this PR? I also need to to calculate a weighted KDE.

axsk · 2016-10-13T09:09:01Z

@lstagner
Hey, I would like to finish this PR, but I still don't know how to handle the uniform weights properly.
I could work around the uniform array allocation by creating a new UniformWeight type, doing the job (but I think this belongs rather to StatsBase, maybe the next step :))
Edit: done

@ararslan
I agree that having a bivariate weighted KDE would be nice, but I would like to leave that for someone else so far..
Edit: well, if @lstagner said it wasn't that difficulat I had to try it, went fast :)

@nalimilan
How do you imagine the tests, I can't think of a simple example I can calculate in my head, so basically I could just test runnability.

lstagner · 2016-10-13T09:27:19Z

@axsk I got a little impatient and implemented it myself. It wasn't super difficult. I also did the bivariate case.

Heres the code
lstagner@b48fb2d

axsk · 2016-10-13T09:38:07Z

So how do we proceed? :)

While I like using StatsBase.WeightsVec internally, I think the API should still allow plain Vectors.
I will push my suggestion :)

lstagner · 2016-10-13T11:00:56Z

src/univariate.jl

+
+typealias Weights Union{UniformWeights, RealVector, WeightVec}
+
+


You could have the length be a part of the type parameter so that its slightly more efficient.

type UniformWeights{N} end UniformWeights(N) = UniformWeights{N}() Base.sum(x::UniformWeights) = 1.0 Base.getindex{N}(x::UniformWeights{N}, i) = 1/N Base.length{N}(x::UniformWeights{N}) = N values{N}(x::UniformWeights{N}) = fill(x[1], N) eltype(x::UniformWeights) = eltype(x[1]) isempty(x::UniformWeights) = false

axsk · 2016-10-13T11:15:05Z

I think its all done, but the tests.
Any recommendations on those?

lstagner · 2016-12-13T00:18:01Z

Bump

lstagner · 2017-01-22T12:51:51Z

I still want this to be included. The only thing holding it back are some tests. @axsk perhaps you should just mimic the current tests to get started.

chriselrod · 2017-05-04T04:16:04Z

Also, how about letting weights be negative?

That would be useful when dealing with SparseGrids, for example.

ararslan · 2017-05-04T05:51:12Z

Negative weights, if they are currently possible, will not be possible soon in StatsBase.

chriselrod · 2017-05-04T08:35:40Z

That's fine, and less interesting than just adding weights in the first place.

A simple ad hoc means of implementing negative weights is to just divide your data into positive and negative weights, create two KDEs, and then use the sum of the respective weights to take a weighted difference of the pdf evaluations. Not saying this should be included in the package, just that if anyone really wants to make plots or something, it's easy enough to work around.

Is adding tests to improve coverage all that remains to be done here?

getting rid of the 4 covarge misses to fix coverage degress.

axsk · 2017-05-18T14:14:58Z

Tests are done, coverage improved, should I squash it all to one?

lstagner · 2017-05-26T17:55:55Z

Now that the tests have been added and all the checks have passed can this please get merged? I use this functionality daily and it would be great if didn't have to use a different branch.

lstagner · 2017-06-07T22:33:04Z

Bump @nalimilan @simonbyrne

lstagner · 2017-06-27T10:23:26Z

🎉 Happy Anniversary!!! 🎉

This PR has been open for 1 whole year!!! This is a major accomplishment. It has really beaten the odds. Honestly, I thought it was finished well over a month ago but I was proven wrong. Here's to another year. 🥂

simonbyrne · 2017-06-27T19:07:35Z

Thanks @axsk

remoore · 2017-08-24T20:46:18Z

I'm trying to make use of the of the weighted KDE functionality described above. I can't seem to get it to work despite having run the following code, which is cut and paste from test/univariate.jl:

using Distributions
using KernelDensity

import KernelDensity: kde_range

r = kde_range((-2.0,2.0), 128)
X = [0.0]   
D = Normal
k6 = kde(X,r;kernel=D, weights=ones(X)/length(X))
k1 = kde([0.0, 1.], r, bandwidth=1, weights=[0,1])

I'm getting the following error message:

MethodError: no method matching kde(::Array{Float64,1}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}; kernel=Distributions.Normal, weights=[1.0])
Closest candidates are:
  kde(::AbstractArray{T,1} where T<:Real, ::Range; bandwidth, kernel) at /Users/remoore/.julia/v0.6/KernelDensity/src/univariate.jl:136 got unsupported keyword argument "weights"
  kde(::AbstractArray{T,1} where T<:Real, ::Range, ::Distributions.Distribution{Distributions.Univariate,S} where S<:Distributions.ValueSupport) at /Users/remoore/.julia/v0.6/KernelDensity/src/univariate.jl:123 got unsupported keyword arguments "kernel", "weights"
  kde(::AbstractArray{T,1} where T<:Real; bandwidth, kernel, npoints, boundary) at /Users/remoore/.julia/v0.6/KernelDensity/src/univariate.jl:143 got unsupported keyword argument "weights"
  ...

Stacktrace:
 [1] (::KernelDensity.#kw##kde)(::Array{Any,1}, ::KernelDensity.#kde, ::Array{Float64,1}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}) at ./<missing>:0
 [2] include_string(::String, ::String) at ./loading.jl:515

Any ideas why this isn't working?

I've only just started using Julia, so apologies in advance if this is a stupid question

lstagner · 2017-08-24T21:00:44Z

The weights keyword accepts a Weights (formally WeightsVec) type from StatsBase

The following works.

using Distributions
using KernelDensity

import KernelDensity: kde_range
import StatsBase: Weights

r = kde_range((-2.0,2.0), 128)
X = [0.0]   
D = Normal
k6 = kde(X,r;kernel=D, weights=Weights(ones(X)/length(X)))
k1 = kde([0.0, 1.], r, bandwidth=1, weights=Weights([0,1]))

remoore · 2017-08-24T21:10:22Z

Thanks for the help, but it's still not behaving:

`MethodError: no method matching kde(::Array{Float64,1}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}; kernel=Distributions.Normal, weights=[1.0])
Closest candidates are:
kde(::AbstractArray{T,1} where T<:Real, ::Range; bandwidth, kernel) at /Users/remoore/.julia/v0.6/KernelDensity/src/univariate.jl:136 got unsupported keyword argument "weights"
kde(::AbstractArray{T,1} where T<:Real, ::Range, ::Distributions.Distribution{Distributions.Univariate,S} where S<:Distributions.ValueSupport) at /Users/remoore/.julia/v0.6/KernelDensity/src/univariate.jl:123 got unsupported keyword arguments "kernel", "weights"
kde(::AbstractArray{T,1} where T<:Real; bandwidth, kernel, npoints, boundary) at /Users/remoore/.julia/v0.6/KernelDensity/src/univariate.jl:143 got unsupported keyword argument "weights"
...

Stacktrace:
[1] (::KernelDensity.#kw##kde)(::Array{Any,1}, ::KernelDensity.#kde, ::Array{Float64,1}, ::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}) at ./:0
[2] include_string(::String, ::String) at ./loading.jl:515`

lstagner · 2017-08-24T21:17:40Z

Well it works on 0.5.1 (I haven't updated to 0.6 yet)

Edit: Actually I forgot I'm using my own version of this functionality.

Edit^2: Your example works on 0.5.1 using this branch (without using Weights as is needed on my branch).

lstagner · 2017-08-24T21:28:14Z

Also it could be that master hasn't been tagged yet. So try doing Pkg.checkout("KernelDensity") to the latest version.

ararslan · 2017-08-24T21:28:56Z

Yes, there have been no tags since this was merged

remoore · 2017-08-24T21:50:59Z

I've just checked out the latest version and it works on Julia 0.6.0.

Thanks for the help

axsk · 2017-10-12T14:22:39Z

Could we tag this version?

ararslan · 2017-10-12T18:21:45Z

JuliaLang/METADATA.jl#11541

axsk added 2 commits June 10, 2016 19:46

extend tabulate, wrap with weights=ones(data)

ce9ca33

finish

903070f

ararslan reviewed Jul 7, 2016
View reviewed changes

nalimilan reviewed Jul 7, 2016
View reviewed changes

axsk mentioned this pull request Jul 8, 2016

KDE with weighted data #25

Closed

Merge branch 'master' into HEAD

3ade534

axsk added 3 commits October 13, 2016 11:38

implement UniformWeights

0af7754

add bivariate weights

fd3a44a

change tabulate() signature; use default_weights()

af24d40

lstagner reviewed Oct 13, 2016

View reviewed changes

add length parameter to type

3ff043e

axsk added 5 commits May 18, 2017 14:07

add test for weighted univ. kde

7ad63b0

Merge branch 'master' into weights

26529d9

fix test bracket

b1491ee

remove unneeded methods from UniformWeights

346874f

getting rid of the 4 covarge misses to fix coverage degress.

add bivariate tests

4a3680a

axsk changed the title ~~RFC: Weighted KDE~~ Weighted KDE May 18, 2017

ararslan approved these changes May 26, 2017

View reviewed changes

ararslan requested review from nalimilan and simonbyrne May 26, 2017 18:00

simonbyrne merged commit 4bcdc23 into JuliaStats:master Jun 27, 2017

Nosferican mentioned this pull request May 27, 2019

density with weights JuliaPlots/StatsPlots.jl#201

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Weighted KDE #26

Weighted KDE #26

axsk commented Jun 27, 2016

ararslan Jul 7, 2016

nalimilan Jul 7, 2016

ararslan commented Jul 7, 2016 •

edited

Loading

nalimilan commented Jul 7, 2016 •

edited

Loading

nalimilan Jul 7, 2016

nalimilan commented Jul 7, 2016

lstagner commented Oct 12, 2016

axsk commented Oct 13, 2016 •

edited

Loading

lstagner commented Oct 13, 2016

axsk commented Oct 13, 2016

lstagner Oct 13, 2016

axsk Oct 13, 2016

axsk commented Oct 13, 2016 •

edited

Loading

lstagner commented Dec 13, 2016

lstagner commented Jan 22, 2017

chriselrod commented May 4, 2017

ararslan commented May 4, 2017

chriselrod commented May 4, 2017 •

edited

Loading

axsk commented May 18, 2017

lstagner commented May 26, 2017

lstagner commented Jun 7, 2017

lstagner commented Jun 27, 2017

simonbyrne commented Jun 27, 2017

remoore commented Aug 24, 2017 •

edited by ararslan

Loading

lstagner commented Aug 24, 2017

remoore commented Aug 24, 2017

lstagner commented Aug 24, 2017 •

edited

Loading

lstagner commented Aug 24, 2017

ararslan commented Aug 24, 2017

remoore commented Aug 24, 2017

axsk commented Oct 12, 2017

ararslan commented Oct 12, 2017


		typealias Weights Union{UniformWeights, RealVector, WeightVec}

Weighted KDE #26

Weighted KDE #26

Conversation

axsk commented Jun 27, 2016

ararslan Jul 7, 2016

Choose a reason for hiding this comment

nalimilan Jul 7, 2016

Choose a reason for hiding this comment

ararslan commented Jul 7, 2016 • edited Loading

nalimilan commented Jul 7, 2016 • edited Loading

nalimilan Jul 7, 2016

Choose a reason for hiding this comment

nalimilan commented Jul 7, 2016

lstagner commented Oct 12, 2016

axsk commented Oct 13, 2016 • edited Loading

lstagner commented Oct 13, 2016

axsk commented Oct 13, 2016

lstagner Oct 13, 2016

Choose a reason for hiding this comment

axsk Oct 13, 2016

Choose a reason for hiding this comment

axsk commented Oct 13, 2016 • edited Loading

lstagner commented Dec 13, 2016

lstagner commented Jan 22, 2017

chriselrod commented May 4, 2017

ararslan commented May 4, 2017

chriselrod commented May 4, 2017 • edited Loading

axsk commented May 18, 2017

lstagner commented May 26, 2017

lstagner commented Jun 7, 2017

lstagner commented Jun 27, 2017

simonbyrne commented Jun 27, 2017

remoore commented Aug 24, 2017 • edited by ararslan Loading

lstagner commented Aug 24, 2017

remoore commented Aug 24, 2017

lstagner commented Aug 24, 2017 • edited Loading

lstagner commented Aug 24, 2017

ararslan commented Aug 24, 2017

remoore commented Aug 24, 2017

axsk commented Oct 12, 2017

ararslan commented Oct 12, 2017

ararslan commented Jul 7, 2016 •

edited

Loading

nalimilan commented Jul 7, 2016 •

edited

Loading

axsk commented Oct 13, 2016 •

edited

Loading

axsk commented Oct 13, 2016 •

edited

Loading

chriselrod commented May 4, 2017 •

edited

Loading

remoore commented Aug 24, 2017 •

edited by ararslan

Loading

lstagner commented Aug 24, 2017 •

edited

Loading