Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecate varm and stdm #736

Merged
merged 2 commits into from
Nov 26, 2021
Merged

Deprecate varm and stdm #736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
38efcf5
Deprecate `varm` and `stdm`
nalimilan Nov 21, 2021
bad23b4
Adjust alignment in test/moments.jl
andreasnoack Nov 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/cov.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ cov(x::DenseMatrix, w::AbstractWeights, dims::Int=1; corrected::DepBool=nothing)

function corm(x::DenseMatrix, mean, w::AbstractWeights, vardim::Int=1)
c = covm(x, mean, w, vardim; corrected=false)
s = stdm(x, w, mean, vardim; corrected=false)
s = std(x, w, vardim; mean=mean, corrected=false)
cov2cor!(c, s)
end

Expand Down
7 changes: 7 additions & 0 deletions src/deprecates.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,10 @@ end
### Deprecated September 2019
@deprecate sum(A::AbstractArray, w::AbstractWeights, dims::Int) sum(A, w, dims=dims)
@deprecate values(wv::AbstractWeights) convert(Vector, wv)

### Deprecated November 2021
@deprecate stdm(x::RealArray, w::AbstractWeights, m::Real; corrected::DepBool=nothing) std(x, w, mean=m, corrected=corrected) false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we disable export? This is breaking I think

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, because it's not exported currently. We still overload the function defined in Statistics though to it can be used after using Statistics.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah - OK. I mixed it up with the fact that Statistics exports it.

@deprecate varm(x::RealArray, w::AbstractWeights, m::Real; corrected::DepBool=nothing) var(x, w, mean=m, corrected=corrected) false
@deprecate stdm(x::RealArray, w::AbstractWeights, m::RealArray, dim::Int; corrected::DepBool=nothing) std(x, w, dim, mean=m, corrected=corrected) false
@deprecate varm(x::RealArray, w::AbstractWeights, m::RealArray, dim::Int; corrected::DepBool=nothing) var(x, w, dim, mean=m, corrected=corrected) false
@deprecate varm!(R::AbstractArray, x::RealArray, w::AbstractWeights, m::RealArray, dim::Int; corrected::DepBool=nothing) var!(R, x, w, dim, mean=m, corrected=corrected) false
99 changes: 19 additions & 80 deletions src/moments.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,26 +1,6 @@
##### Weighted var & std

## var
"""
varm(x::AbstractArray, w::AbstractWeights, m, [dim]; corrected=false)

Compute the variance of a real-valued array `x` with a known mean `m`, optionally
over a dimension `dim`. Observations in `x` are weighted using weight vector `w`.
The uncorrected (when `corrected=false`) sample variance is defined as:
```math
\\frac{1}{\\sum{w}} \\sum_{i=1}^n {w_i\\left({x_i - m}\\right)^2 }
```
where ``n`` is the length of the input. The unbiased estimate (when `corrected=true`) of
the population variance is computed by replacing
``\\frac{1}{\\sum{w}}`` with a factor dependent on the type of weights used:
* `AnalyticWeights`: ``\\frac{1}{\\sum w - \\sum {w^2} / \\sum w}``
* `FrequencyWeights`: ``\\frac{1}{\\sum{w} - 1}``
* `ProbabilityWeights`: ``\\frac{n}{(n - 1) \\sum w}`` where ``n`` equals `count(!iszero, w)`
* `Weights`: `ArgumentError` (bias correction not supported)
"""
varm(v::RealArray, w::AbstractWeights, m::Real; corrected::DepBool=nothing) =
_moment2(v, w, m; corrected=depcheck(:varm, :corrected, corrected))

"""
var(x::AbstractArray, w::AbstractWeights, [dim]; mean=nothing, corrected=false)

Expand All @@ -43,30 +23,22 @@ function var(v::RealArray, w::AbstractWeights; mean=nothing,
corrected = depcheck(:var, :corrected, corrected)

if mean == nothing
varm(v, w, Statistics.mean(v, w); corrected=corrected)
_moment2(v, w, Statistics.mean(v, w); corrected=corrected)
Copy link
Contributor

@bkamins bkamins Nov 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not var with kwarg? same below

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're already in var. Calling ourselves with the computed mean would work but do you think it would really be cleaner?

(I'd like to deprecate _moment2 and similar functions at some point to simplify the API and implementation...)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:)

else
varm(v, w, mean; corrected=corrected)
_moment2(v, w, mean; corrected=corrected)
end
end

## var along dim

function varm!(R::AbstractArray, A::RealArray, w::AbstractWeights, M::RealArray,
dim::Int; corrected::DepBool=nothing)
corrected = depcheck(:varm!, :corrected, corrected)
rmul!(_wsum_centralize!(R, abs2, A, convert(Vector, w), M, dim, true),
varcorrection(w, corrected))
end

function var!(R::AbstractArray, A::RealArray, w::AbstractWeights, dims::Int;
mean=nothing, corrected::DepBool=nothing)
corrected = depcheck(:var!, :corrected, corrected)

if mean == 0
varm!(R, A, w, Base.reducedim_initarray(A, dims, 0, eltype(R)), dims;
corrected=corrected)
elseif mean == nothing
varm!(R, A, w, Statistics.mean(A, w, dims=dims), dims; corrected=corrected)
mean = Base.reducedim_initarray(A, dims, 0, eltype(R))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this not tested. Also the same as above - why is it better than just calling the default replacement?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out it's not tested on master already. This is an undocumented feature of var (which calls var!). I wish it wasn't allowed, but I guess we have keep supporting it now (and add tests for it)...

elseif mean === nothing
mean = Statistics.mean(A, w, dims=dims)
else
# check size of mean
for i = 1:ndims(A)
Expand All @@ -78,15 +50,9 @@ function var!(R::AbstractArray, A::RealArray, w::AbstractWeights, dims::Int;
dM == dA || throw(DimensionMismatch("Incorrect size of mean."))
end
end
varm!(R, A, w, mean, dims; corrected=corrected)
end
end

function varm(A::RealArray, w::AbstractWeights, M::RealArray, dim::Int;
corrected::DepBool=nothing)
corrected = depcheck(:varm, :corrected, corrected)
varm!(similar(A, Float64, Base.reduced_indices(axes(A), dim)), A, w, M,
dim; corrected=corrected)
return rmul!(_wsum_centralize!(R, abs2, A, convert(Vector, w), mean, dims, true),
varcorrection(w, corrected))
end

function var(A::RealArray, w::AbstractWeights, dim::Int; mean=nothing,
Expand All @@ -97,26 +63,6 @@ function var(A::RealArray, w::AbstractWeights, dim::Int; mean=nothing,
end

## std
"""
stdm(x::AbstractArray, w::AbstractWeights, m, [dim]; corrected=false)

Compute the standard deviation of a real-valued array `x` with a known mean `m`,
optionally over a dimension `dim`. Observations in `x` are weighted using weight vector `w`.
The uncorrected (when `corrected=false`) sample standard deviation is defined as:
```math
\\sqrt{\\frac{1}{\\sum{w}} \\sum_{i=1}^n {w_i\\left({x_i - m}\\right)^2 }}
```
where ``n`` is the length of the input. The unbiased estimate (when `corrected=true`) of the
population standard deviation is computed by replacing ``\\frac{1}{\\sum{w}}`` with a factor
dependent on the type of weights used:
* `AnalyticWeights`: ``\\frac{1}{\\sum w - \\sum {w^2} / \\sum w}``
* `FrequencyWeights`: ``\\frac{1}{\\sum{w} - 1}``
* `ProbabilityWeights`: ``\\frac{n}{(n - 1) \\sum w}`` where ``n`` equals `count(!iszero, w)`
* `Weights`: `ArgumentError` (bias correction not supported)
"""
stdm(v::RealArray, w::AbstractWeights, m::Real; corrected::DepBool=nothing) =
sqrt(varm(v, w, m, corrected=depcheck(:stdm, :corrected, corrected)))

"""
std(x::AbstractArray, w::AbstractWeights, [dim]; mean=nothing, corrected=false)

Expand All @@ -138,15 +84,8 @@ weights used:
std(v::RealArray, w::AbstractWeights; mean=nothing, corrected::DepBool=nothing) =
sqrt.(var(v, w; mean=mean, corrected=depcheck(:std, :corrected, corrected)))

stdm(v::RealArray, m::RealArray, dim::Int; corrected::DepBool=nothing) =
sqrt!(varm(v, m, dims=dim, corrected=depcheck(:stdm, :corrected, corrected)))

stdm(v::RealArray, w::AbstractWeights, m::RealArray, dim::Int;
corrected::DepBool=nothing) =
sqrt.(varm(v, w, m, dim; corrected=depcheck(:stdm, :corrected, corrected)))

std(v::RealArray, w::AbstractWeights, dim::Int; mean=nothing,
corrected::DepBool=nothing) =
std(v::RealArray, w::AbstractWeights, dim::Int;
mean=nothing, corrected::DepBool=nothing) =
sqrt.(var(v, w, dim; mean=mean, corrected=depcheck(:std, :corrected, corrected)))

##### Fused statistics
Expand All @@ -161,7 +100,7 @@ See [`var`](@ref) documentation for more details.
"""
function mean_and_var(x; corrected::Bool=true)
m = mean(x)
v = varm(x, m; corrected=corrected)
v = var(x, mean=m, corrected=corrected)
m, v
end

Expand All @@ -177,44 +116,44 @@ See [`std`](@ref) documentation for more details.
"""
function mean_and_std(x; corrected::Bool=true)
m = mean(x)
s = stdm(x, m; corrected=corrected)
s = std(x, mean=m, corrected=corrected)
m, s
end

function mean_and_var(x::RealArray, w::AbstractWeights; corrected::DepBool=nothing)
m = mean(x, w)
v = varm(x, w, m; corrected=depcheck(:mean_and_var, :corrected, corrected))
v = var(x, w, mean=m, corrected=depcheck(:mean_and_var, :corrected, corrected))
m, v
end
function mean_and_std(x::RealArray, w::AbstractWeights; corrected::DepBool=nothing)
m = mean(x, w)
s = stdm(x, w, m; corrected=depcheck(:mean_and_std, :corrected, corrected))
s = std(x, w, mean=m, corrected=depcheck(:mean_and_std, :corrected, corrected))
m, s
end


function mean_and_var(x::RealArray, dim::Int; corrected::Bool=true)
m = mean(x, dims = dim)
v = varm(x, m, dims = dim, corrected=corrected)
m = mean(x, dims=dim)
v = var(x, dims=dim, mean=m, corrected=corrected)
m, v
end
function mean_and_std(x::RealArray, dim::Int; corrected::Bool=true)
m = mean(x, dims = dim)
s = stdm(x, m, dim; corrected=corrected)
m = mean(x, dims=dim)
s = std(x, dims=dim, mean=m, corrected=corrected)
m, s
end


function mean_and_var(x::RealArray, w::AbstractWeights, dims::Int;
corrected::DepBool=nothing)
m = mean(x, w, dims=dims)
v = varm(x, w, m, dims; corrected=depcheck(:mean_and_var, :corrected, corrected))
v = var(x, w, dims, mean=m, corrected=depcheck(:mean_and_var, :corrected, corrected))
m, v
end
function mean_and_std(x::RealArray, w::AbstractWeights, dims::Int;
corrected::DepBool=nothing)
m = mean(x, w, dims=dims)
s = stdm(x, w, m, dims; corrected=depcheck(:mean_and_std, :corrected, corrected))
s = std(x, w, dims, mean=m, corrected=depcheck(:mean_and_std, :corrected, corrected))
m, s
end

Expand Down
20 changes: 16 additions & 4 deletions test/moments.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,13 @@ w = [3.84, 2.70, 8.29, 8.91, 9.71, 0.0]
@testset "Variance" begin
@test var(x, wv; corrected=false) ≈ expected_var
@test var(x, wv; mean=m, corrected=false) ≈ expected_var
@test varm(x, wv, m; corrected=false) ≈ expected_var
end

@testset "Standard Deviation" begin
@test std(x, wv; corrected=false) ≈ expected_std
@test std(x, wv; mean=m, corrected=false) ≈ expected_std
@test stdm(x, wv, m; corrected=false) ≈ expected_std
end

@testset "Mean and Variance" begin
Expand Down Expand Up @@ -62,6 +64,7 @@ expected_std = sqrt.(expected_var)
else
@test var(x, wv; corrected=true) ≈ expected_var[i]
@test var(x, wv; mean=m, corrected=true) ≈ expected_var[i]
@test varm(x, wv, m; corrected=true) ≈ expected_var[i]
end
end

Expand All @@ -71,6 +74,7 @@ expected_std = sqrt.(expected_var)
else
@test std(x, wv; corrected=true) ≈ expected_std[i]
@test std(x, wv; mean=m, corrected=true) ≈ expected_std[i]
@test stdm(x, wv, m; corrected=true) ≈ expected_std[i]
end
end

Expand Down Expand Up @@ -119,17 +123,21 @@ w2 = [3.84, 2.70, 8.29, 8.91, 9.71, 0.0]
expected_std2 = sqrt.(expected_var2)

@testset "Variance" begin
@test var(x, wv1, 1; corrected=false) ≈ expected_var1
@test var(x, wv2, 2; corrected=false) ≈ expected_var2
@test var(x, wv1, 1; corrected=false) ≈ expected_var1
@test var(x, wv2, 2; corrected=false) ≈ expected_var2
@test var(x, wv1, 1; mean=m1, corrected=false) ≈ expected_var1
@test var(x, wv2, 2; mean=m2, corrected=false) ≈ expected_var2
@test varm(x, wv1, m1, 1; corrected=false) ≈ expected_var1
@test varm(x, wv2, m2, 2; corrected=false) ≈ expected_var2
end

@testset "Standard Deviation" begin
@test std(x, wv1, 1; corrected=false) ≈ expected_std1
@test std(x, wv2, 2; corrected=false) ≈ expected_std2
@test std(x, wv1, 1; mean=m1, corrected=false) ≈ expected_std1
@test std(x, wv2, 2; mean=m2, corrected=false) ≈ expected_std2
@test stdm(x, wv1, m1, 1; corrected=false) ≈ expected_std1
@test stdm(x, wv2, m2, 2; corrected=false) ≈ expected_std2
end

@testset "Mean and Variance" begin
Expand Down Expand Up @@ -182,10 +190,12 @@ end
if isa(wv1, Weights)
@test_throws ArgumentError var(x, wv1, 1; corrected=true)
else
@test var(x, wv1, 1; corrected=true) ≈ expected_var1
@test var(x, wv2, 2; corrected=true) ≈ expected_var2
@test var(x, wv1, 1; corrected=true) ≈ expected_var1
@test var(x, wv2, 2; corrected=true) ≈ expected_var2
@test var(x, wv1, 1; mean=m1, corrected=true) ≈ expected_var1
@test var(x, wv2, 2; mean=m2, corrected=true) ≈ expected_var2
@test varm(x, wv1, m1, 1; corrected=true) ≈ expected_var1
@test varm(x, wv2, m2, 2; corrected=true) ≈ expected_var2
end
end

Expand All @@ -197,6 +207,8 @@ end
@test std(x, wv2, 2; corrected=true) ≈ expected_std2
@test std(x, wv1, 1; mean=m1, corrected=true) ≈ expected_std1
@test std(x, wv2, 2; mean=m2, corrected=true) ≈ expected_std2
@test stdm(x, wv1, m1, 1; corrected=true) ≈ expected_std1
@test stdm(x, wv2, m2, 2; corrected=true) ≈ expected_std2
end
end

Expand Down