Definition of == for NullableArray doesn't seem to work

[quinnj]

My guess is we're relying on some AbstractArray behavior, but the fact that we return Nullable{Bool} instead of Bool is throwing something off. To be clear, I'm not advocating for not returning Nullable{Bool}, I think we may just need to define our own == or something where the semantics of Nullable{Bool} vs. Bool are currently breaking things.

julia> a = NullableArrays.NullableArray([1])
1-element NullableArrays.NullableArray{Int64,1}:
 1

julia> b = NullableArrays.NullableArray([1])
1-element NullableArrays.NullableArray{Int64,1}:
 1

julia> a == b
ERROR: TypeError: non-boolean (Nullable{Bool}) used in boolean context
 in == at abstractarray.jl:1041

julia>

[nalimilan]

Good catch. Unfortunately, the only way to fix this everywhere would be to allow Nullable{Bool} in if (JuliaLang/julia#13207). Without it, we need to override ==. I've made a proposal in #84.

(See also #74)

[nalimilan]

There's also the question of what to do with comparisons between NullableArrays and other arrays:

julia> [1,2,3] == NullableArray([1,2,3])
false

This comes from the fact that Nullable(1) == 1 is false. If ==(::Nullable, x) returned a Nullable{Bool}, we would get an error, and require a special function just like the one I proposed above.

[nalimilan]

I've updated the PR with support for ==(::NullableArray, ::AbstractArray). This is the logical complement of #85.

[davidagold]

FWIW I've been using isequal for this sort of functionality:

julia> X = NullableArray([1])
1-element NullableArray{Int64,1}:
 1

julia> Y = NullableArray([1])
1-element NullableArray{Int64,1}:
 1

julia> isequal(X, Y)
true

[andyferris]

I just came across the == problem.

This comes from the fact that Nullable(1) == 1

At first glance that seems insane to me. Don't we have different things like == and isequal to avoid these issues?

[davidagold]

julia> using NullableArrays

julia> Nullable(1) == 1
false

What exactly are you referring to?

[andyferris]

Well, we have 1.0 == 1 as true and many many other examples. == is meant to be the "handwavey", and "its close enough in meaning" (not implementation) type of equality. For an example of different implementations, Diagonal([1,2,3]) == diagm([1,2,3]).

This is different to is/===, and isequal which are more about being implemented the same. My interpretation of Nullable is to formalize the idea of NaN to arbitrary types, just like NA did. But maybe I have the premise incorrect?

[davidagold]

I'm just confused because the quote you provided doesn't seem to be true. Are you saying it's insane that Nullable(1) == 1, in which case I would respond "that would be insane, but it's not true", or are you saying it's insane that it's not the case that Nullable(1) == 1, in which case I'd respond that I think that's actually appropriate.

[nalimilan]

I think he meant the latter. Cf #85.

[andyferris]

Yes, sorry for being unclear, but I did mean the latter. While I do think #85 is a good idea (and I do understand the difficulties and have read a lot about possible approaches, including the current Julia PR about higher order lifting), I feel regardless of all of that we should have == defined to follow the same semantics as NaN does for Float64 (same for isequal).

I suppose the precedent might be 1 == [1] is false, if you want to think of Nullable as a kind of container. Maybe I just have to change my mental model of what a Nullable is.

[davidagold]

How do the semantics of NaN inform the semantics of Nullable(1) == 1?

In any case, here's an argument for Nullable(1) !== 1. One might reasonably say that Nullable(1) has a value that is currently 1 but might have been missing, whereas 1 is just the value 1 and does not satisfy the same counterfactual. One might posit this as sufficient reason for distinguishing them with ==.

I can also see at least two counterarguments:

1. The difference in which counterfactuals objects satisfy shouldn't come to bear on whether or not they satisfy ==. Or, at least, this particular counterfactual shouldn't come to bear.
2. Actually, there is no difference between Nullable(1) and 1 in this regard. Since Nullable is immutable, Nullable(1, true) is a different object than Nullable(1, false), in much the same way 1 and NA or 1 and Nullable(1, false) are different objects. There's no difference between saying Nullable(1, true) could have been Nullable(1, false) and saying 1 could have been NA, except for the difference in type-inferability. But that is simply an implementation detail (albeit an important one) of how we've chosen to represent missingness.

I'm not unsympathetic to these counterarguments; I just think they're a slippery slope. If Nullable(1) == 1, and if I can do 1 + 1, then why can't I do Nullable(1) + 1? One could argue that == is special and, in principle, should have mixed signature methods defined. But it seems like the situations in which one relies on Nullable(1) == 1 are also situations where one wants to do Nullable(1) + 1, and I'm much less in favor of defining the latter.

On the whole, I think the issue of whether or not Nullable(1) == 1 should be discussed in Base. This issue is about the semantics of NullableArray equality. EDIT: Though I do see the relevance.