Inconsistency in push!ing an empty row into a DataFrame

[ericphanson]

julia> df = DataFrame()
0×0 DataFrame

julia> push!(df, NamedTuple(); cols=:union) # does not add a row
0×0 DataFrame

julia> df = DataFrame(:a => [1]) # try again with non-empty df
1×1 DataFrame
 Row │ a     
     │ Int64 
─────┼───────
   1 │     1

julia> push!(df, NamedTuple(); cols=:union) # adds a row
2×1 DataFrame
 Row │ a       
     │ Int64?  
─────┼─────────
   1 │       1
   2 │ missing 

This edge case came up for me when doing

new_df = DataFrame()
for value in old_df.col
    push!(new_df, nt_with_variable_number_cols(value); cols=:union)
end
df = hcat(old_df, new_df; copycols=false)

to try to work around the fact that

transform!(df, :col => ByRow(nt_with_variable_number_cols) => AsTable)

does not work (says you must emit identical columns).

[bkamins]

I can see the problem. The question is what to do about it. If the data frame has no columns then pushing NamedTuple() cannot add a row to it.

On the other hand, if a data frame already has some data and you push NamedTuple() to it with cols=:union I think that it is natural to expect that a row full of missing values will be created.

In general Tables.dictrowtable is a function designed to perform such unioning as you want:

julia> Tables.dictrowtable([NamedTuple(), (a=1,), NamedTuple(), (b=2,), NamedTuple()]) |> DataFrame
5×2 DataFrame
 Row │ a        b
     │ Int64?   Int64?
─────┼──────────────────
   1 │ missing  missing
   2 │       1  missing
   3 │ missing  missing
   4 │ missing        2
   5 │ missing  missing

[bkamins]

I think it should be closed.

@ericphanson - if you think otherwise - please comment.