panel.to_frame() discards nan by default

[hensing]

As already described in issue #3690, panel.to_frame discards all minor entries with nan in the data, which can be very confusing. I believe there should be a warning, first time data is dropped or the opposite should be the default behavior.

The warning could be treated similar to a ZeroDivisionWarning in numpy only on the first occurrence.

See below for an example:

df1 = pd.DataFrame(np.random.randn(2, 3), columns=['A', 'B', 'C'],
                   index=['foo', 'bar'])
df2 = pd.DataFrame(np.random.randn(2, 3), columns=['A', 'B', 'C'],
                   index=['foo', 'bar'])
df2.loc['foo', 'B'] = np.nan
mydict = {'df1': df1, 'df2': df2}

pd.Panel(mydict).to_frame()

Output:

major	minor	df1	df2
foo	A	1.9097545931480682	-0.6710202447941566
foo	C	1.3335254610685865	1.53372538551507
bar	A	0.3145550744497975	-1.7221352144306152
bar	B	-0.15681197178861878	-1.2308510354641322
bar	C	-0.09598971674309852	-0.1268630728124487

Using filter_observations=False, nan won't be dropped:

pd.Panel(mydict).to_frame(filter_observations=False)

Output:

major	minor	df1	df2
foo	A	1.9097545931480682	-0.6710202447941566
foo	B	2.092552358833253
foo	C	1.333525461068586	1.53372538551507
bar	A	0.3145550744497975	-1.7221352144306152
bar	B	-0.15681197178861878	-1.2308510354641322
bar	C	-0.09598971674309852	-0.1268630728124487

[TomAugspurger]

+1 on a warning. This surprises me every time.

[m-novikov]

@jreback
I can fix broken tests, it's for most part causing problems becouse result of to_frame() changed length with new API.
But I'm not sure what to do with SparsePanel.to_frame() method which will raise exeption if filter_observations changed to False.
Else if I left along SpasePanel.to_frame method signatures of Panel.to_frame(filter_observations=False) and SparsePanel.to_frame(filter_observations=True) will be inconsistent.

[jreback]

can you show the error with SparsePanel. That's sort of a problem step-child. Prob noone uses it (and I never fixed it to inherit from NDFrame).

[springcoil]

Sorry, @jreback how exactly do I do that? Do you want me to change some
errors?

On Mon, Nov 3, 2014 at 12:16 AM, jreback notifications@github.com wrote:

can you show the error with SparsePanel. That's sort of a problem
step-child. Prob noone uses it (and I never fixed it to inherit from
NDFrame).

—
Reply to this email directly or view it on GitHub
#7879 (comment).

Peadar Coyle
Skype: springcoilarch
www.twitter.com/springcoil
peadarcoyle.wordpress.com

[jreback]

just show a complete example and the error it outputs (when you change the default)

[eyurtsev]

This just got me. Took a long time to figure out why data was missing. 👍 for any solution. :)

[eyurtsev]

This got me yet one more time! 1 hour of troubleshooting. :)

[jreback]

pull-requests welcome to change this for 0.17.0

[hensing]

Is PR #8063 a solution or is it worth to write a new one or to change the default behavior?

[jreback]

@hensing so options are this

• just change the default and document in the API section. This doesn't lose information, but could be unexpected for users. Though in this case ok with it.
• deprecate filter_observations and replace with dropna (which is more consistent with the kw schema). Then you can change to the new default and show a deprecation warning for the originals.

[hensing]

@jreback solution № 2 (deprecate und change kw) seems for me to be the most elegant solution — I'll give it a try next weekend.

[jreback]

gr8!

[jreback]

@hensing status?

[hensing]

@jreback I couldn't work on that due to bike accident, sorry. Hope I can make it this weekend.

[saddy001]

I am a bit confused: I'm using Pandas 19.2 and still have the default drop-behaviour with the filter_observations-Keyword.

pip3 show pandas
Version: 0.19.2
$ python3
import pandas
pandas.__version__
'0.19.2'

When I look into the source I see no changes regarding dropna etc:
panel.py line 879

What am I missing?

[jreback]

this is an open issue

iirc there a couple of pull requests that were almost there but did not get merged

welcome to have a complete one

[jreback]

closing as Panel deprecated