Adding a warning when dropping NA values for panel.to_frame #7879

[Magellanea]

closes: #7879

I've added a warning when the NaN values are dropped when calling pd.Panel(dict).to_frame() with the param filter_observations true

I've supplied a test case to be used against the example provided in the issue tracker

[TomAugspurger]

You'll need to import warnings in this file too

[TomAugspurger]

Thanks for the PR.

I added a couple comments. Let me know if they don't make sense or you need any help.

[jreback]

their is another pr for this issue

however I think in that one the consensus was to make an API change to make filter_observations to False

instead of doing a warning which I think is a bit odd
as we don't warn on anything else like this

[Magellanea]

@TomAugspurger, I've modified the test as following

def test_to_frame_na_drop_warnings(self):
        df1 = DataFrame(np.random.randn(2, 3), columns=['A', 'B', 'C'],
               index=['foo', 'bar'])
        df2 = DataFrame(np.random.randn(2, 3), columns=['A', 'B', 'C'],
               index=['foo', 'bar'])
        df2.loc['foo', 'B'] = np.nan
        dict_without_dropped_vals = {'df1': df1, 'df2': df2}
        ## A panel without dropped vals shouldn't throw warnings
        with tm.assert_produces_warning(False):
            Panel(dict_without_dropped_vals).to_frame()
        ## A panel with dropped vals should throw a Runtime warning if \
        # filter_observations is True
        df2_with_na_vals = DataFrame(df2)
        df2_with_na_vals.loc['foo', 'B'] = np.nan
        dict_with_dropped_vals = {'df1': df1, 'df2_dropped': df2_with_na_vals}
        with tm.assert_produces_warning(RuntimeWarning):
            Panel(dict_with_dropped_vals).to_frame()
        ##if filter_observations is False, a warning shouldn't be throws
        with tm.assert_produces_warning(False):
            Panel(dict_with_dropped_vals).to_frame(filter_observations=False)

Does this match what you expected?

Thanks for the reply

[Magellanea]

@jreback So you suggest I stop working on that PR as pandas doesn't use warnings by default, Right?
Concerning the other PR, was it accepted, I didn't find any PRs or comments referencing it?
Thanks

[jreback]

see #7970. Its not been accepted as of yet.

pandas does use warnings. Its for this usecase.

I think the API change is better. Really just looking for some feedback (and then impl of course).

[jreback]

@Magellanea pls review that PR and post comments (here is ok).

[Magellanea]

@jreback Concerning the API change - if I'm we're going to neglect the warnings - do you suggest that we evaluate filter_observation to be false regardless of the passed value, Is that what you mean by the API CHANGE, Thanks

[jreback]

@Magellanea no I meant change the default of filter_observations=True to False. I think this would be fine as an API change with an example in v0.15.0.txt. You will have a couple of test fialures I think which will need addressing.

[jreback]

@Magellanea can you rebase / address comments?

[Magellanea]

@jreback I'm sorry for leaving this for a while, the problem was that the travis built failed, and I don't know what the reason was, I've changed the filter_observation default value and in the corresponding test but the built failed for other reasons and irrelevant tests

[jreback]

dupe of #7970