util.testing.assert_almost_equal() gives Key Error

[dickster77]

Test 1 works successfully with a square dataframe

import numpy as np
import pandas as pd

np.random.seed(1)
a = pd.DataFrame(np.random.rand(5, 5))
np.random.seed(1)
b = pd.DataFrame(np.random.rand(5, 5))
b += 0.00001
pd.util.testing.assert_almost_equal(a,b)

Test 2 fails with a dataframe where number rows > number of columns

np.random.seed(1)
a = pd.DataFrame(np.random.rand(6, 5))
np.random.seed(1)
b = pd.DataFrame(np.random.rand(6, 5))
b += 0.00001
pd.util.testing.assert_almost_equal(a,b)

KeyError: 5L

[sinhrks]

You should use pd.util.testing.assert_frame_equal(a,b) for DataFrame comparison.

Better to add docstring of assert_almost_equal to describe it.

[dickster77]

Sorry sinhrks - I meant to highlight the fact there are small differences in the dataframes. Hence i specifically need assert_almost_equal()

[sinhrks]

Did you updated the example? Use pd.util.testing.assert_almost_equal(a.values, b.values).

Strangely assert_frame_equal(a, b, check_less_precise=True) should work for your data by definition, it raises.

[dickster77]

Yes see update in the example b += 0.00001

When I use the .values property to return an ndarray the test assert_almost_equal() works for me.

However I would want to do a dataframe test. One that ensure indexes etc. match. Yet it allows for the small variation in the values.

[sinhrks]

Currently it should be possible by check_less_precise option, but it seems not applied to all cases.

More flexible comparison tolerance is being discussed in #10788.

[dickster77]

Thanks - when should I use .assert_almost_equal() with respect to dataframes?

[sinhrks]

Let me summarize:

• assert_almost_equal shoun't be used for DataFrame
• assert_frame_equal can compare index and columns. It should be always used for DataFrame comparison.
• assert_frame_equal(check_less_precise=True) should compare 3 digits after decimal points (similar to assert_almost_equal). (Your case should be covered by the option, but it looks not work properly because of a bug)

As a workaround until the bug is fixed, you can use assert_almost_equal(a.values, b.values) to compare DataFrame values (ndarray). To compare index and columns, use assert_index_equal each.

[jreback]

see #9457 for the enhancement issue to add equiv of np.allclose to this.

@sinhrks let's rerpose this issue to enable assert_almost_equal to raise a nicer error message (and suggest using one of the assert_*pandasobject*_equal methods

I suppose we could just call one of these methods directly, no?