BUG: 0.12.0 using DataFrame.to_hdf() with mode='a' does not append the data

[bluefir]

to_hdf(....., append=True,table=True) raising, should accept this

Is there a way to use DataFrame.to_hdf() so that data can be appened the data? Somehow to_hdf(filename, table_name, mode='a', table=True) overwrites the data while to_hdf(filename, table_name, table=True, append=True) gives the following error:

Traceback (most recent call last):
File "M:/Projects/PortfolioAnalytics/Code/Python/attribution/TopContributors.py", line 1035, in
portfolio_data.to_hdf(filename, OUTPUT_NAME_PORTFOLIO, table=True, append=Settings.TC_OUTPUT_APPEND)
File "C:\Python27\lib\site-packages\pandas\core\generic.py", line 521, in to_hdf
return pytables.to_hdf(path_or_buf, key, self, *_kwargs)
File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 194, in to_hdf
f(store)
File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 188, in
f = lambda store: store.append(key, value, *_kwargs)
File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 658, in append
self._write_to_group(key, value, table=True, append=True, **kwargs)
TypeError: _write_to_group() got multiple values for keyword argument 'table'

[jreback]

table=True implies append (so don't need append keyword (its really a relic anyhow)), and this is a PER GROUP option

mode has to do with the file, e.g. passing mode=w overwrites the file, normally you should ALWAYS use the default mode=a (as you can read/write with it)

[bluefir]

Well, using

df1.to_hdf(filename, table_name, mode='w', table=True)

and then

df2.to_hdf(filename, table_name, mode='a', table=True)

overwrites df1 data in table_name with df2 data.

pd.read_hdf(filename, table_name)

after df1

<class 'pandas.core.frame.DataFrame'>
MultiIndex: 108879 entries, (2013-08-01 00:00:00, 000312) to (2013-08-09 00:00:00, Y8565N10)
Data columns (total 12 columns):
market_cap 108830 non-null values
alpha 108676 non-null values
gics_code 108830 non-null values
investable 108879 non-null values
issuer_country 108701 non-null values
msci_country 77291 non-null values
universe 108879 non-null values
weight_benchmark 108834 non-null values
weight_portfolio 108834 non-null values
weight_active 108834 non-null values
country 108834 non-null values
return_daily 108649 non-null values
dtypes: bool(2), float64(7), object(3)

after df2

<class 'pandas.core.frame.DataFrame'>
MultiIndex: 62432 entries, (2013-08-12 00:00:00, 000312) to (2013-08-15 00:00:00, Y8565N10)
Data columns (total 12 columns):
market_cap 62432 non-null values
alpha 62344 non-null values
gics_code 62432 non-null values
investable 62432 non-null values
issuer_country 62356 non-null values
msci_country 44268 non-null values
universe 62432 non-null values
weight_benchmark 62432 non-null values
weight_portfolio 62432 non-null values
weight_active 62432 non-null values
country 62432 non-null values
return_daily 62281 non-null values
dtypes: bool(2), float64(7), object(3)

Am I missing something?

[jreback]

try to 2nd one with append=True and leave table=True out. append=True implies table, so you only need one of these.

I think append=True,table=True is a bug (in that it should work and not cause an error)

FYI....prob going to deprecate the table options and instead

format='storer | `table(default will bestorer``)

and append option will still be there

[bluefir]

Thanks!

[jreback]

oh....this is a 'bug'...going to fix..thanks for the report