ENH: Preserve attrs in to_dataframe()

[snowman2]

• Closes ENH: Preserve attrs when converting to pandas dataframe #5327
• Tests added
• Passes pre-commit run --all-files
• User visible changes (including notable bug fixes) are documented in whats-new.rst

[max-sixty]

Thanks @snowman2 !

I added one question.

[max-sixty]

Are these set per column or for the whole dataframe?

[snowman2]

Both actually. It is on the dataframe and the series.

[max-sixty]

Should we test for both?

[snowman2]

I have a test for both in xarray/tests/test_dataset.py. I don't believe it is applicable in xarray/tests/test_dataarray.py .

[max-sixty]

Forgive me if I'm confusing things.

How does pandas handle attrs? Here, it looks like it's the same on both the series and the dataframe — is that always the case? Or do we need to test both?

In [8]: df.attrs = dict(a=2)

In [9]: df
Out[9]:
     test
y x
1 2   0.0
  3   0.0
  4   0.0
  5   0.0
  6   0.0
2 2   0.0
  3   0.0
  4   0.0
  5   0.0
  6   0.0
3 2   0.0
  3   0.0
  4   0.0
  5   0.0
  6   0.0
4 2   0.0
  3   0.0
  4   0.0
  5   0.0
  6   0.0
5 2   0.0
  3   0.0
  4   0.0
  5   0.0
  6   0.0

In [10]: df.attrs
Out[10]: {'a': 2}

In [11]: df['test'].attrs
Out[11]: {'a': 2}

[snowman2]

Added more tests to ensure the expected behavior occurs.

[snowman2]

Here, it looks like it's the same on both the series and the dataframe — is that always the case?

I hadn't checked that behavior. It is different here. 🤷‍♂️

[dcherian]

Thanks @snowman2

If you're interested, see #3497 for the inverse problem of using pandas attrs when constructing Xarray objects (in a future PR) :)

[max-sixty]

Was there any reason this stalled? It looked like a good start!

[keewis]

I believe the issue is that pandas.DataFrame does not support column attrs (or did not? I didn't check whether that changed since then). DataFrame-level attrs should work, though.

[dcherian]

They were thinking of removing it at one point: pandas-dev/pandas#52166, also dask/dask#11146

perhaps we should punt until someone really really wants it?

[max-sixty]

Yes, looks like the conclusion from the pandas issue is they want to keep it but the support is spotty.

Probably we close this unless someone comes to save it, but I would vote to merge a PR that did this — I can't see a downside...

[giovp]

hi, I just saw that this discussion has been picking up. I work on the framework mentioned in this comment pandas-dev/pandas#52166 (comment) and we would be very happy if the dataframe-level attrs would be added back to dask-dataframes. We don't use the column-level attrs but do use the df-level attrs. Currently, the solution we implement to use the latest dask is that we ask users to change the configs like so

dask.config.set({'dataframe.query-planning': False})

Would this PR preserve the attrs also in the dask-expr backend?