Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` #3877

johnomotani · 2020-03-23T01:32:59Z

combine_attrs argument for merge(), concat(), combine_by_coords() and combine_nested() controls what attributes the result is given. Defaults maintain the current behaviour. Possible values (named following compat arguments) are:

'drop': empty attrs on returned Dataset.
'identical': all attrs must be the same on every object.
'no_conflicts': attrs from all objects are combined, any that have the same name must also have the same value.
'override': skip comparing and copy attrs from the first dataset to the result.

Closes merge drops attributes #3865
Tests added
Passes isort -rc . && black . && mypy . && flake8
Fully documented, including whats-new.rst for all changes and api.rst for new API

Adds option 'promote_attrs' to DataArray.to_dataset(). By default promote_attrs=False, maintaining current behaviour. If promote_attrs=True, the attrs of the DataArray are shallow-copied to the Dataset returned by to_dataset().

If the values of any shared key are not equivalent, then raises an error.

Provides several options for how to combine the attributes of the passed objects and give them to the returned Dataset.

Provides several options for how to combine the attributes of the passed objects and give them to the returned DataArray or Dataset.

Provides several options for how to combine the attributes of the passed objects and give them to the returned Dataset.

max-sixty

This looks excellent! Thanks @johnomotani !

Any thoughts from anyone re the kwarg name combine_coords or its options? They seem reasonable to me.

xarray/core/dataarray.py

xarray/core/utils.py

xarray/core/concat.py

xarray/core/combine.py

xarray/core/merge.py

xarray/core/combine.py

xarray/core/concat.py

Apply suggestions from code review Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>

No need for these arguments to be MutableMapping rather than just Mapping.

max-sixty · 2020-03-23T17:20:58Z

Any other thoughts from anyone before we hit the big green button?

Do not use OrderedDicts any more, so name did not make sense.

TomNicholas · 2020-03-23T19:12:19Z

Oh sweet!

Should there perhaps be another option to specify which object to get the attrs from? I'm just thinking by analogy to how open_mfdataset now lets you specify which file you want the attrs from.

johnomotani · 2020-03-23T19:23:47Z

Should there perhaps be another option to specify which object to get the attrs from? I'm just thinking by analogy to how open_mfdataset now lets you specify which file you want the attrs from.

That would actually be nice to have in concat() for a use-case I have. It's not immediately obvious to me how to implement it though. For merge() or concat() you could give an integer index. I think open_mfdataset used the file-name (?), but there's no equivalent for combine_by_coords or combine_nested is there?

johnomotani · 2020-03-24T09:42:11Z

For specifying which object, one possibility would be to pass an int to combine_attrs in merge(), concat() or combine_by_coords(), or a tuple of int to combine_nested, giving the index of the object to use attributes from. This feature would need new tests writing though, so I'd suggest implementing it in a new PR.

TomNicholas · 2020-03-24T11:36:10Z

For specifying which object, one possibility would be to pass an int

I'm not in favour of an integer argument, for the reasons discussed for open_mfdataset here. We maybe just pass the actual dataset/array object though...

This feature would need new tests writing though, so I'd suggest implementing it in a new PR.

Fine to discuss this on a separate PR though.

max-sixty · 2020-03-24T15:11:28Z

doc/whats-new.rst

+- Control over attributes of result in :py:func:`merge`, :py:func:`concat`,
+  :py:func:`combine_by_coords` and :py:func:`combine_nested` using
+  combine_attrs keyword argument. (:issue:`3865`, :pull:`3877`)
+  By `John Omotani <https://github.com/johnomotani>`_


Could we merge master and move this to 0.16.0?

max-sixty · 2020-03-24T15:12:36Z

Let's merge later today unless there are other comments?

max-sixty · 2020-03-24T20:40:07Z

Merging — any other feedback please post and we can follow-up on.

Thanks a lot @johnomotani , great PR!

* upstream/master: (54 commits) Limit repr of arrays containing long strings (pydata#3900) expose a few zarr backend functions as semi-public api (pydata#3897) Use drawstyle instead of linestyle in plot.step. (pydata#3274) Implementation of polyfit and polyval (pydata#3733) misplaced quote in whatsnew (pydata#3889) Rename ordered_dict_intersection -> compat_dict_intersection (pydata#3887) Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` (pydata#3877) xfail test_uamiv_format_write (pydata#3885) Use `fixes` in PR template (pydata#3886) Tweaks to "how_to_release" (pydata#3882) whatsnew section for 0.16.0 Release v0.15.1 whatsnew for 0.15.1 (pydata#3879) update panel documentation (pydata#3880) reword the whats-new entry for unit support (pydata#3878) Raise error when assigning to IndexVariable.values & IndexVariable.data (pydata#3862) Re-enable tests xfailed in pydata#3808 and fix new CFTimeIndex failures due to upstream changes (pydata#3874) add spacing in the versions section of the issue report (pydata#3876) map_blocks: allow user function to add new unindexed dimension. (pydata#3817) Delete associated indexes when deleting coordinate variables. (pydata#3840) ...

* upstream/master: (75 commits) Implement idxmax and idxmin functions (pydata#3871) Update pre-commit-config.yaml (pydata#3911) Revert "Use `fixes` in PR template (pydata#3886)" (pydata#3912) update the docstring of diff (pydata#3909) Un-xfail test_dayofyear_after_cftime_range (pydata#3907) Limit repr of arrays containing long strings (pydata#3900) expose a few zarr backend functions as semi-public api (pydata#3897) Use drawstyle instead of linestyle in plot.step. (pydata#3274) Implementation of polyfit and polyval (pydata#3733) misplaced quote in whatsnew (pydata#3889) Rename ordered_dict_intersection -> compat_dict_intersection (pydata#3887) Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` (pydata#3877) xfail test_uamiv_format_write (pydata#3885) Use `fixes` in PR template (pydata#3886) Tweaks to "how_to_release" (pydata#3882) whatsnew section for 0.16.0 Release v0.15.1 whatsnew for 0.15.1 (pydata#3879) update panel documentation (pydata#3880) reword the whats-new entry for unit support (pydata#3878) ...

johnomotani added 6 commits March 23, 2020 01:27

utils.ordered_dict_union returns the union of two compatible dicts

cae88df

If the values of any shared key are not equivalent, then raises an error.

combine_attrs argument for merge()

5b57976

Provides several options for how to combine the attributes of the passed objects and give them to the returned Dataset.

combine_attrs argument for concat()

4fdab0b

Provides several options for how to combine the attributes of the passed objects and give them to the returned DataArray or Dataset.

combine_attrs argument for combine_by_coords() and combine_nested()

ea73a0e

Provides several options for how to combine the attributes of the passed objects and give them to the returned Dataset.

Add combine_attrs changes to whats-new.rst

190c8ee

max-sixty approved these changes Mar 23, 2020

View reviewed changes

xarray/core/dataarray.py Outdated Show resolved Hide resolved

xarray/core/utils.py Outdated Show resolved Hide resolved

xarray/core/concat.py Outdated Show resolved Hide resolved

johnomotani commented Mar 23, 2020

View reviewed changes

xarray/core/combine.py Outdated Show resolved Hide resolved

xarray/core/merge.py Outdated Show resolved Hide resolved

xarray/core/combine.py Outdated Show resolved Hide resolved

xarray/core/concat.py Outdated Show resolved Hide resolved

johnomotani and others added 2 commits March 23, 2020 09:48

Update docstrings to note default values

41183a0

Apply suggestions from code review Co-Authored-By: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>

First argument of update_safety_check and ordered_dict_union not mutable

23df58e

No need for these arguments to be MutableMapping rather than just Mapping.

johnomotani force-pushed the merge-attrs branch from 854e922 to 5249814 Compare March 23, 2020 17:37

Rename ordered_dict_union -> compat_dict_union

9cc68a9

Do not use OrderedDicts any more, so name did not make sense.

johnomotani force-pushed the merge-attrs branch from 5249814 to 9cc68a9 Compare March 23, 2020 17:57

johnomotani mentioned this pull request Mar 23, 2020

Parallel interpolation boutproject/xBOUT#108

Merged

max-sixty reviewed Mar 24, 2020

View reviewed changes

johnomotani added 3 commits March 24, 2020 15:19

Move combine_attrs to v0.16.0 in whats-new.rst

8a2535c

Merge branch 'master' into merge-attrs

5ec7dc3

Fix merge of whats-new.rst

e415815

max-sixty merged commit d8bb620 into pydata:master Mar 24, 2020

johnomotani mentioned this pull request Mar 24, 2020

Rename ordered_dict_intersection -> compat_dict_intersection #3887

Merged

3 tasks

TomNicholas mentioned this pull request Apr 5, 2020

save "encoding" when using open_mfdataset #2436

Open

TomNicholas added the topic-metadata Relating to the handling of metadata (i.e. attrs and encoding) label Apr 5, 2020

rabernat mentioned this pull request Apr 14, 2020

fix py36 xarray master build MITgcm/xmitgcm#200

Merged

aulemahal mentioned this pull request Apr 27, 2020

Incoherencies between docs in open_mfdataset and combine_by_coords and its behaviour. #4009

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` #3877

Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` #3877

johnomotani commented Mar 23, 2020

max-sixty left a comment

max-sixty commented Mar 23, 2020

TomNicholas commented Mar 23, 2020

johnomotani commented Mar 23, 2020

johnomotani commented Mar 24, 2020

TomNicholas commented Mar 24, 2020 •

edited

Loading

max-sixty Mar 24, 2020

max-sixty commented Mar 24, 2020 •

edited

Loading

max-sixty commented Mar 24, 2020

Control attrs of result in merge(), concat(), combine_by_coords() and combine_nested() #3877

Control attrs of result in merge(), concat(), combine_by_coords() and combine_nested() #3877

Conversation

johnomotani commented Mar 23, 2020

max-sixty left a comment

Choose a reason for hiding this comment

max-sixty commented Mar 23, 2020

TomNicholas commented Mar 23, 2020

johnomotani commented Mar 23, 2020

johnomotani commented Mar 24, 2020

TomNicholas commented Mar 24, 2020 • edited Loading

max-sixty Mar 24, 2020

Choose a reason for hiding this comment

max-sixty commented Mar 24, 2020 • edited Loading

max-sixty commented Mar 24, 2020

Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` #3877

Control attrs of result in `merge()`, `concat()`, `combine_by_coords()` and `combine_nested()` #3877

TomNicholas commented Mar 24, 2020 •

edited

Loading

max-sixty commented Mar 24, 2020 •

edited

Loading