Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserve variables order in Dataset.concat() #1049

Merged
merged 3 commits into from
Nov 2, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion doc/whats-new.rst
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,16 @@ Bug fixes
``missing_value`` are set to ``NaN`` (:issue:`997`).
By `Marco Zühlke <https://github.com/mzuehlke>`_.

- ``.where()`` and ``.fillna()`` now preserve attributes(:issue:`1009`).
- ``.where()`` and ``.fillna()`` now preserve attributes (:issue:`1009`).
By `Fabien Maussion <https://github.com/fmaussion>`_.

- Applying :py:func:`broadcast()` to an xarray object based on the dask backend
won't accidentally convert the array from dask to numpy anymore (:issue:`978`).
By `Guido Imperiale <https://github.com/crusaderky>`_.

- ``Dataset.concat()`` now preserves variables order (:issue:`1027`).
By `Fabien Maussion <https://github.com/fmaussion>`_.

.. _whats-new.0.8.2:

v0.8.2 (18 August 2016)
Expand Down
12 changes: 6 additions & 6 deletions xarray/core/combine.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,6 @@ def differs(vname):
concat_over.update(process_subset_opt(coords, 'coords'))
if dim in datasets[0]:
concat_over.add(dim)

return concat_over


Expand Down Expand Up @@ -264,11 +263,12 @@ def ensure_common_dims(vars):
var = var.expand_dims(common_dims, common_shape)
yield var

# stack up each variable to fill-out the dataset
for k in concat_over:
vars = ensure_common_dims([ds.variables[k] for ds in datasets])
combined = concat_vars(vars, dim, positions)
insert_result_variable(k, combined)
# stack up each variable to fill-out the dataset (in order)
for k in datasets[0].variables:
if k in concat_over:
vars = ensure_common_dims([ds.variables[k] for ds in datasets])
combined = concat_vars(vars, dim, positions)
insert_result_variable(k, combined)

result = Dataset(result_vars, attrs=result_attrs)
result = result.set_coords(result_coord_names)
Expand Down
15 changes: 15 additions & 0 deletions xarray/test/test_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -1858,6 +1858,21 @@ def test_groupby_nan(self):
expected = Dataset({'foo': ('bar', [1.5, 3]), 'bar': [1, 2]})
self.assertDatasetIdentical(actual, expected)

def test_groupby_order(self):
# groupby should preserve variables order

ds = Dataset()
for vn in ['a', 'b', 'c']:
ds[vn] = DataArray(np.arange(10), dims=['t'])
all_vars_ref = list(ds.variables.keys())
data_vars_ref = list(ds.data_vars.keys())
ds = ds.groupby('t').mean()
all_vars = list(ds.variables.keys())
data_vars = list(ds.data_vars.keys())
self.assertEqual(data_vars, data_vars_ref)
# coords are now at the end of the list, so the test below fails
# self.assertEqual(all_vars, all_vars_ref)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think this test passes in this version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It still doesn't pass on my linux machine :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, concat does not currently preserve the order of all variables (yet). But at least we can make it non-stochastic.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could probably do this by consolidating the two loops over all variables. But up to you if you want to go through the trouble.


def test_resample_and_first(self):
times = pd.date_range('2000-01-01', freq='6H', periods=10)
ds = Dataset({'foo': (['time', 'x', 'y'], np.random.randn(10, 5, 3)),
Expand Down