Skip to content
This repository has been archived by the owner on Dec 10, 2018. It is now read-only.

N-D structure to hold the datasets to be concatenated #1

Closed
TomNicholas opened this issue Sep 2, 2018 · 2 comments
Closed

N-D structure to hold the datasets to be concatenated #1

TomNicholas opened this issue Sep 2, 2018 · 2 comments
Assignees

Comments

@TomNicholas
Copy link
Owner

Unsure of what the best structure to hold the datasets to be concatenated is.

I tried a numpy.ndarray but that did not play well with xarray, as discussed at pydata/xarray#2159.

Might have to resort to using a list of lists?
Whatever structure I use should be something that is easy for a user to create if necessary.

@TomNicholas TomNicholas self-assigned this Sep 2, 2018
@TomNicholas
Copy link
Owner Author

This stackoverflow answer seems to say that numpy arrays of dtype='object' will behave badly when trying to store anything which has a __len__ attribute.

Unfortunately the method proposed on the xarray issue above seems way more complicated than necessary for my purposes.

One (hacky) solution might be to use numpy to store an object whose only purpose is to contain a single xarray.Dataset but without exposing a __len__ or __getitem__ attribute, something like

class box:
    def __init__(self, thing):
        self.thing = thing

    def item:
        return self.thing

On the other hand if the only numpy functionality I'm using is np.apply_along_axis and np.ndenumerate then it might be easier to just use a list of lists, and recursively call a function (like this one) which concatenates the objects residing in the deepest lists.

@TomNicholas
Copy link
Owner Author

I went for the "hacky" solution, but simplified by using a dictionary as the container class instead of defining a class just for this purpose, i.e.

split_data = [{'key': ds1}, {'key': ds2}]
ds = _concat_nd(split_data, dims=['dim1'])

This works, and the code can be seen at commit 303bfc8.

Unless someone from the xarray community can suggest a much easier way of doing this then I will close this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant