Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xarray.open_datatree is taking too long to open datatree in a s3 bucket #9376

Closed
aladinor opened this issue Aug 17, 2024 · 1 comment · Fixed by #9377
Closed

xarray.open_datatree is taking too long to open datatree in a s3 bucket #9376

aladinor opened this issue Aug 17, 2024 · 1 comment · Fixed by #9377
Labels
topic-DataTree Related to the implementation of a DataTree class topic-performance topic-zarr Related to zarr storage library

Comments

@aladinor
Copy link
Contributor

What is your issue?

Hi all,

I was trying to open a datatree stored in a s3 bucket but it is taking too long.

from xarray.backends.api import open_datatree

URL = 'https://js2.jetstream-cloud.org:8001/'
path = f'pythia/radar/erad2024'
fs = s3fs.S3FileSystem(anon=True, client_kwargs=dict(endpoint_url=URL))
file = s3fs.S3Map(f"{path}/zarr_radar/erad_2024.zarr", s3=fs)
dt = open_datatree(file, engine='zarr', consolidated=True)

When digging around, I discovered some parameters/arguments such as mode, consolidated, ..., were not being passed to ZarrStore.open_store function here.

stores = ZarrStore.open_store(filename_or_obj, group=parent)

@aladinor aladinor added the needs triage Issue that has not been reviewed by xarray team member label Aug 17, 2024
Copy link

welcome bot commented Aug 17, 2024

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

@TomNicholas TomNicholas added topic-performance topic-DataTree Related to the implementation of a DataTree class topic-zarr Related to zarr storage library and removed needs triage Issue that has not been reviewed by xarray team member labels Aug 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-DataTree Related to the implementation of a DataTree class topic-performance topic-zarr Related to zarr storage library
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants