Add support for byte order (endianness) #132
Merged
+95
−11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
netCDF4 variables are written with a specific byte order (endianness), typically the same as the machine writing to the dataset. In netCDF4-python this shows up in the variable's dtype.byteorder, and may be '<' (little), '>' (big) or '=' (native). For the local storage backend this generally "just works", since the numpy array returned by netCDF4-python has the correct byte ordering. For files in an S3 store, kerchunk uses h5py to read the netCDF4 dataset when converting it to a Zarr array index. In this case the byte order of the dtype on the Zarr array is not always correct, and sometimes shows up as big endian even though little endian was specified. We work around this by capturing the dtype of the dataset when reading other attributes (filters, metadata, etc.). Closes #76
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## s3-missing-data #132 +/- ##
================================================
Coverage 86.52% 86.53%
================================================
Files 8 8
Lines 579 594 +15
================================================
+ Hits 501 514 +13
- Misses 78 80 +2
☔ View full report in Codecov by Sentry. |
valeriupredoi
approved these changes
Aug 11, 2023
|
#130 was merged so am merging this too 👍 |
This was referenced Aug 15, 2023
markgoddard
added a commit
that referenced
this pull request
Aug 15, 2023
Since byte order support was added in #132, we capture the dtype from netCDF metadata and use it in preference to the Zarr dtype when using the S3 storage backend. In the case where missing data values are passed into the Active constructor, we do not load netCDF metadata, and therefore do not capture the dtype. This leads to the following error with S3 storage: AttributeError: 'Active' object has no attribute '_dtype' This change switches to always load the metadata so that we can capture the dtype. This has the downside that it is not necessary when using local storage and missing data has been specified. This allows us to enable the test_daily_data_masked test for S3. Closes #137
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
netCDF4 variables are written with a specific byte order (endianness),
typically the same as the machine writing to the dataset. In
netCDF4-python this shows up in the variable's dtype.byteorder, and may
be '<' (little), '>' (big) or '=' (native).
For the local storage backend this generally "just works", since the
numpy array returned by netCDF4-python has the correct byte ordering.
For files in an S3 store, kerchunk uses h5py to read the netCDF4 dataset
when converting it to a Zarr array index. In this case the byte order of
the dtype on the Zarr array is not always correct, and sometimes shows
up as big endian even though little endian was specified. We work around
this by capturing the dtype of the dataset when reading other attributes
(filters, metadata, etc.).
Closes #76