deduplicate timestamps with check #96
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is failing the tests as xarray compatible with Python 3.7 doesn't have |
|
It appears to work on Python 3.8 and more recent versions. We could skip the call to drop_duplicates if the function does not exist and raise a warning that it's been skipped? |
|
I think we could pin to 3.8. That's almost 3 years old. |
|
I added a check for drop_duplicates that works around the problem. Or would you prefer to drop 3.8 now? |
jklymak
reviewed
Jun 20, 2022
pyglider/seaexplorer.py
Outdated
| raise ValueError(f"{loss_str} Check input data for duplicate timestamps") | ||
| elif proportion_kept < 0.999: | ||
| _log.warning(loss_str) | ||
| if "drop_duplicates" in ds.__dir__(): |
There was a problem hiding this comment.
I think:
Suggested change
| if "drop_duplicates" in ds.__dir__(): | |
| if hasattr(ds, "drop_duplicates"): |
pyglider/seaexplorer.py
Outdated
| else: | ||
| _log.info(loss_str) |
There was a problem hiding this comment.
Can we just write our own drop_duplicates? Its basically good = (ds.time.diff() > 1e-6) ds = ds.isel(time=good), plus or minus one on the index, and maybe keeping the first index.
jklymak
approved these changes
Jun 22, 2022
|
Feel free to self-merge if you are done.... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This resolves #95
Varying response depending on how many samples are removed by this deduplication:
more than half lost - error
more than 0.1 % lost - warning
otherwise log
order by time and drop_duplicates moved before
oxygen_concentration_correctionso the reindex in this function does not breeak.