Ignore nans masking #282
Ignore nans masking #282
Conversation
|
I think @sebhahn removed some functions from pygeobase recently, which are not found anymore by pytesmo... read_ts was deprecated anyway... so maybe we can just make it optional in pytesmo or remove it completely. However, I think read_ts is still in pynetcf (which is between pygeobase and ascat/pytesmo). |
|
That woul
That would work..I will just fallback for now and open an issue |
|
@s-scherrer could you please review and merge if you think it is ok? new tests are included but for some reason coverall is not happy |
|
|
||
| if mask is not None: | ||
| mask = mask & new_mask | ||
| else: | ||
| mask = new_mask | ||
|
|
||
| return data[mask] | ||
| return data[mask].dropna(how="all") |
There was a problem hiding this comment.
What's the reason for adding the dropna here?
There was a problem hiding this comment.
With the ignore_nans option active, the dataframe can be returned with all-nans rows. I think it is quite specific to the SMOSL2 data, where there are a LOT of NaNs in the flagging fields, and a lot of observation with no soil moisture (NaN) and some type of flag value.
In the latter case it can happen that the NaN soil moisture is kept. dropna just eliminates these occurrences
There was a problem hiding this comment.
Sounds okay to me. I will merge it in then. Do you need a new release?
There was a problem hiding this comment.
I guess it would be best, but if you prefer to wait it should work too.
Method for masking to ignore NaNs in filter columns, i.e. observations can be kept if no information is given by the filtering field.