Handle filling aca image array differently #297
Conversation
mica/archive/aca_l0.py
Outdated
| for i, row in enumerate(dat): | ||
| imgraw = imgraws[i].reshape(8, 8) | ||
| sz = imgsizes[i] | ||
| if sz < 8: | ||
| imgraw = imgraw[:sz, :sz] | ||
|
|
||
| meta = {name: col[i] for name, col in cols.items() if col[i] != -9999} | ||
| meta = {name: col[i] for name, col in cols.items() if col[i] != fill_vals[name]} |
There was a problem hiding this comment.
imgraw.fill_value is not -9999, but you filled it with -9999, so the condition if col[i] != fill_vals[name] will never be true for IMGRAW. Will that be a problem? Or IMGRAW is an exception?
It might be better to do
dat["IMGRAW"].fill_value = -9999
imgraws = dat["IMGRAW"].filled()
There was a problem hiding this comment.
I suppose the ambiguity is that I only made this work for the the default columns (IMGRAW isn't in the default columns) and it isn't expected that one would put the 8x8 imgraw data in the metadata for the image. I should probably just remove imgraw from columns if it is in there.
There was a problem hiding this comment.
And your idea to update the fill value for IMGRAW is also a good one. Technically I think I'd need to do that before making that little dictionary I have from column names to fill values to be 100% consistent (but the idea would still be that we don't have to worry about imgraw in the "add meta data to the ACAImage section").
There was a problem hiding this comment.
I think you don't need to do any filling at all except for imgraws. For the meta values for each row do:
names = dat.dtype.names
for i, row in ...:
meta = {name: row[name] for name in names if not dat.mask[name][i]}
Technically speaking if dat["IMGSIZE"] is masked then that entire row is junk and should just be skipped. Using a filled value for that column would lead to undefined behavior.
There was a problem hiding this comment.
And probably the whole thing can be nicer as:
# Only keep rows with non-masked IMGSIZE
dat = dat[~dat["IMGSIZE"].mask]
for row, imgraw in zip(dat, imgraws):
imgraw = imgraw.reshape(8, 8)
sz = row["IMGSIZE"]
if ..:
...
meta = {name: row[name] for name in names if not np.any(row.mask[name])}
...
There was a problem hiding this comment.
Maybe we need to go and find missing data to see which of these filters was needed in general. IIRC IMGSIZE is a calculated quantity from another part of the code, so I didn't see a way that it could have a masked value, for example
There was a problem hiding this comment.
I think that my 2nd suggestion just works in any case and you don't need to spend time digging. It's all going to be fast enough as well.
There was a problem hiding this comment.
Your second suggestion doesn't seem to unmask IMGRAW, and I thought the larger point was not to let the masked array end up as the ACAImage still masked?
There was a problem hiding this comment.
That was assuming your change imgraws = dat["IMGRAW"].filled(-9999). (Note the first sentence in https://github.com/sot/mica/pull/297/files#r1539513631).
There was a problem hiding this comment.
OK. I think I addressed the comments here.
Description
Handle filling aca image array differently
Fixes #293
Interface impacts
Testing
Unit tests
Independent check of unit tests by Javier
Functional tests
No functional testing.