findFeaturesSegment.py Error

[lwellerastro]

ISIS version(s) affected: isis8.1.0-RC2

Description
The newly available findFeaturesSegment script errors when working on overlapping images when not all segments overlap, even when new parameters minarea and/or minthickness are set to values greater than 0.

I tested on images noted in here and displayed in the qmos figure.

Initially I ran the test without any new script parameters assuming the script would test for zero overlap while working on segments, but that failed. I'm having a difficult time isolating the most relevant error message(s) because of all the output (and various warnings for deprecated/unsupported things), but I think the problem is mostly near the end of the script when it is trying to merge expected segment networks that don't exist. Here's a snippet with leading path info removed:

Traceback (most recent call last):
  File "isis8.1.0-RC2/bin/findFeaturesSegment.py", line 383, in <module>
    findFeaturesSegment(ui) 
  File "isis8.1.0-RC2/bin/findFeaturesSegment.py", line 354, in findFeaturesSegment
    output = output.get()
  File "isis8.1.0-RC2/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "isis8.1.0-RC2/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "isis8.1.0-RC2/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "isis8.1.0-RC2/bin/findFeaturesSegment.py", line 234, in generate_cnet
    segmented_net = cnet.from_isis(new_params["ONET"])
  File "isis8.1.0-RC2/lib/python3.9/site-packages/plio/io/io_controlnetwork.py", line 114, in from_isis
    with IsisStore(path, mode='rb') as store:
  File "isis8.1.0-RC2/lib/python3.9/site-packages/plio/io/io_controlnetwork.py", line 206, in __init__
    self._open()
  File "isis8.1.0-RC2/lib/python3.9/site-packages/plio/io/io_controlnetwork.py", line 220, in _open
    self._handle = open(self._path, self._mode)
FileNotFoundError: [Errno 2] No such file or directory: 'M156671044LE_ff_1_from_images_segment2.net'

I do get some basic findfeatures output despite the above errors and get a segment1.net file (see more below on that).

When I chose values>0 for minarea and minthickness I got somewhat different errors. I thought using minarea and minthickness would help in that the script would completely avoid the segments that had zero overlap and avoiding the expectation, but I don't think it even got that far.

DEBUG:root:From images overlapping Match: []
DEBUG:root:output: [None, None, None, None]
DEBUG:root:onets: []
Traceback (most recent call last):
  File "isis8.1.0-RC2/bin/findFeaturesSegment.py", line 383, in <module>
    findFeaturesSegment(ui) 
  File "isis8.1.0-RC2/bin/findFeaturesSegment.py", line 366, in findFeaturesSegment
    final_images = set.union(*tolists)
TypeError: unbound method set.union() needs an argument

The above is right after overlapstats output. There is no call to findfeatures that I can see.
It seems the new parameters don't work as intended or I don't understand how to use them.

I was not able to capture what was going to screen (using 'tee' and running the command directly in my shell), so I had to copy and paste all of it into a text file. (Although the amount of information going to screen is a lot, not having it is a problem - ideas on how to capture it? Essential for debugging.). It's clear from findfeatures program output (via Defaults/) that there are segments it could not find points for (normal 'no points' output and clean exit), but it did find some points for overlapping segments and that should be valid output for this script.

How to reproduce
See my user work area Isis3Tests/FFSegmentScript/Defaults/ for input images, fromlist and command (run.cmd) along with what went to screen in Stdout.txt.

See Isis3Tests/FFSegmentScript/PolyConstraints/ for similar but with values entered for minarea and minthickness. See Stdout.txt there for that screen grab.

The attached qmos screengrab is under the Defaults/ directory. The purple footprint is the 'match' image and the blue and green footprints are in the fromlist. Segment 1 for all images was the successful chunk and are the darker portions of the footprints and have the resulting segment1.net overlain on top. Segment 2 seems to be the problem, but I can't tell if it's across the board or just for green seg 2 and match seg 2 poly's.

Possible Solution
In the Defaults/ case, I can see the findfeatures call for M156671044LE_ff_1_from_images_segment2.net and that "No control points found!!", so maybe it is just the script having the expectation that there will be success and the output network will be created when that is not always going to be the case. So that might need to be a test if not already one.

But I also see via the findfeatures log (M156671044LE_ff.log) that it is indeed trying to work on segments that do not overlap at all which I think we should completely be avoiding. So if not already testing on valid overlaps via the overlapstats info (don't know what is being passed since it's going to /tmp), then for sure via the minarea/minthickness parameters, but test two shows that has some issues still.

Additional Context
I first tested the updated script on a different 3 image set where all segments overlapped completely and a final network was produced. I then ran a test on a larger set of data (dozens of images) that blew up and the above images came from that larger set (plus these are the same images in the mentioned PR). So currently, the script seems to run in default mode for the most ideal situation but not the more realistic complex situations.

[Kelvinrr]

What do you mean you were not able to capture what was going to the screen? The logger should be outputting to stdout, did redirecting output not work? e.g. findfeaturesSegment.py ... >& log.out

[lwellerastro]

The specific errors in my original post are not captured in the print.prt so I tried using 'tee'

This would normally allow for standard output capture to a file while watching the process run in real time, but instead all that was captured in the tee output was "PathRemoved/isis8.1.0-RC2/bin/xml/findFeaturesSegment.xml". Tee normally works for isis applications and I guess it's not working as expected here because this is a python script?

I just used your command (which I was not familiar with) and it worked (but nothing went to screen, only file) - thanks.

[Kelvinrr]

Yeah that command won't let you monitor it while it's released, usually I just output to file, background it and run tail on it on occasion. If tail is simply redirecting streams, I don't know why it wouldn't work. Maybe something weird with how the logger is interacting with tee. Something to look into.

[Kelvinrr]

Alrighty, looking at it, it seems like it's a simple bug of the app trying to merge non-existing networks after a failure. MinThickness doesn't seem to be a very useful metric? I included it originally as it seemed like a simple way to filter out these issues, but it seems letting findfeatures naturally fail is just as good rather than tuning a parameter.

The error from setting minthickness>0 is because of how small the values are. Even in images with good overlaps.

Adding some simple error checking seemed to fix it, PR up in a bit.

[lwellerastro]

Had the opportunity to test the updates and we are very good to go. Awesome work - thanks!

[lwellerastro]

I just tested findFeaturesSegment.py under isis8.2.0-RC1 and got the following error:

(isis8.2.0-RC1) igs{140}> findFeaturesSegment.py 
Traceback (most recent call last):
  File "/usgs/cpkgs/anaconda3_linux/envs/isis8.2.0-RC1/bin/findFeaturesSegment.py", line 7, in <module>
    import kalasiris as kisis
ModuleNotFoundError: No module named 'kalasiris'

Run on a local VM.

[lwellerastro]

I tested these updates under a isis8.2.0 (locally on an astrovm) and get the following error (leading path info removed):

(isis8.2.0) igs{220}> cat LOG_I820_SysScript.out
  File "isis8.2.0/bin/findFeaturesSegment.py", line 389
    log.info(f"COMPLETE, wrote { ui.GetFileName("onet")}")
                                                 ^^^^
SyntaxError: f-string: unmatched '('

I copied the script into my working area and commented out line 389 to get it to run.
Good news is the issue in this particular post has been addressed.

I get a similar SyntaxError under 8.3.0-RC1.

Please let me know if additional information is needed (or if this needs to be a new post).

[Kelvinrr]

@lwellerastro we didn't update that script, the changes haven't gone in yet, are you running the script from the install or running a local copy of it?

[lwellerastro]

This issue was merged back in April and made it to 8.2.0 - an install that was done locally by IT (not me - I don't ever install versions of isis to work with ), as described above.

I saw the update in a recent announcement and thought I would test it.

The issue in this post has been addressed (when I copy the script into my directory and comment out the creating the error) except for the new Syntax Error above (something that we encountered a long time ago so I was surprised to see it here in the astrovm version). Not sure else how to make this clearer.

I guess I'll make a new post since it's a new error?

Version of python if it helps:

conda list | egrep 'isis|python'
# packages in environment at /usgs/cpkgs/anaconda3_linux/envs/isis8.2.0:
isis                      8.2.0                         0    usgs-astrogeology
msgpack-python            1.0.8           py311h52f7536_0    conda-forge
python                    3.11.6          hab00c5b_0_cpython    conda-forge
python-dateutil           2.9.0post0      py311h06a4308_2  
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    5_cp311    conda-forge