hical fails on HIRISE edr image due to missing csv files

[marvins]

ISIS version(s) affected: 8.0.0

Description

While following the MRO tutorial on GitHub link, I am unable to perform radiometric calibration on any EDR images I downloaded from PDS.

(isis)$ hical from=ESP_072111_2645_RED0_0.cub to=ESP_072111_2645_RED0_0.cal.cub
**USER ERROR** Failed in ZeroDark module.
**ERROR** No existing files found with a numerical version matching [B_TDI128_BIN1_hical_????.csv] in 
[/Volumes/data/ISISDATA/mro/calibration/matrices].

The images in question are located here...
https://hirise-pds.lpl.arizona.edu/PDS/EDR/ESP/ORB_072100_072199/ESP_072111_2645/

After downloading the images, I converted them to a .cub format. I then add the spice info. I verified the SPICE information is properly appended by running gdalinfo -mdd all ESP_072111_2645_RED0_0.cub.

I've fully downloaded the entire catalog of MRO ISIS data using the following...

downloadIsisData base ${ISISDATA}
downloadIsisData mro ${ISISDATA}

I also verified the ${ISISDATA}/mro/calibration/matrix folder has data.

(isis) $ ls /Volumes/data/ISISDATA/mro/calibration/matrices/
A_TDI128_BIN1_hical_0006.cub            A_TDI8_BIN2_hical_0001.cub              B_TDI64_BIN4_hical_0005.cub             G_TDI32_BIN8_hical_0007.cub
A_TDI128_BIN2_hical_0006.cub            A_TDI8_BIN4_hical_0005.cub              B_TDI64_BIN8_hical_0005.cub             G_TDI64_BIN1_hical_0008.cub
A_TDI128_BIN4_hical_0005.cub            A_TDI8_BIN8_hical_0001.cub              B_TDI8_BIN1_hical_0005.cub              G_TDI64_BIN2_hical_0008.cub
A_TDI128_BIN8_hical_0001.cub            B_TDI128_BIN1_hical_0006.cub            B_TDI8_BIN2_hical_0005.cub              G_TDI64_BIN4_hical_0008.cub
A_TDI32_BIN1_hical_0005.cub             B_TDI128_BIN2_hical_0005.cub            B_TDI8_BIN4_hical_0005.cub              G_TDI64_BIN8_hical_0007.cub
A_TDI32_BIN2_hical_0005.cub             B_TDI128_BIN4_hical_0005.cub            B_TDI8_BIN8_hical_0005.cub              G_TDI8_BIN1_hical_0007.cub
A_TDI32_BIN4_hical_0005.cub             B_TDI128_BIN8_hical_0005.cub            G_TDI128_BIN1_hical_0008.cub            G_TDI8_BIN2_hical_0007.cub
A_TDI32_BIN8_hical_0001.cub             B_TDI32_BIN1_hical_0005.cub             G_TDI128_BIN2_hical_0008.cub            G_TDI8_BIN4_hical_0007.cub
A_TDI64_BIN1_hical_0005.cub             B_TDI32_BIN2_hical_0005.cub             G_TDI128_BIN4_hical_0008.cub            G_TDI8_BIN8_hical_0007.cub
A_TDI64_BIN2_hical_0005.cub             B_TDI32_BIN4_hical_0005.cub             G_TDI128_BIN8_hical_0007.cub            ReverseClockStatistics.0001.conf
A_TDI64_BIN4_hical_0005.cub             B_TDI32_BIN8_hical_0005.cub             G_TDI32_BIN1_hical_0008.cub             ReverseClockStatistics.0002.conf
A_TDI64_BIN8_hical_0001.cub             B_TDI64_BIN1_hical_0005.cub             G_TDI32_BIN2_hical_0008.cub
A_TDI8_BIN1_hical_0001.cub              B_TDI64_BIN2_hical_0005.cub             G_TDI32_BIN4_hical_0008.cub

Current Hypothesis:

• After fighting with the config-file for a while, I realized that the files I have are of ISIS Cube (.cub) format. The hical application needs the files in a CSV format.
  • Is there a command to downloadIsisData that I'm missing?
  • Is there a command to convert these CUBE files to CSV?
  • I re-ran downloadIsisData multiple times and still not getting these CSVs.
    • This includes with -include='calibration/*' and other tweaks.

Hardware:

• 2019 MacBook Pro (Intel)
• Ventura 13.4.1

Other Notes

If this is a documentation issue or small coding problem, I'm happy to contribute a Pull Request.

Please pass on my thanks to the developers. This documentation is very impressive and for the most part, I've had minimal hassles. Appreciate it.

Marvin

[rbeyer]

@marvins, it appears that you are missing files from your ${ISISDATA}/mro/calibration/matrix/ directory. If you only have .cub files there, you are then missing almost 260 .csv files that should also have been downloaded. There is no "conversion." The .cub files and the .csv files contain different and separate information for ISIS hical to operate. I suspect that this is some problem with downloadIsisData that it is not somehow getting all of the files it needs (or they are somehow not in the Amazon buckets that it is pulling from)? I'm honestly not sure, I don't really know enough about downloadIsisData but that's where the problem is.

[KrisBecker]

@marvins there is a line in the downloadData script that essentially excludes all *.csv files in ISISDATA. This is a problem as it excludes the files mentioned in your post that are needed by hical.

The specific line to remove is:

    '- *.csv',

[marvins]

Worked like a charm. Appreciate the help.

If anyone is interested, I wrote an update for the wiki page here that explains this. I assume that CSV filter is there for a good reason, so I just added a note.

The only quirk is it seems you can't open PRs against wiki pages, so I've attached the block of markdown to copy/paste into the page, starting at line 59 (After the USGS hical link). I cloned the wiki and verified it should render correctly.

Thanks again.

#### Note on `hical` datasets

The `hical` program requires a set of radiometric calibration files which may not be downloaded when you run `downloadIsisData`.  This is because the download script filters out csv files from consideration.  The required files must exist in `${ISISDATA}/mro/calibration/matrices`.

To download the required CSV files, open up the `downloadIsisData` script.  If using conda, this is likely in your environment at `~/miniconda3/envs/isis/bin/downloadIsisData`.  Use `which downloadIsisData` to find the location if using a `bash` terminal.

Inside the file at approximately [line 41](https://github.com/DOI-USGS/ISIS3/blob/dev/isis/scripts/downloadIsisData#L41), remove this entry to stop filtering. 

    '- *.csv',

Next, re-run `downloadIsisData`.

    downloadIsisData mro $ISISDATA --include='calibration/**'

`${ISISDATA}/mro/calibration/matrices` should now have all required files.  Return the filter to the script.  Otherwise, be aware a conda update may revert it back to it's CM configuration.

Re-run your previous hical command. 

[rosykk1]

How to solve ERROR No existing files found with a numerical version matching [hical.????.conf]
Is there another .conf file that I missed?