Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

isisdataeval Updates, fixes and additional documentation #5163

Merged
merged 11 commits into from
May 4, 2023
Merged

isisdataeval Updates, fixes and additional documentation #5163

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
95ba42c
isisdataeval fixes, improvements, updates and docs
KrisBecker Mar 29, 2023
f23a4c0
CHANGELOG.md correction and update
KrisBecker Mar 29, 2023
498c9f4
isisdataeval documentation update via code review
KrisBecker Apr 3, 2023
60f8de5
isisdataeval install $ISISROOT/bin/isisdata_mockup
KrisBecker Apr 3, 2023
c3e033e
isisdataeval mockup script minor docs change
KrisBecker Apr 3, 2023
2cbe597
Rename isisdata_mockup.py to isisdata_mockup
KrisBecker Apr 5, 2023
3cddece
Merge branch 'DOI-USGS:dev' into feature/isisdataeval-update
KrisBecker Apr 10, 2023
07e3139
Merge remote-tracking branch 'origin/dev' into feature/isisdataeval-u…
KrisBecker Apr 30, 2023
76a9d02
Resolved merge conflict and add CHANGELOG.md entry
KrisBecker Apr 30, 2023
bcf1631
Merge branch 'feature/isisdataeval-update' of https://github.com/Kris…
KrisBecker Apr 30, 2023
7345ec0
Removed hardcoded path in isisdata_mockup
KrisBecker May 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ release.
## [Unreleased]

### Changed
- Removed the `.py` extention from the _isisdataeval_ tool `isisdata_mockup` for consistency and install it in $ISISROOT/bin; added the `--tojson` and `--hasher` option to _isisdata_mockup_ tool improve utility; updated the tool `README.md` documentation to reflect this change, removed help output and trimmed example results; fixed paths to test data in `make_isisdata_mockup.sh`. [#5163](https://github.com/DOI-USGS/ISIS3/pull/5163)

### Added

Expand All @@ -49,13 +50,13 @@ release.

### Changed
- Updated download location for Dawn source files to include updated pck from HAMO Dawn mosaic [#4001](https://github.com/USGS-Astrogeology/ISIS3/issues/4001)
- Pinned cspice version to 67 [#5083](https://github.com/USGS-Astrogeology/ISIS3/issues/5083)
- Pinned cspice version to 67 [#5083](https://github.com/USGS-Astrogeology/ISIS3/issues/5083)
- Changed the `rsync` related commands in the ISIS SPICE Web Service document to `downloadIsisData` command
- Updated Geos from version 3.7 to 3.9 [#3627](https://github.com/DOI-USGS/ISIS3/issues/3627)

### Added
- Instructions on setting `channel_priority=flexible` for isis environment manually during installation [#5158](https://github.com/DOI-USGS/ISIS3/issues/5158)
- Added additional filters to downloadIsisData to reduce download of extraneous kernels [#5143](https://github.com/DOI-USGS/ISIS3/issues/5143)
- Added additional filters to downloadIsisData to reduce download of extraneous kernels [#5143](https://github.com/DOI-USGS/ISIS3/issues/5143)

### Deprecated

Expand Down
3 changes: 3 additions & 0 deletions isis/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -600,6 +600,9 @@ install(DIRECTORY ${CMAKE_SOURCE_DIR}/scripts DESTINATION ${CMAKE_INSTA
install(PROGRAMS ${CMAKE_BINARY_DIR}/lib/Camera.plugin DESTINATION ${CMAKE_INSTALL_PREFIX}/lib/)
install(PROGRAMS ${CMAKE_SOURCE_DIR}/scripts/downloadIsisData DESTINATION ${CMAKE_INSTALL_PREFIX}/bin/)
install(FILES ${CMAKE_SOURCE_DIR}/config/rclone.conf DESTINATION ${CMAKE_INSTALL_PREFIX}/etc/isis/)
install(FILES ${CMAKE_SOURCE_DIR}/src/system/apps/isisdataeval/tools/isisdata_mockup
PERMISSIONS OWNER_WRITE OWNER_READ OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
DESTINATION ${CMAKE_INSTALL_PREFIX}/bin/)

# Trigger all post-install behavior.
# - The only way to run commands post-install in CMake is to add a subdirectory at
Expand Down
156 changes: 87 additions & 69 deletions isis/src/system/apps/isisdataeval/tools/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# <a name="#testingisisdataeval">ISISDATA Mockup Procedures for Testing isisdataeval</a>

Proper, thorough testing of ISIS application `isisdataeval` is difficult due to the requirements of having a stable and flexible ISISDATA directory structure to test with. It is unfeasible to rely on the real $ISISDATA due to its volitile and every changing content. One solution is to create a mockup of the ISISDATA directory but mimimize the resources to emulate a real, functioning ISISDATA installation. The system presented here generates a complete ISISDATA directory structure at a signification fraction of the actual dataset. By careful selective culling of many of the existing files in the selection mission datasets, it creates a real time snapshot of ISISDATA to help test `isisdataeval`.
Proper, thorough testing of ISIS application `isisdataeval` is difficult due to the requirements of having a stable and flexible ISISDATA directory structure to test with. It is unfeasible to rely on the real $ISISDATA due to its volitile and every changing content. One solution is to create a mockup of the ISISDATA directory but mimimize the resources to emulate/represent a real, functioning ISISDATA installation. The system presented here generates a complete ISISDATA directory structure at a signification fraction of the actual dataset. By careful selective culling of many of the existing files in the selection mission datasets, it creates a real time snapshot of ISISDATA to help test `isisdataeval` and validate the local ISISDATA install.

The ISISDATA mockup system is a complete copy of every directory and every file in an existing ISISDATA install. However, the contents of every file has been replaced with information about that file, keeping the size of every file to about 300 bytes each. Here is the format of a file prepared by this system:
The ISISDATA mockup system is a complete copy of every directory and every file in an existing ISISDATA install. However, the contents of every file has been replaced with information about that file, keeping the size of every file to about 400 bytes each. Here is the JSON format of a mocked ISISDATA file prepared by this system:

`cat isisdatamockup/base/kernels/spk/de118.bsp`
```
Expand All @@ -17,99 +17,117 @@ The ISISDATA mockup system is a complete copy of every directory and every file
}
```

The size of this file has been reduced to 376 bytes from 4,097,024 bytes. Each file will contain JSON text with details about the original file. The files size, creation and modified dates, and hash values for the md5, sha1 and sha256 hash algorithms computed from the contents of the file using Python tools - and independt. The hash values are expressly provided to provide comparisons of the Qt algorithms used in `isisdataeval`. Note it is possible, but cannot be guaranteed, the _source_ for the file will exists in every ISISDATA install, or even be the same file. But this is a good thing to test.
The size of this file has been reduced to 376 bytes from 4,097,024 bytes. Each file will contain JSON text with details about the original file. The files size, creation and modified dates, and hash values for md5, sha1 and sha256 hash algorithms are computed from the contents of the file using Python tools. This provides a an external source (Python) comparison of the Qt hash algorithnms used in `isisdataeval` for validation tests. Note it is possible, but cannot be guaranteed, the _source_ for the file will exist in every ISISDATA install, or even be the same file. But this is a good thing because it **validates** the local ISISDATA installation and test environment.

# <a name="#isisdatamockuptool">ISISDATA Mockup Tool - isisdata_mockup.py</a>
# <a name="#isisdatamockuptool">ISISDATA Mockup Tool - isisdata_mockup</a>

A tool has been provided with the `isisdataeval` application that generates ISISDATA mockups for this and potentially other purposes. This Python tool will convert an ISISDATA installation into a mockup that is suitable to test the database lookup system and verify the contents and structure of mission kernel configurations in $ISISDATA. However, it has been generalized to use for any directory structure. Here is the application documentation:

`isisdata_mockup.py -h`
A tool has been provided with the `isisdataeval` application that generates ISISDATA mockups for this and potentially other uses. This Python tool will convert an ISISDATA installation into a mockup that is suitable to test the database lookup system and **verify** the contents and structure of mission kernel configurations in $ISISDATA. However, it has been generalized to use for any directory structure. The application help documentation describes the parameter and behavioir of the script. Use the command `isisdata_mockup -h` to produce the documentation.

In general, if you want to produce a complete inventory of the ISISDATA directory, the form to use is:
```
usage: isisdata_mockup.py [-h] --isisdata ISISDATA --outpath OUTPATH [--ghostdir GHOSTDIR]
[--saveconfig] [--dryrun] [--verbose]

Convert ISISDATA directory to test format

optional arguments:
-h, --help show this help message and exit
--isisdata ISISDATA ISISDATA directory root
--outpath OUTPATH NEW ISISDATA directory to create
--ghostdir GHOSTDIR Replaces the --isisdata path with this string to ghost the actual input
directory path
--saveconfig, -s Retain *.db, *.conf files rather than replace with processs info
--dryrun, -n Only print actions but do not execute
--verbose, -v Verbose output

isisdata_mockup.py will create a copy of all the files found in the directory specified by
the --isisdata parameter. All files and directories contained in --isisdata will
be copied to the --outpath destination, which specifies a new directory. This
directory will be created if it does not exist.

All *.db and *.conf files are copied as is, fully intact. Also needed to use this
mock up of ISISDATA is the NAIF LSK. This LSK kernel is always loaded by the iTime
class to concert UTC and Et times. Otherwise, all other files encountered will
be created but the contents are replaced by information regarding the --isisdata
source file. This information includes the original file opath, creation and
modification dates in UTC format, the size of the file and its hash values.

This app sets up a full data directory structure that mimics the contents of
ISISDATA (or any directory for that matter). The size of the file, its creation
and last modification dates and the md5, sha1 and sha256 hash values are created
in a dict which is then written to the new output file if its not a special ISIS
kernel db/conf file or LSK.

Finally, by setting --ghostdir '$ISISDATA', this now provides a connection to
the source file in the ISISDATA directory. This can be used to compare hash values
compute by other sources.

Example:

To provide a full mock up of ISISDATA directory:

isisdata_mockup.py --saveconfig --isisdata /opt/isis/data --outpath $PWD/mockisisdata --ghostdir '$ISISDATA'

Author: Kris J. Becker, University of Arizona
kbecker@orex.lpl.arizona.edu

History 2023-03-15 Kris Becker - Original verison
isisdata_mockup --saveconfig --hasher all --isisdata /opt/isis/data --outpath $PWD/mockisisdata --ghostdir '$ISISDATA' --tojson isisdata_complete_mockup.json
```

You can also choose to process only a single directory, as shown in the following example:

```
isisdata_mockup.py --saveconfig --isisdata /opt/isis/data/voyager1 --outpath $PWD/mockisisdata/voyager1 --ghostdir '$ISISDATA/voyager1'
isisdata_mockup --saveconfig --hasher all --isisdata /opt/isis/data/voyager1 --outpath $PWD/mockisisdata/voyager1 --ghostdir '$ISISDATA/voyager1' --tojson isisdata_voyager_mockup.json
```
Processing times can be significant since this script is computing 3 different hash values per file.
Processing times can be significant since these examples are computing 3 different hash values per file. For this purpose, the `md5` hash algorithm is probably sufficient for ISISDATA file comparisons.


# <a name="#isisdatamockupinstall">ISISDATA Mockup Test Data Preparation</a>

With this tool and the files listed in `isisdataeval_isisdata_mockup_files.lis`, the test for isisdataeval can be recreated from any ISISDATA installation. Note that from time to time, the files used in this ISISDATA test could change which would cause failures. Here are the commands to create the `isisdataeval` test ISISDATA mockup directory - from the Git install directory:
With this tool and the files listed in `isisdataeval_isisdata_mockup_files.lis`, the test for isisdataeval can be recreated from any ISISDATA installation. Note that from time to time, the files used in this ISISDATA test could change which would cause failures. In this situation, maintainers will need to regenerate the ISISDATA mockup if the change is expected/needed. Here are the commands to create the `isisdataeval` test ISISDATA mockup directory originating from the Git install directory:

```
# Create the test data area for isisdata
cd ISIS3/isis/test/isisdata
# From the Git install directory, create the test data mockup in isisdata
cd ISIS3/isis/test/data/isisdata
mkdir -p mockup mockprocessing

# Produce the mockup data. Its assumed isisdata_mockup.py is in a runtime path.
isisdata_mockup.py --saveconfig --isisdata $ISISDATA/base --outpath mockprocessing/isisdatamockup/base --ghostdir '$ISISDATA/base'
isisdata_mockup.py --saveconfig --isisdata $ISISDATA/hayabusa --outpath mockprocessing/isisdatamockup/hayabusa --ghostdir '$ISISDATA/hayabusa'
isisdata_mockup.py --saveconfig --isisdata $ISISDATA/smart1 --outpath mockprocessing/isisdatamockup/smart1 --ghostdir '$ISISDATA/smart1'
isisdata_mockup.py --saveconfig --isisdata $ISISDATA/voyager1 --outpath mockprocessing/isisdatamockup/voyager1 --ghostdir '$ISISDATA/voyager1'
# Produce the mockup data. Its assumed isisdata_mockup is in a runtime path.
isisdata_mockup --saveconfig --hasher all --isisdata $ISISDATA/base --outpath mockprocessing/isisdatamockup/base --ghostdir '$ISISDATA/base' --tojson isisdata_mockup_base.json
isisdata_mockup --saveconfig --hasher all --isisdata $ISISDATA/hayabusa --outpath mockprocessing/isisdatamockup/hayabusa --ghostdir '$ISISDATA/hayabusa' --tojson isisdata_mockup_hayabusa.json
isisdata_mockup --saveconfig --hasher all --isisdata $ISISDATA/smart1 --outpath mockprocessing/isisdatamockup/smart1 --ghostdir '$ISISDATA/smart1' --tojson isisdata_mockup_smart1.json
isisdata_mockup --saveconfig --hasher all --isisdata $ISISDATA/voyager1 --outpath mockprocessing/isisdatamockup/voyager1 --ghostdir '$ISISDATA/voyager1' --tojson isisdata_mockup_voyager1.json

# Copy/install the desired files for the test
rsync -av --files-from=isisdataeval_isisdata_mockup_files.lis mockprocessing/isisdatamockup/ mockup/
rsync -av --files-from=isisdataeval_isisdata_mockup_files.lis mockprocessing/isisdatamockup/ mockup/
/bin/rm -rf mockprocessing

# Run an inventory test for the mockup
# Run an inventory on the local ISISDATA mockup test data
isisdataeval isisdata=mockup datadir=mockup toinventory=isisdata_mockup_inventory.csv toissues=isisdata_mockup_issues.csv toerrors=isisdata_mockup_errors.csv hash=md5
```

# Other products
The `isisdata_mockup` application can also produce a summary procesing log file of the run used to create a mockup. It can also be ran standalone on any system to provide as an comparison dataset for any mockup. This command produces a summary of the full ISISDATA installation without producing the mockup in `--outpath` (it is now an optional parameter), computes only the md5 hash and writes the entire contents to the JSON output file specified in `--tojson`:

```
isisdata_mockup -v --hasher=md5 --isisdata /opt/isis3/data --ghostdir '$ISISDATA' --tojson isisdata_full_md5.json
```

This example provides the results of a much smaller directory, $ISISDATA/base/dems, that also computes the md5 hash, including all `*.db` and `*.conf` files, does not generate, unless given, any absolute paths, and **does not** produce the mockup directory:
```
isisdata_mockup --hasher md5 --isisdata /opt/isis3/data/base/dems --ghostdir '$ISISDATA/base/dems' --tojson isisdata_base_dem_md5.json
```

And here is portion of the contents of the output JSON file, `isisdata_base_dem_md5.json`:
```
{
"program": {
"name": "isisdata_mockup",
"version": "0.2",
"date": "2023-03-29",
"runtime": "2023-03-29T21:02:03.721127 UTC",
"endtime": "2023-03-29T21:02:39.887436 UTC",
"elapsedtime": "0:00:36.166309",
"parameters": {
"isisdata": "/opt/isis3/data/base/dems",
"outpath": null,
"ghostdir": "$ISISDATA/base/dems",
"hasher": [
"md5"
],
"saveconfig": false,
"tojson": "isisdata_base_dem_md5.json",
"dryrun": false,
"verbose": false
}
},
"inventory": {
"hashlist": [
"md5"
],
"missing_count": 0,
"files_count": 33,
"missing": [],
"files": [
{
"source": "$ISISDATA/base/dems/Ceres_Dawn_FC_HAMO_DTM_DLR_Global_60ppd_Oct2016_prep.cub",
"filesize": 467404363,
"createtime": "2016-10-17T21:20:18.000000",
"modifiedtime": "2020-02-13T16:33:27.681881",
"md5hash": "e8e8763630d938caf9cfc52b2820f35b"
},
{
"source": "$ISISDATA/base/dems/LRO_LOLA_LDEM_global_128ppd_20100915.cub",
"filesize": 2141164162,
"createtime": "2011-03-19T19:18:34.000000",
"modifiedtime": "2020-02-13T16:36:48.776844",
"md5hash": "cb36cb569671f4db0badc29d737ed6d5"
},
{
...more file descriptions follow...
}

]
}
}
```

Note this file can then be shared and compared using the same runtime parameters on other platforms. I would refrain from comparing times, but all the other data (i.e., source, filesize, and hash values) can be directly compared for equivalence to help identify any differences from, say, a reputable ISISDATA source.

# <a name="#isisdatamockuptests">Running ISISDATA Mockup Tests</a>
Once the test data is installed, the contents ./ISIS3/isis/tests/data/isisisdata/mockup will contain the test data created above. To explicitly run the `isisdataeval` tests, use `ctest -R IsisData`. If an error occurs, rerun with the command `ctest --rerun-failed --output-on-failure` to get specific information about which tests failed and why.
Once the test data is installed, the contents of ./ISIS3/isis/tests/data/isisisdata/mockup will contain the test data created above. To explicitly run the `isisdataeval` tests, from the build directory, use `ctest -R IsisData`. If an error occurs, rerun with the command `ctest --rerun-failed --output-on-failure` to get specific information about which tests failed and why.

Note that during this test, the contents of files with less than 102400 bytes will have all three hash values compute in the tests and compared with the values stored in hash keywords, _md5hash_, _sha1hash_ and _sha256hash_ for an external validation hashing. Note also that the all files $ISISDATA/base/kernels/lsk are retained since Isis::iTime requires an LSK and it always loads one from $ISISDATA/base/lsk/naif????.tls.
Note that during this test, the contents of files with less than 102,400 bytes will have all three hash values compute in the tests and compared with the values stored in hash keywords, _md5hash_, _sha1hash_ and _sha256hash_ for an external validation of hashing algorithms. Note also that all files in $ISISDATA/base/kernels/lsk are retained since Isis::iTime requires an LSK and it always loads one from $ISISDATA/base/lsk/naif????.tls.

Loading