Provide external data dependencies for FOT Matlab tools

[taldcroft]

NOTE : this is transferred to the Wiki

The following external data files are required for selecting acquisition stars and need to be available to any machine running proseco.

• $SKA/data/agasc/proseco_agasc.h5 (Corresponds to AGASC 1.7 for proseco 4.0 release)
• $SKA/data/agasc/agasc_supplement.h5
• $SKA/data/mica/archive/aca_dark/*/image.fits
• $SKA/data/mica/archive/aca_dark/*/properties.json

The dark current files currently span the entire mission. This isn't really necessary for production planning, but what is necessary is to include new files after a dark cal. When SRDC's become a reality this could mean an update every few days, but we don't need to deal with this for the 4.0 release.

For the optional functionality of selecting a catalog for an archival observation using only the obsid as input, the following files are also needed:

• $SKA/data/cmd_states/cmd_states.db3
• $SKA/data/mica/archive/starcheck/starcheck.db3

These files are needed to perform all tests (they will be skipped if not available).

Assigned to @mbaski.

[mbaski]

@taldcroft Will the copy of the dark current files on the GRETA network be updated timely enough that it can be the source for FOT Windows users?

[taldcroft]

@mbaski - sure, we can make it so.

The files will be on GRETA in those directories with $SKA=/proj/sot/ska. Note, in general there will be soft links from different environments, so e.g. /proj/sot/ska3/flight/data => /proj/sot/ska/data. Apps can expect data to be in $SKA/data where $SKA must be provided by the environment. (In case any of this makes a difference to what you do on the Windows side).

Eventually there will be other files, e.g. the dynamically-updated list of best-available mags for stars that have been observed.

[taldcroft]

So this is also an action for me or @jeanconn to ensure data are available on GRETA. For the 4.0 proseco release it should sufficient to ensure by process that a new dark cal gets copied over in a timely manner.

p.s. 4.x is our version designator for code that is not Py2 / Ska2 compatible.

[mbaski]

Here's the approach I'm implementing for Windows (with the goal being minimal effort and inconvenience for the user):

• At the first startup with the proseco release, the user will be queried to "install" the data files if they don't have them yet
• At future startups, the user will be queried to update the files (they'll have an option to never be asked again)
• The data files will be initially copied and then updated in the future using rsync (using the cygwin rsync executable, which unfortunately brings some baggage but so far looks doable)
• The first time the user will be asked for their login credentials. ssh keys will be generated and setup at that point to (hopefully...) eliminate the need for future credential entry.
• If the file syncing tends to take more than a few seconds, it will be performed in parallel in the background
• This same methodology (and code) is planned to be used for syncing local copies of individual Ska Eng Archive files (to keep a local copy of a subset of the archive)

[taldcroft]

@mbaski - looks like a good plan.

[taldcroft]

@mbaski - I updated the description to include a couple more optional files, for your reference.

[taldcroft]

@mbaski - closing since my understanding is that you have implemented this for Matlab tools. Feel free to re-open if there is any outstanding issue.

[jeanconn]

I just reopened this because I'm not sure where to track that we need the new miniagasc.h5 in the data files for the Windows env.

[jeanconn]

@mbaski On the Windows side do you or will you set a SKA environment variable or a MICA_ARCHIVE variable or both?

[mbaski]

Thanks for re-opening this - it is not actually done (it's about 90% complete - all the various pieces are implemented, I'm just adding niceties for users, e.g. insight into how much data needs to be synced).

@jeanconn one of the files I'm syncing is the miniagasc.h5 file, so it will be taken care of.

[jeanconn]

Regarding the syncing of the miniagasc I wasn't sure if it was an rsync or the like (which should notice the file is different and update). On HEAD and GRETA I've set things now to use the new miniagasc by removing the old miniagasc and setting miniagasc.h5 to be a symlink to the new file.

[mbaski]

@mbaski On the Windows side do you or will you set a SKA environment variable or a MICA_ARCHIVE variable or both?

Both. And also mica.common.MICA_ARCHIVE (for star catalogs from obsid as input).

[mbaski]

This has now been implemented for user's Windows machines. The GRETA network is the source of the files so they need to be kept up to date.

[taldcroft]

@mbaski - which files are being synced?

[taldcroft]

Ah, I see some input from your separate email. It looks like optional files are synced if they are already in the data directory.

[mbaski]

For the record, here's the current list of files/directories synced:

Mandatory (if the directory doesn't exist, it is created):

1. ...\data\mica\archive\aca_dark\ (entire directory)
2. ...\data\agasc\miniagasc.h5 (single file) - this will likely be removed from the list once everyone has it (or the proseco specific version)

Optional (only synced if the directory exists):

1. ...\data\cmd_states\ (entire directory, excluding sub-directories)
2. ...\data\mica\archive\starcheck\starcheck.db3 (single file)
3. ...\data\kadi\ (entire directory excluding sub-directories, *.bak*, *-bak*, and *~)