Ska3 runtime environment for maintainers

This document outlines the process for building and maintaining Ska3 environments and the associated packages.

Package update, testing and control process

Prior to making a release, the update needs to be developed, tested (which may including integration testing), reviewed and approved. This process is summarized below.

Package developer process

All Ska3 packages should:

Have a build recipe in sot/skare3
Use setuptools_scm. This automatically sets version numbers based on git tags.
Have the following workflows:
- build (.github/workflows/build.yml) calls the sot/skare3/package_build.yml workflow.
- release (.github/workflows/release.yml) calls the sot/skare3/package_release.yml workflow.
- documentation (.github/workflows/docs.yml) calls the sot/skare3/package_docs.yml workflow.

Once a package is properly setup, development is as follows:

Develop PR(s), make sure tests are passing on local development platform
Get code review on Github
Do unit and functional tests within git repo on supported platforms, specifically:
1. Mac High Sierra using ska3-flight (or possibly ska3/test if other package dependencies are involved).
2. HEAD linux-64 using ska3-flight (or possibly ska3/test)
3. GRETA linux-64 using ska3-matlab. This should be done using /proj/sot/ska3/matlab32/test, which represents either the current MATLAB environment or the pending one for the next promotion.

If warranted do system integration testing, as shown in the example for xija:

Create a test env and install the package using pip:

conda create -n test-ska ska3-flight
source activate test-ska
conda uninstall --force xija
cd ~/git/xija
pip install --no-deps --verbose --no-binary :all: --no-index .  # add --egg for namespace package

Run ska_testr
Integration functional tests as needed

Seek approval from controlling authority (if needed)
Merge to master. At this point, a conda package is built and copied into the masters conda channel, with a version matching the sha of the merge commit.
Every night, a set of regression tests is run using the latest package from master. The results are displayed in the SkaRE3 dashboard
When ready, make a release on Github. A Github release should trigger the following automated responses:
- the build workflow builds a conda package with the new version
- the documentation workflow builds the HTML documentation and posts it so it appears in [sot.github.io]
- the release workflow does the following:
  - opens an issue in sot/skare3 (with the Package Update tag) to remind the Ska3 maintainer that this package should go in an upcoming release.
  - edits the release description to list all PRs since last release.
Some times things fail. In these cases, the Ska3 maintainer can do it manually:
- If the package didn't build, it will have to either be built manually or using an event dispatch (details below).

Ska3 maintainer process

The Ska3 release process is summarized in the ska3-flight and ska3-matlab issue templates. The following list gives a bit more detail, but it might be outdated in comparison with the issue templates.

Preparation:

Create a milestone for the release (Usually called the same as the release, the milestone name is used below)
Create an issue on sot/skare3 to keep track of the release process and assign it to the milestone. There are templates for releasing ska3-flight and ska3-matlab which include a checklist like the following.
Review the packages with merged PRs and/or new releases in the SkaRE3 dashboard . Password available here
Review the skare3 issues with the Package Update tag and coordinate with developers to see which are ready for release. Assign milestones to the issues.
If the package updates will require recipe changes, do those recipe updates first in new PRs, approve, merge to skare3 master.

Package and Test:

Create a branch named {version}-branch. The name is important, because otherwise the build will not succeed.
Create a PR titled {version} from {version}-branch into master. The name is important, because otherwise the build will not succeed. Take a look at the https://github.com/sot/skare3/blob/master/.github/PULL_REQUEST_TEMPLATE/ska3-flight.md and https://github.com/sot/skare3/blob/master/.github/PULL_REQUEST_TEMPLATE/ska3-matlab.md pull request templates
Make all needed changes to meta.yml.
Create a pre-release {version}rc{N} at the latest commit in the branch. This triggers the automated build/test workflow.
Check that all packages are built and automated tests pass. This can be done in the actions section.

Promote packages from masters to test channel.

skare3-promote --to test --ska3-conda www/ASPECT/ska3-conda ska3-flight=={version}rc{N} [--dry-run]

Install on HEAD and GRETA Linux test environments (ska3/test).
Run testr on HEAD and GRETA

Add related issues to PR

skare3-milestone-issues --repository sot/skare3 --milestone {release}

Document all changes
```
skare3-changes-summary --conda-channel test --meta-package ska3-flight --final-version {version}
```
NOTE: for this command, you will need a Github token passed in the command line or as the GITHUB_TOKEN environment variable.
Write summary and highlight relevant changes.

Document test status

skare3-test-dashboard -i /path/to/test/results

Review PR internally.
Request and wait for approval.

Promotion:

Promote packages to main conda channel.

skare3-promote --to flight --ska3-conda www/ASPECT_ICXC/ska3-conda ska3-flight=={version} [--dry-run]

Notify ACA.
Install ska3/flight on HEAD as aca user details.
Install ska3/flight on GRETA as SOT user details.
Run ska_testr on HEAD and GRETA.

Document test results.

skare3-test-dashboard -i /path/to/test/results

Notify ACA and the community.
Merge PR

Creating and maintaining conda package builds

GitHub process

The next step in building a package is to have that package available on GitHub with the desired release tagged with a version specifier like 3.5.2 or 3.5.2rc1 (for testing). If there has been an update, this must be approved by the relevant controlling authority, merged to master, and tagged on GitHub.

Metapackages

The skare3 build system also includes several recipes for conda metapackages. The ska3-flight metapackage is one such package. If an individual Python module package update being reviewed or is ready to be installed in the flight environment, say on the annie package, for example, then there is also a required step to make and receive approval for a pull request to change the meta.yaml file in the ska3-flight recipe within skare3 to reflect the new annie version and to increment the version of the ska3-flight metapackage itself. After that PR is merged, one would need to use the ska3_builder process (below) build both the annie conda package and the ska3-flight conda package.

An update to a core package in the ska3-core metapackage requires an increment of the version of ska3-core, and update to the ska3-core entry in ska3-flight, an increment in the ska3-flight metapackage version, and a rebuild of both the ska3-core and ska3-flight metapackages.

Setting up build environments

Linux

To support CentOS5, packages should be built from a CentOS5 machine (though for noarch packages it should really make no difference) and the user running the scripts will need a CentOS5-compatible .condarc. First, edit your user ~/.condarc to include:

binstar_upload: false
channels:
  - https://icxc.harvard.edu/aspect/ska3-conda/
  - https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
#  - defaults
#default_channels:
#  - https://repo.continuum.io/pkgs/free
#  - https://repo.continuum.io/pkgs/r
#  - https://repo.continuum.io/pkgs/msys2

(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)

This is already configured for the 'aca' user. Then fetch a miniconda installer still compatible with CentOS5 and install

chmod o-rwx ~/.condarc
curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh
bash Miniconda3-4.3.21-Linux-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # Update Python but keep conda at 4.3.21
conda install ska3_builder  # This package must be installed in a "root" conda environment

For the aca user this install has been done with the ${miniconda_install_prefix} set to /export/aca/ska3 on presto .

OSX

For OSX we could use newer packages in ska3-dev in the modern channels, but by default, we are using the same .condarc used in Linux. This prevents getting carried away with features that we can't actually use on Linux production systems. So, the .condarc is set as such on the skare3 osx build system, zero. (zero is presently not available so this section of the directions is not up-to-date)

binstar_upload: false
channels:
  - https://icxc.harvard.edu/aspect/ska3-conda/
  - https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
#  - defaults
#default_channels:
#  - https://repo.continuum.io/pkgs/free
#  - https://repo.continuum.io/pkgs/r
#  - https://repo.continuum.io/pkgs/msys2

(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)

On zero, the aca user also has a .bash_profile that has the miniconda environment already loaded, with the miniconda_install_prefix set to '/Users/aca/miniconda3/bin/conda'. If this needs to be redone, these would be the steps.

curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh
bash Miniconda3-4.3.21-MacOSX-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # update Python but keep conda at 4.3.21
conda install ska3_builder  # This package must be installed in a "root" conda environment

Building locally

The underlying packages that comprise the ska3 environment are built using the ska_builder.py script, which is available via the skare3 git repository with the setup described above. Several of the packages are private github packages, and for these to clone and build locally you will need working github ssh authentication (this is already set up for the aca user). If you haven't set this up, see https://help.github.com/articles/connecting-to-github-with-ssh/ . You may also want to manage your key with ssh-agent before building packages. See https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#adding-your-ssh-key-to-the-ssh-agent

After ssh is set up get into the appropriate conda environment for building (if not already set from above steps). For builds as aca, that is currently

export PATH=/export/aca/ska3/bin:$PATH

as the aca user on presto (also set as ska3build alias if working in .cshrc)

For 32 bit builds for matlab build as the aca user on quango and set path

export PATH=/export/aca/miniconda3/bin/:$PATH

For osx use the aca user on the HEAD system 'zero', and the PATH is already set.

Then set the build root

export build_root=/export/aca/ska3-conda-builds  # As ACA for production on presto
 OR
export build_root=/export/aca/build_root # As ACA user for 32 bit builds on quango
OR
export build_root=~/ska3-conda-builds  # In user home dir

Then get in to the skare3 repo and do the build. Clone if needed

cd ~/git
git clone git@github.com:sot/skare3

Check out the skare3 packages at the release tag intended for the ska3-flight or ska3-matlab release.

cd ~/git/skare3
python ska_builder.py --build-root=${build_root} <package>

By default the latest tag (sorted by commit date, not tag date) is built. To build a specific tag, specify that with the --tag=<tag> option.

In order to build all available packages at once, just omit the <package>:

cd ~/git/skare3
python ska_builder.py --build-root=${build_root}

This will clone each repo if needed from GitHub or fetch updates, check out at the desired tag, and build the package using the conda build recipe.

The cloned repos are in ${build_root}/srcs and the builds are in ${build_root}/builds.

You can then install the new built package into a non-production environment using:

conda install -c file://${build_root}/builds <package>

Assuming you have built everything, then you can use the meta-package names shown earlier like ska3-flight.

Updating the `ska3-conda` channel

The ska3-conda channel is a non-public channel that is served on the icxc web server and can be accessed via a typical conda channel URL spec.

The channel URL is https://icxc.cfa.harvard.edu/aspect/ska3-conda , corresponding to a file directory of /proj/sot/ska/www/ASPECT_ICXC/ska3-conda.

Linux

For production linux builds, do the following (still as user aca on presto):

# Get into ska3_builder environment if not already there

rsync -ruv ${build_root}/builds/linux-64/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/

For the test channel one would use (note that these are on the public web and not ICXC, they are rsync'd to ICXC from public web):

rsync -ruv ${build_root}/builds/linux-64/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/

MacOSX

For production MacOSX builds, after building the packages as the aca user on the zero system, sync back to the HEAD LAN as as the aca user on the zero system:

ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/

For the test channel one would use (note that these are on public web. A cronjob syncs to ICXC)

ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/

Sherpa package (linux and MacOSX)

The sherpa conda packages are copied in to the ska3-conda channel and served as if they were ska packages. The 4.9.1 Linux version was fetched from https://anaconda.org/CXC/sherpa/files and the 4.9.1 Mac version was fetched from https://anaconda.org/sherpa/sherpa/files. The channel indexes were updated appropriately. We're using the pinned and manually fetched build of 4.9.1 from Linux dev channel to support a custom build from the Sherpa team to support CentOS5. At some point in the future when CentOS5 support is deprecated in skare3, we could update via adding the sherpa channel to the config.

cd /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64
curl -L -O https://anaconda.org/CXC/sherpa/4.9.1/download/linux-64/sherpa-4.9.1-py36he678ed3_0.tar.bz2
cd ..
conda index linux-64
cd osx-64
curl -L -O https://anaconda.org/sherpa/sherpa/4.9.1/download/osx-64/sherpa-4.9.1-py36_0.tar.bz2
cd ..
conda index osx-64

Updating the core conda packages

(still needs work)

The ska3-core package should be updated once or twice a year to prevent core conda packages from getting too stale. The process is to

review the ska3-pinned metapackage

This package holds back packages when newer versions of those packages will break compatibility across our platforms. At the time of the initial release of skare3, we are holding back packages to continue to support CentOS5 and those packages are called out in ska3-pinned. For a core update, the pinned list in the ska3-pinned metapackage should be reviewed, and if any of the restrictions no longer apply, the package should be edited and rebuilt.

install the ska3-dev and ska3-pinned package in an env

On each supported platform, the ska3-dev and ska3-pinned packages should be installed in a fresh conda env.

conda create -n core-update ska3-dev ska3-pinned

This will create a new environment where the packages will have floated to versions as high as allowed by the restrictions in ska3-pinned.

save new platform-specific package list

Where {platform} is replaced with a string to record which platform or machine was used to make the file, do:

source activate core-update
conda list --json > {platform}.json

Then copy the json files so they are in the same directory on a machine with access to Python and a script in the skare3 repo.

generate new combined-arch package list

There is a defunct script, ska_conda/pkg_defs/ska3-core/combine_arch.py, that was used to make the first combined package list when the operating systems were just 'linux' and 'osx'. That script will need rewriting if one desires to programmatically update the ska3-core pkg list from multiple operating systems including linux-32, linux-64, os-x, and win. In the interim, the strategy is to review the individual package listings from the multiple machines (the {platform}.json files or just the output of conda list) and update ska3-core meta.yaml manually.

rebuild and test

Commit the ska3-core changes. Update ska3-flight with the new ska3-core version in the run requirements. Tag the changes in skare3. Then, rebuild ska3_builder, ska3-core, and ska3-flight. Install the updated ska3-flight to a test area and test. If all tests look good put updated packages in the web-accessible repository.

Updating production and test environments

First, notify the other maintainers that you are about to do the actual install (with email to aca@cfa).

HEAD

ska3-flight is installed on HEAD at

/proj/sot/ska3/flight

Login as aca user on presto, then (assuming bash)

source /proj/sot/ska3/flight/bin/ska_envs.sh
conda list ska3-flight  # check version
conda update ska3-flight  # confirm package update looks right

GRETA

Coordinate with FOT-matlab before proceeding.

ska3-flight and ska3-matlab environments GRETA follow similar naming convention:

/proj/sot/ska3/flight
/proj/sot/ska3/matlab/flight
/proj/sot/ska3/matlab/test
/proj/sot/ska3/matlab32/flight
/proj/sot/ska3/matlab32/test

To install ska3-matlab, login as SOT user on chimchim and cheru and do the following (assuming bash):

source /proj/sot/ska3/matlab/flight/bin/ska_envs.sh
conda list ska3-matlab  # check version
conda update ska3-matlab  # confirm package update looks right

Same thing with flight, mutatis mutandi.

For ska3-matlab 32 bit, logged in as SOT user on chimchim (assuming bash):

source /proj/sot/ska3/matlab32/flight/bin/ska_envs.sh
conda update ska3-matlab

A special case is the 32 bit environment on cheru. Logged in as SOT user on cheru:

rsync -av --dry-run --delete chimchim:/proj/sot/ska3/matlab32/flight/ /proj/sot/ska3/matlab32/flight/

Finally:

notify chandracode about the update
for a ska3-flight update, also notify mpweekly about the update.

Setting up a New Production Environment

If reusing an previously-used environment directory, just move the old content.

Then, on GRETA first follow the steps in Syncing conda builds to GRETA. Then follow the directions above to install a miniconda environment, except the miniconda directory will be at the production directory location / prefix, i.e. "/proj/sot/ska3/test" or "/proj/sot/ska3/flight". Then install ska3-flight and ska3_builder into the "root" conda environment with the following as the aca user (set the PATH to get into that conda environment if not already set):

% conda install ska3-flight ska3_builder task_schedule watch_cron_logs

(It is useful to have the ska3_builder environment to guarantee the right version of the conda environment and to have access to "conda index". task_schedule and watch_cron_logs should probably move to a metapackage at some point.)

As a non-privileged user, you can use the environment by sourcing into it:

% source ${prefix}/bin/ska_envs.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ska3 runtime environment for maintainers

Contents

Package update, testing and control process

Package developer process

Ska3 maintainer process

Preparation:

Package and Test:

Promotion:

Creating and maintaining conda package builds

GitHub process

Metapackages

Setting up build environments

Linux

OSX

Building locally

Updating the `ska3-conda` channel

Linux

MacOSX

Sherpa package (linux and MacOSX)

Updating the core conda packages

Updating production and test environments

HEAD

GRETA

Setting up a New Production Environment

For Users

For Maintainers

Continuous Integration

Old Pages

Clone this wiki locally

Ska3 runtime environment for maintainers

Contents

Package update, testing and control process

Package developer process

Ska3 maintainer process

Preparation:

Package and Test:

Promotion:

Creating and maintaining conda package builds

GitHub process

Metapackages

Setting up build environments

Linux

OSX

Building locally

Updating the ska3-conda channel

Linux

MacOSX

Sherpa package (linux and MacOSX)

Updating the core conda packages

Updating production and test environments

HEAD

GRETA

Setting up a New Production Environment

For Users

For Maintainers

Continuous Integration

Old Pages

Clone this wiki locally

Updating the `ska3-conda` channel