Skip to content

Ska3 runtime environment for maintainers

Jump to bottom Edit New page
Jean Connelly edited this page Apr 11, 2022 · 108 revisions

This document provides a desired interface and outlines the process for building and maintaining Ska3 environments and the associated packages.

Contents

Resources

Cron Jobs for Monitoring

Cron Jobs for Management

Package update, testing and control process

Prior to making a release, the update needs to be developed, tested (which may including integration testing), reviewed and approved. This process is summarized below, with further details to be added.

Package developer process

  1. Develop PR(s), make sure tests are passing on local development platform
  2. Get code review
  3. Do unit and functional tests within git repo on supported platforms, specifically:
    1. Mac High Sierra using ska3-flight (or possibly ska3/test if other package dependencies are involved).
    2. HEAD linux-64 using ska3-flight (or possibly ska3/test)
    3. GRETA linux-32 using ska3-matlab. This should be done using /proj/sot/ska3/matlab32/test, which represents either the current MATLAB environment or the pending one for the next promotion.
  4. If warranted do system integration testing, as shown in the example for xija:
    1. Create a test env and install the package using pip: 
      conda create -n test-ska ska3-flight
source activate test-ska
conda uninstall --force xija
cd ~/git/xija
pip install --no-deps --verbose --no-binary :all: --no-index .  # add --egg for namespace package
    2. Run ska_testr
    3. Integration functional tests as needed
  5. In rare cases confirm the conda build process by making an RC tag like 3.4.2rc1 on the branch, then build local conda package, conda install to a test Ska and repeat testing as needed.
  6. Seek approval to install from controlling authority (if needed)
  7. Merge to master
  8. If necessary, do a quick PR to update the package version number. NOTE: this should never be needed. Projects should be using setuptools_scm.
  9. Open an issue in skare3 indicating that the package has changes that should go in an upcoming release.
    1. Tag with Package Update
    2. If this is targeted for a release milestone then tag as such.
    3. If the package update is considered complete and ready for a release, then add a new release using the GitHub Releases => Draft a new release buttons.
    4. Update the issue to indicate the PRs that will be in the anticipated release, or link to a release tag with the same info, and other text that will go into the release notes for the package when released. If the change was low priority, the issue may be revisited to "collect" subsequent PRs for a larger release. Use (replace 'tail -r' with 'tac' on Linux): 
      git log <last_release_tag>...HEAD --merges --format="%s ::: %b"   \
  | sed "s/Merge pull request //"   \
  | sed 's/from .* :://'   | tail -r

Ska3 conda build maintainer process

  1. Review the packages with merged PRs and/or new releases in the skare3 dashboard . Password available here
  2. Review the skare3 issues with the Package Update tag and coordinate with developers to see which are ready for release.
  3. If the package updates will require recipe changes, do those recipe updates first in new PRs, approve, merge to skare3 master.
  4. Build conda packages for the individual packages from the skare3 repository at 'master'
  5. Install conda individual packages to flight conda repository
  6. If needed, install packages to a test environment for conda integration testing
  7. Make PRs against skare3 metapackages as needed to (ska3-flight, ska3-core, ska3-matlab) to support the release
  8. Review PRs internally as needed.
  9. Merge PRs
  10. Build ska3-flight, ska3-core, or ska3-matlab metapackages
  11. Install ska3-flight or ska3-matlab packages to the 'test' channel repository
    1. This process is to copy the package into the test channel and update the conda index.
  12. Install any update of ska3-core metapackage to flight repository
  13. Make a tag or release against skare3 master corresponding to a ska3-flight or ska3-matlab release
  14. Get approval on the ska3-flight skare3 update(s) at LR board or controlling authority.
  15. Install updated ska3-flight package to flight ska3conda repository.
    1. This process is to copy the package into the flight channel and update the conda index.
  16. Install the updated package into the production environments.
  17. Sanity check testing, e.g. run package unit tests from IPython. If this fails then start over and issue a bug fix release. This will most likely have revealed some process error that needs to be corrected or prevented in the future.
  18. Notify community as needed

NOTE: Regarding PRs that contain edits to meta.yaml for ska3-flight, those can still be merged before load review approval, but be aware one cannot blindly rsync ska3-flight built metapackages into the flight package repository, and that the ska3-flight metapackage should only be installed in the flight/ska3-conda repository after approval.

Creating and maintaining conda package builds

GitHub process

The next step in building a package is to have that package available on GitHub with the desired release tagged with a version specifier like 3.5.2 or 3.5.2rc1 (for testing). If there has been an update, this must be approved by the relevant controlling authority, merged to master, and tagged on GitHub.

Metapackages

The skare3 build system also includes several recipes for conda metapackages. The ska3-flight metapackage is one such package. If an individual Python module package update being reviewed or is ready to be installed in the flight environment, say on the annie package, for example, then there is also a required step to make and receive approval for a pull request to change the meta.yaml file in the ska3-flight recipe within skare3 to reflect the new annie version and to increment the version of the ska3-flight metapackage itself. After that PR is merged, one would need to use the ska3_builder process (below) build both the annie conda package and the ska3-flight conda package.

An update to a core package in the ska3-core metapackage requires an increment of the version of ska3-core, and update to the ska3-core entry in ska3-flight, an increment in the ska3-flight metapackage version, and a rebuild of both the ska3-core and ska3-flight metapackages.

Setting up build environments

Linux

To support CentOS5, packages should be built from a CentOS5 machine (though for noarch packages it should really make no difference) and the user running the scripts will need a CentOS5-compatible .condarc. First, edit your user ~/.condarc to include:

binstar_upload: false
channels:
  - https://icxc.harvard.edu/aspect/ska3-conda/
  - https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
#  - defaults
#default_channels:
#  - https://repo.continuum.io/pkgs/free
#  - https://repo.continuum.io/pkgs/r
#  - https://repo.continuum.io/pkgs/msys2

(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)

This is already configured for the 'aca' user. Then fetch a miniconda installer still compatible with CentOS5 and install

chmod o-rwx ~/.condarc
curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh
bash Miniconda3-4.3.21-Linux-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # Update Python but keep conda at 4.3.21
conda install ska3_builder  # This package must be installed in a "root" conda environment

For the aca user this install has been done with the ${miniconda_install_prefix} set to /export/aca/ska3 on presto .

OSX

For OSX we could use newer packages in ska3-dev in the modern channels, but by default, we are using the same .condarc used in Linux. This prevents getting carried away with features that we can't actually use on Linux production systems. So, the .condarc is set as such on the skare3 osx build system, zero. (zero is presently not available so this section of the directions is not up-to-date)

binstar_upload: false
channels:
  - https://icxc.harvard.edu/aspect/ska3-conda/
  - https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
#  - defaults
#default_channels:
#  - https://repo.continuum.io/pkgs/free
#  - https://repo.continuum.io/pkgs/r
#  - https://repo.continuum.io/pkgs/msys2

(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)

On zero, the aca user also has a .bash_profile that has the miniconda environment already loaded, with the miniconda_install_prefix set to '/Users/aca/miniconda3/bin/conda'. If this needs to be redone, these would be the steps.

curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh
bash Miniconda3-4.3.21-MacOSX-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # update Python but keep conda at 4.3.21
conda install ska3_builder  # This package must be installed in a "root" conda environment

Building locally

The underlying packages that comprise the ska3 environment are built using the ska_builder.py script, which is available via the skare3 git repository with the setup described above. Several of the packages are private github packages, and for these to clone and build locally you will need working github ssh authentication (this is already set up for the aca user). If you haven't set this up, see https://help.github.com/articles/connecting-to-github-with-ssh/ . You may also want to manage your key with ssh-agent before building packages. See https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#adding-your-ssh-key-to-the-ssh-agent

After ssh is set up get into the appropriate conda environment for building (if not already set from above steps). For builds as aca, that is currently

export PATH=/export/aca/ska3/bin:$PATH

as the aca user on presto (also set as ska3build alias if working in .cshrc)

For 32 bit builds for matlab build as the aca user on quango and set path

export PATH=/export/aca/miniconda3/bin/:$PATH

For osx use the aca user on the HEAD system 'zero', and the PATH is already set.

Then set the build root

export build_root=/export/aca/ska3-conda-builds  # As ACA for production on presto
 OR
export build_root=/export/aca/build_root # As ACA user for 32 bit builds on quango
OR
export build_root=~/ska3-conda-builds  # In user home dir

Then get in to the skare3 repo and do the build. Clone if needed

cd ~/git
git clone git@github.com:sot/skare3

Check out the skare3 packages at the release tag intended for the ska3-flight or ska3-matlab release.

cd ~/git/skare3
python ska_builder.py --build-root=${build_root} <package>

By default the latest tag (sorted by commit date, not tag date) is built. To build a specific tag, specify that with the --tag=<tag> option.

In order to build all available packages at once, just omit the <package>:

cd ~/git/skare3
python ska_builder.py --build-root=${build_root}

This will clone each repo if needed from GitHub or fetch updates, check out at the desired tag, and build the package using the conda build recipe.

The cloned repos are in ${build_root}/srcs and the builds are in ${build_root}/builds.

You can then install the new built package into a non-production environment using:

conda install -c file://${build_root}/builds <package>

Assuming you have built everything, then you can use the meta-package names shown earlier like ska3-flight.

Updating the ska3-conda channel

The ska3-conda channel is a non-public channel that is served on the icxc web server and can be accessed via a typical conda channel URL spec.

The channel URL is https://icxc.cfa.harvard.edu/aspect/ska3-conda , corresponding to a file directory of /proj/sot/ska/www/ASPECT_ICXC/ska3-conda.

Linux

For production linux builds, do the following (still as user aca on presto):

# Get into ska3_builder environment if not already there

rsync -ruv ${build_root}/builds/linux-64/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/

For the test channel one would use (note that these are on the public web and not ICXC, they are rsync'd to ICXC from public web):

rsync -ruv ${build_root}/builds/linux-64/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/

MacOSX

For production MacOSX builds, after building the packages as the aca user on the zero system, sync back to the HEAD LAN as as the aca user on the zero system:

ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/

For the test channel one would use (note that these are on public web. A cronjob syncs to ICXC)

ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/

Sherpa package (linux and MacOSX)

The sherpa conda packages are copied in to the ska3-conda channel and served as if they were ska packages. The 4.9.1 Linux version was fetched from https://anaconda.org/CXC/sherpa/files and the 4.9.1 Mac version was fetched from https://anaconda.org/sherpa/sherpa/files. The channel indexes were updated appropriately. We're using the pinned and manually fetched build of 4.9.1 from Linux dev channel to support a custom build from the Sherpa team to support CentOS5. At some point in the future when CentOS5 support is deprecated in skare3, we could update via adding the sherpa channel to the config.

cd /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64
curl -L -O https://anaconda.org/CXC/sherpa/4.9.1/download/linux-64/sherpa-4.9.1-py36he678ed3_0.tar.bz2
cd ..
conda index linux-64
cd osx-64
curl -L -O https://anaconda.org/sherpa/sherpa/4.9.1/download/osx-64/sherpa-4.9.1-py36_0.tar.bz2
cd ..
conda index osx-64

Updating the core conda packages

(still needs work)

The ska3-core package should be updated once or twice a year to prevent core conda packages from getting too stale. The process is to

  1. review the ska3-pinned metapackage

This package holds back packages when newer versions of those packages will break compatibility across our platforms. At the time of the initial release of skare3, we are holding back packages to continue to support CentOS5 and those packages are called out in ska3-pinned. For a core update, the pinned list in the ska3-pinned metapackage should be reviewed, and if any of the restrictions no longer apply, the package should be edited and rebuilt.

  1. install the ska3-dev and ska3-pinned package in an env

On each supported platform, the ska3-dev and ska3-pinned packages should be installed in a fresh conda env.

conda create -n core-update ska3-dev ska3-pinned

This will create a new environment where the packages will have floated to versions as high as allowed by the restrictions in ska3-pinned.

  1. save new platform-specific package list

Where {platform} is replaced with a string to record which platform or machine was used to make the file, do:

source activate core-update
conda list --json > {platform}.json

Then copy the json files so they are in the same directory on a machine with access to Python and a script in the skare3 repo.

  1. generate new combined-arch package list

There is a defunct script, ska_conda/pkg_defs/ska3-core/combine_arch.py, that was used to make the first combined package list when the operating systems were just 'linux' and 'osx'. That script will need rewriting if one desires to programmatically update the ska3-core pkg list from multiple operating systems including linux-32, linux-64, os-x, and win. In the interim, the strategy is to review the individual package listings from the multiple machines (the {platform}.json files or just the output of conda list) and update ska3-core meta.yaml manually.

  1. rebuild and test

Commit the ska3-core changes. Update ska3-flight with the new ska3-core version in the run requirements. Tag the changes in skare3. Then, rebuild ska3_builder, ska3-core, and ska3-flight. Install the updated ska3-flight to a test area and test. If all tests look good put updated packages in the web-accessible repository.

Updating production and test environments

First, notify the other maintainers that you are about to do the actual install (with email to aca@cfa).

HEAD

ska3-flight is installed on HEAD at

/proj/sot/ska3/flight

Login as aca user on presto, then (assuming bash)

source /proj/sot/ska3/flight/bin/ska_envs.sh
conda list ska3-flight  # check version
conda update ska3-flight  # confirm package update looks right

GRETA

Coordinate with FOT-matlab before proceeding.

ska3-flight and ska3-matlab environments GRETA follow similar naming convention:

/proj/sot/ska3/flight
/proj/sot/ska3/matlab/flight
/proj/sot/ska3/matlab/test
/proj/sot/ska3/matlab32/flight
/proj/sot/ska3/matlab32/test

To install ska3-matlab, login as SOT user on chimchim and cheru and do the following (assuming bash):

source /proj/sot/ska3/matlab/flight/bin/ska_envs.sh
conda list ska3-matlab  # check version
conda update ska3-matlab  # confirm package update looks right

Same thing with flight, mutatis mutandi.

For ska3-matlab 32 bit, logged in as SOT user on chimchim (assuming bash):

source /proj/sot/ska3/matlab32/flight/bin/ska_envs.sh
conda update ska3-matlab

A special case is the 32 bit environment on cheru. Logged in as SOT user on cheru:

rsync -av --dry-run --delete chimchim:/proj/sot/ska3/matlab32/flight/ /proj/sot/ska3/matlab32/flight/

Finally:

  • notify chandracode about the update
  • for a ska3-flight update, also notify mpweekly about the update.

Setting up a New Production Environment

If reusing an previously-used environment directory, just move the old content.

Then, on GRETA first follow the steps in Syncing conda builds to GRETA. Then follow the directions above to install a miniconda environment, except the miniconda directory will be at the production directory location / prefix, i.e. "/proj/sot/ska3/test" or "/proj/sot/ska3/flight". Then install ska3-flight and ska3_builder into the "root" conda environment with the following as the aca user (set the PATH to get into that conda environment if not already set):

% conda install ska3-flight ska3_builder task_schedule watch_cron_logs

(It is useful to have the ska3_builder environment to guarantee the right version of the conda environment and to have access to "conda index". task_schedule and watch_cron_logs should probably move to a metapackage at some point.)

As a non-privileged user, you can use the environment by sourcing into it:

% source ${prefix}/bin/ska_envs.sh
Add a custom footer

For Users

For Maintainers

Continuous Integration

Old Pages

Clone this wiki locally