Skip to content

Ska3 runtime environment for maintainers

Jump to bottom Edit New page
Javier Gonzalez edited this page Aug 14, 2023 · 108 revisions

Ska3 packages are distributed using conda through several private conda channels that can be found here. The packages are created and tested using Github workflows and a mix of Github-hosted and self-hosted runners. This document outlines the process for building and maintaining Ska3 environments and the associated packages.

Contents

Package update, testing and control process

Prior to making a release, the update needs to be developed, tested (which may including integration testing), reviewed and approved. This process is summarized below.

Package Setup

All Ska3 packages should:

Package developer process

Once a package is properly setup, development is as follows:

  1. Develop PR(s), make sure tests are passing on local development platform
  2. Get code review on Github
  3. Do unit and functional tests within git repo on supported platforms, specifically:
    1. Mac High Sierra using ska3-flight (or possibly ska3/test if other package dependencies are involved).
    2. HEAD linux-64 using ska3-flight (or possibly ska3/test)
    3. GRETA linux-64 using ska3-matlab. This should be done using /proj/sot/ska3/matlab32/test, which represents either the current MATLAB environment or the pending one for the next promotion.
  4. If warranted do system integration testing, as shown in the example for xija:
    1. Create a test env and install the package using pip: 
      conda create -n test-ska ska3-flight
source activate test-ska
conda uninstall --force xija
cd ~/git/xija
pip install --no-deps --verbose --no-binary :all: --no-index .  # add --egg for namespace package
    2. Run ska_testr
    3. Integration functional tests as needed
  5. Seek approval from controlling authority (if needed)
  6. Merge to master. At this point, a conda package is built and copied into the masters conda channel, with a version matching the sha of the merge commit.
  7. Every night, a set of regression tests is run using the latest package from master. The results are displayed in the SkaRE3 dashboard
  8. When ready, make a release on Github. A Github release should trigger the following automated responses:
    • the build workflow builds a conda package with the new version
    • the documentation workflow builds the HTML documentation and posts it so it appears in [sot.github.io]
    • the release workflow does the following:
      • opens an issue in sot/skare3 (with the Package Update tag) to remind the Ska3 maintainer that this package should go in an upcoming release.
      • edits the release description to list all PRs since last release.
  9. Some times things fail. In these cases, the Ska3 maintainer can do it manually:
    • If the package didn't build, it will have to either be built manually or using an event dispatch (details below).

Ska3 release process

The Ska3 release process is summarized in the ska3-flight and ska3-matlab issue templates. The following list gives a bit more detail, but it might be outdated in comparison with the issue templates.

Preparation:

  1. Create a milestone for the release (Usually called the same as the release, the milestone name is used below)
  2. Create an issue on sot/skare3 to keep track of the release process and assign it to the milestone. There are templates for releasing ska3-flight and ska3-matlab which include a checklist like the following.
  3. Review the packages with merged PRs and/or new releases in the SkaRE3 dashboard . Password available here
  4. Review the skare3 issues with the Package Update tag and coordinate with developers to see which are ready for release. Assign milestones to the issues.
  5. If the package updates will require recipe changes, do those recipe updates first in new PRs, approve, merge to skare3 master.

Package and Test:

  1. Create a branch named {version}-branch. The name is important, because otherwise the build will not succeed.
  2. Create a PR titled {version} from {version}-branch into master. The name is important, because otherwise the build will not succeed. Take a look at the https://github.com/sot/skare3/blob/master/.github/PULL_REQUEST_TEMPLATE/ska3-flight.md and https://github.com/sot/skare3/blob/master/.github/PULL_REQUEST_TEMPLATE/ska3-matlab.md pull request templates
  3. Make all needed changes to meta.yml.
  4. Create a pre-release {version}rc{N} at the latest commit in the branch. This triggers the automated build/test workflow.
  5. Check that all packages are built and automated tests pass. This can be done in the actions section.
  6. Promote packages from masters to test channel. 
    skare3-promote --to test --ska3-conda www/ASPECT/ska3-conda ska3-flight=={version}rc{N} [--dry-run]
  7. Install on HEAD and GRETA Linux test environments (ska3/test).
  8. Run testr on HEAD and GRETA
  9. Add related issues to PR 
    skare3-milestone-issues --repository sot/skare3 --milestone {release}
  10. Document all changes 
    skare3-changes-summary --conda-channel test --meta-package ska3-flight --final-version {version}
    NOTE: for this command, you will need a Github token passed in the command line or as the GITHUB_TOKEN environment variable.
  11. Write summary and highlight relevant changes.
  12. Document test status 
    skare3-test-dashboard -i /path/to/test/results
  13. Review PR internally.
  14. Request and wait for approval.

Promotion:

  1. Promote packages to main conda channel. 
    skare3-promote --to flight --ska3-conda www/ASPECT_ICXC/ska3-conda ska3-flight=={version} [--dry-run]
  2. Notify ACA.
  3. Install ska3/flight on HEAD as aca user details.
  4. Install ska3/flight on GRETA as SOT user details.
  5. Run ska_testr on HEAD and GRETA.
  6. Document test results. 
    skare3-test-dashboard -i /path/to/test/results
  7. Notify ACA and the community.
  8. Merge PR

Cron Jobs

Cron Jobs for Monitoring

Cron Jobs for Management

Production and test environments

Updating an environment

Notify the other maintainers that you are about to do the actual install (with email to aca@cfa).

HEAD

ska3-flight is installed on HEAD at

/proj/sot/ska3/flight

Login as aca user on kady, then (assuming bash)

source /proj/sot/ska3/flight/bin/ska_envs.sh
conda list ska3-flight  # check version
conda update ska3-flight  # confirm package update looks right

GRETA

Coordinate with FOT before proceeding.

ska3-flight and ska3-matlab environments GRETA follow similar naming convention:

/proj/sot/ska3/flight
/proj/sot/ska3/test
/proj/sot/ska3/matlab/flight
/proj/sot/ska3/matlab/test

To install ska3-matlab, login as SOT user on chimchim or cheru and do the following (assuming bash):

source /proj/sot/ska3/matlab/flight/bin/ska_envs.sh
conda list ska3-matlab  # check version
conda update ska3-matlab  # confirm package update looks right

Same thing with flight, mutatis mutandi.

Finally:

  • for a ska3-matlab update: notify FOT.
  • for a ska3-flight update: notify sot@cfa and fot@ipa.

Setting up a new environment

All this should be done as the aca/SOT user as in the last section.

  • If reusing a previously-used environment directory, just move the old content.

  • Our production environments are setup at the root of a miniconda installation. For flight and test environments, install a miniconda environment at the destination directory (i.e. "/proj/sot/ska3/test" or "/proj/sot/ska3/flight"):

    curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh
bash Miniconda3-py39_22.11.1-1-Linux-x86_64.sh -b -p ${prefix}

  • Create a condarc file in the environment directory. This file includes the channels where packages are fetched. The flight environment has the following condarc:

    channels:
  - https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight

    For the test environment, it will also include the test channel. This setup should also work on GRETA as long as icxc is accessible.

  • Install ska3-flight (and optionally ska3-perl):

    % conda install ska3-flight ska3-perl

  • Create a link to ska/data

    % ln -s /proj/sot/ska/data ${prefix}/data

As a non-privileged user, you can use the environment by sourcing into it:

% source ${prefix}/bin/ska_envs.sh

Creating the ska3-masters environment

The ska3-masters is a bit different because the environment lives alongside other environments created by continuous integration. It cannot be at the root of a miniconda installation. Creating this environment is similar to the previous section, except:

  • After installing miniconda, create the environment and activate it:

    conda create -y -n ska3-masters
conda activate ska3-masters

  • the condarc file should include the test and masters channels.

  • Install the "latest" versions of the ska3 meta-packages:

    % conda install ska3-core-latest ska3-flight-latest ska3-perl-latest

Creating the trunk environment on GRETA

These are the steps used to create the trunk environment the last time.

As user sot on cheru:

bash Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -b -p  /proj/sot/ska3/matlab/trunk
eval "$(/proj/sot/ska3/matlab/trunk/bin/conda shell.bash hook)"

Prepare a /proj/sot/ska3/matlab/trunk/condarc with the following content:

channels:
  - https://ska:ska-cxc-20y@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/masters
  - https://ska:ska-cxc-20y@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/flight
  - https://ska:ska-cxc-20y@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/test

Install:

conda install mamba
mamba install ska3-core==2023.1 ska3-flight-latest==2023.3
ln -s /proj/sot/ska/data/* /proj/sot/ska3/matlab/trunk/data

To update:

conda clean --all --packages --force-pkgs-dir
mamba update --all

Creating and maintaining Conda packages

Useful Resources

Ska3 Packages

All Ska3 packages are developed around Github, and distributed as conda packages. There are two kinds of conda packages in Ska3:

  • Meta-packages are just empty packages with a list of dependencies. These are useful to manage collections of packages in a coherent way. These packages are hosted in sot/skare3. Their names all start with ska3- and their versions follow a{year}.{integer}[rc{integer}] format, with version numbers like 2022.1 for releases and 2022.1rc1 for pre-releases. Examples of these meta-packages include ska3-flight, ska3-matlab, ska3-core and ska3-perl.
  • Regular packages are hosted on their own Github repositories and their build recipe is hosted in sot/skare3. Their versions follow the PEP 440 specification, which is handled by setuptools_scm. They usually have the following workflows:
    • build (.github/workflows/build.yml) calls the sot/skare3/package_build.yml workflow, which assembles a conda package and uploads it to the Ska3 conda repository.
    • release (.github/workflows/release.yml) calls the sot/skare3/package_release.yml workflow which edits the release description and performs other menial tasks associated with the release.
    • documentation (.github/workflows/docs.yml) calls the sot/skare3/package_docs.yml workflow, which builds the package's sphinx documentation and uploads it so it appears in the Ska3 documentation.

Editing build recipes

Recipes for regular packages are usually modified on a branch, reviewed internally and merged into master without requiring review from out of the Ska development team. Changes to package's build recipe's are very rare.

Edits to the recipes for meta-packages require a version number increase, and are always done on a branch as part of the release process outlined above: edits are made on a branch, the PR is internally reviewed, then an external authority approves. After approval, a release is made, the PR is merged and the package is installed in production environments.

An update to a core package in the ska3-core meta-package requires an increment of the version of ska3-core, and update to the ska3-core entry in ska3-flight, an increment in the ska3-flight meta-package version, and a rebuild of both the ska3-core and ska3-flight meta-packages.

NOTE: a typical issue that arises with this versioning scheme is that version 2022.10 can be interpreted as a float by yaml and converted to "2022.1", so in this particular case the quotation marks are required: "2022.10"

Building conda packages

Every step in this section is usually done automatically as part of the package build. However, sometimes the process fails, in which case it is important to be able to do it manually (hence this section).

Building with repository dispatch

Building manually can be tedious because we might need to build several packages for the different platforms. When something goes wrong in the automated build, it is very often easier to fix the workflow and run it again.

The build workflow can be triggered directly using the Github API. One way to do it is to use skare3_tools:

from skare3_tools import github
r = github.Repository('sot/skare3')
r.dispatch_event(
    event_type='build-package',
    client_payload={
        'package': 'sot/xija',
        'tag': '4.29.1',
        'arch': True,           # this is not a noarch package
        'upload': True,         # upload to Ska3 conda repo
    }
)

Setting up the build environment

Right now this section is outdated. Better check the workflow itself to see how this is done.

Linux

To support CentOS5, packages should be built from a CentOS5 machine (though for noarch packages it should really make no difference) and the user running the scripts will need a CentOS5-compatible .condarc. First, edit your user ~/.condarc to include:

binstar_upload: false
channels:
  - https://icxc.harvard.edu/aspect/ska3-conda/
  - https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
#  - defaults
#default_channels:
#  - https://repo.continuum.io/pkgs/free
#  - https://repo.continuum.io/pkgs/r
#  - https://repo.continuum.io/pkgs/msys2

(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)

This is already configured for the 'aca' user. Then fetch a miniconda installer still compatible with CentOS5 and install

chmod o-rwx ~/.condarc
curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh
bash Miniconda3-4.3.21-Linux-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # Update Python but keep conda at 4.3.21
conda install ska3_builder  # This package must be installed in a "root" conda environment

For the aca user this install has been done with the ${miniconda_install_prefix} set to /export/aca/ska3 on presto .

OSX

For OSX we could use newer packages in ska3-dev in the modern channels, but by default, we are using the same .condarc used in Linux. This prevents getting carried away with features that we can't actually use on Linux production systems. So, the .condarc is set as such on the skare3 osx build system, zero. (zero is presently not available so this section of the directions is not up-to-date)

binstar_upload: false
channels:
  - https://icxc.harvard.edu/aspect/ska3-conda/
  - https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
#  - defaults
#default_channels:
#  - https://repo.continuum.io/pkgs/free
#  - https://repo.continuum.io/pkgs/r
#  - https://repo.continuum.io/pkgs/msys2

(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)

On zero, the aca user also has a .bash_profile that has the miniconda environment already loaded, with the miniconda_install_prefix set to '/Users/aca/miniconda3/bin/conda'. If this needs to be redone, these would be the steps.

curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh
bash Miniconda3-4.3.21-MacOSX-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # update Python but keep conda at 4.3.21
conda install ska3_builder  # This package must be installed in a "root" conda environment

Building locally

The underlying packages that comprise the ska3 environment are built using the ska_builder.py script, which is available via the skare3 git repository with the setup described above. Several of the packages are private github packages, and for these to clone and build locally you will need working github ssh authentication (this is already set up for the aca user). If you haven't set this up, see https://help.github.com/articles/connecting-to-github-with-ssh/ . You may also want to manage your key with ssh-agent before building packages. See https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#adding-your-ssh-key-to-the-ssh-agent

After ssh is set up get into the appropriate conda environment for building (if not already set from above steps). For builds as aca, that is currently

export PATH=/export/aca/ska3/bin:$PATH

as the aca user on presto (also set as ska3build alias if working in .cshrc)

For 32 bit builds for matlab build as the aca user on quango and set path

export PATH=/export/aca/miniconda3/bin/:$PATH

For osx use the aca user on the HEAD system 'zero', and the PATH is already set.

Then set the build root

export build_root=/export/aca/ska3-conda-builds  # As ACA for production on presto
 OR
export build_root=/export/aca/build_root # As ACA user for 32 bit builds on quango
OR
export build_root=~/ska3-conda-builds  # In user home dir

Then get in to the skare3 repo and do the build. Clone if needed

cd ~/git
git clone git@github.com:sot/skare3

Check out the skare3 packages at the release tag intended for the ska3-flight or ska3-matlab release.

cd ~/git/skare3
python ska_builder.py --build-root=${build_root} <package>

By default the latest tag (sorted by commit date, not tag date) is built. To build a specific tag, specify that with the --tag=<tag> option.

In order to build all available packages at once, just omit the <package>:

cd ~/git/skare3
python ska_builder.py --build-root=${build_root}

This will clone each repo if needed from GitHub or fetch updates, check out at the desired tag, and build the package using the conda build recipe.

The cloned repos are in ${build_root}/srcs and the builds are in ${build_root}/builds.

You can then install the new built package into a non-production environment using:

conda install -c file://${build_root}/builds <package>

Assuming you have built everything, then you can use the meta-package names shown earlier like ska3-flight.

Updating the ska3-conda channel

The ska3-conda channel is a non-public channel that is served on the icxc web server and can be accessed via a typical conda channel URL spec.

The channel URL is https://icxc.cfa.harvard.edu/aspect/ska3-conda , corresponding to a file directory of /proj/sot/ska/www/ASPECT_ICXC/ska3-conda.

Linux

For production linux builds, do the following (still as user aca on presto):

# Get into ska3_builder environment if not already there

rsync -ruv ${build_root}/builds/linux-64/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/

For the test channel one would use (note that these are on the public web and not ICXC, they are rsync'd to ICXC from public web):

rsync -ruv ${build_root}/builds/linux-64/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/

MacOSX

For production MacOSX builds, after building the packages as the aca user on the zero system, sync back to the HEAD LAN as as the aca user on the zero system:

ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
          /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/

For the test channel one would use (note that these are on public web. A cronjob syncs to ICXC)

ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
          /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/

Updating the core conda packages

The ska3-core package should be updated once or twice a year to prevent core conda packages from getting too stale. The process update ska3-core/meta.yml is described in Updating the Core Conda Packages

Add a custom footer

For Users

For Maintainers

Continuous Integration

Old Pages

Clone this wiki locally