-
Notifications
You must be signed in to change notification settings - Fork 4
Ska3 runtime environment for maintainers
Ska3 packages are distributed using conda through several private conda channels that can be found here. The packages are created and tested using Github workflows and a mix of Github-hosted and self-hosted runners. This document outlines the process for building and maintaining Ska3 environments and the associated packages.
- Package update, testing and control process
- Cron Jobs
- Production and test environments
- Creating and maintaining Conda packages
Prior to making a release, the update needs to be developed, tested (which may including integration testing), reviewed and approved. This process is summarized below.
All Ska3 packages should:
- Have a build recipe in sot/skare3
- Use setuptools_scm. This automatically sets version numbers based on git tags.
- Have the following workflows:
-
build (
.github/workflows/build.yml
) calls the sot/skare3/package_build.yml workflow. -
release (
.github/workflows/release.yml
) calls the sot/skare3/package_release.yml workflow. -
documentation (
.github/workflows/docs.yml
) calls the sot/skare3/package_docs.yml workflow.
-
build (
Once a package is properly setup, development is as follows:
- Develop PR(s), make sure tests are passing on local development platform
- Get code review on Github
-
Do unit and functional tests within git repo on supported platforms, specifically:
- Mac High Sierra using ska3-flight (or possibly
ska3/test
if other package dependencies are involved). - HEAD linux-64 using ska3-flight (or possibly
ska3/test
) - GRETA linux-64 using ska3-matlab. This should be done using
/proj/sot/ska3/matlab32/test
, which represents either the current MATLAB environment or the pending one for the next promotion.
- Mac High Sierra using ska3-flight (or possibly
- If warranted do system integration testing, as shown in the example for
xija
:- Create a test env and install the package using pip:
conda create -n test-ska ska3-flight source activate test-ska conda uninstall --force xija cd ~/git/xija pip install --no-deps --verbose --no-binary :all: --no-index . # add --egg for namespace package
- Run ska_testr
- Integration functional tests as needed
- Create a test env and install the package using pip:
- Seek approval from controlling authority (if needed)
-
Merge to master. At this point, a conda package is built and copied into the
masters
conda channel, with a version matching the sha of the merge commit. - Every night, a set of regression tests is run using the latest package from master. The results are displayed in the SkaRE3 dashboard
- When ready, make a release on Github. A Github release should trigger the following automated responses:
- the build workflow builds a conda package with the new version
- the documentation workflow builds the HTML documentation and posts it so it appears in [sot.github.io]
- the release workflow does the following:
- opens an issue in sot/skare3 (with the
Package Update
tag) to remind the Ska3 maintainer that this package should go in an upcoming release. - edits the release description to list all PRs since last release.
- opens an issue in sot/skare3 (with the
- Some times things fail. In these cases, the Ska3 maintainer can do it manually:
- If the package didn't build, it will have to either be built manually or using an event dispatch (details below).
The Ska3 release process is summarized in the ska3-flight and ska3-matlab issue templates. The following list gives a bit more detail, but it might be outdated in comparison with the issue templates.
- Create a milestone for the release (Usually called the same as the release, the milestone name is used below)
- Create an issue on sot/skare3 to keep track of the release process and assign it to the milestone. There are templates for releasing ska3-flight and ska3-matlab which include a checklist like the following.
- Review the packages with merged PRs and/or new releases in the SkaRE3 dashboard . Password available here
- Review the skare3 issues with the
Package Update
tag and coordinate with developers to see which are ready for release. Assign milestones to the issues. - If the package updates will require recipe changes, do those recipe updates first in new PRs, approve, merge to skare3 master.
- Create a branch named {version}-branch. The name is important, because otherwise the build will not succeed.
- Create a PR titled {version} from {version}-branch into master. The name is important, because otherwise the build will not succeed. Take a look at the https://github.com/sot/skare3/blob/master/.github/PULL_REQUEST_TEMPLATE/ska3-flight.md and https://github.com/sot/skare3/blob/master/.github/PULL_REQUEST_TEMPLATE/ska3-matlab.md pull request templates
- Make all needed changes to
meta.yml
. -
Create a pre-release
{version}rc{N}
at the latest commit in the branch. This triggers the automated build/test workflow. - Check that all packages are built and automated tests pass. This can be done in the actions section.
-
Promote packages from
masters
totest
channel.skare3-promote --to test --ska3-conda www/ASPECT/ska3-conda ska3-flight=={version}rc{N} [--dry-run]
-
Install on HEAD and GRETA Linux test environments (
ska3/test
). - Run testr on HEAD and GRETA
- Add related issues to PR
skare3-milestone-issues --repository sot/skare3 --milestone {release}
-
Document all changes
NOTE: for this command, you will need a Github token passed in the command line or as the
skare3-changes-summary --conda-channel test --meta-package ska3-flight --final-version {version}
GITHUB_TOKEN
environment variable. - Write summary and highlight relevant changes.
- Document test status
skare3-test-dashboard -i /path/to/test/results
- Review PR internally.
- Request and wait for approval.
-
Promote packages to main conda channel.
skare3-promote --to flight --ska3-conda www/ASPECT_ICXC/ska3-conda ska3-flight=={version} [--dry-run]
- Notify ACA.
- Install ska3/flight on HEAD as aca user details.
- Install ska3/flight on GRETA as SOT user details.
- Run ska_testr on HEAD and GRETA.
- Document test results.
skare3-test-dashboard -i /path/to/test/results
- Notify ACA and the community.
- Merge PR
Cron Jobs for Monitoring
- Ska3 masters testing (the "Regression tests" as Javier).
- Output to Dashboard tests
- Ska3 dashboard (the Update dashboard cron Job)
- Output to Dashboard
- Ska3 flight automated nightly run (the "Regression tests" as jeanconn). Sends email to Jean.
Cron Jobs for Management
- Github Runner see "github-runner-service". Runs not-on-github parts of the github workflows.
- Sync private/public conda channels see "Sync private/public conda channels"
Notify the other maintainers that you are about to do the actual install (with email to aca@cfa).
ska3-flight is installed on HEAD at
/proj/sot/ska3/flight
Login as aca user on kady, then (assuming bash)
source /proj/sot/ska3/flight/bin/ska_envs.sh
conda list ska3-flight # check version
conda update ska3-flight # confirm package update looks right
Coordinate with FOT before proceeding.
ska3-flight and ska3-matlab environments GRETA follow similar naming convention:
/proj/sot/ska3/flight
/proj/sot/ska3/test
/proj/sot/ska3/matlab/flight
/proj/sot/ska3/matlab/test
To install ska3-matlab, login as SOT user on chimchim or cheru and do the following (assuming bash):
source /proj/sot/ska3/matlab/flight/bin/ska_envs.sh
conda list ska3-matlab # check version
conda update ska3-matlab # confirm package update looks right
Same thing with flight, mutatis mutandi.
Finally:
- for a ska3-matlab update: notify FOT.
- for a ska3-flight update: notify sot@cfa and fot@ipa.
All this should be done as the aca
/SOT
user as in the last section.
-
If reusing a previously-used environment directory, just move the old content.
-
Our production environments are setup at the root of a miniconda installation. For flight and test environments, install a miniconda environment at the destination directory (i.e. "/proj/sot/ska3/test" or "/proj/sot/ska3/flight"):
curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh bash Miniconda3-py39_22.11.1-1-Linux-x86_64.sh -b -p ${prefix}
-
Create a
condarc
file in the environment directory. This file includes the channels where packages are fetched. The flight environment has the followingcondarc
:channels: - https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight
For the test environment, it will also include the
test
channel. This setup should also work on GRETA as long as icxc is accessible. A way to create this conda rc:conda config --env --add channels https://icxc.cfa.harvard.edu/aspect/ska3-conda/flight conda config --env --remove channels defaults
-
Install ska3-flight (and optionally ska3-perl):
% conda install ska3-flight ska3-perl
-
Create a link to ska/data
% ln -s /proj/sot/ska/data ${prefix}/data
As a non-privileged user, you can use the environment by sourcing into it:
% source ${prefix}/bin/ska_envs.sh
The ska3-masters
is a bit different because the environment lives alongside other environments created by continuous integration. It cannot be at the root of a miniconda installation. Creating this environment is similar to the previous section, except:
-
After installing miniconda, create the environment and activate it:
conda create -y -n ska3-masters conda activate ska3-masters
-
the
condarc
file should include the test and masters channels. -
Install the "latest" versions of the ska3 meta-packages:
% conda install ska3-core-latest ska3-flight-latest ska3-perl-latest
These are the steps used to create the trunk environment the last time.
As user sot
on cheru
:
bash Miniconda3-py310_23.1.0-1-Linux-x86_64.sh -b -p /proj/sot/ska3/matlab/trunk
eval "$(/proj/sot/ska3/matlab/trunk/bin/conda shell.bash hook)"
Create /proj/sot/ska3/matlab/trunk/condarc
with the following content:
channels:
- https://ska:<password>@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/masters
- https://ska:<password>@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/flight
- https://ska:<password>@cxc.cfa.harvard.edu/mta/ASPECT/ska3-conda/test
Install:
conda install mamba
mamba install ska3-core==2023.1 ska3-flight-latest==2023.3
ln -s /proj/sot/ska/data/* /proj/sot/ska3/matlab/trunk/data
To update:
conda clean --all --packages --force-pkgs-dir
mamba update --all
- Ska and Ska3 environment planning doc (on google drive)
- https://conda-forge.org/docs/recipe.html
- https://conda.io/docs/user-guide/tutorials/index.html
All Ska3 packages are developed around Github, and distributed as conda packages. There are two kinds of conda packages in Ska3:
-
Meta-packages are just empty packages with a list of dependencies. These are useful to manage collections of packages in a coherent way. These packages are hosted in sot/skare3. Their names all start with
ska3-
and their versions follow a{year}.{integer}[rc{integer}]
format, with version numbers like2022.1
for releases and2022.1rc1
for pre-releases. Examples of these meta-packages includeska3-flight
,ska3-matlab
,ska3-core
andska3-perl
. -
Regular packages are hosted on their own Github repositories and their build recipe is hosted in sot/skare3. Their versions follow the PEP 440 specification, which is handled by setuptools_scm.
They usually have the following workflows:
-
build (
.github/workflows/build.yml
) calls the sot/skare3/package_build.yml workflow, which assembles a conda package and uploads it to the Ska3 conda repository. -
release (
.github/workflows/release.yml
) calls the sot/skare3/package_release.yml workflow which edits the release description and performs other menial tasks associated with the release. -
documentation (
.github/workflows/docs.yml
) calls the sot/skare3/package_docs.yml workflow, which builds the package's sphinx documentation and uploads it so it appears in the Ska3 documentation.
-
build (
Recipes for regular packages are usually modified on a branch, reviewed internally and merged into master without requiring review from out of the Ska development team. Changes to package's build recipe's are very rare.
Edits to the recipes for meta-packages require a version number increase, and are always done on a branch as part of the release process outlined above: edits are made on a branch, the PR is internally reviewed, then an external authority approves. After approval, a release is made, the PR is merged and the package is installed in production environments.
An update to a core package in the ska3-core
meta-package requires an increment of the version of ska3-core
, and update to the ska3-core
entry in ska3-flight
, an increment in the ska3-flight
meta-package version, and a rebuild of both the ska3-core
and ska3-flight
meta-packages.
NOTE: a typical issue that arises with this versioning scheme is that version 2022.10
can be interpreted as a float by yaml and converted to "2022.1"
, so in this particular case the quotation marks are required: "2022.10"
Every step in this section is usually done automatically as part of the package build. However, sometimes the process fails, in which case it is important to be able to do it manually (hence this section).
Building manually can be tedious because we might need to build several packages for the different platforms. When something goes wrong in the automated build, it is very often easier to fix the workflow and run it again.
The build workflow can be triggered directly using the Github API. One way to do it is to use skare3_tools:
from skare3_tools import github
r = github.Repository('sot/skare3')
r.dispatch_event(
event_type='build-package',
client_payload={
"package": "obc_rate_noise_plots", # This can be a package name or a repository
"tag": "package", # This can be a branch or tag.
"arch": True, # Optional. Default is False.
"upload": True, # Optional. Default is False.
"skare3_branch": "obc_rate_noise_plots", # Optional. Default is master.
# "python_version": "3.11", # Optional. Default as shown.
# "numpy_version": "1.26.2", # Optional. Default as shown.
# "channel_in": "flight", # Optional. Default as shown.
# "channel_out": "test", # Optional. Default as shown.
}
)
Right now this section is outdated. Better check the workflow itself to see how this is done.
To support CentOS5, packages should be built from a CentOS5 machine (though for noarch packages it should really make no difference) and the user running the scripts will need a CentOS5-compatible .condarc
. First, edit your user ~/.condarc to include:
binstar_upload: false
channels:
- https://icxc.harvard.edu/aspect/ska3-conda/
- https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
# - defaults
#default_channels:
# - https://repo.continuum.io/pkgs/free
# - https://repo.continuum.io/pkgs/r
# - https://repo.continuum.io/pkgs/msys2
(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)
This is already configured for the 'aca' user. Then fetch a miniconda installer still compatible with CentOS5 and install
chmod o-rwx ~/.condarc
curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-Linux-x86_64.sh
bash Miniconda3-4.3.21-Linux-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # Update Python but keep conda at 4.3.21
conda install ska3_builder # This package must be installed in a "root" conda environment
For the aca user this install has been done with the ${miniconda_install_prefix} set to /export/aca/ska3 on presto .
For OSX we could use newer packages in ska3-dev in the modern channels, but by default, we are using the same .condarc used in Linux. This prevents getting carried away with features that we can't actually use on Linux production systems. So, the .condarc is set as such on the skare3 osx build system, zero. (zero is presently not available so this section of the directions is not up-to-date)
binstar_upload: false
channels:
- https://icxc.harvard.edu/aspect/ska3-conda/
- https://icxc.harvard.edu/aspect/ska3-conda/core-pkg-repo
# - defaults
#default_channels:
# - https://repo.continuum.io/pkgs/free
# - https://repo.continuum.io/pkgs/r
# - https://repo.continuum.io/pkgs/msys2
(the defaults channels look to have become incompatible on 2019-07-01 so are commented out for now)
On zero, the aca user also has a .bash_profile that has the miniconda environment already loaded, with the miniconda_install_prefix set to '/Users/aca/miniconda3/bin/conda'. If this needs to be redone, these would be the steps.
curl -O https://repo.continuum.io/miniconda/Miniconda3-4.3.21-MacOSX-x86_64.sh
bash Miniconda3-4.3.21-MacOSX-x86_64.sh
export PATH=${miniconda_install_prefix}:$PATH
conda install python=3.6.2 conda=4.3.21 # update Python but keep conda at 4.3.21
conda install ska3_builder # This package must be installed in a "root" conda environment
The underlying packages that comprise the ska3 environment are built using the
ska_builder.py
script, which is available via the skare3 git repository with the setup described above.
Several of the packages are private github packages, and for these to clone and build locally you will need working github ssh authentication (this is already set up for the aca user). If you haven't set this up, see https://help.github.com/articles/connecting-to-github-with-ssh/ . You may also want to manage your key with ssh-agent before building packages. See https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#adding-your-ssh-key-to-the-ssh-agent
After ssh is set up get into the appropriate conda environment for building (if not already set from above steps). For builds as aca, that is currently
export PATH=/export/aca/ska3/bin:$PATH
as the aca user on presto (also set as ska3build alias if working in .cshrc)
For 32 bit builds for matlab build as the aca user on quango and set path
export PATH=/export/aca/miniconda3/bin/:$PATH
For osx use the aca user on the HEAD system 'zero', and the PATH is already set.
Then set the build root
export build_root=/export/aca/ska3-conda-builds # As ACA for production on presto
OR
export build_root=/export/aca/build_root # As ACA user for 32 bit builds on quango
OR
export build_root=~/ska3-conda-builds # In user home dir
Then get in to the skare3 repo and do the build. Clone if needed
cd ~/git
git clone git@github.com:sot/skare3
Check out the skare3 packages at the release tag intended for the ska3-flight or ska3-matlab release.
cd ~/git/skare3
python ska_builder.py --build-root=${build_root} <package>
By default the latest tag (sorted by commit date, not tag date) is built.
To build a specific tag, specify that with the --tag=<tag>
option.
In order to build all available packages at once, just omit the <package>
:
cd ~/git/skare3
python ska_builder.py --build-root=${build_root}
This will clone each repo if needed from GitHub or fetch updates, check out at the desired tag, and build the package using the conda build recipe.
The cloned repos are in ${build_root}/srcs
and the builds are in
${build_root}/builds
.
You can then install the new built package into a non-production environment using:
conda install -c file://${build_root}/builds <package>
Assuming you have built everything, then you can use the meta-package names
shown earlier like ska3-flight
.
The ska3-conda
channel is a non-public channel that is served
on the icxc
web server and can be accessed via a typical conda channel URL spec.
The channel URL is https://icxc.cfa.harvard.edu/aspect/ska3-conda
,
corresponding to a file directory of /proj/sot/ska/www/ASPECT_ICXC/ska3-conda
.
For production linux builds, do the following (still as user aca
on presto
):
# Get into ska3_builder environment if not already there
rsync -ruv ${build_root}/builds/linux-64/ \
/proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
/proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/noarch/
For the test channel one would use (note that these are on the public web and not ICXC, they are rsync'd to ICXC from public web):
rsync -ruv ${build_root}/builds/linux-64/ \
/proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/linux-64/
rsync -ruv ${build_root}/builds/noarch/ \
/proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/noarch/
For production MacOSX builds, after building the packages as the aca user on the zero system, sync back to the HEAD LAN as as the aca user on the zero system:
ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
/proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT_ICXC/ska3-conda/osx-64/
For the test channel one would use (note that these are on public web. A cronjob syncs to ICXC)
ssh aca@kady % can't be done from zero
rsync -ruv aca@zero:/${build_root}/builds/osx-64/ \
/proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/
export PATH=/disk1/telmon/conda/bin:$PATH
conda index /proj/sot/ska/www/ASPECT/ska3-conda/test/osx-64/
The ska3-core
package should be updated once or twice a year to prevent core conda packages from getting too stale. The process update ska3-core/meta.yml
is described in Updating the Core Conda Packages