Skip to content

Home

Jump to bottom
Dr Owain Kenway edited this page Apr 23, 2019 · 2 revisions

Overview

This repository contains the build scripts used to install software on the UCL RITS HPC/HTC services, as well as issues with bug reports or requests for new installs.

Structure on services

On each UCL service, the repository is checked out in /shared/ucl/apps/build_scripts via https by the role account ccspapp which is used to manage software installs. Correspondingly, there is another repository which holds environment modules. This is updated on services by running /shared/ucl/apps/update_modules as ccspapp.

General script structure

The purpose of each script is to allow us to reproducibly install software across our clusters. In this respect it most likely only works on our clusters but does document the steps required to install a package elsewhere. The scripts vary in format depending on who wrote them but effectively run:

  1. Some initial variable setup, including module requirements in newer scripts. This makes a lot of use of the following construct in bash:
VAR=${VAR:-default}

This sets $VAR to the existing contents of $VAR if already set, or to "default" (replace with what should be the default value) if it is not. This allows you to control the scripts from outside by setting environment variables.

  1. Download and checksum source code (or check it out of github at a given release).

  2. Apply any patches as required. Patches should be included in the repository. Where there is a license issue clever application of sed may be required instead.

  3. Compile and install the package.

Installing a new package

  1. Write the script. I keep a skeleton script that I fill out on my own workstation as a starting point:
#!/usr/bin/env bash

set -e

for i in ${includes_dir:=$(dirname $0 2>/dev/null)/includes}/{module_maker,require}_inc.sh; do . $i; done

require gcc-libs/4.9.2

NAME=${NAME:-}
VERSION=${VERSION:-}
INSTALL_PREFIX=${INSTALL_PREFIX:-/shared/ucl/apps/$NAME/$VERSION/$COMPILER_TAG}
MD5=${MD5:-}
SHA1=${SHA1:-}
SHA256=${SHA256:-}
SHA512=${SHA512:-}

SRC_ARCHIVE=${SRC_ARCHIVE:-}


mkdir -p /dev/shm/$(whoami)/${NAME}

temp_dir=$(mktemp -d -p /dev/shm/$(whoami)/${NAME})

cd $temp_dir

wget $SRC_ARCHIVE
archive=$(basename "${SRC_ARCHIVE}")

md5sum -c <<< "$MD5 $archive"
sha1sum -c <<< "$SHA1 $archive"
sha256sum -c <<< "$SHA256 $archive"
sha512sum -c <<< "$SHA512 $archive"

tar -xvf $archive

cd ${NAME}-${VERSION}

./configure --prefix=$INSTALL_PREFIX
make 
make install

rm -rf $temp_dir

Once done, check it into the repository (remember to push).

  1. Log onto one of the services as ccspapp. Clear out the existing environment. For complex historical reasons ccspapp has different default modules on different services:
module purge
module load rcps-core

This should give you useful tools (like git) but not compilers or other libraries which may conflict with what you are doing.

  1. Update the local copy of the repository:
cd /shared/ucl/apps/build_scripts
git pull

Once this is completed you can run your script. Hopefully, the install completes successfully.

  1. Write a module file (in TCL - sorry) and add it to the rcps-modulefiles repository.

  2. Update the local modules repository:

cd /shared/ucl/apps
./update_modules
  1. Repeat steps 3 + 5 on each service.
Clone this wiki locally