ubuntu 10.04

StarCluster AMI Cookbook Ubuntu 10.04

Below are pseudo-instructions for creating the latest StarCluster AMI (Ubuntu 10.04 Lucid) from scratch:

Note

You should be root when executing any command in this guide

Launch AMI and Install Packages

Launch an alestic ubuntu AMI (x86: ami-2d4aa444, amd64: ami-fd4aa494)
Login via ssh as the ubuntu user and use "sudo -i" to switch to root
Edit /etc/apt/sources.list, add multiverse to the end of all lines
Update the packages to the latest versions:
```
apt-get update
apt-get upgrade
```

Install the necessary build tools:

apt-get --reinstall install build-essential python-dev swig gfortran -y
apt-get remove python-nose g77 python-setuptools -y

Install and configure blas/lapack:

rm -rvf /usr/lib{,64}/atlas/
apt-get remove liblapack"*" -y
apt-get remove libblas"*" -y
apt-get -y --reinstall install libatlas-base-dev libatlas3gf-base

Install and configure gfortran:

apt-get --reinstall install gfortran-4.4 -y
ln -sf /usr/bin/gfortran{-4.4,}

Remove any existing numpy/scipy installations:

rm -rvf /usr/local/lib/python2.6/*-packages/numpy*
rm -rvf /usr/local/lib/python2.6/*-packages/scipy*

Install and configure the base dependencies:

apt-get install python-dev git-core vim mercurial subversion cvs encfs \
ipython openmpi-bin libopenmpi-dev nfs-kernel-server python-django keychain \
screen tmux zsh ksh csh tcsh python-pip python-setuptools python-virtualenv \
python-imaging python-boto unzip rar unace build-essential gfortran \
ec2-api-tools ec2-ami-tools mysql-server mysql-client apache2 \
libapache2-mod-wsgi sysv-rc-conf pssh emacs cython irssi python-distutils-extra \
python-profiler htop vim-scripts python-ctypes

Install latest nose:
```
pip install nose
```

Configure Root Login

The alestic AMI's have been configured to disable root logins. Follow the commands below to undo this behavior:

Edit /etc/cloud/cloud.cfg and set disable_root: 0
Edit /root/.ssh/authorized_keys and remove prefix commands from pubkey entry
Edit /usr/bin/cloud-init, go to line 67 and change 'once-per-instance' to 'always', save and exit

Configure MOTD and Bash Completion

Customize motd in /etc/update-motd.d/* and /etc/motd.tail
Uncomment bash completion section in /etc/bash.bashrc

Configure NFS

StarCluster relies on the /etc/init.d/nfs script to start the NFS server. Run the command below to ensure this exists:

ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs

Configure Sun Grid Engine

Unpack SGE 6u5 distribution files and move to /opt/sge6-fresh
Unpack python-drmaa (0.2) to /opt, set SGE6_ROOT='/opt/sge6-fresh' in setup.py and run:
```
python setup.py install
```
Unpack python-drmaa (0.4b3) to /opt and run the following command inside the source folder:
```
python setup.py install
```

Link the DRMAA library to /usr/lib64/libdrmaa.so:

ln -s /opt/sge6-fresh/lib/lx24-{x86|amd64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so.1.0
ln -s /usr/lib{64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so

Test that the following commands complete without error:
```
python -c "import DRMAA"
python -c "import drmaa"
```

Rebuild Ubuntu's OpenMPI packages with Sun Grid Engine Support

Install the build deps for libopenmpi-dev:
```
apt-get build-dep libopenmpi-dev
```
Get the source for the libopenmpi-dev debian package:
```
cd /usr/local/src
apt-get source libopenmpi-dev
```
Change into the libopenmpi-dev package's debian folder:
```
cd openmpi-1.4.1/debian
```
Modify the 'rules' file and add --with-sge to the configure arguments:
```
vim rules
(add --with-sge to configure args)
```
Rebuild the libopenmpi-dev package:
```
cd ..
dpkg-buildpackage -rfakeroot -b
```
Install the newly rebuild package:
```
cd ..
dpkg -i *.deb
```

Verify Sun Grid Engine support:

ompi_info | grep -i grid
    MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.4.1)

Configure Optimized Atlas/UMFPACK/AMD/FFTW/Scipy/Numpy

The next sections describe installing an optimized scipy/numpy using Atlas/UMFPACK/AMD/FFTW libraries from source. A lot of these instructions were merged from http://scipy.org/Installing_SciPy/Linux

Install lapack/atlas from source

Checkout npinto-toolbox:

cd /usr/local/src
git clone git://github.com/jtriley/npinto-toolbox
cd npinto-toolbox

Run the ATLAS installer:

./install_atlas3.8.3_Ubuntu9.10-{x86,x86_64}-root.bash

Remove /usr/lib/liblapack.so:
```
rm /usr/lib/liblapack.so
```

Update /usr/lib/gfortran.so symlink:

ln -s /usr/lib/libgfortran.so.3 /usr/lib/gfortran.so

Install UMFPACK/AMD/UFconfig from source

Create a directory to unpack the source into:
```
mkdir /usr/local/src/umfpack
```
Get the latest versions of AMD, UFconfig and UMFPACK and untar them into /usr/local/src/umfpack

Modify /usr/local/src/umfpack/UFconfig/UFconfig.mk to look like (change -m64 to -m32 in the flags below for a 32bit system):

CC = gcc
CFLAGS = -O3 -fexceptions -m64 -fPIC -DNCHOLMOD
F77 = gfortran
F77FLAGS = -O -m64 -fPIC

BLAS = -lptf77blas -lptcblas -latlas -lgfortran
LAPACK = -llapack -lptf77blas -lptcblas -latlas -lgfortran

Run 'make' in the /usr/local/src/umfpack/UMFPACK directory:
```
cd /usr/local/src/umfpack/UMFPACK
make
```

Copy resulting libraries and include files:

cp /usr/local/src/umfpack/AMD/Lib/libamd.a /usr/lib
cp /usr/local/src/umfpack/UMFPACK/Lib/libumfpack.a /usr/lib
cp /usr/local/src/umfpack/AMD/Include/amd.h /usr/include
cp /usr/local/src/umfpack/UFconfig/UFconfig.h /usr/include
cp /usr/local/src/umfpack/UMFPACK/Include/*.h /usr/include

Create /usr/lib/libumfpack.so symlink:

ln -s /usr/lib/libumfpack.a /usr/lib/libumfpack.so

Install FFTW from source

Download fftw-3.2.2 from http://www.fftw.org into /usr/local/src and unpack
Change into the fftw source folder:
```
cd /usr/local/src/fftw-3.2.2
```

Configure fftw (change -m64 to -m32 for 32bit):

export CFLAGS="-O3 -fomit-frame-pointer -fstrict-aliasing -ffast-math -pthread -fPIC -m64"
export FFLAGS="-g -O2 -fPIC -m64"
export CXXFLAGS="-g -O2 -fPIC -m64"
./configure --enable-sse2 --enable-threads --with-combined-threads --prefix=/usr

Build fftw:
```
make install
```

Install Scipy/Numpy from source

Download numpy-1.5.0b1/scipy-0.8.0 into /usr/local/src and untar them

Copy numpy_scipy_site.cfg from npinto-toolbox:

cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/numpy-1.5.0b1/site.cfg
cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/scipy-0.8.0/site.cfg

Run the configure check for numpy:

cd /usr/local/src/numpy-1.5.0b1
python setup.py config

Copy the SMCFLAGS from the ATLAS VERSION output and export CFLAGS to be the same as SMCFLAGS
Copy the F77FLAGS from the ATLAS VERSION output and export F77FLAGS to be the same
Install numpy:
```
python setup.py install
```

Run the following command:

python -c "import numpy; numpy.show_config()"

The output of the above command should look something like:

In [3]: import numpy

In [4]: numpy.show_config()
atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = c

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

Install scipy:

cd /usr/local/src/scipy-0.8.0
python setup.py install