ubuntu 10.10

StarCluster AMI Cookbook Ubuntu 10.10

Below are pseudo-instructions for creating the latest StarCluster AMI (Ubuntu 10.04 Lucid) from scratch:

Note

You should be root when executing any command in this guide

Launch AMI and Install Packages

1. Launch an alestic ubuntu AMI (x86: ami-2d4aa444, amd64: ami-fd4aa494)

2. Login via ssh as the ubuntu user and use "sudo -i" to switch to root

3. Edit /etc/apt/sources.list, add multiverse to the end of all lines

4. Update the packages to the latest versions:

   apt-get update
   apt-get upgrade
   

5. Install the necessary build tools

   apt-get --reinstall install build-essential python-dev swig gfortran -y apt-get remove python-nose g77 python-setuptools -y

6. Install and configure blas/lapack:

   rm -rvf /usr/lib{,64}/atlas/
   apt-get remove liblapack"*" -y
   apt-get remove libblas"*" -y
   apt-get -y --reinstall install libatlas-base-dev libatlas3gf-base
   

7. Install and configure gfortran:

   apt-get --reinstall install gfortran-4.4 -y
   ln -sf /usr/bin/gfortran{-4.4,}
   

8. Remove any existing numpy/scipy installations:

   rm -rvf /usr/local/lib/python2.6/*-packages/numpy*
   rm -rvf /usr/local/lib/python2.6/*-packages/scipy*
   

6. Install and configure the base dependencies:

   apt-get install python-dev git-core vim mercurial subversion cvs encfs \
   ipython openmpi-bin libopenmpi-dev nfs-kernel-server python-django keychain \
   screen tmux zsh ksh csh tcsh python-pip python-setuptools python-virtualenv \
   python-imaging python-boto unzip rar unace build-essential gfortran \
   ec2-api-tools ec2-ami-tools mysql-server mysql-client apache2 \
   libapache2-mod-wsgi sysv-rc-conf pssh emacs cython irssi python-distutils-extra \
   python-profiler htop vim-scripts python-ctypes
   

7. Install latest nose:

   pip install nose
   

Configure Root Login

The alestic AMI's have been configured to disable root logins. Follow the commands below to undo this behavior:

1. Edit /etc/cloud/cloud.cfg and set disable_root: 0
2. Edit /root/.ssh/authorized_keys and remove prefix commands from pubkey entry
3. Edit /usr/bin/cloud-init, go to line 67 and change 'once-per-instance' to 'always', save and exit

Configure MOTD and Bash Completion

1. Customize motd in /etc/update-motd.d/* and /etc/motd.tail
2. Uncomment bash completion section in /etc/bash.bashrc

Configure NFS

StarCluster relies on the /etc/init.d/nfs script to start the NFS server. Run the command below to ensure this exists:

ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs

Configure Sun Grid Engine

1. Unpack SGE 6u5 distribution files and move to /opt/sge6-fresh

2. Unpack python-drmaa (0.2) to /opt, set SGE6_ROOT='/opt/sge6-fresh' in setup.py and run:

   python setup.py install
   

3. Unpack python-drmaa (0.4b3) to /opt and run the following command inside the source folder:

   python setup.py install
   

4. Link the DRMAA library to /usr/lib64/libdrmaa.so:

   ln -s /opt/sge6-fresh/lib/lx24-{x86|amd64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so.1.0
   ln -s /usr/lib{64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so
   

6. Test that the following commands complete without error:

   python -c "import DRMAA"
   python -c "import drmaa"
   

Rebuild Ubuntu's OpenMPI packages with Sun Grid Engine Support

1. Install the build deps for libopenmpi-dev:

   apt-get build-dep libopenmpi-dev
   

2. Get the source for the libopenmpi-dev debian package:

   cd /usr/local/src
   apt-get source libopenmpi-dev
   

3. Change into the libopenmpi-dev package's debian folder:

   cd openmpi-1.4.1/debian
   

4. Modify the 'rules' file and add --with-sge to the configure arguments:

   vim rules
   (add --with-sge to configure args)
   

5. Rebuild the libopenmpi-dev package:

   cd ..
   dpkg-buildpackage -rfakeroot -b
   

6. Install the newly rebuild package:

   cd ..
   dpkg -i *.deb
   

7. Verify Sun Grid Engine support:

   ompi_info | grep -i grid
       MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.4.1)
   

Configure Optimized Atlas/UMFPACK/AMD/FFTW/Scipy/Numpy

The next sections describe installing an optimized scipy/numpy using Atlas/UMFPACK/AMD/FFTW libraries from source. A lot of these instructions were merged from http://scipy.org/Installing_SciPy/Linux

Install lapack/atlas from source

1. Checkout npinto-toolbox:

   cd /usr/local/src
   git clone git://github.com/npinto/npinto-toolbox
   cd npinto-toolbox
   

2. Run the ATLAS installer:

   ./install_atlas3.8.3_Ubuntu9.10-{x86,x86_64}-root.bash
   

3. Remove /usr/lib/liblapack.so:

   rm /usr/lib/liblapack.so
   

4. Update /usr/lib/gfortran.so symlink:

   ln -s /usr/lib/libgfortran.so.3 /usr/lib/gfortran.so
   

Install UMFPACK/AMD/UFconfig from source

1. Create a directory to unpack the source into:

   mkdir /usr/local/src/umfpack
   

2. Get the latest versions of AMD, UFconfig and UMFPACK from http://www.cise.ufl.edu/research/sparse/ and untar them into /usr/local/src/umfpack

3. Modify /usr/local/src/umfpack/UFconfig/UFconfig.mk to look like (change -m64 to -m32 in the flags below for a 32bit system):

   CC = gcc
   CFLAGS = -O3 -fexceptions -m64 -fPIC -DNCHOLMOD
   F77 = gfortran
   F77FLAGS = -O -m64 -fPIC
   
   BLAS = -lptf77blas -lptcblas -latlas -lgfortran
   LAPACK = -llapack -lptf77blas -lptcblas -latlas -lgfortran
   

4. Run 'make' in the /usr/local/src/umfpack/UMFPACK directory:

   cd /usr/local/src/umfpack/UMFPACK
   make
   

5. Copy resulting libraries and include files:

   cp /usr/local/src/umfpack/AMD/Lib/libamd.a /usr/lib
   cp /usr/local/src/umfpack/UMFPACK/Lib/libumfpack.a /usr/lib
   cp /usr/local/src/umfpack/AMD/Include/amd.h /usr/include
   cp /usr/local/src/umfpack/UFconfig/UFconfig.h /usr/include
   cp /usr/local/src/umfpack/UMFPACK/Include/*.h /usr/include
   

6. Create /usr/lib/libumfpack.so symlink:

   ln -s /usr/lib/libumfpack.a /usr/lib/libumfpack.so
   

Install FFTW from source

1. Download fftw-3.2.2 from http://www.fftw.org into /usr/local/src and unpack

2. Change into the fftw source folder:

   cd /usr/local/src/fftw-3.2.2
   

3. Configure fftw (change -m64 to -m32 for 32bit):

   export CFLAGS="-O3 -fomit-frame-pointer -fstrict-aliasing -ffast-math -pthread -fPIC -m64"
   export FFLAGS="-g -O2 -fPIC -m64"
   export CXXFLAGS="-g -O2 -fPIC -m64"
   ./configure --enable-sse2 --enable-threads --with-combined-threads --prefix=/usr
   

4. Build fftw:

   make install
   

Install Scipy/Numpy from source

1. Download numpy-1.5.0b1/scipy-0.8.0 into /usr/local/src and untar them

2. Copy numpy_scipy_site.cfg from npinto-toolbox:

   cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/numpy-1.5.0b1/site.cfg
   cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/scipy-0.8.0/site.cfg
   

3. Run the configure check for numpy:

   cd /usr/local/src/numpy-1.5.0b1
   python setup.py config
   

4. Copy the SMCFLAGS from the ATLAS VERSION output and export CFLAGS to be the same as SMCFLAGS

5. Copy the F77FLAGS from the ATLAS VERSION output and export F77FLAGS to be the same

6. Install numpy:

   python setup.py install
   

7. Run the following command:

   python -c "import numpy; numpy.show_config()"
   

8. The output of the above command should look something like:

   In [3]: import numpy
   
   In [4]: numpy.show_config()
   atlas_threads_info:
       libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
       library_dirs = ['/usr/lib']
       language = f77
   
   blas_opt_info:
       libraries = ['ptf77blas', 'ptcblas', 'atlas']
       library_dirs = ['/usr/lib']
       define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
       language = c
   
   atlas_blas_threads_info:
       libraries = ['ptf77blas', 'ptcblas', 'atlas']
       library_dirs = ['/usr/lib']
       language = c
   
   lapack_opt_info:
       libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
       library_dirs = ['/usr/lib']
       define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
       language = f77
   
   lapack_mkl_info:
     NOT AVAILABLE
   
   blas_mkl_info:
     NOT AVAILABLE
   
   mkl_info:
     NOT AVAILABLE
   

9. Install scipy:

   cd /usr/local/src/scipy-0.8.0
   python setup.py install
   

10. Run the following command:

    python -c "import scipy; scipy.show_config()"
    

11. Output of the above command should look something like:

    In [5]: import scipy
    
    In [6]: scipy.show_config()
    amd_info:
        libraries = ['amd']
        library_dirs = ['/usr/lib']
        define_macros = [('SCIPY_AMD_H', None)]
        swig_opts = ['-I/usr/include']
        include_dirs = ['/usr/include']
    
    umfpack_info:
        libraries = ['umfpack', 'gfortran', 'amd']
        library_dirs = ['/usr/lib']
        define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)]
        swig_opts = ['-I/usr/include', '-I/usr/include']
        include_dirs = ['/usr/include']
    
    atlas_threads_info:
        libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
        library_dirs = ['/usr/lib']
        language = f77
    
    blas_opt_info:
        libraries = ['ptf77blas', 'ptcblas', 'atlas']
        library_dirs = ['/usr/lib']
        define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
        language = c
    
    atlas_blas_threads_info:
        libraries = ['ptf77blas', 'ptcblas', 'atlas']
        library_dirs = ['/usr/lib']
        language = c
    
    lapack_opt_info:
        libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
        library_dirs = ['/usr/lib']
        define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
        language = f77
    
    lapack_mkl_info:
      NOT AVAILABLE
    
    blas_mkl_info:
      NOT AVAILABLE
    
    mkl_info:
      NOT AVAILABLE
    

12. Make sure the following commands both output "OK":

    python -c "import numpy; numpy.test()"
    python -c "import scipy; scipy.test()"