# ASKAP SDP HAS Moved!

The **ASKAP SDP project**, which is astronomy data reduction software **ASKAPsoft** and other things, is now being hosted at the CSIRO Bitbucket repository which is publicly accessible. The version of ASKAPsoft in this github repo is old and not supported in any way. Go to the CSIRO Bitbucket to obtain the latest version. Information and links (including Digital Object Identifiers (DOI) for referencing) to various versions can be found if you [search the CSIRO Data Access Portal (DAP)](https://data.csiro.au/dap/search?q=ASKAPsoft). The DAP will also have contact details for queries and issues.

We have moved from github to that system on CSIRO recommendation. This github repo was setup before CSIRO obtained their Bitbucket services.

---------

ASKAPsoft Science Data Processor
================================

These notes provide just a brief outline for building the ASKAPsoft Science Data
Processor codebase. More detailed instructions exist on the ASKAP Computing
Wiki.

Software Requirements
=====================

ASKAPsoft is designed to be built and executed in a standard Unix/Linux
environment. The current official supported environments are:
* Debian 7.x (Wheezy) Linux
* Mac OSX 10.9.x (Mavericks)
* Cray Linux Environment (XC30 platform)

Core dependencies must be fulfilled by the platform. These include, but are not
limited to, a C/C++/Fortran compiler, Make, Python 2.7, Java 7 and MPI. More
specific dependencies are downloaded by the ASKAPsoft build system and are
installed within the ASKAPsoft development tree.

Specific to the Debian platform, after a standard installation of Debian Wheezy
(7.x) the following packages will need to be installed with apt-get:

* g++
* gfortran
* openjdk-7-jdk
* python-dev
* flex
* bison
* openmpi-bin
* libopenmpi-dev
* libfreetype6-dev
* libpng12-dev

Checkout & Bootstrap
====================

The first step of building ASKAPsoft involved checking out a copy of the code
from the Subversion repository and then running the bootstrap script.

Note that during the bootstrap and build process a number of tarballs containing
third party software will be fetched. If Direct HTTP access is not available on
your build host then please refer the the section below titled "Archive
Repository".

The boostrap script build and configures the developer tools necessary for
building the rest of the ASKAPsoft codebase:

    svn co https://svn.atnf.csiro.au/askapsdp/trunk ASKAPsoft
    cd ASKAPsoft
    /usr/bin/python2.7 bootstrap.py -n

The "-n" option to bootstrap says not to use "svn update" to update the
workspace prior to doing the bootstrap.

Building
========

Before building ASKAPsoft, certain environment variable need to be set.  This is
accomplished by sourcing the initaskap.sh script, which will exist in the top
level directory after the bootstrap procedure described above has been
completed. Then the recursive builder (rbuild) can be used to build the entire
ASKAPsoft tree:

    cd ASKAPsoft
    . initaskap.sh
    rbuild -S -a

The "-S" option to rbuild says not to do an "svn update" prior to building.

To build a release tarball for a specific package, or set of packages the
release target option must be set. For example to build the core data processing
applications (assuming initaskap.sh has already been sourced):

    cd $ASKAP_ROOT
    cd Code/Systems/cpapps
    rbuild -S -t release

This will create a filenamed release-<svnrev>.tgz

Archive Repository
==================

ASKAPsoft depends on a number of 3rd party packages. Some of these must be
supplied by the operating platform. For example a compiler such as GCC and an
MPI implementation must be present.

Other software is downloaded during the ASKAPsoft build process, and as such
HTTP access is required. If your build host relies on a HTTP proxy to access the
internet then the http_proxy & https_proxy environment variables must be set in
your environment. Here is an example, where "webproxy.myorg.com" is the hostname
for the proxy server that is running on port 8080

    export http_proxy=http://webproxy.myorg.com:8080
    export https_proxy=$http_proxy

Alternatively, if access to download these files is not available at all on the
build host a local repository can be set up. In this case the bootstrap.ini must
be configured (prior to executing the bootstrap below) to refer to the local
cache via a "file" URL. For example:

    remote_archive = file:///home/joe/askapsoft_dependencies

Documentation
=============

Included within the ASKAPsoft tree is both API documentation and user
documentation. The user documentation can be built like so:

    cd $ASKAP_ROOT/Code/Doc/user/current
    rbuild -n -t doc

The HTML documentation tree will be created at the following location:
$ASKAP_ROOT/Code/Doc/user/current/doc/_build/html

Rather than a single unified API documentation tree, each package hosts its own
documentation. A package can generally be identified by the fact it has a
build.py in its package root directory. The following example shows how to build
the "Synthesis" package API documentation:

    cd $ASKAP_ROOT/Code/Components/Synthesis/synthesis/current
    rbuild -n -t doc

The HTML documentation tree will be created at the following location:
$ASKAP_ROOT/Code/Components/Synthesis/synthesis/current/html

Release Build
=============

While the ASKAPsoft applications can be executed directly out of a development
tree, a deployment consisting of the binaries and libraries is support. A
release tarball can be created for each package separately if desired, however
super-packages are available as a convenient grouping. For example the "cpapps"
package provides the calibration, imaging & source-finding software. It can be
built like so:

    cd $ASKAP_ROOT/Code/Systems/cpapps
    rbuild -n -t release

This will create a file release-1234.tgz (where 1234 is the Subversion release
number) which contains bin & lib directories. This tarball can be extracted in a
convenient location, such as /opt or /usr/local, and PATH/LD_LIBRARY_PATH
environment variables can then be set to include the bin/lib directories
respectively.

In the typical HPC environment, the ASKAPsoft "cpapps" package would be deployed
as an [Environment Module].

[Environment Module]:http://modules.sourceforge.net/

rbuild
======

The 'rbuild' command is the main build command for developers. It has the
ability of updating from the subversion repository and also recursively resolve,
update and build dependencies.

To get help for 'rbuild' simply type 

------------------------------------------------------------------------------
%% rbuild -h
Usage: rbuild [options] [<package_path>]
                                                                                                        
This is the main ASKAPsoft build command for developers. It can handle                                  
dependencies, subversion updates and changes to the build system. There are
two types of build targets: recursive [depends, install, stage, release,
signature] and non-recursive [bclean, clean, doc, format, pylint, functest,
test, deploy]. The non-recursive targets only apply to the current package,                             
while recursive targets are applies to all dependencies of the current                                  
package.  The recursive behaviour may be overridden with appropriate flag. The                          
default <package_path> is the current directory.                                                        
                                                                                                        
Options:
  -h, --help            show this help message and exit                                                                            
  -a, --autobuild       mode where dependencies/packages are computed once.                                                        
                        Additionally turns on no-update and no-recursion flags                                                     
                        i.e. -n -R                                                                                                 
  -f, --force           force building of packages ignoring NO_BUILD files                                                         
  -q, --quiet           do not show all builder output [default=True]
  -v, --verbose         show all builder output                                                                                                      
  -n, --no-update       no svn updates, rebuild of myself or Tools rebuild.                                                                          
                        Equivalent to "-S -M -T"                                                                                                     
  -N, --no-recursive-update                                                                                                                          
                        no svn updates, rebuild of myself, Tools rebuild or                                                                          
                        recursion. Equivalent to "-S -M -T -R -v" or                                                                                 
                        "python build.py TARGET"                                                                                                     
  -p EXTRAOPTS, --pass-options=EXTRAOPTS
                        pass on package specific build options, e.g. "nompi=1"                                                                         
                        or specific functional tests e.g "-t functest -p                                                                             
                        functests/mytest.py"                                                                                                         
  -t TARGET, --target=TARGET                                                                                                                         
                        select TARGET from: depends, install, stage, release,                                                                        
                        signature, bclean, clean, doc, format, pylint,
                        functest, test, deploy [default=install]
  --release-name=RELEASE_NAME
                        the name of the staging directory and the release
                        tarball
  --deploy-targets=DEPLOY_TARGETS
                        the deployement targets to execute. Select any
                        DEPLOY_TARGETS from:
                        authenticate,deploy_local,deploy_remote,verify [defaul
                        t=authenticate,deploy_local,deploy_remote,verify]

  Advanced Options:
    Caution: Use these options at your own risk.

    -D, --debug         do not run the actual package building command
    -M, --no-build-myself
                        do not rebuild myself (rbuild)
    -P, --no-parallel   do not do parallel builds of packages
    -R, --no-recursion  do not apply target recursively to dependencies
    -S, --no-svn-update
                        do not perform subversion update
    -T, --no-tools      do not rebuild Tools
    -U, --update-only   Ignore any target options and just do svn update
    -V, --no-virtualenv
                        do not include virtualenv in a release
    -X, --no-exit-on-error
                        continue building ignoring any individual package
                        build failures

------------------------------------------------------------------------------

rbuild system
=============
The rbuild system is the infrastructure underlying the rbuild command.
At the package level it can be run via the rbuild script or calling the
build.py files directly. In this case there is no recursive building.

For example:
    python build.py [-q|-x] <target>

It provides the following command-line options

-q        suppress messages to stdout and just print errors
-x        exit recursivebuild when an error is encountered

and targets as for rbuild.


Troubleshooting
===============

* Make sure that you don't have a ~/.pydistutils.cfg because this can
  conflict with virtualenv.