lab1-parallel-programming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 1: High-Performance Computing Environment\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, we will set up a PC-based parallel computing environment in our computer laboratory and try some MPI-based parallel computing. In the future, we might extend it to a Hadoop+Spark computational platform.\n",
    "![](images/clusters.png)\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "#### System\n",
    "- CentOS 6.x, CentOS 7.x\n",
    "- RHEL 6.x, 7.x\n",
    "- Scientific Linux 6.x, 7.x\n",
    "- SUSE Linux Enterprise Server 11, 12\n",
    "    * Run daemons: `service` -> `systemctl`\n",
    "    * daemon scripts: `/etc/init.d/` -> `/usr/lib/systemd/system/`\n",
    "\n",
    "#### Software\n",
    "- `yum install libtool libxml2-devel openssl-devel boost-devel`\n",
    "- [Torque](http://www.adaptivecomputing.com)\n",
    "- [MPICH](http://www.mpich.org/)\n",
    "- [Environment Modules](http://modules.sourceforge.net/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Prepation: Architecture settings\n",
    "\n",
    "1. Each PC cluster are comprised of 4 computers - one as the **master node**: (for resource management), the other three as the **slave nodes** (for executing computational tasks).\n",
    "\n",
    "2. The master node communicates with the slave nodes via **SSH**, without entering the password.\n",
    "\n",
    "3. All the nodes share part of the file system using **NFS**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Network configuration\n",
    "\n",
    "On each node, add the names and corresponding IP address of all the nodes in our cluster to `/etc/hosts` file, as shown in the following example:\n",
    "```\n",
    "127.0.0.1    localhost\n",
    "192.168.5.1   node01   # master\n",
    "192.168.5.2   node02   # slave\n",
    "192.168.5.3   node03   # slave\n",
    "192.168.5.4   node04    # slave\n",
    "```\n",
    "and then restart the network:\n",
    "```bash\n",
    "/etc/init.d/network restart\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. SSH setup\n",
    "\n",
    "On `node01` (the master node), run as a non-root user:\n",
    "```bash\n",
    "ssh-keygen -t rsa -C \"your_email@example.com\"\n",
    "```\n",
    "Do NOT enter any password. This will generate two files under your `~/.ssh` directory:\n",
    "- **id_rsa.pub**: public key (公钥)\n",
    "- **id_rsa**: private key (私钥)\n",
    "\n",
    "Copy the public key to another file:\n",
    "```bash\n",
    "cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys\n",
    "```\n",
    "\n",
    "Repeat the above process on all the nodes. And merge the files `~/.ssh/authorized_keys` into a single file and then copy to all the nodes under the same directory `~/.ssh/`.\n",
    "\n",
    "Now you can transfer between these nodes without entering password, cool.\n",
    "\n",
    "```bash\n",
    "# on node 01\n",
    "ssh node02\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Setup NFS for shared file system\n",
    "\n",
    "On node01, add something like the folloiwng to the file `/etc/exports`, set the two file systems `/usr/local` and `/home` shared by the other nodes in our cluster.\n",
    "```\n",
    "/home    192.168.5.0/24(rw,no_root_squash)\n",
    "/usr/local    192.168.5.0/24(rw,no_root_squash)\n",
    "```\n",
    "\n",
    "and then restart the NFS service:\n",
    "```bash\n",
    "chkconfig --add nfs\n",
    "/etc/init.d/nfs restart\n",
    "```\n",
    "\n",
    "For slave nodes, you should add two lines to the file `/etc/fstab`:\n",
    "```bash\n",
    "node01:/home    /home    nfs   default    0 0\n",
    "node01:/usr/local    /usr/local  nfs    default    0 0\n",
    "```\n",
    "\n",
    "and then run\n",
    "```bash\n",
    "mount -a\n",
    "```\n",
    "to mount the files systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Install Torque for Job management\n",
    "\n",
    "### Configuration on the server (node01)\n",
    "\n",
    "On the master node (**node01**), run the following command to install the most popular open-source Job Manager, Torque:\n",
    "```bash\n",
    "yum install git\n",
    "git clone https://github.com/adaptivecomputing/torque.git -b 6.0.2 6.0.2\n",
    "cd 6.0.2\n",
    "./autogen.sh\n",
    "./configure --disable-gui\n",
    "make && make install\n",
    "```\n",
    "\n",
    "And then begin configuration:\n",
    "```bash\n",
    "echo $(hostname) > /var/spool/torque/server_name\n",
    "cp contrib/init.d/trqauthd /etc/init.d/\n",
    "chkconfig --add trqauthd\n",
    "echo /usr/local/lib > /etc/ld.so.conf.d/torque.conf\n",
    "ldconfig\n",
    "/etc/init.d/trqauthd start\n",
    "qterm\n",
    "./torque.setup root\n",
    "```\n",
    "\n",
    "And then add the nodes resources into the file:\n",
    "```bash\n",
    "cat > /var/spool/torque/server_priv/nodes <<EOF\n",
    "node01  np=2\n",
    "node02  np=2\n",
    "node03  np=2\n",
    "node04  np=2\n",
    "EOF\n",
    "```\n",
    "\n",
    "Copy the starting scripts to `/init.d`, and start the service:\n",
    "```bash\n",
    "cp contrib/init.d/pbs_server /etc/init.d/\n",
    "cp contrib/init.d/pbs_sched /etc/init.d/\n",
    "chkconfig --add pbs_server\n",
    "chkconfig --add pbs_sched\n",
    "/etc/init.d/pbs_server restart\n",
    "/etc/init.d/pbs_sched restart\n",
    "```\n",
    "\n",
    "### 5.2 Configuration on the mom nodes (node02, node03, node04)\n",
    "\n",
    "Build the packages for the slave nodes:\n",
    "```bash\n",
    "## run on node01 \n",
    "make packages\n",
    "for i in 2 3 4; do\n",
    "    scp contrib/init.d/pbs_mom node0\"${i}\":/etc/init.d/\n",
    "    scp torque-package-mom-linux-x86_64.sh node0\"${i}\":/root/\n",
    "    scp torque-package-clients-linux-x86_64.sh node0\"${i}\":/root\n",
    "    ssh node0${i} ./torque-package-mom-linux-x86_64.sh --install\n",
    "    ssh node0${i} ./torque-package-clients-linux-x86_64.sh --install\n",
    "    ssh node0${i} chkconfig --add pbs_mom\n",
    "    ssh node0${i} /etc/init.d/pbs_mom restart\n",
    "done\n",
    "```\n",
    "\n",
    "Be sure to check whether the service has started.\n",
    "\n",
    "And then add the following lines to the file `/var/spool/torque/mom_priv/config`:\n",
    "```bash\n",
    "for node in 2 3 4; do\n",
    "    ssh node0${node} echo > /var/spool/torque/mom_priv/config <<EOF\n",
    "\\$pbsserver    node01    # hostname running pbs server     \n",
    "\\$logevent      225         # bitmap of which events to log   \n",
    " EOF\n",
    "done\n",
    "```\n",
    "\n",
    "<font color=\"red\">Note</font>: If you want to run both the server and mom on a single PC, give the server and mom different host names for a same PC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Some important configuration files\n",
    "- `/var/spool/torque/server_name`\n",
    "- `/var/spool/torque/server_priv/nodes`\n",
    "- `/var/spool/torque/mom_priv/config`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Environment Modules – A Great Tool for Clusters\n",
    "\n",
    "The Environment Modules package provides for the dynamic modification of a user's environment via **modulefiles**.\n",
    "\n",
    "You can use Environment Modules to alter or change environment variables such as \\$PATH, \\$MANPATH, \\$LD_LIBRARY_LOAD, and others.\n",
    "\n",
    "### 6.1 Installation and Configuration\n",
    "\n",
    "Download the Environment Modules from [sourceforge.net](http://modules.sourceforge.net)\n",
    "```bash\n",
    "cd /usr/local\n",
    "mkdir Modules\n",
    "cd Modules\n",
    "mkdir src\n",
    "cp modules-3.2.6.tar.gz /usr/local/Modules/src\n",
    "tar xzvf modules-3.2.6.tar.gz\n",
    "cd modules-3.2.6\n",
    "cd /usr/local/Modules/src/modules-3.2.6\n",
    "./configure \n",
    "make && make install\n",
    "\n",
    "cd /usr/local/Modules\n",
    "ln -s 3.2.6 default\n",
    "\n",
    "cp /usr/local/Modules/default/init/sh /etc/profile.d/modules.sh\n",
    "chmod 755 /etc/profile.d/modules.sh\n",
    "```\n",
    "\n",
    "Now users can use Environment Modules by just putting the following in their *.bashrc* or *.profile*:\n",
    "\n",
    "```bash\n",
    ". /etc/profile.d/modules.sh\n",
    "```\n",
    "\n",
    "#### Some configuration files\n",
    "- `/usr/share/Modules/int/`\n",
    "    * `/usr/share/Modules/.modulepath`\n",
    "    * `/usr/share/Modules/init/sh`\n",
    "- `/usr/share/Modules/modulefiles/` directory\n",
    "- `/etc/modulefiles/` directory\n",
    "- `/etc/profile.d/modules.sh`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Using Environment Modules\n",
    "\n",
    "To begin, I’ll assume that Environment Modules is installed and functioning correctly, so you can now test a few of the options typically used. \n",
    "\n",
    "- **List the available modules**:\n",
    "```bash\n",
    "module avail\n",
    "```\n",
    "\n",
    "- **Load the required software**:\n",
    "```bash\n",
    "module load bwa\n",
    "```\n",
    "\n",
    "- **Remove the modules from the current environment**:\n",
    "```bash\n",
    "module unload bwa\n",
    "```\n",
    "\n",
    "- **List the loaded softwares**:\n",
    "```bash\n",
    "module list\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 Writing module files\n",
    "\n",
    "Here is an example for a newer version of `compilers/gcc-4.6.2`:\n",
    "```\n",
    "#%Module1.0#####################################################################\n",
    "##\n",
    "## modules compilers/gcc-4.6.2\n",
    "##\n",
    "## modulefiles/compilers/gcc-4.6.2\n",
    "##\n",
    "proc ModulesHelp { } {\n",
    "        global version modroot\n",
    "\n",
    "        puts stderr \"compilers/gcc-4.6.2 - sets the Environment for GCC 4.6.2 in my home directory\"\n",
    "}\n",
    "\n",
    "module-whatis   \"Sets the environment for using gcc-4.6.2 compilers (C, Fortran)\"\n",
    "\n",
    "# for Tcl script use only\n",
    "set     topdir          /usr/local/gcc-4.6.2\n",
    "set     version         4.6.2\n",
    "set     sys             linux86\n",
    "\n",
    "setenv          CC              $topdir/bin/gcc\n",
    "setenv          GCC             $topdir/bin/gcc\n",
    "setenv          FC              $topdir/bin/gfortran\n",
    "setenv          F77             $topdir/bin/gfortran\n",
    "setenv          F90             $topdir/bin/gfortran\n",
    "prepend-path    PATH            $topdir/include\n",
    "prepend-path    PATH            $topdir/bin\n",
    "prepend-path    MANPATH         $topdir/man\n",
    "prepend-path    LD_LIBRARY_PATH $topdir/lib\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 6.4 Module example together with torque\n",
    "\n",
    "Here is the combination of `Environment Modules` and `Torque`:\n",
    "```bash\n",
    "#PBS -l nodes=2:ppn=8,pmem=1000mb,walltime=8:00:00\n",
    "#PBS -m example\n",
    "#PBS -M sample_email@sjtu.edu.cn\n",
    "\n",
    "module load bwa\n",
    "bwa index input.fasta\n",
    "bwa aln -t $PBS_NP input.fasta input.fq > output.sai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. MPI Parallel Programming Development\n",
    "\n",
    "MPI is the abbreviation of `Message Passing Interface`, \n",
    "\n",
    "### 7.1 Install MPICH2\n",
    "\n",
    "Now installation of MPICH2 has become an easy job, you can run `yum install mpich2` directly as privileged user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Practical MPI programming\n",
    "\n",
    "The details will be illustrated in the [lab notebook](lab1.pdf)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}