00 - Conda Package Management.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Virtual Environments and Package Management"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Virtual environments are a useful tool for isolating and managing the software packages you use. Doing this enables you to track the packages you use for your work (enabling you to create reproducable environments, so that others can also use your code). It also allows you to use packages which might have conflicting requirements (or even different versions of the same package) without the hassle of haveing to install and uninstall these each time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python has a built in tools for managing virtual environments (venv) and packages (pip). We will not cover these tools today, for reasons given below, but if you are interested in them you can get more information here: https://docs.python.org/3/tutorial/venv.html\n",
    "\n",
    "While venv and pip are very useful for managing pure python packages, often we want to use packages which are built on other languages (or, even, are do not use python at all), and they are not very flexible in such a situation. Another tool, conda, has been built to extend their functionality to cover these mixed-language packages, for a wide range of computing platforms, and it is this which what we will cover today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lesson below is not conducted in a python interpreter, but instead in the unix shell. When you are asked to type in code below, please do this in a terminal window, not in this browser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing Conda"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conda package manager is available to install by itself (using the miniconda installer), or with a preprepared set of commonly used packages and a graphical user-interface (GUI) for managing your packages (using the anaconda installer). Miniconda is a good choice if you have limited bandwidth or disk space, and are happy to install packages as you need them using the command line interface (CLI). Anaconda is a good choice if you have more disk space (it requires >3Gb) and want to use a GUI.\n",
    "\n",
    "We will not cover installation in this lession, but more information is available on both of these, as well as links for downloading the installers, is available here: https://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package Channels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conda uses \"channels\", locations where software packages are stored. Conda has a default set of these channels that it searches when you request a package or update. You can modify this list of channels, to add other public or private channels that you wish to use. One of the most commonly added is conda-forge (https://conda-forge.org/), a community channel with thousands of contributors who use its unified, automated build infrastructure to share their software with the world."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see what channels are being used by the command: \n",
    "\n",
    "```bash\n",
    "conda config --show channels\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will (likely) list these as:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "channels:\n",
    "  - defaults"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add conda-forge as a channel by:\n",
    "\n",
    "```bash\n",
    "conda config --add channels conda-forge\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And use the previous command to check that it has now been added to your list of channels."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "channels:\n",
    "  - conda-forge\n",
    "  - defaults"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Note:</b> to get more information on the conda command, and on sub-commands, you can use `--help`. Try these two commands to see what this does.\n",
    "\n",
    "```bash\n",
    "conda --help```\n",
    "\n",
    "```bash\n",
    "conda config --help```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can list the environments that are installed using:\n",
    "\n",
    "```bash\n",
    "conda env list\n",
    "```"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# conda environments:\n",
    "#\n",
    "base                  *  /Users/mbessdl2/anaconda3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows you the environments you have installed, the path where they are installed, and indicates which environment you currently have loaded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A new conda environment can be created using the command:\n",
    "\n",
    "```bash \n",
    "conda create --name myenv\n",
    "```\n",
    "\n",
    "This will create an environment named `myenv` (once you press `[y]` to proceed). Check this using the `conda env list` command."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# conda environments:\n",
    "#\n",
    "base                  *  /Users/mbessdl2/anaconda3\n",
    "myenv                    /Users/mbessdl2/anaconda3/envs/myenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This environment will be empty of all packages though (you will find out later how to check this).\n",
    "\n",
    "We can also specify the packages we want to install when creating an environment. When this is done conda will work out all the necessary supporting packages for you. For example:\n",
    "\n",
    "```bash\n",
    "conda create --name myenv numpy ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will list the packages which are to be downloaded, and those which will be installed, before asking if you wish to continue. Press `[y]` to continue, and then check that the environment exists using `conda env list` as before."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='background:#B1E0A8; padding:10px'>\n",
    "<H4> Challenge </H4>\n",
    "\n",
    "Create the environment that we will use for the rest of session. It should be called `python_adv`, and needs to contain the packages:\n",
    "`jupyter`,`spyder`,`numpy`,`pandas`,`matplotlib`,and `astropy`.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activating and Deactivating Environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you can work in an environment you must activate it. This can be done using:\n",
    "\n",
    "`conda activate myenv`\n",
    "\n",
    "or (if your shell has not been configured to use `conda activate`)\n",
    "\n",
    "`. activate myenv`\n",
    "\n",
    "Once this done you should see the name of the environment you are using in brackets before your command prompt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you've finished working in an environment you can exit it using:\n",
    "\n",
    "`conda deactivate`\n",
    "\n",
    "<b>Note:</b> you should not put the environment name after this command (if you do then you will be prompted to try again without including the environment name)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Managing Packages within an Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To examine the packages installed in environment `myenv` use the command:\n",
    "\n",
    "```bash\n",
    "conda list --name myenv\n",
    "```\n",
    "(if the environment has been activated, as shown above, you can leave out `--name myenv`).\n",
    "\n",
    "This will show you the names, version numbers, builds, and source channel for each package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To install the `numpy` package in environment `myenv` use:\n",
    "\n",
    "`conda install --name myenv numpy`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Note:</b> If you have not added a channel to the channel list you can still install packages from it using `--channel`:\n",
    "\n",
    "`conda install --name myenv numpy --channel conda-forge`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Note:</b> It is good practice, when creating a new environment, to install all packages you want when it is created. This is because conda is better at ensuring there are no conflicts in package requirements when packages are all installed together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To update the `numpy` package use:\n",
    "\n",
    "`conda update --name myenv numpy`\n",
    "\n",
    "To update all packages in the `myenv` environment use:\n",
    "\n",
    "`conda update --name myenv --all`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To uninstall the `numpy` package from the `myenv` environment:\n",
    "\n",
    "`conda uninstall --name myenv numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='background:#B1E0A8; padding:10px'>\n",
    "<H4> Challenge </H4>\n",
    "\n",
    "Did the `conda uninstall numpy` remove every package that the `conda install numpy` command installed? If not, what packages were left, and why do you think they were left?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there is a specific version of a package that you want to install, this can be specified by putting the version number after the package name (separated by =):\n",
    "\n",
    "```bash\n",
    "conda install --name myenv python=2\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style='background:#B1E0A8; padding:10px'>\n",
    "<H4> Challenge </H4>\n",
    "\n",
    "What changes are there to the rest of the installed packages when you change the version of python?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reproducing Environments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can make an exact copy of an environment using:\n",
    "\n",
    "```bash\n",
    "conda create --name myclone --clone myenv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful if you are wanting to make modifications to an existing environment, without losing a setup that you know works. However, to make a record of your setup, which can then be shared with your work, you need to save an explicit list of the installed packages:\n",
    "\n",
    "```bash\n",
    "conda list --explicit > spec-file.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use this file to create an environment, or install packages within an existing environment, you add `--file spec-file.txt` to the command. E.g.:\n",
    "\n",
    "```bash\n",
    "conda create --name myenv2 --file spec-file.txt\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}