notebooks/2_InputOutput.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input Output\n",
    "\n",
    "This notebooks gives you some intuition of functions and Python packages, and then discusses input/output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions\n",
    "\n",
    "Functions are constructs that allows us to organize portions of code more than once in a program. \n",
    "\n",
    "The alternative way to obtain the same results without functions would be to copy the same portion of code every time it is needed. \n",
    "\n",
    "Functions in Python are defined by a `def` statement, following this template:\n",
    "\n",
    "```python\n",
    "def function_name(parameters):\n",
    "    \"\"\"\n",
    "    docstring\n",
    "    \"\"\"\n",
    "    function_body\n",
    "    return result\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The list of the parameters required by the function is reported between round brackets right after the name of the function. Each function may have **zero or more** parameters. When a function is called, its parameters are called **arguments**.\n",
    ">\n",
    "> The (optional) documentation string should be placed immediately after the function definition. There are many way to format your **docstring**, [PEP 287](https://www.python.org/dev/peps/pep-0257/) recommends reStructuredText, but more formats are available. See [this tutorial](http://daouzli.com/blog/docstring.html) for an introduction to the topic.\n",
    ">\n",
    "> The **indented** function body contains all the statements that are executed every time the function is called. When a `return` statement is executed, the function exits and its output is the argument of the `return` statement. \n",
    ">\n",
    "> When there is no return statement in the body function, or when a return statement with no arguments is executed, the function  returns `None`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, the following function calculates the number of characters in a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chars(s):\n",
    "    \"\"\"\n",
    "    Calculates the number of characters in a string\n",
    "    \"\"\"\n",
    "    if not type(s) is str:\n",
    "        return \"This is not a string!\"\n",
    "    r = len(s)\n",
    "    return r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The docstring is saved into a  `__doc__` variable and can be accessed by using the `help()` function or the IPython `?`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Calculates the number of characters in a string\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "# don't use this, it is just to make the point\n",
    "print(chars.__doc__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function chars in module __main__:\n",
      "\n",
      "chars(s)\n",
      "    Calculates the number of characters in a string\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# use one of this two\n",
    "help(chars)\n",
    "chars?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to execute the code included in a function, you have to **call the function**, either in your script or in the interactive shell. For instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars(\"voodoo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is not a string!'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars(1979)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "\n",
    "A function can receive any number of parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def higher(n1, n2, n3):\n",
    "    \"\"\"\n",
    "    find the higher of three numbers\n",
    "    \"\"\"\n",
    "    if n1 > n2 and n2 >= n3:\n",
    "        return n1\n",
    "    if n2 >= n3:\n",
    "        return n2\n",
    "    else:\n",
    "        return n3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a parameter can be passed either by position\n",
    "higher(4, 2, 8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or by name\n",
    "higher(n3 = 8, n1 = 4, n2 = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optional Parameters\n",
    "\n",
    "In some situation it may be useful to have a default parameter value, that is used when a call leaves an arguments **unspecified**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def higher(n1, n2 = 0, n3 = 0):\n",
    "    \"\"\"\n",
    "    find the higher of three numbers\n",
    "    \"\"\"\n",
    "    if n1 > n2 and n2 >= n3:\n",
    "        return n1\n",
    "    if n2 >= n3:\n",
    "        return n2\n",
    "    else:\n",
    "        return n3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "higher(9,4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "higher(-6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arbitrary Number of Parameters\n",
    "\n",
    "A different situation is when we want our function to have an unspecified number of parameters. Python functions admit the so-called \"tuple references\", marked by an asterisk `*` in front of the last parameter  (that becomes a tuple)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_params(*params):\n",
    "    print (\"your input:\")\n",
    "    print (params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "your input:\n",
      "('Down from my ceiling', 'Drips great noise', 'It drips on my head through a hole in the roof')\n"
     ]
    }
   ],
   "source": [
    "print_params(\"Down from my ceiling\", \"Drips great noise\", \"It drips on my head through a hole in the roof\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quiz\n",
    "\n",
    "* Write a function that takes a string as input and returns a dictionary of tokens (sequences of characters separated by whitespace) as keys, and the number of times they occur as values. The `split()` method for string might be useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules and Packages\n",
    "\n",
    "Python modules are groupings of related code that are structures as to facilitate its re-use. \n",
    "\n",
    "Physically, modules are `.py` files implementing a set of **functions, classes or variables**, as well as **executable statements**, that can be accessed from other modules by using the `import` command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `import` command can be used both to import **the whole code** of a module, using the following syntax:\n",
    "\n",
    "```python\n",
    "import module\n",
    "```\n",
    "\n",
    "or just **specific attributes** (one or more functions, variables, classes or a combination of these) with the following syntax:\n",
    "\n",
    "```python\n",
    "from module import name1, name2, name3\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, if order to know what is our current working directory, we can use the function `getcwd()` available `os` module (see below) in two different ways:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/giovannicolavizza/Dropbox/db_projects/Teaching/UvA_CDH_2020/notebooks'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/giovannicolavizza/Dropbox/db_projects/Teaching/UvA_CDH_2020/notebooks'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from os import getcwd\n",
    "getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can think of a **package** as a structured collection of Python modules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File Input/Output\n",
    "\n",
    "A huge portion of our input data will come from files on disk, and a lot of our work will be saved in memory. So, mastering the art of reading and writing is crucial even in programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code opens a file in our filesystem, prints the first 10 lines and closes the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Hitch Hiker's Guide to the Galaxy \n",
      "\n",
      "\n",
      "\n",
      "for Jonny Brock and Clare Gorst \n",
      "\n",
      "and all other Arlingtoniansfor tea, sympathy, and a sofa\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Far out in the uncharted backwaters of the unfashionable  end  of\n",
      "\n",
      "the  western  spiral  arm  of  the Galaxy lies a small unregarded\n",
      "\n",
      "yellow sun.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "infile = open('data/adams-hhgttg.txt', 'r')\n",
    "for i, line in enumerate(infile):\n",
    "    if i == 10:\n",
    "        break\n",
    "    print(line)\n",
    "infile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The key passage here is the one in which the `open()` function opens a file and return a **file object**, and it is commonly used with the following two parameters: the **name of the file** that we want to open and the **mode**. \n",
    "\n",
    "- **filename**: the name of the file to open\n",
    "\n",
    "- the **mode** in which we want to open a file: the most commonly used values are `r` for **reading** (default), `w` for **writing** (overwriting existing files), and `a` for **appending**. (Note that [the documentation](https://docs.python.org/2/library/functions.html#open) report mode values that may be necessary in some exceptional case)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**IMPORTANT**: every opened file should be **closed** by using the function `close()` before the end of the program, or the file could be unavailable to successive manipulations or for other programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other ways to read a text file, among which the use of the methods `read()` and `readlines()`, that would simplify the above function in:\n",
    "\n",
    "```python\n",
    "infile = open('data/adams-hhgttg.txt', 'rt')\n",
    "text = infile.read()\n",
    "print(text[:10])\n",
    "infile.close()\n",
    "```\n",
    "\n",
    "However, these methods **read the whole file at once**, thus creating huge problems when working with big corpora.\n",
    "\n",
    "In the solution we adopt here the input file is read line by line, so that at any given moment **only one line of text** is loaded into memory. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Writing an output file in Python has a structure that is close to that we're ued in our reading examples above. The main difference are \n",
    "\n",
    "- the specification of the **mode** `w`\n",
    "\n",
    "\n",
    "- the use of the function `write()` for each line of text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outfile = open('stuff/output-test-1.txt', 'w')\n",
    "outfile.write(\"My name is:\")\n",
    "outfile.write(\"John\")\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "> When writing line by line, it's up to you to take care of the **newlines** by appending `\\n` to each line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outfile = open('stuff/output-test-2.txt', 'w')\n",
    "outfile.write(\"My name is:\\n\")\n",
    "outfile.write(\"Alexander\")\n",
    "outfile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The with statement \n",
    "\n",
    "A `with` statement is used to wrap the execution of a block of code.\n",
    "\n",
    "Using this construction to open files has three major advantages:\n",
    "\n",
    "- there is no need to explicitly  close the file (the file is automatically closed as soon as the nested code exits)\n",
    "\n",
    "\n",
    "- the file is closed automatically even when unhandled errors cause the program to crash\n",
    "\n",
    "\n",
    "- the code is way clearer (it is trivial to identify where in the code a file is opened ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Hitch Hiker's Guide to the Galaxy \n",
      "\n",
      "\n",
      "\n",
      "for Jonny Brock and Clare Gorst \n",
      "\n",
      "and all other Arlingtoniansfor tea, sympathy, and a sofa\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Far out in the uncharted backwaters of the unfashionable  end  of\n",
      "\n",
      "the  western  spiral  arm  of  the Galaxy lies a small unregarded\n",
      "\n",
      "yellow sun.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# that's how I usually open files\n",
    "with open(\"data/adams-hhgttg.txt\", \"r\") as infile:\n",
    "    for i, line in enumerate(infile):\n",
    "        if i == 10:\n",
    "            break\n",
    "        print(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looping through folders and files\n",
    "\n",
    "We can use the `os` module to loop through a folder and load multiple files in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "gutenberg_books = dict()\n",
    "\n",
    "for root, dirs, files in os.walk(\"data/gutenberg-extension\"):\n",
    "    for file in files:\n",
    "        gutenberg_books[file] = open(os.path.join(root,file)).read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['doyle-sherlock.txt', 'stoker-dracula.txt', 'README', 'shelley-frankestein.txt', 'austen-pride.txt', 'joyce-dubliners.txt'])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gutenberg_books.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "﻿[The Adventures of Sherlock Holmes by Arthur Conan Doyle]\n",
      "\n",
      "ADVENTURE I. A SCANDAL IN BOHEMIA\n",
      "\n",
      "I.\n",
      "\n",
      "To Sherlock Holmes she is always THE woman. I have seldom heard\n",
      "him mention her under any other name. In his eyes she eclipses\n",
      "and predominates the whole of her sex. It was not that he felt\n",
      "any emotion\n"
     ]
    }
   ],
   "source": [
    "print(gutenberg_books['doyle-sherlock.txt'][:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1.\n",
    "\n",
    "The [factorial](https://en.wikipedia.org/wiki/Factorial) of an integer $n$, defined as:\n",
    "\n",
    "$$\n",
    "n! = \\begin{cases}\n",
    "               1               & n = 1\\\\\n",
    "               n * (n-1)! & \\text{n > 1}\n",
    "           \\end{cases}\n",
    "$$\n",
    "\n",
    "is the product of all positive integers less than or equal to $n$. For example:\n",
    "\n",
    "$$4! = 4 * 3 * 2 * 1$$\n",
    "\n",
    "$$3! = 3 * 2 * 1$$\n",
    "\n",
    "The factorial operation can be implemented in Python both as a recursive function and as an iterative functions. \n",
    "\n",
    "Write one factorial function picking the approach you prefer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2.\n",
    "\n",
    "Read the file `data/adams-hhgttg.txt` and:\n",
    "\n",
    "- Count the number of lines in the file\n",
    "\n",
    "- Count the number of non-empty lines\n",
    "\n",
    "- Read each line of the input file, remove its newline character and write it to file `stuff/adams-output.txt`\n",
    "\n",
    "- Compute the average number of alphanumeric characters per line\n",
    "\n",
    "- Identify all the unique words used in the text (no duplicates!) and write them in a text file called `stuff/lexicon.txt` (one word per line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "\n",
    "with open(\"stuff/lexicon.txt\", \"w\") as infile:\n",
    "    infile.write(\"something\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}