completed/basics_complete_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's search some text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You already know the components of programming. You have been exercising the reasoning programming relies on for your entire life, probably without even realizing it. Programming is just a way to take the logic you already use on a daily basis and express it in a way a computer can understand and act upon.\n",
    "\n",
    "It's just learning how to write in a different language.\n",
    "\n",
    "One very important disclaimer before we start doing just that: **Nobody memorizes this stuff**. We all have to look stuff up all the time. We don’t expect you to memorize it, either. Ask questions. Ask us to review things we’ve already told you.\n",
    "\n",
    "(Most of us ask questions we've asked before daily — we just ask them of Google.)\n",
    "\n",
    "Now for some code. Let's say you want to search 130,000 lines of text for certain terms -- which are most common, how frequently do they occur, how often are they used in a way that's concentrated, which might indicate places you want to look more closely.\n",
    "\n",
    "No person wants to do that by hand. And people are bad at precisely that kind of work. But it's perfect for a computer.\n",
    "\n",
    "That length happens to correspond to The Iliad. In groups of two or three, think about a book like that. In your groups, figure out two things:\n",
    "\n",
    "- A whole text is made up of what parts?\n",
    "- What is the first thing you need to know to begin to search a file of text? The second thing? Third thing?\n",
    "\n",
    "Roughly, the steps might look like this:\n",
    "1. open the file\n",
    "2. break the file into individual lines\n",
    "3. begin to examine each line\n",
    "4. if the line contains the term you're looking for, capture that\n",
    "5. does anything else about the line interest you? Is your term there multiple times, for instance?\n",
    "6. if none of your conditions are met, keep going\n",
    "\n",
    "This is a program! See, you already know how to program. Now let’s take a minute to step through this the way a computer might.\n",
    "\n",
    "In Python and other languages, we use the concept of **variables** to store values. A variable is just an easy way to reference a value we want to keep track of. So if we want to store a search term and how often our program has found it, we probably want to assign them to variables to keep track of them.\n",
    "\n",
    "Create a string that represents the search term we want to find and assign it to a variable `search_term`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This could just as easily be 'horse' or 'Helen' or 'Agamemnon' or `sand` -- or 'Trojan'\n",
    "search_term = 'Achilles'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now tell Python where the file is and to open it. The path to the file can be one variable: `file_location`. And we can use that to open the file itself and store that opened file in a variable `file_to_read`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_location = '../basics/data/iliad.txt'\n",
    "file_to_read = open(file_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a variable `term_count` containing the integer value of how many times we've seen it in the text. So far, that's zero"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many times our search_term has occurred\n",
    "term_count = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So any time we want to check to see how many times we've seen our `search_term` or check where our `file_location` is, we can use these variables instead of typing out the card value!\n",
    "\n",
    "If you forget what one of the variables is set to, you can `print` it out. Let's also make a comment to remind us of what this variable does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many lines contain at least two of our search_term\n",
    "multi_term_line = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When it's on multiple lines, note the line number and collect all relevant line numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# so far, zero\n",
    "line_number = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# an empty list we hope to fill with lines we might want to explore in greater detail\n",
    "line_numbers_list = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that a string is just a series of characters, not a word or a sentence.\n",
    "So you can represent those characters as lowercase or uppercase, or see whether the string starts with or ends with specific things.\n",
    "Try to make our `search_term` lowercase:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lowercase because of line.lower() below -- we want to compare lowercase only against lowercase\n",
    "search_term = search_term.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've decided to standaradize our strings for comparison by making them lowercase. Cool. Now we need to do the comparing. Our open file is ready to explore. And to do that, we'll need a `for loop`. The loop will assign each line to a variable on the fly, then reference that variable to do stuff we tell it to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# begin looping line by line through our file\n",
    "for line in file_to_read:\n",
    "    # increment the line_number\n",
    "    line_number += 1\n",
    "    # make the line lowercase\n",
    "    line = line.lower()\n",
    "    # check whether our search_term is in the line\n",
    "    if search_term in line:\n",
    "        # if it is, use a tool Python gives us to count how many times\n",
    "        # and add that to the number of times we've seen already\n",
    "        term_count += line.count(search_term)\n",
    "        # if it has counted more than one in the line, we know it's there multiple times;\n",
    "        # keep track of that, too\n",
    "        if line.count(search_term) > 1:\n",
    "            # print(line)\n",
    "            multi_term_line += 1\n",
    "            # and add that to the list using a tool Python give us for lists\n",
    "            line_numbers_list.append(line_number)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've read through the whole file, but our variable `search_term` still holds the open file. Let's close it explicitly, using a tool Python gives us on files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_to_read.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's set some language so we can make our data more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if this value is zero or more than one or (somehow) negative, this word should be plural\n",
    "if multi_term_line != 1:\n",
    "    times = 'times'\n",
    "else:\n",
    "    times = 'time'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can drop our variables into a sentence to help better make sense of our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "achilles was in The Iliad 418 times\n"
     ]
    }
   ],
   "source": [
    "# we can do it by adding the strings to one another like this:\n",
    "print(search_term + ' was in The Iliad ' + str(term_count) + ' times')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "achilles was in The Iliad 418 times\n"
     ]
    }
   ],
   "source": [
    "# or we can use what Python calls `f-strings`, which allow us to drop variables directly into a string;\n",
    "# doing it this way means we don't have to keep track as much of wayward spaces or\n",
    "# whether one of our variables is an integer\n",
    "print(f'{search_term} was in The Iliad {term_count} times')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And how often was our term on the same line? Which lines?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It was on the same line multiple times 5 times\n",
      "it was on lines [11674, 13351, 16400, 20182, 20640] multiple times\n"
     ]
    }
   ],
   "source": [
    "print(f'It was on the same line multiple times {multi_term_line} {times}')\n",
    "print(f'it was on lines {line_numbers_list} multiple times')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to analyze text is frequency of the words it contains. There may be insights there about what's important, or they may be terms you want to use in a FOIA request:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make a dictionary to keep track of how often all the words in The Iliad occur:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a dictionary to collect words as keys and number of occurrences as the value\n",
    "most_common_words = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, we closed our file, so we'll need to open it again and set it to a variable. This is a time when making the file path its own variable saves us the trouble of finding it again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_to_read = open(file_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we'll need to loop through the lines in the file. This time, we care about inspecting each individual word -- not just whether a term is somewhere in the line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "for line in file_to_read:\n",
    "    line = line.lower()\n",
    "    # make a list of words out of each line using a Python tool for lists\n",
    "    word_list = line.split()\n",
    "    # and loop over each word in the line\n",
    "    for word in word_list:\n",
    "        # if a word is not yet in the most_common_words dictionary, add it\n",
    "        # if the word is there already, increase the count by 1\n",
    "        most_common_words[word] = most_common_words.get(word, 0) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we now have the words we want to analyze further in a dictionary -- so we don't need that file anymore. So let's close it\n",
    "file_to_read.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up our baseline variables -- where we'll want to store the top values we're looking for. We'll need one variable for `most_common_word`, set to `None`, and another for `highest_count`, set to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_common_word = None\n",
    "highest_count = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a dictionary of every word in The Iliad. And we can spot-check the number of times any word we'd like has appeared by using the word as the `key` to access that (just remember we made all the `key`s lowercase):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "103\n",
      "37\n",
      "279\n",
      "28\n",
      "1\n",
      "215\n"
     ]
    }
   ],
   "source": [
    "print(most_common_words[\"homer\"])\n",
    "print(most_common_words['paris'])\n",
    "print(most_common_words['hector'])\n",
    "print(most_common_words['helen'])\n",
    "print(most_common_words['sand'])\n",
    "print(most_common_words['trojan'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word, count in most_common_words.items():\n",
    "    # as we go through the most_common_words dictionary,\n",
    "    # set the word and the count that's the biggest we've seen so far\n",
    "    if highest_count is None or count > highest_count:\n",
    "        most_common_word = word\n",
    "        highest_count = count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The most common word in The Iliad is: the\n",
      "It is in The Iliad 15447 times\n",
      "Wow! How cool is that?\n"
     ]
    }
   ],
   "source": [
    "print(f'The most common word in The Iliad is: {most_common_word}')\n",
    "print(f'It is in The Iliad {highest_count} times')\n",
    "print('Wow! How cool is that?')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you’ve just seen, programming can be pretty tedious when you’re trying to break tasks down. So now that you’ve gotten a little bit of a taste for what writing a program is like, let’s dive into some of [the nitty-gritty basics](../completed/basics_reference_completed.ipynb), like how you strip whitespace from a string and what happens when you mix a float and an integer.\n",
    "\n",
    "That sounds like a lot of fun. It must. It does. We promise.\n",
    "\n",
    "Onward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}