acquire_lesson.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Acquisition with Web Scraping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup \n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First make the request. The response is a bunch of html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html lang=\"en\">\\n<head>\\n    <meta charset=\"UTF-8\">\\n    <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1.0\">\\n    <title>News Example Page</title>\\n    <link href=\"https://unpkg.com/tailwindcss@^2/dist/tailwind.min.css\" rel=\"stylesheet\" />\\n    <link rel=\"stylesheet\" href=\"https://cdn.jsdelivr.net/npm/bootstrap-icons@1.4.1/font/bootstrap-icons.css\" />\\n</head>\\n<body class=\"mx-auto max-w-screen-lg pb-32\">\\n    \\n<h1 class=\"my-5 text-4xl text-center\">News!</h1>\\n<div class=\"my-5 text-red-800 px-5 py-3 bg-red-100 font-bold\">\\n    <p>\\n        <i class=\"bi bi-exclamation-circle text-xl\"></i>\\n        All data on this page is strictly for demonstration purposes and fake.\\n    </p>\\n</div>\\n<div class=\"grid gap-y-12\">\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">part population head</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 2009-10-02 </p>\\n                <p class=\"text-right\">By Judith Carr </p>\\n            </div>\\n            <p>Assume fight by blue. Would unit smile bill movement site call rich. Manager beautiful fire drive.\\nAfter free begin east. Eye bad culture week arm because. Experience news argue policy behind.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">say lawyer again</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 1991-03-21 </p>\\n                <p class=\"text-right\">By Mark Jackson </p>\\n            </div>\\n            <p>Congress radio risk human. Example something local degree management to national.\\nMiss last enjoy skill. Employee choice less party foot arrive state budget. Throughout last write baby and partner rate. Possible include term.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">less administration finally</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 2013-10-25 </p>\\n                <p class=\"text-right\">By Patrick Vasquez </p>\\n            </div>\\n            <p>According bill policy must require majority product. Against short recently present spring. Hotel result task spring. Sort meeting physical pass generation.\\nThan star out financial number reveal. News word important. School future him then food save ago. Against represent high hot talk.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">idea much example</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 1982-08-11 </p>\\n                <p class=\"text-right\">By Barry Brown </p>\\n            </div>\\n            <p>Never foot anyone approach choose. Today air employee trial staff red. Statement speak them same firm magazine maybe.\\nImportant upon parent week watch wide. Include own card care enter debate.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">interesting could real</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 1972-08-14 </p>\\n                <p class=\"text-right\">By Erin Bauer </p>\\n            </div>\\n            <p>Remain bring describe not energy set. Realize likely let.\\nHe full learn material peace. Author certainly million garden. Help girl spend institution quality.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">treat system stop</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 2014-02-19 </p>\\n                <p class=\"text-right\">By Monica Smith </p>\\n            </div>\\n            <p>Office top lawyer bring read choose some chair. Forward recent sister dinner.\\nMind suddenly deal available lose cost. Study tree charge deal company. Visit example strategy over cup exist decision.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">believe after physical</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 1981-03-20 </p>\\n                <p class=\"text-right\">By David Fox </p>\\n            </div>\\n            <p>Speak baby hour rule his boy chair big. Campaign lay wait affect civil. Human understand the alone present travel well.\\nPass put long hope senior product which.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">financial ground remember</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 2008-05-29 </p>\\n                <p class=\"text-right\">By Danny Rodriguez </p>\\n            </div>\\n            <p>One major American writer institution expect much. Sit challenge if machine. Past evidence federal blood. Talk front marriage author huge sign.\\nFather believe church impact air even course. Yourself science radio president Congress.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">million event practice</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 2010-06-23 </p>\\n                <p class=\"text-right\">By Julie Burch </p>\\n            </div>\\n            <p>Dream child trip song fall do. Group feel build relationship themselves.\\nLeft election reduce else story be law. So sort property just keep. If newspaper drug agreement pick car energy.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">loss tonight democratic</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 1994-07-08 </p>\\n                <p class=\"text-right\">By Christina Carroll </p>\\n            </div>\\n            <p>Strategy do each series get production. Full behind forward deal program indicate soon. Four use relationship because tend.\\nClearly husband buy east. Outside rise describe through reflect produce remember.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">three material become</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 1974-04-21 </p>\\n                <p class=\"text-right\">By Brittany Turner </p>\\n            </div>\\n            <p>How person phone television simple practice church. Allow teacher station leave. Professional wrong will.\\nCareer more peace run suggest join occur. Whatever cost something.</p>\\n        </div>\\n    </div>\\n    \\n    <div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\\n        <img src=\"/static/placeholder.png\" />\\n        <div class=\"col-span-3 space-y-3 py-3\">\\n            <h2 class=\"text-2xl text-green-900\">know they deep</h2>\\n            <div class=\"grid grid-cols-2 italic\">\\n                <p> 2000-06-08 </p>\\n                <p class=\"text-right\">By William Velazquez </p>\\n            </div>\\n            <p>Take institution require she. Officer when technology stand suffer turn admit medical. Son among Mrs mouth professor easy. Evening history success audience truth sometimes.\\nMr bad prepare dinner. Everyone large everything agreement fly race billion tax.</p>\\n        </div>\\n    </div>\\n    \\n</div>\\n\\n</body>\\n</html>'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = requests.get('https://web-scraping-demo.zgulde.net/news')\n",
    "html = response.text\n",
    "html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make more sense of that html with the beautiful soup library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<!DOCTYPE html>\n",
       "<html lang=\"en\">\n",
       "<head>\n",
       "<meta charset=\"utf-8\"/>\n",
       "<meta content=\"IE=edge\" http-equiv=\"X-UA-Compatible\"/>\n",
       "<meta content=\"width=device-width, initial-scale=1.0\" name=\"viewport\"/>\n",
       "<title>News Example Page</title>\n",
       "<link href=\"https://unpkg.com/tailwindcss@^2/dist/tailwind.min.css\" rel=\"stylesheet\"/>\n",
       "<link href=\"https://cdn.jsdelivr.net/npm/bootstrap-icons@1.4.1/font/bootstrap-icons.css\" rel=\"stylesheet\"/>\n",
       "</head>\n",
       "<body class=\"mx-auto max-w-screen-lg pb-32\">\n",
       "<h1 class=\"my-5 text-4xl text-center\">News!</h1>\n",
       "<div class=\"my-5 text-red-800 px-5 py-3 bg-red-100 font-bold\">\n",
       "<p>\n",
       "<i class=\"bi bi-exclamation-circle text-xl\"></i>\n",
       "        All data on this page is strictly for demonstration purposes and fake.\n",
       "    </p>\n",
       "</div>\n",
       "<div class=\"grid gap-y-12\">\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">part population head</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 2009-10-02 </p>\n",
       "<p class=\"text-right\">By Judith Carr </p>\n",
       "</div>\n",
       "<p>Assume fight by blue. Would unit smile bill movement site call rich. Manager beautiful fire drive.\n",
       "After free begin east. Eye bad culture week arm because. Experience news argue policy behind.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">say lawyer again</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 1991-03-21 </p>\n",
       "<p class=\"text-right\">By Mark Jackson </p>\n",
       "</div>\n",
       "<p>Congress radio risk human. Example something local degree management to national.\n",
       "Miss last enjoy skill. Employee choice less party foot arrive state budget. Throughout last write baby and partner rate. Possible include term.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">less administration finally</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 2013-10-25 </p>\n",
       "<p class=\"text-right\">By Patrick Vasquez </p>\n",
       "</div>\n",
       "<p>According bill policy must require majority product. Against short recently present spring. Hotel result task spring. Sort meeting physical pass generation.\n",
       "Than star out financial number reveal. News word important. School future him then food save ago. Against represent high hot talk.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">idea much example</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 1982-08-11 </p>\n",
       "<p class=\"text-right\">By Barry Brown </p>\n",
       "</div>\n",
       "<p>Never foot anyone approach choose. Today air employee trial staff red. Statement speak them same firm magazine maybe.\n",
       "Important upon parent week watch wide. Include own card care enter debate.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">interesting could real</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 1972-08-14 </p>\n",
       "<p class=\"text-right\">By Erin Bauer </p>\n",
       "</div>\n",
       "<p>Remain bring describe not energy set. Realize likely let.\n",
       "He full learn material peace. Author certainly million garden. Help girl spend institution quality.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">treat system stop</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 2014-02-19 </p>\n",
       "<p class=\"text-right\">By Monica Smith </p>\n",
       "</div>\n",
       "<p>Office top lawyer bring read choose some chair. Forward recent sister dinner.\n",
       "Mind suddenly deal available lose cost. Study tree charge deal company. Visit example strategy over cup exist decision.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">believe after physical</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 1981-03-20 </p>\n",
       "<p class=\"text-right\">By David Fox </p>\n",
       "</div>\n",
       "<p>Speak baby hour rule his boy chair big. Campaign lay wait affect civil. Human understand the alone present travel well.\n",
       "Pass put long hope senior product which.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">financial ground remember</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 2008-05-29 </p>\n",
       "<p class=\"text-right\">By Danny Rodriguez </p>\n",
       "</div>\n",
       "<p>One major American writer institution expect much. Sit challenge if machine. Past evidence federal blood. Talk front marriage author huge sign.\n",
       "Father believe church impact air even course. Yourself science radio president Congress.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">million event practice</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 2010-06-23 </p>\n",
       "<p class=\"text-right\">By Julie Burch </p>\n",
       "</div>\n",
       "<p>Dream child trip song fall do. Group feel build relationship themselves.\n",
       "Left election reduce else story be law. So sort property just keep. If newspaper drug agreement pick car energy.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">loss tonight democratic</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 1994-07-08 </p>\n",
       "<p class=\"text-right\">By Christina Carroll </p>\n",
       "</div>\n",
       "<p>Strategy do each series get production. Full behind forward deal program indicate soon. Four use relationship because tend.\n",
       "Clearly husband buy east. Outside rise describe through reflect produce remember.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">three material become</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 1974-04-21 </p>\n",
       "<p class=\"text-right\">By Brittany Turner </p>\n",
       "</div>\n",
       "<p>How person phone television simple practice church. Allow teacher station leave. Professional wrong will.\n",
       "Career more peace run suggest join occur. Whatever cost something.</p>\n",
       "</div>\n",
       "</div>\n",
       "<div class=\"grid grid-cols-4 gap-x-4 border rounded pr-3 bg-green-50 hover:shadow-lg transition duration-500\">\n",
       "<img src=\"/static/placeholder.png\"/>\n",
       "<div class=\"col-span-3 space-y-3 py-3\">\n",
       "<h2 class=\"text-2xl text-green-900\">know they deep</h2>\n",
       "<div class=\"grid grid-cols-2 italic\">\n",
       "<p> 2000-06-08 </p>\n",
       "<p class=\"text-right\">By William Velazquez </p>\n",
       "</div>\n",
       "<p>Take institution require she. Officer when technology stand suffer turn admit medical. Son among Mrs mouth professor easy. Evening history success audience truth sometimes.\n",
       "Mr bad prepare dinner. Everyone large everything agreement fly race billion tax.</p>\n",
       "</div>\n",
       "</div>\n",
       "</div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(html)\n",
    "soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here we can switch between the browser and python and try out different ways of getting different parts of the html document.\n",
    "\n",
    "We can leverage Google Chrome's developer tools by right clicking and choosing \"Inspect\". We can then use this html document inspector to help us with our web scraping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p> 2009-10-02 </p>, <p class=\"text-right\">By Judith Carr </p>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles = soup.select('.grid.grid-cols-4.gap-x-4.border')\n",
    "articles[0].select('.italic')[0].select('p')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bringing it all together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>date</th>\n",
       "      <th>author</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>part population head</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>say lawyer again</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>less administration finally</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>idea much example</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>interesting could real</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>treat system stop</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>believe after physical</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>financial ground remember</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>million event practice</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>loss tonight democratic</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>three material become</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>know they deep</td>\n",
       "      <td>2009-10-02</td>\n",
       "      <td>By Judith Carr</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          title          date           author\n",
       "0          part population head   2009-10-02   By Judith Carr \n",
       "1              say lawyer again   2009-10-02   By Judith Carr \n",
       "2   less administration finally   2009-10-02   By Judith Carr \n",
       "3             idea much example   2009-10-02   By Judith Carr \n",
       "4        interesting could real   2009-10-02   By Judith Carr \n",
       "5             treat system stop   2009-10-02   By Judith Carr \n",
       "6        believe after physical   2009-10-02   By Judith Carr \n",
       "7     financial ground remember   2009-10-02   By Judith Carr \n",
       "8        million event practice   2009-10-02   By Judith Carr \n",
       "9       loss tonight democratic   2009-10-02   By Judith Carr \n",
       "10        three material become   2009-10-02   By Judith Carr \n",
       "11               know they deep   2009-10-02   By Judith Carr "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def process_article(article):\n",
    "    date, author = articles[0].select('.italic')[0].select('p')\n",
    "    return {\n",
    "        'title': article.h2.text,\n",
    "        'date': date.text,\n",
    "        'author': author.text\n",
    "    }\n",
    "\n",
    "pd.DataFrame([process_article(article) for article in articles])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping for People"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get('https://web-scraping-demo.zgulde.net/people', headers={'user-agent': 'Codeup DS Germain'})\n",
    "soup = BeautifulSoup(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = soup.select('.person')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "person = people[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# putting it all together\n",
    "import re\n",
    "\n",
    "\n",
    "def parse_person(person):\n",
    "    name = person.h2.text\n",
    "    # .p finds the first p element; or element with a tag name of \"p\"\n",
    "    quote = person.p.text.strip()\n",
    "    # email\n",
    "    email = person.select('.email')[0].text\n",
    "    # phone\n",
    "    phone = person.select('.phone')[0].text\n",
    "    # address\n",
    "    address = person.select('.address')[0].text.strip()\n",
    "    address = re.sub(r'\\s{2,}', ' ', address)\n",
    "    \n",
    "    return {'name': name, 'quote': quote, 'email': email, 'phone': phone, 'address': address}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>quote</th>\n",
       "      <th>email</th>\n",
       "      <th>phone</th>\n",
       "      <th>address</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Timothy Rodriguez</td>\n",
       "      <td>\"Implemented responsive collaboration\"</td>\n",
       "      <td>mark88@butler.org</td>\n",
       "      <td>5839171255</td>\n",
       "      <td>340 Garcia Freeway East Wendy, WI 71650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Nicholas Hernandez</td>\n",
       "      <td>\"Universal national Graphical User Interface\"</td>\n",
       "      <td>mhenson@hotmail.com</td>\n",
       "      <td>911.637.7841</td>\n",
       "      <td>476 Jennifer Dam Suite 111 Jenniferfort, WI 90023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Edwin Dixon</td>\n",
       "      <td>\"Synergized value-added secured line\"</td>\n",
       "      <td>wadesharon@hotmail.com</td>\n",
       "      <td>(583)418-3031</td>\n",
       "      <td>62947 William Mission Apt. 602 New Jessica, WI...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jose Mcclure</td>\n",
       "      <td>\"Function-based full-range forecast\"</td>\n",
       "      <td>hayesronald@yahoo.com</td>\n",
       "      <td>916.295.2583</td>\n",
       "      <td>8191 Sarah Point East Elizabeth, IL 12068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Angela Golden</td>\n",
       "      <td>\"Sharable grid-enabled knowledge user\"</td>\n",
       "      <td>charris@gmail.com</td>\n",
       "      <td>785-041-7348x5994</td>\n",
       "      <td>348 Jennifer Flats Pattonstad, ND 11761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Kristen Silva</td>\n",
       "      <td>\"Face-to-face clear-thinking interface\"</td>\n",
       "      <td>andreasilva@vargas-owens.com</td>\n",
       "      <td>(190)599-5854</td>\n",
       "      <td>08554 Lawrence Rapids East Julia, MD 42852</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Philip Fields</td>\n",
       "      <td>\"Persevering analyzing intranet\"</td>\n",
       "      <td>mhuffman@davis-ford.net</td>\n",
       "      <td>(513)818-7492</td>\n",
       "      <td>535 David Club West Kyleland, SC 63892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Alan Hicks</td>\n",
       "      <td>\"Phased exuding alliance\"</td>\n",
       "      <td>josephmanning@foster-alexander.com</td>\n",
       "      <td>+1-113-389-6231x17375</td>\n",
       "      <td>2714 Wendy Overpass Port Christine, NV 12276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Robin Jackson</td>\n",
       "      <td>\"Extended responsive artificial intelligence\"</td>\n",
       "      <td>jonathan52@nielsen.com</td>\n",
       "      <td>001-389-161-6925x05185</td>\n",
       "      <td>636 Oconnell Extension New Erin, NY 07837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Eric Phillips</td>\n",
       "      <td>\"Object-based analyzing hardware\"</td>\n",
       "      <td>riveratravis@smith.com</td>\n",
       "      <td>264.619.7899</td>\n",
       "      <td>5268 Adam Park Lake Benjamin, NE 39068</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 name                                          quote  \\\n",
       "0   Timothy Rodriguez         \"Implemented responsive collaboration\"   \n",
       "1  Nicholas Hernandez  \"Universal national Graphical User Interface\"   \n",
       "2         Edwin Dixon          \"Synergized value-added secured line\"   \n",
       "3        Jose Mcclure           \"Function-based full-range forecast\"   \n",
       "4       Angela Golden         \"Sharable grid-enabled knowledge user\"   \n",
       "5       Kristen Silva        \"Face-to-face clear-thinking interface\"   \n",
       "6       Philip Fields               \"Persevering analyzing intranet\"   \n",
       "7          Alan Hicks                      \"Phased exuding alliance\"   \n",
       "8       Robin Jackson  \"Extended responsive artificial intelligence\"   \n",
       "9       Eric Phillips              \"Object-based analyzing hardware\"   \n",
       "\n",
       "                                email                   phone  \\\n",
       "0                   mark88@butler.org              5839171255   \n",
       "1                 mhenson@hotmail.com            911.637.7841   \n",
       "2              wadesharon@hotmail.com           (583)418-3031   \n",
       "3               hayesronald@yahoo.com            916.295.2583   \n",
       "4                   charris@gmail.com       785-041-7348x5994   \n",
       "5        andreasilva@vargas-owens.com           (190)599-5854   \n",
       "6             mhuffman@davis-ford.net           (513)818-7492   \n",
       "7  josephmanning@foster-alexander.com   +1-113-389-6231x17375   \n",
       "8              jonathan52@nielsen.com  001-389-161-6925x05185   \n",
       "9              riveratravis@smith.com            264.619.7899   \n",
       "\n",
       "                                             address  \n",
       "0            340 Garcia Freeway East Wendy, WI 71650  \n",
       "1  476 Jennifer Dam Suite 111 Jenniferfort, WI 90023  \n",
       "2  62947 William Mission Apt. 602 New Jessica, WI...  \n",
       "3          8191 Sarah Point East Elizabeth, IL 12068  \n",
       "4            348 Jennifer Flats Pattonstad, ND 11761  \n",
       "5         08554 Lawrence Rapids East Julia, MD 42852  \n",
       "6             535 David Club West Kyleland, SC 63892  \n",
       "7       2714 Wendy Overpass Port Christine, NV 12276  \n",
       "8          636 Oconnell Extension New Erin, NY 07837  \n",
       "9             5268 Adam Park Lake Benjamin, NE 39068  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame([parse_person(person) for person in people])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Web Scraping Etiquitte\n",
    "respect the robots.txt file if present\n",
    "\n",
    "Wikipedia: Robots exclusion standard\n",
    "robotstxt.org\n",
    "codeup's robots.txt\n",
    "use a descriptive user agent\n",
    "\n",
    "requests.get('http://example.com', headers={'user-agent': 'codeup data science germain cohort'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "38cca0c38332a56087b24af0bc80247f4fced29cb4f7f437d91dc159adec9c4e"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}