1_data_collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LSA9HAs2fS1D"
   },
   "source": [
    "# Text Analysis: 50 yrs. of Magazine Issues\n",
    "\n",
    "**Author:** [Ryan Parker](https://github.com/rparkr)\n",
    "\n",
    "**Data source:** Data scraped from magazines of The Church of Jesus Christ of Latter-day Saints from 1971-2021. Starting point: [Church Magazines](https://www.churchofjesuschrist.org/study/magazines?lang=eng).\n",
    "\n",
    "**Objective**: understand trends in topics addressed through the content of four magazines produced by [The Church of Jesus Christ of Latter-day Saints](https://churchofjesuschrist.org).\n",
    "\n",
    "In addition to collecting data, I'll analyze the data and implement various machine learning algorithms to understand the data, uncover insights, and make predictions.\n",
    "\n",
    "## Steps\n",
    "1. <strong><span style=\"color:rgb(82, 191, 127)\">Data collection (this notebook)</strong></span>: Gather data through webscraping, lightly process it, and save it to .csv files.\n",
    "2. Topic modeling: assign a topic to each document (article) and label the topics, then append the topic label as a new feature (column) in the dataset.\n",
    "3. Data augmentation: modify and add additional features (columns) to the dataset to enhance its analysis.\n",
    "4. Analysis: explore the dataset through visualization, and make predictions using supervised and unsupervised machine learing algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zs_A4uC-fLuT"
   },
   "source": [
    "# Data scraping\n",
    "**Note:** I reviewed the site's [`robots.txt` page](https://www.churchofjesuschrist.org/robots.txt) and verified that the pages used for this data collection exercise are allowed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import required packages for this section"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\Ryan\\anaconda3\\envs\\stapps\\lib\\site-packages\\tqdm\\auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "# Built-in packages\n",
    "import json                     # parse JSON to Python dictionaries\n",
    "import os                       # file management and other operating system utilities\n",
    "import csv                      # save data to .csv file\n",
    "import re                       # text-based pattern matching\n",
    "import time                     # pause between each webpage request\n",
    "import random                   # sample from a list\n",
    "\n",
    "# Third-party packages\n",
    "import requests                 # send and receive data over HTTP(S)\n",
    "from bs4 import BeautifulSoup   # class used to parse HTML document object model (DOM)\n",
    "from tqdm.auto import tqdm      # progress bars\n",
    "import pandas as pd             # tabular data analysis\n",
    "tqdm.pandas()                   # register df.progress_apply() to add progress bars to DataFrame.apply() methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional alternative to requests, for returning content rendered by JavaScript.\n",
    "# Does not work in a Jupyter notebook, only from a Python script (.py file).\n",
    "\n",
    "# %pip install requests_html\n",
    "# from requests_html import HTMLSession   # requests, with support for rendering JavaScript\n",
    "# See: https://github.com/psf/requests-html\n",
    "#      https://stackoverflow.com/questions/45448994/wait-page-to-load-before-getting-data-with-requests-get-in-python-3/67778709#67778709\n",
    "#      https://stackoverflow.com/questions/8049520/web-scraping-javascript-page-with-python/49865035#49865035\n",
    "# Usage:\n",
    "# session = HTMLSession()\n",
    "# response = session.get(URL_TO_PAGE)\n",
    "# html = response.html.render()\n",
    "# parsed_page = BeautifulSoup(html, features='lxml')\n",
    "# print(parsed_page.text)\n",
    "\n",
    "# Another alternative: Selenium with Chrome in headless mode\n",
    "# See, for example: https://stackoverflow.com/questions/45448994/wait-page-to-load-before-getting-data-with-requests-get-in-python-3/68787500#68787500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q2ksqwiT9ICu"
   },
   "source": [
    "Test sites with interesting structures to account for: \n",
    "* [2017 collection of _Liahona_ issues](https://www.churchofjesuschrist.org/study/magazines/liahona/2017?lang=eng), has Special Issues in addition to standard monthly issues. See also [2018](https://www.churchofjesuschrist.org/study/magazines/liahona/2018?lang=eng) for other examples of special editions. \n",
    "* [August 2021 example](https://www.churchofjesuschrist.org/study/liahona/2021/08/hear-him?lang=eng) of an article with no text content\n",
    "* [Do Not Copy photo notice example](https://www.churchofjesuschrist.org/study/liahona/2018/02/a-release-is-a-beginning-not-an-end?lang=eng)\n",
    "* [Nested category example](https://www.churchofjesuschrist.org/study/liahona/2018/02/october-2017-conference-notebook/how-can-we-bring-the-savior-into-our-lives?lang=eng), where categories form the part of the URL after the month (e.g., `.../liahona/2018/02/additional_category/article?lang=eng`)\n",
    "* [Extra Contents section example](https://www.churchofjesuschrist.org/study/liahona/2010/10/contents?lang=eng)\n",
    "\n",
    "\n",
    "Things to account for: \n",
    "* All links on a Year page, including special editions\n",
    "* All articles, using links from left-side navigation (including nested categories like Youth, Adults, Children, etc.)\n",
    "* Multiple pictures\n",
    "* Author bios, if present (bios are present if you can find the author's name at the beginning of the sidebar text)\n",
    "* Author locations, if present\n",
    "* Author titles, if present (e.g., \"Of the Quorum of the Twelve Apostles\" or \"Church magazines\")\n",
    "* Article summaries, if present\n",
    "* Don't include an entry for an article that has no text; e.g., one that is just a picture and a caption\n",
    "\n",
    "## Dataset features scraped from the web\n",
    "* Magazine (_Liahona_, _The Friend_, _Ensign_, _New Era_)\n",
    "* URL\n",
    "* Year\n",
    "* Month\n",
    "* Issue (month/year combo)\n",
    "* Author name\n",
    "* Author's role\n",
    "* Author bio (if present)\n",
    "* Article source (e.g., \"from a devotional address given at BYU on...\")\n",
    "* Article summary\n",
    "* Article text\n",
    "* Article main topic (uses Python function to generate topic from synonym dictionary)\n",
    "* Article category (Main, Young Adults, Children, etc.)\n",
    "* Word count\n",
    "* Number of links on each page\n",
    "* Image captions\n",
    "* Image descriptions (Alt text) --> actually, alt text is generated by a JavaScript script client-side, so the alt text appears as an empty string ('') in the HTML retrieved by `requests.get()`. For more info, see this [Stack Overflow question](https://stackoverflow.com/questions/53469230/beautifulsoup-not-extracting-image-alt-text)\n",
    "\n",
    "<img alt=\"DevTools window showing that alt text is written by JavaScript after HTML DOM is loaded\" href=\"images/alt_text_attribute_assigned_by_JavaScript.png\" width=300>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data collection methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "kx29NBPY-cGk"
   },
   "outputs": [],
   "source": [
    "class MagazineData():\n",
    "    '''\n",
    "    A class to collect and store magazine data by scraping webpages.\n",
    "    '''\n",
    "    def __init__(self, root_url, magazine_names, magazine_urls):\n",
    "        '''\n",
    "        Create an object that will collect and store magazine data.\n",
    "\n",
    "        Parameters\n",
    "        ---\n",
    "\n",
    "        `root_url`: str\n",
    "        The URL of the main page of the website. Necessary because the links\n",
    "        given throughout the website are relative links that start at the end\n",
    "        of the root URL (e.g., a relative link looks like: '/study/liahona/')\n",
    "\n",
    "        `magazine_names`: list\n",
    "        A list of the names for each magazine. Will be used as the first-level\n",
    "        keys in the returned dictionary. Please ensure that the number of\n",
    "        elements in `magazine_names` matches the number of elements in\n",
    "        `magazine_urls`.\n",
    "        \n",
    "        `magazine_urls`: list\n",
    "        A list of the main URLs for the magainzes, where the main URL is the page\n",
    "        with links to each year for the magazine.\n",
    "        '''\n",
    "        # Set the root_url, magazine_names, and magazine_urls properties of\n",
    "        # the object. Same as self.root_url = root_url, etc. for each parameter.\n",
    "        self.__dict__.update(locals())\n",
    "        self.all_years = None\n",
    "        self.all_issues = None\n",
    "        self.article_data = None\n",
    "        self.article_text = None\n",
    "\n",
    "        # Alternative form\n",
    "        # self.root_url = root_url\n",
    "        # self.magazine_names = magazine_names\n",
    "        # self.magazine_urls = magazine_urls\n",
    "\n",
    "        # Enforce lists for the input parameters\n",
    "        if type(self.magazine_names) == str:\n",
    "            self.magazine_names = list(self.magazine_names)\n",
    "        if type(self.magazine_urls) == str:\n",
    "            self.magazine_urls = list(self.magazine_urls)\n",
    "        \n",
    "        # Ensure lists passed are of equal length\n",
    "        assert len(self.magazine_urls) == len(self.magazine_names), (\n",
    "            f\"The number of items passed to 'magazine_urls' ({len(self.magazine_urls)})\"\n",
    "            f\"\\ndoesn't match the number of items passed to 'magazine_names' ({len(self.magazine_names)}).\"\n",
    "            f\"\\nPlease re-initialize the object with an equal number of items in each list.\"\n",
    "        )\n",
    "\n",
    "\n",
    "\n",
    "    # Collect links to magazines, grouped by year\n",
    "    def get_magazine_years(\n",
    "        self,\n",
    "        return_data: bool=False,\n",
    "        crawl_delay: float=0.1) -> dict:\n",
    "        '''\n",
    "        Creates a dictionary of dictionaries that organizes URLs to the year\n",
    "        page for each magazine.\n",
    "\n",
    "        Parameters\n",
    "        ---\n",
    "\n",
    "        `return_data`: bool, default=False\n",
    "        Whether to return the data that this function collects. Either way, the\n",
    "        data is stored in the `all_years` property of the `MagazineData` object.\n",
    "\n",
    "        `crawl_delay`: float, default=0.1\n",
    "        The number of seconds to pause prior to making each web request. Implemented\n",
    "        to reduce the load on the web server. This delay separates each request by a\n",
    "        minimum of `crawl_delay` seconds, since the code's execution time also creates\n",
    "        a delay between consecutive requests.\n",
    "\n",
    "        Return value\n",
    "        ---\n",
    "        Dictionary of the form:\n",
    "        ```python\n",
    "        {\n",
    "            'Liahona': {\n",
    "                '2021': {'url': 'https://www.example.org/'},\n",
    "                '2020': {'url': 'https://www.example.org/'}\n",
    "            },\n",
    "            'Friend': {\n",
    "                '2021': {'url': 'https://www.example.org/'},\n",
    "                '2020': {'url': 'https://www.example.org/'}\n",
    "            },\n",
    "        }\n",
    "        ```\n",
    "        '''\n",
    "        # Create a dictionary to store the links to each year for each magazine\n",
    "        magazine_years = {name: {} for name in self.magazine_names}\n",
    "\n",
    "        # Start a web session\n",
    "        sesh = requests.session()\n",
    "\n",
    "        # Loop through all provided links\n",
    "        for i, main_link in enumerate(self.magazine_urls):\n",
    "            mag_name = self.magazine_names[i]\n",
    "\n",
    "            # Access the magazine's main page\n",
    "            response = sesh.get(main_link)\n",
    "\n",
    "            # Parse the HTML into a Python-navigable structure using BeautifulSoup.\n",
    "            # For best speed, use the 'lxml' parser. \n",
    "            #  See: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-parser\n",
    "            #  Alternative: built-in parser 'html.parser'\n",
    "            page = BeautifulSoup(response.text, 'lxml')\n",
    "\n",
    "            for link in page.find('section', id='main').find_all('a'):\n",
    "                # Make sure the link is to a full year, not a month from the current year.\n",
    "                # Full years have only 4 characters (the digits of the year), and months\n",
    "                #  of the current year have a month name prefixing the year.\n",
    "                if link.text.isdigit():\n",
    "                    magazine_years[mag_name][link.text] = {'url': self.root_url + link.get('href')}\n",
    "            \n",
    "            # Pause between page requests as a courtesy to reduce server load\n",
    "            time.sleep(crawl_delay)\n",
    "        \n",
    "        # Save data in the all_years property of the object\n",
    "        self.all_years = magazine_years\n",
    "\n",
    "        if return_data:\n",
    "            return magazine_years\n",
    "        else:\n",
    "            return None\n",
    "\n",
    "\n",
    "\n",
    "    # Collect links to magazine issues\n",
    "    def get_magazine_issues(\n",
    "        self,\n",
    "        return_data: bool=False,\n",
    "        force_retrieve: bool=False,\n",
    "        crawl_delay: float=0.1):\n",
    "        '''\n",
    "        Creates a dictionary with the following information for each year of\n",
    "        a magazine:\n",
    "        * 'url': link to a page that groups links for each issue of the year\n",
    "        * 'issues': list of titles of issues for that year (e.g., 'April 2018')\n",
    "        * 'issue_urls': list of links to each issue for that year\n",
    "\n",
    "        Parameters\n",
    "        ---\n",
    "        `return_data`: bool, default=False\n",
    "        Whether to return the data that this function collects. Either way, the\n",
    "        data is stored in the `all_issues` property of the `MagazineData` object.\n",
    "\n",
    "        `force_retrieve`: bool, default=False\n",
    "        If True, will collect the data regardless of whether the data has already\n",
    "        been scraped.\n",
    "\n",
    "        `crawl_delay`: float, default=0.1\n",
    "        The number of seconds to pause prior to making each web request. Implemented\n",
    "        to reduce the load on the web server. This delay separates each request by a\n",
    "        minimum of `crawl_delay` seconds, since the code's execution time also creates\n",
    "        a delay between consecutive requests.\n",
    "\n",
    "        Return value\n",
    "        ---\n",
    "        Dictionary of the form:\n",
    "        ```python\n",
    "        {\n",
    "            'Liahona': {\n",
    "                '2020': {\n",
    "                    'url': 'https://www.example.org/',\n",
    "                    'issues': ['January 2020', 'February 2020'],\n",
    "                    'issue_urls': ['https://www.example.org/', 'https://www.example.org/']\n",
    "                    },\n",
    "                '2021': {\n",
    "                    'url': 'https://www.example.org/',\n",
    "                    'issues': ['January 2021', 'February 2021'],\n",
    "                    'issue_urls': ['https://www.example.org/', 'https://www.example.org/']\n",
    "                    }\n",
    "            },\n",
    "            'Friend': {\n",
    "                '2020': {\n",
    "                    'url': 'https://www.example.org/',\n",
    "                    'issues': ['January 2020', 'February 2020'],\n",
    "                    'issue_urls': ['https://www.example.org/', 'https://www.example.org/']\n",
    "                    },\n",
    "                '2021': {\n",
    "                    'url': 'https://www.example.org/',\n",
    "                    'issues': ['January 2021', 'February 2021'],\n",
    "                    'issue_urls': ['https://www.example.org/', 'https://www.example.org/']\n",
    "                    }\n",
    "            }\n",
    "        }\n",
    "        ```\n",
    "        '''\n",
    "        # Check if the magazine issues have already been collected and saved\n",
    "        if os.path.exists('data/magazine_issues.json') and force_retrieve == False:\n",
    "            with open('data/magazine_issues.json', mode='rt', encoding='utf-8') as jsonfile:\n",
    "                self.all_issues = json.load(jsonfile)\n",
    "            print(f'magazine_issues.json file found; data loaded to `all_issues` attribute of the MagazineData object.')\n",
    "        else:\n",
    "            if self.all_years == None:\n",
    "                print(\"Running the get_magazine_years() function to find URLs for the year pages of each magazine.\")\n",
    "                self.get_magazine_years()\n",
    "            \n",
    "            # Start the dictionary with the data already scraped and saved in the all_years attribute.\n",
    "            magazine_issues = self.all_years\n",
    "\n",
    "            total_mag_years = 0\n",
    "\n",
    "            # Create lists for issue titles and URLs\n",
    "            for mag_name, yrs in magazine_issues.items():\n",
    "                total_mag_years += len(yrs)\n",
    "                for yr in yrs:\n",
    "                    magazine_issues[mag_name][yr]['issues'] = []\n",
    "                    magazine_issues[mag_name][yr]['issue_urls'] = []\n",
    "\n",
    "            # Initialize progress bar that counts the number of issues\n",
    "            progress_bar = tqdm(total=total_mag_years)\n",
    "\n",
    "            # Create a web session\n",
    "            sesh = requests.session()\n",
    "\n",
    "            total_issues = 0\n",
    "\n",
    "            for mag_name, yrs in magazine_issues.items():\n",
    "                for yr, properties in yrs.items():\n",
    "                    progress_bar.set_description(f\"Retrieving issues for: {mag_name}, {yr}\")\n",
    "                    # Get webpage with links to issues for a given year\n",
    "                    response = sesh.get(properties['url'])\n",
    "                    response.encoding = 'utf-8'\n",
    "                    page = BeautifulSoup(response.text, 'lxml')\n",
    "                    for link in page.find('section', id='main').find_all('a'):\n",
    "                        total_issues += 1\n",
    "                        # Store issue titles and URLs in the list for the magazine year\n",
    "                        magazine_issues[mag_name][yr]['issues'].append(link.text)\n",
    "                        magazine_issues[mag_name][yr]['issue_urls'].append(self.root_url + link.get('href'))\n",
    "                    progress_bar.update()\n",
    "\n",
    "                    # Pause between page requests as a courtesy to reduce server load\n",
    "                    time.sleep(crawl_delay)\n",
    "\n",
    "            print(f\"There are a total of {total_issues:,.0f} magazine issues.\")\n",
    "\n",
    "            self.all_issues = magazine_issues\n",
    "\n",
    "            # Save data to a file so it doesn't need to be scraped again\n",
    "            with open('data/magazine_issues.json', mode='wt', encoding='utf-8') as jsonfile:\n",
    "                json.dump(obj=magazine_issues, fp=jsonfile)\n",
    "\n",
    "            if return_data:\n",
    "                return magazine_issues\n",
    "            else:\n",
    "                return None\n",
    "\n",
    "\n",
    "\n",
    "    # Scrape data on articles from each magazine issue\n",
    "    def get_article_data(\n",
    "        self,\n",
    "        save_progress: bool=True,\n",
    "        return_data: bool=False,\n",
    "        crawl_delay: float=0.1,\n",
    "        force_retrieve: bool=False,\n",
    "        continue_last_saved: bool=False):\n",
    "        '''\n",
    "        Scrape article data from all issues in the `all_issues` attribute of the\n",
    "        MagazineData object.\n",
    "        \n",
    "        The resulting data is saved to this `MagazineData` object\n",
    "        as a dictionary of columns (keys) and rows (values, as lists).\n",
    "        After running this function, access the data using the `.article_data`\n",
    "        and `.article_text` attributes of the `MagazineData` object.\n",
    "        \n",
    "        Also saves the data to .csv files called: \"data/article_data.csv\"\n",
    "        and \"data/article_text.csv\" in the current working directory.\n",
    "\n",
    "        **NOTE:** This function will take approximately 12-18 hours to scrape\n",
    "        data for all 2,302 magazine issues released between 1970 and 2020 across\n",
    "        all four magazines. It is best to use the .csv files already scraped:\n",
    "        the data likely won't change since the articles are historical.\n",
    "\n",
    "        On my 2015 Windows 10 laptop with a 2.20GHz Intel Core i5-5200U CPU,\n",
    "        8GB of RAM, and 150Mb/s internet connection, the scraping process\n",
    "        took about 13 hours in total (conducted over multiple sessions).\n",
    "\n",
    "        There are about 67,145 articles in the dataset, and each article\n",
    "        took approximately 0.7 seconds to scrape.\n",
    "\n",
    "\n",
    "        Parameters\n",
    "        ---\n",
    "        `save_progress`: bool, default=True\n",
    "            Whether to save the scraped data to .csv files during the\n",
    "            function's execution or to hold all data in memory until the end.\n",
    "            Due to the relative speed of RAM compared with disk access,\n",
    "            the in-memory option is faster. Saving progress to disk at \n",
    "            each step safeguards against function failure partway at \n",
    "            the expense of addititional computation time.\n",
    "            \n",
    "            Approximately 332MB of data are collected from all magazines\n",
    "            (the Friend, New Era, Liahona, and Ensign) from 1970-2020.\n",
    "            Regardless of the `save_progress` setting, the collected \n",
    "            data is held in-memory in the `.article_data` \n",
    "            and `.article_text` attributes of the `MagazineData` object\n",
    "            \n",
    "        `return_data`: bool, default=False\n",
    "        Whether to return the data that this function collects. Either way, the\n",
    "        data is stored in the `article_data` and `.article_text` attributes\n",
    "        of the `MagazineData` object. If True, returns a DataFrame.\n",
    "\n",
    "        `force_retrieve`: bool, default=False\n",
    "        Whether to force the function to scrape data and save it, even though\n",
    "        `data/article_data.csv` file already exists. Will overwrite existing data.\n",
    "\n",
    "        `crawl_delay`: float, default=0.1\n",
    "        The number of seconds to pause prior to making each web request. Implemented\n",
    "        to reduce the load on the web server. This delay separates each request by a\n",
    "        minimum of `crawl_delay` seconds, since the code's execution time also creates\n",
    "        a delay between consecutive requests.\n",
    "\n",
    "        `continue_last_saved`: bool, default=True\n",
    "        Whether to continue scraping starting with the last scraped article. Useful\n",
    "        because the scraping takes many hours, so this enables a developer to continue\n",
    "        at the last saved point. `force_retrive` must be set to `False` if this\n",
    "        parameter is set to `True`.\n",
    "\n",
    "\n",
    "        Return value\n",
    "        ---\n",
    "        `None`, unless `return_data` is set to `True`, in which case the function returns\n",
    "        a pandas DataFrame with data scraped from each article (one row per article).\n",
    "        '''\n",
    "        \n",
    "        if force_retrieve:\n",
    "            assert not continue_last_saved, (\n",
    "                \"Conflicting arguments: if `force_retrieve` is `True`, `continue_last_saved` cannot also be `True`.\"\n",
    "                + \"\\nPlease set one of the arguments to `False` to proceed.\")\n",
    "\n",
    "        # Dictionary to hold data during function execution.\n",
    "        # Each key is a column, with the value as a list of the rows in that column.\n",
    "        columns = [\n",
    "            'magazine', 'year', 'month', 'issue', 'extra_issue', 'url',\n",
    "            'section', 'series', 'title', 'subtitle', 'source',\n",
    "            'author_name', 'author_role', 'author_bio',\n",
    "            'summary', 'word_count', 'references_count',\n",
    "            'image_alt_text', 'image_captions'\n",
    "        ]\n",
    "\n",
    "        text_columns = ['url', 'text']\n",
    "\n",
    "        if self.all_issues == None:\n",
    "            print(\"Running the get_magazine_issues() function to find issue URLs for each magazine.\")\n",
    "            self.get_magazine_issues()\n",
    "            print(\"Now, running the get_article_data function.\")\n",
    "\n",
    "        # Check if the article data has already been scraped and saved\n",
    "        if os.path.exists('data/article_data.csv') and os.path.exists('data/article_text.csv') and not force_retrieve:\n",
    "            if not self.article_data:\n",
    "                self.article_data = pd.read_csv('data/article_data.csv').to_dict(orient='list')\n",
    "                print(\"article_data.csv file found; data loaded to `article_data` attribute of the MagazineData object.\")\n",
    "            if not self.article_text:\n",
    "                self.article_text = pd.read_csv('data/article_text.csv').to_dict(orient='list')\n",
    "                print(\"article_text.csv file found; data loaded to `article_text` attribute of the MagazineData object.\")\n",
    "            \n",
    "            if continue_last_saved:\n",
    "                print(\"Continuing to scrape article data starting from the last retrieved article.\")\n",
    "            else:\n",
    "                if return_data:\n",
    "                    print(\"Merging article_data and article_text dictionaries and returning a single DataFrame with all data.\")\n",
    "                    return pd.DataFrame({**self.article_data, **self.article_text})\n",
    "                else:\n",
    "                    return None\n",
    "        else:\n",
    "            self.article_data = {col: [] for col in columns}\n",
    "            self.article_text = {col: [] for col in text_columns}\n",
    "            if save_progress:\n",
    "                # Create CSV files and write the header row in each file\n",
    "                with open('data/article_data.csv', mode='wt', encoding='utf-8', newline='') as csvfile:\n",
    "                    csv_writer = csv.writer(csvfile)\n",
    "                    csv_writer.writerow(columns)\n",
    "                with open('data/article_text.csv', mode='wt', encoding='utf-8', newline='') as csvfile:\n",
    "                    csv_writer = csv.writer(csvfile)\n",
    "                    csv_writer.writerow(text_columns)\n",
    "    \n",
    "        # Start a web requests session\n",
    "        sesh = requests.session()\n",
    "\n",
    "        # Headers updated in Jul 2022\n",
    "        headers = {\n",
    "            'Accept': 'text/html,application/xhtml+xml,application/xml,application/json;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',\n",
    "            'Accept-Encoding': 'gzip, deflate, br',\n",
    "            'Accept-Language': 'en-US,en;q=0.9',\n",
    "            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36 Edg/103.0.1264.71',\n",
    "            # 'Referer': 'https://www.churchofjesuschrist.org/study?lang=eng&platform=web'\n",
    "        }\n",
    "        \n",
    "        # Copy the all_issues dictionary to modify it if necessary,\n",
    "        # which enables the continue_last_saved feature.\n",
    "        # The pattern below is: iterate through the all_issues dict (the copy)\n",
    "        # and remove items whose index is less than the index of the last saved item;\n",
    "        # that is, remove from the copy the items that have already been scraped.\n",
    "        all_issues = self.all_issues.copy()\n",
    "        if continue_last_saved:\n",
    "            last_magazine = self.article_data['magazine'][-1]\n",
    "            last_year = str(self.article_data['year'][-1])\n",
    "            last_issue = self.article_data['issue'][-1]\n",
    "\n",
    "            # Remove magazines from the all_issues dictionary if\n",
    "            # all articles from those magazines have already been scraped\n",
    "            del_keys = []\n",
    "            index_last_mag = list(all_issues.keys()).index(last_magazine)\n",
    "            for key in all_issues.keys():\n",
    "                if list(all_issues.keys()).index(key) < index_last_mag:\n",
    "                    del_keys.append(key)\n",
    "            for key in del_keys:\n",
    "                # Note that I am deleting keys (magazines) from the COPY, not the original\n",
    "                del all_issues[key]\n",
    "            \n",
    "            # Remove years for the magazine that have already been scraped.\n",
    "            del_yrs = []\n",
    "            index_last_year = list(all_issues[last_magazine]).index(last_year)\n",
    "            for yr_num in all_issues[last_magazine]:\n",
    "                if list(all_issues[last_magazine]).index(yr_num) < index_last_year:\n",
    "                    del_yrs.append(yr_num) \n",
    "            for yr_num in del_yrs:\n",
    "                del all_issues[last_magazine][yr_num]\n",
    "            \n",
    "            # Remove issues for the magazine that have already been scraped\n",
    "            del_issue_nums = []\n",
    "            index_last_issue = all_issues[last_magazine][last_year]['issues'].index(last_issue)\n",
    "            for i_num, issue in enumerate(all_issues[last_magazine][last_year]['issues']):\n",
    "                if all_issues[last_magazine][last_year]['issues'].index(issue) < index_last_issue:\n",
    "                    del_issue_nums.append(i_num)\n",
    "            # When deleting items from a list using the index number,\n",
    "            # items must be deleted in reverse order; otherwise the indices\n",
    "            # of later elements shift down with each deletion.\n",
    "            for i_num in reversed(del_issue_nums):\n",
    "                del all_issues[last_magazine][last_year]['issues'][i_num]\n",
    "                del all_issues[last_magazine][last_year]['issue_urls'][i_num]\n",
    "\n",
    "        # Count the number of issues\n",
    "        issue_count = 0\n",
    "        for mag_name in all_issues:\n",
    "            for yr_num in all_issues[mag_name]:\n",
    "                issue_count += len(all_issues[mag_name][yr_num]['issues'])\n",
    "        \n",
    "        # When scraping article text within the <div class-\"body-block\"> by\n",
    "        # looping through each direct child element in body_block.findChildren(recursive=False),\n",
    "        # if an element is an <aside>, <figure>, or <figcaption> tag,\n",
    "        # or if an element is a <div> tag with class \"credit\" or \"imageWrapper...\",\n",
    "        # ignore its text since it isn't part of the article's main body text.\n",
    "        tags_to_omit = ['aside', 'figure', 'figcaption']\n",
    "        classes_to_omit = ['credit', re.compile('imageWrapper.*')]\n",
    "\n",
    "        # When scraping image alt text, the text is prefixed\n",
    "        # with \"Image\", which I later strip out when saving the data.\n",
    "        img_prefix_chars = len('Image')\n",
    "        \n",
    "        # Create progress bar\n",
    "        pbar = tqdm(total=issue_count)\n",
    "\n",
    "        month_names = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']\n",
    "\n",
    "        # Loop through each issue in all_issues dict\n",
    "        for mag_name in all_issues:\n",
    "            for yr_num in all_issues[mag_name]:\n",
    "                for n, url in enumerate(all_issues[mag_name][yr_num]['issue_urls']):\n",
    "                    issue = all_issues[mag_name][yr_num]['issues'][n]\n",
    "                    # The postion of the last question mark in the URL\n",
    "                    q_pos = url.rfind('?')\n",
    "                    # The position of the last forward slash\n",
    "                    start_pos = url.rfind('/')\n",
    "                    month_num = url[start_pos + 1 : q_pos]\n",
    "                    if month_num.isdigit():\n",
    "                        month_name = issue.split(' ')[0]\n",
    "                        extra_issue = False\n",
    "                    else:\n",
    "                        extra_issue = True\n",
    "                        if month_num[:2].isdigit():\n",
    "                            month_name = month_names[int(month_num[:2]) - 1]\n",
    "                        else:\n",
    "                            month_name = None\n",
    "\n",
    "                    # Pause between page requests as a courtesy to reduce server load\n",
    "                    time.sleep(crawl_delay)\n",
    "                    # Find all articles in the issue\n",
    "                    response = sesh.get(url=url, headers=headers)\n",
    "                    retry_count = 5\n",
    "                    while not response.ok and retry_count > 0:\n",
    "                        time.sleep(1)\n",
    "                        response = sesh.get(url=url, headers=headers)\n",
    "                        retry_count -= 1\n",
    "                    response.encoding = 'utf-8'\n",
    "                    page = BeautifulSoup(response.text, 'lxml')\n",
    "                    sections = []\n",
    "                    article_urls = []\n",
    "                    section = \"general\"\n",
    "                    try:\n",
    "                        for li in page.find('div', class_='body').find('nav', class_='manifest').find_all('li'):\n",
    "                            # A single HTML element can have multiple classes,\n",
    "                            # so BeautifulSoup returns a list of classes when accessing the \"class\" attribute\n",
    "                            n_n_class = li.next.next.get('class')  # -> list of classes or None\n",
    "                            if n_n_class:\n",
    "                                if any(nnc in ['title'] for nnc in n_n_class):\n",
    "                                    # Update section to match current section\n",
    "                                    section = li.find('p', class_='title').text\n",
    "                            a_url = li.find('a', {'href': True})\n",
    "                            if a_url.text != \"Contents\" and (self.root_url + a_url.get('href') not in article_urls):\n",
    "                                sections.append(section)\n",
    "                                article_urls.append(self.root_url + a_url.get('href'))\n",
    "                            else:\n",
    "                                continue\n",
    "                    except AttributeError:\n",
    "                        # An AttributeError means that either:\n",
    "                        # 1. page is None because BeautifulSoup didn't have any HTML to parse (unlikely because requests retries 5 times if the response fails)\n",
    "                        # 2. there is no <div class=\"body\"> element on the page\n",
    "                        # 3. there is no <nav class=\"manifest\"> (i.e., the \"Contents\" section) within the body of the page\n",
    "                        # For any of those errors, simply skip this issue and move on to the next one.\n",
    "                        continue\n",
    "\n",
    "                    # Loop through each article\n",
    "                    for a_num, a_url in enumerate(article_urls):\n",
    "                        pbar.set_description(\n",
    "                            f\"Working on {mag_name}, \"\n",
    "                            + f\"issue {issue}, \"\n",
    "                            + f\"article {a_num + 1:,d}/{len(article_urls):,d}\")\n",
    "                        # Skip article if its data has already been scraped\n",
    "                        if continue_last_saved:\n",
    "                            if a_url in self.article_data['url']:\n",
    "                                continue\n",
    "                        # Pause between page requests as a courtesy to reduce server load\n",
    "                        time.sleep(crawl_delay)\n",
    "                        # Open webpage\n",
    "                        response = sesh.get(url=a_url, headers=headers, timeout=None)\n",
    "                        retry_count = 5\n",
    "                        while not response.ok and retry_count > 0:\n",
    "                            time.sleep(1)\n",
    "                            response = sesh.get(url=a_url, headers=headers)\n",
    "                            retry_count -= 1\n",
    "                        # Skip article if it still isn't loading\n",
    "                        if not response.ok:\n",
    "                            print(f\"Failed to retrieve article with status code {response.status_code} at URL: {a_url}\")\n",
    "                            continue\n",
    "                        response.encoding = 'utf-8'\n",
    "                        page = BeautifulSoup(response.text, 'lxml')\n",
    "                        body = page.find('div', class_='body')\n",
    "                        # If no body element is found on the page, skip to next article\n",
    "                        if not body:\n",
    "                            continue\n",
    "                        # Sometimes, an article has mutliple body-block div tags. In that case, use the last one (i.e., index [-1])\n",
    "                        # See, for example: https://www.churchofjesuschrist.org/study/liahona/2021/01/unlocking-the-door-to-personal-revelation?lang=eng\n",
    "                        body_block = body.find_all('div', class_='body-block')\n",
    "\n",
    "                        if body_block:\n",
    "                            body_block = body_block[-1]\n",
    "                            # A better approach might be to merge the body-block sections.\n",
    "                            # For example, see:\n",
    "                            # https://stackoverflow.com/questions/50026264/beautifulsoup-combine-consecutive-tags\n",
    "                            # https://stackoverflow.com/questions/35531608/how-can-i-merge-two-beautiful-soup-tags\n",
    "                            \n",
    "                        else:\n",
    "                            # print(f\"No <div class=\\\"body-block\\\"> tag found on page. Could not load text from article: {a_url}\")\n",
    "                            # Skip this article since it has no content\n",
    "                            continue\n",
    "                        \n",
    "                        series = body.find('p', class_='series-title')\n",
    "                        # BeautifulSoup returns None if no element is found.\n",
    "                        # None is a \"falsey\" value that evaluates to False.\n",
    "                        # Thus, we can write \"if series:\" and Python will return True\n",
    "                        # if series is an object (that is, a match was found) or False\n",
    "                        # if series is None (that is, no match was found).\n",
    "                        # pandas treats None as a missing value (NaN), which is the intended behavior\n",
    "                        # in this dataset. See: https://note.nkmk.me/en/python-pandas-nan-none-na/#none-is-also-considered-a-missing-value\n",
    "                        series = series.text.strip() if series else None\n",
    "\n",
    "                        title = body.find(id='title1')\n",
    "                        title = title.text.strip() if title else None\n",
    "                        subtitle = body.find(class_='subtitle')\n",
    "                        subtitle = subtitle.text.strip() if subtitle else None\n",
    "\n",
    "                        byline = body.find('div', class_='byline')\n",
    "                        if byline:\n",
    "                            byline_vals = byline.find_all('p')\n",
    "                            author_name = byline_vals[0].text\n",
    "                            if author_name.startswith(\"By \"):\n",
    "                                # Trim the leading \"By \"\n",
    "                                author_name = author_name[3:]\n",
    "                            author_role = byline_vals[1].text if len(byline_vals) > 1 else None\n",
    "                        else:\n",
    "                            author_name = None\n",
    "                            author_role = None\n",
    "\n",
    "                        author_bio = None\n",
    "                        asides = body_block.find_all('aside', class_='sidebar')\n",
    "                        # asides is a list of elements found. If no elements\n",
    "                        # were found, it will be an empty list.\n",
    "                        # Empty lists (i.e., [] or list()) evaluate to False.\n",
    "                        if asides and author_name:\n",
    "                            for aside_box in asides:\n",
    "                                for p_elem in aside_box.find_all('p'):\n",
    "                                    if author_name in p_elem.text:\n",
    "                                        author_bio = p_elem.text\n",
    "                                        break\n",
    "                        else:\n",
    "                            # Check if there is a \"bio\" class\n",
    "                            bio_div = body.find('div', class_='bio')\n",
    "                            if bio_div:\n",
    "                                author_bio = bio_div.text\n",
    "                        \n",
    "                        # Trim leading or trailing newline characters\n",
    "                        if author_bio:\n",
    "                            author_bio = author_bio.strip()\n",
    "                    \n",
    "                        summary = body.find('p', class_='kicker')\n",
    "                        if summary:\n",
    "                            summary = summary.text\n",
    "                        else:\n",
    "                            # Check if there is an epigraph rather than a \"kicker\"\n",
    "                            summary = body.find('p', class_='epigraph')\n",
    "                            summary = summary.text.strip() if summary else None\n",
    "                        \n",
    "                        # Source describes where the article came from, if adapted\n",
    "                        # from a prior address given by the author.\n",
    "                        source = body.find('div', class_='event')\n",
    "                        source = source.text.strip() if source else None\n",
    "                        \n",
    "                        references_count = len(body_block.find_all('a', class_='scripture-ref'))\n",
    "                        references_count += len(body_block.find_all('a', class_='note-ref'))\n",
    "                        references_count += len(body_block.find_all('a', class_='cross-ref'))\n",
    "\n",
    "                        # Use the .text attribute of the container that holds\n",
    "                        # the image, which is a <div> element with class that begins with \"imageWrapper\"\n",
    "                        images = body.find_all(class_=re.compile('imageWrapper.*'))\n",
    "                        if images:\n",
    "                            image_alt_text = [img.text.strip() for img in images]\n",
    "                        else:\n",
    "                            image_alt_text = None\n",
    "                            \n",
    "                        image_captions = body.find_all('figcaption')\n",
    "                        if image_captions:\n",
    "                            image_captions = [fig.text.strip() for fig in image_captions]\n",
    "                        else:\n",
    "                            image_captions = None\n",
    "                        \n",
    "                        # The code below applies filters to refrain from collecting\n",
    "                        # text that is not part of the article (image captions, section headings, asides).\n",
    "                        text = []\n",
    "                        for ch in body_block.findChildren(recursive=False):\n",
    "                            ch_class = ch.get('class')  # -> a list, or None\n",
    "                            if ch_class:\n",
    "                                skip_class = any(cc in classes_to_omit for cc in ch_class)\n",
    "                            else:\n",
    "                                skip_class = False\n",
    "                            \n",
    "                            if ch.name in tags_to_omit or skip_class:\n",
    "                                continue\n",
    "                            else:\n",
    "                                text.append(ch.text.strip())\n",
    "\n",
    "                        # Combine into a single string, with paragraphs delimitted by newlines.\n",
    "                        text = '\\n'.join(text)\n",
    "                        # Replace multiple newlines with a single newline.\n",
    "                        text = re.sub('[\\n]{2,}', '\\n', text)\n",
    "\n",
    "                        # Remove unwanted text that was included through child elements\n",
    "                        # (e.g., a section that, along with article text, \n",
    "                        # contained text from unwanted elements like images, credits, or asides)\n",
    "                        credit_elems = body_block.find_all('div', class_='credit')\n",
    "                        if credit_elems:\n",
    "                            for cr in credit_elems:\n",
    "                                text = text.replace(cr.text.strip(), '', 1)\n",
    "                        \n",
    "                        aside_elems = body_block.find_all('aside')\n",
    "                        if aside_elems:\n",
    "                            for ae in aside_elems:\n",
    "                                text = text.replace(ae.text.strip(), '', 1)\n",
    "\n",
    "                        if image_alt_text:\n",
    "                            for iat in image_alt_text:\n",
    "                                text = text.replace(iat, '', 1)\n",
    "\n",
    "                        if image_captions:\n",
    "                            for ic in image_captions:\n",
    "                                text = text.replace(ic, '', 1)\n",
    "                        \n",
    "                        # Replace multiple whitespace characters with a single space\n",
    "                        text = re.sub('[\\s]{2,}', ' ', text)\n",
    "                        \n",
    "                        if text:\n",
    "                            # Trim leading newline character, if present\n",
    "                            if text[0] == '\\n':\n",
    "                                text = text[1:]\n",
    "                        # else:\n",
    "                        #     # Skip to next article, since there is no text in this article\n",
    "                        #     continue\n",
    "\n",
    "                        # Convert less-common Unicode characters to simpler ones (e.g., á -> a)\n",
    "                        # text = unicode_to_ascii(text)\n",
    "                        \n",
    "                        word_count = len(text.split()) if text else None\n",
    "\n",
    "                        # Clean image_alt_text by removing the \"Image\" prefix.\n",
    "                        if image_alt_text:\n",
    "                            image_alt_text = [iat[img_prefix_chars:] for iat in image_alt_text]\n",
    "                    \n",
    "\n",
    "                        # Add info to article_data\n",
    "                        self.article_data['magazine'].append(mag_name)\n",
    "                        self.article_data['year'].append(yr_num)\n",
    "                        self.article_data['month'].append(month_name)\n",
    "                        self.article_data['issue'].append(issue)\n",
    "                        self.article_data['extra_issue'].append(extra_issue)\n",
    "                        self.article_data['url'].append(a_url)\n",
    "                        self.article_data['section'].append(sections[a_num])\n",
    "                        self.article_data['series'].append(series)\n",
    "                        self.article_data['source'].append(source)\n",
    "                        self.article_data['title'].append(title)\n",
    "                        self.article_data['subtitle'].append(subtitle)\n",
    "                        self.article_data['author_name'].append(author_name)\n",
    "                        self.article_data['author_role'].append(author_role)\n",
    "                        self.article_data['author_bio'].append(author_bio)\n",
    "                        self.article_data['summary'].append(summary)\n",
    "                        self.article_data['word_count'].append(word_count)\n",
    "                        self.article_data['references_count'].append(references_count)\n",
    "                        self.article_data['image_alt_text'].append('\\n'.join(image_alt_text) if image_alt_text else None)\n",
    "                        self.article_data['image_captions'].append('\\n'.join(image_captions) if image_captions else None)\n",
    "\n",
    "                        self.article_text['url'].append(a_url)\n",
    "                        self.article_text['text'].append(text)\n",
    "\n",
    "                        if save_progress:\n",
    "                            # Write a new row with data scraped from the current article\n",
    "                            values = [self.article_data[col][-1] for col in self.article_data.keys()]\n",
    "                            text_values = [self.article_text[col][-1] for col in self.article_text.keys()]\n",
    "                            # Append data to the .csv files\n",
    "                            with open('data/article_data.csv', mode='a', encoding='utf-8', newline='') as csvfile:\n",
    "                                csv_writer = csv.writer(csvfile)\n",
    "                                csv_writer.writerow(values)\n",
    "                            with open('data/article_text.csv', mode='a', encoding='utf-8', newline='') as csvfile:\n",
    "                                csv_writer = csv.writer(csvfile)\n",
    "                                csv_writer.writerow(text_values)\n",
    "\n",
    "                    # Completed scraping data for all articles in the current issue.\n",
    "                    # Update the progress bar and move to the next issue.\n",
    "                    pbar.update()\n",
    "        \n",
    "        # Completed scraping data from all issuee\n",
    "        # Save to .csv files if files were not already created\n",
    "        if not save_progress:\n",
    "            df_data = pd.DataFrame(self.article_data)\n",
    "            df_text = pd.DataFrame(self.article_text)\n",
    "            df_data.to_csv('data/article_data.csv', index=False)\n",
    "            df_text.to_csv('data/article_text.csv', index=False)\n",
    "        if return_data:\n",
    "            # Return a single DataFrame by merging the dictionaries using unpacking (**)\n",
    "            return pd.DataFrame({**self.article_data, **self.article_text})\n",
    "        else:\n",
    "            return None\n",
    "\n",
    "\n",
    "    # Load already-scraped data to the MagazineData object\n",
    "    def load_article_data(\n",
    "        self,\n",
    "        return_data: bool=False):\n",
    "        '''\n",
    "        Loads data from the .csv files: 'data/article_data.csv' and 'data/article_text.csv'.\n",
    "\n",
    "        If `return_data` is `False` (default), the data is loaded as attributes of the MagazineData\n",
    "        object but no data is returned. If `True`, the data is loaded as attributes of the MagazineData\n",
    "        object _and_ returned as a single pandas DataFrame combining both datasets.\n",
    "        '''\n",
    "\n",
    "        if os.path.exists('data/article_data.csv'):\n",
    "            if not self.article_data:\n",
    "                self.article_data = pd.read_csv('data/article_data.csv').to_dict(orient='list')\n",
    "                print(\"article_data.csv file found; data loaded to `article_data` attribute of the MagazineData object.\")\n",
    "        else:\n",
    "            print(\"Could not find the file 'data/article_data.csv'. Consider running the `get_article_data` method to create the file and scrape article data.\")\n",
    "            return\n",
    "        \n",
    "        if os.path.exists('data/article_text.csv'):\n",
    "            if not self.article_text:\n",
    "                self.article_text = pd.read_csv('data/article_text.csv').to_dict(orient='list')\n",
    "                print(\"article_text.csv file found; data loaded to `article_text` attribute of the MagazineData object.\")\n",
    "        else:\n",
    "            print(\"Could not find the file 'data/article_text.csv'. Consider running the `get_article_data` method to create the file and scrape article text.\")\n",
    "            return\n",
    "        \n",
    "        if return_data:\n",
    "            print(\"Merging article_data and article_text dictionaries and returning a single DataFrame with all data...\")\n",
    "            return pd.DataFrame({**self.article_data, **self.article_text})\n",
    "        else:\n",
    "            return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collect data using the `MagazineData` class\n",
    "This process will likely take 12-15 hours to complete. There are about 2,700 magazine issues and over 67,000 articles. The scraped dataset will be approximately 300 MB.\n",
    "\n",
    "Note that after scraping and saving the data to .csv files, you can safely delete the `MagazineData` instance (here, I've called it `magdata`) to release its memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "ROOT_URL = 'https://www.churchofjesuschrist.org'\n",
    "MAGAZINE_NAMES = ['Liahona', 'Ensign', 'New Era', 'Friend']\n",
    "MAGAZINE_URLS = [\n",
    "    'https://www.churchofjesuschrist.org/study/magazines/liahona',\n",
    "    'https://www.churchofjesuschrist.org/study/magazines/ensign-19712020',\n",
    "    'https://www.churchofjesuschrist.org/study/magazines/for-the-strength-of-youth/new-era-19712020?lang=eng',\n",
    "    'https://www.churchofjesuschrist.org/study/magazines/friend']\n",
    "\n",
    "magdata = MagazineData(\n",
    "    root_url=ROOT_URL,\n",
    "    magazine_names=MAGAZINE_NAMES,\n",
    "    magazine_urls=MAGAZINE_URLS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running the get_magazine_issues() function to find issue URLs for each magazine.\n",
      "magazine_issues.json file found; data loaded to `all_issues` attribute of the MagazineData object.\n",
      "Now, running the get_article_data function.\n",
      "article_data.csv file found; data loaded to `article_data` attribute of the MagazineData object.\n",
      "article_text.csv file found; data loaded to `article_text` attribute of the MagazineData object.\n",
      "Continuing to scrape article data starting from the last retrieved article.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d276f80112904d2b8773fa02300931a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1091 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "magdata.get_article_data(\n",
    "    save_progress=True,\n",
    "    crawl_delay=0.0,\n",
    "    continue_last_saved=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missed articles during scraping due to server error\n",
    "- Failed to retrieve article with status code 500 at URL: https://www.churchofjesuschrist.org/study/ensign/1990/04/portraits/ed-rawley-a-steel-grip-on-family-history\n",
    "- Failed to retrieve article with status code 500 at URL: https://www.churchofjesuschrist.org/study/new-era/2014/09/what-is-the-work-of-salvation\n",
    "- Failed to retrieve article with status code 500 at URL: https://www.churchofjesuschrist.org/study/new-era/2013/10/come-follow-me"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Count of articles with text extracted: 67143\n",
      "Count of articles with data extracted: 67143\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Ryan\\AppData\\Local\\Temp\\ipykernel_10972\\3912052598.py:2: DtypeWarning: Columns (10,13) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  print(f\"Count of articles with data extracted: {len(pd.read_csv('data/article_data.csv').index)}\")\n"
     ]
    }
   ],
   "source": [
    "print(f\"Count of articles with text extracted: {len(pd.read_csv('data/article_text.csv').index)}\")\n",
    "print(f\"Count of articles with data extracted: {len(pd.read_csv('data/article_data.csv').index)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset processing: article section\n",
    "After collecting the data, I noticed that some of the articles with specific sections should be re-classified in the \"general\" section based on their URL pattern. This post-collection data processing will enhance the quality of the dataset.\n",
    "\n",
    "After running this code, I'll update the data collection class so any future runs will correctly classify the section of each magazine article; then I'll move this code to the \"Experiments\" section below.\n",
    "\n",
    "**What I've learned so far**<br>\n",
    "One way to know the section is to open the _contents_ page of each issue (the main URL of the issue) and scrape links to each article from the `<nav class=\"tableOfContents-HYfFr\">` element on the left of the screen. Each article (`<a>` element) will have an attribute that identifies its connection to the header it falls under. If the element's `style` attribute is `style=\"padding-inline-start:16px\"`, the element is part of the \"general\" section of magazine articles, but if its `style` attribute is `style=\"padding-inline-start:32px\"` (i.e., with `32px` of indentation or more), the element's section is the same as the nearest section header above it (a `<span class=\"sectionTitle-_Dn99 sectionLabel-HdHsm\">` element that is a parent element to the article).\n",
    "> Key takeaway: the level of indentation is the surest way to identify the section that an article belongs to; it's difficult to figure out from the URL alone or from the combination of the URL and the series.\n",
    "\n",
    "Aside from re-scraping the data on each article to determine the section based on its level of indentation, there might not be a sure way to accurately determine the correct section (that is, assign the section \"general\" rather than a more specific category when the article belongs to the general magazine articles and not to a specific category).\n",
    "\n",
    "The code below is my best attempt to clean up many of the incorrect sections labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logic:\n",
    "- If `section` is not 'general'\n",
    "- AND `series` is `NaN`\n",
    "- AND the `url` has no forward slashes (`/`) after the issue (which is formatted as: '2021/01/article_title', '1987/12/article_title', or '2017/11-se/article_title'). There are 7 forward slashes in the URL for an article in the \"general\" section (and for many articles that have a specific section as well -- 7 is the minimum number of forward slashes).\n",
    "- THEN reclassify the article as the 'general' section and keep track of what its section was to check if the following sections should be updated.\n",
    "- If the article follows a section that was changed to 'general' and the article's section is the same as the one that was changed to 'general', then change the current article's section to 'general' as well -- since articles don't switch back and forth between the 'general' category and a more specific category (I think -- from the samples I've looked at so far, that seems to hold true).\n",
    "\n",
    "Then reclassify the section as \"general\".\n",
    "\n",
    "This should correctly update the section classification for most articles that were misclassified before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "df_data = pd.read_csv('data/article_data.csv', low_memory=False)\n",
    "\n",
    "# Ensure entire URLs can be viewed\n",
    "pd.options.display.max_colwidth = None\n",
    "# Reset to default\n",
    "# pd.reset_option('display.max_colwidth')\n",
    "\n",
    "# Count the number of forward slashes in each URL\n",
    "df_data['slash_count'] = df_data['url'].str.count(r'/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_rows = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of articles in the 'general' category: 22,399/67,143\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>section</th>\n",
       "      <th>series</th>\n",
       "      <th>slash_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/pointing-us-all-to-jesus-christ</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/hear-him</td>\n",
       "      <td>general</td>\n",
       "      <td>Hear Him</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/a-new-publication-for-a-worldwide-church</td>\n",
       "      <td>general</td>\n",
       "      <td>Welcome to This Issue</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/grow-into-the-principle-of-revelation</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/god-speaks-to-us-today</td>\n",
       "      <td>general</td>\n",
       "      <td>Gospel Basics</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/the-power-of-personal-revelation</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/a-better-choice</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/deneto-forde-saint-catherine-jamaica</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/the-anchor-of-my-life-and-faith</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/home-centered-church-away-from-home</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/a-bit-of-missionary-heaven</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/women-and-covenant-power</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-used-her-spiritual-gifts</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Early Women of the Restoration</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/how-does-the-restored-gospel-bless-us</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/how-can-i-know-for-myself</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/what-did-moroni-teach-joseph-smith</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/what-greater-witness-can-you-have-than-from-god</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/how-studying-church-history-strengthens-my-faith</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/including-everyone</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Inclusion</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/be-with-and-strengthen-one-another</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Ministering Principles</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/enjoying-the-later-years</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Aging Faithfully</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/what-sweeping-taught-me-about-parenthood</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Parenting Help</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/prophets-and-revelation</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>For Parents</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/unlocking-the-door-to-personal-revelation</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/making-the-doctrine-and-covenants-personal-to-you</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/a-note-and-a-spark-in-my-soul</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-mack-smith-a-faithful-witness</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only: Early Women of the Restoration</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/grappling-with-unanswered-gospel-questions</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/recovering-from-spiritual-numbness</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/waiting-for-answers-without-doubting</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/more-for-you</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/helping-loved-ones-face-questions-and-doubts-about-faith</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/my-questions-and-christs-love</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/i-had-left-the-church-so-why-did-my-husband-want-to-join-it</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/faith-the-antidote-to-uncertainty</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/unclouding-my-vision</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/finding-answers-to-gospel-questions</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/how-the-miracle-my-ancestor-experienced-during-the-spanish-flu-pandemic-is-inspiring-me</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/the-doctrine-and-covenants-through-the-years</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-001</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Area Presidency Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-002</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Priesthood Leader Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-003</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Church History</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-004</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Member Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-005</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Teachings of the Presidents of the Church</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-001</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Area Presidency Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-002</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Local Leader Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-003</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Church History</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-004</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Member Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-005</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Teachings of the Presidents of the Church</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-001</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Area Leadership Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-002</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-003</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-004</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-005</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-001</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Area Leadership Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-002</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-003</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-004</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-001</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>Area Leadership Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-002</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-003</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-004</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-005</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-006</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-007</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-008</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-009</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-010</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-011</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-012</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-013</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-014</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-015</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/what-is-the-united-states-and-canada-section</td>\n",
       "      <td>United States and Canada Section</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-past-present-and-future</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-and-the-liahona</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/is-this-the-liahona</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/what-happens-when-you-read-church-magazines</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/could-i-honor-my-heritage-as-a-descendant-of-lehi</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/an-obscure-boy</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/compassion-for-those-affected-by-crime-and-incarceration</td>\n",
       "      <td>general</td>\n",
       "      <td>Welcome to This issue</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/vestland-norway</td>\n",
       "      <td>general</td>\n",
       "      <td>The Church Is Here</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/hear-him</td>\n",
       "      <td>general</td>\n",
       "      <td>Hear Him</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/god-told-us-to-be-baptized</td>\n",
       "      <td>general</td>\n",
       "      <td>Gospel Basics</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/the-gift-of-forgiveness</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/a-fulness-of-joy</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/the-strength-to-move-forward</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/finding-the-strength-and-courage-to-fly</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/what-if-i-hadnt-gone</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/the-worth-of-each-soul</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants 18–19</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/your-worth-is-great</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>For Parents</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/helping-children-prepare-for-baptism</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/eliza-spoke-with-authority</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Early Women of the Restoration</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/how-does-the-lords-wisdom-bless-our-lives</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/what-can-we-do-to-follow-the-prophet</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/who-was-david-whitmer</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/what-is-the-gospel-of-jesus-christ-worth-to-you</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-with-financial-challenges</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Ministering Principles</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-who-are-incarcerated</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                  url  \\\n",
       "0                                                                           https://www.churchofjesuschrist.org/study/liahona/2021/01/pointing-us-all-to-jesus-christ   \n",
       "1                                                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/01/hear-him   \n",
       "2                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/01/a-new-publication-for-a-worldwide-church   \n",
       "3                                                                     https://www.churchofjesuschrist.org/study/liahona/2021/01/grow-into-the-principle-of-revelation   \n",
       "4                                                                                    https://www.churchofjesuschrist.org/study/liahona/2021/01/god-speaks-to-us-today   \n",
       "5                                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/the-power-of-personal-revelation   \n",
       "6                                                                   https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/a-better-choice   \n",
       "7                                              https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/deneto-forde-saint-catherine-jamaica   \n",
       "8                                                   https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/the-anchor-of-my-life-and-faith   \n",
       "9                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/home-centered-church-away-from-home   \n",
       "10                                                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/a-bit-of-missionary-heaven   \n",
       "11                                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/women-and-covenant-power   \n",
       "12                                                                            https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-used-her-spiritual-gifts   \n",
       "13                                                                    https://www.churchofjesuschrist.org/study/liahona/2021/01/how-does-the-restored-gospel-bless-us   \n",
       "14                                                                                https://www.churchofjesuschrist.org/study/liahona/2021/01/how-can-i-know-for-myself   \n",
       "15                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/what-did-moroni-teach-joseph-smith   \n",
       "16                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/what-greater-witness-can-you-have-than-from-god   \n",
       "17                                                         https://www.churchofjesuschrist.org/study/liahona/2021/01/how-studying-church-history-strengthens-my-faith   \n",
       "18                                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/including-everyone   \n",
       "19                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/be-with-and-strengthen-one-another   \n",
       "20                                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/enjoying-the-later-years   \n",
       "21                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/what-sweeping-taught-me-about-parenthood   \n",
       "22                                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/01/prophets-and-revelation   \n",
       "23                                                                https://www.churchofjesuschrist.org/study/liahona/2021/01/unlocking-the-door-to-personal-revelation   \n",
       "24                                                        https://www.churchofjesuschrist.org/study/liahona/2021/01/making-the-doctrine-and-covenants-personal-to-you   \n",
       "25                                                                            https://www.churchofjesuschrist.org/study/liahona/2021/01/a-note-and-a-spark-in-my-soul   \n",
       "26                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-mack-smith-a-faithful-witness   \n",
       "27                                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/grappling-with-unanswered-gospel-questions   \n",
       "28                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/recovering-from-spiritual-numbness   \n",
       "29                                                        https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/waiting-for-answers-without-doubting   \n",
       "30                                                                                https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/more-for-you   \n",
       "31                                    https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/helping-loved-ones-face-questions-and-doubts-about-faith   \n",
       "32                                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/my-questions-and-christs-love   \n",
       "33                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/i-had-left-the-church-so-why-did-my-husband-want-to-join-it   \n",
       "34                                                           https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/faith-the-antidote-to-uncertainty   \n",
       "35                                                                        https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/unclouding-my-vision   \n",
       "36                                                         https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/finding-answers-to-gospel-questions   \n",
       "37     https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/how-the-miracle-my-ancestor-experienced-during-the-spanish-flu-pandemic-is-inspiring-me   \n",
       "38                                                             https://www.churchofjesuschrist.org/study/liahona/2021/01/the-doctrine-and-covenants-through-the-years   \n",
       "39                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-001   \n",
       "40                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-002   \n",
       "41                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-003   \n",
       "42                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-004   \n",
       "43                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-005   \n",
       "44                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-001   \n",
       "45                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-002   \n",
       "46                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-003   \n",
       "47                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-004   \n",
       "48                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-005   \n",
       "49                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-001   \n",
       "50                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-002   \n",
       "51                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-003   \n",
       "52                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-004   \n",
       "53                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-005   \n",
       "54                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-001   \n",
       "55                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-002   \n",
       "56                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-003   \n",
       "57                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-004   \n",
       "58                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-001   \n",
       "59                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-002   \n",
       "60                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-003   \n",
       "61                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-004   \n",
       "62                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-005   \n",
       "63                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-006   \n",
       "64                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-007   \n",
       "65                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-008   \n",
       "66                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-009   \n",
       "67                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-010   \n",
       "68                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-011   \n",
       "69                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-012   \n",
       "70                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-013   \n",
       "71                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-014   \n",
       "72                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-015   \n",
       "73                            https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/what-is-the-united-states-and-canada-section   \n",
       "74                                              https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona   \n",
       "75          https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-past-present-and-future   \n",
       "76                  https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-and-the-liahona   \n",
       "77                          https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/is-this-the-liahona   \n",
       "78  https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/what-happens-when-you-read-church-magazines   \n",
       "79                       https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/could-i-honor-my-heritage-as-a-descendant-of-lehi   \n",
       "80                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/an-obscure-boy   \n",
       "81                                                 https://www.churchofjesuschrist.org/study/liahona/2021/02/compassion-for-those-affected-by-crime-and-incarceration   \n",
       "82                                                                                          https://www.churchofjesuschrist.org/study/liahona/2021/02/vestland-norway   \n",
       "83                                                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/02/hear-him   \n",
       "84                                                                               https://www.churchofjesuschrist.org/study/liahona/2021/02/god-told-us-to-be-baptized   \n",
       "85                                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/02/the-gift-of-forgiveness   \n",
       "86                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/a-fulness-of-joy   \n",
       "87                                                     https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/the-strength-to-move-forward   \n",
       "88                                          https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/finding-the-strength-and-courage-to-fly   \n",
       "89                                                             https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/what-if-i-hadnt-gone   \n",
       "90                                                                                   https://www.churchofjesuschrist.org/study/liahona/2021/02/the-worth-of-each-soul   \n",
       "91                                                                                      https://www.churchofjesuschrist.org/study/liahona/2021/02/your-worth-is-great   \n",
       "92                                                                     https://www.churchofjesuschrist.org/study/liahona/2021/02/helping-children-prepare-for-baptism   \n",
       "93                                                                               https://www.churchofjesuschrist.org/study/liahona/2021/02/eliza-spoke-with-authority   \n",
       "94                                                                https://www.churchofjesuschrist.org/study/liahona/2021/02/how-does-the-lords-wisdom-bless-our-lives   \n",
       "95                                                                     https://www.churchofjesuschrist.org/study/liahona/2021/02/what-can-we-do-to-follow-the-prophet   \n",
       "96                                                                                    https://www.churchofjesuschrist.org/study/liahona/2021/02/who-was-david-whitmer   \n",
       "97                                                          https://www.churchofjesuschrist.org/study/liahona/2021/02/what-is-the-gospel-of-jesus-christ-worth-to-you   \n",
       "98                                                           https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-with-financial-challenges   \n",
       "99                                                                https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-who-are-incarcerated   \n",
       "\n",
       "                               section  \\\n",
       "0                              general   \n",
       "1                              general   \n",
       "2                              general   \n",
       "3                              general   \n",
       "4                              general   \n",
       "5                              general   \n",
       "6              Latter-day Saint Voices   \n",
       "7              Latter-day Saint Voices   \n",
       "8              Latter-day Saint Voices   \n",
       "9              Latter-day Saint Voices   \n",
       "10             Latter-day Saint Voices   \n",
       "11             Latter-day Saint Voices   \n",
       "12             Latter-day Saint Voices   \n",
       "13                     Come, Follow Me   \n",
       "14                     Come, Follow Me   \n",
       "15                     Come, Follow Me   \n",
       "16                     Come, Follow Me   \n",
       "17                     Come, Follow Me   \n",
       "18                     Come, Follow Me   \n",
       "19                     Come, Follow Me   \n",
       "20                     Come, Follow Me   \n",
       "21                     Come, Follow Me   \n",
       "22                     Come, Follow Me   \n",
       "23                        Digital Only   \n",
       "24                        Digital Only   \n",
       "25                        Digital Only   \n",
       "26                        Digital Only   \n",
       "27                        Digital Only   \n",
       "28                        Young Adults   \n",
       "29                        Young Adults   \n",
       "30                        Young Adults   \n",
       "31          Digital Only: Young Adults   \n",
       "32          Digital Only: Young Adults   \n",
       "33          Digital Only: Young Adults   \n",
       "34          Digital Only: Young Adults   \n",
       "35          Digital Only: Young Adults   \n",
       "36          Digital Only: Young Adults   \n",
       "37          Digital Only: Young Adults   \n",
       "38          Digital Only: Young Adults   \n",
       "39          Africa Central Local Pages   \n",
       "40          Africa Central Local Pages   \n",
       "41          Africa Central Local Pages   \n",
       "42          Africa Central Local Pages   \n",
       "43          Africa Central Local Pages   \n",
       "44            Africa South Local Pages   \n",
       "45            Africa South Local Pages   \n",
       "46            Africa South Local Pages   \n",
       "47            Africa South Local Pages   \n",
       "48            Africa South Local Pages   \n",
       "49               Australia Local Pages   \n",
       "50               Australia Local Pages   \n",
       "51               Australia Local Pages   \n",
       "52               Australia Local Pages   \n",
       "53               Australia Local Pages   \n",
       "54             New Zealand Local Pages   \n",
       "55             New Zealand Local Pages   \n",
       "56             New Zealand Local Pages   \n",
       "57             New Zealand Local Pages   \n",
       "58  United Kingdom/Ireland Local Pages   \n",
       "59  United Kingdom/Ireland Local Pages   \n",
       "60  United Kingdom/Ireland Local Pages   \n",
       "61  United Kingdom/Ireland Local Pages   \n",
       "62  United Kingdom/Ireland Local Pages   \n",
       "63  United Kingdom/Ireland Local Pages   \n",
       "64  United Kingdom/Ireland Local Pages   \n",
       "65  United Kingdom/Ireland Local Pages   \n",
       "66  United Kingdom/Ireland Local Pages   \n",
       "67  United Kingdom/Ireland Local Pages   \n",
       "68  United Kingdom/Ireland Local Pages   \n",
       "69  United Kingdom/Ireland Local Pages   \n",
       "70  United Kingdom/Ireland Local Pages   \n",
       "71  United Kingdom/Ireland Local Pages   \n",
       "72  United Kingdom/Ireland Local Pages   \n",
       "73    United States and Canada Section   \n",
       "74          Welcome to the New Liahona   \n",
       "75          Welcome to the New Liahona   \n",
       "76          Welcome to the New Liahona   \n",
       "77          Welcome to the New Liahona   \n",
       "78          Welcome to the New Liahona   \n",
       "79          Welcome to the New Liahona   \n",
       "80          Welcome to the New Liahona   \n",
       "81                             general   \n",
       "82                             general   \n",
       "83                             general   \n",
       "84                             general   \n",
       "85                             general   \n",
       "86             Latter-day Saint Voices   \n",
       "87             Latter-day Saint Voices   \n",
       "88             Latter-day Saint Voices   \n",
       "89             Latter-day Saint Voices   \n",
       "90             Latter-day Saint Voices   \n",
       "91             Latter-day Saint Voices   \n",
       "92             Latter-day Saint Voices   \n",
       "93             Latter-day Saint Voices   \n",
       "94                     Come, Follow Me   \n",
       "95                     Come, Follow Me   \n",
       "96                     Come, Follow Me   \n",
       "97                     Come, Follow Me   \n",
       "98                     Come, Follow Me   \n",
       "99                     Come, Follow Me   \n",
       "\n",
       "                                           series  slash_count  \n",
       "0                                             NaN            7  \n",
       "1                                        Hear Him            7  \n",
       "2                           Welcome to This Issue            7  \n",
       "3                                             NaN            7  \n",
       "4                                   Gospel Basics            7  \n",
       "5                                             NaN            7  \n",
       "6                         Latter-day Saint Voices            8  \n",
       "7                         Latter-day Saint Voices            8  \n",
       "8                         Latter-day Saint Voices            8  \n",
       "9                         Latter-day Saint Voices            8  \n",
       "10                                            NaN            7  \n",
       "11                                            NaN            7  \n",
       "12                 Early Women of the Restoration            7  \n",
       "13        Come, Follow Me: Doctrine and Covenants            7  \n",
       "14        Come, Follow Me: Doctrine and Covenants            7  \n",
       "15        Come, Follow Me: Doctrine and Covenants            7  \n",
       "16        Come, Follow Me: Doctrine and Covenants            7  \n",
       "17                                            NaN            7  \n",
       "18                                      Inclusion            7  \n",
       "19                         Ministering Principles            7  \n",
       "20                               Aging Faithfully            7  \n",
       "21                                 Parenting Help            7  \n",
       "22                                    For Parents            7  \n",
       "23                                   Digital Only            7  \n",
       "24                                   Digital Only            7  \n",
       "25                                   Digital Only            7  \n",
       "26   Digital Only: Early Women of the Restoration            7  \n",
       "27                                   Digital Only            7  \n",
       "28                                   Young Adults            8  \n",
       "29                                   Young Adults            8  \n",
       "30                                   Young Adults            8  \n",
       "31                     Digital Only: Young Adults            8  \n",
       "32                     Digital Only: Young Adults            8  \n",
       "33                     Digital Only: Young Adults            8  \n",
       "34                     Digital Only: Young Adults            8  \n",
       "35                     Digital Only: Young Adults            8  \n",
       "36                     Digital Only: Young Adults            8  \n",
       "37                     Digital Only: Young Adults            8  \n",
       "38                                            NaN            7  \n",
       "39                        Area Presidency Message            8  \n",
       "40                      Priesthood Leader Message            8  \n",
       "41                                 Church History            8  \n",
       "42                                  Member Voices            8  \n",
       "43      Teachings of the Presidents of the Church            8  \n",
       "44                        Area Presidency Message            8  \n",
       "45                           Local Leader Message            8  \n",
       "46                                 Church History            8  \n",
       "47                                  Member Voices            8  \n",
       "48      Teachings of the Presidents of the Church            8  \n",
       "49                        Area Leadership Message            8  \n",
       "50                                    Local Pages            8  \n",
       "51                                    Local Pages            8  \n",
       "52                                    Local Pages            8  \n",
       "53                                    Local Pages            8  \n",
       "54                        Area Leadership Message            8  \n",
       "55                                    Local Pages            8  \n",
       "56                                    Local Pages            8  \n",
       "57                                    Local Pages            8  \n",
       "58                        Area Leadership Message            8  \n",
       "59                                    Local Pages            8  \n",
       "60                                            NaN            8  \n",
       "61                                            NaN            8  \n",
       "62                                            NaN            8  \n",
       "63                                            NaN            8  \n",
       "64                                            NaN            8  \n",
       "65                                            NaN            8  \n",
       "66                                            NaN            8  \n",
       "67                                            NaN            8  \n",
       "68                                            NaN            8  \n",
       "69                                            NaN            8  \n",
       "70                                            NaN            8  \n",
       "71                                            NaN            8  \n",
       "72                                            NaN            8  \n",
       "73                                            NaN            8  \n",
       "74                                            NaN            8  \n",
       "75                                            NaN            9  \n",
       "76                                            NaN            9  \n",
       "77                                            NaN            9  \n",
       "78                                            NaN            9  \n",
       "79                                            NaN            8  \n",
       "80                                            NaN            8  \n",
       "81                          Welcome to This issue            7  \n",
       "82                             The Church Is Here            7  \n",
       "83                                       Hear Him            7  \n",
       "84                                  Gospel Basics            7  \n",
       "85                                            NaN            7  \n",
       "86                        Latter-day Saint Voices            8  \n",
       "87                        Latter-day Saint Voices            8  \n",
       "88                        Latter-day Saint Voices            8  \n",
       "89                        Latter-day Saint Voices            8  \n",
       "90  Come, Follow Me: Doctrine and Covenants 18–19            7  \n",
       "91                                    For Parents            7  \n",
       "92                                            NaN            7  \n",
       "93                 Early Women of the Restoration            7  \n",
       "94                                Come, Follow Me            7  \n",
       "95                                Come, Follow Me            7  \n",
       "96                                Come, Follow Me            7  \n",
       "97                                Come, Follow Me            7  \n",
       "98                         Ministering Principles            7  \n",
       "99                                            NaN            7  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(f\"Number of articles in the 'general' category: {df_data[(df_data.section=='general')]['section'].count():,.0f}/{len(df_data.index):,.0f}\")\n",
    "df_data[['url', 'section', 'series', 'slash_count']].head(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of articles in the 'general' category: 41,495/67,143\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>section</th>\n",
       "      <th>series</th>\n",
       "      <th>slash_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/pointing-us-all-to-jesus-christ</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/hear-him</td>\n",
       "      <td>general</td>\n",
       "      <td>Hear Him</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/a-new-publication-for-a-worldwide-church</td>\n",
       "      <td>general</td>\n",
       "      <td>Welcome to This Issue</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/grow-into-the-principle-of-revelation</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/god-speaks-to-us-today</td>\n",
       "      <td>general</td>\n",
       "      <td>Gospel Basics</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/the-power-of-personal-revelation</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/a-better-choice</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/deneto-forde-saint-catherine-jamaica</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/the-anchor-of-my-life-and-faith</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/home-centered-church-away-from-home</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/a-bit-of-missionary-heaven</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/women-and-covenant-power</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-used-her-spiritual-gifts</td>\n",
       "      <td>general</td>\n",
       "      <td>Early Women of the Restoration</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/how-does-the-restored-gospel-bless-us</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/how-can-i-know-for-myself</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/what-did-moroni-teach-joseph-smith</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/what-greater-witness-can-you-have-than-from-god</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/how-studying-church-history-strengthens-my-faith</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/including-everyone</td>\n",
       "      <td>general</td>\n",
       "      <td>Inclusion</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/be-with-and-strengthen-one-another</td>\n",
       "      <td>general</td>\n",
       "      <td>Ministering Principles</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/enjoying-the-later-years</td>\n",
       "      <td>general</td>\n",
       "      <td>Aging Faithfully</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/what-sweeping-taught-me-about-parenthood</td>\n",
       "      <td>general</td>\n",
       "      <td>Parenting Help</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/prophets-and-revelation</td>\n",
       "      <td>general</td>\n",
       "      <td>For Parents</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/unlocking-the-door-to-personal-revelation</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/making-the-doctrine-and-covenants-personal-to-you</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/a-note-and-a-spark-in-my-soul</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-mack-smith-a-faithful-witness</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only: Early Women of the Restoration</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/grappling-with-unanswered-gospel-questions</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>Digital Only</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/recovering-from-spiritual-numbness</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/waiting-for-answers-without-doubting</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/more-for-you</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/helping-loved-ones-face-questions-and-doubts-about-faith</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/my-questions-and-christs-love</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/i-had-left-the-church-so-why-did-my-husband-want-to-join-it</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/faith-the-antidote-to-uncertainty</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/unclouding-my-vision</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/finding-answers-to-gospel-questions</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/how-the-miracle-my-ancestor-experienced-during-the-spanish-flu-pandemic-is-inspiring-me</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>Digital Only: Young Adults</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/the-doctrine-and-covenants-through-the-years</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-001</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Area Presidency Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-002</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Priesthood Leader Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-003</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Church History</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-004</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Member Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-005</td>\n",
       "      <td>Africa Central Local Pages</td>\n",
       "      <td>Teachings of the Presidents of the Church</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-001</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Area Presidency Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-002</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Local Leader Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-003</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Church History</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-004</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Member Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-005</td>\n",
       "      <td>Africa South Local Pages</td>\n",
       "      <td>Teachings of the Presidents of the Church</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-001</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Area Leadership Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-002</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-003</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-004</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-005</td>\n",
       "      <td>Australia Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-001</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Area Leadership Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-002</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-003</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-004</td>\n",
       "      <td>New Zealand Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-001</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>Area Leadership Message</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-002</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>Local Pages</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-003</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-004</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-005</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-006</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-007</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-008</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-009</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-010</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-011</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-012</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-013</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-014</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-015</td>\n",
       "      <td>United Kingdom/Ireland Local Pages</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/what-is-the-united-states-and-canada-section</td>\n",
       "      <td>United States and Canada Section</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-past-present-and-future</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-and-the-liahona</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/is-this-the-liahona</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/what-happens-when-you-read-church-magazines</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/could-i-honor-my-heritage-as-a-descendant-of-lehi</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/an-obscure-boy</td>\n",
       "      <td>Welcome to the New Liahona</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/compassion-for-those-affected-by-crime-and-incarceration</td>\n",
       "      <td>general</td>\n",
       "      <td>Welcome to This issue</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/vestland-norway</td>\n",
       "      <td>general</td>\n",
       "      <td>The Church Is Here</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/hear-him</td>\n",
       "      <td>general</td>\n",
       "      <td>Hear Him</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/god-told-us-to-be-baptized</td>\n",
       "      <td>general</td>\n",
       "      <td>Gospel Basics</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/the-gift-of-forgiveness</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/a-fulness-of-joy</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/the-strength-to-move-forward</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/finding-the-strength-and-courage-to-fly</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/what-if-i-hadnt-gone</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/the-worth-of-each-soul</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>Come, Follow Me: Doctrine and Covenants 18–19</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/your-worth-is-great</td>\n",
       "      <td>Latter-day Saint Voices</td>\n",
       "      <td>For Parents</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/helping-children-prepare-for-baptism</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/eliza-spoke-with-authority</td>\n",
       "      <td>general</td>\n",
       "      <td>Early Women of the Restoration</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/how-does-the-lords-wisdom-bless-our-lives</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/what-can-we-do-to-follow-the-prophet</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/who-was-david-whitmer</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/what-is-the-gospel-of-jesus-christ-worth-to-you</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-with-financial-challenges</td>\n",
       "      <td>Come, Follow Me</td>\n",
       "      <td>Ministering Principles</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-who-are-incarcerated</td>\n",
       "      <td>general</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                  url  \\\n",
       "0                                                                           https://www.churchofjesuschrist.org/study/liahona/2021/01/pointing-us-all-to-jesus-christ   \n",
       "1                                                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/01/hear-him   \n",
       "2                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/01/a-new-publication-for-a-worldwide-church   \n",
       "3                                                                     https://www.churchofjesuschrist.org/study/liahona/2021/01/grow-into-the-principle-of-revelation   \n",
       "4                                                                                    https://www.churchofjesuschrist.org/study/liahona/2021/01/god-speaks-to-us-today   \n",
       "5                                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/the-power-of-personal-revelation   \n",
       "6                                                                   https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/a-better-choice   \n",
       "7                                              https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/deneto-forde-saint-catherine-jamaica   \n",
       "8                                                   https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/the-anchor-of-my-life-and-faith   \n",
       "9                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/latter-day-saint-voices/home-centered-church-away-from-home   \n",
       "10                                                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/a-bit-of-missionary-heaven   \n",
       "11                                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/women-and-covenant-power   \n",
       "12                                                                            https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-used-her-spiritual-gifts   \n",
       "13                                                                    https://www.churchofjesuschrist.org/study/liahona/2021/01/how-does-the-restored-gospel-bless-us   \n",
       "14                                                                                https://www.churchofjesuschrist.org/study/liahona/2021/01/how-can-i-know-for-myself   \n",
       "15                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/what-did-moroni-teach-joseph-smith   \n",
       "16                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/what-greater-witness-can-you-have-than-from-god   \n",
       "17                                                         https://www.churchofjesuschrist.org/study/liahona/2021/01/how-studying-church-history-strengthens-my-faith   \n",
       "18                                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/including-everyone   \n",
       "19                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/be-with-and-strengthen-one-another   \n",
       "20                                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/enjoying-the-later-years   \n",
       "21                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/what-sweeping-taught-me-about-parenthood   \n",
       "22                                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/01/prophets-and-revelation   \n",
       "23                                                                https://www.churchofjesuschrist.org/study/liahona/2021/01/unlocking-the-door-to-personal-revelation   \n",
       "24                                                        https://www.churchofjesuschrist.org/study/liahona/2021/01/making-the-doctrine-and-covenants-personal-to-you   \n",
       "25                                                                            https://www.churchofjesuschrist.org/study/liahona/2021/01/a-note-and-a-spark-in-my-soul   \n",
       "26                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-mack-smith-a-faithful-witness   \n",
       "27                                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/grappling-with-unanswered-gospel-questions   \n",
       "28                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/recovering-from-spiritual-numbness   \n",
       "29                                                        https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/waiting-for-answers-without-doubting   \n",
       "30                                                                                https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/more-for-you   \n",
       "31                                    https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/helping-loved-ones-face-questions-and-doubts-about-faith   \n",
       "32                                                               https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/my-questions-and-christs-love   \n",
       "33                                 https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/i-had-left-the-church-so-why-did-my-husband-want-to-join-it   \n",
       "34                                                           https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/faith-the-antidote-to-uncertainty   \n",
       "35                                                                        https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/unclouding-my-vision   \n",
       "36                                                         https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/finding-answers-to-gospel-questions   \n",
       "37     https://www.churchofjesuschrist.org/study/liahona/2021/01/young-adults/how-the-miracle-my-ancestor-experienced-during-the-spanish-flu-pandemic-is-inspiring-me   \n",
       "38                                                             https://www.churchofjesuschrist.org/study/liahona/2021/01/the-doctrine-and-covenants-through-the-years   \n",
       "39                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-001   \n",
       "40                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-002   \n",
       "41                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-003   \n",
       "42                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-004   \n",
       "43                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afc-eng-local-pages/local-news-005   \n",
       "44                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-001   \n",
       "45                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-002   \n",
       "46                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-003   \n",
       "47                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-004   \n",
       "48                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/afs-eng-local-pages/local-news-005   \n",
       "49                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-001   \n",
       "50                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-002   \n",
       "51                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-003   \n",
       "52                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-004   \n",
       "53                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/aus-eng-local-pages/local-news-005   \n",
       "54                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-001   \n",
       "55                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-002   \n",
       "56                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-003   \n",
       "57                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/nzl-eng-local-pages/local-news-004   \n",
       "58                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-001   \n",
       "59                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-002   \n",
       "60                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-003   \n",
       "61                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-004   \n",
       "62                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-005   \n",
       "63                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-006   \n",
       "64                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-007   \n",
       "65                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-008   \n",
       "66                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-009   \n",
       "67                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-010   \n",
       "68                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-011   \n",
       "69                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-012   \n",
       "70                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-013   \n",
       "71                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-014   \n",
       "72                                                                       https://www.churchofjesuschrist.org/study/liahona/2021/01/eur-eng-local-pages/local-news-015   \n",
       "73                            https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/what-is-the-united-states-and-canada-section   \n",
       "74                                              https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona   \n",
       "75          https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-past-present-and-future   \n",
       "76                  https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/the-liahona-and-the-liahona   \n",
       "77                          https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/is-this-the-liahona   \n",
       "78  https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/welcome-to-the-new-liahona/what-happens-when-you-read-church-magazines   \n",
       "79                       https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/could-i-honor-my-heritage-as-a-descendant-of-lehi   \n",
       "80                                                          https://www.churchofjesuschrist.org/study/liahona/2021/01/united-states-and-canada-section/an-obscure-boy   \n",
       "81                                                 https://www.churchofjesuschrist.org/study/liahona/2021/02/compassion-for-those-affected-by-crime-and-incarceration   \n",
       "82                                                                                          https://www.churchofjesuschrist.org/study/liahona/2021/02/vestland-norway   \n",
       "83                                                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/02/hear-him   \n",
       "84                                                                               https://www.churchofjesuschrist.org/study/liahona/2021/02/god-told-us-to-be-baptized   \n",
       "85                                                                                  https://www.churchofjesuschrist.org/study/liahona/2021/02/the-gift-of-forgiveness   \n",
       "86                                                                 https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/a-fulness-of-joy   \n",
       "87                                                     https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/the-strength-to-move-forward   \n",
       "88                                          https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/finding-the-strength-and-courage-to-fly   \n",
       "89                                                             https://www.churchofjesuschrist.org/study/liahona/2021/02/latter-day-saint-voices/what-if-i-hadnt-gone   \n",
       "90                                                                                   https://www.churchofjesuschrist.org/study/liahona/2021/02/the-worth-of-each-soul   \n",
       "91                                                                                      https://www.churchofjesuschrist.org/study/liahona/2021/02/your-worth-is-great   \n",
       "92                                                                     https://www.churchofjesuschrist.org/study/liahona/2021/02/helping-children-prepare-for-baptism   \n",
       "93                                                                               https://www.churchofjesuschrist.org/study/liahona/2021/02/eliza-spoke-with-authority   \n",
       "94                                                                https://www.churchofjesuschrist.org/study/liahona/2021/02/how-does-the-lords-wisdom-bless-our-lives   \n",
       "95                                                                     https://www.churchofjesuschrist.org/study/liahona/2021/02/what-can-we-do-to-follow-the-prophet   \n",
       "96                                                                                    https://www.churchofjesuschrist.org/study/liahona/2021/02/who-was-david-whitmer   \n",
       "97                                                          https://www.churchofjesuschrist.org/study/liahona/2021/02/what-is-the-gospel-of-jesus-christ-worth-to-you   \n",
       "98                                                           https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-with-financial-challenges   \n",
       "99                                                                https://www.churchofjesuschrist.org/study/liahona/2021/02/ministering-to-those-who-are-incarcerated   \n",
       "\n",
       "                               section  \\\n",
       "0                              general   \n",
       "1                              general   \n",
       "2                              general   \n",
       "3                              general   \n",
       "4                              general   \n",
       "5                              general   \n",
       "6              Latter-day Saint Voices   \n",
       "7              Latter-day Saint Voices   \n",
       "8              Latter-day Saint Voices   \n",
       "9              Latter-day Saint Voices   \n",
       "10                             general   \n",
       "11                             general   \n",
       "12                             general   \n",
       "13                     Come, Follow Me   \n",
       "14                     Come, Follow Me   \n",
       "15                     Come, Follow Me   \n",
       "16                     Come, Follow Me   \n",
       "17                             general   \n",
       "18                             general   \n",
       "19                             general   \n",
       "20                             general   \n",
       "21                             general   \n",
       "22                             general   \n",
       "23                        Digital Only   \n",
       "24                        Digital Only   \n",
       "25                        Digital Only   \n",
       "26                        Digital Only   \n",
       "27                        Digital Only   \n",
       "28                        Young Adults   \n",
       "29                        Young Adults   \n",
       "30                        Young Adults   \n",
       "31          Digital Only: Young Adults   \n",
       "32          Digital Only: Young Adults   \n",
       "33          Digital Only: Young Adults   \n",
       "34          Digital Only: Young Adults   \n",
       "35          Digital Only: Young Adults   \n",
       "36          Digital Only: Young Adults   \n",
       "37          Digital Only: Young Adults   \n",
       "38                             general   \n",
       "39          Africa Central Local Pages   \n",
       "40          Africa Central Local Pages   \n",
       "41          Africa Central Local Pages   \n",
       "42          Africa Central Local Pages   \n",
       "43          Africa Central Local Pages   \n",
       "44            Africa South Local Pages   \n",
       "45            Africa South Local Pages   \n",
       "46            Africa South Local Pages   \n",
       "47            Africa South Local Pages   \n",
       "48            Africa South Local Pages   \n",
       "49               Australia Local Pages   \n",
       "50               Australia Local Pages   \n",
       "51               Australia Local Pages   \n",
       "52               Australia Local Pages   \n",
       "53               Australia Local Pages   \n",
       "54             New Zealand Local Pages   \n",
       "55             New Zealand Local Pages   \n",
       "56             New Zealand Local Pages   \n",
       "57             New Zealand Local Pages   \n",
       "58  United Kingdom/Ireland Local Pages   \n",
       "59  United Kingdom/Ireland Local Pages   \n",
       "60  United Kingdom/Ireland Local Pages   \n",
       "61  United Kingdom/Ireland Local Pages   \n",
       "62  United Kingdom/Ireland Local Pages   \n",
       "63  United Kingdom/Ireland Local Pages   \n",
       "64  United Kingdom/Ireland Local Pages   \n",
       "65  United Kingdom/Ireland Local Pages   \n",
       "66  United Kingdom/Ireland Local Pages   \n",
       "67  United Kingdom/Ireland Local Pages   \n",
       "68  United Kingdom/Ireland Local Pages   \n",
       "69  United Kingdom/Ireland Local Pages   \n",
       "70  United Kingdom/Ireland Local Pages   \n",
       "71  United Kingdom/Ireland Local Pages   \n",
       "72  United Kingdom/Ireland Local Pages   \n",
       "73    United States and Canada Section   \n",
       "74          Welcome to the New Liahona   \n",
       "75          Welcome to the New Liahona   \n",
       "76          Welcome to the New Liahona   \n",
       "77          Welcome to the New Liahona   \n",
       "78          Welcome to the New Liahona   \n",
       "79          Welcome to the New Liahona   \n",
       "80          Welcome to the New Liahona   \n",
       "81                             general   \n",
       "82                             general   \n",
       "83                             general   \n",
       "84                             general   \n",
       "85                             general   \n",
       "86             Latter-day Saint Voices   \n",
       "87             Latter-day Saint Voices   \n",
       "88             Latter-day Saint Voices   \n",
       "89             Latter-day Saint Voices   \n",
       "90             Latter-day Saint Voices   \n",
       "91             Latter-day Saint Voices   \n",
       "92                             general   \n",
       "93                             general   \n",
       "94                     Come, Follow Me   \n",
       "95                     Come, Follow Me   \n",
       "96                     Come, Follow Me   \n",
       "97                     Come, Follow Me   \n",
       "98                     Come, Follow Me   \n",
       "99                             general   \n",
       "\n",
       "                                           series  slash_count  \n",
       "0                                             NaN            7  \n",
       "1                                        Hear Him            7  \n",
       "2                           Welcome to This Issue            7  \n",
       "3                                             NaN            7  \n",
       "4                                   Gospel Basics            7  \n",
       "5                                             NaN            7  \n",
       "6                         Latter-day Saint Voices            8  \n",
       "7                         Latter-day Saint Voices            8  \n",
       "8                         Latter-day Saint Voices            8  \n",
       "9                         Latter-day Saint Voices            8  \n",
       "10                                            NaN            7  \n",
       "11                                            NaN            7  \n",
       "12                 Early Women of the Restoration            7  \n",
       "13        Come, Follow Me: Doctrine and Covenants            7  \n",
       "14        Come, Follow Me: Doctrine and Covenants            7  \n",
       "15        Come, Follow Me: Doctrine and Covenants            7  \n",
       "16        Come, Follow Me: Doctrine and Covenants            7  \n",
       "17                                            NaN            7  \n",
       "18                                      Inclusion            7  \n",
       "19                         Ministering Principles            7  \n",
       "20                               Aging Faithfully            7  \n",
       "21                                 Parenting Help            7  \n",
       "22                                    For Parents            7  \n",
       "23                                   Digital Only            7  \n",
       "24                                   Digital Only            7  \n",
       "25                                   Digital Only            7  \n",
       "26   Digital Only: Early Women of the Restoration            7  \n",
       "27                                   Digital Only            7  \n",
       "28                                   Young Adults            8  \n",
       "29                                   Young Adults            8  \n",
       "30                                   Young Adults            8  \n",
       "31                     Digital Only: Young Adults            8  \n",
       "32                     Digital Only: Young Adults            8  \n",
       "33                     Digital Only: Young Adults            8  \n",
       "34                     Digital Only: Young Adults            8  \n",
       "35                     Digital Only: Young Adults            8  \n",
       "36                     Digital Only: Young Adults            8  \n",
       "37                     Digital Only: Young Adults            8  \n",
       "38                                            NaN            7  \n",
       "39                        Area Presidency Message            8  \n",
       "40                      Priesthood Leader Message            8  \n",
       "41                                 Church History            8  \n",
       "42                                  Member Voices            8  \n",
       "43      Teachings of the Presidents of the Church            8  \n",
       "44                        Area Presidency Message            8  \n",
       "45                           Local Leader Message            8  \n",
       "46                                 Church History            8  \n",
       "47                                  Member Voices            8  \n",
       "48      Teachings of the Presidents of the Church            8  \n",
       "49                        Area Leadership Message            8  \n",
       "50                                    Local Pages            8  \n",
       "51                                    Local Pages            8  \n",
       "52                                    Local Pages            8  \n",
       "53                                    Local Pages            8  \n",
       "54                        Area Leadership Message            8  \n",
       "55                                    Local Pages            8  \n",
       "56                                    Local Pages            8  \n",
       "57                                    Local Pages            8  \n",
       "58                        Area Leadership Message            8  \n",
       "59                                    Local Pages            8  \n",
       "60                                            NaN            8  \n",
       "61                                            NaN            8  \n",
       "62                                            NaN            8  \n",
       "63                                            NaN            8  \n",
       "64                                            NaN            8  \n",
       "65                                            NaN            8  \n",
       "66                                            NaN            8  \n",
       "67                                            NaN            8  \n",
       "68                                            NaN            8  \n",
       "69                                            NaN            8  \n",
       "70                                            NaN            8  \n",
       "71                                            NaN            8  \n",
       "72                                            NaN            8  \n",
       "73                                            NaN            8  \n",
       "74                                            NaN            8  \n",
       "75                                            NaN            9  \n",
       "76                                            NaN            9  \n",
       "77                                            NaN            9  \n",
       "78                                            NaN            9  \n",
       "79                                            NaN            8  \n",
       "80                                            NaN            8  \n",
       "81                          Welcome to This issue            7  \n",
       "82                             The Church Is Here            7  \n",
       "83                                       Hear Him            7  \n",
       "84                                  Gospel Basics            7  \n",
       "85                                            NaN            7  \n",
       "86                        Latter-day Saint Voices            8  \n",
       "87                        Latter-day Saint Voices            8  \n",
       "88                        Latter-day Saint Voices            8  \n",
       "89                        Latter-day Saint Voices            8  \n",
       "90  Come, Follow Me: Doctrine and Covenants 18–19            7  \n",
       "91                                    For Parents            7  \n",
       "92                                            NaN            7  \n",
       "93                 Early Women of the Restoration            7  \n",
       "94                                Come, Follow Me            7  \n",
       "95                                Come, Follow Me            7  \n",
       "96                                Come, Follow Me            7  \n",
       "97                                Come, Follow Me            7  \n",
       "98                         Ministering Principles            7  \n",
       "99                                            NaN            7  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_section = 'general'\n",
    "update_following_labels = False\n",
    "\n",
    "for n, row in enumerate(df_data.itertuples()):\n",
    "    if row.section != 'general':\n",
    "        # Check the next rows that have the same section,\n",
    "        # since they will also need to be update to 'general' \n",
    "        # until the section changes.\n",
    "        if update_following_labels:\n",
    "            if row.section == current_section:\n",
    "                # Change this section to 'general' since\n",
    "                # it is in the same section as a previous\n",
    "                # article that was reclassified as 'general'\n",
    "                df_data.loc[n, 'section'] = 'general'\n",
    "                continue\n",
    "            else:\n",
    "                # Update the current section\n",
    "                current_section = row.section\n",
    "                # Don't change the next labels to 'general' since we're in a new section\n",
    "                update_following_labels = False\n",
    "        \n",
    "        if pd.isna(row.series):\n",
    "            if row.slash_count == 7:\n",
    "                # Update the section\n",
    "                df_data.loc[n, 'section'] = 'general'\n",
    "                current_section = row.section\n",
    "                update_following_labels = True\n",
    "\n",
    "print(f\"Number of articles in the 'general' category: {df_data[(df_data.section=='general')]['section'].count():,.0f}/{len(df_data.index):,.0f}\")\n",
    "df_data[['url', 'section', 'series', 'slash_count']].head(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How many topics should I use?\n",
    "> Answer: use the Gospel Topics section on https://ChurchOfJesusChrist.org/study to find out!\n",
    "\n",
    "In this section, I scrape article data for topics, to be used for automatically labeling topics during the topic modeling phase of this analysis.\n",
    "\n",
    "Much of the code below is messy and written in a one-off fashion -- I was exploring an idea more than developing a re-usable solution, so some of the code is redundant and certainly not optimized, but it worked for the task of creating a list of topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the number of topics listed on the General Conference Topics page\n",
    "\n",
    "# import requests\n",
    "# from bs4 import BeautifulSoup\n",
    "# URL = \"https://www.churchofjesuschrist.org/study/general-conference/topics?lang=eng\"\n",
    "\n",
    "# response = requests.get(URL)\n",
    "# response.encoding = 'utf-8'\n",
    "# page = BeautifulSoup(response.text)\n",
    "\n",
    "# topics = [a.text for a in page.find_all('a', class_='sc-omeqik-0 ksscEV tile-P903U listTile-WHLxI')]\n",
    "# print(f\"Number of topics: {len(topics)}\")\n",
    "# print(\"\\nTopics:\")\n",
    "# print(topics)\n",
    "\n",
    "# Number of topics: 317\n",
    "\n",
    "# Topics:\n",
    "# ['Aaronic Priesthood', 'Adam and Eve', 'Articles of Faith', 'Atonement', ...]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 224 topics in Gospel Topics.\n",
      "There are 173 topics in True to the Faith.\n"
     ]
    }
   ],
   "source": [
    "# First, using the Gospel Topics articles\n",
    "URL = \"https://www.churchofjesuschrist.org/study/manual/gospel-topics\"\n",
    "\n",
    "response = requests.get(URL)\n",
    "response.encoding = 'utf-8'\n",
    "page = BeautifulSoup(response.text)\n",
    "\n",
    "gt_topic_links = [a.get('href') for a in page.find('div', class_='body').find('ul', class_='doc-map').find_all('a')]\n",
    "print(f\"There are {len(gt_topic_links)} topics in Gospel Topics.\")\n",
    "\n",
    "# Next, using True to the Faith\n",
    "URL = \"https://www.churchofjesuschrist.org/study/manual/true-to-the-faith\"\n",
    "response = requests.get(URL)\n",
    "response.encoding = 'utf-8'\n",
    "page = BeautifulSoup(response.text)\n",
    "\n",
    "tf_topic_links = [a.get('href') for a in page.find('div', class_='body').find('ul', class_='doc-map').find_all('a')]\n",
    "print(f\"There are {len(tf_topic_links)} topics in True to the Faith.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scrape data from topics in [_True to the Faith_](https://www.churchofjesuschrist.org/study/manual/true-to-the-faith) to use for automatic topic labeling after performing LDA or NMF.\n",
    "\n",
    "Most of this code is adapted from the \"Data Scraping > Data collection class\" section at the beginning of this notebook (see the `get_article_data` method of that class)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<>:105: DeprecationWarning: invalid escape sequence \\s\n",
      "<>:105: DeprecationWarning: invalid escape sequence \\s\n",
      "C:\\Users\\Ryan\\AppData\\Local\\Temp\\ipykernel_2096\\1536269445.py:105: DeprecationWarning: invalid escape sequence \\s\n",
      "  text = re.sub('[\\s]{2,}', ' ', text)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dccacadc921d4070a001f3384d753f69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/171 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a .csv file to store the data\n",
    "with open('data/topics_tf.csv', mode='w', encoding='utf-8', newline='') as csvfile:\n",
    "    csv_writer = csv.writer(csvfile)\n",
    "    csv_writer.writerow(['url', 'topic', 'text'])\n",
    "\n",
    "ROOT_URL = \"https://www.churchofjesuschrist.org\"\n",
    "response = requests.get(ROOT_URL + \"/study/manual/true-to-the-faith\")\n",
    "response.encoding = 'utf-8'\n",
    "page = BeautifulSoup(response.text)\n",
    "\n",
    "skip_topics = ['Title Page', 'Message from the First Presidency']\n",
    "# When scraping article text within the <div class-\"body-block\"> by\n",
    "# looping through each direct child element in body_block.findChildren(recursive=False),\n",
    "# if an element is an <aside>, <figure>, or <figcaption> tag,\n",
    "# or if an element is a <div> tag with class \"credit\" or \"imageWrapper...\",\n",
    "# ignore its text since it isn't part of the article's main body text.\n",
    "tags_to_omit = ['aside', 'figure', 'figcaption']\n",
    "classes_to_omit = ['credit', re.compile('imageWrapper.*')]\n",
    "\n",
    "# When scraping image alt text, the text is prefixed\n",
    "# with \"Image\", which I later strip out when saving the data.\n",
    "img_prefix_chars = len('Image')\n",
    "\n",
    "topic_links = []\n",
    "topic_titles = []\n",
    "for a in page.find('div', class_='body').find('ul', class_='doc-map').find_all('a'):\n",
    "    if a.text not in skip_topics:\n",
    "        topic_links.append(ROOT_URL + a.get('href'))\n",
    "        topic_titles.append(a.text.strip())\n",
    "\n",
    "total_topics = len(topic_links)\n",
    "pbar = tqdm(total=total_topics)\n",
    "\n",
    "for n, url in enumerate(topic_links):\n",
    "    response = requests.get(url)\n",
    "    response.encoding = 'utf-8'\n",
    "    page = BeautifulSoup(response.text)\n",
    "    body = page.find('div', class_='body')\n",
    "    body_block = body.find('div', class_='body-block')\n",
    "\n",
    "    pbar.set_description(f\"Working on topic {n + 1}/{total_topics}, {topic_titles[n]}\")\n",
    "\n",
    "    title = body.find(id='title1')\n",
    "    if title:\n",
    "        title = title.text.strip()\n",
    "\n",
    "    # Use the .text attribute of the container that holds\n",
    "    # the image, which is a <div> element with class that begins with \"imageWrapper\"\n",
    "    images = body.find_all(class_=re.compile('imageWrapper.*'))\n",
    "    if images:\n",
    "        image_alt_text = [img.text.strip() for img in images]\n",
    "    else:\n",
    "        image_alt_text = None\n",
    "        \n",
    "    image_captions = body.find_all('figcaption')\n",
    "    if image_captions:\n",
    "        image_captions = [fig.text.strip() for fig in image_captions]\n",
    "    else:\n",
    "        image_captions = None\n",
    "\n",
    "    # The code below applies filters to refrain from collecting\n",
    "    # text that is not part of the article (image captions, section headings, asides).\n",
    "    title = body.find(id='title1')\n",
    "    title = title.text.strip() if title else None\n",
    "    text = []\n",
    "    for ch in body_block.findChildren(recursive=False):\n",
    "        ch_class = ch.get('class')  # -> a list, or None\n",
    "        if ch_class:\n",
    "            skip_class = any(cc in classes_to_omit for cc in ch_class)\n",
    "        else:\n",
    "            skip_class = False\n",
    "        \n",
    "        if ch.name in tags_to_omit or skip_class:\n",
    "            continue\n",
    "        else:\n",
    "            text.append(ch.text.strip())\n",
    "\n",
    "    # Combine into a single string, with paragraphs delimitted by newlines.\n",
    "    text = '\\n'.join(text)\n",
    "    # Replace multiple newlines with a single newline.\n",
    "    text = re.sub(r'[\\n]{2,}', '\\n', text)\n",
    "\n",
    "    # Remove unwanted text that was included through child elements\n",
    "    # (e.g., a section that, along with article text, \n",
    "    # contained text from unwanted elements like images, credits, or asides)\n",
    "    credit_elems = body_block.find_all('div', class_='credit')\n",
    "    if credit_elems:\n",
    "        for cr in credit_elems:\n",
    "            text = text.replace(cr.text.strip(), '', 1)\n",
    "    \n",
    "    aside_elems = body_block.find_all('aside')\n",
    "    if aside_elems:\n",
    "        for ae in aside_elems:\n",
    "            text = text.replace(ae.text.strip(), '', 1)\n",
    "\n",
    "    if image_alt_text:\n",
    "        for iat in image_alt_text:\n",
    "            text = text.replace(iat, '', 1)\n",
    "\n",
    "    if image_captions:\n",
    "        for ic in image_captions:\n",
    "            text = text.replace(ic, '', 1)\n",
    "    \n",
    "    # Replace multiple whitespace characters with a single space\n",
    "    text = re.sub(r'[\\s]{2,}', ' ', text)\n",
    "    \n",
    "    if text:\n",
    "        # Trim leading newline character, if present\n",
    "        if text[0] == '\\n':\n",
    "            text = text[1:]\n",
    "    # else:\n",
    "    #     # Skip to next article, since there is no text in this article\n",
    "    #     continue\n",
    "    \n",
    "    word_count = len(text.split()) if text else 0\n",
    "\n",
    "    pbar.update(n=1)\n",
    "\n",
    "    if word_count > 0:\n",
    "        # Save to .csv file\n",
    "        # Create a .csv file to store the data\n",
    "        with open('data/topics_tf.csv', mode='a', encoding='utf-8', newline='') as csvfile:\n",
    "            csv_writer = csv.writer(csvfile)\n",
    "            csv_writer.writerow([url, title, text])\n",
    "    # else:\n",
    "        # Don't save data; skip this article to move to the next one\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     108.000000\n",
       "mean      497.500000\n",
       "std       388.832581\n",
       "min        44.000000\n",
       "25%       224.750000\n",
       "50%       356.500000\n",
       "75%       629.250000\n",
       "max      1966.000000\n",
       "Name: word_count, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_df = pd.read_csv('data/topics_tf.csv')\n",
    "\n",
    "topics_df['word_count'] = topics_df['text'].apply(lambda x: len(x.split()))\n",
    "topics_df['word_count'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scrape data from topics in [_Gospel Topics_](https://www.churchofjesuschrist.org/study/manual/gospel-topics) to use for automatic topic labeling after performing LDA or NMF.\n",
    "\n",
    "Most of this code is adapted from the \"Data Scraping > Data collection class\" section at the beginning of this notebook (see the `get_article_data` method of that class)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "046a6c3c6815436c8da37fb497c191aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/223 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a .csv file to store the data\n",
    "with open('data/topics_gt.csv', mode='w', encoding='utf-8', newline='') as csvfile:\n",
    "    csv_writer = csv.writer(csvfile)\n",
    "    csv_writer.writerow(['url', 'topic', 'text'])\n",
    "\n",
    "ROOT_URL = \"https://www.churchofjesuschrist.org\"\n",
    "response = requests.get(ROOT_URL + \"/study/manual/gospel-topics\")\n",
    "response.encoding = 'utf-8'\n",
    "page = BeautifulSoup(response.text)\n",
    "\n",
    "skip_topics = ['Contents', 'Introduction to Gospel Topics']\n",
    "# When scraping article text within the <div class-\"body-block\"> by\n",
    "# looping through each direct child element in body_block.findChildren(recursive=False),\n",
    "# if an element is an <aside>, <figure>, or <figcaption> tag,\n",
    "# or if an element is a <div> tag with class \"credit\" or \"imageWrapper...\",\n",
    "# ignore its text since it isn't part of the article's main body text.\n",
    "tags_to_omit = ['button', 'aside', 'figure', 'figcaption']\n",
    "classes_to_omit = ['credit', re.compile('imageWrapper.*')]\n",
    "\n",
    "# When scraping image alt text, the text is prefixed\n",
    "# with \"Image\", which I later strip out when saving the data.\n",
    "img_prefix_chars = len('Image')\n",
    "\n",
    "topic_links = []\n",
    "topic_titles = []\n",
    "for a in page.find('div', class_='body').find('ul', class_='doc-map').find_all('a'):\n",
    "    ttl = a.find('p', class_='title').text.strip()\n",
    "    if ttl not in skip_topics:\n",
    "        topic_links.append(ROOT_URL + a.get('href'))\n",
    "        topic_titles.append(ttl)\n",
    "\n",
    "# Determine the length of the longest title \n",
    "# to use for left-justifying the names of \n",
    "# other titles when displaying on the progress bar\n",
    "max_title_chars = 0\n",
    "for item in topic_titles:\n",
    "    if len(item) > max_title_chars:\n",
    "        max_title_chars = len(item)\n",
    "\n",
    "total_topics = len(topic_links)\n",
    "pbar = tqdm(total=total_topics)\n",
    "\n",
    "for n, url in enumerate(topic_links):\n",
    "    response = requests.get(url)\n",
    "    response.encoding = 'utf-8'\n",
    "    page = BeautifulSoup(response.text)\n",
    "    body = page.find('div', class_='body')\n",
    "    body_block = body.find('div', class_='body-block')\n",
    "\n",
    "    justified_title = topic_titles[n].ljust(max_title_chars)\n",
    "    pbar.set_description(f\"Working on topic {n + 1}/{total_topics}, {justified_title}\")\n",
    "\n",
    "    title = body.find(id='title1')\n",
    "    if title:\n",
    "        title = title.text.strip()\n",
    "\n",
    "    # Use the .text attribute of the container that holds\n",
    "    # the image, which is a <div> element with class that begins with \"imageWrapper\"\n",
    "    images = body.find_all(class_=re.compile('imageWrapper.*'))\n",
    "    if images:\n",
    "        image_alt_text = [img.text.strip() for img in images]\n",
    "    else:\n",
    "        image_alt_text = None\n",
    "        \n",
    "    image_captions = body.find_all('figcaption')\n",
    "    if image_captions:\n",
    "        image_captions = [fig.text.strip() for fig in image_captions]\n",
    "    else:\n",
    "        image_captions = None\n",
    "\n",
    "    # The code below applies filters to refrain from collecting\n",
    "    # text that is not part of the article (image captions, section headings, asides).\n",
    "    title = body.find(id='title1')\n",
    "    title = title.text.strip() if title else None\n",
    "    text = []\n",
    "    for ch in body_block.findChildren(recursive=False):\n",
    "        ch_class = ch.get('class')  # -> a list, or None\n",
    "        if ch_class:\n",
    "            skip_class = any(cc in classes_to_omit for cc in ch_class)\n",
    "        else:\n",
    "            skip_class = False\n",
    "        \n",
    "        if ch.name in tags_to_omit or skip_class:\n",
    "            continue\n",
    "        else:\n",
    "            text.append(ch.text.strip())\n",
    "\n",
    "    # Combine into a single string, with paragraphs delimitted by newlines.\n",
    "    text = '\\n'.join(text)\n",
    "\n",
    "    # Remove unwanted text that was included through child elements\n",
    "    # (e.g., a section that, along with article text, \n",
    "    # contained text from unwanted elements like images, credits, or asides)\n",
    "    credit_elems = body_block.find_all('div', class_='credit')\n",
    "    if credit_elems:\n",
    "        for cr in credit_elems:\n",
    "            text = text.replace(cr.text.strip(), '', 1)\n",
    "    \n",
    "    aside_elems = body_block.find_all('aside')\n",
    "    if aside_elems:\n",
    "        for ae in aside_elems:\n",
    "            text = text.replace(ae.text.strip(), '', 1)\n",
    "\n",
    "    if image_alt_text:\n",
    "        for iat in image_alt_text:\n",
    "            text = text.replace(iat, '', 1)\n",
    "\n",
    "    if image_captions:\n",
    "        for ic in image_captions:\n",
    "            text = text.replace(ic, '', 1)\n",
    "    \n",
    "    # Replace multiple whitespace characters with a single space\n",
    "    text = re.sub(r'[\\s]{2,}', ' ', text)\n",
    "    \n",
    "    if text:\n",
    "        # Trim leading newline character, if present\n",
    "        if text[0] == '\\n':\n",
    "            text = text[1:]\n",
    "        if text[:len('Overview')] == 'Overview':\n",
    "            # Remove the \"Overview\\n\" at the beginning of the article\n",
    "            text = text[len('Overview'):]\n",
    "        \n",
    "        end_of_article = text.find('Related Topics')\n",
    "        text = text[:end_of_article]\n",
    "\n",
    "        end_of_article = text.find('Record Your Impressions')\n",
    "        # str.find() returns -1 if the text isn't found, so\n",
    "        # anything above 0 means a match.\n",
    "        if end_of_article > 0:\n",
    "            text = text[:end_of_article]\n",
    "\n",
    "        # Replace multiple newlines with a single newline.\n",
    "        text = re.sub(r'[\\n]{2,}', r'\\n', text)\n",
    "\n",
    "        end_of_article = text.find('Scriptures Scripture References')\n",
    "        if end_of_article > 0:\n",
    "            text = text[:end_of_article]\n",
    "\n",
    "        # Trim leading newline character, if present\n",
    "        if text[0] == '\\n':\n",
    "            text = text[1:]\n",
    "        \n",
    "        text = text.strip()\n",
    "\n",
    "        # Prefix topic text with topic name in case \n",
    "        # the topic is not mentioned within the text. \n",
    "        # Increases score for TFIDF normalization.\n",
    "        text = f\"{title}: {text}\"\n",
    "    \n",
    "    word_count = len(text.split()) if text else 0\n",
    "\n",
    "    pbar.update(n=1)\n",
    "\n",
    "    if word_count > 0:\n",
    "        # Save to .csv file\n",
    "        # Create a .csv file to store the data\n",
    "        with open('data/topics_gt.csv', mode='a', encoding='utf-8', newline='') as csvfile:\n",
    "            csv_writer = csv.writer(csvfile)\n",
    "            csv_writer.writerow([url, title, text])\n",
    "    # else:\n",
    "        # Don't save data; skip this article to move to the next one\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     223.000000\n",
       "mean      457.547085\n",
       "std       583.696188\n",
       "min         9.000000\n",
       "25%       172.000000\n",
       "50%       306.000000\n",
       "75%       472.500000\n",
       "max      4146.000000\n",
       "Name: word_count, dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_df = pd.read_csv('data/topics_gt.csv')\n",
    "\n",
    "topics_df['word_count'] = topics_df['text'].apply(lambda x: len(x.split()))\n",
    "topics_df['word_count'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>178.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>253.926966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>152.539669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>139.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>263.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>386.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>541.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       word_count\n",
       "count  178.000000\n",
       "mean   253.926966\n",
       "std    152.539669\n",
       "min      9.000000\n",
       "25%    139.500000\n",
       "50%    263.500000\n",
       "75%    386.500000\n",
       "max    541.000000"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Consider removing the top 20% longest articles,\n",
    "# since they tend to be overweighted when creating\n",
    "# the bag-of-words (vectorized) representation.\n",
    "\n",
    "# NEVERMIND: I corrected that problem by normalizing the topics' doc-terms matrix\n",
    "# by the word count of each topic, so the sum across the terms for each topic\n",
    "# equals 1.\n",
    "\n",
    "# cutoff_wrdcnt = topics_df['word_count'].quantile(0.80)\n",
    "# cutoff_wrdcnt\n",
    "\n",
    "# trimmed_df = topics_df[topics_df['word_count'] < cutoff_wrdcnt]\n",
    "# trimmed_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manually adjust the dataset\n",
    "For example, to replace article text \"See ___ in Gospel Topics Essays\" with the actual text from the Gospel Topics essay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>topic</th>\n",
       "      <th>text</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Transgression</td>\n",
       "      <td>Violation or breaking of a commandment or law.</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Becoming Like God</td>\n",
       "      <td>See Becoming Like God in the Gospel Topics Essays</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Mother in Heaven</td>\n",
       "      <td>See Heavenly Parents and Mother in Heaven (Gos...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Priesthood and Race</td>\n",
       "      <td>See Race and the Priesthood in the Gospel Topi...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Joseph Smith’s Teachings about Priesthood, Tem...</td>\n",
       "      <td>See Joseph Smith’s Teachings about Priesthood,...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Abraham, Book of</td>\n",
       "      <td>See Translation and Historicity of the Book of...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Mormons</td>\n",
       "      <td>This is a commonly used term for members of Th...</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Sealing</td>\n",
       "      <td>An ordinance performed in the temple eternally...</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Patriarch</td>\n",
       "      <td>A patriarch is a priesthood holder who is orda...</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Mormonism</td>\n",
       "      <td>A common term used to describe the teachings a...</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   url  \\\n",
       "209  https://www.churchofjesuschrist.org/study/manu...   \n",
       "17   https://www.churchofjesuschrist.org/study/manu...   \n",
       "122  https://www.churchofjesuschrist.org/study/manu...   \n",
       "148  https://www.churchofjesuschrist.org/study/manu...   \n",
       "100  https://www.churchofjesuschrist.org/study/manu...   \n",
       "2    https://www.churchofjesuschrist.org/study/manu...   \n",
       "120  https://www.churchofjesuschrist.org/study/manu...   \n",
       "177  https://www.churchofjesuschrist.org/study/manu...   \n",
       "134  https://www.churchofjesuschrist.org/study/manu...   \n",
       "119  https://www.churchofjesuschrist.org/study/manu...   \n",
       "\n",
       "                                                 topic  \\\n",
       "209                                      Transgression   \n",
       "17                                   Becoming Like God   \n",
       "122                                   Mother in Heaven   \n",
       "148                                Priesthood and Race   \n",
       "100  Joseph Smith’s Teachings about Priesthood, Tem...   \n",
       "2                                     Abraham, Book of   \n",
       "120                                            Mormons   \n",
       "177                                            Sealing   \n",
       "134                                          Patriarch   \n",
       "119                                          Mormonism   \n",
       "\n",
       "                                                  text  word_count  \n",
       "209     Violation or breaking of a commandment or law.           8  \n",
       "17   See Becoming Like God in the Gospel Topics Essays           9  \n",
       "122  See Heavenly Parents and Mother in Heaven (Gos...          10  \n",
       "148  See Race and the Priesthood in the Gospel Topi...          10  \n",
       "100  See Joseph Smith’s Teachings about Priesthood,...          14  \n",
       "2    See Translation and Historicity of the Book of...          14  \n",
       "120  This is a commonly used term for members of Th...          17  \n",
       "177  An ordinance performed in the temple eternally...          17  \n",
       "134  A patriarch is a priesthood holder who is orda...          19  \n",
       "119  A common term used to describe the teachings a...          19  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_df.sort_values(by='word_count', axis=0, ascending=True).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# topics_df = pd.read_csv('data/topics_gt_mod.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Family'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the index of a topic\n",
    "# topics_df['topic'].values.tolist().index('SEARCH TOPIC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text to add to the article (topic) text.\n",
    "manual_adjustment_text = \"\"\"\n",
    "THE FAMILY\n",
    "A PROCLAMATION TO THE WORLD\n",
    "\n",
    "The First Presidency and Council of the Twelve Apostles of The Church of Jesus Christ of Latter-day Saints\n",
    "\n",
    "We, the First Presidency and the Council of the Twelve Apostles of The Church of Jesus Christ of Latter-day Saints, solemnly proclaim that marriage between a man and a woman is ordained of God and that the family is central to the Creator’s plan for the eternal destiny of His children.\n",
    "\n",
    "All human beings—male and female—are created in the image of God. Each is a beloved spirit son or daughter of heavenly parents, and, as such, each has a divine nature and destiny. Gender is an essential characteristic of individual premortal, mortal, and eternal identity and purpose.\n",
    "\n",
    "In the premortal realm, spirit sons and daughters knew and worshipped God as their Eternal Father and accepted His plan by which His children could obtain a physical body and gain earthly experience to progress toward perfection and ultimately realize their divine destiny as heirs of eternal life. The divine plan of happiness enables family relationships to be perpetuated beyond the grave. Sacred ordinances and covenants available in holy temples make it possible for individuals to return to the presence of God and for families to be united eternally.\n",
    "\n",
    "The first commandment that God gave to Adam and Eve pertained to their potential for parenthood as husband and wife. We declare that God’s commandment for His children to multiply and replenish the earth remains in force. We further declare that God has commanded that the sacred powers of procreation are to be employed only between man and woman, lawfully wedded as husband and wife.\n",
    "\n",
    "We declare the means by which mortal life is created to be divinely appointed. We affirm the sanctity of life and of its importance in God’s eternal plan.\n",
    "\n",
    "Husband and wife have a solemn responsibility to love and care for each other and for their children. “Children are an heritage of the Lord” (Psalm 127:3). Parents have a sacred duty to rear their children in love and righteousness, to provide for their physical and spiritual needs, and to teach them to love and serve one another, observe the commandments of God, and be law-abiding citizens wherever they live. Husbands and wives—mothers and fathers—will be held accountable before God for the discharge of these obligations.\n",
    "\n",
    "The family is ordained of God. Marriage between man and woman is essential to His eternal plan. Children are entitled to birth within the bonds of matrimony, and to be reared by a father and a mother who honor marital vows with complete fidelity. Happiness in family life is most likely to be achieved when founded upon the teachings of the Lord Jesus Christ. Successful marriages and families are established and maintained on principles of faith, prayer, repentance, forgiveness, respect, love, compassion, work, and wholesome recreational activities. By divine design, fathers are to preside over their families in love and righteousness and are responsible to provide the necessities of life and protection for their families. Mothers are primarily responsible for the nurture of their children. In these sacred responsibilities, fathers and mothers are obligated to help one another as equal partners. Disability, death, or other circumstances may necessitate individual adaptation. Extended families should lend support when needed.\n",
    "\n",
    "We warn that individuals who violate covenants of chastity, who abuse spouse or offspring, or who fail to fulfill family responsibilities will one day stand accountable before God. Further, we warn that the disintegration of the family will bring upon individuals, communities, and nations the calamities foretold by ancient and modern prophets.\n",
    "\n",
    "We call upon responsible citizens and officers of government everywhere to promote those measures designed to maintain and strengthen the family as the fundamental unit of society.\n",
    "\"\"\"\n",
    "topics_df.at[47, \"text\"] = topics_df.at[47, \"text\"] + manual_adjustment_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>topic</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>example.org/1</td>\n",
       "      <td>one</td>\n",
       "      <td>one: This is topic one.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>example.org/2</td>\n",
       "      <td>two</td>\n",
       "      <td>two: This is topic two.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             url topic                     text\n",
       "0  example.org/1   one  one: This is topic one.\n",
       "1  example.org/2   two  two: This is topic two."
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prefix topic text with topic name in case \n",
    "# the topic is not mentioned within the text. \n",
    "# Increases score for TFIDF normalization.\n",
    "\n",
    "test_data = {\n",
    "    \"url\": [\"example.org/1\", \"example.org/2\"],\n",
    "    \"topic\": [\"one\", \"two\"],\n",
    "    \"text\": [\"This is topic one.\", \"This is topic two.\"]\n",
    "}\n",
    "\n",
    "test_df = pd.DataFrame(test_data)\n",
    "\n",
    "def prefix_topic(series):\n",
    "    x = series.copy()\n",
    "    x['text'] = f\"{x['topic']}: {x['text']}\"\n",
    "    return x\n",
    "\n",
    "test_df = test_df.apply(prefix_topic, axis=1)\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>topic</th>\n",
       "      <th>text</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Aaronic Priesthood</td>\n",
       "      <td>Aaronic Priesthood: The priesthood is the eter...</td>\n",
       "      <td>406</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Abortion</td>\n",
       "      <td>Abortion: See the Church’s official statement ...</td>\n",
       "      <td>276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Abraham, Book of</td>\n",
       "      <td>Abraham, Book of: The Church of Jesus Christ o...</td>\n",
       "      <td>2947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Abrahamic Covenant</td>\n",
       "      <td>Abrahamic Covenant: Abraham made covenants wit...</td>\n",
       "      <td>160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Abuse</td>\n",
       "      <td>Abuse: Abuse is the mistreatment or neglect of...</td>\n",
       "      <td>525</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 url               topic  \\\n",
       "0  https://www.churchofjesuschrist.org/study/manu...  Aaronic Priesthood   \n",
       "1  https://www.churchofjesuschrist.org/study/manu...            Abortion   \n",
       "2  https://www.churchofjesuschrist.org/study/manu...    Abraham, Book of   \n",
       "3  https://www.churchofjesuschrist.org/study/manu...  Abrahamic Covenant   \n",
       "4  https://www.churchofjesuschrist.org/study/manu...               Abuse   \n",
       "\n",
       "                                                text  word_count  \n",
       "0  Aaronic Priesthood: The priesthood is the eter...         406  \n",
       "1  Abortion: See the Church’s official statement ...         276  \n",
       "2  Abraham, Book of: The Church of Jesus Christ o...        2947  \n",
       "3  Abrahamic Covenant: Abraham made covenants wit...         160  \n",
       "4  Abuse: Abuse is the mistreatment or neglect of...         525  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def prefix_topic(series):\n",
    "    '''No longer needed; I added the topic prefix in the data collection function.'''\n",
    "    x = series.copy()\n",
    "    x['text'] = f\"{x['topic']}: {x['text']}\"\n",
    "    return x\n",
    "\n",
    "topics_df2 = topics_df.copy()\n",
    "# topics_df2 = topics_df2.apply(prefix_topic, axis=1)\n",
    "topics_df2['word_count'] = topics_df2['text'].apply(lambda x: len(x.split()))\n",
    "topics_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>topic</th>\n",
       "      <th>text</th>\n",
       "      <th>word_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Answering Gospel Questions</td>\n",
       "      <td>Answering Gospel Questions: The Lord encourage...</td>\n",
       "      <td>4146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>206</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>The Manifesto and the End of Plural Marriage</td>\n",
       "      <td>The Manifesto and the End of Plural Marriage: ...</td>\n",
       "      <td>3893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Peace and Violence among 19th-Century Latter-d...</td>\n",
       "      <td>Peace and Violence among 19th-Century Latter-d...</td>\n",
       "      <td>3632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Becoming Like God</td>\n",
       "      <td>Becoming Like God: One of the most common imag...</td>\n",
       "      <td>3488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>210</th>\n",
       "      <td>https://www.churchofjesuschrist.org/study/manu...</td>\n",
       "      <td>Translation and Historicity of the Book of Abr...</td>\n",
       "      <td>Translation and Historicity of the Book of Abr...</td>\n",
       "      <td>2952</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   url  \\\n",
       "9    https://www.churchofjesuschrist.org/study/manu...   \n",
       "206  https://www.churchofjesuschrist.org/study/manu...   \n",
       "137  https://www.churchofjesuschrist.org/study/manu...   \n",
       "17   https://www.churchofjesuschrist.org/study/manu...   \n",
       "210  https://www.churchofjesuschrist.org/study/manu...   \n",
       "\n",
       "                                                 topic  \\\n",
       "9                           Answering Gospel Questions   \n",
       "206       The Manifesto and the End of Plural Marriage   \n",
       "137  Peace and Violence among 19th-Century Latter-d...   \n",
       "17                                   Becoming Like God   \n",
       "210  Translation and Historicity of the Book of Abr...   \n",
       "\n",
       "                                                  text  word_count  \n",
       "9    Answering Gospel Questions: The Lord encourage...        4146  \n",
       "206  The Manifesto and the End of Plural Marriage: ...        3893  \n",
       "137  Peace and Violence among 19th-Century Latter-d...        3632  \n",
       "17   Becoming Like God: One of the most common imag...        3488  \n",
       "210  Translation and Historicity of the Book of Abr...        2952  "
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_df2.sort_values(by='word_count', ascending=False).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save modified version\n",
    "topics_df2 = topics_df2.drop(columns=[\"word_count\"])\n",
    "topics_df2.to_csv('data/topics_gt_mod.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove unncessary topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success! Filtered to 158 topics.\n"
     ]
    }
   ],
   "source": [
    "# Load the modified version to make additional changes (e.g., removing topics)\n",
    "topics_df = pd.read_csv('data/topics_gt_mod_old.csv')\n",
    "\n",
    "# The following topics don't represent true topics \n",
    "# for magazine articles, but rather are more like terms in a glossary\n",
    "topics_to_remove = [\n",
    "    'Aaronic Priesthood',\n",
    "    'Abraham, Book of',\n",
    "    'Apostle',\n",
    "    'Are “Mormons” Christian?',\n",
    "    'Articles of Faith',\n",
    "    'Bible, Inerrancy of',\n",
    "    'Birth Control',\n",
    "    'Bishop',\n",
    "    'Book of Mormon and DNA Studies',\n",
    "    'Book of Mormon Geography',\n",
    "    'Book of Mormon Translation',\n",
    "    'Church Finances—Commercial Businesses',\n",
    "    'Cross',\n",
    "    'Daughters in My Kingdom',\n",
    "    'Deacon',\n",
    "    'Elder',\n",
    "    'First Presidency',\n",
    "    'First Vision Accounts',\n",
    "    'Gambling',\n",
    "    'Gold Plates',\n",
    "    'High Council',\n",
    "    'High Priest',\n",
    "    'Jesus Christ Chosen as Savior',\n",
    "    \"Joseph Smith’s Teachings about Priesthood, Temple, and Women\",\n",
    "    'Journal of Discourses',\n",
    "    'Laying On of Hands',\n",
    "    'Melchizedek Priesthood',\n",
    "    'Membership Councils',\n",
    "    'Mormon Church',\n",
    "    'Mormonism',\n",
    "    'Mormons',\n",
    "    'Mother in Heaven',\n",
    "    'Mountain Meadows Massacre',\n",
    "    'Noah',\n",
    "    'Original Sin',\n",
    "    'Patriarch',\n",
    "    'Peace and Violence among 19th-Century Latter-day Saints',\n",
    "    'Plural Marriage in The Church of Jesus Christ of Latter-day Saints',\n",
    "    'Priest',\n",
    "    'Priesthood Blessing',\n",
    "    'Priesthood and Race',\n",
    "    'Primary',\n",
    "    'Prison Ministry',\n",
    "    'Prophecy',\n",
    "    'Quorum',\n",
    "    'Quorum of the Twelve Apostles',\n",
    "    'Race and the Priesthood',\n",
    "    'Relief Society',\n",
    "    'Restoration of the Priesthood',\n",
    "    'Sacrament Meeting',\n",
    "    'Same-Sex Marriage',\n",
    "    'Satan',\n",
    "    'Signs',\n",
    "    'Spaulding Manuscript',\n",
    "    'Stake',\n",
    "    'Tattooing and Body Piercing',\n",
    "    'Teacher (Aaronic Priesthood)',\n",
    "    'Standard Works',\n",
    "    'Telestial Kingdom',\n",
    "    'Terrestrial Kingdom',\n",
    "    'The Manifesto and the End of Plural Marriage',\n",
    "    'Transgression',\n",
    "    'Translation and Historicity of the Book of Abraham',\n",
    "    'Unwed Pregnancy',\n",
    "    'Urim and Thummim'\n",
    "]\n",
    "\n",
    "topics_to_remove = [item.lower() for item in topics_to_remove]\n",
    "filter = topics_df['topic'].apply(lambda x: x.lower() not in topics_to_remove)\n",
    "topics_df = topics_df[filter.values]\n",
    "\n",
    "topics_df.to_csv('data/topics_gt_mod.csv', index=False)\n",
    "print(f\"Success! Filtered to {len(topics_df.index)} topics.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine similar topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Built-in packages\n",
    "import time                     # measure code execution time\n",
    "import re                       # text-based pattern matching\n",
    "import unicodedata              # Unicode character reference (database on character properties)\n",
    "                                # for normalizing characters. See also unicodedata2: https://pypi.org/project/unicodedata2/\n",
    "\n",
    "# Third-party packages\n",
    "import numpy as np              # array-based data storage and computation\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd             # tabular data analysis\n",
    "import seaborn as sns           # statistical data visualization\n",
    "\n",
    "# Create a doc-terms matrix for the topics, to find out which ones are most similar\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "pd.options.display.precision = 3  # Show 3 decimal places of precision in DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Text pre-processing\n",
    "\n",
    "Function to normalize text (convert Unicode to ASCII equivalents), convert to lowercase, and remove non-alphabetic characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_text(orig_str):\n",
    "    # Normalize after lowercasing\n",
    "    # Third-party method: works with most Latin-looking characters and punctuation.\n",
    "    # new_str = unidecode.unidecode(orig_str.lower())\n",
    "\n",
    "    # Built-in normalizing method: fails to convert “” to \"\" and ‘’ to '',\n",
    "    # but good enough for my purposes since I remove those characters anyways.\n",
    "    # TfidfVectorizer also doesn't remove those characters, or numbers either.\n",
    "    new_str = unicodedata.normalize(\n",
    "        'NFKD',\n",
    "        orig_str.lower()\n",
    "    ).encode(encoding='ascii', errors='ignore').decode()\n",
    "\n",
    "    # Remove digits and punctuation (and anything else besides lowercase a-z and whitespace)\n",
    "    new_str = re.sub(pattern=r'[^a-z\\s]', repl=' ', string=new_str)\n",
    "    \n",
    "    # Replace multiple whitespace chars with a single space\n",
    "    new_str = re.sub(pattern=r'[\\s]{2,}', repl=' ', string=new_str)\n",
    "\n",
    "    return new_str"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic consolidation\n",
    "Create two DataFrames: one with all topics that will be considered (`full_topics_df`), and one with only the topics that will be kept in the final, cleaned version of the topic articles (`topics_df`).\n",
    "\n",
    "For topics that are in `full_topics_df` but not in `topics_df`, the text of those excluded topics will be joined (appended to) the most similar topic kept in `topics_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting number of topics: 223.\n",
      "After filtering, 183 topics remain.\n"
     ]
    }
   ],
   "source": [
    "# Load the modified version to make additional changes (e.g., removing topics)\n",
    "full_topics_df = pd.read_csv('data/topics_gt_mod_old.csv')\n",
    "print(f\"Starting number of topics: {len(full_topics_df.index)}.\")\n",
    "\n",
    "# The following topics are overrepresented in the data and don't represent\n",
    "# true topics for magazine articles, but rather are more like terms in a glossary\n",
    "topics_to_remove = [\n",
    "    # 'Aaronic Priesthood',\n",
    "    'Abraham, Book of',\n",
    "    # 'Apostle',\n",
    "    'Are “Mormons” Christian?',\n",
    "    'Articles of Faith',\n",
    "    'Bible, Inerrancy of',\n",
    "    'Birth Control',\n",
    "    # 'Bishop',\n",
    "    'Book of Mormon and DNA Studies',\n",
    "    'Book of Mormon Geography',\n",
    "    'Book of Mormon Translation',\n",
    "    'Church Finances—Commercial Businesses',\n",
    "    # 'Cross',\n",
    "    'Daughters in My Kingdom',\n",
    "    # 'Deacon',\n",
    "    # 'Elder',\n",
    "    # 'First Presidency',\n",
    "    # 'First Vision Accounts',\n",
    "    'Gambling',\n",
    "    'Gold Plates',\n",
    "    'High Council',\n",
    "    # 'High Priest',\n",
    "    # 'Jesus Christ Chosen as Savior',\n",
    "    \"Joseph Smith’s Teachings about Priesthood, Temple, and Women\",\n",
    "    'Journal of Discourses',\n",
    "    'Laying On of Hands',\n",
    "    # 'Melchizedek Priesthood',\n",
    "    'Membership Councils',\n",
    "    'Mormon Church',\n",
    "    'Mormonism',\n",
    "    'Mormons',\n",
    "    # 'Mother in Heaven',\n",
    "    'Mountain Meadows Massacre',\n",
    "    'Noah',\n",
    "    'Original Sin',\n",
    "    # 'Patriarch',\n",
    "    'Peace and Violence among 19th-Century Latter-day Saints',\n",
    "    'Plural Marriage in The Church of Jesus Christ of Latter-day Saints',\n",
    "    # 'Priest',\n",
    "    # 'Priesthood Blessing',\n",
    "    # 'Priesthood and Race',\n",
    "    # 'Primary',\n",
    "    'Prison Ministry',\n",
    "    # 'Prophecy',\n",
    "    'Quorum',\n",
    "    # 'Quorum of the Twelve Apostles',\n",
    "    'Race and the Priesthood',\n",
    "    # 'Relief Society',\n",
    "    # 'Restoration of the Priesthood',\n",
    "    'Sacrament Meeting',\n",
    "    'Same-Sex Marriage',\n",
    "    'Satan',\n",
    "    'Signs',\n",
    "    'Spaulding Manuscript',\n",
    "    'Stake',\n",
    "    'Tattooing and Body Piercing',\n",
    "    # 'Teacher (Aaronic Priesthood)',\n",
    "    # 'Standard Works',\n",
    "    # 'Telestial Kingdom',\n",
    "    # 'Terrestrial Kingdom',\n",
    "    'The Manifesto and the End of Plural Marriage',\n",
    "    'Transgression',\n",
    "    'Translation and Historicity of the Book of Abraham',\n",
    "    'Unwed Pregnancy',\n",
    "    'Urim and Thummim'\n",
    "]\n",
    "\n",
    "topics_to_remove = [item.lower() for item in topics_to_remove]\n",
    "filter = full_topics_df['topic'].apply(lambda x: x.lower() not in topics_to_remove)\n",
    "full_topics_df = full_topics_df[filter.values]\n",
    "\n",
    "print(f\"After filtering, {len(full_topics_df.index)} topics remain.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting number of topics: 183.\n",
      "After filtering, 75 topics remain.\n"
     ]
    }
   ],
   "source": [
    "print(f\"Starting number of topics: {len(full_topics_df.index)}.\")\n",
    "\n",
    "# The topics_to_keep list is meant to group together\n",
    "# similar topics (e.g., Celestial Kingdom into Kingdoms of Glory)\n",
    "topics_to_keep = [\n",
    "    'Addiction',\n",
    "    'Adversity',\n",
    "    'Agency and Accountability',\n",
    "    'Atonement of Jesus Christ',\n",
    "    'Baptism',\n",
    "    'Bible',\n",
    "    'Book of Mormon',\n",
    "    'Charity',\n",
    "    'Chastity',\n",
    "    'Christmas',\n",
    "    'Citizenship',\n",
    "    'Communication',\n",
    "    'Conversion',\n",
    "    'Covenant',\n",
    "    'Dating and Courtship',\n",
    "    'Diversity and Unity in The Church of Jesus Christ of Latter-day Saints',\n",
    "    'Easter',\n",
    "    'Education',\n",
    "    'Emergency Preparedness',\n",
    "    'Employment',\n",
    "    'Eternal Life',\n",
    "    'Faith in Jesus Christ',\n",
    "    'Family',\n",
    "    'Family Finances',\n",
    "    'Family History',\n",
    "    'Fasting and Fast Offerings',\n",
    "    'Forgiveness',\n",
    "    'Gospel',\n",
    "    'Grace',\n",
    "    'Gratitude',\n",
    "    'Grief',\n",
    "    'Happiness',\n",
    "    'Health',\n",
    "    'Heavenly Parents',\n",
    "    'Holy Ghost',\n",
    "    'Honesty',\n",
    "    'Hope',\n",
    "    'Humility',\n",
    "    'Jesus Christ',\n",
    "    'Joseph Smith',\n",
    "    'Judging Others',\n",
    "    'Kingdoms of Glory',\n",
    "    'Love',\n",
    "    'Marriage',\n",
    "    'Media',\n",
    "    'Mercy',\n",
    "    'Ministering',\n",
    "    'Modesty',\n",
    "    'Music',\n",
    "    'Obedience',\n",
    "    'Parenting',\n",
    "    'Peace',\n",
    "    'Peer Pressure',\n",
    "    'Plan of Salvation',\n",
    "    'Prayer',\n",
    "    'Priesthood',\n",
    "    'Prophets',\n",
    "    'Repentance',\n",
    "    'Restoration of the Church',\n",
    "    'Revelation',\n",
    "    'Sabbath Day',\n",
    "    'Scriptures',\n",
    "    'Second Coming of Jesus Christ',\n",
    "    'Self-Reliance',\n",
    "    'Service',\n",
    "    'Spiritual Gifts',\n",
    "    'Teaching the Gospel',\n",
    "    'Temples',\n",
    "    'Testimony',\n",
    "    'Tithing',\n",
    "    'Vicarious Work',\n",
    "    'Virtue',\n",
    "    'Women in the Church',\n",
    "    'Word of Wisdom',\n",
    "    'Worship'\n",
    "]\n",
    "\n",
    "\n",
    "topics_df = full_topics_df[full_topics_df['topic'].isin(topics_to_keep)]\n",
    "\n",
    "print(f\"After filtering, {len(topics_df.index)} topics remain.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data and create doc-terms matrix from the `full_topics_df` DataFrame. Later, that will be consolidated into the abbreviated list of topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Number of docs by number of terms:\n",
      "(183, 34083)\n",
      "\n",
      "Approximate size in memory: 464,928 bytes\n"
     ]
    }
   ],
   "source": [
    "# print(f\"Topic dataset loaded. Number of topics: {len(topics_df.index)}\")\n",
    "\n",
    "NGRAMS = 2\n",
    "\n",
    "count_vectorizer = CountVectorizer(\n",
    "    preprocessor = preprocess_text,\n",
    "    ngram_range = (1, NGRAMS),\n",
    "    stop_words = 'english'\n",
    ")\n",
    "\n",
    "# Fit the doc-terms matrix\n",
    "# dtm = count_vectorizer.fit_transform(topics_df['text'])\n",
    "dtm = count_vectorizer.fit_transform(full_topics_df['text'])\n",
    "\n",
    "# Normalize by word count. The .reshape turns the sum into a column vector used in the divide operation\n",
    "dtm_normalized =  dtm / dtm.sum(axis=1).reshape(-1, 1)\n",
    "\n",
    "# Check vocabulary size\n",
    "print(f\"\\nNumber of docs by number of terms:\\n{dtm.shape}\")\n",
    "\n",
    "# View the size, in bytes, of the doc-to-terms matrix\n",
    "print(f\"\\nApproximate size in memory: {dtm.data.nbytes:,.0f} bytes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare topic similarity using matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of correlation matrix: (183, 183)\n"
     ]
    }
   ],
   "source": [
    "# Matrix multiplication using the @ operator.\n",
    "corr_matrix = dtm_normalized @ dtm_normalized.transpose()\n",
    "corr_matrix = np.array(corr_matrix)\n",
    "print(f\"Shape of correlation matrix: {corr_matrix.shape}\")\n",
    "\n",
    "df_corr = pd.DataFrame(\n",
    "    data=np.array(corr_matrix),\n",
    "    index=full_topics_df['topic'].values,\n",
    "    columns=full_topics_df['topic'].values\n",
    ")\n",
    "\n",
    "# Save for later use\n",
    "# df_corr.to_csv('experiments/topic_correlations_for_consolidation.csv')\n",
    "\n",
    "# Display heatmap of correlation matrix\n",
    "# fig, ax = plt.subplots(figsize=(30, 30))\n",
    "# sns.heatmap(df_corr)\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the most similar topics for each topic\n",
    "\n",
    "df_similar_topics = pd.DataFrame(\n",
    "    data = np.argsort(corr_matrix, axis=1)[:, ::-1], # reverse the order of the sorted columns so the max ones are first\n",
    "    index = full_topics_df['topic'].values\n",
    ").apply(\n",
    "    # Replace the index number (from argsort) with the word\n",
    "    lambda s: pd.Series([full_topics_df['topic'].values[i] for i in s]),\n",
    "    axis = 1)\n",
    "\n",
    "df_similar_topics.to_csv('experiments/similar_topics.csv')\n",
    "\n",
    "# Add score\n",
    "# df_similar_topics = df_similar_topics.join(\n",
    "#     pd.DataFrame(\n",
    "#         data = np.sort(corr_matrix, axis=1)[:, ::-1], # reverse the order of the sorted columns so the max ones are first\n",
    "#         index = full_topics_df['topic'].values\n",
    "#     ),\n",
    "#     rsuffix='_score'\n",
    "# )\n",
    "\n",
    "# df_similar_topics.to_csv('experiments/similar_topics_with_scores.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine text from excluded articles based on their most-similar included article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_topic_list = {k: [k] for k in topics_df['topic'].values}\n",
    "\n",
    "for row in df_similar_topics.itertuples():\n",
    "    # row[0] is the index (the topic word), and row[1:] contains the next most similar topics\n",
    "    if row[0] not in final_topic_list:\n",
    "        # This topic is excluded from the final topic list; append it to the next most simlar topic that is included\n",
    "        for t in row[1:]:\n",
    "            if t in final_topic_list:\n",
    "                final_topic_list[t].append(row[0])\n",
    "                break\n",
    "\n",
    "# View the topics included in the final topic list\n",
    "final_topic_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the similar-scored topics as a starting point, I now make a few manual adjustments to the categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_topic_list = {\n",
    "    'Addiction': [\n",
    "        'Addiction',\n",
    "        'Pornography'],\n",
    "    'Adversity': ['Adversity'],\n",
    "    'Agency and Accountability': [\n",
    "        'Agency and Accountability',\n",
    "        'Temptation',],\n",
    "    'Atonement of Jesus Christ': [\n",
    "        'Atonement of Jesus Christ',\n",
    "        'Cross',\n",
    "        'Fall of Adam and Eve',\n",
    "        'Sacrifice',\n",
    "        'Salvation'],\n",
    "    'Baptism': ['Baptism'],\n",
    "    'Bible': [\n",
    "        'Bible',\n",
    "        'New Testament',\n",
    "        'Old Testament'],\n",
    "    'Book of Mormon': ['Book of Mormon'],\n",
    "    'Charity': ['Charity'],\n",
    "    'Chastity': ['Chastity'],\n",
    "    'Christmas': ['Christmas'],\n",
    "    'Citizenship': [\n",
    "        'Citizenship',\n",
    "        'Religious Freedom'],\n",
    "    'Communication': ['Communication'],\n",
    "    'Conversion': ['Conversion'],\n",
    "    'Covenant': [\n",
    "        'Covenant', \n",
    "        'Abrahamic Covenant', \n",
    "        'Endowment',\n",
    "        'Ordinances'],\n",
    "    'Dating and Courtship': ['Dating and Courtship'],\n",
    "    'Diversity and Unity in The Church of Jesus Christ of Latter-day Saints': [\n",
    "        'Diversity and Unity in The Church of Jesus Christ of Latter-day Saints',\n",
    "        'Disabilities',\n",
    "        'Priesthood and Race',\n",
    "        'Racial and Cultural Prejudice',\n",
    "        'Same-Sex Attraction',\n",
    "        'Single Adult Members of the Church',\n",
    "        'Transgender',\n",
    "        'Unity',\n",
    "        'War'],\n",
    "    'Easter': [\n",
    "        'Easter', \n",
    "        'Resurrection'],\n",
    "    'Education': ['Education'],\n",
    "    'Emergency Preparedness': [\n",
    "        'Emergency Preparedness',\n",
    "        'Food Storage',\n",
    "        'Emergency Response'],\n",
    "    'Employment': ['Employment'],\n",
    "    'Eternal Life': ['Eternal Life'],\n",
    "    'Faith in Jesus Christ': [\n",
    "        'Faith in Jesus Christ', \n",
    "        'Answering Gospel Questions'],\n",
    "    'Family': [\n",
    "        'Family', \n",
    "        'Abortion',\n",
    "        'Family Councils',\n",
    "        'Home Evening',\n",
    "        'Single-Parent Families'],\n",
    "    'Family Finances': [\n",
    "        'Family Finances', \n",
    "        'Debt'],\n",
    "    'Family History': ['Family History'],\n",
    "    'Fasting and Fast Offerings': ['Fasting and Fast Offerings'],\n",
    "    'Forgiveness': ['Forgiveness'],\n",
    "    'Gospel': ['Gospel'],\n",
    "    'Grace': [\n",
    "        'Grace',\n",
    "        'Abuse',\n",
    "        'Miracles',\n",
    "        'Suicide'],\n",
    "    'Gratitude': ['Gratitude'],\n",
    "    'Grief': ['Grief'],\n",
    "    'Happiness': ['Happiness'],\n",
    "    'Health': ['Health'],\n",
    "    'Heavenly Parents': [\n",
    "        'Heavenly Parents',\n",
    "        'Mother in Heaven',\n",
    "        'God the Father',\n",
    "        'Spirit',\n",
    "        'Spirit Children of Heavenly Parents'],\n",
    "    'Holy Ghost': [\n",
    "        'Holy Ghost',\n",
    "        'Godhead',\n",
    "        'Light of Christ',\n",
    "        'Spiritual Experiences'],\n",
    "    'Honesty': ['Honesty'],\n",
    "    'Hope': ['Hope'],\n",
    "    'Humility': ['Humility'],\n",
    "    'Jesus Christ': ['Jesus Christ'],\n",
    "    'Joseph Smith': [\n",
    "        'Joseph Smith',\n",
    "        'First Vision',\n",
    "        'First Vision Accounts'],\n",
    "    'Judging Others': ['Judging Others'],\n",
    "    'Kingdoms of Glory': [\n",
    "        'Kingdoms of Glory',\n",
    "        'Celestial Kingdom',\n",
    "        'Heaven',\n",
    "        'Telestial Kingdom',\n",
    "        'Terrestrial Kingdom'],\n",
    "    'Love': ['Love'],\n",
    "    'Marriage': [\n",
    "        'Marriage',\n",
    "        'Divorce'],\n",
    "    'Media': [\n",
    "        'Media',\n",
    "        'Movies and Television'],\n",
    "    'Mercy': [\n",
    "        'Mercy',\n",
    "        'Justice'],\n",
    "    'Ministering': [\n",
    "        'Ministering',\n",
    "        'Bishop'],\n",
    "    'Modesty': [\n",
    "        'Modesty',\n",
    "        'Profanity'],\n",
    "    'Music': ['Music'],\n",
    "    'Obedience': [\n",
    "        'Obedience',\n",
    "        'Stewardship',\n",
    "        'Ten Commandments'],\n",
    "    'Parenting': [\n",
    "        'Parenting',\n",
    "        'Adoption',\n",
    "        'Primary',\n",
    "        'Sex Education and Behavior'],\n",
    "    'Peace': [\n",
    "        'Peace',\n",
    "        'Conscience'],\n",
    "    'Peer Pressure': ['Peer Pressure'],\n",
    "    'Plan of Salvation': [\n",
    "        'Plan of Salvation',\n",
    "        'Becoming Like God',\n",
    "        'Council in Heaven',\n",
    "        'Creation',\n",
    "        'Environmental Stewardship and Conservation',\n",
    "        'Death, Physical',\n",
    "        'Death, Spiritual',\n",
    "        'Foreordination',\n",
    "        'Hell',\n",
    "        'Immortality',\n",
    "        'Jesus Christ Chosen as Savior',\n",
    "        'Millennium',\n",
    "        'Mortality',\n",
    "        'Paradise',\n",
    "        'Postmortality',\n",
    "        'Premortality',\n",
    "        'Soul',\n",
    "        'Spirit World',\n",
    "        'War in Heaven'],\n",
    "    'Prayer': ['Prayer'],\n",
    "    'Priesthood': [\n",
    "        'Priesthood',\n",
    "        'Aaronic Priesthood',\n",
    "        'Deacon',\n",
    "        'Elder',\n",
    "        'High Priest',\n",
    "        'Patriarch',\n",
    "        'Priest',\n",
    "        'Priesthood Blessing',\n",
    "        'Melchizedek Priesthood',\n",
    "        'Teacher (Aaronic Priesthood)'],\n",
    "    'Prophets': [\n",
    "        'Prophets',\n",
    "        'Apostle',\n",
    "        'First Presidency',\n",
    "        'Prophecy',\n",
    "        'Quorum of the Twelve Apostles'],\n",
    "    'Repentance': [\n",
    "        'Repentance',\n",
    "        'Sin'],\n",
    "    'Restoration of the Church': [\n",
    "        'Restoration of the Church',\n",
    "        'Apostasy',\n",
    "        'Dispensations',\n",
    "        'Doctrine and Covenants',\n",
    "        'Pearl of Great Price',\n",
    "        'Restoration of the Priesthood',\n",
    "        'Zion'],\n",
    "    'Revelation': [\n",
    "        'Revelation',\n",
    "        'Patriarchal Blessings'],\n",
    "    'Sabbath Day': [\n",
    "        'Sabbath Day',\n",
    "        'Sacrament'],\n",
    "    'Scriptures': [\n",
    "        'Scriptures',\n",
    "        'Standard Works'],\n",
    "    'Second Coming of Jesus Christ': ['Second Coming of Jesus Christ'],\n",
    "    'Self-Reliance': [\n",
    "        'Self-Reliance',\n",
    "        'Gardening'],\n",
    "    'Service': ['Service'],\n",
    "    'Spiritual Gifts': ['Spiritual Gifts'],\n",
    "    'Teaching the Gospel': [\n",
    "        'Teaching the Gospel',\n",
    "        'Missionary Work'],\n",
    "    'Temples': [\n",
    "        'Temples',\n",
    "        'Garments',\n",
    "        'Sealing'],\n",
    "    'Testimony': [\n",
    "        'Testimony',\n",
    "        'Witness',],\n",
    "    'Tithing': [\n",
    "        'Tithing',\n",
    "        'Church Councils',\n",
    "        'Consecration'],\n",
    "    'Vicarious Work': [\n",
    "        'Vicarious Work',\n",
    "        'Baptisms for the Dead',\n",
    "        'Proxy Baptism'],\n",
    "    'Virtue': ['Virtue'],\n",
    "    'Women in the Church': [\n",
    "        'Women in the Church',\n",
    "        'Relief Society'],\n",
    "    'Word of Wisdom': ['Word of Wisdom'],\n",
    "    'Worship': [\n",
    "        'Worship',\n",
    "        'Reverence']\n",
    " }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new DataFrame with the final topic list, and save to file: `data/topics_gt_mod.csv` to indicate that this is a modified version of the topics list from the Gospel Topics articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set index of full_topics_df to 'topic' so items can be accessed more succinctly\n",
    "full_topics_df = full_topics_df.set_index('topic')\n",
    "\n",
    "data_dict = {\n",
    "    'url': topics_df['url'].values.tolist(),\n",
    "    'topic': topics_df['topic'].values.tolist(),\n",
    "    'text': []\n",
    "}\n",
    "\n",
    "for n, t in enumerate(final_topic_list):\n",
    "    text_to_add = ''\n",
    "    for subtopic in final_topic_list[t]:\n",
    "        text_to_add += '\\n' + full_topics_df.at[subtopic, 'text']\n",
    "    # Add the combined text to its position in the data dictionary, trimming the leading newline\n",
    "    data_dict['text'].append(text_to_add.strip())\n",
    "\n",
    "# Create DataFrame from dictionary\n",
    "final_topics_df = pd.DataFrame.from_dict(data_dict)\n",
    "\n",
    "# Save resulting topics list\n",
    "final_topics_df.to_csv('data/topics_gt_mod.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# <strong><span style=\"color: rgba(220, 85, 80, 1.0)\"> 🛑 STOP HERE</span></strong>\n",
    "\n",
    "When running the entire notebook, stop before this section to save computation time.\n",
    "\n",
    "I conduct problem-solving through experiments, exploring ideas and testing different approaches. I usually make note of what I learned from the experiments and save the code I wrote in case I want to refer to it later, but I keep the best part of it for the finished analysis, which is in the sections above.\n",
    "\n",
    "The sections below contain valuable notes documenting my problem-solving methodology and explaining what worked and what didn't. Review the code below to understand the steps I carried out before arriving at the finished code above, but **comment out the below sections if you plan to \"Run All\" cells in the notebook from top to bottom** to avoid redundant computation on problems that have already been explored and resolved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiments: data scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test for scraping image alt text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Liahona', 'Human Race', 'First Presidency 2018 Official Portraits Photography', 'Smartphone', 'Taking Notes']\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.churchofjesuschrist.org/study/liahona/2021/01/pointing-us-all-to-jesus-christ?lang=eng'\n",
    "test_response = requests.get(url)\n",
    "test_response.encoding = 'utf-8'\n",
    "test_page = BeautifulSoup(test_response.text)\n",
    "\n",
    "body_block = test_page.find('div', class_='body-block')\n",
    "img_alt_text = []\n",
    "prefix_chars = len('Image')\n",
    "for img in body_block.find_all(class_=re.compile('imageWrapper.*')):\n",
    "    img_alt_text.append(img.text.strip()[prefix_chars:])\n",
    "\n",
    "# img_alt_text = '\\n'.join(img_alt_text)\n",
    "# img_alt_text = re.sub('[\\n]{2,}', '\\n', img_alt_text)\n",
    "print(img_alt_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test for scraping article body text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://www.churchofjesuschrist.org/study/liahona/2021/04/united-states-and-canada-section/finding-blessings-in-tragedy?lang=eng'\n",
    "url = 'https://www.churchofjesuschrist.org/study/liahona/2021/01/lucy-mack-smith-a-faithful-witness'\n",
    "test_response = requests.get(url)\n",
    "test_response.encoding = 'utf-8'\n",
    "test_page = BeautifulSoup(test_response.text)\n",
    "\n",
    "body = test_page.find('div', class_='body')\n",
    "body_block = test_page.find('div', class_='body-block')\n",
    "\n",
    "images = body.find_all(class_=re.compile('imageWrapper.*'))\n",
    "if images:\n",
    "    image_alt_text = [img.text.strip() for img in images]\n",
    "else:\n",
    "    image_alt_text = None\n",
    "    \n",
    "image_captions = body.find_all('figcaption')\n",
    "if image_captions:\n",
    "    image_captions = [fig.text.strip() for fig in image_captions]\n",
    "else:\n",
    "    image_captions = None\n",
    "\n",
    "# If the element is an <aside>, <figure>, or <figcaption> tag,\n",
    "# or if the element is a <div> tag with class \"credit\" or \"imageWrapper...\",\n",
    "# ignore its text since it isn't part of the article's main body text.\n",
    "tags_to_omit = ['aside', 'figure', 'figcaption']\n",
    "classes_to_omit = ['credit', re.compile('imageWrapper.*')]\n",
    "text = []\n",
    "\n",
    "for ch in body_block.findChildren(recursive=False):\n",
    "    ch_class = ch.get('class')  # -> a list, or None\n",
    "    \n",
    "    if ch_class:\n",
    "        skip_class = any(cc in classes_to_omit for cc in ch_class)\n",
    "    else:\n",
    "        skip_class = False\n",
    "    \n",
    "    if ch.name in tags_to_omit or skip_class:\n",
    "        continue\n",
    "    else:\n",
    "        text.append(ch.text.strip())\n",
    "\n",
    "# Combine into a single string, with paragraphs delimitted by newlines.\n",
    "text = '\\n'.join(text)\n",
    "# Replace multiple newlines with a single newline.\n",
    "text = re.sub('[\\n]{2,}', '\\n', text)\n",
    "\n",
    "# Remove unwanted text that was included through child elements\n",
    "# (e.g., a section that, along with text, contained unwanted child elements like images or credits)\n",
    "credit_elems = body_block.find_all('div', class_='credit')\n",
    "if credit_elems:\n",
    "    for cr in credit_elems:\n",
    "        text = text.replace(cr.text.strip(), '')\n",
    "\n",
    "aside_elems = body_block.find_all('aside')\n",
    "if aside_elems:\n",
    "    for ae in aside_elems:\n",
    "        text = text.replace(ae.text.strip(), '')\n",
    "\n",
    "if image_alt_text:\n",
    "    for iat in image_alt_text:\n",
    "        text = text.replace(iat, '')\n",
    "\n",
    "if image_captions:\n",
    "    for ic in image_captions:\n",
    "        text = text.replace(ic, '')\n",
    "# Replace multiple whitespace characters with a single space\n",
    "text = re.sub('[\\s]{2,}', ' ', text)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test for how to delete items from a dictionary, which is used in the logic for implementing the `continue_last_saved` feature (i.e., picking up where the scraping left off)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Liahona': {'2020': {'url': 'https://www.example.org/'}}, 'Friend': {'2021': {'url': 'https://www.example.org/'}, '2020': {'url': 'https://www.example.org/'}}}\n"
     ]
    }
   ],
   "source": [
    "test_dict = {\n",
    "    'Liahona': {\n",
    "        '2021': {'url': 'https://www.example.org/'},\n",
    "        '2020': {'url': 'https://www.example.org/'}\n",
    "    },\n",
    "    'Friend': {\n",
    "        '2021': {'url': 'https://www.example.org/'},\n",
    "        '2020': {'url': 'https://www.example.org/'}\n",
    "    },\n",
    "}\n",
    "\n",
    "new_test_dict = test_dict.copy()\n",
    "\n",
    "test_index = list(test_dict['Liahona']).index('2020')\n",
    "\n",
    "del_years = []\n",
    "for yr_num in test_dict['Liahona']:\n",
    "    if list(test_dict['Liahona']).index(yr_num) < test_index:\n",
    "        del_years.append(yr_num)\n",
    "\n",
    "for dy in del_years:\n",
    "    del new_test_dict['Liahona'][dy]\n",
    "\n",
    "print(new_test_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tVR1Azfrrcwk"
   },
   "source": [
    "Test: extracting the month from the URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8ME9mz4Ck6rB",
    "outputId": "71f406cd-f31a-4cbb-db41-dda3d82095c8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "01\t01\thttps://www.churchofjesuschrist.org/study/liahona/2017/01?lang=eng\n",
      "02\t02\thttps://www.churchofjesuschrist.org/study/liahona/2017/02?lang=eng\n",
      "03\t03\thttps://www.churchofjesuschrist.org/study/liahona/2017/03?lang=eng\n",
      "04\t04\thttps://www.churchofjesuschrist.org/study/liahona/2017/04?lang=eng\n",
      "05\t05\thttps://www.churchofjesuschrist.org/study/liahona/2017/05?lang=eng\n",
      "06\t06\thttps://www.churchofjesuschrist.org/study/liahona/2017/06?lang=eng\n",
      "07\t07\thttps://www.churchofjesuschrist.org/study/liahona/2017/07?lang=eng\n",
      "08\t08\thttps://www.churchofjesuschrist.org/study/liahona/2017/08?lang=eng\n",
      "09\t09\thttps://www.churchofjesuschrist.org/study/liahona/2017/09?lang=eng\n",
      "10\t10\thttps://www.churchofjesuschrist.org/study/liahona/2017/10?lang=eng\n",
      "11-se\t11-se\thttps://www.churchofjesuschrist.org/study/liahona/2017/11-se?lang=eng\n",
      "11\t11\thttps://www.churchofjesuschrist.org/study/liahona/2017/11?lang=eng\n",
      "12\t12\thttps://www.churchofjesuschrist.org/study/liahona/2017/12?lang=eng\n",
      "digital\tdigital\thttps://www.churchofjesuschrist.org/study/liahona/2017/digital?lang=eng\n"
     ]
    }
   ],
   "source": [
    "all_matches = []\n",
    "issue_months = []\n",
    "\n",
    "for url in magdata.all_issues['Liahona']['2017']['issue_urls']:\n",
    "    # Version using string methods\n",
    "    q_pos = url.rfind('?')\n",
    "    # The last forward slash\n",
    "    start_pos = url.rfind('/')\n",
    "    issue_months.append(url[start_pos + 1:q_pos])\n",
    "\n",
    "    # Version using RegEx\n",
    "    # What this re is doing:\n",
    "    # starts with (?<=...) four digits followed by a forward slash (/)\n",
    "    # ends with (?=...) a question mark\n",
    "    # capture everything in between that start and end point\n",
    "    pattern = re.compile(r'(?<=[0-9]{4}[/]).+(?=\\?)')\n",
    "    match = re.search(pattern, url)\n",
    "    if match:\n",
    "        all_matches.append(match.group(0))\n",
    "\n",
    "for string_methods_month, month, url in zip(issue_months, all_matches, magdata.all_issues['Liahona']['2017']['issue_urls']):\n",
    "    print(f\"{string_methods_month}\\t{month}\\t{url}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\nAny suffering “can be made right through the Atonement of Jesus Christ.”\\n', '\\n“When [God] hath tried me,” Job learned, “I shall come forth as gold.”\\n']\n",
      "[]\n",
      "[<img alt=\"\" height=\"119\" width=\"100\"/>, <img alt=\"\" height=\"500\" width=\"500\"/>, <img alt=\"\" height=\"500\" width=\"500\"/>]\n"
     ]
    }
   ],
   "source": [
    "# Test retrieving alt text for img tags\n",
    "# Appears to be that alt text is generated by a JavaScript script and is\n",
    "# not present in the HTML DOM. Perhaps that helps accommodate multiple languages?\n",
    "\n",
    "url = 'https://www.churchofjesuschrist.org/study/liahona/2022/08/03_trust-god-and-let-him-prevail?lang=eng'\n",
    "# url = 'https://www.churchofjesuschrist.org/study/liahona/2008/11/you-know-enough?lang=eng'\n",
    "# url = 'https://www.churchofjesuschrist.org/study/liahona/1991/02/the-towers-of-chartres?lang=eng'\n",
    "response = requests.get(url, timeout=None)\n",
    "response.encoding = 'utf-8'\n",
    "page = BeautifulSoup(response.text)\n",
    "\n",
    "image_captions = page.find('div', class_='body').find_all('figcaption')\n",
    "if image_captions:\n",
    "    image_captions = [fig.text for fig in image_captions]\n",
    "    print(image_captions)\n",
    "else:\n",
    "    image_captions = None\n",
    "    print(\"No captions found\")\n",
    "\n",
    "images = page.find('div', class_='body').find_all('img', {'alt': True})\n",
    "if images:\n",
    "    image_alt_text = [img.get('alt', {'alt': True}) for img in images if img.get('alt')]\n",
    "    # Convert to a string representation, delimitted by \"; \"\n",
    "    # image_alt_text = '; '.join(image_alt_text)\n",
    "    print(image_alt_text)\n",
    "    print(images)\n",
    "else:\n",
    "    image_alt_text = None\n",
    "    print(\"No images found\")\n",
    "\n",
    "\n",
    "# Image alt text is generated by JavaScript; it is not available in the HTML,\n",
    "# so I am commenting out this section for now.\n",
    "# The alt text could be retrieved by using Selenium in headless mode or \n",
    "#  by using the requests-html package and rendering the page before extracting\n",
    "#  alt text, but Selenium takes more memory than requests, and requests-html does not work in a Jupyter notebook.\n",
    "# See, for example: https://stackoverflow.com/questions/53469230/beautifulsoup-not-extracting-image-alt-text\n",
    "# image_alt_text = None\n",
    "# images = body.find_all('img', alt=True)\n",
    "# if images:\n",
    "#     image_alt_text = [\n",
    "#         img.get('alt') \n",
    "#         for img in images]\n",
    "#     if image_alt_text:\n",
    "#         # Convert to a string representation delimitted by \";\"\n",
    "#         image_alt_text = ';'.join(image_alt_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NBuS9cpCHfR6"
   },
   "source": [
    "### Estimate memory requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "15xv6sPYHnOS"
   },
   "outputs": [],
   "source": [
    "def estimate_memory_usage(magazine_issues: dict, root_url: str, sample_size: int=10) -> float:\n",
    "    '''\n",
    "    Estimates the memory (RAM) needed to store all the data scraped from\n",
    "    magazine issues listed in the `magazine_issues` dictionary.\n",
    "    \n",
    "    Each character of text requires approximately 1 byte of memory,\n",
    "    in addition to the memory overhead for Python objects. This function simply\n",
    "    counts the number of characters in a sample of `sample_size` number of\n",
    "    issues from the `magazine_issues` dictionary, then uses the average number\n",
    "    of characters per issue to estimate the total number of characters of\n",
    "    text in all articles.\n",
    "\n",
    "    Note that this function underestimates memory usage, since it does not\n",
    "    account for fields to be captured in addition to article text, such as\n",
    "    author name, article URL, or magazine issue number.\n",
    "\n",
    "    The estimate serves as a guideline to the user, who determines whether\n",
    "    to save the scraped data to a .csv file during the web scraping process or\n",
    "    to hold all data in memory until the end, when a .csv file will be created\n",
    "    before the memory is released. Due to the relative speed of RAM compared\n",
    "    with disk access, the in-memory option is much faster, but will result in\n",
    "    greater memory usage as the function runs. Saving progress to disk at \n",
    "    each step reduces the memory footprint (and safeguards against function\n",
    "    failure partway) at the expense of addititional computation time.\n",
    "\n",
    "    Parameters\n",
    "    ---\n",
    "    `magazine_issues`: dict\n",
    "        The dictionary returned by the function: `get_magazine_issues`\n",
    "    \n",
    "    `root_url`: str\n",
    "        The URL to the domain's main page. Necessary because links on any\n",
    "        issue or article page are relative links, rather than absolute.\n",
    "\n",
    "    `sample_size`: int, default=10\n",
    "        The number of magazine issues to sample prior to making an estimate.\n",
    "        This number of issues will be sampled for each magazine in the\n",
    "        `magazine_issues` dictionary.\n",
    "\n",
    "    Return value\n",
    "    ---\n",
    "        Returns a float approximating the number of bytes required to store\n",
    "        all magazine data in RAM.\n",
    "    '''\n",
    "\n",
    "    # Start a web requests session\n",
    "    sesh = requests.session()\n",
    "\n",
    "    chars_per_issue = {mag_name: [] for mag_name in magazine_issues}\n",
    "    articles_per_issue = {mag_name: [] for mag_name in magazine_issues}\n",
    "    \n",
    "    # Create progress bar\n",
    "    pbar = tqdm(total=(sample_size * len(magazine_issues)))\n",
    "    \n",
    "    for mag_name in magazine_issues:\n",
    "        sampled_years = random.sample(\n",
    "            list(magazine_issues[mag_name].keys()),\n",
    "            k=sample_size)\n",
    "        for yr_num, yr in enumerate(sampled_years):\n",
    "            # Select a random issue from that year\n",
    "            issue_url = random.choice(magazine_issues[mag_name][yr]['issue_urls'])\n",
    "            issue_chars = 0\n",
    "            # Open the webpage with the magazine issue's contents,\n",
    "            #  with no timeout (so Requests waits until the page is returned)\n",
    "            #  See: https://requests.readthedocs.io/en/master/user/advanced/#timeouts\n",
    "            issue_page = sesh.get(issue_url, timeout=None)\n",
    "            issue_page.encoding = 'utf-8'\n",
    "            # Parse the page's text\n",
    "            issue_page = BeautifulSoup(issue_page.text, 'lxml')\n",
    "            # Find links to all articles in the issue\n",
    "            issue_contents = issue_page.find('nav', {'class': 'manifest'})\n",
    "            if issue_contents != None:\n",
    "                # 💡 Consider using a standard for loop and only prepending\n",
    "                #  the root_url if the link is a relative link,\n",
    "                #  that is: a['href'].startswith('/')\n",
    "                article_links = [\n",
    "                    root_url + a['href'] \n",
    "                    for a in issue_contents.find_all('a') \n",
    "                    if a.text != \"Contents\"]\n",
    "                articles_per_issue[mag_name].append(len(article_links))\n",
    "            else:\n",
    "                print(f\"No <nav class=\\\"manifest\\\"> tag found on page. {issue_url}\")\n",
    "                # Move to next issue\n",
    "                continue\n",
    "            # Access each article to determine its length (in characters)\n",
    "            for article_num, link in enumerate(article_links):\n",
    "                pbar.set_description(\n",
    "                    f\"Working on {mag_name}, \"\n",
    "                  + f\"issue {yr_num + 1:,d}/{sample_size:,d}, \"\n",
    "                  + f\"article {article_num + 1:,d}/{len(article_links):,d}\")\n",
    "                # Pause for 100ms between page requests as a courtesy to reduce server load\n",
    "                time.sleep(0.1)\n",
    "                # Open webpage\n",
    "                response = sesh.get(link, timeout=None)\n",
    "                # Set the encoding used when returning text from the page\n",
    "                response.encoding = 'utf-8'\n",
    "                article_page = BeautifulSoup(response.text, 'lxml')\n",
    "                article_text = article_page.find('div', class_='body-block')\n",
    "                if article_text != None:\n",
    "                    issue_chars += len(article_text.text.strip())\n",
    "                else:\n",
    "                    print(f\"No <div class=\\\"body-block\\\"> tag found on page. Could not load text from article: {link}\")\n",
    "                   \n",
    "            # Add the number of characters in this issue to the running list\n",
    "            chars_per_issue[mag_name].append(issue_chars)\n",
    "            pbar.update(n=1)\n",
    "    \n",
    "    # Compute the average characters per issue from the sampled issues\n",
    "    for mag_name in chars_per_issue:\n",
    "        chars_per_issue[mag_name] = round(\n",
    "            sum(chars_per_issue[mag_name]) / len(chars_per_issue[mag_name]), 0)\n",
    "    \n",
    "    # Count the number of issues in each magazine\n",
    "    issues_per_mag = {}\n",
    "\n",
    "    for mag_name in magazine_issues:\n",
    "        issue_count = 0\n",
    "        for yr in magazine_issues[mag_name]:\n",
    "            issue_count += len(magazine_issues[mag_name][yr]['issue_urls'])\n",
    "        issues_per_mag[mag_name] = issue_count\n",
    "\n",
    "    # Estimate the characters in all issues\n",
    "    total_chars = 0\n",
    "    for mag_name in chars_per_issue:\n",
    "        total_chars += (chars_per_issue[mag_name] * issues_per_mag[mag_name])\n",
    "    \n",
    "    print(f\"Estimated memory required: {total_chars:,.0f} bytes.\")\n",
    "    print(f\"Number of articles per issue:\\n{articles_per_issue}\")\n",
    "\n",
    "    return total_chars"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 356,
     "referenced_widgets": [
      "da558aa344b04828857710d3573fbb85",
      "ef3443e9468d48b7ba62a29ecc5b3b81",
      "83bd8ea9dca5464cb55e496317ed3ce2",
      "f8ffb95649d548ee8b35100a0ce283a4",
      "49fba9efa7a24fb4876d7a5539ad4bf4",
      "fbcd728592eb41a5839aeb9f731ffda1",
      "6c7854baf0ad4470ba6706519c0ce54f",
      "15c8e940283f424caccd30c44ab37cf9",
      "47d387bde88644f899ca71d4bea46231",
      "b512ef8ebd8c42218dc92072153b8ad4",
      "aaa721c0e5f149639d15e9d4a079954d"
     ]
    },
    "id": "9zwBtM8x84EK",
    "outputId": "b1b3a129-dd38-4c3e-9edf-bb592ebbaf62"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da558aa344b04828857710d3573fbb85",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/40 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1988/10/tambulilit\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/2000/11/the-friend\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1980/02/childrens-section\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1984/01/tambulilit\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1994/11/tambulilit\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1989/09/tambulilit\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1993/08/tambulilit\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/liahona/1981/12/childrens-section\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/ensign/2006/10/latter-day-saint-voices\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/ensign/2007/12/latter-day-saint-voices\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/new-era/2009/05/whats-up\n",
      "No <div class=\"body-block\"> tag found on page. Could not load text from article: https://www.churchofjesuschrist.org/study/new-era/2009/05/instant-messages\n",
      "Estimated memory required: 316,160,272 bytes.\n",
      "Number of articles per issue:\n",
      "{'Liahona': [20, 21, 15, 15, 18, 41, 22, 23, 14, 64], 'Ensign': [38, 43, 31, 45, 39, 31, 33, 47, 55, 40], 'New Era': [20, 18, 19, 29, 14, 16, 29, 16, 18, 14], 'Friend': [25, 27, 24, 41, 26, 24, 28, 35, 28, 27]}\n"
     ]
    }
   ],
   "source": [
    "est_memory_reqd = estimate_memory_usage(\n",
    "    magazine_issues=magdata.all_issues, root_url=ROOT_URL, sample_size=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "tu9th4YSoRU3",
    "outputId": "a8acfdb4-c7bd-47d4-b675-ff8cf2b779aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Estimated memory required to store text from all issues of all magazines from 1970-2020:\n",
      "316.16 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"Estimated memory required to store text from all issues of all magazines\"\n",
    "   + f\" from 1970-2020:\\n{(est_memory_reqd / 10**6):,.2f} MB\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sGLnXhOVqTtw",
    "outputId": "32d00452-f7b4-4bff-a5f1-dafd870c2fe6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average number of articles per magazine:\n",
      "----------------------------------------\n",
      "Liahona: 25\n",
      "Ensign: 40\n",
      "New Era: 19\n",
      "Friend: 28\n"
     ]
    }
   ],
   "source": [
    "# I copied this dictionary from the output printed two cells above\n",
    "article_count = {\n",
    "    'Liahona': [20, 21, 15, 15, 18, 41, 22, 23, 14, 64],\n",
    "    'Ensign': [38, 43, 31, 45, 39, 31, 33, 47, 55, 40],\n",
    "    'New Era': [20, 18, 19, 29, 14, 16, 29, 16, 18, 14],\n",
    "    'Friend': [25, 27, 24, 41, 26, 24, 28, 35, 28, 27]\n",
    "    }\n",
    "\n",
    "msg = \"Average number of articles per magazine:\"\n",
    "print(msg, '\\n', '-' * len(msg), sep='')\n",
    "for mag_name in article_count:\n",
    "    total_articles = sum(article_count[mag_name])\n",
    "    total_issues = len(article_count[mag_name])\n",
    "    avg = total_articles / total_issues\n",
    "    print(f\"{mag_name}: {avg:,.0f}\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "40 yrs of Magazine Issues.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "2beeefc7dc4dab53ba5ba31461f2b0762373c43d11ea67239efa966a02061b9c"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0e31aa237db740178870fbc4c9648375": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_81eb07cd8167457198593a185f2cb4bc",
      "max": 196,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_99b93b6cf506497f9b0423f056eb4010",
      "value": 196
     }
    },
    "15c8e940283f424caccd30c44ab37cf9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "20e2229011e04365a674a0ba58cbd478": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "23a28b228afa441db03fa8602a676c41": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "36225da9ff9e4d488d72b12e039c9cee": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f4b63e282796488f8a5febfbd1c0417a",
       "IPY_MODEL_0e31aa237db740178870fbc4c9648375",
       "IPY_MODEL_b2adfdbea95a4cda86363d7117857f88"
      ],
      "layout": "IPY_MODEL_4a2e558a80774b8283cc0e5f3a9e8b20"
     }
    },
    "47d387bde88644f899ca71d4bea46231": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "49fba9efa7a24fb4876d7a5539ad4bf4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4a2e558a80774b8283cc0e5f3a9e8b20": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c0d34606c0e43bfab79fa351b1cdf73": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c7854baf0ad4470ba6706519c0ce54f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "81eb07cd8167457198593a185f2cb4bc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "83bd8ea9dca5464cb55e496317ed3ce2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_15c8e940283f424caccd30c44ab37cf9",
      "max": 40,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_47d387bde88644f899ca71d4bea46231",
      "value": 40
     }
    },
    "8f9e62fc2edb4a7ea1a007df6ee2f2c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "99b93b6cf506497f9b0423f056eb4010": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "aaa721c0e5f149639d15e9d4a079954d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b2adfdbea95a4cda86363d7117857f88": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_23a28b228afa441db03fa8602a676c41",
      "placeholder": "​",
      "style": "IPY_MODEL_8f9e62fc2edb4a7ea1a007df6ee2f2c9",
      "value": " 196/196 [13:04&lt;00:00,  1.31it/s]"
     }
    },
    "b512ef8ebd8c42218dc92072153b8ad4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "da558aa344b04828857710d3573fbb85": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ef3443e9468d48b7ba62a29ecc5b3b81",
       "IPY_MODEL_83bd8ea9dca5464cb55e496317ed3ce2",
       "IPY_MODEL_f8ffb95649d548ee8b35100a0ce283a4"
      ],
      "layout": "IPY_MODEL_49fba9efa7a24fb4876d7a5539ad4bf4"
     }
    },
    "ef3443e9468d48b7ba62a29ecc5b3b81": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fbcd728592eb41a5839aeb9f731ffda1",
      "placeholder": "​",
      "style": "IPY_MODEL_6c7854baf0ad4470ba6706519c0ce54f",
      "value": "Working on Friend, issue 10/10, article 27/27: 100%"
     }
    },
    "f4b63e282796488f8a5febfbd1c0417a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6c0d34606c0e43bfab79fa351b1cdf73",
      "placeholder": "​",
      "style": "IPY_MODEL_20e2229011e04365a674a0ba58cbd478",
      "value": "Retrieving issues for: Friend, 1971: 100%"
     }
    },
    "f8ffb95649d548ee8b35100a0ce283a4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b512ef8ebd8c42218dc92072153b8ad4",
      "placeholder": "​",
      "style": "IPY_MODEL_aaa721c0e5f149639d15e9d4a079954d",
      "value": " 40/40 [28:20&lt;00:00, 24.79s/it]"
     }
    },
    "fbcd728592eb41a5839aeb9f731ffda1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}