Warm_Up_Project_Boilerplate.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Warm_Up_Project_Boilerplate.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdSng4W0E3Xy"
      },
      "source": [
        "Installing the Spark Dependancies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ZioYLN4EIgt"
      },
      "source": [
        "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n",
        "!wget -q https://downloads.apache.org/spark/spark-3.0.3/spark-3.0.3-bin-hadoop3.2.tgz\n",
        "!tar xf spark-3.0.3-bin-hadoop3.2.tgz\n",
        "!pip install -q findspark\n",
        "!pip install pyspark==3.0.3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "glH6hqJzFA4P"
      },
      "source": [
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
        "os.environ[\"SPARK_HOME\"] = \"/content/spark-3.0.3-bin-hadoop3.2\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ZkRDJa9Fd8M"
      },
      "source": [
        "Copy the Dataset into the local store (stored under \"/content\" folder in Colab)\n",
        "\n",
        "Dataset description can be found here: https://github.com/nytimes/covid-19-data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qadIjn_Z6Giz"
      },
      "source": [
        "! wget https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iOq5eADQhbyw"
      },
      "source": [
        "**Start your program**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u6dZKZx1FAt1"
      },
      "source": [
        "import findspark\n",
        "findspark.init()\n",
        "from pyspark import SparkContext, SparkConf\n",
        "from pyspark.sql import SQLContext, SparkSession"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6WY_BKuuE94M"
      },
      "source": [
        "Create a Spark Session"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oc1d5Df2FMJ5"
      },
      "source": [
        "conf = SparkConf().set('spark.ui.port', '4050')\n",
        "sc = SparkContext(conf=conf)\n",
        "spark = SparkSession.builder.master('local[*]').getOrCreate()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RMLGiIF0Z2wf"
      },
      "source": [
        "Read the dataset as a Dataframe"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C-_0QqoE48AL"
      },
      "source": [
        "# Load the dataset\n",
        "data = spark.read.load('/content/us-counties.csv', format='csv', inferSchema=True, header=True)\n",
        "\n",
        "# Print schema\n",
        "data.printSchema()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K9fE-buM7rOl"
      },
      "source": [
        "# See first 10 rows of the dataset\n",
        "data.head(10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mZS23AZqZ81a"
      },
      "source": [
        "**Task 0. Find the daily new cases across the entire US and plot**\n",
        "(*you DO not need to do this, this code is given for your assistance*)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pce2JYkU6dR1"
      },
      "source": [
        "from pyspark.sql import functions as F\n",
        "\n",
        "# Aggregate by day, sum the cases for all counties for each day\n",
        "daily_cumulative = data.groupby('date').agg(F.sum('cases').alias('total_cases'))\n",
        "daily_cumulative = daily_cumulative.sort('date')\n",
        "daily_cumulative.show(10)\n",
        "\n",
        "# Convert Spark dataframe to a Panda dataframe to plot\n",
        "plot_data = daily_cumulative.toPandas()\n",
        "dates = plot_data['date']\n",
        "values = plot_data['total_cases']\n",
        "\n",
        "# Find daily news cases from cumulative cases\n",
        "daily_cases = [values[i+1] - values[i] for i in range(len(values)-1)]\n",
        "ddates = [dates[i+1] for i in range(len(values)-1)]\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "plt.plot(ddates, daily_cases)\n",
        "plt.xlabel('Date')\n",
        "plt.ylabel('Daily new cases')\n",
        "plt.xticks(np.arange(1, len(daily_cases),30), [ddates[i] for i in np.arange(1, len(daily_cases),30)], rotation=90)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uSO9J5J5HN0n"
      },
      "source": [
        "Now, solve any **three** tasks from the follwing (BONUS score if you do all!).\n",
        "\n",
        "*Feel free to add more code blocks withinn each task to spereate the codes for better clarity and understanding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FHPGd1tDHrtV"
      },
      "source": [
        "**Task 1: Find the total number of new cases added in the entire US in the  month of March 2020.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RaaFOC-xHy7e"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JCD7_J82HzqE"
      },
      "source": [
        "**Task 2: Calculate the total new cases added in three consecutive months of June, July, and August of 2020 in Jackson county, Missouri (fips code 29095).**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WGFyaUb0IOb4"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JtcLCf3HIPnx"
      },
      "source": [
        "**Task 3: Find the daily new cases per month per 1000 population in Missouri state (MO) since the beginning of the pandemic (assume MO's population is 6,154,913). [Plot the data]**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U7qQr0QhIVI-"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "neUh4Ez3IV8o"
      },
      "source": [
        "**Task 4:  On which date all 50 US states have at least 100 cases? At least one death?**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YedPgvtyIaY0"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9i7LuBxIa2C"
      },
      "source": [
        "**Task 5: Which single day in the year 2020 had the largest number of deaths in the entire US (if there are multiple such dates, choose the earliest one)?**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kY8peDkmIgCd"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iWPsgNrbIihP"
      },
      "source": [
        "Your Warm-up Project ends here.\n",
        "Thank you."
      ]
    }
  ]
}