8_Data_Cleaning_&_Missing_Values_Project.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Data Cleaning & Missing Values - Project.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2zxREd_otAge"
      },
      "source": [
        "![Immoscout](https://raw.githubusercontent.com/juliandnl/redi_ss20/master/image.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YAmw_s-VtIaV"
      },
      "source": [
        "# Project - Cleaning the Expanded Immobilien Scout 24 Dataset\n",
        "\n",
        "--- \n",
        "After exploring the original dataset and trying to build a predictive model with it, it was discovered that we need more data. Unfortunately we lost access to Immobilien Scout's API so we had to scrape the data. While this worked, the data is quite messy and needs to be cleaned before we can use it. In this exercise you'll clean the remaining of the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cDxFVRHbq6eR"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import plotly.express as px"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GD2ryFwto1a"
      },
      "source": [
        "# Read in dataset\n",
        "df = pd.read_pickle(\"/content/berlin_housing_with_scraped_class_cleaned.pkl\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 360
        },
        "id": "EFdOoJcZVrc1",
        "outputId": "0ba13f0d-dac4-4b6a-d16c-25d770fa2023"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>url</th>\n",
              "      <th>region</th>\n",
              "      <th>condition</th>\n",
              "      <th>rooms</th>\n",
              "      <th>rent</th>\n",
              "      <th>year_construction</th>\n",
              "      <th>space</th>\n",
              "      <th>cold_rent</th>\n",
              "      <th>utilities</th>\n",
              "      <th>heating_costs</th>\n",
              "      <th>security_deposit</th>\n",
              "      <th>parking_space</th>\n",
              "      <th>renovated_date</th>\n",
              "      <th>property_condition</th>\n",
              "      <th>furnishing</th>\n",
              "      <th>heating_type</th>\n",
              "      <th>energy_sources</th>\n",
              "      <th>energy_certificate</th>\n",
              "      <th>type_of_certificate</th>\n",
              "      <th>energy_requirement</th>\n",
              "      <th>efficiency_class</th>\n",
              "      <th>deposit_cleaned</th>\n",
              "      <th>n_missing</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.immobilienscout24.de/expose/116051687</td>\n",
              "      <td>Mitte</td>\n",
              "      <td>first_time_use</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2659.00</td>\n",
              "      <td>2019</td>\n",
              "      <td>117.20</td>\n",
              "      <td>2199.000000</td>\n",
              "      <td>+460€</td>\n",
              "      <td>in</td>\n",
              "      <td>3</td>\n",
              "      <td>120€</td>\n",
              "      <td>NaN</td>\n",
              "      <td>New</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Underfloor Heating</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Bedarfsausweis</td>\n",
              "      <td>NaN</td>\n",
              "      <td>B</td>\n",
              "      <td>7977.00</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.immobilienscout24.de/expose/115338103</td>\n",
              "      <td>Kreuzberg</td>\n",
              "      <td>first_time_use</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1200.00</td>\n",
              "      <td>2020</td>\n",
              "      <td>29.33</td>\n",
              "      <td>973.510000</td>\n",
              "      <td>+226,49€</td>\n",
              "      <td>in</td>\n",
              "      <td>2920.53</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>New</td>\n",
              "      <td>Gehobene Qualität</td>\n",
              "      <td>Central Heating</td>\n",
              "      <td>Fernwärme Energieausweis liegt vor</td>\n",
              "      <td>liegt vor</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2920.53</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.immobilienscout24.de/expose/116458710</td>\n",
              "      <td>Köpenick</td>\n",
              "      <td>well_kept</td>\n",
              "      <td>2.0</td>\n",
              "      <td>979.00</td>\n",
              "      <td>1997</td>\n",
              "      <td>83.61</td>\n",
              "      <td>799.000000</td>\n",
              "      <td>+90€</td>\n",
              "      <td>+90€</td>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Groomed</td>\n",
              "      <td>Normale Qualität</td>\n",
              "      <td>Central Heating</td>\n",
              "      <td>Fernwärme Energieausweis liegt vor</td>\n",
              "      <td>liegt vor</td>\n",
              "      <td>Verbrauchsausweis Energie­ver­brauchs­kennwert...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2937.00</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.immobilienscout24.de/expose/116573177</td>\n",
              "      <td>Wilmersdorf</td>\n",
              "      <td>well_kept</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1830.22</td>\n",
              "      <td>1900</td>\n",
              "      <td>171.18</td>\n",
              "      <td>911.544216</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>nan</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Unknown</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5490.66</td>\n",
              "      <td>13</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.immobilienscout24.de/expose/115925878</td>\n",
              "      <td>Kreuzberg</td>\n",
              "      <td>first_time_use</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2272.00</td>\n",
              "      <td>2020</td>\n",
              "      <td>88.27</td>\n",
              "      <td>1131.573505</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>nan</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>New</td>\n",
              "      <td>Gehobene Qualität</td>\n",
              "      <td>Central Heating</td>\n",
              "      <td>Fernwärme Energieausweis liegt zur Besichtigun...</td>\n",
              "      <td>liegt zur Besichtigung vor</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6816.00</td>\n",
              "      <td>8</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 url  ... n_missing\n",
              "0  https://www.immobilienscout24.de/expose/116051687  ...         5\n",
              "1  https://www.immobilienscout24.de/expose/115338103  ...         5\n",
              "2  https://www.immobilienscout24.de/expose/116458710  ...         4\n",
              "3  https://www.immobilienscout24.de/expose/116573177  ...        13\n",
              "4  https://www.immobilienscout24.de/expose/115925878  ...         8\n",
              "\n",
              "[5 rows x 23 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PZmVxvM_4C5V"
      },
      "source": [
        "## 0. Before cleaning, print out how many missing values (sorted for readability) each column has:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OELxGQNs-AhR"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UeSRMJD94EZ7"
      },
      "source": [
        "## 1. Clean: `heating_type`\n",
        "\n",
        "Please explain why you choose your strategy for filling the missing values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BpfSyexk-Cju"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RLAa7wgu4E0V"
      },
      "source": [
        "## 2. Clean: `renovated_date`\n",
        "\n",
        "Please explain why you choose your strategy for filling the missing values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HOpffadx-EhM"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BMqdgnI3Y7BN"
      },
      "source": [
        "## 3. Clean: `parking_space`\n",
        "\n",
        "Please explain why you choose your strategy for filling the missing values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PjMfP4MvY5Fr"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-FJavHVZM16"
      },
      "source": [
        "## 4. Clean: `utilities`\n",
        "\n",
        "Please explain why you choose your strategy for filling the missing values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1gCHzQ5JY5IS"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZrL7CM4ZRuA"
      },
      "source": [
        "## 5. Clean: `energy_requirement`\n",
        "\n",
        "Please explain why you choose your strategy for filling the missing values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A__3sJhWZR3P"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HtIY4a3v9uas"
      },
      "source": [
        "## (Bonus) Clean: `type_of_certificate`\n",
        "\n",
        "Hint: could two features be created from this data?\n",
        "\n",
        "As with before, explain why you chose your strategy :)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "letuSY50-9w3"
      },
      "source": [
        "## code here"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xnQMN5CQt90T"
      },
      "source": [
        "--- \n",
        "# Review\n",
        "\n",
        "This section is only for the reviewing team!\n",
        "\n",
        "Guideline how to review:\n",
        "https://docs.google.com/presentation/d/1YORFwlfVQo9ogj7jR9t6_pqxmGIlpBSGubbp1UdtcBQ/edit?usp=sharing\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        },
        "id": "t_oU8YSOt-b0",
        "outputId": "55a6f79b-d2a4-4707-f67f-01e452a6d4bb"
      },
      "source": [
        "#@title Review Criteria:\n",
        "%%html\n",
        "<h3><input type=\"checkbox\"> 1. Are all questions answered? <br></h3>\n",
        "<h3><input type=\"checkbox\"> 2. Does all code run through? <br></h3>\n",
        "<h3><input type=\"checkbox\"> 3. Are explanations of fill strategy provided for all relevant questions?  <br></h3>\n",
        "<h3><input type=\"checkbox\"> 4. Are different strategies used for continuous vs categorical data?  <br></h3>\n",
        "<h3><input type=\"checkbox\"> 5. Is the bonus question answered?  <br></h3> "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<h3><input type=\"checkbox\"> 1. Are all questions answered? <br></h3>\n",
              "<h3><input type=\"checkbox\"> 2. Does all code run through? <br></h3>\n",
              "<h3><input type=\"checkbox\"> 3. Are explanations of fill strategy provided for all relevant questions?  <br></h3>\n",
              "<h3><input type=\"checkbox\"> 4. Are different strategies used for continuous vs categorical data?  <br></h3>\n",
              "<h3><input type=\"checkbox\"> 5. Is the bonus question answered?  <br></h3> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vrmrmS4BavqM"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}