Notebook_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ecommerce Churn Assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aim of the assignment is to build a model that predicts whether a person purchases an item after it has been added to the cart or not. Being a classification problem, you are expected to use your understanding of all the three models covered till now. You must select the most robust model and provide a solution that predicts the churn in the most suitable manner. \n",
    "\n",
    "For this assignment, you are provided the data associated with an e-commerce company for the month of October 2019. Your task is to first analyse the data, and then perform multiple steps towards the model building process.\n",
    "\n",
    "The broad tasks are:\n",
    "- Data Exploration\n",
    "- Feature Engineering\n",
    "- Model Selection\n",
    "- Model Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset stores the information of a customer session on the e-commerce platform. It records the activity and the associated parameters with it.\n",
    "\n",
    "- **event_time**: Date and time when user accesses the platform\n",
    "- **event_type**: Action performed by the customer\n",
    "            - View\n",
    "            - Cart\n",
    "            - Purchase\n",
    "            - Remove from cart\n",
    "- **product_id**: Unique number to identify the product in the event\n",
    "- **category_id**: Unique number to identify the category of the product\n",
    "- **category_code**: Stores primary and secondary categories of the product\n",
    "- **brand**: Brand associated with the product\n",
    "- **price**: Price of the product\n",
    "- **user_id**: Unique ID for a customer\n",
    "- **user_session**: Session ID for a user\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialising the SparkSession"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset provided is 5 GBs in size. Therefore, it is expected that you increase the driver memory to a greater number. The m4.xlarge instance holds 16 GB memory and we are allocating 14 GB to the driver."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "34ea99f058b1491399365bc2b6f82d3c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Spark application\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>Current session?</th></tr><tr><td>2</td><td>application_1595617311353_0003</td><td>pyspark</td><td>idle</td><td><a target=\"_blank\" href=\"http://ip-172-31-42-82.ap-southeast-1.compute.internal:20888/proxy/application_1595617311353_0003/\">Link</a></td><td><a target=\"_blank\" href=\"http://ip-172-31-34-31.ap-southeast-1.compute.internal:8042/node/containerlogs/container_1595617311353_0003_01_000001/livy\">Link</a></td><td>✔</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparkSession available as 'spark'.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# initialising the session with 14 GB driver memory\n",
    "MAX_MEMORY = \"14g\"\n",
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.appName('ECom').config(\"spark.driver.memory\", MAX_MEMORY).getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e8199bc74e9642f083c2da6ea9d68f92",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# installing required libraries\n",
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml.classification import LogisticRegression\n",
    "from pyspark.mllib.evaluation import BinaryClassificationMetrics\n",
    "from pyspark.mllib.evaluation import MulticlassMetrics\n",
    "from pyspark.ml.tuning import ParamGridBuilder, CrossValidator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3639238f818a45908cc11e4a0f06784c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# loading the dataset from the parquet file\n",
    "df = spark.read.parquet(\"s3://asr-aiml-2020/processed.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "966c0cd8c8c44136b69850dde423aaa1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = df.withColumnRenamed(\"is_purchased\",\"label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bef1ceb2a0c745dbb76aa37a590bae53",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyspark.sql.types import DoubleType\n",
    "\n",
    "df = df.withColumn(\"label\", df[\"label\"].cast(DoubleType()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if the dataframe is correctly loaded\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89673097035f4999b794ca9bffe5f66a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- brand: string (nullable = true)\n",
      " |-- price: double (nullable = true)\n",
      " |-- Category: string (nullable = true)\n",
      " |-- SubCategory: string (nullable = true)\n",
      " |-- ActivityCount: integer (nullable = true)\n",
      " |-- ProductView: integer (nullable = true)\n",
      " |-- SubCategoryViewCount: integer (nullable = true)\n",
      " |-- AvgSpendingOnCategoty: double (nullable = true)\n",
      " |-- EventHour: integer (nullable = true)\n",
      " |-- label: double (nullable = true)\n",
      " |-- EventHour_Buckets: integer (nullable = true)\n",
      " |-- brand_idx: double (nullable = true)\n",
      " |-- Category_idx: double (nullable = true)\n",
      " |-- SubCategory_idx: double (nullable = true)\n",
      " |-- brand_Enc: vector (nullable = true)\n",
      " |-- Category_enc: vector (nullable = true)\n",
      " |-- SubCategory_enc: vector (nullable = true)\n",
      " |-- features: vector (nullable = true)"
     ]
    }
   ],
   "source": [
    "# exploring the dataframe - schema\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6a6254265ac43cea809ca34b6139504",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784361, 18)"
     ]
    }
   ],
   "source": [
    "# Number of rows and columns in the dataset\n",
    "print((df.count(), len(df.columns)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3: Model Selection\n",
    "3 models for classification:\t\n",
    "- Logistic Regression\n",
    "- Decision Tree\n",
    "- Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model 1: Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3aecd91c47f3472c80f37e98d9ebf8cf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Splitting the data into train and test (Remember you are expected to compare the model later)\n",
    "train, test = df.randomSplit([0.7,0.3], seed= 5043)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79628bba42804fbe9e80a48fa68ed27b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "549630 234731"
     ]
    }
   ],
   "source": [
    "# Number of rows in train and test data\n",
    "print(train.count(), test.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "651b005b30da4f10b7f79aac7873da2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Building the model\n",
    "lr = LogisticRegression(featuresCol='features', labelCol='label')\n",
    "lrevaluator = BinaryClassificationEvaluator(rawPredictionCol=\"rawPrediction\")\n",
    "\n",
    "paramGrid = ParamGridBuilder()\\\n",
    "    .addGrid(lr.threshold, [0.2, 0.4, 0.6, 0.8 ]) \\\n",
    "    .addGrid(lr.regParam, [0.001, 0.01, 0.1, 1.0, 2.0]) \\\n",
    "    .addGrid(lr.maxIter, [1, 5, 10]) \\\n",
    "    .build()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68a004b93d854439a11d1b72b3453df9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-10:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n",
      "Exception in thread cell_monitor-11:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n",
      "Exception in thread cell_monitor-9:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Fitting the model on transformed df\n",
    "cv = CrossValidator(estimator=lr, estimatorParamMaps=paramGrid, evaluator=lrevaluator, numFolds=4)\n",
    "# Run cross validations\n",
    "cvModel = cv.fit(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef6bed64a63e4ea099ac90fbf5ad1c01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-12:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "bestModel = cvModel.bestModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Required Steps:\n",
    "- Fit on test data\n",
    "- Performance analysis\n",
    "    - Appropriate Metric with reasoning\n",
    "    - Threshold selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef58152589154a76abeb0763a59f503c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-13:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lrpredictions=bestModel.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "103a6f7235b74ce5ba1d6fe1a5ef0ab2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-14:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.5788662890456998"
     ]
    }
   ],
   "source": [
    "print('Accuracy:', lrevaluator.evaluate(lrpredictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cd33229981dc4880b4ca49fd89cb0d8f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-15:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC: 0.6592417023777871"
     ]
    }
   ],
   "source": [
    "print('AUC:', BinaryClassificationMetrics(lrpredictions['label','prediction'].rdd).areaUnderROC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Updated model, if any\n",
    "Repeat the steos "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "479f06291d1349b39c43b7d1828a0607",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-16:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.mllib.evaluation import MulticlassMetrics\n",
    "def confusion_matrix(pred_df):\n",
    "    rdd = pred_df.select(['prediction', 'label']).rdd.map(tuple)\n",
    "    metrics = MulticlassMetrics(rdd)\n",
    "    return metrics.confusionMatrix().toArray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4ea443f59a9c48d2a27ecc54272d953d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-17:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.12926e+05 4.70000e+01]\n",
      " [1.21517e+05 2.41000e+02]]"
     ]
    }
   ],
   "source": [
    "print(confusion_matrix(lrpredictions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "64c25deace7e433197d69235cd93fb71",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-18:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Using cached https://files.pythonhosted.org/packages/af/f3/683bf2547a3eaeec15b39cef86f61e921b3b187f250fcd2b5c5fb4386369/pandas-1.0.5-cp37-cp37m-manylinux1_x86_64.whl\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/site-packages (from pandas)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib64/python3.7/site-packages (from pandas)\n",
      "Collecting python-dateutil>=2.6.1 (from pandas)\n",
      "  Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas)\n",
      "Installing collected packages: python-dateutil, pandas\n",
      "Successfully installed pandas-1.0.5 python-dateutil-2.8.1"
     ]
    }
   ],
   "source": [
    "sc.install_pypi_package(\"pandas\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc015b4017af45598eaf0c7f38fc0a29",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-19:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "modelcoefficients=bestModel.coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "289320054bb145a89ee609f4635c9a13",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-20:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "def ExtractFeatureImp(modelcoefficients, dataset, featuresCol):\n",
    "    list_extract = []\n",
    "    for i in dataset.schema[featuresCol].metadata[\"ml_attr\"][\"attrs\"]:\n",
    "        list_extract = list_extract + dataset.schema[featuresCol].metadata[\"ml_attr\"][\"attrs\"][i]\n",
    "    varlist = pd.DataFrame(list_extract)\n",
    "    varlist['coefficient'] = varlist['idx'].apply(lambda x: modelcoefficients[x])\n",
    "    return(varlist.sort_values('coefficient', ascending = False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef9e546f1d76489cad48594986856a72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    idx                           name  coefficient\n",
      "74   68           SubCategory_enc_sock     4.092197\n",
      "72   66       SubCategory_enc_bathroom     1.295500\n",
      "67   61          SubCategory_enc_swing     0.274485\n",
      "62   56  SubCategory_enc_ironing_board     0.263187\n",
      "24   18                brand_Enc_vitek     0.240822\n",
      "70   64     SubCategory_enc_lawn_mower     0.196477\n",
      "66   60        SubCategory_enc_trainer     0.187937\n",
      "37   31      Category_enc_country_yard     0.160358\n",
      "38   32     SubCategory_enc_smartphone     0.131685\n",
      "7     1                brand_Enc_apple     0.118908"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread cell_monitor-21:\n",
      "Traceback (most recent call last):\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 926, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"/opt/conda/lib/python3.7/threading.py\", line 870, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in cell_monitor\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "  File \"/opt/conda/lib/python3.7/site-packages/awseditorssparkmonitoringwidget-1.0-py3.7.egg/awseditorssparkmonitoringwidget/cellmonitor.py\", line 154, in <listcomp>\n",
      "    job_group_filtered_jobs = [job for job in jobs_data if job['jobGroup'] == str(statement_id)]\n",
      "KeyError: 'jobGroup'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ExtractFeatureImp(modelcoefficients, df, \"features\").head(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PySpark",
   "language": "",
   "name": "pysparkkernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "python",
    "version": 2
   },
   "mimetype": "text/x-python",
   "name": "pyspark",
   "pygments_lexer": "python2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}