filter_swiss_helpful_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following script extracts the (more) helpful reviews from the swiss reviews and saves them locally.\n",
    "From the extracted reviews it also saves a list with their asin identifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The list of asin identifiers will be later used to to find the average review rating for the respective products."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the swiss reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/swiss-reviews.txt\", 'r') as fp:\n",
    "    swiss_rev = fp.readlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "304739"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(swiss_rev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"reviewerID\": \"A0009686KROLKEH2EHF4\", \"asin\": \"B000I7GST2\", \"reviewerName\": \"shahrizal abdullah\", \"helpful\": [1, 1], \"reviewText\": \"i like it so much, i hope Kuhn Rikon is available in my place in the future keep it up the good quality and company image thank you..\", \"overall\": 5.0, \"summary\": \"Very stylish and practical to use\", \"unixReviewTime\": 1372723200, \"reviewTime\": \"07 2, 2013\"}\\n'"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "swiss_rev[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The filter_helpful function keeps only the reviews which had at least 5 flags/votes in the helpfulness field.\n",
    "This amounts to a subset of around 23000 reviews. A smaller subset of around 10000 reviews was obtained as well by only keeping reviews with 10 flags/votes. The main advantage of the smaller subset is that it contains better quality reviews while its drawback is, of course, the reduced size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Extract the helpful reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def filter_helpful(line):\n",
    "    l = line.rstrip('\\n')\n",
    "    l = yaml.load(l)\n",
    "    if('helpful' in l.keys()):\n",
    "        if(l['helpful'][1] >= 5):\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "    else:\n",
    "        print(\"Review does not have helpful score key: \"+line)\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the filter_helpful to each swiss product review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_helpful(data):\n",
    "    res = []\n",
    "    counter = 1\n",
    "    i = 0\n",
    "    for line in data:\n",
    "        i += 1\n",
    "        if(filter_helpful(line)):\n",
    "            if(counter % 1000 == 0):\n",
    "                print(\"Count \"+str(counter)+\" / \"+str(i))\n",
    "            counter += 1\n",
    "            res.append(line)\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Count 1000 / 13319\n",
      "Count 2000 / 25440\n",
      "Count 3000 / 38733\n",
      "Count 4000 / 50934\n",
      "Count 5000 / 63854\n",
      "Count 6000 / 77771\n",
      "Count 7000 / 90390\n",
      "Count 8000 / 103006\n",
      "Count 9000 / 116094\n",
      "Count 10000 / 129210\n",
      "Count 11000 / 141829\n",
      "Count 12000 / 154550\n",
      "Count 13000 / 167209\n",
      "Count 14000 / 179988\n",
      "Count 15000 / 192203\n",
      "Count 16000 / 204764\n",
      "Count 17000 / 218146\n",
      "Count 18000 / 231084\n",
      "Count 19000 / 243987\n",
      "Count 20000 / 256821\n",
      "Count 21000 / 268986\n",
      "Count 22000 / 281442\n",
      "Count 23000 / 295562\n"
     ]
    }
   ],
   "source": [
    "swiss_reviews_helpful = get_helpful(swiss_rev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23755"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(swiss_reviews_helpful)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the subset with helpful swiss product reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "write_file = open('data/swiss-reviews-helpful-correct-bigger.txt', 'w')\n",
    "for item in swiss_reviews_helpful:\n",
    "  write_file.write(item)\n",
    "write_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Extract the asins of the products which the helpful reviews correspond to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('data/swiss-reviews-helpful-correct-bigger.txt', 'r') as fp:\n",
    "    swiss_reviews_helpful = fp.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function simply extracts the 'asin' from the helpful reviews.\n",
    "Repetitions of the asins are of no consequence, as the list is just meant to be a check up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def filter_asin(line):\n",
    "    l = line.rstrip('\\n')\n",
    "    l = yaml.load(l)\n",
    "    if('asin' in l.keys()):\n",
    "        return l['asin']\n",
    "    else:\n",
    "        return ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "500\n",
      "1000\n",
      "1500\n",
      "2000\n",
      "2500\n",
      "3000\n",
      "3500\n",
      "4000\n",
      "4500\n",
      "5000\n",
      "5500\n",
      "6000\n",
      "6500\n",
      "7000\n",
      "7500\n",
      "8000\n",
      "8500\n",
      "9000\n",
      "9500\n",
      "10000\n",
      "10500\n",
      "11000\n",
      "11500\n",
      "12000\n",
      "12500\n",
      "13000\n",
      "13500\n",
      "14000\n",
      "14500\n",
      "15000\n",
      "15500\n",
      "16000\n",
      "16500\n",
      "17000\n",
      "17500\n",
      "18000\n",
      "18500\n",
      "19000\n",
      "19500\n",
      "20000\n",
      "20500\n",
      "21000\n",
      "21500\n",
      "22000\n",
      "22500\n",
      "23000\n",
      "23500\n"
     ]
    }
   ],
   "source": [
    "helpful_asins = []\n",
    "counter = 1\n",
    "for item in swiss_reviews_helpful:\n",
    "    if(counter%500 == 0):\n",
    "        print(counter)\n",
    "    counter += 1\n",
    "    x = filter_asin(item)\n",
    "    if(len(x) > 0):\n",
    "        helpful_asins.append(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the list of asins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open('data/helpful_asins_bigger.pickle', 'wb') as fp:\n",
    "    pickle.dump(helpful_asins, fp)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}