Trip Analysis.ipynb

{
  "cells": [
    {
      "metadata": {
        "_uuid": "394b19a07771c3e936cf3a75378442f834e3ff75",
        "_cell_guid": "a5e5cf94-d1dd-406a-837b-e50c7eaa797e"
      },
      "cell_type": "markdown",
      "source": "<h1>New York City Trip Duration</h1>\n<subtitle>A Beginner's Perspective</subtitle>\n\n<h2> Introduction</h2>\n<p>The dataset contains data about the taxi services in New York City. The data is split into two parts <b>test.csv</b> and <b>train.csv</b> the roles of which are self-explanatory. The aim is to predict the duration of a taxi trip based on certain parameters such as: <b> Pickup location, drop-off location, pickup date and time, number of passengers, vendor</b> etc. The impact of these factors on the trip duration will be assessed using machine learning models and then the model will be used to predict a value for the trip duration.</p>\n\n"
    },
    {
      "metadata": {
        "_kg_hide-output": false,
        "_uuid": "b604f659145981957f823360782249cfc723ff96",
        "_cell_guid": "86ac325e-75ca-4d10-a7de-bba04bd00593",
        "_kg_hide-input": false,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport time\nfrom sklearn.metrics import mean_squared_error as mse\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))\n\n# Any results you write to the current directory are saved as output.",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ce96dd0f7e84182ede2e116df74cbfcedb3ed29a",
        "_cell_guid": "dae07cad-6427-4511-90a4-ce9d29f03b3e",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "#Import training data\ntrain = pd.read_csv('../input/train.csv')\ntrain.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "190a9a1c990faeec43d92c390ff4735931fdee8f",
        "_cell_guid": "c91b570d-9f2f-46fe-9af1-c21d8915b046",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "train.info()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7f262a94d9cd88e3ec626e246df613924959b5a2",
        "_cell_guid": "aaaa6d71-3a45-4f71-9671-59994c209ded"
      },
      "cell_type": "markdown",
      "source": "The store_and_fwd_flag is of object type so we convert it into integer type by replacing 'N' by 0 and 'Y' by 1"
    },
    {
      "metadata": {
        "_uuid": "0850e165353a87b43b07ca91c40c9b06ec7e17f6",
        "_cell_guid": "7e48f94c-89ae-4793-adc5-d68155c739d7",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "#Convert store_and_fwd_flag to categorical binary data. Also, datatype is converted to integer\ntrains = train.copy()\ntrains.loc[trains.store_and_fwd_flag == 'N', 'store_and_fwd_flag'] = 0\ntrains.loc[trains.store_and_fwd_flag == 'Y', 'store_and_fwd_flag'] = 1\ntrains['store_and_fwd_flag'] = pd.to_numeric(trains['store_and_fwd_flag'])\ntrains.info()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "02f26549e6ece7e4e191d514d6752f004ed88df6",
        "_cell_guid": "94bdc435-4130-41ef-8cc1-5019eacd2b82"
      },
      "cell_type": "markdown",
      "source": "Convert pickup and dropoff datetimes to pandas datetime"
    },
    {
      "metadata": {
        "_uuid": "71896cfbc53cc0d80278c1af0c8016190b216fca",
        "collapsed": true,
        "_cell_guid": "c2b1ec0a-c0c8-41e1-b863-b742d8377bbc",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Convert pickup and dropoff date_time to datetime format\ntrains['pickup_datetime'] = pd.to_datetime(trains.pickup_datetime)\ntrains['dropoff_datetime'] = pd.to_datetime(trains.dropoff_datetime)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "5745b6aef2d7d6b340232c23163f064eaaa34bb4",
        "_cell_guid": "eb5f9f57-f8e0-43eb-bcdc-1de11a9368d6"
      },
      "cell_type": "markdown",
      "source": "Calculate pickup_time as hour*3600+minute*60+seconds"
    },
    {
      "metadata": {
        "_uuid": "e41af0912f8f6d7230f7dddf2c538a0c19a1d5da",
        "_cell_guid": "f2b50183-c018-475a-82f6-780319dd5e57",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "#import datetime as dt\nptime = trains.pickup_datetime.dt.hour*3600+trains.pickup_datetime.dt.minute*60+trains.pickup_datetime.dt.second\n%matplotlib inline\nsns.set(style=\"white\", palette=\"muted\", color_codes=True)\nf, axes = plt.subplots(1, 1, figsize=(11, 7), sharex=True)\nsns.despine(left=True)\nsns.distplot(ptime, axlabel = 'Pickup Time', label = 'Pickup Time', bins = 20, color=\"r\")\ntrains.loc[:, 'pickup_time'] = ptime\nplt.setp(axes, yticks=[])\nplt.tight_layout()\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "70d58f3be561e12b9d65d93340ea36a3c0fb8b8e",
        "_cell_guid": "649366ab-5553-4bd3-8860-d3920f1c31f1",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "trains.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "918b935c424a3ed433bdebb5bfee80ac811df333",
        "_cell_guid": "db472534-f771-4898-a319-3cac46e6e2ce"
      },
      "cell_type": "markdown",
      "source": "We find out that there exist some instances when the duration between the pickup and dropoff time was not equal to the actual trip_duration. Quite strange. Thankfully the difference was not very high (in the order of 10^-14) so we need not worry about it."
    },
    {
      "metadata": {
        "_uuid": "7e8b9434b7e260a6435cd70dece6db2399731940",
        "_cell_guid": "0c5fd630-bc82-4714-b2f5-a75829e5a6fb",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "trains.loc[(trains.dropoff_datetime - trains.pickup_datetime).dt.total_seconds() != trains.trip_duration]\ndiffer = (trains.dropoff_datetime - trains.pickup_datetime).dt.total_seconds() - trains.trip_duration\nsns.distplot(np.log10(differ.loc[((trains.dropoff_datetime - trains.pickup_datetime).dt.total_seconds()- trains.trip_duration)!=0]), axlabel = 'log10(Duration_Difference)')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a81c46d82e0debe5491695e0cd91a779486738cd",
        "_cell_guid": "a81950c8-7063-4fa7-a55c-22aa8b168360"
      },
      "cell_type": "markdown",
      "source": "We calculate the haversine distance between the pickup and dropoff points"
    },
    {
      "metadata": {
        "_uuid": "3a7236c3f909f28f5458e56d8e58605cfa4b80a6",
        "collapsed": true,
        "_cell_guid": "5ff725e2-7c0d-4934-b79d-e0354f43e925",
        "trusted": false
      },
      "cell_type": "code",
      "source": "def haversine_(lat1, lng1, lat2, lng2):\n    \"\"\"function to calculate haversine distance between two co-ordinates\"\"\"\n    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))\n    AVG_EARTH_RADIUS = 6371  # in km\n    lat = lat2 - lat1\n    lng = lng2 - lng1\n    d = np.sin(lat * 0.5) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin(lng * 0.5) ** 2\n    h = 2 * AVG_EARTH_RADIUS * np.arcsin(np.sqrt(d))\n    return(h)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "53fac3c0569574d2c2af8cba588beb5f25a65e9c",
        "collapsed": true,
        "_cell_guid": "59be055d-628f-4c1f-afbd-ccf19ae05896",
        "trusted": false
      },
      "cell_type": "code",
      "source": "lon1 = trains['pickup_longitude']\nlat1 = trains['pickup_latitude']\nlon2 = trains['dropoff_longitude']\nlat2 = trains['dropoff_latitude']\ntrains['haversine'] = abs(haversine_(lon1,lat1,lon2,lat2))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a54f6e6c363ab52cedafda18f2e4ae6b6e4f6cf1",
        "collapsed": true,
        "_cell_guid": "b6fc1c5a-a6f1-4d62-a7e1-2711258d8265",
        "trusted": false
      },
      "cell_type": "code",
      "source": "trains = trains.loc[trains.haversine!=0]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8bd334c1b458c4c143f387edcc6bdf5264c6b91b",
        "_cell_guid": "803a2957-2489-4c1d-b902-e6f8aefc851d",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "%matplotlib inline\nsns.set(style=\"white\", palette=\"muted\", color_codes=True)\nf, axes = plt.subplots(1, 1, figsize=(11, 7), sharex=True)\nsns.despine(left=True)\nsns.distplot(np.log10(trains.haversine), axlabel = 'log10(haversine)', label = 'Haversine', bins = 50, color=\"b\")\nplt.setp(axes, yticks=[])\nplt.tight_layout()\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "aa771db7af5f7a4df722f35086d552506bac0c1d",
        "_cell_guid": "cf6f693f-9d83-4cdb-8fef-db139b03a4bb"
      },
      "cell_type": "markdown",
      "source": "We can spot outliers in the trip_duration value. Some trips indicate trip_durations of more than 9 hours !!"
    },
    {
      "metadata": {
        "_uuid": "21a8e7bc72a0354fc6d115313dbf6c4a10dbac87",
        "_cell_guid": "9d51cfbc-358b-47cf-8fc9-8313c2410d73",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "%matplotlib inline\nsns.set(style=\"white\", palette=\"muted\", color_codes=True)\nf, axes = plt.subplots(1, 1, figsize=(11, 7), sharex=True)\nsns.despine(left=True)\nsns.distplot(np.log10(trains.trip_duration), axlabel = 'log10(Trip Duration)', label = 'Trip Duration', bins = 50, color=\"b\")\nplt.setp(axes, yticks=[])\nplt.tight_layout()\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d38e0f56aa058ce3fb9150665bd5cf322c5d4266",
        "_cell_guid": "1e27738a-7de1-4999-9030-0a003e026d8f"
      },
      "cell_type": "markdown",
      "source": "Also the distribution would have been almost gaussian but due to outliers, the mean and median are pushed far away"
    },
    {
      "metadata": {
        "_uuid": "41d8321f847758443297d8fd2a4a808bdcce9019",
        "_cell_guid": "725881a6-8023-4be0-9f75-7b2e7657a78c",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "trains.trip_duration.describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0a62f7ea52de9e433333925569d04eb9cf3a45d1",
        "_cell_guid": "673c7c0a-f000-4b53-9a13-071bf317f8bd"
      },
      "cell_type": "markdown",
      "source": "Thus we remove outliers by packing the duration between 2 minutes and 9 hours (almost). \nThis brings the mean and median much closer"
    },
    {
      "metadata": {
        "_uuid": "e4f6402741710a7cb7d15beb80329f2b37f5c517",
        "_cell_guid": "f8f0fd09-0259-4912-850c-e259a9b4838b",
        "_kg_hide-input": false,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "trains_drop1 = trains.loc[(1.7<=np.log10(trains.trip_duration)) & (np.log10(trains.trip_duration)<=4.0)]\ntrains_drop1.trip_duration.describe()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "1497ff36321b305191d97b9427ac468717db6385",
        "_cell_guid": "d314ec1d-1dd5-4a03-a983-40fcb26226ae",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "%matplotlib inline\nsns.set(style=\"white\", palette=\"muted\", color_codes=True)\nf, axes = plt.subplots(1, 1, figsize=(11, 7), sharex=True)\nsns.despine(left=True)\nsns.distplot(np.log10(trains_drop1.trip_duration), axlabel = 'log10(Trip Duration)', label = 'Trip Duration', bins = 50, color=\"b\")\nplt.setp(axes, yticks=[])\nplt.tight_layout()\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8ccc809a62b6076c9c69eadf823a95da1d4111a4",
        "_cell_guid": "a25c57ce-068f-4760-8e37-865f9d439891"
      },
      "cell_type": "markdown",
      "source": "Adding new features of month, weekofyear, dayofyear, dayofweek"
    },
    {
      "metadata": {
        "_uuid": "3b236bbf263a6053476964eaf17dd06dcc292149",
        "_cell_guid": "7deb59c8-c027-4278-9d54-20b4f3347284",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "trains_drop1.loc[:, 'month'] = trains_drop1['pickup_datetime'].dt.month\ntrains_drop1.loc[:, 'week_of_year'] = trains_drop1['pickup_datetime'].dt.weekofyear\ntrains_drop1.loc[:, 'day_of_month'] = trains_drop1['pickup_datetime'].dt.day\ntrains_drop1.loc[:, 'day_of_week'] = trains_drop1['pickup_datetime'].dt.dayofweek+1\ntrains_drop1.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "42264014a3bcbc69578f45f0586f186c0f30b307",
        "_cell_guid": "627a6c4c-9b30-4f0b-99fd-e3e017dd7197"
      },
      "cell_type": "markdown",
      "source": "In the plot below we can spot that there exist certain hotspots from where the maximum pickups and dropoffs take place"
    },
    {
      "metadata": {
        "_uuid": "c5a861c1091c5992680e65ba246aaaa806a6ef9a",
        "_cell_guid": "334a2d04-8ed2-4d49-87d6-153173f741cc",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "sns.set(style=\"white\", palette=\"muted\", color_codes=True)\nf, axes = plt.subplots(2,2,figsize=(10, 10), sharex=False, sharey = False)\nsns.despine(left=True)\nsns.distplot(trains_drop1['pickup_latitude'].values, label = 'pickup_latitude',color=\"m\",bins = 100, ax=axes[0,0])\nsns.distplot(trains_drop1['pickup_longitude'].values, label = 'pickup_longitude',color=\"m\",bins =100, ax=axes[0,1])\nsns.distplot(trains_drop1['dropoff_latitude'].values, label = 'dropoff_latitude',color=\"m\",bins =100, ax=axes[1, 0])\nsns.distplot(trains_drop1['dropoff_longitude'].values, label = 'dropoff_longitude',color=\"m\",bins =100, ax=axes[1, 1])\nplt.setp(axes, yticks=[])\nplt.tight_layout()\nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "c88421067370b22e147d7c9a72445ce9c3cec9dd",
        "_cell_guid": "2fdbc742-13f6-4c6a-a3b9-902a6922b555"
      },
      "cell_type": "markdown",
      "source": "The plot below shows the number of passengers acroos trips. Looks like a lot of passengers go on solo trips. Interestingly there are also trips with zero passengers."
    },
    {
      "metadata": {
        "_uuid": "a6c846f07276edf6ede0088abc323f45fc61b865",
        "_cell_guid": "b325ee57-89dd-4867-a70c-eda2e8c17dc8",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "objects = trains_drop1['passenger_count'].unique()\ny_pos = np.arange(len(objects))\nperformance = trains_drop1['passenger_count'].value_counts()\nplt.bar(y_pos, performance, align='center', alpha=0.5)\nplt.xticks(y_pos, objects)\nplt.ylabel('Number of trips')\nplt.title('Passenger Count') \nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "9bff1cf0116f25518a75adfd1c48c6b20a866cca",
        "_cell_guid": "45fb34eb-857a-4943-a019-f69e109f19a4"
      },
      "cell_type": "markdown",
      "source": "Thus, we will have to drop the trips having zero passengers"
    },
    {
      "metadata": {
        "_uuid": "3b72fef88d2e8b2f96e7c87f623456f85ac02a50",
        "collapsed": true,
        "_cell_guid": "ad763a62-2031-4b75-90d8-89c8190dbf25",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Dropping the trips where passenger count is zero\ntrains_drop2 = trains_drop1.loc[trains_drop1.passenger_count != 0]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "b7bed4f139e44adf3f18b406c6356f50638c65c9",
        "_cell_guid": "ba6fd94a-8cf5-4c81-ab78-74448b7d3739",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "objects = trains_drop2['passenger_count'].unique()\ny_pos = np.arange(len(objects))\nperformance = trains_drop2['passenger_count'].value_counts()\nplt.bar(y_pos, performance, align='center', alpha=0.5)\nplt.xticks(y_pos, objects)\nplt.ylabel('Number of trips')\nplt.ylabel('Passenger Count')\nplt.title('Passenger Count') \nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d7880bdbb9ad7c0898582f0ad8e950f89e385ffd",
        "_cell_guid": "83c8d304-3b2e-4cfd-a0ea-69a5600e4542"
      },
      "cell_type": "markdown",
      "source": "Vendor 2 has significantly more trips than vendor 1"
    },
    {
      "metadata": {
        "_uuid": "fd87181c5e4b251d9e0914b24f6f9e730498b67b",
        "_cell_guid": "9d740cea-d447-41e5-8dbe-50401a8d388c",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "objects = trains_drop2['vendor_id'].unique()\ny_pos = np.arange(len(objects))\nperformance = trains_drop2['vendor_id'].value_counts()\nplt.bar(y_pos, performance, align='center', alpha=0.5, color = ['b','r'])\nplt.xticks(y_pos, objects)\nplt.ylabel('Number of trips')\nplt.title('Vendor_ID') \nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3f01a77fad1f3a13159d8f62b6c7f82432f04015",
        "_cell_guid": "147ace55-8cdf-44c1-9ab3-82f6982b697b"
      },
      "cell_type": "markdown",
      "source": "Not many trips have store_and_fwd_flag as 1"
    },
    {
      "metadata": {
        "_uuid": "ac2a054ba947d7a39de4863b8a921f5e049a1646",
        "_cell_guid": "53f7695a-01fe-4ed3-b7dd-7a5e9680f3ef",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "objects = trains_drop2['store_and_fwd_flag'].unique()\ny_pos = np.arange(len(objects))\nperformance = trains_drop2['store_and_fwd_flag'].value_counts()\nplt.bar(y_pos, np.log10(performance), align='center', alpha=0.5, color = ['b','r'])\nplt.xticks(y_pos, objects)\nplt.ylabel('Number of flags')\nplt.title('store_and_fwd_flag') \nplt.show()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "_uuid": "fa6df06d87e10093c654d8e47578b1b1feb798d3",
        "_cell_guid": "2a14288a-2e77-4565-8dac-96698b0b2615",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "trains_drop2.columns",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "255580d03cd8703595497a81fa9285d8c5c9fd6b",
        "_cell_guid": "53c4b722-4d15-47b9-9e8c-c4f882993bfe"
      },
      "cell_type": "markdown",
      "source": "Lets get down to work. We choose the features we consider are important for predicting the trip duration"
    },
    {
      "metadata": {
        "_uuid": "f85173ca179dbc10875ff43a035be02adba1585d",
        "_cell_guid": "458dc4b4-42a0-42dc-8c98-c360896647b2",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "important_features = ['vendor_id', 'pickup_longitude', 'pickup_latitude',\n                      'dropoff_longitude', 'dropoff_latitude','pickup_time','haversine',\n                      'month','week_of_year','day_of_month','day_of_week']\nX = trains_drop2[important_features]\nX.info()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "14b78dd28589626bdeaf72550f443628e207097a",
        "_cell_guid": "7d88992f-c885-43ce-8b58-23dd1bb2f05f"
      },
      "cell_type": "markdown",
      "source": "Spli the data into train and test data"
    },
    {
      "metadata": {
        "_uuid": "f8b8f56a47b8e2baa1e6e4b4704ed55ae62c566e",
        "collapsed": true,
        "_cell_guid": "e0a6eda4-e08e-479c-9692-11153e998ca7",
        "trusted": false
      },
      "cell_type": "code",
      "source": "y = trains_drop2['trip_duration']\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=60)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7f45cd23ff53b18a2dbe579e1679afa5eed65519",
        "_cell_guid": "296affb8-4ac3-4275-b76f-190f9084d9f6"
      },
      "cell_type": "markdown",
      "source": "Train the regressor"
    },
    {
      "metadata": {
        "_uuid": "acc520851c8178eb7b5c957395a80dae13943975",
        "_cell_guid": "2bbdec85-730a-4fc0-8db9-d248aadac43f",
        "_kg_hide-input": false,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "start=time.time()\nfrom xgboost import XGBRegressor\nxgb_pr = XGBRegressor()\nxgb_pr.fit(X_train,y_train)\nend=time.time()\nprint(\" completed in \",end-start,\" seconds\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0c207c4752ea6a5508abd023f7e87ff618d99003",
        "_cell_guid": "5705ed20-851a-42cc-b7ff-dc68b3c1b465"
      },
      "cell_type": "markdown",
      "source": "Predict the output"
    },
    {
      "metadata": {
        "_uuid": "edc78690f52d85bc429983cbf84719f80da3e010",
        "_cell_guid": "2758e616-913e-4ee5-90bb-bb7fd28cb978",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "start=time.time()\ny_predtrain = xgb_pr.predict(X_test)\nend=time.time()\nprint(\" completed in \",end-start,\" seconds\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "_uuid": "b06506db49cbee1bc76f1e947eafc15b8249fd9d",
        "_cell_guid": "59b68871-ac07-477b-aca6-dda2a91861cc",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "y_predtrain",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "_uuid": "87286f9df547be2cda888308e63ec899a6a1b3ec",
        "_cell_guid": "fb489d4d-8e68-49ef-9d85-4cec4faab96a",
        "_kg_hide-input": false,
        "collapsed": true,
        "trusted": false
      },
      "cell_type": "code",
      "source": "y_tests = np.array(y_test)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "_uuid": "3718958f263d09243b1c5dd15a7947809a74fc4e",
        "_cell_guid": "267262c2-d800-4988-9917-d5884e6246a6",
        "_kg_hide-input": true,
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "len(y_predtrain)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ee06b70c3ed2e79ddad6a103f54152ef24bdae4f",
        "_cell_guid": "8313af5e-71fa-4c4f-a8ee-1e6401363f7d"
      },
      "cell_type": "markdown",
      "source": "The absolute difference between the actual value and the predicted value"
    },
    {
      "metadata": {
        "_uuid": "0cb8bc1f8a57e9f8f25ad7f29b2d16d9d1a206aa",
        "_cell_guid": "4de52bb9-d030-4fdd-8d74-26bd63823be6",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "from scipy import stats\ndiffer = abs(y_tests - y_predtrain.astype(int))\nstats.describe(differ)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8426b9aca516347aaa02a19647e8529efc3b67d0",
        "_cell_guid": "6fb859d6-3887-4a04-bccc-899a3da12730"
      },
      "cell_type": "markdown",
      "source": "Calculate the mean square error"
    },
    {
      "metadata": {
        "_uuid": "20e5de201f081311e817ae6d8c6dc21a52ecd7f1",
        "_cell_guid": "d7ad3df6-15e3-4756-a075-98b18c2c3390",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "mse(y_tests,y_predtrain.astype(int))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_kg_hide-output": true,
        "_uuid": "ca817d8b9100b1538b5dbf8efb896554263e4741",
        "_cell_guid": "8dff6b68-0f7d-4a66-94d3-0eeb72fd09a2",
        "_kg_hide-input": true,
        "collapsed": true
      },
      "cell_type": "markdown",
      "source": "Replace all negative values with zero"
    },
    {
      "metadata": {
        "_uuid": "1dff91da5bea51dd66c9165600973b722f4d23ca",
        "collapsed": true,
        "_cell_guid": "070ffc83-67e7-4988-9f47-78b5425ba337",
        "trusted": false
      },
      "cell_type": "code",
      "source": "y_predtrain2 = y_predtrain.astype(int).copy()\nfor index,row in enumerate(y_predtrain2):\n    if(row<0):\n        y_predtrain2[index]=0\nfor index,row in enumerate(y_predtrain2):\n    if(row<0):\n        print(row)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "2dcb69eb25ef59a3b19446228d0d80f6d5d0c081",
        "collapsed": true,
        "_cell_guid": "8ecbedb2-c043-442f-b975-bdd821713362"
      },
      "cell_type": "markdown",
      "source": "Calculate the Mean Square Log Error for our result (training data)"
    },
    {
      "metadata": {
        "_uuid": "c5fc3557279ae05150a531cd60d5bab1cff6168e",
        "_cell_guid": "a28d169d-6265-4fc2-a85f-7eb4cbd30a9f",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "from sklearn.metrics import mean_squared_log_error as msle\nmsle(y_tests,y_predtrain2)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4e4a097aed77a20752ed7b74870b74fdb490e910",
        "_cell_guid": "c88e4846-c01b-469f-9a2b-6fff6c46fa51",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "test = pd.read_csv('../input/test.csv')\ntests = test.copy()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "a75f1ae8ac9908feb400f757091d0b4fc62b7367",
        "_cell_guid": "4a86abd6-93aa-4425-a25c-9ffe47bf3e7d"
      },
      "cell_type": "markdown",
      "source": "Prepare test data for prediction"
    },
    {
      "metadata": {
        "_uuid": "cc17a6b444c53a93cdffbdfd8b58f54cc739dc96",
        "_cell_guid": "9e0e2ad2-974c-4b08-bfe6-306f1b38ce19",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "tests['pickup_datetime'] = pd.to_datetime(tests.pickup_datetime)\nptime = tests.pickup_datetime.dt.hour*3600+tests.pickup_datetime.dt.minute*60+tests.pickup_datetime.dt.second\ntests.loc[:, 'pickup_time'] = ptime\ntests.loc[tests.store_and_fwd_flag == 'N', 'store_and_fwd_flag'] = 0\ntests.loc[tests.store_and_fwd_flag == 'Y', 'store_and_fwd_flag'] = 1\ntests['store_and_fwd_flag'] = pd.to_numeric(tests['store_and_fwd_flag'])\nlon1 = trains['pickup_longitude']\nlat1 = trains['pickup_latitude']\nlon2 = trains['dropoff_longitude']\nlat2 = trains['dropoff_latitude']\ntests['haversine'] = abs(haversine_(lon1,lat1,lon2,lat2))\ntests.loc[:, 'month'] = tests['pickup_datetime'].dt.month\ntests.loc[:, 'week_of_year'] = tests['pickup_datetime'].dt.weekofyear\ntests.loc[:, 'day_of_month'] = tests['pickup_datetime'].dt.day\ntests.loc[:, 'day_of_week'] = tests['pickup_datetime'].dt.dayofweek+1\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d3faee54d8a1c4f95cdb654f37360ac99a992bfe",
        "collapsed": true,
        "_cell_guid": "f4b6dda3-3869-4786-8aa5-dd2c064ce41a"
      },
      "cell_type": "markdown",
      "source": "Select important features"
    },
    {
      "metadata": {
        "_uuid": "5072c6953da7bdce12887accc94116b3770a0d5f",
        "_cell_guid": "b85c03da-ff86-41b4-b005-1e52d07922cc",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "X_tests = tests[important_features]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "d3ae77a943eda8c5bc91846c8dae109be34a1a9f",
        "_cell_guid": "fe7f93bf-376b-40c9-8558-b7f8aa9b8ca3"
      },
      "cell_type": "markdown",
      "source": "Predict"
    },
    {
      "metadata": {
        "_uuid": "21711f475c046aef384242fcea136273ccb5162d",
        "_cell_guid": "1a670063-5549-40e6-b096-f298562208ba",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "start=time.time()\ny_tpred = xgb_pr.predict(X_tests)\nend=time.time()\nprint(\" completed in \",end-start,\" seconds\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "7abdd1a8c229f3aea241b9296f3ddab516979430",
        "_cell_guid": "a19d5747-913f-405e-b852-a5146e75d7bc",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "y_tpred",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "fc0c36afde56f7832390e8839d1538b0a1855afe",
        "_cell_guid": "0c629c90-789f-403d-8632-9dc6df93572a"
      },
      "cell_type": "markdown",
      "source": "Remove negative values"
    },
    {
      "metadata": {
        "_uuid": "d132f5c5d421c51bed383bffdd0ddf50bd5ef237",
        "_cell_guid": "20f77e81-c475-4cfb-bb4b-d0f3fdef20ce",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "for index,row in enumerate(y_tpred):\n    if(row<0):\n        print(row)\nfor index,row in enumerate(y_tpred):\n    if(row<0):\n        y_tpred[index]=0",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ddd2e814e1f823547f7fe961f9cc0d780a746a26",
        "collapsed": true,
        "_cell_guid": "a762b8e7-a116-450f-bdfc-a0677de4c5db"
      },
      "cell_type": "markdown",
      "source": "Write to output file"
    },
    {
      "metadata": {
        "_uuid": "74b29919e7ce956feb6f011cd7805fab3526f503",
        "collapsed": true,
        "_cell_guid": "ce861110-1075-43d4-a588-9b915cdd4547",
        "trusted": false
      },
      "cell_type": "code",
      "source": "df_benchmark = pd.DataFrame()\ndf_benchmark['id'] = tests['id']\ndf_benchmark['trip_duration'] = y_tpred",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "f62e42776463ff78ca61a853e7b521f2e5b0e0cb",
        "_cell_guid": "54e51eed-67ab-4223-a33b-c19f94eb5753",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "len(y_tpred)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "49f30fa5acb05ae1b66e9b6d8bbc9cafc50e2c3a",
        "_cell_guid": "78abbc6d-7c3b-40a3-aa20-6e1e16ecf8f3",
        "scrolled": true,
        "collapsed": true,
        "trusted": false
      },
      "cell_type": "code",
      "source": "df_benchmark.to_csv(\"sample_submission5.csv\",index=False)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3532299db75d3268930c88e6f5ceba36905ded61",
        "collapsed": true,
        "_cell_guid": "0276f805-0d60-4b73-a832-f06d38c84d22",
        "trusted": false
      },
      "cell_type": "code",
      "source": "",
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}