Day 5.4 Case Study Sentiment Analysis on IMDB Movie Reviews Using a LSTM Neural Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "48f510f7",
   "metadata": {},
   "source": [
    "## The goal of this case study is to develop a sentiment analysis model that classifies movie reviews as positive or negative using a Long Short-Term Memory (LSTM) neural network. Sentiment analysis is a common Natural Language Processing (NLP) task that helps in understanding the emotional tone behind a series of words.\n",
    "\n",
    "### Dataset:\n",
    "The IMDB movie reviews dataset is a popular dataset for binary sentiment classification. It contains 50,000 movie reviews from the IMDB website, divided into 25,000 reviews for training and 25,000 reviews for testing. Each review is labeled as either positive or negative.\n",
    "\n",
    "### Load and Preprocess the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "faa78f1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load and Preprocess the Data\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing import sequence\n",
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding, LSTM, Dense\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "# Parameters\n",
    "max_features = 10000  # Number of words to consider as features\n",
    "maxlen = 500  # Cut texts after this number of words\n",
    "\n",
    "# Load the IMDB dataset\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "# Pad sequences to ensure uniform input size\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206124b2",
   "metadata": {},
   "source": [
    "## Visualize the Data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6504b5c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAiU0lEQVR4nO3debhcRZ3/8fcHwg5hS2BCFsISlIAYJWQYYWTTYREIIkgYlYhokEEEAZUIKgzyk0VgzPwGNAqyKSHsEcO+O7IlQAgBI2GPwSSsCYhAyHf+qGpzcunu2zfn9r1p7+f1PP306Tqn6lTdm9xvV9U5dRQRmJmZLasVursCZmbW2hxIzMysFAcSMzMrxYHEzMxKcSAxM7NSHEjMzKwUBxIrTdLPJH2/k8oaJOlNSSvmz3dJ+mpnlJ3Lu1HS6M4qrwPn/ZGklyX9pYvP+6akTbvynN2hs/+dWMc4kFhdkp6T9LakhZJel/QHSV+X9Pd/OxHx9Yg4tcGyPlXvmIh4ISLWjIj3O6HuJ0u6rE35e0bExWXL7mA9BgLHAUMj4p+q7N9Z0uL8R3+hpJmSDu2Mc+ef5TOdUVY11X7GzdYd57T6HEisEftExFrAxsDpwHeBCzr7JJJ6dXaZy4mNgVciYl6dY+ZExJpAb+BbwC8kfahLamdWkgOJNSwi3oiIScBBwGhJWwNIukjSj/J2H0k35N7Lq5LulbSCpEuBQcBv8zfv70gaLCkkHSbpBeCOQloxqGwm6UFJb0i6XtJ6+Vw7S5pdrGOl1yNpD+B7wEH5fNPy/r8PgeR6nSTpeUnzJF0iae28r1KP0ZJeyMNSJ9b62UhaO+efn8s7KZf/KeBWYKNcj4va+RlHREwGXgW2KdTzBElPS3pF0sTCz+AmSd9oU5dpkvbP2yFp87y9iqSf5PbMzUOSq+V9d0v6XN7eMefbK3/+lKRH69W7xs9k+9yDfT3XaefCvrsknSrpf3Mv7BZJfQr7D8k/x1ckfb+932u2cbXyJK0q6bJc1uuSHpK0YUfbY7U5kFiHRcSDwGzgX6vsPi7v6wtsSPpPHxHxJeAFUu9mzYg4s5BnJ2BLYPcapzwE+AqwEbAIGNdAHW8C/h9wRT7fR6sc9uX82gXYFFgT+P9tjtkR+BCwG/ADSVvWOOV/A2vncnbKdT40Im4D9iT3OCLiy/XqnYPGvkAfYFZO/iawXy53I+A14H/yvt8ABxfyDyX1gH5XpfgzgC2AYcDmQH/gB3nf3cDOefuTwDP5fJXPd9erd5V29M91+BGwHnA8cLWkvoXD/h04FNgAWDkfU2nDecAXgH6kn2t/aPf3WrU8YHQuYyCwPvB14O2OtMfqcyCxZTWH9AeirfdI//k3joj3IuLeaH9Bt5Mj4q2IqPWf+9KIeDwi3gK+D3xeeTK+pC8A50TEMxHxJjAWGNWmN3RKRLwdEdOAacAHAlKuy0HA2IhYGBHPAWcDX+pAXTaS9DrpD9y1wLER8UjedzhwYkTMjoh3gJOBA3I9rwWGSdq40KZr8nHFOgr4GvCtiHg1IhaS/iCPyofczdKB48eFzzvRwUACfBGYHBGTI2JxRNwKTAH2Khzzq4j4U/69TyQFOIADgN9GxO8j4l1SsGtkUcBa5b1HCiCbR8T7ETE1IhZ0sD1WhwOJLav+pOGXts4ifZO+RdIzkk5ooKwXO7D/eWAl0jf2sjbK5RXL7kXqSVUUr7L6K6nX0lYf0jfgtmX170Bd5kTEOqQ5knHAroV9GwPX5mGZ14EngfeBDXNA+B1LAsIo4NdVyu8LrA5MLZRzU04HuA/YIg/5DAMuAQbm4aERwD0daEulzgdWzpXPtyPpS0ZFrZ/tRhR+5xHxV+CVBs5Zq7xLgZuBCZLmSDpT0kodaYzV50BiHSZpO9Ifyd+33Ze/kR8XEZsC+wDHStqtsrtGke192xxY2B5E+ob5MvAW6Y9jpV4rsuQPYyPlziH9wSuWvQiY206+tl7OdWpb1p87WA65J/Fd4COS9svJLwJ7RsQ6hdeqEVEp/3LgYEn/AqwG3Fmjjm8DWxXKWDtP8Ff+WE8FjgYezz2BPwDHAk9HxMsdbMqLpJ5ksc5rRMTpDeR9CRhQ+ZDncdYv7O/QkuW5Z3xKRAwFPgHsTRp6tE7iQGINk9Rb0t7ABOCyiJhe5Zi9JW2eh1IWkL45Vy7lnUuaQ+ioL0oaKml14D+Bq/LlwX8CVpX0mfwN8yRglUK+ucBgFS5VbuNy4FuSNpG0JkvG3hd1pHK5LhOB0yStlYeZjgWW6RLV/Ef8bJbMX/wsl70xgKS+kkYWskwmBbH/zPVfXKXMxcAvgHMlbZDL6S+pOC91N/ANlgxj3dXmcy0r5AntymsVUtv3kbS7pBVz+s6SBrRTFsBVOe8nJK0MnAKosL+93+tSJO0i6SP5i8YCUtAvfXm5LeFAYo34raSFpG+ZJwLnkCY1qxkC3Aa8SRouOS8i7sr7fgyclIc6jq+Rv5pLgYtIQxerkiafiYg3gP8Afkn69v8WaaK/4sr8/oqkh6uUe2Eu+x7gWeBvwFEdqFfRUfn8z5B6ar/J5S+rC4FBkvYBfgpMIg0XLgTuB/65cmDuxVwDfCqft5bvkoYd75e0gPR7Kl5ifDewFkuGsdp+ruVgUm+n8no6Il4ERpIutphP+rfzbRr4mxMRM0g/zwmk3slCYB5Qmfdp7/fa1j+RgtMC0rDg3SxjkLfq5AdbmdnyLPcWXweGRMSz3Vwdq8I9EjNb7kjaR9LqktYAfgJMB57r3lpZLQ4kZrY8Gkm6GGIOabh0VAOXkVs38dCWmZmV4h6JmZmV8o+6SF5Nffr0icGDB3d3NczMWsrUqVNfjoi+1fb1uEAyePBgpkyZ0t3VMDNrKZKer7XPQ1tmZlaKA4mZmZXiQGJmZqU4kJiZWSkOJGZmVooDiZmZldK0QCJpoKQ7JT0paYako3P6yZL+LOnR/NqrkGespFmSZhaXt5a0raTped+4vER55RnUV+T0ByQNblZ7zMysumb2SBYBx0XElsD2wJH5WcwA50bEsPyaDH9/TvMoYCtgD+C8wuNUzwfGkNbcGZL3AxwGvBYRmwPnkp5JbWZmXahpgSQiXoqIh/P2QtJzAOo9enQkMCEi3slLRc8CRkjqB/SOiPvyom2XAPsV8lyct68Cdqv0VszMrGt0yZ3tecjpY8ADwA7ANyQdAkwh9VpeIwWZ+wvZZue091j6YUWVdPL7iwARsUjSG6RHci71WFBJY0g9GgYNGtSZTTPrNINP+F3Nfc+d/pkurIlZxzR9sj0/lOZq4JiIWEAaptoMGEZ6+tnZlUOrZI866fXyLJ0QMT4ihkfE8L59qy4VY2Zmy6ipgSQ/R/tq4NcRcQ1ARMyNiPcLz5AekQ+fDQwsZB9AehbB7LzdNn2pPJJ6AWsDrzanNWZmVk0zr9oScAHwZEScU0jvVzjss8DjeXsSMCpfibUJaVL9wYh4CVgoaftc5iHA9YU8o/P2AcAdfviNmVnXauYcyQ7Al4Dpkh7Nad8DDpY0jDQE9RxwOEBEzJA0EXiCdMXXkRHxfs53BHARsBpwY35BClSXSppF6omMamJ7zMysiqYFkoj4PdXnMCbXyXMacFqV9CnA1lXS/wYcWKKaZmZWku9sNzOzUhxIzMysFAcSMzMrxYHEzMxKcSAxM7NSHEjMzKwUBxIzMyvFgcTMzEpxIDEzs1IcSMzMrBQHEjMzK8WBxMzMSnEgMTOzUhxIzMysFAcSMzMrxYHEzMxKcSAxM7NSHEjMzKwUBxIzMyvFgcTMzEpxIDEzs1IcSMzMrBQHEjMzK8WBxMzMSnEgMTOzUhxIzMysFAcSMzMrxYHEzMxKcSAxM7NSHEjMzKwUBxIzMyvFgcTMzEppWiCRNFDSnZKelDRD0tE5fT1Jt0p6Kr+vW8gzVtIsSTMl7V5I31bS9LxvnCTl9FUkXZHTH5A0uFntMTOz6prZI1kEHBcRWwLbA0dKGgqcANweEUOA2/Nn8r5RwFbAHsB5klbMZZ0PjAGG5NceOf0w4LWI2Bw4Fzijie0xM7MqmhZIIuKliHg4by8EngT6AyOBi/NhFwP75e2RwISIeCcingVmASMk9QN6R8R9ERHAJW3yVMq6Ctit0lsxM7Ou0SVzJHnI6WPAA8CGEfESpGADbJAP6w+8WMg2O6f1z9tt05fKExGLgDeA9aucf4ykKZKmzJ8/v5NaZWZm0AWBRNKawNXAMRGxoN6hVdKiTnq9PEsnRIyPiOERMbxv377tVdnMzDqgqYFE0kqkIPLriLgmJ8/Nw1Xk93k5fTYwsJB9ADAnpw+okr5UHkm9gLWBVzu/JWZmVkszr9oScAHwZEScU9g1CRidt0cD1xfSR+UrsTYhTao/mIe/FkraPpd5SJs8lbIOAO7I8yhmZtZFejWx7B2ALwHTJT2a074HnA5MlHQY8AJwIEBEzJA0EXiCdMXXkRHxfs53BHARsBpwY35BClSXSppF6omMamJ7zMysiqYFkoj4PdXnMAB2q5HnNOC0KulTgK2rpP+NHIjMzKx7+M52MzMrxYHEzMxKcSAxM7NSHEjMzKyUDgUSSStI6t2sypiZWetpN5BI+o2k3pLWIF2aO1PSt5tfNTMzawWN9EiG5qVN9gMmA4NI94eYmZk1FEhWykud7AdcHxHvUWU9KzMz65kaCSQ/B54D1gDukbQxUG/xRTMz60HaDSQRMS4i+kfEXnkdqxeAXZpfNTMzawXtLpEi6WngfuBe4J6IqKyFZWZm1thkO2l4a33gJ5KekXRtc6tlZmatopFA8j7wXn5fDMxlyTNEzMysh2tk9d8FwHTgHOAXEfFKc6tkZmatpJEeycHAPcB/ABMknSKp6jLwZmbW87TbI4mI64HrJX0Y2BM4BvgO6SFTZmbWwzWyRMrV+cqtn5LuJTkEWLfZFTMzs9bQyBzJ6cDDhcfempmZ/V0jcyQzgLGSxgNIGiJp7+ZWy8zMWkUjgeRXwLvAJ/Ln2cCPmlYjMzNrKY0Eks0i4kzSvSRExNuAmlorMzNrGY0EknclrUZe8VfSZsA7Ta2VmZm1jEYm238I3AQMlPRrYAfgy82slJmZtY5G7iO5VdLDwPakIa2jI+LlptfMzMxaQs2hrXwDIpI+DmwMvATMAQblNDMzs7o9kmOBMcDZVfYFsGtTamRmZi2lZiCJiDH53Q+xMjOzmhpZImWapLH5ai0zM7OlNHL5776kZ5FMlPSQpOMlDWpyvczMrEU08sz25yPizIjYFvh3YBvg2abXzMzMWkIj95EgaTDweeAgUu/kO02sk5mZtZB2A4mkB4CVgCuBAyPimabXyszMWkYjcySjI+LjEfHjjgQRSRdKmifp8ULayZL+LOnR/NqrsG+spFmSZkravZC+raTped84Scrpq0i6Iqc/kHtNZmbWxRoJJK9JukDSjQCShko6rIF8FwF7VEk/NyKG5dfkSpnAKGCrnOc8SSvm488n3c8yJL8qZR4GvBYRmwPnAmc0UCczM+tkjQSSi4CbgY3y5z+RHrdbV0TcA7zaYD1GAhMi4p2IeBaYBYyQ1A/oHRH3RUQAlwD7FfJcnLevAnar9FbMzKzrNBJI+kTERGAxQEQsIk24L6tvSHosD31VHtnbH3ixcMzsnNY/b7dNXypPrtMbwPol6mVmZsugkUDylqT1WbKM/PakP9rL4nxgM2AYae2uyvIr1XoSUSe9Xp4PkDRG0hRJU+bPn9+hCpuZWX2NBJJjgUnAZpL+lzS8dNSynCwi5kbE+xGxGPgFMCLvmg0MLBw6gLRA5Oy83TZ9qTySegFrU2MoLSLGR8TwiBjet2/fZam6mZnV0MgNiQ8DO5EetXs4aUJ8rWU5WZ7zqPgsULmiaxIwKl+JtQlpUv3BiHgJWChp+zz/cQhwfSHP6Lx9AHBHnkcxM7MuVPM+knzV1OdJcxE3RsQMSXsD44HVgI/VK1jS5cDOQB9Js0kPyNpZ0jDSENRzpMBELnsi8ASwCDgyIirzMEeQJvxXA27ML4ALgEslzSL1REZ1oN1mZtZJ6t2QeAFp6OhB4L8lPU96uNXYiLiuvYIj4uAaZdY6/jTgtCrpU4Ctq6T/DTiwvXqYmVlz1Qskw4FtImKxpFWBl4HNI+IvXVM1MzNrBfXmSN7Nk+KVb/9/chAxM7O26vVIPizpsbwt0lVbj+XtiIhtml47MzNb7tULJFt2WS3MzKxl1XvU7vNdWREzM2tNjdyQaGZmVpMDiZmZlVIzkEi6Pb97eXYzM6up3mR7P0k7AftKmkCbRRLz0ilmZtbD1QskPwBOIC2UeE6bfQHs2qxKmZlZ66h31dZVwFWSvh8Rp3ZhnczMrIXU65EAEBGnStoX+GROuisibmhutczMrFW0e9WWpB8DR5NW5n0CODqnmZmZtd8jAT4DDKusuyXpYuARYGwzK2ZmZq2h0ftI1ilsr92EepiZWYtqpEfyY+ARSXeSLgH+JO6NmJlZ1shk++WS7gK2IwWS73o5eTMzq2ikR0J+dvqkJtfFzMxakNfaMjOzUhxIzMyslLqBRNIKkh7vqsqYmVnrqRtI8r0j0yQN6qL6mJlZi2lksr0fMEPSg8BblcSI2LdptTIzs5bRSCA5pem1MDOzltXIfSR3S9oYGBIRt0laHVix+VUzM7NW0MiijV8DrgJ+npP6A9c1sU5mZtZCGrn890hgB2ABQEQ8BWzQzEqZmVnraCSQvBMR71Y+SOpFekKimZlZQ4HkbknfA1aT9GngSuC3za2WmZm1ikYCyQnAfGA6cDgwGTipmZUyM7PW0chVW4vzw6weIA1pzYwID22ZmRnQQCCR9BngZ8DTpGXkN5F0eETc2OzKmZnZ8q+RGxLPBnaJiFkAkjYDfgc4kJiZWUNzJPMqQSR7BpjXXiZJF0qaV1z0UdJ6km6V9FR+X7ewb6ykWZJmStq9kL6tpOl53zhJyumrSLoipz8gaXAjDTYzs85VM5BI2l/S/qR1tiZL+rKk0aQrth5qoOyLgD3apJ0A3B4RQ4Db82ckDQVGAVvlPOdJqtw9fz4wBhiSX5UyDwNei4jNgXOBMxqok5mZdbJ6PZJ98mtVYC6wE7Az6QqudWtnSyLiHuDVNskjgYvz9sXAfoX0CRHxTkQ8C8wCRkjqB/SOiPvyBP8lbfJUyroK2K3SWzEzs65Tc44kIg5twvk2zI/tJSJeklS5Q74/cH/huNk57b283Ta9kufFXNYiSW8A6wMvtz2ppDGkXg2DBnlFfDOzztTIVVubAEcBg4vHd/Iy8tV6ElEnvV6eDyZGjAfGAwwfPtyXLpuZdaJGrtq6DriANDeyuOT55krql3sj/VgyaT8bGFg4bgAwJ6cPqJJezDM7L9uyNh8cSjMzsyZr5Kqtv0XEuIi4MyLurryW8XyTgNF5ezRwfSF9VL4SaxPSpPqDeRhsoaTt8/zHIW3yVMo6ALjDN0qamXW9RnokP5X0Q+AW4J1KYkQ8XC+TpMtJk/N9JM0GfgicDkyUdBjwAnBgLmuGpInAE8Ai4MiIeD8XdQTpCrDVSPeuVO5fuQC4VNIsUk9kVANtMTOzTtZIIPkI8CVgV5YMbUX+XFNEHFxj1241jj8NOK1K+hRg6yrpfyMHIjMz6z6NBJLPApsWl5I3MzOraGSOZBqwTpPrYWZmLaqRHsmGwB8lPcTScySdefmvmZm1qEYCyQ+bXgszM2tZjTyPZFkv9TUzsx6gkTvbF7LkjvGVgZWAtyKidzMrZmZmraGRHslaxc+S9gNGNKtCZmbWWhq5amspEXEd7dxDYmZmPUcjQ1v7Fz6uAAynxuKIZmbW8zRy1dY+he1FwHOkZ4GYmZk1NEfSjOeSmJnZP4iagUTSD+rki4g4tQn1MTOzFlOvR/JWlbQ1SM9KXx9wIDEzs7qP2j27si1pLeBo4FBgAnB2rXxmZtaz1J0jkbQecCzwBeBi4OMR8VpXVMzMzFpDvTmSs4D9Sc86/0hEvNlltTIzs5ZR74bE44CNgJOAOZIW5NdCSQu6pnpmZra8qzdH0uG73s3MrOdxsDAzs1IcSMzMrBQHEjMzK8WBxMzMSnEgMTOzUhxIzMysFAcSMzMrxYHEzMxKcSAxM7NSHEjMzKwUBxIzMyvFgcTMzEpxIDEzs1IcSMzMrJRuCSSSnpM0XdKjkqbktPUk3Srpqfy+buH4sZJmSZopafdC+ra5nFmSxklSd7THzKwn684eyS4RMSwihufPJwC3R8QQ4Pb8GUlDgVHAVsAewHmSVsx5zgfGAEPya48urL+ZmbF8DW2NJD0Xnvy+XyF9QkS8ExHPArOAEZL6Ab0j4r6ICOCSQh4zM+si3RVIArhF0lRJY3LahhHxEkB+3yCn9wdeLOSdndP65+226R8gaYykKZKmzJ8/vxObYWZmNR+122Q7RMQcSRsAt0r6Y51jq817RJ30DyZGjAfGAwwfPrzqMWZmtmy6pUcSEXPy+zzgWmAEMDcPV5Hf5+XDZwMDC9kHAHNy+oAq6WZm1oW6PJBIWkPSWpVt4N+Ax4FJwOh82Gjg+rw9CRglaRVJm5Am1R/Mw18LJW2fr9Y6pJDHzMy6SHcMbW0IXJuv1O0F/CYibpL0EDBR0mHAC8CBABExQ9JE4AlgEXBkRLyfyzoCuAhYDbgxv8zMrAt1eSCJiGeAj1ZJfwXYrUae04DTqqRPAbbu7DqamVnjlqfLf83MrAU5kJiZWSkOJGZmVooDiZmZleJAYmZmpTiQmJlZKQ4kZmZWigOJmZmV4kBiZmalOJCYmVkpDiRmZlaKA4mZmZXiQGJmZqU4kJiZWSkOJGZmVooDiZmZleJAYmZmpTiQmJlZKQ4kZmZWigOJmZmV4kBiZmalOJCYmVkpDiRmZlaKA4mZmZXiQGJmZqU4kJiZWSkOJGZmVooDiZmZleJAYmZmpTiQmJlZKQ4kZmZWigOJmZmV4kBiZmaltHwgkbSHpJmSZkk6obvrY2bW07R0IJG0IvA/wJ7AUOBgSUO7t1ZmZj1LSwcSYAQwKyKeiYh3gQnAyG6uk5lZj9KruytQUn/gxcLn2cA/tz1I0hhgTP74pqSZXVC3ztYHeLm7K9HFelqba7ZXZ3RxTbpOT/sdQ+u2eeNaO1o9kKhKWnwgIWI8ML751WkeSVMiYnh316Mr9bQ297T2gtv8j6LVh7ZmAwMLnwcAc7qpLmZmPVKrB5KHgCGSNpG0MjAKmNTNdTIz61FaemgrIhZJ+gZwM7AicGFEzOjmajVLSw/NLaOe1uae1l5wm/8hKOIDUwpmZmYNa/WhLTMz62YOJGZmVooDSTeRtKKkRyTdkD9/VNJ9kqZL+q2k3jl9ZUm/yunTJO1cp8yj8nIxMySd2TUtaVxnt1nSMEn3S3pU0hRJI7quNe2T9Fxuw6OSpuS09STdKump/L5u4fixeamfmZJ2r1FmzfzLgya1+SxJf5T0mKRrJa3TRc1pSDPaXDj2eEkhqU+z21FKRPjVDS/gWOA3wA3580PATnn7K8CpeftI4Fd5ewNgKrBClfJ2AW4DVqkc291t7II23wLsmbf3Au7q7ja2qd9zQJ82aWcCJ+TtE4Az8vZQYBqwCrAJ8DSwYpUyq+ZfXl5NavO/Ab3y9hk9oc352IGkC4meb1v+8vZyj6QbSBoAfAb4ZSH5Q8A9eftW4HN5eyhwO0BEzANeB6rdzHQEcHpEvFM4drnRpDYH0Dtvr01r3EM0Erg4b18M7FdInxAR70TEs8As0hJAjeZfnpVqc0TcEhGL8sf7SfeLLe/K/p4BzgW+Q5WbrJc3DiTd479I/0AWF9IeB/bN2wey5EbLacBISb0kbQJsy9I3YVZsAfyrpAck3S1pu6bUfNn9F53f5mOAsyS9CPwEGNv51S4lgFskTc3L9ABsGBEvAeT3DXJ6teV++lcps1b+5UUz2lz0FeDGTqxvZ+j0NkvaF/hzRExrXrU7T0vfR9KKJO0NzIuIqW3G/r8CjJP0A9JNle/m9AuBLYEppC7uH4BFfFAvYF1ge2A7YKKkTSP3kbtTE9t8BPCtiLha0ueBC4BPNaURy2aHiJgjaQPgVkl/rHNsQ8v9tICmtVnSiaR/B78uWcfO1qltlrQ6cCJpSK8lOJB0vR2AfSXtBawK9JZ0WUR8kfwPR9IWpGEgcpf+W5XMkv4APFWl3NnANTlwPChpMWlxuPnNbEyDmtXm0cDReftKlh4263YRMSe/z5N0LWkIY66kfhHxkqR+QGUIstHlfmrlXy40qc1IGg3sDey2PHw5KmpCmzcjzZ9Mk1Q55mFJIyLiL01syjLz0FYXi4ixETEgIgaTlnS5IyK+mL/NIGkF4CTgZ/nz6pLWyNufBhZFxBNVir4O2DUftwWwMsvJCqNNbPMcYKe8vSvVg023kLSGpLUq26SA+Tip5zU6HzYauD5vTwJGSVolD+cNAR6sUnSt/N2uWW2WtAfwXWDfiPhrc1vRMc1oc0RMj4gNImJw/j8zG/j48hpEAF+11Z0vYGeWXMF0NPCn/DqdJasODAZmAk+SrsrauJD/l8DwvL0ycBnpH/HDwK7d3b4uaPOOpCu6pgEPANt2d/sK9dw012saMAM4MaevT7qQ4Kn8vl4hz4mkq3hmkq9Gq9Lmmvm7+9XENs8izSs8ml8/6+62NrvNbc7xHMv5VVteIsXMzErx0JaZmZXiQGJmZqU4kJiZWSkOJGZmVooDiZmZleJAYj2CpDebXP4x+Y7k0ufL9xjclleTPajNvoskPZv3TZO02zKeY7ikcctaR7Mi39lu1jmOId3H0xk3zH0MWCkihtXY/+2IuErSLqTHtg7p6AkiYgppCRqz0twjsR5L0maSbsqL7d0r6cM5/SJJ4yT9QdIzkg7I6StIOk/peS83SJos6QBJ3wQ2Au6UdGeh/NNyr+F+SRtWOf96kq5Tes7G/ZK2yXf7XwYMy72Ozeo04T7ygn9Kz3o5S9JDubzDc/oVeWmayjkvkvQ5STtryXNh1pB0Yc77iKSROX2ypG3y9iN5TTQknSrpq8v+k7d/NA4k1pONB46KiG2B44HzCvv6ke6c35t01z3A/qS77j8CfBX4F4CIGEdarmWXiNglH7sGcH9EfJS0VP7Xqpz/FOCRiNgG+B5wSaRl878K3BsRwyLi6Tr134O0NA7AYcAbEbEdadHOr+UlOCYAB0F6YBiwGzC5TTknkpat2Y70XJuz8nIf95BWlO5NWixxh3z8jsC9deplPYyHtqxHkrQm8AngyrwwHqSHDVVcFxGLgScKvYkdgStz+l+KvY8q3gVuyNtTgU9XOWZH8jNYIuIOSetLWruB6p+l9ATMDUirPUNa42mbSu+J9HyWIaQl18dJWoUUeO6JiLcLba7k3VfS8fnzqsAgUrD4JvAs8Dvg03keaHBEzGygntZDOJBYT7UC8HqdeYh3Cttq896I92LJ+kPvU/3/2rIuHf9t4BrSH/mLSc9rEal3dfMHTiLdBexO6plcXqMen2sbHHIPZjjwDOnBY31IPaupDdTRehAPbVmPFBELgGclHQig5KPtZPs98Lk8V7IhaQHKioXAWh2sxj3AF/L5dwZezvVqV+4V/RRYQem53zcDR0haKZe3RWUFZdLw1qHAv+bj2roZOEq5myLpY/kc75IWS/w86cmE95KGAD2sZUtxILGeYnVJswuvY0l/xA+TVFm5dWQ7ZVxNWtL7ceDnpBWH38j7xgM3tjPc1dbJwHBJj5HmYUbXP3xpucfzI9KTJ38JPEF6bkWlfpVe0C3AJ4HbcnBo61RgJeCxnPfUwr57gbmRlm+/l/RsDAcSW4pX/zXrAElrRsSbktYnPUdih1ienxNh1gU8R2LWMTdIWof0/JdTHUTM3CMxM7OSPEdiZmalOJCYmVkpDiRmZlaKA4mZmZXiQGJmZqX8H1J+o6R4DeLTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Visualize the distribution of review lengths\n",
    "plt.hist([len(x) for x in x_train], bins=50)\n",
    "plt.xlabel('Length of Review')\n",
    "plt.ylabel('Number of Reviews')\n",
    "plt.title('Distribution of Review Lengths')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "raw",
   "id": "73540adb",
   "metadata": {},
   "source": [
    "\n",
    "pip install wordcloud --trusted-host pypi.org --trusted-host files.pythonhosted.org pydot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07d43e8d",
   "metadata": {},
   "source": [
    "## Build the model\n",
    "\n",
    "\n",
    "- def create_model(embedding_size=128, lstm_units=64): declares a function named create_model with two optional arguments: embedding_size and lstm_units, which determine the dimensions of the embedding layer and LSTM layer respectively.\n",
    "\n",
    "## Model Construction:\n",
    "\n",
    "- **model = Sequential()**: initializes a sequential model, which allows stacking layers sequentially.\n",
    "- **model.add(Embedding(max_features, embedding_size, input_length=maxlen))**:  adds an embedding layer to the model. This layer converts input integer sequences into dense vectors of fixed size (embedding_size), useful for handling text data. max_features represents the size of the vocabulary, and maxlen specifies the maximum length of input sequences.\n",
    "- **model.add(LSTM(lstm_units, dropout=0.2, recurrent_dropout=0.2))**:  adds a Long Short-Term Memory (LSTM) layer to the model. LSTMs are a type of recurrent neural network (RNN) that are capable of learning long-term dependencies in sequential data. The dropout parameter sets the dropout rate for input units, while recurrent_dropout sets the dropout rate for recurrent units, both helping prevent overfitting.\n",
    "- **model.add(Dense(1, activation='sigmoid'))**: adds a dense output layer with a single neuron and a sigmoid activation function. This layer produces binary classification probabilities, with values between 0 and 1.\n",
    "\n",
    "- Model Compilation:**model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])** :compiles the model, specifying the loss function (binary_crossentropy for binary classification), the optimizer (adam), and the evaluation metric (accuracy). This step configures the model for training.\n",
    "\n",
    "\n",
    "\n",
    "### Dropout: \n",
    "In TensorFlow, dropout randomly deactivates a fraction of neurons during training, preventing overfitting by encouraging network robustness. It discourages co-adaptation among neurons, fostering independence and generalization. Dropout acts as a form of ensemble learning, training multiple architectures and combining them during inference. Implementation in TensorFlow involves inserting a dropout layer with a specified dropout rate within the neural network architecture. Its role is vital for improving model performance and preventing overfitting in deep learning tasks.\n",
    "\n",
    "\n",
    "-  This dropout rate is a scalar value representing the fraction of neurons to randomly drop during training. It typically ranges between 0 and 1, where:\n",
    "\n",
    "- 0 means no neurons are dropped, resulting in no regularization effect.\n",
    " - 1 means all neurons are dropped, effectively disabling the network.\n",
    " - Commonly used dropout rates fall between 0.1 and 0.5, although the optimal rate may vary depending on the specific task, dataset, and network architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b3743bdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the model\n",
    "def create_model(embedding_size=128, lstm_units=64):\n",
    "    model = Sequential()\n",
    "    model.add(Embedding(max_features, embedding_size, input_length=maxlen))\n",
    "    model.add(LSTM(lstm_units, dropout=0.2, recurrent_dropout=0.2))\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "    model.compile(loss='binary_crossentropy',\n",
    "                  optimizer='adam',\n",
    "                  metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "model = create_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "31a6d5f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the model\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# Visualize the model architecture\n",
    "#plot_model(model, show_shapes=True)\n",
    "#plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b855a851",
   "metadata": {},
   "source": [
    "#### Windows instructions to see model Architecture\n",
    "- Download the Graphviz installer from the Graphviz download page.\n",
    "- Run the installer and follow the installation instructions.\n",
    "- Add the Graphviz bin directory to your system PATH. Typically, this path is C:\\Program Files (x86)\\GraphvizX.X\\bin.\n",
    "#### MacOS:\n",
    "bash\n",
    "brew install graphviz"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1525c8f",
   "metadata": {},
   "source": [
    "## Hyperparameter tuning\n",
    "Hypertuning, is the process of optimizing the hyperparameters of a machine learning model to achieve the best performance. Hyperparameters are settings that govern the training process and model architecture, which are not learned from the data. In the context of text data, hypertuning is particularly important due to the complexity and high dimensionality of textual features. Here’s a detailed explanation of hypertuning with respect to text data:\n",
    "\n",
    "## Key Hyperparameters for Text Data\n",
    "### 1.Vectorization Parameters:\n",
    "\n",
    "- Vocabulary Size: Limits the number of unique words or tokens to consider.\n",
    "- n-gram Range: Specifies the range of n-grams (e.g., unigrams, bigrams) to include.\n",
    "- Max Document Frequency: Filters out words that appear in a high percentage of documents.\n",
    "- Min Document Frequency: Filters out words that appear in a low percentage of documents.\n",
    "\n",
    "### 2. Model Parameters:\n",
    "- Regularization Strength (C or alpha): Controls the degree of regularization applied to avoid overfitting.\n",
    "- Learning Rate: Governs the step size in the gradient descent optimization.\n",
    "- Number of Layers and Units: Determines the depth and width of neural networks.\n",
    "- Dropout Rate: Prevents overfitting by randomly dropping units during training.\n",
    "- Batch Size: Specifies the number of samples processed before the model’s internal parameters are updated.\n",
    "- Number of Epochs: Determines the number of times the learning algorithm will work through the entire training dataset.\n",
    "\n",
    "### 3/ Preprocessing Parameters:\n",
    "\n",
    "- Text Cleaning Options: Choices such as lowercasing, removing punctuation, and stemming/lemmatization.\n",
    "- Tokenization Method: The technique used to split text into tokens (e.g., word-level, character-level).\n",
    "### Hypertuning Techniques\n",
    "- Grid Search:\n",
    "Exhaustively searches through a specified subset of hyperparameters.\n",
    "Example: Testing combinations of n-gram ranges (1,1), (1,2), (1,3) and regularization strengths 0.1, 1, 10.\n",
    "Random Search:\n",
    "\n",
    "- Randomly samples hyperparameter combinations.\n",
    "Can be more efficient than grid search when the search space is large.\n",
    "\n",
    "- Bayesian Optimization:\n",
    "Uses probabilistic models to predict performance and select the next hyperparameter set to try.\n",
    "More efficient than grid and random search as it focuses on the most promising areas of the search space.\n",
    "\n",
    "- Gradient-Based Optimization:\n",
    "Techniques like Hyperband or Population Based Training (PBT) that dynamically allocate resources and adapt hyperparameters during training.\n",
    "\n",
    "### Process of Hypertuning\n",
    "- Define the Hyperparameter Space:\n",
    "\n",
    "- Identify the hyperparameters to tune and their possible values or ranges.\n",
    "- Example for text classification: {'max_features': [1000, 5000, 10000], 'ngram_range': [(1,1), (1,2)], 'alpha': [0.001, 0.01, 0.1, 1]}.\n",
    "- Split the Data:\n",
    "Divide the text data into training, validation, and test sets.\n",
    "Use the training set to train models and the validation set to evaluate hyperparameter combinations.\n",
    "- Search and Evaluate:\n",
    "Use a hypertuning method (e.g., grid search) to train models with different hyperparameter combinations.\n",
    "Evaluate each combination using a performance metric like accuracy, F1 score, or log loss.\n",
    "- Select the Best Hyperparameters:\n",
    "\n",
    "- Choose the hyperparameters that yield the best performance on the validation set.\n",
    "Train a final model on the combined training and validation sets using these hyperparameters.\n",
    "- Test the Final Model:\n",
    "\n",
    "Evaluate the final model on the test set to ensure it generalizes well to unseen data."
   ]
  },
  {
   "cell_type": "raw",
   "id": "09909ced",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning\n",
    "model = KerasClassifier(build_fn=create_model, verbose=0)\n",
    "\n",
    "param_dist = {\n",
    "    'embedding_size': [64, 128],\n",
    "    'lstm_units': [32, 64],\n",
    "    'batch_size': [32],\n",
    "    'epochs': [3]\n",
    "}\n",
    "\n",
    "# Reduce n_iter and cv for faster tuning\n",
    "random_search = RandomizedSearchCV(estimator=model, param_distributions=param_dist, n_iter=2, cv=2, verbose=2, n_jobs=1)\n",
    "\n",
    "# Use a smaller subset of the data for quicker hyperparameter tuning\n",
    "x_train_subset = x_train[:2000]\n",
    "y_train_subset = y_train[:2000]\n",
    "\n",
    "random_search_result = random_search.fit(x_train_subset, y_train_subset)\n",
    "\n",
    "# Print the best parameters\n",
    "print(\"Best parameters found: \", random_search_result.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c2af847",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1000\n",
      " 38/125 [========>.....................] - ETA: 5:59 - loss: 0.6295 - accuracy: 0.6458"
     ]
    }
   ],
   "source": [
    "#  Train the model using the best parameters\n",
    "#best_params = random_search_result.best_params_\n",
    "#model = create_model(embedding_size=best_params['embedding_size'], lstm_units=best_params['lstm_units'])\n",
    "model.fit(x_train, y_train,\n",
    "          #batch_size=best_params['batch_size'],\n",
    "          batch_size = 200,\n",
    "          epochs=1000,\n",
    "          validation_data=(x_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94ead96f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the model\n",
    "score, acc = model.evaluate(x_test, y_test,\n",
    "                            batch_size=best_params['batch_size'])\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e45d2317",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample new data for prediction\n",
    "new_reviews = [\"This movie was excellent!\", \"The movie was terrible...\"]\n",
    "\n",
    "# Preprocess the new data\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_features)\n",
    "tokenizer.fit_on_texts(new_reviews)\n",
    "new_sequences = tokenizer.texts_to_sequences(new_reviews)\n",
    "new_padded_sequences = pad_sequences(new_sequences, maxlen=maxlen)\n",
    "\n",
    "# Make predictions\n",
    "predictions = model.predict(new_padded_sequences)\n",
    "print(\"Predictions:\", predictions)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}