L5-Evaluation.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "52824b89-532a-4e54-87e9-1410813cd39e",
   "metadata": {},
   "source": [
    "# LangChain: Evaluation\n",
    "When building a complex application using an LLM on of the important but sometimes tricky steps is how do you evaluate how your application performance? \n",
    "Also, if you decide to change your implementation, maybe change the parameters of your system or the strategy of how you use a vector database or something else, how do you know if you're making it better or worse?\n",
    "\n",
    "In this notebook, we will dive into some frameworks for evaluating a LLM-based application as well as some tools to help you do that.\n",
    "## Outline:\n",
    "\n",
    "* Example generation\n",
    "* Manual evaluation (and debuging)\n",
    "* LLM-assisted evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b7ed03ed-1322-49e3-b2a2-33e94fb592ef",
   "metadata": {
    "height": 81,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import openai\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "import os\n",
    "\n",
    "_ = load_dotenv(find_dotenv())  # add .env to .gitignore\n",
    "openai.api_key = os.getenv(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "28008949",
   "metadata": {},
   "source": [
    "## Create our Q&A application"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "22cac268",
   "metadata": {},
   "source": [
    "First, we need to have the chain or the application that we're going to evaluate in the first place. We're going to use the document question answering chain from the previous lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "974acf8e-8f88-42de-88f8-40a82cb58e8b",
   "metadata": {
    "height": 98
   },
   "outputs": [],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.document_loaders import CSVLoader\n",
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "from langchain.vectorstores import DocArrayInMemorySearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9ec1106d",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "file = 'OutdoorClothingCatalog_1000.csv'\n",
    "loader = CSVLoader(file_path=file, encoding='utf-8')\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b31c218f",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "# Create index\n",
    "index = VectorstoreIndexCreator(\n",
    "    vectorstore_cls=DocArrayInMemorySearch\n",
    ").from_loaders([loader])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "86f6afd7",
   "metadata": {},
   "source": [
    "Then, we're going to reate the retrieval QA chain by specifying the language model, the chain type, the retriever, and then the verbosity that we're going to print out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a2006054",
   "metadata": {
    "height": 183
   },
   "outputs": [],
   "source": [
    "\n",
    "llm = ChatOpenAI(temperature = 0.0)\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm, \n",
    "    chain_type=\"stuff\", \n",
    "    retriever=index.vectorstore.as_retriever(), \n",
    "    verbose=True,\n",
    "    chain_type_kwargs = {\n",
    "        \"document_separator\": \"<<<<>>>>>\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "333d1bad",
   "metadata": {},
   "source": [
    " The first thing we need to do is we need to really figure out what are some data points that we want to evaluate it on.\n",
    " So there's a few different methods that we're going to cover for doing this:\n",
    " "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "791ebd73",
   "metadata": {},
   "source": [
    "## Example Generation\n",
    "\n",
    "### 1. Manual Generation (not practical)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ddc2e400",
   "metadata": {},
   "source": [
    "The first is the most simple, which is basically we're going to come up with data points that we think are good examples ourselves. So to do that, we can just look at some of the data and come up with example questions and then example ground truth answers that we can later use to evaluate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fb04a0f9",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=\": 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.\", metadata={'source': 'OutdoorClothingCatalog_1000.csv', 'row': 10})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fe4a88c2",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content=': 11\\nname: Ultra-Lofty 850 Stretch Down Hooded Jacket\\ndescription: This technical stretch down jacket from our DownTek collection is sure to keep you warm and comfortable with its full-stretch construction providing exceptional range of motion. With a slightly fitted style that falls at the hip and best with a midweight layer, this jacket is suitable for light activity up to 20° and moderate activity up to -30°. The soft and durable 100% polyester shell offers complete windproof protection and is insulated with warm, lofty goose down. Other features include welded baffles for a no-stitch construction and excellent stretch, an adjustable hood, an interior media port and mesh stash pocket and a hem drawcord. Machine wash and dry. Imported.', metadata={'source': 'OutdoorClothingCatalog_1000.csv', 'row': 11})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[11]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5262e669",
   "metadata": {},
   "source": [
    "And from these details, we can create some example query and answer pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c2d59bf2",
   "metadata": {
    "height": 217
   },
   "outputs": [],
   "source": [
    "examples = [\n",
    "    {\n",
    "        \"query\": \"Do the Cozy Comfort Pullover Set\\\n",
    "        have side pockets?\",\n",
    "        \"answer\": \"Yes\"\n",
    "    },\n",
    "    {\n",
    "        \"query\": \"What collection is the Ultra-Lofty \\\n",
    "        850 Stretch Down Hooded Jacket from?\",\n",
    "        \"answer\": \"The DownTek collection\"\n",
    "    }\n",
    "]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "69414d5a",
   "metadata": {},
   "source": [
    " But this doesn't really scale that well. We can automate it is with language models themselves.\n",
    " So we have a chain in LingChain that can do exactly that. In other words, the LLM will take in documents and it will create a question answer pair from each document."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c7ce3e4f",
   "metadata": {},
   "source": [
    "### 2. LLM-Generated examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d44f8376",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAGenerateChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "34e87816",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "# Create a QA pair for each document\n",
    "example_gen_chain = QAGenerateChain.from_llm(ChatOpenAI())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "62abae09",
   "metadata": {
    "height": 64
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 9c4dbbadad49795476659b4db8ba8027 in your message.).\n"
     ]
    }
   ],
   "source": [
    "new_examples = example_gen_chain.apply_and_parse( # to apply output parser to the results to get dict not str\n",
    "    [{\"doc\": t} for t in data[:5]]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "97ab28b5",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'query': \"What is the weight of one pair of Women's Campside Oxfords?\",\n",
       "  'answer': \"The approximate weight of one pair of Women's Campside Oxfords is 1 lb.1 oz.\"},\n",
       " {'query': 'What are the dimensions of the small and medium Recycled Waterhog Dog Mat, Chevron Weave?',\n",
       "  'answer': 'The dimensions of the small Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\" and the dimensions of the medium mat are 22.5\" x 34.5\".'},\n",
       " {'query': \"What features does the Infant and Toddler Girls' Coastal Chill Swimsuit have?\",\n",
       "  'answer': \"The swimsuit has bright colors, ruffles, and exclusive whimsical prints. It is made of four-way-stretch and chlorine-resistant fabric that keeps its shape and resists snags. The swimsuit is also UPF 50+ rated, providing the highest rated sun protection possible, blocking 98% of the sun's harmful rays. It has crossover no-slip straps and a fully lined bottom to ensure a secure fit and maximum coverage. The swimsuit can be machine washed and line dried for best results.\"},\n",
       " {'query': 'What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts?',\n",
       "  'answer': 'The body of the tankini is made of 82% recycled nylon with 18% Lycra® spandex, and it is lined with 90% recycled nylon with 10% Lycra® spandex.'},\n",
       " {'query': 'What is the main feature of the EcoFlex 3L Storm Pants?',\n",
       "  'answer': 'The main feature of the EcoFlex 3L Storm Pants is the state-of-the-art TEK O2 technology that offers the most breathability ever tested.'}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_examples"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6bbb78af",
   "metadata": {},
   "source": [
    "We just generated a bunch of question answer pairs without writting them ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ada2a3fc",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "# Combine examples \n",
    "examples += new_examples"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a5ba97af",
   "metadata": {},
   "source": [
    "So we got these examples now, but how exactly do we evaluate what's going on? \n",
    "\n",
    "* The first thing we want to do is just run an example through the chain and take a look at the output it produces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9cdf5cf5",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The Cozy Comfort Pullover Set, Stripe has side pockets.'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.run(examples[0][\"query\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "952c4e0a",
   "metadata": {},
   "source": [
    "But it's oftentimes **not enough** to just look at the final answer to understand what is or could be going wrong in the chain.\n",
    "\n",
    "We want to see what is actually happening inside the chain. For example, \n",
    "* What is the actual prompt that's going into the language model? \n",
    "* What are the documents that it retrieves? \n",
    "* If this were a more complex chain with multiple steps in it, what are the intermediate results?\n",
    "\n",
    "So to do that, we can use a ``langchain.debug``:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "63f3cb08",
   "metadata": {},
   "source": [
    "## Manual Evaluation\n",
    "### 1. Manual Evaluation (not practical)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "540a686e",
   "metadata": {},
   "source": [
    "Set ``langchain.debug`` equals true and we now rerun the same example as above, we can see that it starts printing out a lot more information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fcaf622e",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "import langchain\n",
    "langchain.debug = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8a142638",
   "metadata": {
    "height": 30,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"query\": \"Do the Cozy Comfort Pullover Set        have side pockets?\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain] Entering Chain run with input:\n",
      "\u001b[0m[inputs]\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"question\": \"Do the Cozy Comfort Pullover Set        have side pockets?\",\n",
      "  \"context\": \": 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.<<<<>>>>>: 73\\nname: Cozy Cuddles Knit Pullover Set\\ndescription: Perfect for lounging, this knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out. \\n\\nSize & Fit \\nPants are Favorite Fit: Sits lower on the waist. \\nRelaxed Fit: Our most generous fit sits farthest from the body. \\n\\nFabric & Care \\nIn the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features \\nRelaxed fit top with raglan sleeves and rounded hem. \\nPull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg. \\nImported.<<<<>>>>>: 632\\nname: Cozy Comfort Fleece Pullover\\ndescription: The ultimate sweater fleece – made from superior fabric and offered at an unbeatable price. \\n\\nSize & Fit\\nSlightly Fitted: Softly shapes the body. Falls at hip. \\n\\nWhy We Love It\\nOur customers (and employees) love the rugged construction and heritage-inspired styling of our popular Sweater Fleece Pullover and wear it for absolutely everything. From high-intensity activities to everyday tasks, you'll find yourself reaching for it every time.\\n\\nFabric & Care\\nRugged sweater-knit exterior and soft brushed interior for exceptional warmth and comfort. Made from soft, 100% polyester. Machine wash and dry.\\n\\nAdditional Features\\nFeatures our classic Mount Katahdin logo. Snap placket. Front princess seams create a feminine shape. Kangaroo handwarmer pockets. Cuffs and hem reinforced with jersey binding. Imported.\\n\\n – Official Supplier to the U.S. Ski Team\\nTHEIR WILL TO WIN, WOVEN RIGHT IN. LEARN MORE<<<<>>>>>: 151\\nname: Cozy Quilted Sweatshirt\\ndescription: Our sweatshirt is an instant classic with its great quilted texture and versatile weight that easily transitions between seasons. With a traditional fit that is relaxed through the chest, sleeve, and waist, this pullover is lightweight enough to be worn most months of the year. The cotton blend fabric is super soft and comfortable, making it the perfect casual layer. To make dressing easy, this sweatshirt also features a snap placket and a heritage-inspired Mt. Katahdin logo patch. For care, machine wash and dry. Imported.\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain > 4:llm:ChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"System: Use the following pieces of context to answer the users question. \\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\\n----------------\\n: 10\\nname: Cozy Comfort Pullover Set, Stripe\\ndescription: Perfect for lounging, this striped knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out.\\n\\nSize & Fit\\n- Pants are Favorite Fit: Sits lower on the waist.\\n- Relaxed Fit: Our most generous fit sits farthest from the body.\\n\\nFabric & Care\\n- In the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features\\n- Relaxed fit top with raglan sleeves and rounded hem.\\n- Pull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg.\\n\\nImported.<<<<>>>>>: 73\\nname: Cozy Cuddles Knit Pullover Set\\ndescription: Perfect for lounging, this knit set lives up to its name. We used ultrasoft fabric and an easy design that's as comfortable at bedtime as it is when we have to make a quick run out. \\n\\nSize & Fit \\nPants are Favorite Fit: Sits lower on the waist. \\nRelaxed Fit: Our most generous fit sits farthest from the body. \\n\\nFabric & Care \\nIn the softest blend of 63% polyester, 35% rayon and 2% spandex.\\n\\nAdditional Features \\nRelaxed fit top with raglan sleeves and rounded hem. \\nPull-on pants have a wide elastic waistband and drawstring, side pockets and a modern slim leg. \\nImported.<<<<>>>>>: 632\\nname: Cozy Comfort Fleece Pullover\\ndescription: The ultimate sweater fleece – made from superior fabric and offered at an unbeatable price. \\n\\nSize & Fit\\nSlightly Fitted: Softly shapes the body. Falls at hip. \\n\\nWhy We Love It\\nOur customers (and employees) love the rugged construction and heritage-inspired styling of our popular Sweater Fleece Pullover and wear it for absolutely everything. From high-intensity activities to everyday tasks, you'll find yourself reaching for it every time.\\n\\nFabric & Care\\nRugged sweater-knit exterior and soft brushed interior for exceptional warmth and comfort. Made from soft, 100% polyester. Machine wash and dry.\\n\\nAdditional Features\\nFeatures our classic Mount Katahdin logo. Snap placket. Front princess seams create a feminine shape. Kangaroo handwarmer pockets. Cuffs and hem reinforced with jersey binding. Imported.\\n\\n – Official Supplier to the U.S. Ski Team\\nTHEIR WILL TO WIN, WOVEN RIGHT IN. LEARN MORE<<<<>>>>>: 151\\nname: Cozy Quilted Sweatshirt\\ndescription: Our sweatshirt is an instant classic with its great quilted texture and versatile weight that easily transitions between seasons. With a traditional fit that is relaxed through the chest, sleeve, and waist, this pullover is lightweight enough to be worn most months of the year. The cotton blend fabric is super soft and comfortable, making it the perfect casual layer. To make dressing easy, this sweatshirt also features a snap placket and a heritage-inspired Mt. Katahdin logo patch. For care, machine wash and dry. Imported.\\nHuman: Do the Cozy Comfort Pullover Set        have side pockets?\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain > 4:llm:ChatOpenAI] [1.68s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"The Cozy Comfort Pullover Set, Stripe has side pockets on the pull-on pants.\",\n",
      "        \"generation_info\": null,\n",
      "        \"message\": {\n",
      "          \"content\": \"The Cozy Comfort Pullover Set, Stripe has side pockets on the pull-on pants.\",\n",
      "          \"additional_kwargs\": {},\n",
      "          \"example\": false\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 734,\n",
      "      \"completion_tokens\": 18,\n",
      "      \"total_tokens\": 752\n",
      "    },\n",
      "    \"model_name\": \"gpt-3.5-turbo\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain > 3:chain:LLMChain] [1.69s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"text\": \"The Cozy Comfort Pullover Set, Stripe has side pockets on the pull-on pants.\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 2:chain:StuffDocumentsChain] [1.69s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"output_text\": \"The Cozy Comfort Pullover Set, Stripe has side pockets on the pull-on pants.\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA] [1.91s] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"result\": \"The Cozy Comfort Pullover Set, Stripe has side pockets on the pull-on pants.\"\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'The Cozy Comfort Pullover Set, Stripe has side pockets on the pull-on pants.'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa.run(examples[0][\"query\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "054226b9",
   "metadata": {},
   "source": [
    "Similar to creating the examples, that starts to get a little bit tedious over time. And so let's go back to our favorite solution.\n",
    "Can we ask a language model to do it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b3d6bef0",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "# Turn off the debug mode\n",
    "langchain.debug = False"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d5bdbdce",
   "metadata": {},
   "source": [
    "### 2. LLM assisted evaluation\n",
    "First, we need to create predictions for all the examples (we have 7 examples). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a4dca05a",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "predictions = qa.apply(examples)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9318709f",
   "metadata": {},
   "source": [
    "Now that we've got these examples, we can think about evaluating them. So we're going to import the ``QAEvalChain``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6012a3e0",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAEvalChain"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5808993e",
   "metadata": {},
   "source": [
    "We are going to create this chain with a language model, because again, we're going to be using a language model to help do the evaluation. And then we're going to call evaluate on this chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "724b1c0b",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "llm = ChatOpenAI(temperature=0)\n",
    "eval_chain = QAEvalChain.from_llm(llm)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e910229b",
   "metadata": {},
   "source": [
    " We're going to pass in examples and predictions, and we're going to get back a bunch of graded outputs. And so in order to see what exactly is going on for each example, we're going to loop through them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8b46ae55",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "graded_outputs = eval_chain.evaluate(examples, predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3437cfbe",
   "metadata": {
    "height": 132
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example 0:\n",
      "Question: Do the Cozy Comfort Pullover Set        have side pockets?\n",
      "Real Answer: Yes\n",
      "Predicted Answer: The Cozy Comfort Pullover Set, Stripe has side pockets.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 1:\n",
      "Question: What collection is the Ultra-Lofty         850 Stretch Down Hooded Jacket from?\n",
      "Real Answer: The DownTek collection\n",
      "Predicted Answer: The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 2:\n",
      "Question: What is the weight of one pair of Women's Campside Oxfords?\n",
      "Real Answer: The approximate weight of one pair of Women's Campside Oxfords is 1 lb.1 oz.\n",
      "Predicted Answer: The weight of one pair of Women's Campside Oxfords is approximately 1 lb. 1 oz.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 3:\n",
      "Question: What are the dimensions of the small and medium Recycled Waterhog Dog Mat, Chevron Weave?\n",
      "Real Answer: The dimensions of the small Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\" and the dimensions of the medium mat are 22.5\" x 34.5\".\n",
      "Predicted Answer: The dimensions of the small Recycled Waterhog Dog Mat, Chevron Weave are 18\" x 28\" and the dimensions of the medium Recycled Waterhog Dog Mat, Chevron Weave are 22.5\" x 34.5\".\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 4:\n",
      "Question: What features does the Infant and Toddler Girls' Coastal Chill Swimsuit have?\n",
      "Real Answer: The swimsuit has bright colors, ruffles, and exclusive whimsical prints. It is made of four-way-stretch and chlorine-resistant fabric that keeps its shape and resists snags. The swimsuit is also UPF 50+ rated, providing the highest rated sun protection possible, blocking 98% of the sun's harmful rays. It has crossover no-slip straps and a fully lined bottom to ensure a secure fit and maximum coverage. The swimsuit can be machine washed and line dried for best results.\n",
      "Predicted Answer: The Infant and Toddler Girls' Coastal Chill Swimsuit is a two-piece swimsuit with bright colors, ruffles, and exclusive whimsical prints. It has four-way-stretch and chlorine-resistant fabric that keeps its shape and resists snags. The UPF 50+ rated fabric provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The crossover no-slip straps and fully lined bottom ensure a secure fit and maximum coverage. It is machine washable and should be line dried for best results.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 5:\n",
      "Question: What is the fabric composition of the Refresh Swimwear, V-Neck Tankini Contrasts?\n",
      "Real Answer: The body of the tankini is made of 82% recycled nylon with 18% Lycra® spandex, and it is lined with 90% recycled nylon with 10% Lycra® spandex.\n",
      "Predicted Answer: The Refresh Swimwear, V-Neck Tankini Contrasts is made of 82% recycled nylon with 18% Lycra® spandex for the body and 90% recycled nylon with 10% Lycra® spandex for the lining.\n",
      "Predicted Grade: CORRECT\n",
      "\n",
      "Example 6:\n",
      "Question: What is the main feature of the EcoFlex 3L Storm Pants?\n",
      "Real Answer: The main feature of the EcoFlex 3L Storm Pants is the state-of-the-art TEK O2 technology that offers the most breathability ever tested.\n",
      "Predicted Answer: The main feature of the EcoFlex 3L Storm Pants is the state-of-the-art TEK O2 technology that offers the most breathability ever tested, making them great for a variety of outdoor activities year-round.\n",
      "Predicted Grade: CORRECT\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, eg in enumerate(examples):\n",
    "    print(f\"Example {i}:\")\n",
    "    print(\"Question: \" + predictions[i]['query'])\n",
    "    print(\"Real Answer: \" + predictions[i]['answer'])\n",
    "    print(\"Predicted Answer: \" + predictions[i]['result'])\n",
    "    print(\"Predicted Grade: \" + graded_outputs[i]['text'])\n",
    "    print()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "11b3083b",
   "metadata": {},
   "source": [
    "#### Why we need to use the LLM for the evaluation? Let's take a look at the first example:\n",
    "* The real answer and the predicted answer are very different. One's really short, one's really long. So if we were to try to do some string matching or exact matching or even some regexes here, it wouldn't know what to do. This shows off the **importance** of using the **LLM** to do evaluation here. As long as they have the same semantic meaning, they should be graded as being similar."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "88752dfc",
   "metadata": {},
   "source": [
    "### [LangChain Evaluation Platform](https://blog.langchain.dev/auto-evaluator-opportunities/)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "82c10d81",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}