all_prompt_engineering_techniques/evaluating-prompt-effectiveness.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Prompt Effectiveness\n",
    "\n",
    "## Overview\n",
    "This tutorial focuses on methods and techniques for evaluating the effectiveness of prompts in AI language models. We'll explore various metrics for measuring prompt performance and discuss both manual and automated evaluation techniques.\n",
    "\n",
    "## Motivation\n",
    "As prompt engineering becomes increasingly crucial in AI applications, it's essential to have robust methods for assessing prompt effectiveness. This enables developers and researchers to optimize their prompts, leading to better AI model performance and more reliable outputs.\n",
    "\n",
    "## Key Components\n",
    "1. Metrics for measuring prompt performance\n",
    "2. Manual evaluation techniques\n",
    "3. Automated evaluation techniques\n",
    "4. Practical examples using OpenAI and LangChain\n",
    "\n",
    "## Method Details\n",
    "We'll start by setting up our environment and introducing key metrics for evaluating prompts. We'll then explore manual evaluation techniques, including human assessment and comparative analysis. Next, we'll delve into automated evaluation methods, utilizing techniques like perplexity scoring and automated semantic similarity comparisons. Throughout the tutorial, we'll provide practical examples using OpenAI's GPT models and LangChain library to demonstrate these concepts in action.\n",
    "\n",
    "## Conclusion\n",
    "By the end of this tutorial, you'll have a comprehensive understanding of how to evaluate prompt effectiveness using both manual and automated techniques. You'll be equipped with practical tools and methods to optimize your prompts, leading to more efficient and accurate AI model interactions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First, let's import the necessary libraries and set up our environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from langchain_openai import ChatOpenAI\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sentence_transformers import SentenceTransformer\n",
    "import numpy as np\n",
    "\n",
    "# Load environment variables\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "\n",
    "# Set up OpenAI API key\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')\n",
    "\n",
    "# Initialize the language model\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
    "\n",
    "# Initialize sentence transformer for semantic similarity\n",
    "sentence_model = SentenceTransformer('all-MiniLM-L6-v2')\n",
    "\n",
    "def semantic_similarity(text1, text2):\n",
    "    \"\"\"Calculate semantic similarity between two texts using cosine similarity.\"\"\"\n",
    "    embeddings = sentence_model.encode([text1, text2])\n",
    "    return cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Metrics for Measuring Prompt Performance\n",
    "\n",
    "Let's define some key metrics for evaluating prompt effectiveness:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relevance_score(response, expected_content):\n",
    "    \"\"\"Calculate relevance score based on semantic similarity to expected content.\"\"\"\n",
    "    return semantic_similarity(response, expected_content)\n",
    "\n",
    "def consistency_score(responses):\n",
    "    \"\"\"Calculate consistency score based on similarity between multiple responses.\"\"\"\n",
    "    if len(responses) < 2:\n",
    "        return 1.0  # Perfect consistency if there's only one response\n",
    "    similarities = []\n",
    "    for i in range(len(responses)):\n",
    "        for j in range(i+1, len(responses)):\n",
    "            similarities.append(semantic_similarity(responses[i], responses[j]))\n",
    "    return np.mean(similarities)\n",
    "\n",
    "def specificity_score(response):\n",
    "    \"\"\"Calculate specificity score based on response length and unique word count.\"\"\"\n",
    "    words = response.split()\n",
    "    unique_words = set(words)\n",
    "    return len(unique_words) / len(words) if words else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manual Evaluation Techniques\n",
    "\n",
    "Manual evaluation involves human assessment of prompt-response pairs. Let's create a function to simulate this process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt: Explain the concept of machine learning in simple terms.\n",
      "Response: Machine learning is a type of computer technology that allows computers to learn from data and improve their performance over time without being explicitly programmed for every specific task. \n",
      "\n",
      "In simple terms, imagine teaching a child to recognize different animals. Instead of giving them a detailed description of each animal, you show them many pictures of cats, dogs, and birds. Over time, the child learns to identify these animals based on patterns they see in the images, like shapes, colors, and sizes. \n",
      "\n",
      "In the same way, machine learning involves feeding a computer lots of data (like pictures, numbers, or text) and letting it figure out patterns and make decisions on its own. For example, a machine learning model can be trained to recognize spam emails by analyzing examples of both spam and non-spam messages. Once trained, it can then automatically identify new emails as spam or not.\n",
      "\n",
      "So, in essence, machine learning is about teaching computers to learn from experience, adapt to new information, and make predictions or decisions based on what they’ve learned.\n",
      "\n",
      "Evaluation Criteria:\n",
      "Clarity: 5.0/10\n",
      "Accuracy: 5.0/10\n",
      "Simplicity: 5.0/10\n",
      "\n",
      "Additional Comments:\n",
      "Comments: 5\n"
     ]
    }
   ],
   "source": [
    "def manual_evaluation(prompt, response, criteria):\n",
    "    \"\"\"Simulate manual evaluation of a prompt-response pair.\"\"\"\n",
    "    print(f\"Prompt: {prompt}\")\n",
    "    print(f\"Response: {response}\")\n",
    "    print(\"\\nEvaluation Criteria:\")\n",
    "    for criterion in criteria:\n",
    "        score = float(input(f\"Score for {criterion} (0-10): \"))\n",
    "        print(f\"{criterion}: {score}/10\")\n",
    "    print(\"\\nAdditional Comments:\")\n",
    "    comments = input(\"Enter any additional comments: \")\n",
    "    print(f\"Comments: {comments}\")\n",
    "\n",
    "# Example usage\n",
    "prompt = \"Explain the concept of machine learning in simple terms.\"\n",
    "response = llm.invoke(prompt).content\n",
    "criteria = [\"Clarity\", \"Accuracy\", \"Simplicity\"]\n",
    "manual_evaluation(prompt, response, criteria)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automated Evaluation Techniques\n",
    "\n",
    "Now, let's implement some automated evaluation techniques:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt: What are the three main types of machine learning?\n",
      "Response: The three main types of machine learning are:\n",
      "\n",
      "1. **Supervised Learning**: In supervised learning, the model is trained on a labeled dataset, which means that the input data is paired with the correct output. The goal is for the model to learn to map inputs to the correct outputs so that it can make predictions on new, unseen data. Common applications include classification (e.g., spam detection) and regression (e.g., predicting house prices).\n",
      "\n",
      "2. **Unsupervised Learning**: In unsupervised learning, the model is trained on data that does not have labeled outputs. The goal is to identify patterns, structures, or relationships within the data. Common techniques include clustering (e.g., grouping customers based on purchasing behavior) and dimensionality reduction (e.g., reducing the number of features while retaining important information).\n",
      "\n",
      "3. **Reinforcement Learning**: In reinforcement learning, an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties based on its actions, and it aims to maximize the cumulative reward over time. This type of learning is commonly used in applications like game playing (e.g., AlphaGo) and robotics.\n",
      "\n",
      "These three types represent different approaches to learning from data and are used in various applications across multiple domains.\n",
      "\n",
      "Relevance Score: 0.74\n",
      "Specificity Score: 0.64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'relevance': 0.73795843, 'specificity': 0.6403940886699507}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def automated_evaluation(prompt, response, expected_content):\n",
    "    \"\"\"Perform automated evaluation of a prompt-response pair.\"\"\"\n",
    "    relevance = relevance_score(response, expected_content)\n",
    "    specificity = specificity_score(response)\n",
    "    \n",
    "    print(f\"Prompt: {prompt}\")\n",
    "    print(f\"Response: {response}\")\n",
    "    print(f\"\\nRelevance Score: {relevance:.2f}\")\n",
    "    print(f\"Specificity Score: {specificity:.2f}\")\n",
    "    \n",
    "    return {\"relevance\": relevance, \"specificity\": specificity}\n",
    "\n",
    "# Example usage\n",
    "prompt = \"What are the three main types of machine learning?\"\n",
    "expected_content = \"The three main types of machine learning are supervised learning, unsupervised learning, and reinforcement learning.\"\n",
    "response = llm.invoke(prompt).content\n",
    "automated_evaluation(prompt, response, expected_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparative Analysis\n",
    "\n",
    "Let's compare the effectiveness of different prompts for the same task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt: List the types of machine learning.\n",
      "Response: Machine learning can be broadly categorized into several types, each serving different purposes and applications. The main types of machine learning are:\n",
      "\n",
      "1. **Supervised Learning**:\n",
      "   - Involves training a model on a labeled dataset, where the input data is paired with the correct output. The model learns to map inputs to outputs, and its performance is evaluated based on how accurately it predicts the outcomes for new, unseen data.\n",
      "   - Common algorithms: Linear regression, logistic regression, decision trees, support vector machines, neural networks.\n",
      "\n",
      "2. **Unsupervised Learning**:\n",
      "   - Involves training a model on data without labeled responses. The model tries to learn the underlying structure or distribution in the data, often identifying patterns, clusters, or relationships.\n",
      "   - Common algorithms: K-means clustering, hierarchical clustering, principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE).\n",
      "\n",
      "3. **Semi-Supervised Learning**:\n",
      "   - Combines both labeled and unlabeled data for training. This approach is useful when obtaining a fully labeled dataset is expensive or time-consuming. The model leverages both types of data to improve learning accuracy.\n",
      "   - Common applications include image classification, text classification, and speech recognition.\n",
      "\n",
      "4. **Reinforcement Learning**:\n",
      "   - Involves training an agent to make decisions by interacting with an environment. The agent learns to achieve a goal by receiving feedback in the form of rewards or penalties. The learning process is based on trial and error.\n",
      "   - Common applications: Game playing (e.g., AlphaGo), robotics, recommendation systems.\n",
      "\n",
      "5. **Self-Supervised Learning**:\n",
      "   - A subset of unsupervised learning where the model generates its own labels from the input data, allowing it to learn representations of the data without needing labeled examples. It is often used in natural language processing and computer vision.\n",
      "   - Common techniques: Contrastive learning, predicting masked parts of input data (e.g., masked language modeling).\n",
      "\n",
      "6. **Multi-Instance Learning**:\n",
      "   - A type of learning where the model is trained on bags of instances rather than individual labeled instances. Each bag is labeled, but individual instances within the bag may not be labeled.\n",
      "   - Common applications: Drug activity prediction, image classification tasks.\n",
      "\n",
      "7. **Transfer Learning**:\n",
      "   - Involves taking a pre-trained model on one task and fine-tuning it on a different but related task. This approach is particularly useful when labeled data for the new task is scarce.\n",
      "   - Commonly used in deep learning applications, especially in computer vision and natural language processing.\n",
      "\n",
      "These types of machine learning can be applied in various domains, including healthcare, finance, marketing, and more, depending on the specific requirements of the task at hand.\n",
      "\n",
      "Relevance Score: 0.74\n",
      "Specificity Score: 0.57\n",
      "Prompt: What are the main categories of machine learning algorithms?\n",
      "Response: Machine learning algorithms can be broadly categorized into several main categories based on their learning styles and the types of problems they are designed to solve. Here are the primary categories:\n",
      "\n",
      "1. **Supervised Learning**: \n",
      "   - In this category, the algorithm is trained on labeled data, meaning that each training example is paired with an output label. The goal is to learn a mapping from inputs to outputs.\n",
      "   - Common algorithms include:\n",
      "     - Linear Regression\n",
      "     - Logistic Regression\n",
      "     - Decision Trees\n",
      "     - Support Vector Machines (SVM)\n",
      "     - Neural Networks\n",
      "     - Random Forests\n",
      "     - Gradient Boosting Machines (e.g., XGBoost)\n",
      "\n",
      "2. **Unsupervised Learning**: \n",
      "   - This type of learning deals with unlabeled data, where the algorithm tries to learn the underlying structure or distribution of the data without explicit outputs.\n",
      "   - Common algorithms include:\n",
      "     - K-Means Clustering\n",
      "     - Hierarchical Clustering\n",
      "     - Principal Component Analysis (PCA)\n",
      "     - t-Distributed Stochastic Neighbor Embedding (t-SNE)\n",
      "     - Autoencoders\n",
      "\n",
      "3. **Semi-Supervised Learning**: \n",
      "   - This category combines both labeled and unlabeled data during training. It is particularly useful when acquiring a fully labeled dataset is expensive or time-consuming.\n",
      "   - Common approaches include variations of supervised algorithms that incorporate unlabeled data to improve learning.\n",
      "\n",
      "4. **Reinforcement Learning**: \n",
      "   - In reinforcement learning, an agent learns to make decisions by taking actions in an environment to maximize a cumulative reward. The learning process involves exploration and exploitation.\n",
      "   - Common algorithms include:\n",
      "     - Q-Learning\n",
      "     - Deep Q-Networks (DQN)\n",
      "     - Policy Gradients\n",
      "     - Proximal Policy Optimization (PPO)\n",
      "     - Actor-Critic Methods\n",
      "\n",
      "5. **Self-Supervised Learning**: \n",
      "   - This is a form of unsupervised learning where the system generates its own supervisory signal from the input data. It’s particularly popular in natural language processing and computer vision.\n",
      "   - Techniques often involve predicting parts of the input data from other parts (e.g., masked language modeling in transformers).\n",
      "\n",
      "6. **Transfer Learning**: \n",
      "   - This approach involves taking a pre-trained model (often trained on a large dataset) and fine-tuning it on a smaller, task-specific dataset. This is especially useful in deep learning applications.\n",
      "\n",
      "7. **Ensemble Learning**: \n",
      "   - Ensemble methods combine multiple models to produce a better performance than any individual model. This can involve techniques such as bagging, boosting, and stacking.\n",
      "   - Common algorithms include Random Forests (bagging) and AdaBoost (boosting).\n",
      "\n",
      "These categories encompass a wide range of algorithms, each suited for different types of tasks and datasets. The choice of algorithm often depends on the problem at hand, the nature of the data, and the desired outcome.\n",
      "\n",
      "Relevance Score: 0.68\n",
      "Specificity Score: 0.60\n",
      "Prompt: Explain the different approaches to machine learning.\n",
      "Response: Machine learning (ML) is a subset of artificial intelligence that focuses on building systems that can learn from and make decisions based on data. There are several key approaches to machine learning, which can be broadly categorized into the following types:\n",
      "\n",
      "### 1. Supervised Learning\n",
      "In supervised learning, the model is trained on a labeled dataset, which means that each training example is associated with a corresponding output label. The goal is to learn a mapping from inputs to outputs so that the model can predict the label of new, unseen data.\n",
      "\n",
      "- **Examples**: \n",
      "  - Classification (e.g., spam detection, image recognition)\n",
      "  - Regression (e.g., predicting house prices, temperature forecasting)\n",
      "\n",
      "- **Common Algorithms**: \n",
      "  - Linear Regression\n",
      "  - Logistic Regression\n",
      "  - Decision Trees\n",
      "  - Support Vector Machines (SVM)\n",
      "  - Neural Networks\n",
      "\n",
      "### 2. Unsupervised Learning\n",
      "Unsupervised learning involves training a model on data that does not have labeled outputs. The goal is to find patterns, structures, or relationships within the data without explicit guidance on what to look for.\n",
      "\n",
      "- **Examples**: \n",
      "  - Clustering (e.g., customer segmentation, grouping similar items)\n",
      "  - Dimensionality Reduction (e.g., Principal Component Analysis, t-SNE)\n",
      "  - Anomaly Detection (e.g., fraud detection)\n",
      "\n",
      "- **Common Algorithms**: \n",
      "  - K-Means Clustering\n",
      "  - Hierarchical Clustering\n",
      "  - DBSCAN (Density-Based Spatial Clustering of Applications with Noise)\n",
      "  - Autoencoders\n",
      "\n",
      "### 3. Semi-Supervised Learning\n",
      "Semi-supervised learning is a hybrid approach that combines both labeled and unlabeled data for training. It is particularly useful when obtaining a fully labeled dataset is expensive or time-consuming. The model leverages the labeled data to guide the learning process while also benefiting from the structure present in the unlabeled data.\n",
      "\n",
      "- **Examples**: \n",
      "  - Text classification where only a few documents are labeled\n",
      "  - Image recognition tasks with limited labeled images\n",
      "\n",
      "- **Common Algorithms**: \n",
      "  - Self-training\n",
      "  - Co-training\n",
      "  - Graph-based methods\n",
      "\n",
      "### 4. Reinforcement Learning\n",
      "Reinforcement learning (RL) is a type of ML where an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards or penalties based on its actions, allowing it to learn an optimal policy for maximizing cumulative rewards over time.\n",
      "\n",
      "- **Examples**: \n",
      "  - Game playing (e.g., AlphaGo)\n",
      "  - Robotics (e.g., robotic control systems)\n",
      "  - Autonomous vehicles\n",
      "\n",
      "- **Common Algorithms**: \n",
      "  - Q-Learning\n",
      "  - Deep Q-Networks (DQN)\n",
      "  - Proximal Policy Optimization (PPO)\n",
      "  - Actor-Critic methods\n",
      "\n",
      "### 5. Self-Supervised Learning\n",
      "Self-supervised learning is a technique where the model generates its own labels from the input data. This approach is often used in natural language processing and computer vision, where the model learns to predict missing parts of the input or to perform transformations on the input data.\n",
      "\n",
      "- **Examples**: \n",
      "  - Predicting the next word in a sentence (language models like GPT)\n",
      "  - Image inpainting where parts of an image are filled in\n",
      "\n",
      "- **Common Algorithms**: \n",
      "  - Contrastive Learning\n",
      "  - Masked Language Modeling\n",
      "\n",
      "### 6. Transfer Learning\n",
      "Transfer learning involves taking a pre-trained model (usually trained on a large dataset) and fine-tuning it on a smaller, specific dataset. This approach is particularly useful when the target domain has limited data, as it allows leveraging knowledge gained from a related task.\n",
      "\n",
      "- **Examples**: \n",
      "  - Using a model trained on ImageNet for a specific image classification task\n",
      "  - Fine-tuning a language model on domain-specific text\n",
      "\n",
      "- **Common Frameworks**: \n",
      "  - TensorFlow and PyTorch often provide pre-trained models for various tasks.\n",
      "\n",
      "### Conclusion\n",
      "Each of these approaches has its strengths and weaknesses, and the choice of which to use depends on the nature of the data, the specific problem being addressed, and the available resources. Many practical applications of machine learning may involve a combination of these approaches to achieve the best results.\n",
      "\n",
      "Relevance Score: 0.69\n",
      "Specificity Score: 0.52\n",
      "Prompt Comparison Results:\n",
      "\n",
      "1. Prompt: List the types of machine learning.\n",
      "   Relevance: 0.74\n",
      "   Specificity: 0.57\n",
      "\n",
      "2. Prompt: Explain the different approaches to machine learning.\n",
      "   Relevance: 0.69\n",
      "   Specificity: 0.52\n",
      "\n",
      "3. Prompt: What are the main categories of machine learning algorithms?\n",
      "   Relevance: 0.68\n",
      "   Specificity: 0.60\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'prompt': 'List the types of machine learning.',\n",
       "  'relevance': 0.73586243,\n",
       "  'specificity': 0.5693430656934306},\n",
       " {'prompt': 'Explain the different approaches to machine learning.',\n",
       "  'relevance': 0.68791693,\n",
       "  'specificity': 0.5223880597014925},\n",
       " {'prompt': 'What are the main categories of machine learning algorithms?',\n",
       "  'relevance': 0.67862606,\n",
       "  'specificity': 0.6039603960396039}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def compare_prompts(prompts, expected_content):\n",
    "    \"\"\"Compare the effectiveness of multiple prompts for the same task.\"\"\"\n",
    "    results = []\n",
    "    for prompt in prompts:\n",
    "        response = llm.invoke(prompt).content\n",
    "        evaluation = automated_evaluation(prompt, response, expected_content)\n",
    "        results.append({\"prompt\": prompt, **evaluation})\n",
    "    \n",
    "    # Sort results by relevance score\n",
    "    sorted_results = sorted(results, key=lambda x: x['relevance'], reverse=True)\n",
    "    \n",
    "    print(\"Prompt Comparison Results:\")\n",
    "    for i, result in enumerate(sorted_results, 1):\n",
    "        print(f\"\\n{i}. Prompt: {result['prompt']}\")\n",
    "        print(f\"   Relevance: {result['relevance']:.2f}\")\n",
    "        print(f\"   Specificity: {result['specificity']:.2f}\")\n",
    "    \n",
    "    return sorted_results\n",
    "\n",
    "# Example usage\n",
    "prompts = [\n",
    "    \"List the types of machine learning.\",\n",
    "    \"What are the main categories of machine learning algorithms?\",\n",
    "    \"Explain the different approaches to machine learning.\"\n",
    "]\n",
    "expected_content = \"The main types of machine learning are supervised learning, unsupervised learning, and reinforcement learning.\"\n",
    "compare_prompts(prompts, expected_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting It All Together\n",
    "\n",
    "Now, let's create a comprehensive prompt evaluation function that combines both manual and automated techniques:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Automated Evaluation:\n",
      "Prompt: Explain the concept of overfitting in machine learning.\n",
      "Response: Overfitting is a common problem in machine learning where a model learns not only the underlying patterns in the training data but also the noise and random fluctuations. This leads to a model that performs exceptionally well on the training dataset but poorly on unseen data or the test dataset. In essence, the model becomes overly complex, capturing details that do not generalize to new data points.\n",
      "\n",
      "### Key Aspects of Overfitting:\n",
      "\n",
      "1. **Complexity of the Model**: Overfitting often occurs when a model is too complex relative to the amount of training data available. For example, a high-degree polynomial regression may fit a small set of data points perfectly but will not generalize well to new data.\n",
      "\n",
      "2. **Training vs. Validation Performance**: A clear sign of overfitting is when the performance metrics (such as accuracy, loss, etc.) on the training data are significantly better than those on the validation or test data. This disparity indicates that the model is not learning the true underlying relationships but rather memorizing the training examples.\n",
      "\n",
      "3. **Noise**: Overfitted models may learn from noise in the training data, treating random variations as important signals, which can lead to poor predictive performance.\n",
      "\n",
      "### Visual Representation:\n",
      "When visualizing the performance of a model, overfitting can often be seen in a plot where the model fits the training data very closely (high accuracy on training data) but diverges significantly on validation data, leading to a U-shaped curve when plotting training and validation performance against model complexity.\n",
      "\n",
      "### Mitigation Strategies:\n",
      "Several techniques can help mitigate overfitting:\n",
      "\n",
      "1. **Regularization**: Techniques like L1 (Lasso) and L2 (Ridge) regularization add a penalty for larger coefficients in the model, discouraging overly complex models.\n",
      "\n",
      "2. **Cross-Validation**: Using k-fold cross-validation helps ensure that the model's performance is consistent across different subsets of the data.\n",
      "\n",
      "3. **Pruning**: In decision trees, pruning can be used to remove branches that have little importance, simplifying the model.\n",
      "\n",
      "4. **Early Stopping**: In iterative models like neural networks, training can be halted when performance on a validation set begins to degrade, preventing the model from fitting too closely to the training data.\n",
      "\n",
      "5. **Data Augmentation**: Increasing the size of the training dataset through data augmentation techniques can help the model generalize better.\n",
      "\n",
      "6. **Simplifying the Model**: Choosing a simpler model that captures the essential features of the data can reduce the risk of overfitting.\n",
      "\n",
      "### Conclusion:\n",
      "In summary, overfitting is a critical issue in machine learning that impacts a model's ability to generalize to new, unseen data. It is essential for practitioners to recognize the signs of overfitting and implement strategies to mitigate it, ensuring that the models they create are robust and reliable.\n",
      "\n",
      "Relevance Score: 0.82\n",
      "Specificity Score: 0.54\n",
      "\n",
      "Manual Evaluation:\n",
      "Prompt: Explain the concept of overfitting in machine learning.\n",
      "Response: Overfitting is a common problem in machine learning where a model learns not only the underlying patterns in the training data but also the noise and random fluctuations. This leads to a model that performs exceptionally well on the training dataset but poorly on unseen data or the test dataset. In essence, the model becomes overly complex, capturing details that do not generalize to new data points.\n",
      "\n",
      "### Key Aspects of Overfitting:\n",
      "\n",
      "1. **Complexity of the Model**: Overfitting often occurs when a model is too complex relative to the amount of training data available. For example, a high-degree polynomial regression may fit a small set of data points perfectly but will not generalize well to new data.\n",
      "\n",
      "2. **Training vs. Validation Performance**: A clear sign of overfitting is when the performance metrics (such as accuracy, loss, etc.) on the training data are significantly better than those on the validation or test data. This disparity indicates that the model is not learning the true underlying relationships but rather memorizing the training examples.\n",
      "\n",
      "3. **Noise**: Overfitted models may learn from noise in the training data, treating random variations as important signals, which can lead to poor predictive performance.\n",
      "\n",
      "### Visual Representation:\n",
      "When visualizing the performance of a model, overfitting can often be seen in a plot where the model fits the training data very closely (high accuracy on training data) but diverges significantly on validation data, leading to a U-shaped curve when plotting training and validation performance against model complexity.\n",
      "\n",
      "### Mitigation Strategies:\n",
      "Several techniques can help mitigate overfitting:\n",
      "\n",
      "1. **Regularization**: Techniques like L1 (Lasso) and L2 (Ridge) regularization add a penalty for larger coefficients in the model, discouraging overly complex models.\n",
      "\n",
      "2. **Cross-Validation**: Using k-fold cross-validation helps ensure that the model's performance is consistent across different subsets of the data.\n",
      "\n",
      "3. **Pruning**: In decision trees, pruning can be used to remove branches that have little importance, simplifying the model.\n",
      "\n",
      "4. **Early Stopping**: In iterative models like neural networks, training can be halted when performance on a validation set begins to degrade, preventing the model from fitting too closely to the training data.\n",
      "\n",
      "5. **Data Augmentation**: Increasing the size of the training dataset through data augmentation techniques can help the model generalize better.\n",
      "\n",
      "6. **Simplifying the Model**: Choosing a simpler model that captures the essential features of the data can reduce the risk of overfitting.\n",
      "\n",
      "### Conclusion:\n",
      "In summary, overfitting is a critical issue in machine learning that impacts a model's ability to generalize to new, unseen data. It is essential for practitioners to recognize the signs of overfitting and implement strategies to mitigate it, ensuring that the models they create are robust and reliable.\n",
      "\n",
      "Evaluation Criteria:\n",
      "Clarity: 6.0/10\n",
      "Accuracy: 7.0/10\n",
      "Relevance: 6.0/10\n",
      "\n",
      "Additional Comments:\n",
      "Comments: no\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'prompt': 'Explain the concept of overfitting in machine learning.',\n",
       " 'response': \"Overfitting is a common problem in machine learning where a model learns not only the underlying patterns in the training data but also the noise and random fluctuations. This leads to a model that performs exceptionally well on the training dataset but poorly on unseen data or the test dataset. In essence, the model becomes overly complex, capturing details that do not generalize to new data points.\\n\\n### Key Aspects of Overfitting:\\n\\n1. **Complexity of the Model**: Overfitting often occurs when a model is too complex relative to the amount of training data available. For example, a high-degree polynomial regression may fit a small set of data points perfectly but will not generalize well to new data.\\n\\n2. **Training vs. Validation Performance**: A clear sign of overfitting is when the performance metrics (such as accuracy, loss, etc.) on the training data are significantly better than those on the validation or test data. This disparity indicates that the model is not learning the true underlying relationships but rather memorizing the training examples.\\n\\n3. **Noise**: Overfitted models may learn from noise in the training data, treating random variations as important signals, which can lead to poor predictive performance.\\n\\n### Visual Representation:\\nWhen visualizing the performance of a model, overfitting can often be seen in a plot where the model fits the training data very closely (high accuracy on training data) but diverges significantly on validation data, leading to a U-shaped curve when plotting training and validation performance against model complexity.\\n\\n### Mitigation Strategies:\\nSeveral techniques can help mitigate overfitting:\\n\\n1. **Regularization**: Techniques like L1 (Lasso) and L2 (Ridge) regularization add a penalty for larger coefficients in the model, discouraging overly complex models.\\n\\n2. **Cross-Validation**: Using k-fold cross-validation helps ensure that the model's performance is consistent across different subsets of the data.\\n\\n3. **Pruning**: In decision trees, pruning can be used to remove branches that have little importance, simplifying the model.\\n\\n4. **Early Stopping**: In iterative models like neural networks, training can be halted when performance on a validation set begins to degrade, preventing the model from fitting too closely to the training data.\\n\\n5. **Data Augmentation**: Increasing the size of the training dataset through data augmentation techniques can help the model generalize better.\\n\\n6. **Simplifying the Model**: Choosing a simpler model that captures the essential features of the data can reduce the risk of overfitting.\\n\\n### Conclusion:\\nIn summary, overfitting is a critical issue in machine learning that impacts a model's ability to generalize to new, unseen data. It is essential for practitioners to recognize the signs of overfitting and implement strategies to mitigate it, ensuring that the models they create are robust and reliable.\",\n",
       " 'relevance': 0.82301676,\n",
       " 'specificity': 0.5372460496613995}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def evaluate_prompt(prompt, expected_content, manual_criteria=['Clarity', 'Accuracy', 'Relevance']):\n",
    "    \"\"\"Perform a comprehensive evaluation of a prompt using both manual and automated techniques.\"\"\"\n",
    "    response = llm.invoke(prompt).content\n",
    "    \n",
    "    print(\"Automated Evaluation:\")\n",
    "    auto_results = automated_evaluation(prompt, response, expected_content)\n",
    "    \n",
    "    print(\"\\nManual Evaluation:\")\n",
    "    manual_evaluation(prompt, response, manual_criteria)\n",
    "    \n",
    "    return {\"prompt\": prompt, \"response\": response, **auto_results}\n",
    "\n",
    "# Example usage\n",
    "prompt = \"Explain the concept of overfitting in machine learning.\"\n",
    "expected_content = \"Overfitting occurs when a model learns the training data too well, including its noise and fluctuations, leading to poor generalization on new, unseen data.\"\n",
    "evaluate_prompt(prompt, expected_content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}