08_retrieval_augemented_generation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNNwS5q8GJEmKMr//M4JNUW",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/badlogic/genai-workshop/blob/main/08_retrieval_augemented_generation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Retrieval-augmented generation\n",
        "LLMs are limited by the data they were trained on, potentially leading to outdated, generic, or contextually shallow outputs. Adding additional information to the LLM through full pre-training or fine tuning is costly, and can not be done frequently. For individuals and companies that want to get their domain specific knowledge into an LLM, pre-training is out of the question, and fine-tuning is ill-fit to teach the LLM new information. Even if this were possible, we'd still face the problem of model parameters encoding information lossily.\n",
        "\n",
        "**Retrieval-augmented generation (RAG)** addresses these issues by employing **grounding** (discussed in the last section) to provide the LLM with up-to-date, domain and user query specific information as part of the prompt.\n",
        "\n",
        "The basic architecture of a RAG system is deceptively simple:\n",
        "\n",
        "<center><img src=\"https://marioslab.io/uploads/genai/rag.png\"></center>\n",
        "\n",
        "Given a user query, the goal of a RAG system is it, to find relevant information from **domain specific content** stored in documents, databases, or accessible via APIs, and pass that information along with the user query to an LLM. The LLM then references the information in the prompt to answer the user query.\n",
        "\n",
        "From a birds eye view, a RAG system consists of two separate services running in parallel:\n",
        "\n",
        "1. **Index**: the domain specific content is ingested from various data sources, like databases, files, or APIs, pre-processed, and stored in one or more sparse and/or dense retrieval systems. This is a recurring process, to keep the indices up-to-data. The indexing service provides functionality to retrieve relevant information based on a query.\n",
        "\n",
        "2. **Query answering**: A user submits a query to the system. The following steps are executed:\n",
        "  1. **Retrieval of relevant information**: the query is used to retrieve relevant information from the indexing service.    \n",
        "  2. **Prompt construction**: The retrieved information, the conversation history, and the query are combined into a prompt, with instructions for the LLM on how to answer the query based on the retrieved information.\n",
        "  3. **Response**: the prompt is input into an LLM and the response is returned to the user and recorded for the next conversation turn.\n",
        "\n",
        "\n",
        "Let's have a look at the components and processes in a RAG.\n",
        "\n",
        "> **Note:** please execute the next code cells before you continue. They contain helper functions used by the code examples below.\n"
      ],
      "metadata": {
        "id": "zWFIzM1r7Tcz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip -q install openai tiktoken"
      ],
      "metadata": {
        "id": "ZS1581slTalJ"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "> **Note:** Enter your own OpenAI API key below!"
      ],
      "metadata": {
        "id": "iu0HN4A-H-0a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from openai import OpenAI\n",
        "import tiktoken\n",
        "\n",
        "# Use your own OpenAI API key here.\n",
        "client = OpenAI(api_key = \"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\")\n",
        "\n",
        "messages = []\n",
        "model_name=\"gpt-3.5-turbo\"\n",
        "max_tokens = 12000\n",
        "temperature=0\n",
        "\n",
        "# Uncomment to use a model served locally via `ollama serve`\n",
        "# client = OpenAI(\n",
        "#    base_url = 'http://localhost:11434/v1',\n",
        "#    api_key='ollama', # required, but unused\n",
        "# )\n",
        "# model_name=\"mixtral:latest\"\n",
        "\n",
        "enc = tiktoken.get_encoding(\"cl100k_base\")\n",
        "def num_tokens(message):\n",
        "    return len(enc.encode(message))\n",
        "\n",
        "def truncate_messages(messages, max_tokens):\n",
        "    total_tokens = sum(num_tokens(message[\"content\"]) for message in messages)\n",
        "    if total_tokens <= max_tokens:\n",
        "        return messages\n",
        "\n",
        "    truncated_messages = messages[:1]\n",
        "    remaining_tokens = max_tokens - num_tokens(truncated_messages[0][\"content\"])\n",
        "    for message in reversed(messages[1:]):\n",
        "        tokens = num_tokens(message[\"content\"])\n",
        "        if remaining_tokens >= tokens:\n",
        "            truncated_messages.insert(1, message)\n",
        "            remaining_tokens -= tokens\n",
        "        else:\n",
        "            break\n",
        "    return truncated_messages\n",
        "\n",
        "def complete(message, max_response_tokens=2048):\n",
        "    global messages\n",
        "    messages.append({\"role\": \"user\", \"content\": message})\n",
        "    truncated_messages = truncate_messages(messages, max_tokens=max_tokens)\n",
        "    stream = client.chat.completions.create(\n",
        "        model=model_name,\n",
        "        messages=truncated_messages,\n",
        "        stream=True,\n",
        "        temperature=temperature,\n",
        "        max_tokens=max_response_tokens\n",
        "    )\n",
        "    reply = \"\"\n",
        "    for response in stream:\n",
        "        token = response.choices[0].delta.content\n",
        "        if (token is None):\n",
        "            break\n",
        "        reply += token\n",
        "        print(token, end='')\n",
        "\n",
        "    reply = {\"role\": \"assistant\", \"content\": reply}\n",
        "    messages.append(reply)\n",
        "    total_tokens = sum(num_tokens(message[\"content\"]) for message in truncated_messages)\n",
        "    print(f'\\nTokens: {total_tokens}')\n",
        "\n",
        "def clear_history():\n",
        "  global messages\n",
        "  messages = [];\n",
        "\n",
        "def print_history():\n",
        "  global messages\n",
        "  for message in messages:\n",
        "    print(\"<\" + message[\"role\"] + \">\")\n",
        "    print(message[\"content\"])\n",
        "    print()\n",
        "\n",
        "def system_prompt(message):\n",
        "  global messages\n",
        "  prompt = { \"role\": \"system\", \"content\": message }\n",
        "  if (len(messages) == 0):\n",
        "    messages.append(prompt)\n",
        "  else:\n",
        "    messages[0] = prompt"
      ],
      "metadata": {
        "id": "V8Du_kxXTdkA"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Index\n",
        "The index stores domain specific content, usually just referred to as **documents**. It allows **retrieval of the most relevant documents** for given a user query. A full RAG system can have more than one index.\n",
        "\n",
        "The process of storing documents in the index is called **ingestion**, or **indexing**. Indexing should happen frequently, or whenever information is added or updated, so we can provide up-to-date information to the LLM.\n",
        "\n",
        "Since we will stuff domain specific information into the prompt passed to the LLM, we need to ensure that this information fits into the context window. We thus don't just index and retrieve full documents, but document **chunks**.\n",
        "\n",
        "A chunk is a part of a document, which is big enough to contribute information for question answering by the LLM, but not so big, that it can't fit into the LLMs token window along-side the conversation history and user question.\n",
        "\n",
        "Cutting a document up into these chunks is called **chunking**, for which various strategies exist, e.g.\n",
        "\n",
        "* **Character or token based chunking**: the content is split into equally sized chunks of `n` characters or tokens each. Chunks may also overlap. A basic evaluation of the effect of different chunk sizes can be found [here](https://blog.llamaindex.ai/evaluating-the-ideal-chunk-size-for-a-rag-system-using-llamaindex-6207e5d3fec5)\n",
        "* **Semantic chunking**: the content is split into sentences. Each sentence is embedded as a vector. The semantic chunker then accumulates sentences until either a maximum chunk size is reaached, or the similarity between the current set of sentences with the next sentence is smaller than a threshold. See [semantic chunking](https://docs.llamaindex.ai/en/stable/examples/node_parsers/semantic_chunking.html) in the LlamaIndex documentation for more information.\n",
        "* **Structured chunking**: the content is often structured into sections and paragraphs, e.g. if encoded as HTML or markdown, which imply semantic relatedness. Structured chunking splits content into chunks based on this structure.\n",
        "\n",
        "Each document chunk is stored in the index separately, usually including metadata that stores information like where the chunk came from. When we retrieve relevant content for a query, we actually retrieve chunks, not the full document, since we will stuff chunks into the LLM prompt.\n",
        "\n",
        "Indices are usually **sparse or dense retrieval systems**, which we've already investigated in the section on embeddings. [Elastic Seach](https://www.elastic.co/elasticsearch) is a popular sparse retrieval system. For dense retrieval systems like [Chroma](https://www.trychroma.com/), [Pinecone](https://www.pinecone.io/), or [Milvus](https://milvus.io/) are popular.\n",
        "\n",
        "While sparse retrieval systems usually can be directly fed each chunks text, dense retrieval systems require an additional pre-procesing step in form of embedding the chunk texts to a latent vector space. We've already seen how to embed text into a latent vector space using an embedding model. Another popular choice is to use [OpenAI's embedding API](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings). The [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) on Hugging Face gives a good overview of embedding models and their relative performance.\n",
        "\n",
        "When we retrieve relevant documents for a user query from a dense retrieval system, we also need to **embed the user query using the same embedding model**!"
      ],
      "metadata": {
        "id": "RzrgkDWAI2nF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Retrieval of relevant information\n",
        "This step is usually encapsulate in a system component called **the retriever** sits on top of the indices. It is responsible for preprocessing the user query, using it to retrieve relevant chunks from each index, and ranking and selecting the top-k chunks to be passed along in the LLM prompt.\n",
        "\n",
        "One crucial pre-processing step found in most retrievers is **query expansion**. This takes the user query, and augments it with additional information, so it is more likely we can find relevant information. This is especially helpful in turn-by-turn conversations. Consider this example:\n",
        "\n",
        "```\n",
        "<user>\n",
        "Who is the CEO of the company ETM. Also tell me what products to provide.\n",
        "\n",
        "<assistant>\n",
        "The CEO of ETM is Dipl.-Ing. Dr.techn. Bernhard Reichl. ETM develops the SCADA system SIMATIC WinCC Open Architecture, which is designed for applications requiring a high degree of client-specific adaptability, large and/or complex applications, and projects with specific system requirements and functions.\n",
        "\n",
        "<user>\n",
        "What is his age?\n",
        "```\n",
        "\n",
        "The first user query contains enough information to retrieve meaningful information from the index.\n",
        "\n",
        "However, the second user question is problematic. We will not get any relevant chunks for the query `What is his age?`, as it lacks contextual information, such as who \"he\" is.\n",
        "\n",
        "We can use an LLM to expand this query based on the conversation history.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "JeywdqX4PLaK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clear_history()\n",
        "complete(\"\"\"You are given a conversation and new message, both delimited by triple backticks.\n",
        "Expand the new message by resolving and references to persons, entities or locations, in the\n",
        "conversation with their full name.\n",
        "\n",
        "Cconversation:\n",
        "```\n",
        "<user>\n",
        "Who is the CEO of the company ETM. Also tell me what products to provide.\n",
        "\n",
        "<assistant>\n",
        "The CEO of ETM is Dipl.-Ing. Dr.techn. Bernhard Reichl. ETM develops the SCADA system SIMATIC WinCC Open Architecture, which is designed for applications requiring a high degree of client-specific adaptability, large and/or complex applications, and projects with specific system requirements and functions.\n",
        "```\n",
        "\n",
        "New Message:\n",
        "```\n",
        "What is his age?\n",
        "```\n",
        "\"\"\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "echxO-8kTwxO",
        "outputId": "ee0e0f24-57b6-44ce-b4ce-8fde35259fbe"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "What is Dipl.-Ing. Dr.techn. Bernhard Reichl's age?\n",
            "Tokens: 163\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This expanded query is much more likely to return relevant chunks from the indices! The expanded query can be used to improve retrieval performance for both sparse and dense indices.\n",
        "\n",
        "Once we've retrieved all relevant chunks from the indices, we need to select the ones we will pass along in the prompt to the LLM.\n",
        "\n",
        "Usually, sparse and dense retrieval systems will already sort their results such that the most similar chunks come first. A simple approach is thus to just select the top-k chunks that fit into the LLM token window, along with the new user query, the conversation history, and the system prompt that instructs the LLM how to answer the query.\n",
        "\n",
        "However, a problem called **[Lost in the middle](https://arxiv.org/abs/2307.03172)** might sometimes make this not the best option. Empirical analysis has shown, that many LLMs focus on the beginning and end of a prompt more than on the mid-section. This means that relevant chunks in the middle of the prompt, which may perfectly answer the user query, may not get enough attention. The solution: take the chunks in the middle, and swap them with chunks at the end.\n",
        "\n",
        "Not all ranking problems are this simple. If we **retrieve relevant chunks from multiple indices**, we get one list of chunks per index. Each list is sorted according to the similarity measure used by the respective index, so similarities between lists are not comparable.\n",
        "\n",
        "In order to pick the top-k relevant chunks across multiple, separately scored lists of chunks, we need to emply **model-based reranking**: the chunk lists are combined into a single list. This combined list and the user query are then presented to a reranking model, which sorts the chunks in the list according to their relevance to the user query. We can then pick the top-k chunks from this reranked list. Services like [Cohere Rerank](https://txt.cohere.com/rerank/) can provide this functionality in a single line of code.\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "zBMPShVWTyio"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt construction\n",
        "At this stage, we have the list of top-k relevant chunks, the user query, and  conversation history. These need to be combined into a prompt along with instructions for the LLM how to answer the user query.\n",
        "\n",
        "We can use **grounding** prompt engineering techniques to construct the prompt from these pieces of information. Here is an example prompt:\n",
        "\n",
        "```\n",
        "You are provided with a conversation history, a set of relevant information, and a user query.\n",
        "\n",
        "The conversation history:\n",
        "\"\"\"\n",
        "<user>\n",
        "...\n",
        "<assistant>\n",
        "...\n",
        "<user>\n",
        "...\n",
        "<assistant>\n",
        "...\n",
        "\"\"\"\n",
        "\n",
        "The relevant information:\n",
        "\"\"\"\n",
        "[\n",
        "  {\n",
        "    \"url\": \"file://documents/do...\",\n",
        "    \"information\": \"...\"\n",
        "  },\n",
        "  {\n",
        "  \"url\": \"http://domain.com/xyz.html\",\n",
        "  \"information\": \"...\"\n",
        "  },\n",
        "]\n",
        "\"\"\"\n",
        "\n",
        "The query:\n",
        "\"\"\"\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "Answer the query based on the relevant information and conversation history.\n",
        "Output your answer in Markdown.\n",
        "Cite the relevant information by adding Markdown links where appropriate.\n",
        "```\n",
        "\n",
        "Ideally, the LLM will follow the instructions and produce an answer in Markdown format, including links to the documents that informed its answer where appropriate.\n",
        "\n",
        "This technique thus not only allows us to ground the LLM in our own data, but also gives the user citations to explore the information in more depth.\n",
        "\n",
        "Of course, we need to ensure that the prompt fits into the context window of the LLM. We can use a truncation strategy like we've implemented in the last section, to truncate both the conversation history, as well as the contextual information, until the prompt is small enough.\n",
        "\n",
        "Ideally, we provide the LLM with as much information as we can, so in general, we'll use most of the token window. However, this comes at a price: both financially (if you use paid LLM services) as well as in terms of response times."
      ],
      "metadata": {
        "id": "5AvJUU01dfHS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Response\n",
        "Now that we have the prompt, we can simply input it into the LLM to get a hopefully correct, and ground response, including citations.\n",
        "\n",
        "The user query as well as the LLM's answer are added to the conversation history and the system is ready for the next conversation turn.*italicized text*"
      ],
      "metadata": {
        "id": "64oWj759gVj6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluation\n",
        "RAG systems can be very sensitive to the smallest changes. E.g. making small changes to instructions in or format of the prompt can have a huge impact on the answer quality. Similarly, changing the chunking strategy, or using a diffeernt embedding strategy will change which chunks will be retrieved for a query, and will also influence the order of the retrieved chunks.\n",
        "\n",
        "As such, it is advisable to set up a continuous evaluation pipeline, with which we can monitor the impact of changes to the RAG system along two dimensions:\n",
        "\n",
        "* **Retriever evaluation**: given a query, and a query expansion reranker mechanism, does the retriever return:\n",
        "  * the most **relevant** information?  \n",
        "* **Response evaluation**: given a query, a set of relevant information, and the instructions, does the LLM respond\n",
        "  * **faithfully** based only on the provided information or did it hallucinate?\n",
        "  * **relevant** to the query?\n",
        "  * **correctly**\n",
        "\n",
        "[LlamaIndex](https://docs.llamaindex.ai/en/stable/module_guides/evaluating/root.html), a framework for building production RAG system, has a comprehensive evaluation suite integrated, with which the above described evaluation metrics can be evaluated.\n",
        "\n",
        "Notably, RAG evaluation systems as implemented in LlamaIndex often do not require human labels, such as query/response pairs, but rely on \"gold\" LLMs to generate test data, and judge metrics like faithfulness or correctness."
      ],
      "metadata": {
        "id": "h9C2_vk0jpyi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## WinCC OA JS Coding Buddy - RAG without retrieval\n",
        "A full-blown RAG system is only needed if the data we want the LLM to use as grounding does not fit into the token window.\n",
        "\n",
        "Before we build a simple RAG system, let us see how far we can get if we can indeed fit our grounding data entirely into the prompt. We are still augmenting the answer generation, but we are not retrieving anything.\n",
        "\n",
        "ETM provides a simple JavaScript API for their WinCC OA system. We want to create a coding assistant that knows this API and can help users write applications that use this API.\n"
      ],
      "metadata": {
        "id": "3WD2fLMooyEb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data\n",
        "We'll use the [WinCC OA JS API documentation](https://www.winccoa.com/documentation/WinCCOA/3.18/en_US/oaJsApi/oaJsApi.html) as our data source. It is a single HTML file, which contains the full source code of the API, including JSDoc strings.\n",
        "\n",
        "We are not interested in the actual implementation, but only in the JSDoc strings, which document the entire API surface and includes examples for each function. Perfect!\n",
        "\n",
        "Let's write a function that preprocesses this HTML file by extracting the JSDoc strings."
      ],
      "metadata": {
        "id": "TTPOeuhBpmot"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "import os\n",
        "import re\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "def download_api_docs():\n",
        "    url = \"https://www.winccoa.com/documentation/WinCCOA/3.18/en_US/oaJsApi/oaJsApi.class.js.html\"\n",
        "    response = requests.get(url)\n",
        "    html_content = response.text\n",
        "    soup = BeautifulSoup(html_content)\n",
        "    code_element = soup.find('code')\n",
        "    if code_element:\n",
        "        code_text = code_element.text\n",
        "    else:\n",
        "        return \"No <code> tag found.\"\n",
        "\n",
        "    pattern = r'/\\*\\*(?:.|\\n)*?\\*/'\n",
        "    matches = re.findall(pattern, code_text)\n",
        "    api = \"\"\n",
        "    for match in matches:\n",
        "        api += match + \"\\n\"\n",
        "\n",
        "    return api\n",
        "\n",
        "api_docs = download_api_docs()\n",
        "print(api_docs[:1000])"
      ],
      "metadata": {
        "id": "Rh9s--AEqQio",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "38e8f3fe-e55c-40e6-83be-9d1049c671e9"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/**\n",
            " *\n",
            " * @class oaJsApi\n",
            " */\n",
            "/**\n",
            "   *\n",
            "   *  This Callback will be fired in case of an exception. If no errorCallback is registered, oaJsApi will console.error to console.\n",
            "   *  Arguments can be different, depends who calls the error handler. Could be catch block of javascript or WinCCOA.\n",
            "   * @callback errorCallback\n",
            "\n",
            "   * @example\n",
            "   *\n",
            "   * oaJsApi.registerListeners({\n",
            "   *  error: function()\n",
            "   *  {\n",
            "   *    console.error(arguments);\n",
            "   *  }\n",
            "   * });\n",
            "   */\n",
            "/**\n",
            "   * Opens a WebSock Object to the given baseUrl.\n",
            "   * @function oaJsApi.connect\n",
            "\n",
            "   * @param {String} baseUrl\n",
            "   * @param {Object=} options Configuration Options\n",
            "   * @param {Object=} options.webViewId WebView ID\n",
            "   * @param {Object=} options.listeners register global listeners\n",
            "   * @param {requestCallback=} options.listeners.success success Callback of connect\n",
            "   * @private\n",
            "   * @example\n",
            "   *\n",
            "   *  oaJsApi.connect(\"wss://localhost:12345\", {\n",
            "   *    baseParams: {\n",
            "   *       webViewId: 1\n",
            "   *    },\n",
            "   *    listeners: {\n",
            "   *       \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Looking good! We can use the `num_tokens` function to count the tokens we will use up when passing the entire API documentation as part of the prompt:"
      ],
      "metadata": {
        "id": "hJw9vaW9qVju"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "num_tokens(api_docs)"
      ],
      "metadata": {
        "id": "3vQsiEFnqe_1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "00d3975b-0312-49e6-cc1f-4a65152ba527"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "7293"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use OpenAI's `gpt-3.5-turbo` model to answer coding related questions. Each request will have at least 7293 tokens. What will that cost us?\n",
        "\n",
        "Looking at the [pricing page](https://openai.com/pricing), 1000 tokens passed to `gpt-3.5-turbo` cost $0.0005"
      ],
      "metadata": {
        "id": "gJPtyMo4qixb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(f'Minimum cost per query: ${0.0005 * num_tokens(api_docs) / 1000: 3f}')"
      ],
      "metadata": {
        "id": "85d6gPM2rJed",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eaa1ab90-94e7-4aff-9f33-2f472cf37c8b"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Minimum cost per query: $ 0.003647\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This cost doesn't include the additional tokens we need to send the conversation history, instructions, and latest query to the model. It also doesn't include the cost incurred by the response. But it gives us a good ballpark figure to work with.\n",
        "\n",
        "If we used a RAG instead, we would try to fill the token window as well, so the cost compared to a full RAG would likely be similar."
      ],
      "metadata": {
        "id": "9x9zzddAra6w"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Prompt construction\n",
        "Let's create our system prompt."
      ],
      "metadata": {
        "id": "PT3s0sAStMPf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "system_prompt(f\"\"\"\n",
        "You are a helpful assistant. You know about the WinCC OA JS API and can answer questions related to it.\n",
        "\n",
        "Here is the API documentation:\n",
        "\n",
        "```\n",
        "{api_docs}\n",
        "```\n",
        "\n",
        "Only answer questions related to the API above and its usage.\n",
        "\"\"\")"
      ],
      "metadata": {
        "id": "WeA1ACugrv0U"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We just smack the the entire API documentation right in the middle of the system prompt. Let's give it a try."
      ],
      "metadata": {
        "id": "WGdy2FKvsApl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Trying it out"
      ],
      "metadata": {
        "id": "U9u0A09QtaD3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"Who are you?\")"
      ],
      "metadata": {
        "id": "TYmS2IGmrvx2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "141a49dd-4060-499d-e984-3ba993e4b462"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "I am a helpful assistant knowledgeable about the WinCC OA JS API. I can provide information and answer questions related to the API and its usage.\n",
            "Tokens: 7389\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"How can I enumerate all datapoints?\")"
      ],
      "metadata": {
        "id": "TS29Tbs5rvVv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b202017d-b3ba-43e1-fcdb-124c987d847c"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To enumerate all data points, you can use the `dpNames` function provided by the WinCC OA JS API. Here is an example of how you can use it:\n",
            "\n",
            "```javascript\n",
            "oaJsApi.dpNames('*', null, {\n",
            "   success: function(data) {\n",
            "      console.log(data);\n",
            "   },\n",
            "   error: function() {\n",
            "      console.error(arguments);\n",
            "   }\n",
            "});\n",
            "```\n",
            "\n",
            "In this example, the `dpNames` function is called with the pattern `'*'` to match all data points. The `null` parameter is used to retrieve all data point types. The success callback function will log the data points to the console.\n",
            "Tokens: 7526\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"I want to list all datapoints, their description, and the historic values for the last 7 days\")"
      ],
      "metadata": {
        "id": "hDS4QyEJsckH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b2439bee-34ad-44f2-e492-a6f8f6575762"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To achieve this, you can use multiple functions provided by the WinCC OA JS API. You can first use the `dpNames` function to get a list of all data points, then for each data point, you can use the `dpGetDescription` function to get its description, and finally, you can use the `dpGetPeriod` function to retrieve historic values for the last 7 days.\n",
            "\n",
            "Here is an example of how you can accomplish this:\n",
            "\n",
            "```javascript\n",
            "// Step 1: Get a list of all data points\n",
            "oaJsApi.dpNames('*', null, {\n",
            "   success: function(dataPoints) {\n",
            "      // Step 2: For each data point, get its description and historic values for the last 7 days\n",
            "      dataPoints.forEach(function(dp) {\n",
            "         // Get description of the data point\n",
            "         oaJsApi.dpGetDescription(dp, null, {\n",
            "            success: function(description) {\n",
            "               console.log('Data Point: ' + dp);\n",
            "               console.log('Description: ' + description);\n",
            "\n",
            "               // Get historic values for the last 7 days\n",
            "               let endTime = new Date(); // Current time\n",
            "               let startTime = new Date(endTime.getTime() - 7 * 24 * 60 * 60 * 1000); // 7 days ago\n",
            "\n",
            "               oaJsApi.dpGetPeriod(startTime, endTime, 7, dp, {\n",
            "                  success: function(historicValues) {\n",
            "                     console.log('Historic Values for the last 7 days: ' + historicValues);\n",
            "                  },\n",
            "                  error: function() {\n",
            "                     console.error(arguments);\n",
            "                  }\n",
            "               });\n",
            "            },\n",
            "            error: function() {\n",
            "               console.error(arguments);\n",
            "            }\n",
            "         });\n",
            "      });\n",
            "   },\n",
            "   error: function() {\n",
            "      console.error(arguments);\n",
            "   }\n",
            "});\n",
            "```\n",
            "\n",
            "In this example, we first retrieve a list of all data points using `dpNames`, then for each data point, we get its description using `dpGetDescription`, and finally, we retrieve the historic values for the last 7 days using `dpGetPeriod`.\n",
            "Tokens: 7964\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"Can you rewrite this in async/await style?\")"
      ],
      "metadata": {
        "id": "1-Oj25ccsjtr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b1f834c2-55d0-4f14-edd1-e14f18329722"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Certainly! Here is the rewritten code using async/await syntax:\n",
            "\n",
            "```javascript\n",
            "async function getDataPointsInfo() {\n",
            "   try {\n",
            "      // Step 1: Get a list of all data points\n",
            "      const dataPoints = await new Promise((resolve, reject) => {\n",
            "         oaJsApi.dpNames('*', null, {\n",
            "            success: resolve,\n",
            "            error: reject\n",
            "         });\n",
            "      });\n",
            "\n",
            "      // Step 2: For each data point, get its description and historic values for the last 7 days\n",
            "      for (const dp of dataPoints) {\n",
            "         // Get description of the data point\n",
            "         const description = await new Promise((resolve, reject) => {\n",
            "            oaJsApi.dpGetDescription(dp, null, {\n",
            "               success: resolve,\n",
            "               error: reject\n",
            "            });\n",
            "         });\n",
            "\n",
            "         console.log('Data Point: ' + dp);\n",
            "         console.log('Description: ' + description);\n",
            "\n",
            "         // Get historic values for the last 7 days\n",
            "         const endTime = new Date(); // Current time\n",
            "         const startTime = new Date(endTime.getTime() - 7 * 24 * 60 * 60 * 1000); // 7 days ago\n",
            "\n",
            "         const historicValues = await new Promise((resolve, reject) => {\n",
            "            oaJsApi.dpGetPeriod(startTime, endTime, 7, dp, {\n",
            "               success: resolve,\n",
            "               error: reject\n",
            "            });\n",
            "         });\n",
            "\n",
            "         console.log('Historic Values for the last 7 days: ' + historicValues);\n",
            "      }\n",
            "   } catch (error) {\n",
            "      console.error(error);\n",
            "   }\n",
            "}\n",
            "\n",
            "// Call the async function to retrieve data points information\n",
            "getDataPointsInfo();\n",
            "```\n",
            "\n",
            "In this rewritten code, the `getDataPointsInfo` function uses async/await syntax to handle asynchronous operations. It first retrieves the list of data points, then for each data point, it gets the description and historic values using async/await syntax.\n",
            "Tokens: 8362\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"hm shouldn't you wrap the existing api, then use async/Await?\")"
      ],
      "metadata": {
        "id": "moND-1bEsoLy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e98280a8-29f6-4b4f-ba56-b317aa0753bd"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Yes, you are correct. To use async/await with the existing API functions that follow a callback-based pattern, you can wrap those functions in Promises. Here is the updated code with the API functions wrapped in Promises for use with async/await:\n",
            "\n",
            "```javascript\n",
            "function dpNamesAsync() {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpNames('*', null, {\n",
            "         success: resolve,\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "function dpGetDescriptionAsync(dp) {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpGetDescription(dp, null, {\n",
            "         success: resolve,\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "function dpGetPeriodAsync(startTime, endTime, dp) {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpGetPeriod(startTime, endTime, 7, dp, {\n",
            "         success: resolve,\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "async function getDataPointsInfo() {\n",
            "   try {\n",
            "      const dataPoints = await dpNamesAsync();\n",
            "\n",
            "      for (const dp of dataPoints) {\n",
            "         const description = await dpGetDescriptionAsync(dp);\n",
            "\n",
            "         console.log('Data Point: ' + dp);\n",
            "         console.log('Description: ' + description);\n",
            "\n",
            "         const endTime = new Date(); // Current time\n",
            "         const startTime = new Date(endTime.getTime() - 7 * 24 * 60 * 60 * 1000); // 7 days ago\n",
            "\n",
            "         const historicValues = await dpGetPeriodAsync(startTime, endTime, dp);\n",
            "\n",
            "         console.log('Historic Values for the last 7 days: ' + historicValues);\n",
            "      }\n",
            "   } catch (error) {\n",
            "      console.error(error);\n",
            "   }\n",
            "}\n",
            "\n",
            "// Call the async function to retrieve data points information\n",
            "getDataPointsInfo();\n",
            "```\n",
            "\n",
            "In this updated code, the API functions `dpNames`, `dpGetDescription`, and `dpGetPeriod` are wrapped in Promises to enable the use of async/await syntax for handling asynchronous operations.\n",
            "Tokens: 8792\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"Can you rewrite this in TypeScript?\")"
      ],
      "metadata": {
        "id": "HzKfCkcvssea",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1cb17aa1-46c9-4cb4-e3bb-10dc2cff9cee"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Certainly! Here is the rewritten code using TypeScript:\n",
            "\n",
            "```typescript\n",
            "function dpNamesAsync(): Promise<string[]> {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpNames('*', null, {\n",
            "         success: (data: string[]) => resolve(data),\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "function dpGetDescriptionAsync(dp: string): Promise<string> {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpGetDescription(dp, null, {\n",
            "         success: (description: string) => resolve(description),\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "function dpGetPeriodAsync(startTime: Date, endTime: Date, dp: string): Promise<number[]> {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpGetPeriod(startTime, endTime, 7, dp, {\n",
            "         success: (historicValues: number[]) => resolve(historicValues),\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "async function getDataPointsInfo() {\n",
            "   try {\n",
            "      const dataPoints: string[] = await dpNamesAsync();\n",
            "\n",
            "      for (const dp of dataPoints) {\n",
            "         const description: string = await dpGetDescriptionAsync(dp);\n",
            "\n",
            "         console.log('Data Point: ' + dp);\n",
            "         console.log('Description: ' + description);\n",
            "\n",
            "         const endTime: Date = new Date(); // Current time\n",
            "         const startTime: Date = new Date(endTime.getTime() - 7 * 24 * 60 * 60 * 1000); // 7 days ago\n",
            "\n",
            "         const historicValues: number[] = await dpGetPeriodAsync(startTime, endTime, dp);\n",
            "\n",
            "         console.log('Historic Values for the last 7 days: ' + historicValues);\n",
            "      }\n",
            "   } catch (error) {\n",
            "      console.error(error);\n",
            "   }\n",
            "}\n",
            "\n",
            "// Call the async function to retrieve data points information\n",
            "getDataPointsInfo();\n",
            "```\n",
            "\n",
            "In this TypeScript version of the code, the functions `dpNamesAsync`, `dpGetDescriptionAsync`, and `dpGetPeriodAsync` are defined with explicit types for parameters and return values. The async/await syntax is used to handle asynchronous operations in a more readable and structured manner.\n",
            "Tokens: 9242\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"Please remove all comments\")"
      ],
      "metadata": {
        "id": "vAJQfUcvsxTD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "87b14458-4743-4b69-cde0-b3237fae83b4"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Here is the TypeScript code without comments:\n",
            "\n",
            "```typescript\n",
            "function dpNamesAsync(): Promise<string[]> {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpNames('*', null, {\n",
            "         success: (data: string[]) => resolve(data),\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "function dpGetDescriptionAsync(dp: string): Promise<string> {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpGetDescription(dp, null, {\n",
            "         success: (description: string) => resolve(description),\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "function dpGetPeriodAsync(startTime: Date, endTime: Date, dp: string): Promise<number[]> {\n",
            "   return new Promise((resolve, reject) => {\n",
            "      oaJsApi.dpGetPeriod(startTime, endTime, 7, dp, {\n",
            "         success: (historicValues: number[]) => resolve(historicValues),\n",
            "         error: reject\n",
            "      });\n",
            "   });\n",
            "}\n",
            "\n",
            "async function getDataPointsInfo() {\n",
            "   try {\n",
            "      const dataPoints: string[] = await dpNamesAsync();\n",
            "\n",
            "      for (const dp of dataPoints) {\n",
            "         const description: string = await dpGetDescriptionAsync(dp);\n",
            "\n",
            "         console.log('Data Point: ' + dp);\n",
            "         console.log('Description: ' + description);\n",
            "\n",
            "         const endTime: Date = new Date(); // Current time\n",
            "         const startTime: Date = new Date(endTime.getTime() - 7 * 24 * 60 * 60 * 1000); // 7 days ago\n",
            "\n",
            "         const historicValues: number[] = await dpGetPeriodAsync(startTime, endTime, dp);\n",
            "\n",
            "         console.log('Historic Values for the last 7 days: ' + historicValues);\n",
            "      }\n",
            "   } catch (error) {\n",
            "      console.error(error);\n",
            "   }\n",
            "}\n",
            "\n",
            "getDataPointsInfo();\n",
            "```\n",
            "Tokens: 9617\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "complete(\"What other APIs exist that you haven't shown me yet?\")"
      ],
      "metadata": {
        "id": "Tq6xBXPcs34h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c47e9f54-0fae-4677-d483-44f6dd164c70"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Here are some additional APIs from the WinCC OA JS API documentation that have not been covered yet:\n",
            "\n",
            "1. `toCtrl`: Triggers the messageReceived event on the WebView Ewo.\n",
            "2. `toCtrlFn`: Triggers an existing Control-Function.\n",
            "3. `msgToWebViewEwo`: Forwards the given params to another WebView Ewo.\n",
            "4. `dpConnect`: Calls a callback function whenever the passed data point values/attributes change.\n",
            "5. `dpDisconnect`: Disconnects the hotlink.\n",
            "6. `dpSet`: Assigns values to data point attributes.\n",
            "7. `dpGet`: Reads values of data point attributes.\n",
            "8. `setValue`: Sets any number of graphics attributes of a graphics object.\n",
            "9. `getValue`: Reads graphics attribute values for a graphics object in variables.\n",
            "10. `dpGetAsynch`: Returns the historic values that were valid at a specific time.\n",
            "11. `customFunction`: Allows calling a self-implemented method in the WssUserRequestHandler.\n",
            "12. `cnsGetCompletions`: Looks up a pattern in CNS identifiers and IDs.\n",
            "13. `dpQuery`: Retrieves attribute values with the help of SQL statements.\n",
            "14. `dpGetAlias`: Get aliases of data point elements.\n",
            "15. `dpGetDescription`: Get descriptions of data point elements.\n",
            "16. `dpGetUnit`: Get units of data point elements.\n",
            "17. `dpGetFormat`: Get output format of data point elements.\n",
            "18. `setLocale`: Set locale to be used for returning langStrings.\n",
            "19. `registerListeners`: Register a listener.\n",
            "\n",
            "These APIs provide various functionalities for interacting with data points, graphics objects, and other elements in the WinCC OA system.\n",
            "Tokens: 9977\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This looks pretty good!"
      ],
      "metadata": {
        "id": "o0b1zJELtCIo"
      }
    }
  ]
}