Integrations/Unstructured/Dropbox_Clarifai_Unstructured_integration_example.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4NmDOxAHzIxx"
      },
      "source": [
        "# **Clarifai as destination connector using Unstructured.IO**\n",
        "\n",
        "This Notebook walk you through the step by step guide on how to utilise Clarifai as your destination connector using Unstructured.IO and import raw files from various source connectors. For the demo we will be using Dropbox as our source connector which has raw text files."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8b-x7tHrABfw"
      },
      "source": [
        "### Setup\n",
        "Install necessary libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m8bVpqw63PdF"
      },
      "outputs": [],
      "source": [
        "! pip install \"unstructured[clarifai]\" #make sure the unstructured version is 0.13 or above"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install \"unstructured[dropbox]\"\n",
        "!pip install httpx"
      ],
      "metadata": {
        "id": "pHVZfmF8NR2r"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Px-O0dMNAEHF"
      },
      "source": [
        "## Dropbox access token"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jVxZI-5EuCYI"
      },
      "outputs": [],
      "source": [
        "DROPBOX_ACCESS_TOKEN=\"sl*********************************************************\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QCPbc8KFAGr1"
      },
      "source": [
        "## Clarifai and dropbox imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t8G_ONxWuLDN"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from unstructured.ingest.connector.fsspec.dropbox import DropboxAccessConfig, SimpleDropboxConfig\n",
        "from unstructured.ingest.interfaces import (\n",
        "    PartitionConfig,\n",
        "    ProcessorConfig,\n",
        "    ReadConfig,\n",
        ")\n",
        "from unstructured.ingest.runner import DropboxRunner\n",
        "\n",
        "from unstructured.ingest.connector.clarifai import (\n",
        "    ClarifaiAccessConfig,\n",
        "    ClarifaiWriteConfig,\n",
        "    SimpleClarifaiConfig,\n",
        ")\n",
        "\n",
        "from unstructured.ingest.runner.writers.base_writer import Writer\n",
        "from unstructured.ingest.runner.writers.clarifai import (\n",
        "    ClarifaiWriter,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P5Jskd-V1PGS"
      },
      "source": [
        "## Clarifai_writer func which will configure the target clarifai app where the ingested documents will be loaded with initialising clarifai PAT as `api_key`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "32whBy_puaEs"
      },
      "outputs": [],
      "source": [
        "def clarifai_writer() -> Writer:\n",
        "    return ClarifaiWriter(\n",
        "        connector_config=SimpleClarifaiConfig(\n",
        "            access_config=ClarifaiAccessConfig(\n",
        "                api_key= \"YOUR_CLARIFAI_PAT\"\n",
        "            ),\n",
        "            app_id= \"YOUR_CLARIFAI_APP_ID\",\n",
        "            user_id= \"YOUR_CLARIFAI_USER_ID\"\n",
        "            ),\n",
        "        write_config=ClarifaiWriteConfig()\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GABTXfWJ1pZr"
      },
      "source": [
        "## Let's now package the writer and runner together, make sure you have your dropbox URL ready."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "32eJrsPSubk4"
      },
      "outputs": [],
      "source": [
        "if __name__ == \"__main__\":\n",
        "    writer = clarifai_writer()\n",
        "    runner = DropboxRunner(\n",
        "        processor_config=ProcessorConfig(\n",
        "            verbose=True,\n",
        "            output_dir=\"dropbox-output\",\n",
        "            num_processes=2,\n",
        "        ),\n",
        "        read_config=ReadConfig(),\n",
        "        partition_config=PartitionConfig(),\n",
        "        connector_config=SimpleDropboxConfig(\n",
        "            access_config=DropboxAccessConfig(token=DROPBOX_ACCESS_TOKEN),\n",
        "            remote_url=\"dropbox://test\",\n",
        "            recursive=True,\n",
        "        ),\n",
        "        writer=writer,\n",
        "        writer_kwargs={},\n",
        "    )\n",
        "    runner.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Let's do RAG over ingested documents from dropbox in just less than 4 lines of code."
      ],
      "metadata": {
        "id": "mMmdK-tDzKPI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "#Replace your PAT\n",
        "os.environ['CLARIFAI_PAT'] = \"YOUR_CLARIFAI_PAT\""
      ],
      "metadata": {
        "id": "ySWJDSBg8GwF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from clarifai.rag import RAG\n",
        "rag_agent = RAG.setup(app_url=\"YOUR CLARIFAI APP URL\",\n",
        "                      llm_url = \"https://clarifai.com/mistralai/completion/models/mistral-large\") # visit clarifai community to get more details of other available models."
      ],
      "metadata": {
        "id": "ydExCbGfadgG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "result=rag_agent.chat(messages=[{\"role\":\"human\", \"content\":\"How to check the tire pressure?\"}])\n",
        "(result[0][\"content\"].split('\\n'))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rq2P8U7nbgQd",
        "outputId": "5031b09f-52b6-44fe-fe60-442e7185da04"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"Sure, I'd be happy to help you understand how to check tire pressure based on the information provided. Here are the steps:\",\n",
              " '',\n",
              " '1. **Prepare your tire pressure gauge**: You can purchase a tire pressure gauge at auto parts stores or service stations. They are usually about the size of a large pen, but can come in other forms. Some gas stations also have a gauge built into their air pumps.',\n",
              " '',\n",
              " \"2. **Remove the valve cap**: Each tire on your vehicle has a valve stem. It's a small black or chrome-colored tube that protrudes from the tire. At the end of the valve stem, there's a cap that you'll need to remove.\",\n",
              " '',\n",
              " \"3. **Align the gauge with the valve**: Place the end of your tire pressure gauge onto the valve stem. You want to ensure it's aligned correctly to get an accurate reading.\",\n",
              " '',\n",
              " '4. **Press the gauge onto the valve**: Apply firm, direct pressure to the valve stem with the gauge. You should hear a brief hiss of air escaping, which is normal.',\n",
              " '',\n",
              " \"5. **Read the pressure**: The measuring stick on the gauge (if it's a pen-style gauge) will be pushed out by the air pressure in the tire. The reading on the gauge indicates the current tire pressure.\",\n",
              " '',\n",
              " \"6. **Adjust the pressure if necessary**: If the pressure is lower than the recommended level for your vehicle, you can add air using an air pump. Recheck the pressure after adding air to ensure you've reached the desired level.\",\n",
              " '',\n",
              " \"7. **Replace the valve cap**: Once you've finished checking or adjusting the tire pressure, don't forget to replace the valve cap. It helps to keep dirt and debris out of the valve stem.\"]"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "MxkQumFe90kD"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}