Data_Utils/Ingestion pipelines/Ready_to_use_foundational_pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<td>\n",
    "   <a target=\"_blank\" href=\"https://www.clarifai.com/\" ><img src=\"https://upload.wikimedia.org/wikipedia/commons/b/bc/Clarifai_Logo_FC_Web.png\" width=256/></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ready to use foundational pipelines\n",
    "* This notebook consists of how to leverage ready-to-use pipelines using clarifai-datautils repo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install dependancies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install clarifai-datautils==0.0.4\n",
    "!pip install 'unstructured[pdf]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sudo apt-get update\n",
    "!sudo apt-get install -y libgl1-mesa-glx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building basic pipelines\n",
    "* PDF Partition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline: basic_pdf\n",
      "\t<clarifai_datautils.multimodal.pipeline.PDF.PDFPartition object at 0x774f7256eab0>\n"
     ]
    }
   ],
   "source": [
    "from clarifai_datautils.multimodal import Pipeline\n",
    "\n",
    "# use predefine the pipeline\n",
    "pipeline = Pipeline.load(name = 'basic_pdf')\n",
    "\n",
    "#view the pipeline\n",
    "pipeline.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from clarifai.client import Dataset\n",
    "dataset = Dataset(url=DATASET_URL, pat=PAT)\n",
    "dataset.upload_dataset(pipeline.run(files=PDF_FILE, loader=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard pipeline\n",
    "* PDF partition\n",
    "* Clean extra whitespace\n",
    "* Group broken paragraphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use predefine the pipeline\n",
    "pipeline = Pipeline.load(name = 'standard_pdf')\n",
    "\n",
    "dataset = Dataset(url=DATASET_URL, pat=PAT)\n",
    "dataset.upload_dataset(pipeline.run(files=PDF_FILE, loader=True))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Structured pipeline\n",
    "* PDF Partition\n",
    "* Clean extra whitespace\n",
    "* Extract Email address\n",
    "* Extract Date time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use predefine the pipeline\n",
    "pipeline = Pipeline.load(name = 'stuctured_pdf')\n",
    "\n",
    "dataset = Dataset(url=DATASET_URL, pat=PAT)\n",
    "dataset.upload_dataset(pipeline.run(files=PDF_FILE, loader=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}