100_stateless_sequence_classification_task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "distributed-means",
   "metadata": {},
   "source": [
    "# Sequence Classification task\n",
    "> Accelerated model evaluation using ONNX Runtime in the stateless cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb34242c-a208-4693-a9e3-a238a4573c12",
   "metadata": {},
   "source": [
    "Vespa has [implemented](https://blog.vespa.ai/stateless-model-evaluation/)\n",
    "accelerated model evaluation using ONNX Runtime in the stateless cluster.\n",
    "This opens up new usage areas for Vespa, such as serving model predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "resistant-bachelor",
   "metadata": {},
   "source": [
    "## Define the model server\n",
    "\n",
    "The `SequenceClassification` task takes a text input and returns an array of floats that depends on the model used to solve the task. The `model` argument can be the id of the model as defined by the huggingface model hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dedicated-shoot",
   "metadata": {},
   "outputs": [],
   "source": [
    "from learntorank.ml import SequenceClassification\n",
    "\n",
    "task = SequenceClassification(\n",
    "    model_id=\"bert_tiny\", \n",
    "    model=\"google/bert_uncased_L-2_H-128_A-2\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aquatic-robinson",
   "metadata": {},
   "source": [
    "A `ModelServer` is a simplified application package focused on stateless model evaluation. It can take as many tasks as we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cloudy-refund",
   "metadata": {},
   "outputs": [],
   "source": [
    "from learntorank.ml import ModelServer\n",
    "\n",
    "model_server = ModelServer(\n",
    "    name=\"bertModelServer\",\n",
    "    tasks=[task],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "apart-blair",
   "metadata": {},
   "source": [
    "## Deploy the model server\n",
    "\n",
    "We can either host our model server on [Vespa Cloud](https://pyvespa.readthedocs.io/en/latest/deploy-vespa-cloud.html) or deploy it locally using a Docker container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "furnished-diana",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using framework PyTorch: 1.12.1\n",
      "Found input input_ids with shape: {0: 'batch', 1: 'sequence'}\n",
      "Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}\n",
      "Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}\n",
      "Found output output_0 with shape: {0: 'batch'}\n",
      "Ensuring inputs are in correct order\n",
      "position_ids is not present in the generated input list.\n",
      "Generated inputs order: ['input_ids', 'attention_mask', 'token_type_ids']\n",
      "Waiting for configuration server, 0/300 seconds...\n",
      "Waiting for configuration server, 5/300 seconds...\n",
      "Waiting for application status, 0/300 seconds...\n",
      "Waiting for application status, 5/300 seconds...\n",
      "Finished deployment.\n"
     ]
    }
   ],
   "source": [
    "from vespa.deployment import VespaDocker\n",
    "\n",
    "vespa_docker = VespaDocker()\n",
    "app = vespa_docker.deploy(application_package=model_server)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regulated-cleanup",
   "metadata": {},
   "source": [
    "## Get model information\n",
    "\n",
    "Get models available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "lovely-scale",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bert_tiny': 'http://localhost:8080/model-evaluation/v1/bert_tiny'}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "app.get_model_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "integral-wells",
   "metadata": {},
   "source": [
    "Get information about a specific model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "digital-sewing",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model': 'bert_tiny',\n",
       " 'functions': [{'function': 'output_0',\n",
       "   'info': 'http://localhost:8080/model-evaluation/v1/bert_tiny/output_0',\n",
       "   'eval': 'http://localhost:8080/model-evaluation/v1/bert_tiny/output_0/eval',\n",
       "   'arguments': [{'name': 'input_ids', 'type': 'tensor(d0[],d1[])'},\n",
       "    {'name': 'attention_mask', 'type': 'tensor(d0[],d1[])'},\n",
       "    {'name': 'token_type_ids', 'type': 'tensor(d0[],d1[])'}]}]}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "app.get_model_endpoint(model_id=\"bert_tiny\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "defined-behalf",
   "metadata": {},
   "source": [
    "## Get predictions\n",
    "\n",
    "Get a prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "significant-great",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-0.00954509899020195, 0.2504960000514984]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "app.predict(x=\"this is a test\", model_id=\"bert_tiny\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "390c5206",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a7dee01",
   "metadata": {},
   "outputs": [],
   "source": [
    "from shutil import rmtree\n",
    "\n",
    "vespa_docker.container.stop(timeout=600)\n",
    "vespa_docker.container.remove()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "learntorank",
   "language": "python",
   "name": "learntorank"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}