010_passage_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "aad52bcf-6204-4340-9342-9c1ed83344de",
   "metadata": {},
   "source": [
    "---\n",
    "output-file: passage_dataset\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "885c918b-5721-4a0d-947e-62451a6d4c33",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ec96dcb-56ee-412f-bc63-253c5b1a19b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "import os\n",
    "from nbdev import show_doc, nbdev_export"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f3e965-5f42-45ff-ae0c-c85d97563f3a",
   "metadata": {},
   "source": [
    "# Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e388723-b12f-4ba6-b596-f2452b677a88",
   "metadata": {},
   "source": [
    "For the passage ranking use case, we will use the [MS MARCO passage dataset](https://ir-datasets.com/msmarco-passage.html) [^1] through the `ir_datasets` library. Besides being convenient, `ir_datasets` solves encoding errors in the original dataset source files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0815567e-6c49-4dd0-af4c-c2ad4ca63b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ir_datasets\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4e442b2-8e42-4b94-be2e-9c950cdc4298",
   "metadata": {},
   "source": [
    "## Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0227e56f-e1a1-4d19-afa1-20211a1e9235",
   "metadata": {},
   "source": [
    "### Document corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6b03190-4dd2-4847-a150-306a5dc23221",
   "metadata": {},
   "source": [
    "Start by loading the data. The dataset will be downloaded once and cached on disk for future use, so it takes a while the first time the command below is run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90fca331-09ff-49f0-a259-4d366fdafa51",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_corpus = ir_datasets.load(\"msmarco-passage\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "315ebc84-1efa-4dfb-b47e-3b91f38b7db4",
   "metadata": {},
   "source": [
    "Number of passages in the document corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4eaba1b-2d89-44cc-bc5f-1696ffd57153",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8841823"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_corpus.docs_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a204afd-9612-4942-a004-d35ae2ab1237",
   "metadata": {},
   "source": [
    "Sample a few passages of the document corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "827c5f03-cf49-4f75-b01a-1469087ba078",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>The presence of communication amid scientific ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>The Manhattan Project and its atomic bomb help...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Essay on The Manhattan Project - The Manhattan...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>The Manhattan Project was the name for a proje...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>versions of each volume as well as complementa...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  doc_id                                               text\n",
       "0      0  The presence of communication amid scientific ...\n",
       "1      1  The Manhattan Project and its atomic bomb help...\n",
       "2      2  Essay on The Manhattan Project - The Manhattan...\n",
       "3      3  The Manhattan Project was the name for a proje...\n",
       "4      4  versions of each volume as well as complementa..."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "pd.DataFrame(passage_corpus.docs_iter()[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b44d967-1a5a-4d5b-a6f9-0acb10e5e20b",
   "metadata": {},
   "source": [
    "### Training data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b6cd0da-0a97-4f4a-b557-e095684ed0cb",
   "metadata": {},
   "source": [
    "Load the training data. We use the `judged` version that only include queries with at least one relevance judgement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c350128d-efae-4dde-9c4c-5bbee0f435e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_train = ir_datasets.load(\"msmarco-passage/train/judged\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b097f1d-9016-454b-b85f-002aba27a400",
   "metadata": {},
   "source": [
    "#### Relevant documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edbd7231-49d4-49cc-ac41-1ea57bcefb35",
   "metadata": {},
   "source": [
    "Number of relevant judgements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99f66d75-7f81-481c-aa6c-8601c5e09ff7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "532761"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_train.qrels_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e63b35-fae2-4c25-b547-4824fcd6a84a",
   "metadata": {},
   "source": [
    "For each query id, there is a dict of relevant documents containing the document id as key and the relevance score as value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13937d6b-0884-4a21-bf0d-a536282c0872",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'1038069': {'2293922': 1},\n",
       " '700425': {'4351261': 1},\n",
       " '926242': {'3500124': 1},\n",
       " '690553': {'2877918': 1},\n",
       " '411317': {'2230220': 1}}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "from learntorank.passage import sample_dict_items\n",
    "\n",
    "train_qrels_dict = passage_train.qrels_dict()\n",
    "sample_dict_items(train_qrels_dict, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df34f83f-744d-48a8-9ffb-3dfc99bc9750",
   "metadata": {},
   "source": [
    "It is interesting to check what is the range of values of the relevance score. The code below shows that the only score available is 1, indicating that the particular document id is relevant to the query id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b35f9c0a-d7db-4620-a7e2-9ad65184c4c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "set([score \n",
    "     for relevant in train_qrels_dict.values() \n",
    "     for score in relevant.values()]\n",
    "   )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a648c1f4-a4df-4bd4-ba73-b7ea775ce599",
   "metadata": {},
   "source": [
    "#### Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1f81d5c-0e0e-44b6-950e-382fb900836e",
   "metadata": {},
   "source": [
    "Number of training queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce44d430-06b2-4c18-9e2d-a6cb1545dccb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "502939"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_train.queries_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18b3c0d-dfcd-4f39-bab0-a55b6334223e",
   "metadata": {},
   "source": [
    "The number of queries differs from the number of relevant documents because some of the queries have more than one relevant document associated with it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f72f1e25-baa2-4400-81f5-de6e5cba8514",
   "metadata": {},
   "source": [
    "Each query contains a query id and a query text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22332eb7-3f6a-480b-a512-2146dcad4f75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>121352</td>\n",
       "      <td>define extreme</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>634306</td>\n",
       "      <td>what does chattel mean on credit history</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>920825</td>\n",
       "      <td>what was the great leap forward brainly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>510633</td>\n",
       "      <td>tattoo fixers how much does it cost</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>737889</td>\n",
       "      <td>what is decentralization process.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  query_id                                      text\n",
       "0   121352                            define extreme\n",
       "1   634306  what does chattel mean on credit history\n",
       "2   920825   what was the great leap forward brainly\n",
       "3   510633       tattoo fixers how much does it cost\n",
       "4   737889         what is decentralization process."
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "training_queries = pd.DataFrame(passage_train.queries_iter())\n",
    "training_queries.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63f76ea-15b4-40e2-9be3-800f94e828e1",
   "metadata": {},
   "source": [
    "### Development data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e6aa16c-f441-4b70-96c4-c72baf24d9d0",
   "metadata": {},
   "source": [
    "Similarly to the training data, we can load the judged development data and take a look at the queries and relevance judgements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7666049-32cf-44a6-b6dd-d81aee63beae",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_dev = ir_datasets.load(\"msmarco-passage/dev/judged\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c2e5a2-9c7d-41e6-bf2e-18ab7651a4eb",
   "metadata": {},
   "source": [
    "#### Relevant documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b86c85d2-9e8b-47ae-b624-555694ddf166",
   "metadata": {},
   "source": [
    "Number of relevant judgements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4d016f4-f92c-4c44-9a31-7bc1271cb2d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "59273"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_dev.qrels_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5322826-2374-4f4b-8f81-94300dd7fe97",
   "metadata": {},
   "source": [
    "For each query id, there is a dict of relevant documents containing the document id as key and the relevance score as value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3970cf61-1d7b-493d-83be-90b8e9e966b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'255': {'7629892': 1},\n",
       " '611327': {'7610137': 1},\n",
       " '584695': {'7408281': 1},\n",
       " '300246': {'7814106': 1, '7814107': 1},\n",
       " '739094': {'7640560': 1}}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "dev_qrels_dict = passage_dev.qrels_dict()\n",
    "sample_dict_items(dev_qrels_dict, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9708c50-50e6-47b1-b9e1-ddc085f7adca",
   "metadata": {},
   "source": [
    "#### Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93d0d05e-812b-4e35-8813-f8271de3ed4c",
   "metadata": {},
   "source": [
    "Number of dev queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52af5cb3-1650-4df3-b4b3-0c1541d617f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55578"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "passage_dev.queries_count()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be1778e0-8984-4139-8cec-38d87b4ca91e",
   "metadata": {},
   "source": [
    "Each query contains a query id and a query text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "416bac2e-8aba-43e3-b18d-06becf15cf09",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1048578</td>\n",
       "      <td>cost of endless pools/swim spa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1048579</td>\n",
       "      <td>what is pcnt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1048582</td>\n",
       "      <td>what is paysky</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1048583</td>\n",
       "      <td>what is paydata</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1048585</td>\n",
       "      <td>what is paula deen's brother</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  query_id                            text\n",
       "0  1048578  cost of endless pools/swim spa\n",
       "1  1048579                    what is pcnt\n",
       "2  1048582                  what is paysky\n",
       "3  1048583                 what is paydata\n",
       "4  1048585    what is paula deen's brother"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "dev_queries = pd.DataFrame(passage_dev.queries_iter())\n",
    "dev_queries.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19bc32ca-5166-4199-b69d-c48370fcb0e9",
   "metadata": {},
   "source": [
    "## Data Manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc77ec99-0bcf-4264-a9c5-33d257a8e549",
   "metadata": {},
   "source": [
    "### Sample data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54a7efb9-9ccc-4728-9e5f-0af8a45d7a71",
   "metadata": {},
   "source": [
    "Given the large amount of data, it is useful to properly sample data when prototyping, which can be done with the `sample_data` function. This might take same time in case the full dataset needs to be downloaded for the first time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76df394d-85e1-49d4-8745-a851f218b14f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|eval:false\n",
    "#|notest\n",
    "from learntorank.passage import sample_data\n",
    "\n",
    "passage_sample = sample_data(n_relevant=100, n_irrelevant=800)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d73211b1-97d4-405c-bf7f-3970fb699430",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "from learntorank.passage import PassageData\n",
    "\n",
    "passage_sample = PassageData.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "939034ce-0123-48c1-b535-d837ed7670b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PassageData(corpus, train_qrels, train_queries, dev_qrels, dev_queries)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "passage_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a55a35-1705-414d-b7b5-5bdd66d845de",
   "metadata": {},
   "source": [
    "#### Save"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa6885f4-c02a-4492-84c7-fedb7bc28542",
   "metadata": {},
   "source": [
    "We can save the sampled data to disk to avoid regenerating it everytime we need to use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dec8a2cb-39ba-4112-b48f-99a07f58b905",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "assert os.path.exists(\"sample.json\") == False, \"File exists.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c239a82a-048d-4c10-9265-bbd4360352b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "passage_sample.save(\"sample.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5ea5e63-cd4d-48b0-9cd7-4008e543a6cf",
   "metadata": {},
   "source": [
    "#### Load"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c1d8032-b2b4-44bb-96cf-31451367c50d",
   "metadata": {},
   "source": [
    "Load the data back when needed with `PassageData.load`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ef8af40-61db-4957-b51c-7f3c1999777b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from learntorank.passage import PassageData\n",
    "\n",
    "loaded_sample = PassageData.load(file_path=\"sample.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63081808-b74b-44d2-b4af-8da36a7dacc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "assert passage_sample == loaded_sample, \"saved data is different from loaded data.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbbcbb3d-9466-4454-bf21-567d02171007",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "os.remove(\"sample.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a002279-dae8-466a-8e26-441dbf2546f6",
   "metadata": {},
   "source": [
    "[^1]: [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://arxiv.org/abs/1611.09268)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e58812c0-9a5d-42d9-a03b-7c19455141fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "#|hide\n",
    "nbdev_export()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "learntorank",
   "language": "python",
   "name": "learntorank"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}