notebooks/python-api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kipoi python API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three basic building blocks in kipoi:\n",
    "\n",
    "- **Source** - provides Models and DataLoaders.\n",
    "- **Model** - makes the prediction given the numpy arrays. \n",
    "- **Dataloader** - loads the data from raw files and transforms them into a form that is directly consumable by the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](../docs/theme_dir/img/kipoi-workflow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List of main commands\n",
    "\n",
    "\n",
    "Get/list sources\n",
    "- `kipoi.list_sources()`\n",
    "- `kipoi.get_source()`\n",
    "\n",
    "\n",
    "List models/dataloaders\n",
    "- `kipoi.list_models()`\n",
    "- `kipoi.list_dataloaders()`\n",
    "\n",
    "Get model/dataloader\n",
    "- `kipoi.get_model()`\n",
    "- `kipoi.get_dataloader_factory()`\n",
    "\n",
    "Load only model/dataloader description from the yaml file without loading the model\n",
    "\n",
    "- `kipoi.get_model_descr()`  \n",
    "- `kipoi.get_dataloader_descr()`\n",
    "\n",
    "Install the dependencies\n",
    "- `kipoi.install_model_dependencies()`\n",
    "- `kipoi.install_dataloader_dependencies()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kipoi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Source\n",
    "\n",
    "Available sources are specified in the config file located at: `~/.kipoi/config.yaml`. Here is an example config file:\n",
    "\n",
    "```yaml\n",
    "model_sources:\n",
    "    kipoi: # default\n",
    "        type: git-lfs # git repository with large file storage (git-lfs)\n",
    "        remote_url: git@github.com:kipoi/models.git # git remote\n",
    "        local_path: ~/.kipoi/models/ # local storage path\n",
    "    gl:\n",
    "        type: git-lfs  # custom model\n",
    "        remote_url: https://i12g-gagneurweb.informatik.tu-muenchen.de/gitlab/gagneurlab/model-zoo.git\n",
    "        local_path: /s/project/model-zoo\n",
    "```\n",
    "\n",
    "There are three different model sources possible: \n",
    "\n",
    "- **`git-lfs`** - git repository with source files tracked normally by git and all the binary files like model weights (located in `files*` directories) are tracked by [git-lfs](https://git-lfs.github.com). \n",
    "  - Requires `git-lfs` to be installed.\n",
    "- **`git`** - all the files including weights (not recommended)\n",
    "- **`local`** - local directory containing models defined in subdirectories\n",
    "\n",
    "For **`git-lfs`** source type, larger files tracked by `git-lfs` will be downloaded into the specified directory `local_path` only after the model has been requested (when invoking `kipoi.get_model()`).\n",
    "\n",
    "#### Note\n",
    "\n",
    "A particular model/dataloader is defined by its source (say `kipoi` or `my_git_models`) and the relative path of the desired model directory from the model source root (say `rbp/`).\n",
    "\n",
    "A directory is considered a model if it contains a `model.yaml` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import kipoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import logging\n",
    "logging.disable(1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source</th>\n",
       "      <th>type</th>\n",
       "      <th>location</th>\n",
       "      <th>local_size</th>\n",
       "      <th>n_models</th>\n",
       "      <th>n_dataloaders</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kipoi</td>\n",
       "      <td>git-lfs</td>\n",
       "      <td>/home/avsec/.kipoi/mo...</td>\n",
       "      <td>1,2G</td>\n",
       "      <td>780</td>\n",
       "      <td>780</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  source     type                  location local_size  n_models  n_dataloaders\n",
       "0  kipoi  git-lfs  /home/avsec/.kipoi/mo...       1,2G       780            780"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kipoi.list_sources()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = kipoi.get_source(\"kipoi\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GitLFSSource(remote_url='git@github.com:kipoi/models.git', local_path='/home/avsec/.kipoi/models/')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source</th>\n",
       "      <th>model</th>\n",
       "      <th>version</th>\n",
       "      <th>authors</th>\n",
       "      <th>contributors</th>\n",
       "      <th>doc</th>\n",
       "      <th>type</th>\n",
       "      <th>inputs</th>\n",
       "      <th>targets</th>\n",
       "      <th>postproc_score_variants</th>\n",
       "      <th>license</th>\n",
       "      <th>cite_as</th>\n",
       "      <th>trained_on</th>\n",
       "      <th>training_procedure</th>\n",
       "      <th>tags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>kipoi</td>\n",
       "      <td>DeepSEAKeras</td>\n",
       "      <td>0.1</td>\n",
       "      <td>[Author(name='Jian Zh...</td>\n",
       "      <td>[Author(name='Lara Ur...</td>\n",
       "      <td>This CNN is based on ...</td>\n",
       "      <td>keras</td>\n",
       "      <td>seq</td>\n",
       "      <td>TFBS_DHS_probs</td>\n",
       "      <td>True</td>\n",
       "      <td>MIT</td>\n",
       "      <td>https://doi.org/10.10...</td>\n",
       "      <td>ENCODE and Roadmap Ep...</td>\n",
       "      <td>https://www.nature.co...</td>\n",
       "      <td>[Histone modification...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>kipoi</td>\n",
       "      <td>extended_coda</td>\n",
       "      <td>0.1</td>\n",
       "      <td>[Author(name='Pang We...</td>\n",
       "      <td>[Author(name='Johnny ...</td>\n",
       "      <td>Single bp resolution ...</td>\n",
       "      <td>keras</td>\n",
       "      <td>[H3K27AC_subsampled]</td>\n",
       "      <td>[H3K27ac]</td>\n",
       "      <td>False</td>\n",
       "      <td>MIT</td>\n",
       "      <td>https://doi.org/10.10...</td>\n",
       "      <td>Described in https://...</td>\n",
       "      <td>Described in https://...</td>\n",
       "      <td>[Histone modification]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kipoi</td>\n",
       "      <td>DeepCpG_DNA/Hou2016_m...</td>\n",
       "      <td>1.0.4</td>\n",
       "      <td>[Author(name='Christo...</td>\n",
       "      <td>[Author(name='Roman K...</td>\n",
       "      <td>This is the extractio...</td>\n",
       "      <td>keras</td>\n",
       "      <td>[dna]</td>\n",
       "      <td>[cpg/mESC1, cpg/mESC2...</td>\n",
       "      <td>True</td>\n",
       "      <td>MIT</td>\n",
       "      <td>https://doi.org/10.11...</td>\n",
       "      <td>scBS-seq and scRRBS-s...</td>\n",
       "      <td>Described in https://...</td>\n",
       "      <td>[DNA methylation]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>kipoi</td>\n",
       "      <td>DeepCpG_DNA/Smallwood...</td>\n",
       "      <td>1.0.4</td>\n",
       "      <td>[Author(name='Christo...</td>\n",
       "      <td>[Author(name='Roman K...</td>\n",
       "      <td>This is the extractio...</td>\n",
       "      <td>keras</td>\n",
       "      <td>[dna]</td>\n",
       "      <td>[cpg/BS24_1_2I, cpg/B...</td>\n",
       "      <td>True</td>\n",
       "      <td>MIT</td>\n",
       "      <td>https://doi.org/10.11...</td>\n",
       "      <td>scBS-seq and scRRBS-s...</td>\n",
       "      <td>Described in https://...</td>\n",
       "      <td>[DNA methylation]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>kipoi</td>\n",
       "      <td>DeepCpG_DNA/Hou2016_H...</td>\n",
       "      <td>1.0.4</td>\n",
       "      <td>[Author(name='Christo...</td>\n",
       "      <td>[Author(name='Roman K...</td>\n",
       "      <td>This is the extractio...</td>\n",
       "      <td>keras</td>\n",
       "      <td>[dna]</td>\n",
       "      <td>[cpg/HepG21, cpg/HepG...</td>\n",
       "      <td>True</td>\n",
       "      <td>MIT</td>\n",
       "      <td>https://doi.org/10.11...</td>\n",
       "      <td>scBS-seq and scRRBS-s...</td>\n",
       "      <td>Described in https://...</td>\n",
       "      <td>[DNA methylation]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  source                     model version                   authors              contributors  \\\n",
       "0  kipoi              DeepSEAKeras     0.1  [Author(name='Jian Zh...  [Author(name='Lara Ur...   \n",
       "1  kipoi             extended_coda     0.1  [Author(name='Pang We...  [Author(name='Johnny ...   \n",
       "2  kipoi  DeepCpG_DNA/Hou2016_m...   1.0.4  [Author(name='Christo...  [Author(name='Roman K...   \n",
       "3  kipoi  DeepCpG_DNA/Smallwood...   1.0.4  [Author(name='Christo...  [Author(name='Roman K...   \n",
       "4  kipoi  DeepCpG_DNA/Hou2016_H...   1.0.4  [Author(name='Christo...  [Author(name='Roman K...   \n",
       "\n",
       "                        doc   type                inputs                   targets  \\\n",
       "0  This CNN is based on ...  keras                   seq            TFBS_DHS_probs   \n",
       "1  Single bp resolution ...  keras  [H3K27AC_subsampled]                 [H3K27ac]   \n",
       "2  This is the extractio...  keras                 [dna]  [cpg/mESC1, cpg/mESC2...   \n",
       "3  This is the extractio...  keras                 [dna]  [cpg/BS24_1_2I, cpg/B...   \n",
       "4  This is the extractio...  keras                 [dna]  [cpg/HepG21, cpg/HepG...   \n",
       "\n",
       "   postproc_score_variants license                   cite_as                trained_on  \\\n",
       "0                     True     MIT  https://doi.org/10.10...  ENCODE and Roadmap Ep...   \n",
       "1                    False     MIT  https://doi.org/10.10...  Described in https://...   \n",
       "2                     True     MIT  https://doi.org/10.11...  scBS-seq and scRRBS-s...   \n",
       "3                     True     MIT  https://doi.org/10.11...  scBS-seq and scRRBS-s...   \n",
       "4                     True     MIT  https://doi.org/10.11...  scBS-seq and scRRBS-s...   \n",
       "\n",
       "         training_procedure                      tags  \n",
       "0  https://www.nature.co...  [Histone modification...  \n",
       "1  Described in https://...    [Histone modification]  \n",
       "2  Described in https://...         [DNA methylation]  \n",
       "3  Described in https://...         [DNA methylation]  \n",
       "4  Described in https://...         [DNA methylation]  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kipoi.list_models().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "Let's choose to use the `rbp_eclip/XRCC6` model from kipoi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL = \"rbp_eclip/XRCC6\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** If you are using python2, use a different model like `MaxEntScan/3prime` to following this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note. Install all the dependencies for that model:\n",
    "# add --gpu flag to install gpu-compatible dependencies (e.g. installs tensorflow-gpu instead of tensorflow)\n",
    "!kipoi env create {MODEL}\n",
    "!source activate kipoi-{MODEL}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = kipoi.get_model(MODEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Available fields:\n",
    "\n",
    "#### Model\n",
    "\n",
    "- type\n",
    "- args\n",
    "- info\n",
    "  - authors\n",
    "  - name\n",
    "  - version\n",
    "  - tags\n",
    "  - doc\n",
    "- schema\n",
    "  - inputs\n",
    "  - targets\n",
    "- default_dataloader - loaded dataloader class\n",
    "\n",
    "\n",
    "- predict_on_batch()\n",
    "- source\n",
    "- source_dir\n",
    "- pipeline\n",
    "  - predict()\n",
    "  - predict_example()\n",
    "  - predict_generator()\n",
    "  \n",
    "#### Dataloader\n",
    "\n",
    "- type\n",
    "- defined_as\n",
    "- args\n",
    "- info (same as for the model)\n",
    "- output_schema\n",
    "  - inputs\n",
    "  - targets\n",
    "  - metadata\n",
    "\n",
    "\n",
    "- source\n",
    "- source_dir\n",
    "- example_kwargs\n",
    "- init_example()\n",
    "- batch_iter()\n",
    "- batch_train_iter()\n",
    "- batch_predict_iter()\n",
    "- load_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<kipoi.model.KerasModel at 0x7f95b27af2b0>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'keras'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ModelInfo(authors=[Author(name='Ziga Avsec', github='avsecz', email=None)], doc='\\'RBP binding model from Avsec et al: \"Modeling positional effects of regulatory sequences with spline transformations increases prediction accuracy of deep neural networks\". \\'\\n', name=None, version='0.1', license='MIT', tags=['RNA binding'], contributors=[Author(name='Ziga Avsec', github='avsecz', email=None)], cite_as='https://doi.org/10.1093/bioinformatics/btx727', trained_on='RBP occupancy peaks measured by eCLIP-seq (Van Nostrand et al., 2016 - https://doi.org/10.1038/nmeth.3810), https://github.com/gagneurlab/Manuscript_Avsec_Bioinformatics_2017\\n', training_procedure='Single task training with ADAM')"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.1'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.info.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dist_exon_intron': ArraySchema(shape=(1, 10), doc='Distance the nearest exon_intron (splice donor) site transformed with B-splines', name='dist_exon_intron', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_gene_end': ArraySchema(shape=(1, 10), doc='Distance the nearest gene end transformed with B-splines', name='dist_gene_end', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_gene_start': ArraySchema(shape=(1, 10), doc='Distance the nearest gene start transformed with B-splines', name='dist_gene_start', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_intron_exon': ArraySchema(shape=(1, 10), doc='Distance the nearest intron_exon (splice acceptor) site transformed with B-splines', name='dist_intron_exon', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_polya': ArraySchema(shape=(1, 10), doc='Distance the nearest Poly-A site transformed with B-splines', name='dist_polya', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_start_codon': ArraySchema(shape=(1, 10), doc='Distance the nearest start codon transformed with B-splines', name='dist_start_codon', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_stop_codon': ArraySchema(shape=(1, 10), doc='Distance the nearest stop codon transformed with B-splines', name='dist_stop_codon', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'dist_tss': ArraySchema(shape=(1, 10), doc='Distance the nearest TSS site transformed with B-splines', name='dist_tss', special_type=None, associated_metadata=[], column_labels=None),\n",
       " 'seq': ArraySchema(shape=(101, 4), doc='One-hot encoded RNA sequence', name='seq', special_type=<ArraySpecialType.DNASeq: 'DNASeq'>, associated_metadata=[], column_labels=None)}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dict(model.schema.inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ArraySchema(shape=(1,), doc='Predicted binding strength', name=None, special_type=None, associated_metadata=[], column_labels=None)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.schema.targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default dataloader\n",
    "\n",
    "Model already has the default dataloder present. To use it, specify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/avsec/.kipoi/models/rbp_eclip/XRCC6'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.source_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dataloader.SeqDistDataset"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.default_dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Info(authors=[Author(name='Ziga Avsec', github='avsecz', email=None)], doc='RBP binding model taking as input 101nt long sequence as well as 8 distances to nearest genomic landmarks -  tss, poly-A, exon-intron boundary, intron-exon boundary, start codon, stop codon, gene start, gene end\\n', name=None, version='0.1', license='MIT', tags=[])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.default_dataloader.info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict_on_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method KerasModel.predict_on_batch of <kipoi.model.KerasModel object at 0x7f95b27af2b0>>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_on_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Others"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GitLFSSource(remote_url='git@github.com:kipoi/models.git', local_path='/home/avsec/.kipoi/models/')"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Model source\n",
    "model.source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/avsec/.kipoi/models/rbp_eclip/XRCC6'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# model location directory\n",
    "model.source_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "DataLoader = kipoi.get_dataloader_factory(MODEL)\n",
    "# same as DataLoader = model.default_dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dataloader will most likely require input arguments in which the input files are defined, for example input fasta files or bed files, based on which the model input is generated. There are several options where the dataloader input keyword arguments are displayed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Args:\n",
      "        intervals_file: file path; tsv file\n",
      "            Assumes bed-like `chrom start end id score strand` format.\n",
      "        fasta_file: file path; Genome sequence\n",
      "        gtf_file: file path; Genome annotation GTF file.\n",
      "        filter_protein_coding: Considering genomic landmarks only for protein coding genes\n",
      "        preproc_transformer: file path; tranformer used for pre-processing.\n",
      "        target_file: file path; path to the targets\n",
      "        batch_size: int\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "# Display information about the dataloader\n",
    "print(DataLoader.__doc__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keyword argument: `intervals_file`\n",
      "    doc: bed6 file with `chrom start end id score strand` columns\n",
      "    type: str\n",
      "    optional: False\n",
      "    example: example_files/intervals.bed\n",
      "Keyword argument: `fasta_file`\n",
      "    doc: Reference genome sequence\n",
      "    type: str\n",
      "    optional: False\n",
      "    example: example_files/hg38_chr22.fa\n",
      "Keyword argument: `gtf_file`\n",
      "    doc: file path; Genome annotation GTF file\n",
      "    type: str\n",
      "    optional: False\n",
      "    example: example_files/gencode.v24.annotation_chr22.gtf\n",
      "Keyword argument: `filter_protein_coding`\n",
      "    doc: Considering genomic landmarks only for protein coding genes when computing the distances to the nearest genomic landmark.\n",
      "    type: str\n",
      "    optional: True\n",
      "    example: True\n",
      "Keyword argument: `target_file`\n",
      "    doc: path to the targets (txt) file\n",
      "    type: str\n",
      "    optional: True\n",
      "    example: example_files/targets.tsv\n",
      "Keyword argument: `use_linecache`\n",
      "    doc: if True, use linecache https://docs.python.org/3/library/linecache.html to access bed file rows\n",
      "    type: str\n",
      "    optional: True\n",
      "--------------------------------------------------------------------------------\n",
      "Example keyword arguments are: {'intervals_file': 'example_files/intervals.bed', 'fasta_file': 'example_files/hg38_chr22.fa', 'gtf_file': 'example_files/gencode.v24.annotation_chr22.gtf', 'filter_protein_coding': True, 'target_file': 'example_files/targets.tsv'}\n"
     ]
    }
   ],
   "source": [
    "# Alternatively the dataloader keyword arguments can be displayed using the function:\n",
    "kipoi.print_dl_kwargs(DataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run dataloader on some examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fasta_file': 'example_files/hg38_chr22.fa',\n",
       " 'filter_protein_coding': True,\n",
       " 'gtf_file': 'example_files/gencode.v24.annotation_chr22.gtf',\n",
       " 'intervals_file': 'example_files/intervals.bed',\n",
       " 'target_file': 'example_files/targets.tsv'}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# each dataloader already provides example files which can be used to illustrate its use:\n",
    "DataLoader.example_kwargs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cd into the source directory \n",
    "os.chdir(DataLoader.source_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".\n",
      "├── custom_keras_objects.py -> ../template/custom_keras_objects.py\n",
      "├── dataloader_files\n",
      "│   └── position_transformer.pkl\n",
      "├── dataloader.py -> ../template/dataloader.py\n",
      "├── dataloader.yaml -> ../template/dataloader.yaml\n",
      "├── example_files -> ../template/example_files\n",
      "├── model_files\n",
      "│   └── model.h5\n",
      "├── model.yaml -> ../template/model.yaml\n",
      "└── __pycache__\n",
      "    ├── custom_keras_objects.cpython-36.pyc\n",
      "    └── dataloader.cpython-36.pyc\n",
      "\n",
      "4 directories, 8 files\n"
     ]
    }
   ],
   "source": [
    "!tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = DataLoader(**DataLoader.example_kwargs)\n",
    "# could be also done with DataLoader.init_example()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['inputs', 'targets', 'metadata'])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This particular dataloader is of type Dataset\n",
    "# i.e. it implements the __getitem__ method:\n",
    "dl[0].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25]], dtype=float32)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl[0][\"inputs\"][\"seq\"][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25],\n",
       "       [0.25, 0.25, 0.25, 0.25]], dtype=float32)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dl[0][\"inputs\"][\"seq\"][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the whole dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00,  6.24it/s]\n"
     ]
    }
   ],
   "source": [
    "whole_data = dl.load_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['inputs', 'targets', 'metadata'])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "whole_data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14, 101, 4)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "whole_data[\"inputs\"][\"seq\"].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the iterator to run predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "it = dl.batch_iter(batch_size=1, shuffle=False, num_workers=0, drop_last=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 101, 4)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(it)[\"inputs\"][\"seq\"].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.1351]], dtype=float32)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_on_batch(next(it)[\"inputs\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline\n",
    "\n",
    "Pipeline object will take the dataloader arguments and run the whole pipeline directly:\n",
    "\n",
    "```\n",
    "dataloader arguments --Dataloader-->  numpy arrays --Model--> prediction\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_kwargs = model.default_dataloader.example_kwargs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00,  6.78it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0.4208],\n",
       "       [0.0005],\n",
       "       [0.0005],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.0005],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208]], dtype=float32)"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = model.pipeline.predict_example()\n",
    "preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1it [00:01,  1.56s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([0.4208, 0.0005, 0.0005, 0.4208, 0.4208, 0.4208, 0.0005, 0.4208, 0.4208, 0.4208, 0.4208,\n",
       "       0.4208, 0.4208, 0.4208], dtype=float32)"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.pipeline.predict(example_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.4208],\n",
       "       [0.0005]], dtype=float32)"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(model.pipeline.predict_generator(example_kwargs, batch_size=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.4208],\n",
       "       [0.0005],\n",
       "       [0.0005],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.0005],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208],\n",
       "       [0.4208]], dtype=float32)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from kipoi_utils.data_utils import numpy_collate\n",
    "numpy_collate_concat(list(model.pipeline.predict_generator(example_kwargs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re-train the Keras model\n",
    "\n",
    "Keras model is stored under the `.model` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.model.compile(\"adam\", \"binary_crossentropy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_it = dl.batch_train_iter(batch_size=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model.model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "3/3 [==============================] - 1s 291ms/step - loss: 1.3592\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f95b0095fd0>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.model.fit_generator(train_it, steps_per_epoch=3, epochs=1)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}