Slides-handout_version.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "id": "d065de53-0cd6-4a74-82af-9912a6ab320d",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"Diffusion models\"\n",
    "author: \n",
    "  - \"Laura Sánchez García\"\n",
    "  - \"Julio Antonio Soto Vicente\"\n",
    "institute: \"IE University (C4_466671 - Advanced Artificial Intelligence)\"\n",
    "bibliography: references_slides.bib\n",
    "csl: association-for-computing-machinery.csl\n",
    "format: \n",
    "  beamer:\n",
    "    quarto-required: \"1.5.56\"\n",
    "    pdf-engine: xelatex\n",
    "    classoption: \"handout\"\n",
    "    slide-level: 2\n",
    "    fontsize: \"10pt\"\n",
    "    latex-auto-install: false\n",
    "    toc: true\n",
    "    toc-title: Outline\n",
    "    colortheme: magpie\n",
    "    innertheme: circles\n",
    "    outertheme: infolinescustom\n",
    "    navigation: empty\n",
    "    papersize: a4\n",
    "    template-partials:\n",
    "      - toc.tex\n",
    "header-includes: |\n",
    "  \\titlegraphic{Fall 2024}\n",
    "  \\usepackage[dvipsnames]{xcolor}\n",
    "  \\usepackage{ifplatform}\n",
    "  \\usepackage{mathtools}\n",
    "  \\usepackage{amsmath}\n",
    "  \\usepackage{annotate-equations}\n",
    "  \\setbeamertemplate{headline}{}\n",
    "  \\renewcommand{\\eqnhighlightheight}{\\mathstrut}\n",
    "  \\renewcommand{\\eqnhighlightshade}{47}\n",
    "  \\let\\oldfootnote\\footnote\n",
    "  \\renewcommand{\\footnote}{\\only<+->\\oldfootnote}\n",
    "  \\newcommand\\unfootnote[1]{\\begingroup \\renewcommand\\thefootnote{}\\footnote{#1} \\addtocounter{footnote}{-1} \\endgroup}\n",
    "  \\let\\oldfootnotesize\\footnotesize\n",
    "  \\renewcommand*{\\footnotesize}{\\oldfootnotesize\\scriptsize}\n",
    "  \\setbeamerfont{caption}{size=\\tiny}\n",
    "  \\definecolor{BlueMean}{HTML}{004ce9}\n",
    "  \\definecolor{LightBlueMean}{HTML}{70ccf9}\n",
    "  \\definecolor{OrangeVar}{HTML}{ff854c}\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5a8531c-f316-42fa-8e82-9bbb579df149",
   "metadata": {},
   "source": [
    "# Intro"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a81b399-5010-4ea6-97f4-8ba5c49481a1",
   "metadata": {},
   "source": [
    "## Generative models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb07a44c-00ac-45c0-9bde-1be05fd5277e",
   "metadata": {},
   "source": [
    "A (certainly not complete) list:\n",
    "\n",
    "- Latent Variable models (incl. VAEs)\n",
    "- Autoregressive models (incl. GPT-style Language Models)\n",
    "- GANs\n",
    "- Flow-based models (incl. Normalizing Flows)\n",
    "- Energy-Based Models (incl. Score-based models)\n",
    "- **Diffusion models** (kind of mix of all previous points)\n",
    "- Combinations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c499ea57-14a5-41bc-b2f4-4bbba885950f",
   "metadata": {},
   "source": [
    "## Image generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f077aa7-4833-490a-8fef-29b14c83501b",
   "metadata": {},
   "source": [
    ":::: {.columns}\n",
    "\n",
    "::: {.column width=\"50%\" align=center}\n",
    "![](images/DDPM_celebA.png){width=60% fig-align=\"center\" fig-alt=\"DDPM CelebA\"}\n",
    "\\ifwindows \n",
    "\\vspace{-14pt} \n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Ho el al. [2020]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    ":::\n",
    "\n",
    "::: {.column width=\"50%\" align=center}\n",
    "\\small \\textit{\"A photo of a Corgi dog riding a bike in Times Square. It is wearing sunglasses and a beach hat.\"}\n",
    "\n",
    "\\tiny &nbsp;\n",
    "\n",
    "![](images/corgi_imagen.jpg){width=100% fig-alt=\"Imagen corgi\"}\n",
    "\\ifwindows \n",
    "\\vspace{-12pt} \n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Saharia et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    ":::\n",
    "\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4634fd11-be38-4b48-b5b5-466934b1b4e7",
   "metadata": {},
   "source": [
    "## Video generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef971783-a529-4b6c-ba42-ca5d3fe66462",
   "metadata": {},
   "source": [
    ":::: {.columns align=center}\n",
    "::: {.column align=center width=100%}\n",
    "[![](images/sora_1.png){width=100% fig-alt=\"OpenAI Sora\" fig-align=\"center\"}](https://player.vimeo.com/video/913331489?h=d6b3d4c2bd)\n",
    "\\ifwindows \n",
    "\\vspace{-12pt} \n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Brooks et al. [2024]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    ":::\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a9f959-b4f9-46c0-b7a1-394e5a5b4f92",
   "metadata": {},
   "source": [
    "# Denoising Diffusion Probabilistic Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38b3dc8-038f-4806-a5b2-6977e6c1cd95",
   "metadata": {},
   "source": [
    "## Denoising Diffusion Probabilistic Models\n",
    "\n",
    "\\large Outline\n",
    "\n",
    "\\normalsize\n",
    "\n",
    "- The forward process\n",
    "- The Nice™ property\n",
    "- The reverse process\n",
    "- Loss function\n",
    "- Training algorithm\n",
    "- The model\n",
    "- Sampling algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deda108e-0e62-4a09-b4f9-6ce8446fd4a7",
   "metadata": {},
   "source": [
    "## Denoising Diffusion Probabilistic Models (DDPMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c4f1e05-f6a6-4f64-bc89-c12ae97a9a4c",
   "metadata": {},
   "source": [
    ":::: {.columns align=center}\n",
    "::: {.column align=center width=100%}\n",
    "[![](images/ddpm_paper.png){height=80% fig-align=\"center\"}](https://arxiv.org/abs/2006.11239)\n",
    ":::\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2f6e482-dbc4-41df-b15c-8a75075495ae",
   "metadata": {},
   "source": [
    "## Denoising Diffusion Probabilistic Models (DDPMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbd6f5ef-0490-46e3-be10-838a60f8f833",
   "metadata": {},
   "source": [
    "\\begin{center}\n",
    "\\textit{\"Creating noise from data is easy;\\\\creating data from noise is generative modeling.\"}\n",
    "\\end{center}\n",
    "\\ifwindows \n",
    "\\vspace{-4pt} \n",
    "\\fi\n",
    "\\hfill Song et al., 2020\n",
    "\n",
    ". . .\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{10pt} \n",
    "\\fi\n",
    "\n",
    "DDPMs progressively generate images out of noise\n",
    "\n",
    "![](images/ddpm_paper_process_no_notation.png){height=7em}\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-10pt} \n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Ho et al. [2020]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d7636e7-bb05-4842-9fb5-40413b029322",
   "metadata": {},
   "source": [
    "## Denoising Diffusion Probabilistic Models (DDPMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8cf2a36-8600-4ae2-a333-7e8a216851f1",
   "metadata": {},
   "source": [
    "In order for a model to learn how to do that, we need a process to generate suitable training data in the form of $(\\text{noise}, \\text{image})$ pairs\n",
    "\n",
    ". . .\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{10pt} \n",
    "\\fi\n",
    "\n",
    "We will refer to the process of creating the training data as the **forward** diffusion process, which will progressively make an image noisier\n",
    "\n",
    ". . .\n",
    "\n",
    "![](images/ddpm_paper_forward_process_no_notation.png){height=7em}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad4742a-f3d8-4cb6-8b4f-043ac04e8b59",
   "metadata": {},
   "source": [
    "## Denoising Diffusion Probabilistic Models (DDPMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b219d2-f4be-4756-b02a-e88ef25a8fdb",
   "metadata": {},
   "source": [
    "Our model will learn how to revert that noising process, through what it is known as the **reverse** diffusion process → progressively denoising a noisy image\n",
    "\n",
    ". . .\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{10pt} \n",
    "\\fi\n",
    "\n",
    "![](images/ddpm_paper_process_no_notation.png){height=7em}\n",
    "\n",
    ". . .\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{10pt} \n",
    "\\fi\n",
    "\n",
    "Once trained, the model should have therefore learned how to denoise images\n",
    "\n",
    "So we can generate some purely random noise, run it through our model and get back an image generated out of pure noise!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa088fdd-0ec2-4b9b-a813-b82d32560899",
   "metadata": {},
   "source": [
    "## Denoising Diffusion Probabilistic Models (DDPMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56c80c52-fe5b-43ad-969c-d07a8ab4bb8c",
   "metadata": {},
   "source": [
    "More formally, DDPMs work through many steps $t$ which are $0, 1, \\ldots ,T$\n",
    "\n",
    "![](images/ddpm_paper_process.png){height=7em}\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{6pt} \n",
    "\\fi\n",
    "```\n",
    "\n",
    "- $\\mathbf{x}_0$ is the original image\n",
    "- $q(\\mathbf{x}_t \\mid \\mathbf{x}_{t-1})$ is the **forward** diffusion process\n",
    "- $p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_{t})$ will be the **reverse** diffusion process (learned by our model with weights $\\theta$)\n",
    "\n",
    ". . .\n",
    "\n",
    "During forward diffusion we add Gaussian (Normal) noise to the image in every $t$, producing noisy images $\\mathbf{x}_1, \\mathbf{x}_2, \\ldots \\mathbf{x}_T$\n",
    "\n",
    "As $t$ becomes higher, the image becomes more and more noisy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec928474-e277-4a92-a904-c2cb900bcb44",
   "metadata": {},
   "source": [
    "## The forward process {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d468dbe5-f13d-4350-9687-b05af716c7b9",
   "metadata": {},
   "source": [
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "![](images/ddpm_paper_forward_process_highlighted.png){height=4.5em fig-align=\"center\"}\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-16pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "$$q(\\mathbf{x}_t \\mid \\mathbf{x}_{t-1}) \\coloneqq \\sqrt{1 - \\beta_t} \\cdot \\mathbf{x}_{t-1} + \\mathcal{N}(0, \\beta_t \\mathbf{I})$$\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "- Take an image at some point $t-1$ as $\\mathbf{x}_{t-1}$\n",
    "- Generate Gaussian noise from an isotropic [\\color{SkyBlue}{multivariate Normal}](https://julioasotodv.github.io/interactive-demos/mvn/multivariate_normal.html) of size $\\mathbf{x}_t$, with mean $0$ and **variance** $\\beta_t$ \n",
    "- Scale $\\mathbf{x}_{t-1}$ values by $\\sqrt{1 - \\beta_t}$ (so data scale does not grow as we add noise)\n",
    "- Add the noise the the scaled image\n",
    "\n",
    ". . .\n",
    "\n",
    "It can be directly computed as $q(\\mathbf{x}_t \\mid \\mathbf{x}_{t-1}) \\coloneqq \\mathcal{N}(\\mathbf{x}_t; \\sqrt{1 - \\beta_t} \\mathbf{x}_{t-1}, \\beta_t \\mathbf{I})$\n",
    "\n",
    ". . .\n",
    "\n",
    "$\\beta_t$ is called the *variance schedule* $\\beta_1, \\ldots, \\beta_T$ that effectively controls how much noise is added in each step $t$ \\footnote{In the paper it is made to grow linearly from $\\beta_1 = 10^{-4}$ to $\\beta_T = 0.02$ for $T=1000$}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e007c35f-a7a0-4072-bb91-5403defe441c",
   "metadata": {},
   "source": [
    "## The forward process {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47be66f8-dc7c-4eff-b10a-94bbcbe64a88",
   "metadata": {},
   "source": [
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "![](images/ddpm_paper_forward_process_highlighted.png){height=4.5em fig-align=\"center\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf069ca-7d27-4e58-829e-e84a7f6120ab",
   "metadata": {},
   "source": [
    "The full forward process is therefore:\n",
    "\n",
    "$$q(\\mathbf{x}_{1:T} \\mid \\mathbf{x}_0) \\coloneqq \\prod^T_{t=1} q(\\mathbf{x}_{t} \\mid \\mathbf{x}_{t-1})$$\n",
    "\n",
    "For a large $T$, the final image is basically only noise (all original image info is essentially lost), so it becomes roughly $\\mathbf{x}_T \\sim \\mathcal{N}(0, \\mathbf{I})$\n",
    "\n",
    "Demo [\\color{SkyBlue}{here}](https://julioasotodv.github.io/ie-c4-466671-diffusion-models/forward_diffusion_demo/Forward_diffusion_demo.html)!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "862ae496-bdd7-40fa-8781-62ffa688b948",
   "metadata": {},
   "source": [
    "## The Nice™ property {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7730dce2-916c-4929-b9fd-f49f9271156f",
   "metadata": {},
   "source": [
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "![](images/ddpm_paper_nice_property.png){height=5.9em fig-align=\"center\"}\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-4pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "Trick to get any $\\mathbf{x}_t$ from $\\mathbf{x}_0$ without having to compute the intermediate steps. Let's define $\\alpha_t \\coloneqq 1 - \\beta_t$ and $\\bar\\alpha_t \\coloneqq \\prod^t_{s=1} \\alpha_s$. We can use the [\\color{SkyBlue}{reparametrization trick for the Normal distribution}](https://sassafras13.github.io/ReparamTrick/#the-math-behind-the-curtain) to get: \\unfootnote{In the paper this is described as \\textit{\"A notable property\"}. I believe that the first to call this as a nice property was \\href{https://lilianweng.github.io/posts/2021-07-11-diffusion-models/}{\\color{SkyBlue}{Weng [2021]}}. We will call it the Nice™ property}\n",
    "\n",
    ":::: {.columns}\n",
    "\n",
    "::: {.column width=\"85%\" align=center}\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{0pt}\n",
    "\\fi\n",
    "```\n",
    "$$q(\\mathbf{x}_t \\mid \\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_t; \\sqrt{\\bar\\alpha_t}\\mathbf{x}_0, (1 - \\bar\\alpha_t)\\mathbf{I})$$ \n",
    "\n",
    "```{=latex}\n",
    "\\onslide<+-> % like pause, but it does not add duplicate slides when tables and footnotes and pauses co-appear (see https://github.com/jgm/pandoc/issues/4390)\n",
    "```\n",
    "\n",
    "- Easier, faster computation\n",
    "- Any image state $\\mathbf{x}_t$ comes from a (Normal) probability distribution, drastically simplyfing derivations\n",
    "\n",
    ":::\n",
    "\n",
    "::: {.column width=\"15%\" align=center}\n",
    "###\n",
    "\\small Details in [\\color{SkyBlue}{Appendix A}](https://julioasotodv.github.io/ie-c4-466671-diffusion-models/Appendices for lectures on diffusion models.html)! \n",
    ":::\n",
    "\n",
    "::::\n",
    "\n",
    "Demo [\\color{SkyBlue}{here}](https://julioasotodv.github.io/ie-c4-466671-diffusion-models/the_nice_property_demo/The_Nice_property_demo.html)!\n",
    "\n",
    "```{=latex}\n",
    "\\label<2>{np}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3471584-ddbd-4821-a743-a59d9b7c2ce5",
   "metadata": {},
   "source": [
    "## The reverse process {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b70d1f8-981a-43d6-a41f-c4dffee04568",
   "metadata": {},
   "source": [
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "![](images/ddpm_paper_reverse_process_highlighted.png){height=4.5em fig-align=center}\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-2pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "We will train a model $p_\\theta$ to learn to perform the reverse process\n",
    "\n",
    "Starting from $p(\\mathbf{x}_T) = \\mathcal{N}(0, \\mathbf{I})$, it will try to recreate the image!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd625755-6ccb-4f6f-bf00-913d736ed0a8",
   "metadata": {},
   "source": [
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-2pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "```{=latex}\n",
    "\\renewcommand{\\eqnhighlightshade}{100}\n",
    "\\renewcommand{\\eqnhighlightheight}{\\vphantom{\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)}\\mathstrut}\n",
    "\n",
    "\\begin{equation*}\n",
    "p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t) \\coloneqq \\mathcal{N}(\\mathbf{x}_{t-1}; \\eqnmark[Green]{mean}{\\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)}, \\eqnmark[Thistle]{variance}{\\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)})\n",
    "\\end{equation*}\n",
    "\\annotate[yshift=-0.5em]{below,left}{mean}{Will be a neural network prediction}\n",
    "\\annotate[yshift=-2em]{below,left}{variance}{Will be set to a value \\begin{math}\\sigma_t^2 \\mathbf{I}\\end{math} based on \\begin{math}\\beta_t\\end{math}}\n",
    "\\renewcommand{\\eqnhighlightshade}{17}\n",
    "\\label<2>{rp}\n",
    "```\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{22pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    ". . .\n",
    "\n",
    "And\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-16pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "$$p_{\\theta}(\\mathbf{x}_{0:T}) \\coloneqq p(\\mathbf{x}_T) \\prod_{t=1}^T p_{\\theta}(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_{t})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9fdecec-d93e-480a-9483-8f03deffa595",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00078c72-8efc-40d2-ab81-5739f2f4989e",
   "metadata": {},
   "source": [
    "![](images/ddpm_process_and_equations.png){width=100% fig-align=\"center\"}\n",
    "\n",
    ". . .\n",
    "\n",
    "The *forward process posterior* is the ground-truth reverse diffusion process that the model will learn to approximate!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d985c40f-7f27-4b14-a6b3-6691970abd02",
   "metadata": {},
   "source": [
    "## Loss function {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "324ad5e3-0b69-4144-8543-9f8f6a21d840",
   "metadata": {},
   "source": [
    "Just like in VAEs, the loss function is based on the Evidence Lower Bound (ELBO):\n",
    "\n",
    "$$\\text{ELBO} = \\mathbb{E}_{q(\\mathbf{x}_{1:T} \\mid \\mathbf{x}_0)} \\left[ \\log \\frac{p_\\theta(\\mathbf{x}_{0 : T})}{q(\\mathbf{x}_{1:T} \\mid \\mathbf{x}_0)} \\right]$$\n",
    "\n",
    ". . .\n",
    "\n",
    "Which becomes:\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-4pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "\\footnotesize $$\\mathbb{E}_q \\left[ \\underbrace {\\mathcal{D}_{\\text{KL}}(q(\\mathbf{x}_T \\mid \\mathbf{x}_0) \\mid \\mid p(\\mathbf{x}_T))}_{L_T} + \\sum_{t=2}^T{\\underbrace{\\mathcal{D}_{\\text{KL}}(q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t, \\mathbf{x}_0) \\mid \\mid p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t))}_{L_{t-1}}} \\underbrace{ - \\log p_\\theta(\\mathbf{x}_0 \\mid \\mathbf{x}_1)}_{L_0} \\right]$$\n",
    "\n",
    ":::: {.columns}\n",
    "\n",
    "::: {.column width=\"83%\" align=center}\n",
    "\n",
    "\\normalsize\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\begin{itemize}\n",
    "\\tightlist\n",
    "\\item\n",
    "  \\(L_T\\) → prior matching term. Has no learnable parameters, so we\n",
    "  ignore\n",
    "\\item\n",
    "  \\(L_{t-1}\\) → denoising term\n",
    "\\item\n",
    "  \\(L_0\\) → reconstruction term. Only learning how to go from\n",
    "  \\(\\mathbf{x}_1\\) to \\(\\mathbf{x}_0\\), so authors ended up ignoring it\n",
    "  (simpler and better results)\n",
    "\\end{itemize}\n",
    "```\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27e33d48-33a4-4546-8c38-c747d6f90e1b",
   "metadata": {},
   "source": [
    "::: {.column width=\"17%\" align=center}\n",
    "\n",
    "```{=latex}\n",
    "\\phantom{\\includegraphics[width=\\linewidth]{images/confused_nick_young_gif/frame_0.png}}\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{-4pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "```{=latex}\n",
    "\\begin{block}{}\n",
    "\\small Details in \\href{https://julioasotodv.github.io/ie-c4-466671-diffusion-models/Appendices\\%20for\\%20lectures\\%20on\\%20diffusion\\%20models.html\\#b.-diffusion-loss-function-elbo-derivation}{\\color{SkyBlue}{Appendix B}}!\n",
    "\\end{block}\n",
    "\\normalsize\n",
    "```\n",
    ":::\n",
    "\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa32ba3c-f6d3-435f-8167-2ec0ce775e38",
   "metadata": {},
   "source": [
    "## Loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c5b8da-e640-4ab3-98d5-0d6002788499",
   "metadata": {},
   "source": [
    "Loss therefore focuses on $L_{t-1}$:\n",
    "\n",
    "$$\\mathcal{D}_{\\text{KL}}(q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t, \\mathbf{x}_0) \\mid \\mid p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t))$$\n",
    "Where:\n",
    "\n",
    "- $q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t, \\mathbf{x}_0)$ is the *forward process posterior* (i.e. what would be the *prefect*, ground-truth reverse process) conditioned on $\\mathbf{x}_0$\n",
    "- $p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t)$ will be our learned reverse process as seen in \\hyperlink{rp}{{\\color{SkyBlue}slide 17}}\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3183cf8a-1f08-45a5-bb13-273acfb15892",
   "metadata": {},
   "source": [
    ":::: {.columns}\n",
    "\n",
    "::: {.column width=\"83%\" align=center}\n",
    "\n",
    ". . .\n",
    "\n",
    "The forward process posterior is tractable and can be computed as:\n",
    "\n",
    "$$q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t, \\mathbf{x}_0) = \\mathcal{N}(\\mathbf{x}_{t-1};{\\color{BlueMean}\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_{t},\\mathbf{x}_{0})},{\\color{red}\\tilde{\\beta}_t \\mathbf{I}}) \\label<2>{fpp}$$\n",
    "\n",
    "Where\n",
    "\\small $${\\color{BlueMean}\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_{t},\\mathbf{x}_{0})} \\coloneqq {\\color{LightBlueMean} \\frac{1}{\\sqrt{\\alpha_t}} \\Big(\\mathbf{x}_t -  \\frac{\\beta_t}{\\sqrt{1-\\bar{\\alpha}_t}}\\boldsymbol{\\epsilon} \\Big)} \\quad \\text{and} \\quad {\\color{red}\\tilde{\\beta}_t} \\coloneqq {\\color{OrangeVar}\\frac{1 - \\bar{\\alpha}_{t-1}}{1 - \\bar{\\alpha}_t} \\cdot \\beta_t}$$ \\normalsize\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2a28176-82a6-41d4-89c5-a2c829b243de",
   "metadata": {},
   "source": [
    "::: {.column width=\"17%\" align=center}\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\normalsize\n",
    "\\ifwindows \n",
    "\\vspace{36pt}\n",
    "\\fi\n",
    "\\phantom{\\includegraphics[width=\\linewidth]{images/confused_math_lady_gif/frame_0.png}}\n",
    "\\ifwindows \n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{block}{}\n",
    "\\small Details in \\href{https://julioasotodv.github.io/ie-c4-466671-diffusion-models/Appendices\\%20for\\%20lectures\\%20on\\%20diffusion\\%20models.html\\#c.-the-forward-process-posterior}{\\color{SkyBlue}{Appendix C}}!\n",
    "\\end{block}\n",
    "\\normalsize\n",
    "```\n",
    "\n",
    ":::\n",
    "\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48abc71e-9b9e-496d-b948-95635e18a45b",
   "metadata": {},
   "source": [
    "## Loss function {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee316958-19b0-46ce-994c-0b9c74934259",
   "metadata": {},
   "source": [
    "Loss is therefore the KL divergence between two Normals: the forward process posterior $q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t, \\mathbf{x}_0)$ and the reverse process that our model will learn $p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t)$\n",
    "\n",
    ". . .\n",
    "\n",
    "\n",
    "Since both are Normal distributions, this KL divergence is:\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-7pt}\n",
    "\\fi\n",
    "\\renewcommand{\\eqnhighlightshade}{100}\n",
    "\\renewcommand{\\eqnhighlightheight}{}\n",
    "\\begin{equation*}\n",
    "\\mathbb{E}_{q} \\left[ \\frac{1}{2 \\sigma_t^2} \\left \\lVert \\eqnmark[BlueMean]{postmean}{\\tilde{\\boldsymbol{\\mu}}_t(\\mathbf{x}_{t},\\mathbf{x}_{0})} - \\eqnmark[Green]{mean}{ \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)} \\right \\rVert^2_2 \\right]\n",
    "\\end{equation*}\n",
    "\\annotate[yshift=-0.5em]{below,left}{postmean}{Forward process posterior mean \\hyperlink{fpp}{({\\color{SkyBlue}slide 20})}}\n",
    "\\annotate[yshift=-2em]{below,left}{mean}{From model's prediction \\hyperlink{rp}{({\\color{SkyBlue}slide 17})}}\n",
    "\\renewcommand{\\eqnhighlightshade}{17}\n",
    "\\ifwindows \n",
    "\\vspace{22pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    ". . .\n",
    "\n",
    "\n",
    "However: authors decide instead to **predict the noise** added during the forward process. Reformulating:\n",
    "\n",
    ":::: {.columns}\n",
    "\n",
    ":::{.column width=\"83%\" align=center}\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-24pt}\n",
    "\\fi\n",
    "\\renewcommand{\\eqnhighlightshade}{100}\n",
    "\\renewcommand{\\eqnhighlightheight}{\\vphantom{\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}\\mathstrut}\n",
    "\\begin{equation*}\n",
    "L_\\text{simple} \\coloneqq \\mathbb{E}_{q} \\left[ \\left \\lVert \\eqnmark[LightBlueMean]{eps}{\\boldsymbol{\\epsilon}} - \\eqnmark[GreenYellow]{epstheta}{\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)} \\right \\rVert^2_2 \\right]\n",
    "\\end{equation*}\n",
    "\\annotate[yshift=-0.5em]{below,left}{eps}{Added noise in forward pass}\n",
    "\\annotate[yshift=-2em]{below,left}{epstheta}{Model predicting the noise using \\begin{math} \\mathbf{x}_t \\end{math} and \\begin{math} t \\end{math} as features}\n",
    "\\renewcommand{\\eqnhighlightshade}{17}\n",
    "```\n",
    "\n",
    ":::\n",
    "\n",
    ":::{.column width=\"17%\" align=center}\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-10pt} \n",
    "\\fi \n",
    "\\phantom{\\includegraphics[width=\\linewidth]{images/spongebob_magic_gif/frame_8.png}}\n",
    "\\ifwindows \n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{block}{}\n",
    "\\small Details in \\href{https://julioasotodv.github.io/ie-c4-466671-diffusion-models/Appendices\\%20for\\%20lectures\\%20on\\%20diffusion\\%20models.html\\#d.-objective-the-training-procedure}{\\color{SkyBlue}{Appendix D}}!\n",
    "\\end{block}\n",
    "\\normalsize\n",
    "```\n",
    "\n",
    ":::\n",
    "\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e78734e-5712-49f6-8466-f1b02cd8442a",
   "metadata": {},
   "source": [
    "## Training algorithm {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c9d0101-c793-4e9c-98e5-c0e7a59737ad",
   "metadata": {},
   "source": [
    "![](images/ddpm_training_algo.png){height=50% fig-align=\"center\"}\n",
    "```{=latex}\n",
    "\\ifwindows \n",
    "\\vspace{-18pt} \n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Ho et al. [2020]\n",
    "\\end{center}\n",
    "\\normalsize \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af6048c3-cbc3-4e37-86e0-37bcdb594c8d",
   "metadata": {},
   "source": [
    ". . .\n",
    "\n",
    "Where $\\sqrt{\\bar\\alpha_t}\\mathbf{x}_0 + \\sqrt{1 + \\bar\\alpha_t}\\boldsymbol\\epsilon$ is just $\\mathbf{x}_t$ computed through \\hyperlink{np}{{\\color{SkyBlue}the Nice™ property}}!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fec3a2a2-bbdf-4367-bfdd-e5a3de2086fa",
   "metadata": {},
   "source": [
    "## The model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1adcf15d-65f5-45e4-a636-2c81fd1c978e",
   "metadata": {},
   "source": [
    "Proposed model is a U-Net architecture ([\\color{SkyBlue}{Ronneberger et al. [2015]}](https://arxiv.org/abs/1505.04597)) that includes self-attention blocks\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{-6pt}\n",
    "\\fi\n",
    "\n",
    "![](images/ddpm_unet.drawio.png){height=55% fig-align=\"center\"}\n",
    "\n",
    ". . .\n",
    "\n",
    "\\ifwindows \n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\n",
    "They also include GroupNorm ([\\color{SkyBlue}{Wu and He [2018]}](https://arxiv.org/abs/1803.08494v3)) in ResNet and self-attention blocks\n",
    "\n",
    "$t$ is added on every ResNet block through positional encoding ([\\color{SkyBlue}{Vaswani et al. [2017]}](https://arxiv.org/abs/1706.03762))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d744f16-1aa1-4f81-acc4-85243c086ae2",
   "metadata": {},
   "source": [
    "## Sampling algorithm {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4002049d-5379-4b90-bec4-fbd7753fc235",
   "metadata": {},
   "source": [
    "Once the model is trained, we can generate new images by:\n",
    "\n",
    "![](images/ddpm_sampling_algo.png){height=50% fig-align=\"center\"}\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Ho et al. [2020]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "\\label{sampling}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5976acb-8982-405d-b0d1-71f71dd2debc",
   "metadata": {},
   "source": [
    ". . .\n",
    "\n",
    "Step 4 just applies the reparametrization trick to the \\hyperlink{rp}{{\\color{SkyBlue}learned reverse process}} $p_\\theta(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t) = \\mathcal{N}(\\mathbf{x}_{t-1}; {\\color{Green} \\boldsymbol{\\mu}_\\theta(\\mathbf{x}_t, t)}, {\\color{Thistle} \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)})$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55217e4f-545b-4afe-b1a2-af31353909df",
   "metadata": {},
   "source": [
    "## Sampling algorithm {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2220eb0-32ee-4276-8999-feef91d8d3a3",
   "metadata": {},
   "source": [
    "Sampling is an iterative process: we progressively remove predicted noise\n",
    "\n",
    "![](images/ddpm_sampling_algo_highlight.png){height=50% fig-align=\"center\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c6edea8-a050-47fb-84c7-cdad24615141",
   "metadata": {},
   "source": [
    ":::: {.columns}\n",
    "\n",
    ":::{.column width=\"83%\" align=center}\n",
    "\n",
    "\\ifwindows\n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "\n",
    ". . .\n",
    "\n",
    "You may wonder:\n",
    "\n",
    "> *If at any single step we are predicting the full added noise ${\\color{LightBlueMean}\\boldsymbol{\\epsilon}}$, why don't we remove it completely in a single step?*\n",
    "\n",
    ":::\n",
    "\n",
    ":::{.column width=\"17%\" align=center}\n",
    "\n",
    "\\ifwindows\n",
    "\\vspace{0pt}\n",
    "\\fi\n",
    "\n",
    ". . .\n",
    "\n",
    "Answer: \n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-4pt}\n",
    "\\fi\n",
    "\\begin{block}{}\n",
    "\\small Details in \\href{https://julioasotodv.github.io/ie-c4-466671-diffusion-models/Appendices\\%20for\\%20lectures\\%20on\\%20diffusion\\%20models.html\\#e.-the-sampling-procedure}{\\color{SkyBlue}{Appendix E}}!\n",
    "\\end{block}\n",
    "\\normalsize\n",
    "```\n",
    ":::\n",
    "\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3febbdde-6be7-43f6-8202-94225ca3cd97",
   "metadata": {},
   "source": [
    "# Advancements and improvements"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccf872c4-b0a7-417a-8405-3fc84834cf05",
   "metadata": {},
   "source": [
    "## Advancements and improvements\n",
    "\n",
    "\\large Outline\n",
    "\n",
    "\\normalsize\n",
    "\n",
    "- Variance/noise schedulers\n",
    "- Learning the reverse process variance\n",
    "- Faster sampling: DDIMs\n",
    "- Conditional generation\n",
    "  - Classifier Guidance\n",
    "  - Classifier-Free Guidance\n",
    "  - Conditioning on images\n",
    "  - ControlNet\n",
    "  - Conditioning on text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30b007d7-c34d-46c5-95de-d92ff30edeb6",
   "metadata": {},
   "source": [
    "## Variance/noise schedulers {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "661fe480-4353-4664-a52d-82beb73a5789",
   "metadata": {},
   "source": [
    "[\\color{SkyBlue}{Nichol and Dhariwal [2021]}](https://arxiv.org/abs/2102.09672) propose a different way to set $\\beta_t$:\n",
    "\n",
    "$$\\bar{\\alpha}_t = \\frac{f(t)}{f(0)}, \\qquad \\text{where} f(t) = \\cos\\left( \\frac{t / T + s}{1+s} \\cdot \\frac{\\pi}{2}  \\right)^2$$\n",
    "\n",
    "And then $\\beta_t = \\min\\left( 1 - \\frac{\\bar{\\alpha}_t}{\\bar{\\alpha}_{t-1}} ,\\, 0.999\\right )$. $s$ is a small offset to prevent $\\beta_t$ from being tiny when $t$ is close to 0 (they set it to $s=0.008$)\n",
    "\n",
    "![](images/cosine_schedule.png){height=11em fig-align=\"center\"}\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-10pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Comparison between scheduler in DDPM and Nichol \\& Dhariwal's \\\\ cosine scheduler proposal. Source: Nichol \\& Dhariwal [2021]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3757aa45-6cf0-48de-9098-ed794661276d",
   "metadata": {},
   "source": [
    "## Variance/noise schedulers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c737e97-20de-4ab6-b0c1-f7ff2251d763",
   "metadata": {},
   "source": [
    "![](images/cosine_scheduler_dogs.png){height=11em fig-align=\"center\"}\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Nichol \\& Dhariwal [2021]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d5d7355-1fc8-4ecb-8395-b2ac5f5e10f4",
   "metadata": {},
   "source": [
    "More progressive forward diffusion process, especially for large values of $t$\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Demo [\\color{SkyBlue}{here}](https://julioasotodv.github.io/ie-c4-466671-diffusion-models/the_nice_property_demo/The_Nice_property_demo.html)!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b66b8e6b-e681-49c3-a024-f90c73e1e95b",
   "metadata": {},
   "source": [
    "## Learning the reverse process variance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce89a95-a300-47da-8593-1c338f6c6db3",
   "metadata": {},
   "source": [
    "Nichol \\& Dhariwal [2021] also proposed to learn the \\hyperlink{rp}{{\\color{SkyBlue}reverse process variance}} ${\\color{Thistle} \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)}$ instead of setting it upfront based on $\\beta_t$:\n",
    "\n",
    "$$ {\\color{Thistle} \\boldsymbol{\\Sigma}_\\theta(\\mathbf{x}_t, t)} = \\exp(v \\log \\beta_t + (1- v) \\log {\\color{red}\\tilde{\\beta_t}})$$\n",
    "\n",
    "So the learned variance is an interpolation between $\\beta_t$ and ${\\color{red}\\tilde{\\beta_t}}$ controlled by $v$, which is a mixing vector predicted by the model\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    ". . .\n",
    "\n",
    "Loss function is changed accordingly to include this (called now $L_{\\text{hybrid}}$ in the paper)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf3a1c7f-9d75-47ad-80a6-4a107d30ed54",
   "metadata": {},
   "source": [
    "## Faster sampling: DDIMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec9f0bf6-b1eb-4cac-850e-6c85dd70ee8a",
   "metadata": {},
   "source": [
    "The sampling algorithm in DDPMs is slow, as it requires $T$ iterations → slow image generation\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "In *Denoising Diffusion Implicit Models* or **DDIMs**, [\\color{SkyBlue}{Song et al. [2020]}](https://arxiv.org/abs/2010.02502) made the diffusion process non-Markovian, so each step $t$ does not depend only on last step\n",
    "\n",
    ". . .\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "This allows us to \"skip\" steps during the sampling process\n",
    "\n",
    ". . .\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "How? Predicting how to get from $\\mathbf{x_t}$ to $\\mathbf{x}_0$, and then \"go back\" from $\\mathbf{x}_0$ to for instance a $\\mathbf{x}_{t-2}$. Therefore, that iteration brings us from $\\mathbf{x_t}$ directly to $\\mathbf{x}_{t-2}$ (skipping 1 sampling step)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e913104e-ea7d-4a14-a62c-da555d451af7",
   "metadata": {},
   "source": [
    "## Faster sampling: DDIMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6da13a45-bef5-4906-beca-ce0a5d3f680a",
   "metadata": {},
   "source": [
    "![](images/ddim.png){height=9em fig-align=\"center\"}\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny DDIM for accelerated sampling (in this diagram it is used to jump from \\begin{math}\\mathbf{x}_t\\end{math} directly to \\begin{math}\\mathbf{x}_{t-2}\\end{math})\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "```\n",
    "\\small $$\\mathbf{x}_{t-2} = \\sqrt{\\bar\\alpha_{t-2}} \\underbrace{\\left( \\frac{\\mathbf{x}_t - \\sqrt{1 - \\bar\\alpha_t} {\\color{GreenYellow} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}}{\\sqrt{\\bar\\alpha_t}} \\right)}_{\\text{predicted } \\mathbf{x}_0} + \\underbrace{\\sqrt{1 - \\bar\\alpha_{t-2} - \\sigma^2_t} \\cdot {\\color{GreenYellow} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}}_{\\text{direction pointing to } \\mathbf{x_{t}}} + \\underbrace{\\sigma_t \\mathbf{z}}_{\\text{random noise}}$$\n",
    "\n",
    ". . .\n",
    "\n",
    "We can choose freely how many steps to skip per iteration (but quality can decrease if we skip too many!)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af613274-3075-4776-9aea-e531020086f7",
   "metadata": {},
   "source": [
    "## Faster sampling: DDIMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c67e2bc5-a36c-4d68-ad04-28c6a3b197b8",
   "metadata": {},
   "source": [
    "Most models can generate good quality images with few steps (such as 20 or 50)\n",
    "\n",
    "![](images/ddim_samples.png){height=12em fig-align=\"center\"}\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Song et al. [2020]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "```\n",
    ". . .\n",
    "\n",
    "Many other sampling algorithm variants have been proposed after DDIM—especially [\\color{SkyBlue}{for Stable Diffusion models}](https://stable-diffusion-art.com/samplers/)!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c05283-153d-467e-871d-a6923aa5ea20",
   "metadata": {},
   "source": [
    "## Faster sampling: DDIMs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f66ec3f-6160-46cd-a737-920ae5a29080",
   "metadata": {},
   "source": [
    "\\small $$\\mathbf{x}_{t-2} = \\sqrt{\\bar\\alpha_{t-2}} \\underbrace{\\left( \\frac{\\mathbf{x}_t - \\sqrt{1 - \\bar\\alpha_t} {\\color{GreenYellow} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}}{\\sqrt{\\bar\\alpha_t}} \\right)}_{\\text{predicted } \\mathbf{x}_0} + \\underbrace{\\sqrt{1 - \\bar\\alpha_{t-2} - \\sigma^2_t} \\cdot {\\color{GreenYellow} \\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)}}_{\\text{direction pointing to } \\mathbf{x_{t}}} + \\underbrace{\\sigma_t \\mathbf{z}}_{\\text{random noise}}$$\n",
    "\n",
    "\\normalsize Authors also state that $\\sigma_t$ can be set to $0$, making the sampling algorithm deterministic\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    ". . .\n",
    "\n",
    "If done, the model is an *implicit probabilistic* one, hence the \"I\" in the name DDIM\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    ". . .\n",
    "\n",
    "Training does not change (same as in regular DDPMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49f6784c-4839-4a4b-9ad1-bd37eebf3c8d",
   "metadata": {},
   "source": [
    "## Conditional generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9ea8ab-fcb1-4652-afd0-6ba564c37159",
   "metadata": {},
   "source": [
    "So far the diffusion models we have seen generate *any* image from pure noise; we don't have control over it\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "But it is much more useful to tell the model in some way what kind of image we want to generate (e.g. dog, house, \"a polar bear with sunglasses surfing in space\"...)\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "**Conditional generation**: allows us to \"inject\" additional data to the model to obtain a specific kind of image\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "That additional data can be a class label (e.g. 0→dog, 1→cat, 2→house), a text prompt (a polar bear with sunglasses surfing in space), another image..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7398043f-bf34-4b90-869d-eb2d05f8fd3a",
   "metadata": {},
   "source": [
    "## Conditional generation: Classifier Guidance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5438a691-0011-4fe5-bc68-0a94b50a6db8",
   "metadata": {},
   "source": [
    "To perform conditional generation, [\\color{SkyBlue}{Dhariwal and Nichol [2021]}](https://arxiv.org/abs/2105.05233) trained a regular image classifier $p_\\phi(y \\mid \\mathbf{x}_t, t)$ using partially noisy images\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "They used the classifier gradients $\\nabla_{\\mathbf{x}_t} \\log{p_\\phi(y \\mid \\mathbf{x}_t, t)}$ to guide the sampling algorithm towards $y$ (the class label e.g. 0→dog, 1→cat, 2→house)\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "To do so, for the \\hyperlink{sampling}{{\\color{SkyBlue}sampling algorithm}} they replace ${\\color{GreenYellow}\\boldsymbol{\\epsilon}_\\theta}$ with $\\hat{\\boldsymbol\\epsilon}$, which is:\n",
    "\n",
    "$$ \\hat{\\boldsymbol\\epsilon}(\\mathbf{x}_t, t) = {\\color{GreenYellow}\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t)} - \\sqrt{1 - \\bar\\alpha_t} \\cdot s \\cdot \\nabla_{\\mathbf{x}_t} \\log{p_\\phi(y \\mid \\mathbf{x}_t, t)}$$\n",
    "\n",
    "Where $s$ is a weight/scale factor that controls the guidance strength. Higher $s$→higher fidelity (but less diverse) images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe41e9c7-b75e-445a-95f7-5a782a67cc45",
   "metadata": {},
   "source": [
    "## Conditional generation: Classifier-Free Guidance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fca8bb5-b8c2-4fc9-8fcf-ed5fdd59fa47",
   "metadata": {},
   "source": [
    "Cons of Classifier Guidance → having to train a separate classifier. Also: most information in the image $\\mathbf{x}_t$ is not relevant for predicting $y$, and therefore taking its gradient w.r.t. $\\mathbf{x}_t$ can yield somehow arbitrary guidance\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "[\\color{SkyBlue}{Ho and Salimans [2022]}](https://arxiv.org/abs/2207.12598) proposed Classifier-Free Guidance, which avoids training a separate classifier\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Instead, we can get an \"implicit classifier\" by jointly training a conditional and unconditional diffusion model. Applying Bayes' theorem:\n",
    "\n",
    "$$\\underbrace{p(y \\mid \\mathbf{x}_t, t)}_\\text{classifier} \\propto \\underbrace{p_\\theta(\\mathbf{x}_t \\mid y, t)}_{\\substack{\\text{conditional} \\\\ \\text{diffusion model}}} \\,\\, / \\underbrace{p_\\theta(\\mathbf{x}_t \\mid t)}_{\\substack{\\text{unconditional} \\\\ \\text{diffusion model}}}$$\n",
    "\n",
    "Both conditional and unconditional models can be the same one by training a conditional model $p_\\theta(\\mathbf{x}_t \\mid y, t)$ for which the class $y$ gets dropped at random during training with some probability (similar to what happens in dropout)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d3800f-f262-42ee-a9d6-bf124b3b6c1d",
   "metadata": {},
   "source": [
    "## Conditional generation: Classifier-Free Guidance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18ad21fa-7032-42eb-b494-663a28bcbf50",
   "metadata": {},
   "source": [
    "Specifically, we can say that our model becomes ${\\color{GreenYellow}\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t}, y{\\color{GreenYellow})}$ with some probability of $y = \\varnothing$ (this is, we sometimes feed the model with a special null class identifier instead of the real one during training) \\unfootnote{The paper uses slightly different notation: they use $\\mathbf{c}$ instead of $y$ and $t$ for the class and timestep feature respectively (they group them into $\\mathbf{c}$), and use $\\mathbf{z}_\\lambda$ instead of $\\mathbf{x}_t$}\n",
    "\n",
    "```{=latex}\n",
    "\\onslide<+->\n",
    "```\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "In the \\hyperlink{sampling}{{\\color{SkyBlue}sampling algorithm}} they replace ${\\color{GreenYellow}\\boldsymbol{\\epsilon}_\\theta}$ with $\\tilde{\\boldsymbol{\\epsilon}}_\\theta$, which is a combination of the unconditional and conditional predictions:\n",
    "\n",
    "$$\\tilde{\\boldsymbol{\\epsilon}}_\\theta(\\mathbf{x}_t, t, y) = (1 + w) \\cdot {\\color{GreenYellow}\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t}, y{\\color{GreenYellow})} - w \\cdot {\\color{GreenYellow}\\boldsymbol{\\epsilon}_\\theta(\\mathbf{x}_t, t}, y=\\varnothing{\\color{GreenYellow})}$$\n",
    "\n",
    "Where $w$ controls the guidance strength"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3b77d90-48d1-44b9-a678-ad7825bcef69",
   "metadata": {},
   "source": [
    "## Conditional generation: Classifier-Free Guidance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "188b1f23-4c62-4563-a293-12a7d990babb",
   "metadata": {},
   "source": [
    "```{=latex}\n",
    "\n",
    "\\begin{center}\n",
    "\\tiny Source: Ho \\& Salimans [2022]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "\\ifwindows\n",
    "\\vspace{-14pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    ":::: {.columns}\n",
    "\n",
    "::: {.column width=\"33%\" align=center}\n",
    "![](images/classifier_guidance_1.png){width=100% fig-align=\"center\"}\n",
    "\\begin{center}No guidance\\end{center}\n",
    ":::\n",
    "\n",
    "::: {.column width=\"33%\" align=center}\n",
    "![](images/classifier_guidance_2.png){width=100% fig-align=\"center\"}\n",
    "\\begin{center}$w=1$\\end{center}\n",
    ":::\n",
    "\n",
    "::: {.column width=\"33%\" align=center}\n",
    "![](images/classifier_guidance_3.png){width=100% fig-align=\"center\"}\n",
    "\\begin{center}$w=3$\\end{center}\n",
    ":::\n",
    "\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "214277dd-5d8c-45a8-922f-8c8d0823a6f8",
   "metadata": {},
   "source": [
    "## Conditional generation: Conditioning on images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8e89b74-73f3-4616-8b28-5124004363dd",
   "metadata": {},
   "source": [
    "We can condition a diffusion model on something other than a class label. For instance, with other images\n",
    "\n",
    "Palette by [\\color{SkyBlue}{Saharia et al. [2022]}](https://arxiv.org/abs/2111.05826) performs image-to-image translation by training $p_\\theta(\\mathbf{x}_t \\mid y, t)$ where $y$ is an image we use to condition generation (given as input data to the model both during training and sampling)\n",
    "\n",
    ". . .\n",
    "\n",
    "They use it to perform image colorization, where $y$ is a black and white image that we want to colorize. Also for inpainting, image restoration...\n",
    "\n",
    ":::: {.columns}\n",
    "\n",
    "::: {.column width=\"50%\" align=center}\n",
    "![](images/palette_1.png){width=100% fig-align=\"center\"}\n",
    ":::\n",
    "\n",
    "::: {.column width=\"50%\" align=center}\n",
    "![](images/palette_2.png){width=100% fig-align=\"center\"}\n",
    ":::\n",
    "\n",
    "::::\n",
    "\\ifwindows\n",
    "\\vspace{-10pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Saharia et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43c0a32-29ee-42fd-818c-726436dcb134",
   "metadata": {},
   "source": [
    "## Conditional generation: ControlNet {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76389bbd-aab3-47c8-8f9d-bf2b0e355bdd",
   "metadata": {},
   "source": [
    "[\\color{SkyBlue}{Zhang et al. [2023]}](https://arxiv.org/abs/2302.05543) proposed ControlNet, where they fine-tuned a diffusion model to include extra conditioning such as sketch images, depth maps, human pose data, image segmentation data and others\n",
    "\n",
    ". . .\n",
    "\n",
    "They did architectural changes to the diffusion model to accomodate the additional conditioning info, and then fine-tuned the model with datasets that include that extra conditioning data:\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-4pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "![](images/controlnet_arch.png){height=10em fig-align=\"center\"}\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-18pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny ControlNet block. They freeze the original pre-trained model layer, and make a trainable copy adding zero convolutions (1x1 conv layers with weights intialized to zeros) before and after the copied layer. $x$ is what we know as $\\mathbf{x}_t$ and $\\mathbf{c}$ is the extra conditioning image/data. Frozen and ControlNet block outputs are added together. Source: Zhang et al. [2023]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55b27647-921f-42d7-89c6-290c51d1bad6",
   "metadata": {},
   "source": [
    "## Conditional generation: ControlNet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0c90558-3922-4f54-8b7f-238bbb8f37af",
   "metadata": {},
   "source": [
    "![](images/controlnet_examples.png){width=100% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Zhang et al. [2023]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29c5d24e-5913-4342-a964-d8261de17a7c",
   "metadata": {},
   "source": [
    "## Conditional generation: Conditioning on text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca8c2063-2f9b-413a-9bf7-b83fe0c8f2ec",
   "metadata": {},
   "source": [
    "Diffusion models can be also conditioned in natural language through a *prompt* e.g. \"a polar bear with sunglasses surfing in space\"\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "This can be done with Classifier-Free Guidance, but making $y$ be text (a list of tokens) instead of a class label, and adding some text-processing specific layers to the diffusion model (such as Transformer-like layers)\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Another option is to use a pre-trained text model and use its outputs to condition the diffusion model\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "A way to do this is to use *CLIP guidance*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b30dfdf-c868-40fe-811f-245b2faace28",
   "metadata": {},
   "source": [
    "## Conditional generation: Conditioning on text {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2385e11-e7e3-487f-8643-1a53d0626d17",
   "metadata": {},
   "source": [
    "CLIP by [\\color{SkyBlue}{Radford et al. [2021]}](https://arxiv.org/abs/2103.00020) is a discriminative (non-generative) model that uses contrastive learning to match images with their natural language descriptions\n",
    "\n",
    "Made of a text encoder (Transformer) and an image encoder (Vision Transformer), it embeds both text and images into a common dimensional space\n",
    "\n",
    ". . .\n",
    "\n",
    "\n",
    "The model is trained to maximize the (cosine) similarity between matching (image, text) pairs in this dimensional space\n",
    "\n",
    "```{=latex}\n",
    "\\ifwindows\n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "```\n",
    "\n",
    "![](images/clip.png){height=12em fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny CLIP learns to maximize the cosine similarity (which can be thought of as correlation) between images and texts that go together. It therefore learns a sort of \"correlation matrix\" between training images $\\text{I}_1, \\text{I}_2, \\ldots, \\text{I}_N$ and training textual descriptions $\\text{T}_1, \\text{T}_2, \\ldots, \\text{T}_N$. Source: Radford et al. [2021]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92a320a5-8f2e-4d9d-848f-aff36ed47de3",
   "metadata": {},
   "source": [
    "## Conditional generation: Conditioning on text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "593efd9d-2e82-4612-981d-d8da49c0ebfa",
   "metadata": {},
   "source": [
    "Once the CLIP model is trained, we can feed a text prompt to the text encoder and use the output as the conditioning for a diffusion model\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "CLIP guidance is applied in large diffusion models such as GLIDE and DALL-E 2, which we will cover next"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68ce8cd2-f629-4222-8522-bfc4ba161592",
   "metadata": {},
   "source": [
    "# Large diffusion models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb080885-90e6-4edf-bb86-8d86827766a2",
   "metadata": {},
   "source": [
    "## Large diffusion models\n",
    "\n",
    "\\large Outline\n",
    "\n",
    "\\normalsize\n",
    "\n",
    "- GLIDE\n",
    "- DALL-E\n",
    "- Imagen\n",
    "- Stable Diffusion\n",
    "- Flux\n",
    "- Sana"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6541a32-7074-44fa-8d3c-c7df5b96139c",
   "metadata": {},
   "source": [
    "## GLIDE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c44e2414-61e6-4b65-b600-92b6b7f84815",
   "metadata": {},
   "source": [
    "GLIDE (**G**uided **L**anguage to **I**mage **D**iffusion for generation and **E**diting) by [\\color{SkyBlue}{Nichol et al. [2021]}](https://arxiv.org/abs/2112.10741) is a text-to-image model made of three sub-models:\n",
    "\n",
    "- A base diffusion model that generates 64x64 images\n",
    "- An upsampling diffusion model that increases the resolution to 256x256\n",
    "- A Transformer-based text encoder to condition generation on text prompts (Classifier-Free Guidance)\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "They also tried to replace the Transformer with a pre-trained CLIP, but the Classifier-Free Guidance method worked better"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5736c858-805f-4cce-b0e7-e1ee860a2454",
   "metadata": {},
   "source": [
    "## GLIDE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3537a00a-1e9f-4230-93ea-73d74026beb9",
   "metadata": {},
   "source": [
    "![](images/glide.png){width=90% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Nichol et al. [2021]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bb6c482-6c86-47f9-be1b-4e7030486075",
   "metadata": {},
   "source": [
    "## DALL-E"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13993098",
   "metadata": {},
   "source": [
    "Dall-E 1 by  [\\color{SkyBlue}{Ramesh et al. [2021]}](https://arxiv.org/abs/2102.12092) was not a diffusion model, but Dall-E 2 and 3 are\n",
    "\n",
    "Dall-E 2 (aka unCLIP) by [\\color{SkyBlue}{Ramesh et al. [2022]}](https://arxiv.org/abs/2204.06125) uses a pre-trained CLIP model, and the text-to-image model itself is made of two main components:\n",
    "\n",
    "- A *prior* model which produces image embeddings based on the text encoded by CLIP\n",
    "- A *decoder* model that generates the actual image based on the Prior's output and the original prompt\n",
    "\n",
    ". . .\n",
    "\n",
    "![](images/dalle2_annotated_1.png){width=80% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: adapted from Ramesh et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "031d16fa",
   "metadata": {},
   "source": [
    "## DALL-E {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27cac0bf",
   "metadata": {},
   "source": [
    "![](images/dalle2_annotated_2.png){width=80% fig-align=\"center\"}\n",
    "\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\n",
    "Authors tried two variants for the *prior* model:\n",
    "\n",
    "- An autoregressive one (top one inside the red rectangle), where the image embeddings are generated as a discrete set of tokens (like in LLMs)\n",
    "- A diffusion one (bottom on inside the red rectangle)\n",
    "\n",
    "The diffusion prior worked better for them"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc6b3182",
   "metadata": {},
   "source": [
    "## DALL-E {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344e7421",
   "metadata": {},
   "source": [
    "![](images/dalle2_annotated_3.png){width=80% fig-align=\"center\"}\n",
    "\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\n",
    "The decoder model is actually 3 diffusion models:\n",
    "\n",
    "- One to generate 64x64 images\n",
    "- Another one to upscale to 256x256\n",
    "- A third one to upscale to 1024x1024\n",
    "\n",
    "The prior output intermediate representation can be tweaked, allowing for text-guided image manipulation tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f07cc7f",
   "metadata": {},
   "source": [
    "## DALL-E"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d72709",
   "metadata": {},
   "source": [
    "![](images/dalle2_examples_1.png){width=100% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Ramesh et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f285504a-6883-4ea8-9e9f-9019f587b664",
   "metadata": {},
   "source": [
    "## DALL-E"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9469619",
   "metadata": {},
   "source": [
    "![](images/dalle2_examples_2.png){height=85% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-14pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Ramesh et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "491148b2",
   "metadata": {},
   "source": [
    "## DALL-E"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f54c6218",
   "metadata": {},
   "source": [
    "DALL-E 3 by [\\color{SkyBlue}{Betker et al. [2023]}](https://cdn.openai.com/papers/dall-e-3.pdf) improves on DALL-E 2 mostly by curating a better dataset\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "They train an image captioner to generate better textual descriptions of images\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "\n",
    "They also use GPT-4 to \"upsample\" (extend) text prompts before generating the image\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Model architecture is undisclosed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dde49a8",
   "metadata": {},
   "source": [
    "## Imagen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "068b8c14-d190-402b-9e2d-fbc46603ef4c",
   "metadata": {},
   "source": [
    "Imagen by [\\color{SkyBlue}{Saharia et al. [2022]}](https://arxiv.org/abs/2205.11487) also uses \"cascaded\" diffusion models\n",
    "\n",
    "\n",
    "Uses a pre-trained text encoder LLM (T5-XXL by [\\color{SkyBlue}{Raffel et al. [2019]}](https://arxiv.org/abs/1910.10683)) to generate text embeddings → much more effective than CLIP and quality improves with the LLM size (increasing text LLM size > increasing diffusion models size)\n",
    "\n",
    ". . .\n",
    "\n",
    "Used models are *Efficient U-Nets* which slightly alter the original U-Net\n",
    "\n",
    "\\ifwindows\n",
    "\\vspace{-4pt}\n",
    "\\fi\n",
    "\n",
    "![](images/imagen_1.png){height=55% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-13pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Saharia et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a06ca00-9cb0-45e4-9ab1-c61e19b01e47",
   "metadata": {},
   "source": [
    "## Imagen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad87d5c1-99ee-40e8-abb5-bffc42228814",
   "metadata": {},
   "source": [
    "![](images/imagen_examples_1.png){height=80% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Saharia et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f19ad886-2f2b-4b3e-b313-9f21bed061fa",
   "metadata": {},
   "source": [
    "## Imagen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a44fed71-62c8-4173-8f1b-49176e284c45",
   "metadata": {},
   "source": [
    "Imagen 2 (2024) and 3 ([\\color{SkyBlue}{Imagen 3 team [2024]}](https://arxiv.org/abs/2408.07009)) are more recent, but their architectures remain largely undisclosed\n",
    "\n",
    "![](images/imagen_examples_2.png){height=65% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Imagen 3 team [2024]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7763c76-067f-402e-9036-8d8e10eb67bf",
   "metadata": {},
   "source": [
    "## Stable Diffusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ebdbeed-e227-4fcd-9edf-46b663b85613",
   "metadata": {},
   "source": [
    "[\\color{SkyBlue}{Rombach et al. [2022]}](https://arxiv.org/abs/2408.07009) published a **Latent Diffusion Model** (LDM), which would serve as the basis for Stable Diffusion\n",
    "\n",
    "LDM makes image generation more efficient by running the diffusion process in a compressed, latent space instead of in the original image space\n",
    "\n",
    ". . .\n",
    "\n",
    "It uses an Autoencoder to project images into that latent space, and runs the diffusion there\n",
    "\n",
    "![](images/latent_diffusion.png){height=40% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny (Read in order: top-left→top-right→bottom-right→bottom-left): An encoder $\\mathcal{E}$ projects an image $x$ into a latent representation of it named as $z$, which then goes through the forward diffusion process (called Diffusion Process in the image) to get the noisy latent $z_T$. Then, the U-Net performs the reverse diffusion process to try to recover $z$, which gets unprojected using the decoder $\\mathcal{D}$ back to a generated image $\\tilde{x}$. Different conditionings (text, other images, etc) can be used to guide the generation process thanks to the condition encoders $\\tau_{\\theta}$ and the crossattention layers. Source: Rombach et al. [2022]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d84f3851-559c-431f-b59a-7533fe08e6d7",
   "metadata": {},
   "source": [
    "## Stable Diffusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6aa7898-610c-4558-ac7c-17b0a1c8e071",
   "metadata": {},
   "source": [
    "In June 2022, [\\color{SkyBlue}{Stability AI}](https://stability.ai/) and [\\color{SkyBlue}{Runway}](https://runwayml.com/) released an open LDM which they called Stable Diffusion\n",
    "\n",
    "[\\color{SkyBlue}{Versions 1.X}](https://github.com/CompVis/stable-diffusion) (June - Oct 2022) used OpenAI's pretrained CLIP for text conditioning and were capable of generating 512x512 images\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "[\\color{SkyBlue}{Versions 2.X}](https://github.com/Stability-AI/stablediffusion) (Nov - Dec 2022) used a CLIP model trained by themselves, which was open sourced as [\\color{SkyBlue}{OpenCLIP}](https://github.com/mlfoundations/open_clip). Also increased resolution to 768x768\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "XL versions (July - Nov 2023, [\\color{SkyBlue}{Podell et al. [2023]}](https://arxiv.org/abs/2307.01952)) made the U-Net 3x larger and introduced a two-stage process: a base model that generates an image, and a *refiner* model that additional high-quality details to it. Resolution was increased to 1024x1024\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Version 3 ([\\color{SkyBlue}{Esser et al. [2024]}](https://arxiv.org/abs/2403.03206)) made many architectural changes, such as replacing CLIP as the text encoder with T5-XXL ([\\color{SkyBlue}{Raffel et al. [2020]}](https://arxiv.org/abs/1910.10683)). But its release was controversial (both because of its unclear licensing and [\\color{SkyBlue}{generation issues}](https://www.bentoml.com/blog/stable-diffusion-3-text-master-prone-problems))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e359d217-6b99-4fd7-adea-50ea90d08886",
   "metadata": {},
   "source": [
    "## Stable Diffusion"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d8297d50-02ce-4f38-a346-9a50cd475dd9",
   "metadata": {},
   "source": [
    "![](images/sd_generations.jpeg){height=80% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Sources: https://github.com/CompVis/stable-diffusion, https://github.com/Stability-AI/stablediffusion, https://github.com/huggingface/diffusers/blob/main/docs/source/en/using-diffusers/sdxl.md, https://huggingface.co/stabilityai/stable-diffusion-3-medium [2024]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46608e34-342e-4652-b0d9-c8fa21edd0bf",
   "metadata": {},
   "source": [
    "## Flux"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95f8f9c0-4401-405b-8800-76f4759f342c",
   "metadata": {},
   "source": [
    "Authors of the LDM model founded [\\color{SkyBlue}{Black Forest Labs}](https://blackforestlabs.ai/announcing-black-forest-labs/) in Aug 2024 and released inference code for a model called [\\color{SkyBlue}{Flux}](https://github.com/black-forest-labs/flux)\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Architecturally very similar to Stable Diffusion 3, it replaces the U-Net with a **Diffusion Transformer (DiT)** ([\\color{SkyBlue}{Peebles and Xie [2023]}](https://arxiv.org/abs/2212.09748))\n",
    "\n",
    "![](images/DiT.png){height=50% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-12pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Just like LDM, DiT operates on the latent space. Like Vision Transformers (ViTs), it transforms the data into a sequence of \"patches\", which go through $N$ DiT blocks. They tried three different block architectures, being adaLN(adaptive Layer Norm)-Zero the one which gave the best results, as depicted in the right. Source: Peebles and Xie [2023]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "845ba03e-4930-45ba-9f17-966e7f7f8f58",
   "metadata": {},
   "source": [
    "## Flux"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a9dafc-e1fc-4890-b6a7-6a13ce15d3d5",
   "metadata": {},
   "source": [
    "Like Stable Diffusion 3, Flux uses **Flow Matching** ([\\color{SkyBlue}{Lipman et al. [2023]}](https://arxiv.org/abs/2210.02747)) to formulate, train and sample from the model\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Flow matching is based on Continuous Normalizing Flows, and leverages generative modeling by progressively morphing a distribution into another one\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    ". . .\n",
    "\n",
    "Diffusion is a specific case of Flow Matching, where the morphing is done by adding random Normal noise to the data\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Other Flow Matching formulations (like when using *Conditional Optimal Transport*) can be more efficient to train and sample from, at the expense of a bit of quality (which can be offset by more data)\n",
    "\n",
    ". . .\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "We refer to the paper above and [\\color{SkyBlue}{this video}](https://www.youtube.com/watch?v=5ZSwYogAxYg) by Lipman for further details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9e52742-28f1-4047-b18e-a2caa60879b0",
   "metadata": {},
   "source": [
    "## Flux"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a722e0cb-c58a-4a62-8372-2fe163eba39d",
   "metadata": {},
   "source": [
    "![](images/flux_examples.png){height=80% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: https://blackforestlabs.ai/announcing-black-forest-labs/ [2024]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08620d48-cd81-4078-8d52-e21398a97a7c",
   "metadata": {},
   "source": [
    "## Sana"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec68373e-f28e-4b02-b601-66fc3ef96031",
   "metadata": {},
   "source": [
    "Sana by [\\color{SkyBlue}{Xie et al. [2024]}](https://arxiv.org/abs/2410.10629) includes innovations such as:\n",
    "\n",
    "- A *Deep Compression Autoencoder* (DC-AE), which compresses/decompresses aggresively input images up to 32x → efficient generation of 4K images\n",
    "\n",
    "- A linear DiT that performs efficient sub-quadratic attention\n",
    "\n",
    "- Uses Google's pre-trained Gemma-2 ([\\color{SkyBlue}{Gemma Team [2024]}](https://arxiv.org/abs/2408.00118)) decoder-only LLM to perform text conditioning → they extract last layer's output as textual embeddings\n",
    "\n",
    "\\ifwindows\n",
    "\\vspace{-8pt}\n",
    "\\fi\n",
    "![](images/sana_diagram.png){height=40% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-16pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Figure (a) shows the model architecture, including a sample prompt for Gemma-2. Positional embedding is not required in the linear DiT. Figure (b) shows details on the linear DiT block. Source: Xie et al. [2024]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b017d6ce-281e-4ad1-9653-d6c34a312f21",
   "metadata": {},
   "source": [
    "## Sana"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf868f25-e761-417b-a956-f10362a4f8af",
   "metadata": {},
   "source": [
    "![](images/sana_examples.png){height=80% fig-align=\"center\"}\n",
    "\\ifwindows\n",
    "\\vspace{-20pt}\n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: Xie et al. [2024]\n",
    "\\end{center}\n",
    "\\normalsize"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83b452c1-b1c9-45c7-a035-cee8f3a7a6e3",
   "metadata": {},
   "source": [
    "# Beyond image generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee849648-d84f-409e-a1a0-67546b2b07ba",
   "metadata": {},
   "source": [
    "## Diffusion for video"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cc14384-43dc-4ea1-b0d5-3b52ecad00f2",
   "metadata": {},
   "source": [
    "Image generator diuffusion models work with 4D data in the form of $(\\textit{Batch, Channels, Height, Width})$ \n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Diffusion-based video generation models work by mostly adapting the model to handle 5D data $(\\textit{Batch, Time, Channels, Height, Width})$ in various ways\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    ". . .\n",
    "\n",
    "Some of the most prominent publications are:\n",
    "\n",
    "- [\\color{SkyBlue}{Video Diffusion Models}](https://arxiv.org/abs/2204.03458) by Ho et al. [2022]\n",
    "  \n",
    "-  [\\color{SkyBlue}{Imagen Video: High Definition Video Generation with Diffusion Models}](https://arxiv.org/abs/2210.02303) by Ho et al. [2022] and [\\color{SkyBlue}{Veo}](https://deepmind.google/technologies/veo/) by Veo team [2024]\n",
    "  \n",
    "-  [\\color{SkyBlue}{Photorealistic Video Generation with Diffusion Models}](https://arxiv.org/abs/2312.06662) by Gupta et al. [2023]\n",
    "  \n",
    "- [\\color{SkyBlue}{Sora}](https://openai.com/research/video-generation-models-as-world-simulators) by Brooks et al. [2024]\n",
    "  \n",
    "- [\\color{SkyBlue}{Movie Gen}](https://ai.meta.com/static-resource/movie-gen-research-paper) by The Movie Gen Team @ Meta [2024]\n",
    "\n",
    "- [\\color{SkyBlue}{CogVideoX}](https://arxiv.org/abs/2408.06072) by Yang et al. [2024]\n",
    "\n",
    "- [\\color{SkyBlue}{Pyramid Flow}](https://arxiv.org/abs/2410.05954) by Jin et al. [2024]\n",
    "\n",
    "- [\\color{SkyBlue}{Mochi 1}](https://www.genmo.ai/blog) by Genmo Team [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16cb687e-c0ea-4413-af15-05ab91098e55",
   "metadata": {},
   "source": [
    "## Diffusion for video"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81af6218-47dc-4afc-b03b-f47ce56090b8",
   "metadata": {},
   "source": [
    ":::: {.columns align=center}\n",
    "::: {.column align=center width=100%}\n",
    "[![](images/movie_gen_example_1.png){width=100% fig-alt=\"Movie Gen\" fig-align=\"center\"}](https://imgur.com/7yw8MSc)\n",
    "\\ifwindows \n",
    "\\vspace{-14pt} \n",
    "\\fi\n",
    "\\begin{center}\n",
    "\\tiny Source: The Movie Gen Team @ Meta [2024]\n",
    "\\end{center}\n",
    "\\normalsize\n",
    ":::\n",
    "::::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b909b81e-7821-4b8f-af6f-715d7c5dd1e3",
   "metadata": {},
   "source": [
    "## Diffusion for other applications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c48fdde-61e6-4339-86d4-1474aed6c743",
   "metadata": {},
   "source": [
    "Some examples (by no means a thorough list):\n",
    "\n",
    "3D rendering and reconstruction:\n",
    "\n",
    "- [\\color{SkyBlue}{DreamFusion: Text-to-3D using 2D Diffusion}](https://arxiv.org/abs/2209.14988) by Poole et al. [2022]\n",
    "\n",
    "- [\\color{SkyBlue}{CAT3D: Create Anything in 3D with Multi-View Diffusion Models}](https://arxiv.org/abs/2405.10314) by Gao et al. [2024]\n",
    "\n",
    "Text generation:\n",
    "\n",
    "- [\\color{SkyBlue}{Diffusion-LM Improves Controllable Text Generation}](https://arxiv.org/abs/2205.14217) by Lisa Li et al. [2024]\n",
    "\n",
    "Music and audio generation:\n",
    "\n",
    "- [\\color{SkyBlue}{Noise2Music: Text-conditioned Music Generation with Diffusion Models}](https://arxiv.org/abs/2302.03917) by Huang et al. [2023]\n",
    "  \n",
    "- [\\color{SkyBlue}{Fast Timing-Conditioned Latent Audio Diffusion}](https://arxiv.org/abs/2402.04825) by Evans et al. [2024]\n",
    "  \n",
    "- [\\color{SkyBlue}{QA-MDT: Quality-aware Masked Diffusion Transformer for Enhanced Music Generation}](https://arxiv.org/abs/2405.15863) by Li et al. [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e6823a-4d43-480a-998a-f7dcb064fa53",
   "metadata": {},
   "source": [
    "## Diffusion for other applications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ea09f97-5ab3-4d12-bce5-606aacce9d2a",
   "metadata": {},
   "source": [
    "Text-to-speech:\n",
    "\n",
    "- [\\color{SkyBlue}{Better speech synthesis through scaling}](https://arxiv.org/abs/2305.07243) by Betker [2023]\n",
    "  \n",
    "- [\\color{SkyBlue}{NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models}](https://arxiv.org/abs/2403.03100) by Ju et al. [2024]\n",
    "\n",
    "- [\\color{SkyBlue}{F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching}](https://arxiv.org/abs/2403.03100) by Chen et al. [2024]\n",
    "\n",
    "\n",
    "Life sciences:\n",
    "\n",
    "- [\\color{SkyBlue}{DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking}](https://arxiv.org/abs/2210.01776) by Corso et al. [2022]\n",
    "\n",
    "- [\\color{SkyBlue}{Protein generation with evolutionary diffusion: sequence is all you need}](https://www.biorxiv.org/content/10.1101/2023.09.11.556673v1) by Alamdari et al. [2023]\n",
    "\n",
    "- [\\color{SkyBlue}{Accurate structure prediction of biomolecular interactions with AlphaFold 3}](https://www.nature.com/articles/s41586-024-07487-w) by Abramson et al. [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "926b261d-86f3-4010-95d7-9c7913c9b5cd",
   "metadata": {},
   "source": [
    "## Diffusion for other applications"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54cd8680-c3aa-4477-b5a1-e7c95d8ca782",
   "metadata": {},
   "source": [
    "Robotics:\n",
    "\n",
    "- [\\color{SkyBlue}{Diffusion Policy: Visuomotor Policy Learning via Action Diffusion}](https://arxiv.org/abs/2303.04137v5) by Chi et al. [2023]\n",
    "\n",
    "Videogame engine creation:\n",
    "\n",
    "- [\\color{SkyBlue}{Diffusion Models Are Real-Time Game Engines}](https://arxiv.org/abs/2408.14837) by Valevski et al. [2024]\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "And many more (as [\\color{SkyBlue}{there are more than 4,000 papers on arXiv}](https://vsehwag.github.io/blog/2023/2/all_papers_on_diffusion.html) at the time of writing!)\n",
    "\n",
    "```{=latex}\n",
    "\\, \\\\\n",
    "```\n",
    "\n",
    "Recent survey papers like [\\color{SkyBlue}{A Survey on Generative Diffusion Models}](https://arxiv.org/abs/2209.02646) by Cao et al. [2023] can be useful to stay up to date"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6da65660-20a9-456d-be44-6f17a6153ead",
   "metadata": {},
   "source": [
    "## How to use diffusion models?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c7a60a7-0b96-48dc-9898-9b69213dc987",
   "metadata": {},
   "source": [
    "Do-It-Yourself:\n",
    "\n",
    "- PyTorch\n",
    "\n",
    "Training and fine-tuning of existing architectures:\n",
    "\n",
    "- [\\color{SkyBlue}{\\raisebox{-2pt}{\\includegraphics[height=12pt]{images/hugging-face-emoji.png}} Diffusers}](https://huggingface.co/docs/diffusers)\n",
    "- [\\color{SkyBlue}{Ostris AI toolkit}](https://github.com/ostris/ai-toolkit)\n",
    "- [\\color{SkyBlue}{Bghira's SimpleTuner}](https://github.com/bghira/SimpleTuner)\n",
    "- [\\color{SkyBlue}{Kohya}](https://github.com/bmaltais/kohya_ss)\n",
    "\n",
    "Inference and image generation GUIs (no-code or low-code), from simpler to more complex:\n",
    "\n",
    "- [\\color{SkyBlue}{Automatic1111}](https://github.com/AUTOMATIC1111/stable-diffusion-webui)\n",
    "- [\\color{SkyBlue}{ForgeUI}](https://github.com/lllyasviel/stable-diffusion-webui-forge) → very similar to A1111, but faster and more modern\n",
    "- [\\color{SkyBlue}{SwarmUI}](https://github.com/mcmonkeyprojects/SwarmUI)\n",
    "- [\\color{SkyBlue}{ComfyUI}](https://github.com/comfyanonymous/ComfyUI) → tons of features; professional tool\n",
    "\n",
    "\\center \\large This list gets outdated very quickly! Check for community updates on [\\color{SkyBlue}{r/StableDiffusion}](https://www.reddit.com/r/StableDiffusion/) and [\\color{SkyBlue}{Civitai}](https://civitai.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e6911d-6ae5-4f9b-ac08-227ab694dffd",
   "metadata": {},
   "source": [
    "## Some interesting topics we skipped"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffa72b79-0b6c-495d-af5c-9557d2635091",
   "metadata": {},
   "source": [
    "\\small Connection to score-based generative models (what if $t$ was continuous instead of discrete?). Diffusion expressed as Ordinary and Stochastic Differential Equations:\n",
    "\n",
    "- [\\color{SkyBlue}{Generative Modeling by Estimating Gradients of the Data Distribution}](https://arxiv.org/abs/1907.05600) by Song and Ermon [2019]\n",
    "- [\\color{SkyBlue}{Improved Techniques for Training Score-Based Generative Models}](https://arxiv.org/abs/2006.09011) by Song and Ermon [2020]\n",
    "- [\\color{SkyBlue}{Score-Based Generative Modeling through Stochastic Differential Equations}](https://arxiv.org/abs/2011.13456) by Song et al. [2020]\n",
    "- [\\color{SkyBlue}{Elucidating the Design Space of Diffusion-Based Generative Models}](https://arxiv.org/abs/2206.00364) by Karras et al. [2022]\n",
    "\n",
    "Perform sampling in even less steps:\n",
    "\n",
    "- [\\color{SkyBlue}{Progressive Distillation for Fast Sampling of Diffusion Models}](https://arxiv.org/abs/2202.00512) by Salimans and Ho [2022]\n",
    "- [\\color{SkyBlue}{Consistency Models}](https://arxiv.org/abs/2303.01469) by Song et al. [2023]\n",
    "- [\\color{SkyBlue}{Simple and Fast Distillation of Diffusion Models}](https://www.arxiv.org/abs/2409.19681) by Zhou et al. [2024]\n",
    "\n",
    "Zero-terminal Signal-to-Noise Ratio (SNR):\n",
    "\n",
    "- [\\color{SkyBlue}{Common Diffusion Noise Schedules and Sample Steps are Flawed}](https://arxiv.org/abs/2305.08891) by Lin et al. [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e16576-ed06-4fff-a6db-d65cf17f3840",
   "metadata": {},
   "source": [
    "## Some interesting topics we skipped"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "772379e4-72c3-443e-94b8-8436f37fc83f",
   "metadata": {},
   "source": [
    "\\small Improve resolution through cascaded generation, used in e.g. DALL-E 2 and Imagen:\n",
    "\n",
    "- [\\color{SkyBlue}{Cascaded Diffusion Models for High Fidelity Image Generation}](https://arxiv.org/abs/2106.15282) by Ho et al. [2022]\n",
    "\n",
    "Subject-driven image generation through fine-tuning (add specific characters to images):\n",
    "\n",
    "- [\\color{SkyBlue}{An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion}](https://arxiv.org/abs/2208.01618) by Gal et al. [2022]\n",
    "- [\\color{SkyBlue}{DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation}](https://arxiv.org/abs/2208.12242) by Ruiz et al. [2022]\n",
    "\n",
    "Image and video editing:\n",
    "\n",
    "- [\\color{SkyBlue}{SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations}](https://arxiv.org/abs/2108.01073) by Meng et al. [2021]\n",
    "- [\\color{SkyBlue}{Prompt-to-Prompt Image Editing with Cross Attention Control}](https://arxiv.org/abs/2208.01626) by Hertz et al. [2022]\n",
    "- [\\color{SkyBlue}{RePaint: Inpainting using Denoising Diffusion Probabilistic Models}](https://arxiv.org/abs/2201.09865) by Lugmayr et al. [2022]\n",
    "- [\\color{SkyBlue}{Differential Diffusion: Giving Each Pixel Its Strength}](https://arxiv.org/abs/2306.00950) by Levin et al. [2023]\n",
    "- [\\color{SkyBlue}{Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation}](https://arxiv.org/abs/2212.11565) by Wu et al. [2023]\n",
    "- [\\color{SkyBlue}{Dreamix: Video Diffusion Models are General Video Editors}](https://arxiv.org/abs/2302.01329) by Moland et al. [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e8a395-8184-4c25-adb8-a19768cf6fbf",
   "metadata": {},
   "source": [
    "## Some interesting topics we skipped"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0238d0d9-99ec-42e3-9e4e-1b4509b80c9b",
   "metadata": {},
   "source": [
    "\\small Diffusion Transformer variants and improvements:\n",
    "\n",
    "- [\\color{SkyBlue}{PixArt-$\\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis}](https://arxiv.org/abs/2310.00426) by Chen et al. [2024]\n",
    "- [\\color{SkyBlue}{Dynamic Diffusion Transformer}](https://arxiv.org/abs/2410.03456) by Zhao et al. [2024]\n",
    "\n",
    "ControlNet variants:\n",
    "\n",
    "- [\\color{SkyBlue}{Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models}](https://arxiv.org/abs/2305.16322) by Zhao et al. [2023]\n",
    "- [\\color{SkyBlue}{ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback}](https://arxiv.org/abs/2404.07987) by Li et al. [2024]\n",
    "- [\\color{SkyBlue}{CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation}](https://arxiv.org/abs/2410.09400) by Xu et al. [2024]\n",
    "\n",
    "Flow Matching variants and improvements:\n",
    "\n",
    "- [\\color{SkyBlue}{InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation}](https://arxiv.org/abs/2309.06380) by Liu et al. [2023]\n",
    "- [\\color{SkyBlue}{Rectified Diffusion: Straightness Is Not Your Need in Rectified Flow}](https://arxiv.org/abs/2410.07303) by Wang et al. [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b38505-e648-4f99-89bf-284739613730",
   "metadata": {},
   "source": [
    "## Some interesting topics we skipped {.t}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19c5aa13-1eb3-479c-b0ae-672fa2c4676d",
   "metadata": {},
   "source": [
    "\\small Regional generation:\n",
    "\n",
    "- [\\color{SkyBlue}{Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs}](https://arxiv.org/abs/2401.11708) by Yang et al. [2024]\n",
    "- [\\color{SkyBlue}{Training-free Regional Prompting for Diffusion Transformers}](https://arxiv.org/abs/2411.02395) by Chen et al. [2024]\n",
    "\n",
    "General multi-task diffusion models:\n",
    "\n",
    "- [\\color{SkyBlue}{OmniGen: Unified Image Generation}](https://arxiv.org/abs/2409.11340) by Xiao et al. [2024]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8359921b-b857-4f2e-a1f2-710d1bc3f118",
   "metadata": {},
   "source": [
    "## References {.allowframebreaks}\n",
    "\n",
    "---\n",
    "nocite: |\n",
    "  @*\n",
    "---\n",
    "\\tiny\n",
    "::: {#refs}\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cab69403-8ea7-463d-a4b7-78a1d05b9e48",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4b8f6ec-5ac8-4bd3-917a-d26f50ccebad",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}