fgsm_tutorial_pytorch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/csce585-mlsystems/tutorials/blob/main/fgsm_tutorial_pytorch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t95mePLCHgA4"
      },
      "outputs": [],
      "source": [
        "# For tips on running notebooks in Google Colab, see\n",
        "# https://pytorch.org/tutorials/beginner/colab\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w2-Bg40WHgA5"
      },
      "source": [
        "\n",
        "# Adversarial Example Generation\n",
        "\n",
        "If you are reading this, hopefully you can appreciate how effective some\n",
        "machine learning models are. Research is constantly pushing ML models to\n",
        "be faster, more accurate, and more efficient. However, an often\n",
        "overlooked aspect of designing and training models is security and\n",
        "robustness, especially in the face of an adversary who wishes to fool\n",
        "the model.\n",
        "\n",
        "This tutorial will raise your awareness to the security vulnerabilities\n",
        "of ML models, and will give insight into the hot topic of adversarial\n",
        "machine learning. You may be surprised to find that adding imperceptible\n",
        "perturbations to an image *can* cause drastically different model\n",
        "performance. Given that this is a tutorial, we will explore the topic\n",
        "via example on an image classifier. Specifically, we will use one of the\n",
        "first and most popular attack methods, the Fast Gradient Sign Attack\n",
        "(FGSM), to fool an MNIST classifier.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e4KoDubTHgA6"
      },
      "source": [
        "## Threat Model\n",
        "\n",
        "For context, there are many categories of adversarial attacks, each with\n",
        "a different goal and assumption of the attacker’s knowledge. However, in\n",
        "general the overarching goal is to add the least amount of perturbation\n",
        "to the input data to cause the desired misclassification. There are\n",
        "several kinds of assumptions of the attacker’s knowledge, two of which\n",
        "are: **white-box** and **black-box**. A *white-box* attack assumes the\n",
        "attacker has full knowledge and access to the model, including\n",
        "architecture, inputs, outputs, and weights. A *black-box* attack assumes\n",
        "the attacker only has access to the inputs and outputs of the model, and\n",
        "knows nothing about the underlying architecture or weights. There are\n",
        "also several types of goals, including **misclassification** and\n",
        "**source/target misclassification**. A goal of *misclassification* means\n",
        "the adversary only wants the output classification to be wrong but does\n",
        "not care what the new classification is. A *source/target\n",
        "misclassification* means the adversary wants to alter an image that is\n",
        "originally of a specific source class so that it is classified as a\n",
        "specific target class.\n",
        "\n",
        "In this case, the FGSM attack is a *white-box* attack with the goal of\n",
        "*misclassification*. With this background information, we can now\n",
        "discuss the attack in detail.\n",
        "\n",
        "## Fast Gradient Sign Attack\n",
        "\n",
        "One of the first and most popular adversarial attacks to date is\n",
        "referred to as the *Fast Gradient Sign Attack (FGSM)* and is described\n",
        "by Goodfellow et. al. in [Explaining and Harnessing Adversarial\n",
        "Examples](https://arxiv.org/abs/1412.6572)_. The attack is remarkably\n",
        "powerful, and yet intuitive. It is designed to attack neural networks by\n",
        "leveraging the way they learn, *gradients*. The idea is simple, rather\n",
        "than working to minimize the loss by adjusting the weights based on the\n",
        "backpropagated gradients, the attack *adjusts the input data to maximize\n",
        "the loss* based on the same backpropagated gradients. In other words,\n",
        "the attack uses the gradient of the loss w.r.t the input data, then\n",
        "adjusts the input data to maximize the loss.\n",
        "\n",
        "Before we jump into the code, let’s look at the famous\n",
        "[FGSM](https://arxiv.org/abs/1412.6572)_ panda example and extract\n",
        "some notation.\n",
        "\n",
        "![Adversarial Example](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/generative/images/adversarial_example.png?raw=1)\n",
        "\n",
        "From the figure, $\\mathbf{x}$ is the original input image\n",
        "correctly classified as a “panda”, $y$ is the ground truth label\n",
        "for $\\mathbf{x}$, $\\mathbf{\\theta}$ represents the model\n",
        "parameters, and $J(\\mathbf{\\theta}, \\mathbf{x}, y)$ is the loss\n",
        "that is used to train the network. The attack backpropagates the\n",
        "gradient back to the input data to calculate\n",
        "$\\nabla_{x} J(\\mathbf{\\theta}, \\mathbf{x}, y)$. Then, it adjusts\n",
        "the input data by a small step ($\\epsilon$ or $0.007$ in the\n",
        "picture) in the direction (i.e.\n",
        "$sign(\\nabla_{x} J(\\mathbf{\\theta}, \\mathbf{x}, y))$) that will\n",
        "maximize the loss. The resulting perturbed image, $x'$, is then\n",
        "*misclassified* by the target network as a “gibbon” when it is still\n",
        "clearly a “panda”.\n",
        "\n",
        "Hopefully now the motivation for this tutorial is clear, so lets jump\n",
        "into the implementation.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ogXCkH7IHgA7"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "from torchvision import datasets, transforms\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8AJScl1-HgA7"
      },
      "source": [
        "## Implementation\n",
        "\n",
        "In this section, we will discuss the input parameters for the tutorial,\n",
        "define the model under attack, then code the attack and run some tests.\n",
        "\n",
        "### Inputs\n",
        "\n",
        "There are only three inputs for this tutorial, and are defined as\n",
        "follows:\n",
        "\n",
        "-  ``epsilons`` - List of epsilon values to use for the run. It is\n",
        "   important to keep 0 in the list because it represents the model\n",
        "   performance on the original test set. Also, intuitively we would\n",
        "   expect the larger the epsilon, the more noticeable the perturbations\n",
        "   but the more effective the attack in terms of degrading model\n",
        "   accuracy. Since the data range here is $[0,1]$, no epsilon\n",
        "   value should exceed 1.\n",
        "\n",
        "-  ``pretrained_model`` - path to the pretrained MNIST model which was\n",
        "   trained with\n",
        "   [pytorch/examples/mnist](https://github.com/pytorch/examples/tree/master/mnist)_.\n",
        "   For simplicity, download the pretrained model [here](https://drive.google.com/file/d/1HJV2nUHJqclXQ8flKvcWmjZ-OU5DGatl/view?usp=drive_link)_.\n",
        "\n",
        "-  ``use_cuda`` - boolean flag to use CUDA if desired and available.\n",
        "   Note, a GPU with CUDA is not critical for this tutorial as a CPU will\n",
        "   not take much time.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JqekMf4ZHgA7"
      },
      "outputs": [],
      "source": [
        "epsilons = [0, .05, .1, .15, .2, .25, .3]\n",
        "pretrained_model = \"data/lenet_mnist_model.pth\"\n",
        "use_cuda=True\n",
        "# Set random seed for reproducibility\n",
        "torch.manual_seed(42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7uXo69EbHgA7"
      },
      "source": [
        "### Model Under Attack\n",
        "\n",
        "As mentioned, the model under attack is the same MNIST model from\n",
        "[pytorch/examples/mnist](https://github.com/pytorch/examples/tree/master/mnist)_.\n",
        "You may train and save your own MNIST model or you can download and use\n",
        "the provided model. The *Net* definition and test dataloader here have\n",
        "been copied from the MNIST example. The purpose of this section is to\n",
        "define the model and dataloader, then initialize the model and load the\n",
        "pretrained weights.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3pMBYd8eHgA7"
      },
      "outputs": [],
      "source": [
        "# LeNet Model definition\n",
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n",
        "        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n",
        "        self.dropout1 = nn.Dropout(0.25)\n",
        "        self.dropout2 = nn.Dropout(0.5)\n",
        "        self.fc1 = nn.Linear(9216, 128)\n",
        "        self.fc2 = nn.Linear(128, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.conv1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.conv2(x)\n",
        "        x = F.relu(x)\n",
        "        x = F.max_pool2d(x, 2)\n",
        "        x = self.dropout1(x)\n",
        "        x = torch.flatten(x, 1)\n",
        "        x = self.fc1(x)\n",
        "        x = F.relu(x)\n",
        "        x = self.dropout2(x)\n",
        "        x = self.fc2(x)\n",
        "        output = F.log_softmax(x, dim=1)\n",
        "        return output\n",
        "\n",
        "# MNIST Test dataset and dataloader declaration\n",
        "test_loader = torch.utils.data.DataLoader(\n",
        "    datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([\n",
        "            transforms.ToTensor(),\n",
        "            transforms.Normalize((0.1307,), (0.3081,)),\n",
        "            ])),\n",
        "        batch_size=1, shuffle=True)\n",
        "\n",
        "# Define what device we are using\n",
        "print(\"CUDA Available: \",torch.cuda.is_available())\n",
        "device = torch.device(\"cuda\" if use_cuda and torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Initialize the network\n",
        "model = Net().to(device)\n",
        "\n",
        "# Load the pretrained model\n",
        "model.load_state_dict(torch.load(pretrained_model, map_location=device))\n",
        "\n",
        "# Set the model in evaluation mode. In this case this is for the Dropout layers\n",
        "model.eval()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3GgROLwsHgA8"
      },
      "source": [
        "### FGSM Attack\n",
        "\n",
        "Now, we can define the function that creates the adversarial examples by\n",
        "perturbing the original inputs. The ``fgsm_attack`` function takes three\n",
        "inputs, *image* is the original clean image ($x$), *epsilon* is\n",
        "the pixel-wise perturbation amount ($\\epsilon$), and *data_grad*\n",
        "is gradient of the loss w.r.t the input image\n",
        "($\\nabla_{x} J(\\mathbf{\\theta}, \\mathbf{x}, y)$). The function\n",
        "then creates perturbed image as\n",
        "\n",
        "\\begin{align}perturbed\\_image = image + epsilon*sign(data\\_grad) = x + \\epsilon * sign(\\nabla_{x} J(\\mathbf{\\theta}, \\mathbf{x}, y))\\end{align}\n",
        "\n",
        "Finally, in order to maintain the original range of the data, the\n",
        "perturbed image is clipped to range $[0,1]$.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2eGE47B-HgA8"
      },
      "outputs": [],
      "source": [
        "# FGSM attack code\n",
        "def fgsm_attack(image, epsilon, data_grad):\n",
        "    # Collect the element-wise sign of the data gradient\n",
        "    sign_data_grad = data_grad.sign()\n",
        "    # Create the perturbed image by adjusting each pixel of the input image\n",
        "    perturbed_image = image + epsilon*sign_data_grad\n",
        "    # Adding clipping to maintain [0,1] range\n",
        "    perturbed_image = torch.clamp(perturbed_image, 0, 1)\n",
        "    # Return the perturbed image\n",
        "    return perturbed_image\n",
        "\n",
        "# restores the tensors to their original scale\n",
        "def denorm(batch, mean=[0.1307], std=[0.3081]):\n",
        "    \"\"\"\n",
        "    Convert a batch of tensors to their original scale.\n",
        "\n",
        "    Args:\n",
        "        batch (torch.Tensor): Batch of normalized tensors.\n",
        "        mean (torch.Tensor or list): Mean used for normalization.\n",
        "        std (torch.Tensor or list): Standard deviation used for normalization.\n",
        "\n",
        "    Returns:\n",
        "        torch.Tensor: batch of tensors without normalization applied to them.\n",
        "    \"\"\"\n",
        "    if isinstance(mean, list):\n",
        "        mean = torch.tensor(mean).to(device)\n",
        "    if isinstance(std, list):\n",
        "        std = torch.tensor(std).to(device)\n",
        "\n",
        "    return batch * std.view(1, -1, 1, 1) + mean.view(1, -1, 1, 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tErAJlcZHgA8"
      },
      "source": [
        "### Testing Function\n",
        "\n",
        "Finally, the central result of this tutorial comes from the ``test``\n",
        "function. Each call to this test function performs a full test step on\n",
        "the MNIST test set and reports a final accuracy. However, notice that\n",
        "this function also takes an *epsilon* input. This is because the\n",
        "``test`` function reports the accuracy of a model that is under attack\n",
        "from an adversary with strength $\\epsilon$. More specifically, for\n",
        "each sample in the test set, the function computes the gradient of the\n",
        "loss w.r.t the input data ($data\\_grad$), creates a perturbed\n",
        "image with ``fgsm_attack`` ($perturbed\\_data$), then checks to see\n",
        "if the perturbed example is adversarial. In addition to testing the\n",
        "accuracy of the model, the function also saves and returns some\n",
        "successful adversarial examples to be visualized later.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "emBcLpUxHgA8"
      },
      "outputs": [],
      "source": [
        "def test( model, device, test_loader, epsilon ):\n",
        "\n",
        "    # Accuracy counter\n",
        "    correct = 0\n",
        "    adv_examples = []\n",
        "\n",
        "    # Loop over all examples in test set\n",
        "    for data, target in test_loader:\n",
        "\n",
        "        # Send the data and label to the device\n",
        "        data, target = data.to(device), target.to(device)\n",
        "\n",
        "        # Set requires_grad attribute of tensor. Important for Attack\n",
        "        data.requires_grad = True\n",
        "\n",
        "        # Forward pass the data through the model\n",
        "        output = model(data)\n",
        "        init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability\n",
        "\n",
        "        # If the initial prediction is wrong, don't bother attacking, just move on\n",
        "        if init_pred.item() != target.item():\n",
        "            continue\n",
        "\n",
        "        # Calculate the loss\n",
        "        loss = F.nll_loss(output, target)\n",
        "\n",
        "        # Zero all existing gradients\n",
        "        model.zero_grad()\n",
        "\n",
        "        # Calculate gradients of model in backward pass\n",
        "        loss.backward()\n",
        "\n",
        "        # Collect ``datagrad``\n",
        "        data_grad = data.grad.data\n",
        "\n",
        "        # Restore the data to its original scale\n",
        "        data_denorm = denorm(data)\n",
        "\n",
        "        # Call FGSM Attack\n",
        "        perturbed_data = fgsm_attack(data_denorm, epsilon, data_grad)\n",
        "\n",
        "        # Reapply normalization\n",
        "        perturbed_data_normalized = transforms.Normalize((0.1307,), (0.3081,))(perturbed_data)\n",
        "\n",
        "        # Re-classify the perturbed image\n",
        "        output = model(perturbed_data_normalized)\n",
        "\n",
        "        # Check for success\n",
        "        final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability\n",
        "        if final_pred.item() == target.item():\n",
        "            correct += 1\n",
        "            # Special case for saving 0 epsilon examples\n",
        "            if epsilon == 0 and len(adv_examples) < 5:\n",
        "                adv_ex = perturbed_data.squeeze().detach().cpu().numpy()\n",
        "                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )\n",
        "        else:\n",
        "            # Save some adv examples for visualization later\n",
        "            if len(adv_examples) < 5:\n",
        "                adv_ex = perturbed_data.squeeze().detach().cpu().numpy()\n",
        "                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )\n",
        "\n",
        "    # Calculate final accuracy for this epsilon\n",
        "    final_acc = correct/float(len(test_loader))\n",
        "    print(f\"Epsilon: {epsilon}\\tTest Accuracy = {correct} / {len(test_loader)} = {final_acc}\")\n",
        "\n",
        "    # Return the accuracy and an adversarial example\n",
        "    return final_acc, adv_examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "duxgt2XvHgA9"
      },
      "source": [
        "### Run Attack\n",
        "\n",
        "The last part of the implementation is to actually run the attack. Here,\n",
        "we run a full test step for each epsilon value in the *epsilons* input.\n",
        "For each epsilon we also save the final accuracy and some successful\n",
        "adversarial examples to be plotted in the coming sections. Notice how\n",
        "the printed accuracies decrease as the epsilon value increases. Also,\n",
        "note the $\\epsilon=0$ case represents the original test accuracy,\n",
        "with no attack.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OIkLi3PgHgA9"
      },
      "outputs": [],
      "source": [
        "accuracies = []\n",
        "examples = []\n",
        "\n",
        "# Run test for each epsilon\n",
        "for eps in epsilons:\n",
        "    acc, ex = test(model, device, test_loader, eps)\n",
        "    accuracies.append(acc)\n",
        "    examples.append(ex)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ai4xWtgpHgA9"
      },
      "source": [
        "## Results\n",
        "\n",
        "### Accuracy vs Epsilon\n",
        "\n",
        "The first result is the accuracy versus epsilon plot. As alluded to\n",
        "earlier, as epsilon increases we expect the test accuracy to decrease.\n",
        "This is because larger epsilons mean we take a larger step in the\n",
        "direction that will maximize the loss. Notice the trend in the curve is\n",
        "not linear even though the epsilon values are linearly spaced. For\n",
        "example, the accuracy at $\\epsilon=0.05$ is only about 4% lower\n",
        "than $\\epsilon=0$, but the accuracy at $\\epsilon=0.2$ is 25%\n",
        "lower than $\\epsilon=0.15$. Also, notice the accuracy of the model\n",
        "hits random accuracy for a 10-class classifier between\n",
        "$\\epsilon=0.25$ and $\\epsilon=0.3$.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2CEiV6-GHgA9"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(5,5))\n",
        "plt.plot(epsilons, accuracies, \"*-\")\n",
        "plt.yticks(np.arange(0, 1.1, step=0.1))\n",
        "plt.xticks(np.arange(0, .35, step=0.05))\n",
        "plt.title(\"Accuracy vs Epsilon\")\n",
        "plt.xlabel(\"Epsilon\")\n",
        "plt.ylabel(\"Accuracy\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A1OuO9UYHgA9"
      },
      "source": [
        "### Sample Adversarial Examples\n",
        "\n",
        "Remember the idea of no free lunch? In this case, as epsilon increases\n",
        "the test accuracy decreases **BUT** the perturbations become more easily\n",
        "perceptible. In reality, there is a tradeoff between accuracy\n",
        "degradation and perceptibility that an attacker must consider. Here, we\n",
        "show some examples of successful adversarial examples at each epsilon\n",
        "value. Each row of the plot shows a different epsilon value. The first\n",
        "row is the $\\epsilon=0$ examples which represent the original\n",
        "“clean” images with no perturbation. The title of each image shows the\n",
        "“original classification -> adversarial classification.” Notice, the\n",
        "perturbations start to become evident at $\\epsilon=0.15$ and are\n",
        "quite evident at $\\epsilon=0.3$. However, in all cases humans are\n",
        "still capable of identifying the correct class despite the added noise.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ke3NPL4kHgA9"
      },
      "outputs": [],
      "source": [
        "# Plot several examples of adversarial samples at each epsilon\n",
        "cnt = 0\n",
        "plt.figure(figsize=(8,10))\n",
        "for i in range(len(epsilons)):\n",
        "    for j in range(len(examples[i])):\n",
        "        cnt += 1\n",
        "        plt.subplot(len(epsilons),len(examples[0]),cnt)\n",
        "        plt.xticks([], [])\n",
        "        plt.yticks([], [])\n",
        "        if j == 0:\n",
        "            plt.ylabel(f\"Eps: {epsilons[i]}\", fontsize=14)\n",
        "        orig,adv,ex = examples[i][j]\n",
        "        plt.title(f\"{orig} -> {adv}\")\n",
        "        plt.imshow(ex, cmap=\"gray\")\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DTYLAU6SHgA9"
      },
      "source": [
        "## Where to go next?\n",
        "\n",
        "Hopefully this tutorial gives some insight into the topic of adversarial\n",
        "machine learning. There are many potential directions to go from here.\n",
        "This attack represents the very beginning of adversarial attack research\n",
        "and since there have been many subsequent ideas for how to attack and\n",
        "defend ML models from an adversary. In fact, at NIPS 2017 there was an\n",
        "adversarial attack and defense competition and many of the methods used\n",
        "in the competition are described in this paper: [Adversarial Attacks and\n",
        "Defences Competition](https://arxiv.org/pdf/1804.00097.pdf)_. The work\n",
        "on defense also leads into the idea of making machine learning models\n",
        "more *robust* in general, to both naturally perturbed and adversarially\n",
        "crafted inputs.\n",
        "\n",
        "Another direction to go is adversarial attacks and defense in different\n",
        "domains. Adversarial research is not limited to the image domain, check\n",
        "out [this](https://arxiv.org/pdf/1801.01944.pdf)_ attack on\n",
        "speech-to-text models. But perhaps the best way to learn more about\n",
        "adversarial machine learning is to get your hands dirty. Try to\n",
        "implement a different attack from the NIPS 2017 competition, and see how\n",
        "it differs from FGSM. Then, try to defend the model from your own\n",
        "attacks.\n",
        "\n",
        "A further direction to go, depending on available resources, is to modify\n",
        "the code to support processing work in batch, in parallel, and or distributed\n",
        "vs working on one attack at a time in the above for each ``epsilon test()`` loop.\n",
        "\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}