03.2 - Try it on your own.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep learning on MNIST\n",
    "In this exercise we will try to apply what we learnt so far on some numpy computations, but before that, we will do some profiling to check understand the performance and the inner working of the computations that we are going to apply Numba for.\n",
    "\n",
    "The exercise here uses the [Deep learning on MNIST tutorial form Numpy](https://numpy.org/numpy-tutorials/content/tutorial-deep-learning-on-mnist.html). For the mathmetical details, you can check out the [original tutorial](https://numpy.org/numpy-tutorials/content/tutorial-deep-learning-on-mnist.html). Or even better, do that tutorial first before trying this one. For convinence, I have downloaded it and [included in this repo for you](./03.1%20-%20tutorial-deep-learning-on-mnist.ipynb).\n",
    "\n",
    "Just like th original tutorial, we have to use Requests, Numpy and Matplotlib. We also have to import `jit` from Numba. If you don't have Matplotlib and/or Requests installed, uncommned that following and install it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install matplotlib requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and process the MNIST dataset\n",
    "\n",
    "This is the part that we will just copy what the tutorial did. (but using more than 1000 samples.) For detailed explanation of what is done, please see the [original tutorial](./tutorial-deep-learning-on-mnist.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import requests\n",
    "\n",
    "data_sources = {\n",
    "    \"training_images\": \"train-images-idx3-ubyte.gz\",  # 60,000 training images.\n",
    "    \"test_images\": \"t10k-images-idx3-ubyte.gz\",  # 10,000 test images.\n",
    "    \"training_labels\": \"train-labels-idx1-ubyte.gz\",  # 60,000 training labels.\n",
    "    \"test_labels\": \"t10k-labels-idx1-ubyte.gz\",  # 10,000 test labels.\n",
    "}\n",
    "headers = {\n",
    "    \"User-Agent\": \"Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20100101 Firefox/10.0\"\n",
    "}\n",
    "request_opts = {\n",
    "    \"headers\": headers,\n",
    "    \"params\": {\"raw\": \"true\"},\n",
    "}\n",
    "\n",
    "# First check if the data is stored locally; if not, then download it.\n",
    "\n",
    "data_dir = \"../_data\"\n",
    "os.makedirs(data_dir, exist_ok=True)\n",
    "\n",
    "base_url = \"https://github.com/rossbar/numpy-tutorial-data-mirror/blob/main/\"\n",
    "\n",
    "for fname in data_sources.values():\n",
    "    fpath = os.path.join(data_dir, fname)\n",
    "    if not os.path.exists(fpath):\n",
    "        print(\"Downloading file: \" + fname)\n",
    "        resp = requests.get(base_url + fname, stream=True, **request_opts)\n",
    "        resp.raise_for_status()  # Ensure download was succesful\n",
    "        with open(fpath, \"wb\") as fh:\n",
    "            for chunk in resp.iter_content(chunk_size=128):\n",
    "                fh.write(chunk)\n",
    "\n",
    "# Decompress the 4 files and create 4 ndarrays\n",
    "                \n",
    "mnist_dataset = {}\n",
    "\n",
    "# Images\n",
    "for key in (\"training_images\", \"test_images\"):\n",
    "    with gzip.open(os.path.join(data_dir, data_sources[key]), \"rb\") as mnist_file:\n",
    "        mnist_dataset[key] = np.frombuffer(\n",
    "            mnist_file.read(), np.uint8, offset=16\n",
    "        ).reshape(-1, 28 * 28)\n",
    "# Labels\n",
    "for key in (\"training_labels\", \"test_labels\"):\n",
    "    with gzip.open(os.path.join(data_dir, data_sources[key]), \"rb\") as mnist_file:\n",
    "        mnist_dataset[key] = np.frombuffer(mnist_file.read(), np.uint8, offset=8)\n",
    "\n",
    "# Split the data into training and test sets\n",
    "        \n",
    "x_train, y_train, x_test, y_test = (\n",
    "    mnist_dataset[\"training_images\"],\n",
    "    mnist_dataset[\"training_labels\"],\n",
    "    mnist_dataset[\"test_images\"],\n",
    "    mnist_dataset[\"test_labels\"],\n",
    ")\n",
    "\n",
    "# Normalize the arrays by dividing them by 255\n",
    "\n",
    "training_sample, test_sample = 5000, 5000\n",
    "training_images = x_train[0:training_sample] / 255\n",
    "test_images = x_test[0:test_sample] / 255\n",
    "\n",
    "# Encode the labels\n",
    "\n",
    "def one_hot_encoding(labels, dimension=10):\n",
    "    # Define a one-hot variable for an all-zero vector\n",
    "    # with 10 dimensions (number labels from 0 to 9).\n",
    "    one_hot_labels = labels[..., None] == np.arange(dimension)[None]\n",
    "    # Return one-hot encoded labels.\n",
    "    return one_hot_labels.astype(np.float64)\n",
    "\n",
    "training_labels = one_hot_encoding(y_train[:training_sample])\n",
    "test_labels = one_hot_encoding(y_test[:test_sample])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build and train a small neural network\n",
    "\n",
    "Just like the original tutorial, we build a neural network from scratch. We make some adjustment, especially breaking the training process into seperate functions, so the profilling later would be easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create seed\n",
    "seed = 884736743\n",
    "rng = np.random.default_rng(seed)\n",
    "\n",
    "# Define hyperparameters\n",
    "learning_rate = 0.005\n",
    "epochs = 20\n",
    "hidden_size = 100\n",
    "pixels_per_image = 784\n",
    "num_labels = 10\n",
    "\n",
    "class NN:\n",
    "    def __init__(self):\n",
    "        self.weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1\n",
    "        self.weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1\n",
    "\n",
    "    # Define ReLU that returns the input if it's positive and 0 otherwise.\n",
    "    def _relu(self, x):\n",
    "        return (x >= 0) * x\n",
    "\n",
    "    # Set up a derivative of the ReLU function that returns 1 for a positive input\n",
    "    # and 0 otherwise.\n",
    "    def _relu2deriv(self, output):\n",
    "        return output >= 0\n",
    "        \n",
    "    def _forward_prop(self, i):\n",
    "        # Forward propagation/forward pass:\n",
    "        # 1. The input layer:\n",
    "        #    Initialize the training image data as inputs.\n",
    "        self.layer_0 = training_images[i]\n",
    "        # 2. The hidden layer:\n",
    "        #    Take in the training image data into the middle layer by\n",
    "        #    matrix-multiplying it by randomly initialized weights.\n",
    "        self.layer_1 = np.dot(self.layer_0, self.weights_1)\n",
    "        # 3. Pass the hidden layer's output through the ReLU activation function.\n",
    "        self.layer_1 = self._relu(self.layer_1)\n",
    "        # 4. Define the dropout function for regularization.\n",
    "        self.dropout_mask = rng.integers(low=0, high=2, size=self.layer_1.shape)\n",
    "        # 5. Apply dropout to the hidden layer's output.\n",
    "        self.layer_1 *= self.dropout_mask * 2\n",
    "        # 6. The output layer:\n",
    "        #    Ingest the output of the middle layer into the the final layer\n",
    "        #    by matrix-multiplying it by randomly initialized weights.\n",
    "        #    Produce a 10-dimension vector with 10 scores.\n",
    "        self.layer_2 = np.dot(self.layer_1, self.weights_2)\n",
    "        \n",
    "    def _back_prop(self, i):\n",
    "        # Backpropagation/backward pass:\n",
    "        # 1. Measure the training error (loss function) between the actual\n",
    "        #    image labels (the truth) and the prediction by the model.\n",
    "        self.training_loss += np.sum((training_labels[i] - self.layer_2) ** 2)\n",
    "        # 2. Increment the accurate prediction count.\n",
    "        self.training_accurate_predictions += int(\n",
    "            np.argmax(self.layer_2) == np.argmax(training_labels[i])\n",
    "        )\n",
    "        # 3. Differentiate the loss function/error.\n",
    "        layer_2_delta = training_labels[i] - self.layer_2\n",
    "        # 4. Propagate the gradients of the loss function back through the hidden layer.\n",
    "        layer_1_delta = np.dot(self.weights_2, layer_2_delta) * self._relu2deriv(self.layer_1)\n",
    "        # 5. Apply the dropout to the gradients.\n",
    "        layer_1_delta *= self.dropout_mask\n",
    "        # 6. Update the weights for the middle and input layers\n",
    "        #    by multiplying them by the learning rate and the gradients.\n",
    "        self.weights_1 += learning_rate * np.outer(self.layer_0, layer_1_delta)\n",
    "        self.weights_2 += learning_rate * np.outer(self.layer_1, layer_2_delta)\n",
    "        \n",
    "    def _training_step(self):\n",
    "        # Set the initial loss/error and the number of accurate predictions to zero.\n",
    "        self.training_loss = 0.0\n",
    "        self.training_accurate_predictions = 0\n",
    "\n",
    "        # For all images in the training set, perform a forward pass\n",
    "        # and backpropagation and adjust the weights accordingly.\n",
    "        for i in range(len(training_images)):\n",
    "            self._forward_prop(i)\n",
    "            self._back_prop(i)\n",
    "\n",
    "        # Store training set losses and accurate predictions.\n",
    "        self.store_training_loss.append(self.training_loss)\n",
    "        self.store_training_accurate_pred.append(self.training_accurate_predictions)\n",
    "        \n",
    "    def _evaluation_step(self):\n",
    "        results = self._relu(test_images @ self.weights_1) @ self.weights_2\n",
    "\n",
    "        # Measure the error between the actual label (truth) and prediction values.\n",
    "        test_loss = np.sum((test_labels - results) ** 2)\n",
    "\n",
    "        # Measure prediction accuracy on test set\n",
    "        test_accurate_predictions = np.sum(\n",
    "            np.argmax(results, axis=1) == np.argmax(test_labels, axis=1)\n",
    "        )\n",
    "\n",
    "        # Store test set losses and accurate predictions.\n",
    "        self.store_test_loss.append(test_loss)\n",
    "        self.store_test_accurate_pred.append(test_accurate_predictions)\n",
    "        \n",
    "    def train(self, show_epochs=True):\n",
    "\n",
    "        # To store training and test set losses and accurate predictions\n",
    "        # for visualization.\n",
    "        self.store_training_loss = []\n",
    "        self.store_training_accurate_pred = []\n",
    "        self.store_test_loss = []\n",
    "        self.store_test_accurate_pred = []\n",
    "\n",
    "        # This is a training loop.\n",
    "        # Run the learning experiment for a defined number of epochs (iterations).\n",
    "        for j in range(epochs):\n",
    "\n",
    "            #################\n",
    "            # Training step #\n",
    "            #################\n",
    "\n",
    "            self._training_step()\n",
    "\n",
    "            ###################\n",
    "            # Evaluation step #\n",
    "            ###################\n",
    "\n",
    "            # Evaluate model performance on the test set at each epoch.\n",
    "\n",
    "            # Unlike the training step, the weights are not modified for each image\n",
    "            # (or batch). Therefore the model can be applied to the test images in a\n",
    "            # vectorized manner, eliminating the need to loop over each image\n",
    "            # individually:\n",
    "\n",
    "            self._evaluation_step()\n",
    "\n",
    "            # Summarize error and accuracy metrics at each epoch\n",
    "            if show_epochs:\n",
    "                print(\n",
    "                    (\n",
    "                        f\"Epoch: {j}\\n\"\n",
    "                        f\"  Training set error: {store_training_loss[-1] / len(training_images):.3f}\\n\"\n",
    "                        f\"  Training set accuracy: {store_training_accurate_pred[-1] / len(training_images)}\\n\"\n",
    "                        f\"  Test set error: {store_test_loss[-1] / len(test_images):.3f}\\n\"\n",
    "                        f\"  Test set accuracy: {store_test_accurate_pred[-1] / len(test_images)}\"\n",
    "                    )\n",
    "                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Profile the training process\n",
    "\n",
    "Now we use [cProfile](https://docs.python.org/3/library/profile.html) to profile the training process. The code below will take a while to run (~1 min) if you want to test it out at first, you can selected less sample in the data just like the original tutorial does.\n",
    "\n",
    "After running the profiling process, can you identify which process in the `train()` function take the most time? Can you identify which Numpy process has been called the most? With this information, can you formulate a plan of which part of the process we can compile with jit in Numba?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cProfile, pstats\n",
    "profiler = cProfile.Profile()\n",
    "model = NN()\n",
    "profiler.enable()\n",
    "model.train(show_epochs=False)\n",
    "profiler.disable()\n",
    "stats = pstats.Stats(profiler).sort_stats('cumtime')\n",
    "stats.print_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's use jit to speed up some of the process. Remember what we have learnt so far. Use the tricks that we learnt and that may require rewriting of some code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numba import jit, njit\n",
    "\n",
    "#### TODO: use @jit or @njit to speed up the training process\n",
    "\n",
    "class NN_jit:\n",
    "    def __init__(self):\n",
    "        self.weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1\n",
    "        self.weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1\n",
    "        \n",
    "    def reset(self):\n",
    "        # extra helper function to reset the training\n",
    "        self.weights_1 = 0.2 * rng.random((pixels_per_image, hidden_size)) - 0.1\n",
    "        self.weights_2 = 0.2 * rng.random((hidden_size, num_labels)) - 0.1\n",
    "\n",
    "    # Define ReLU that returns the input if it's positive and 0 otherwise.\n",
    "    def _relu(self, x):\n",
    "        return (x >= 0) * x\n",
    "\n",
    "    # Set up a derivative of the ReLU function that returns 1 for a positive input\n",
    "    # and 0 otherwise.\n",
    "    def _relu2deriv(self, output):\n",
    "        return output >= 0\n",
    "\n",
    "    def _forward_prop(self, i):\n",
    "        # Forward propagation/forward pass:\n",
    "        # 1. The input layer:\n",
    "        #    Initialize the training image data as inputs.\n",
    "        self.layer_0 = training_images[i]\n",
    "        # 2. The hidden layer:\n",
    "        #    Take in the training image data into the middle layer by\n",
    "        #    matrix-multiplying it by randomly initialized weights.\n",
    "        self.layer_1 = np.dot(self.layer_0, self.weights_1)\n",
    "        # 3. Pass the hidden layer's output through the ReLU activation function.\n",
    "        self.layer_1 = self._relu(self.layer_1)\n",
    "        # 4. Define the dropout function for regularization.\n",
    "        self.dropout_mask = rng.integers(low=0, high=2, size=self.layer_1.shape)\n",
    "        # 5. Apply dropout to the hidden layer's output.\n",
    "        self.layer_1 *= self.dropout_mask * 2\n",
    "        # 6. The output layer:\n",
    "        #    Ingest the output of the middle layer into the the final layer\n",
    "        #    by matrix-multiplying it by randomly initialized weights.\n",
    "        #    Produce a 10-dimension vector with 10 scores.\n",
    "        self.layer_2 = np.dot(self.layer_1, self.weights_2)\n",
    "\n",
    "    def _back_prop(self, i):\n",
    "        # Backpropagation/backward pass:\n",
    "        # 1. Measure the training error (loss function) between the actual\n",
    "        #    image labels (the truth) and the prediction by the model.\n",
    "        self.training_loss += np.sum((training_labels[i] - self.layer_2) ** 2)\n",
    "        # 2. Increment the accurate prediction count.\n",
    "        self.training_accurate_predictions += int(\n",
    "            np.argmax(self.layer_2) == np.argmax(training_labels[i])\n",
    "        )\n",
    "        # 3. Differentiate the loss function/error.\n",
    "        layer_2_delta = training_labels[i] - self.layer_2\n",
    "        # 4. Propagate the gradients of the loss function back through the hidden layer.\n",
    "        layer_1_delta = np.dot(self.weights_2, layer_2_delta) * self._relu2deriv(self.layer_1)\n",
    "        # 5. Apply the dropout to the gradients.\n",
    "        layer_1_delta *= self.dropout_mask\n",
    "        # 6. Update the weights for the middle and input layers\n",
    "        #    by multiplying them by the learning rate and the gradients.\n",
    "        self.weights_1 += learning_rate * np.outer(self.layer_0, layer_1_delta)\n",
    "        self.weights_2 += learning_rate * np.outer(self.layer_1, layer_2_delta)\n",
    "        \n",
    "    def _training_step(self):\n",
    "        # Set the initial loss/error and the number of accurate predictions to zero.\n",
    "        self.training_loss = 0.0\n",
    "        self.training_accurate_predictions = 0\n",
    "\n",
    "        # For all images in the training set, perform a forward pass\n",
    "        # and backpropagation and adjust the weights accordingly.\n",
    "        for i in range(len(training_images)):\n",
    "            self._forward_prop(i)\n",
    "            self._back_prop(i)\n",
    "\n",
    "        # Store training set losses and accurate predictions.\n",
    "        self.store_training_loss.append(self.training_loss)\n",
    "        self.store_training_accurate_pred.append(self.training_accurate_predictions)\n",
    "        \n",
    "    def _evaluation_step(self):\n",
    "        results = self._relu(test_images @ self.weights_1) @ self.weights_2\n",
    "\n",
    "        # Measure the error between the actual label (truth) and prediction values.\n",
    "        test_loss = np.sum((test_labels - results) ** 2)\n",
    "\n",
    "        # Measure prediction accuracy on test set\n",
    "        test_accurate_predictions = np.sum(\n",
    "            np.argmax(results, axis=1) == np.argmax(test_labels, axis=1)\n",
    "        )\n",
    "\n",
    "        # Store test set losses and accurate predictions.\n",
    "        self.store_test_loss.append(test_loss)\n",
    "        self.store_test_accurate_pred.append(test_accurate_predictions)\n",
    "        \n",
    "    def train(self, show_epochs=True):\n",
    "\n",
    "        # To store training and test set losses and accurate predictions\n",
    "        # for visualization.\n",
    "        self.store_training_loss = []\n",
    "        self.store_training_accurate_pred = []\n",
    "        self.store_test_loss = []\n",
    "        self.store_test_accurate_pred = []\n",
    "\n",
    "        # This is a training loop.\n",
    "        # Run the learning experiment for a defined number of epochs (iterations).\n",
    "        for j in range(epochs):\n",
    "\n",
    "            #################\n",
    "            # Training step #\n",
    "            #################\n",
    "\n",
    "            self._training_step()\n",
    "\n",
    "            ###################\n",
    "            # Evaluation step #\n",
    "            ###################\n",
    "\n",
    "            # Evaluate model performance on the test set at each epoch.\n",
    "\n",
    "            # Unlike the training step, the weights are not modified for each image\n",
    "            # (or batch). Therefore the model can be applied to the test images in a\n",
    "            # vectorized manner, eliminating the need to loop over each image\n",
    "            # individually:\n",
    "\n",
    "            self._evaluation_step()\n",
    "\n",
    "            # Summarize error and accuracy metrics at each epoch\n",
    "            if show_epochs:\n",
    "                print(\n",
    "                    (\n",
    "                        f\"Epoch: {j}\\n\"\n",
    "                        f\"  Training set error: {store_training_loss[-1] / len(training_images):.3f}\\n\"\n",
    "                        f\"  Training set accuracy: {store_training_accurate_pred[-1] / len(training_images)}\\n\"\n",
    "                        f\"  Test set error: {store_test_loss[-1] / len(test_images):.3f}\\n\"\n",
    "                        f\"  Test set accuracy: {store_test_accurate_pred[-1] / len(test_images)}\"\n",
    "                    )\n",
    "                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Profile again, remember to run it once to compile first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile the process\n",
    "\n",
    "model_jit = NN_jit()\n",
    "model_jit.train(show_epochs=False) # or just run the jit'ed methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiler = cProfile.Profile()\n",
    "model_jit.reset()\n",
    "model_jit.train(show_epochs=False)\n",
    "profiler.enable()\n",
    "model_jit.train(show_epochs=False)\n",
    "profiler.disable()\n",
    "stats = pstats.Stats(profiler).sort_stats('cumtime')\n",
    "stats.print_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you managed to get some performance improvement? Have you try with `@njit` or `@jit(nopython=True)`? Challange yourself by using no Python mode and rewrite some of the code. See if that would improve the performance further. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the result\n",
    "\n",
    "This part has nothing to do with the profilling and compiling for speed up but just to check the training result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# The training set metrics.\n",
    "y_training_error = [\n",
    "    model.store_training_loss[i] / float(len(training_images))\n",
    "    for i in range(len(model.store_training_loss))\n",
    "]\n",
    "x_training_error = range(1, len(model.store_training_loss) + 1)\n",
    "y_training_accuracy = [\n",
    "    model.store_training_accurate_pred[i] / float(len(training_images))\n",
    "    for i in range(len(model.store_training_accurate_pred))\n",
    "]\n",
    "x_training_accuracy = range(1, len(model.store_training_accurate_pred) + 1)\n",
    "\n",
    "# The test set metrics.\n",
    "y_test_error = [\n",
    "    model.store_test_loss[i] / float(len(test_images)) for i in range(len(model.store_test_loss))\n",
    "]\n",
    "x_test_error = range(1, len(model.store_test_loss) + 1)\n",
    "y_test_accuracy = [\n",
    "    model.store_training_accurate_pred[i] / float(len(training_images))\n",
    "    for i in range(len(model.store_training_accurate_pred))\n",
    "]\n",
    "x_test_accuracy = range(1, len(model.store_test_accurate_pred) + 1)\n",
    "\n",
    "# Display the plots.\n",
    "fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15, 5))\n",
    "axes[0].set_title(\"Training set error, accuracy\")\n",
    "axes[0].plot(x_training_accuracy, y_training_accuracy, label=\"Training set accuracy\")\n",
    "axes[0].plot(x_training_error, y_training_error, label=\"Training set error\")\n",
    "axes[0].set_xlabel(\"Epochs\")\n",
    "axes[1].set_title(\"Test set error, accuracy\")\n",
    "axes[1].plot(x_test_accuracy, y_test_accuracy, label=\"Test set accuracy\")\n",
    "axes[1].plot(x_test_error, y_test_error, label=\"Test set error\")\n",
    "axes[1].set_xlabel(\"Epochs\")\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}