ch04/01_main-chapter-code/ch04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ce9295b2-182b-490b-8325-83a67c4a001d",
   "metadata": {},
   "source": [
    "# 章节 4：从零开始实现 GPT 模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7da97ed-e02f-4d7f-b68e-a0eba3716e02",
   "metadata": {},
   "source": [
    "- 在本章中，我们将设计一个类似 GPT 的大型语言模型（LLM）架构；下一章则将聚焦于该模型的训练。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d4f11e0-4434-4979-9dee-e1207df0eb01",
   "metadata": {},
   "source": [
    "<img src=\"figures/mental-model.webp\" width=450px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53fe99ab-0bcf-4778-a6b5-6db81fb826ef",
   "metadata": {},
   "source": [
    "## 4.1 设计LLM的架构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad72d1ff-d82d-4e33-a88e-3c1a8831797b",
   "metadata": {},
   "source": [
    "- 第1章探讨了如GPT与Llama等模型，这些模型基于transformer架构的decoder部分，并按顺序生成文本。\n",
    "- 因此，这些LLM经常被称为decoder-only LLM。\n",
    "- 与传统的深度学习模型相比，LLM更大，这是因为它们有更多的参数，而不是代码量。\n",
    "- 而在LLM的架构中，有许多元素是重复的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5213e9-bd1c-437e-aee8-f5e8fb717251",
   "metadata": {},
   "source": [
    "<img src=\"figures/mental-model-2.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d43f5e2-fb51-434a-b9be-abeef6b98d99",
   "metadata": {},
   "source": [
    "- 在前几章中，为了方便展示，我们使用了较小的嵌入（embedding）维度来处理token的输入和输出。\n",
    "- 在本章中，我们将考虑与GPT2-small模型类似的嵌入和模型大小。\n",
    "- 我们将具体实现最小的GPT2-small模型（124M参数）的架构，如Radford等人在[《Language Models are Unsupervised Multitask Learners》](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)中概述的那样（注意，GPT2-small的参数量曾被错误的统计为117M参数，后被更正为124M）。\n",
    "- 第6章将展示如何将预训练权重加载到我们实现的GPT2中，并兼容345、762和1542M参数的模型大小。\n",
    "\n",
    "> 译者注：GPT2的论文《Language Models are Unsupervised Multitask Learners》中错误统计了GPT2系列模型的参数量，这一错误后续在模型仓库中被偷偷修正了。\n",
    "> \n",
    "> 错误的参数量：Small (117M)\tMedium (345M)\tLarge (762M)\tXL (1542M)\n",
    ">\n",
    "> 正确的参数量：Small (124M)\tMedium (355M)\tLarge (774M)\tXL (1558M)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21baa14d-24b8-4820-8191-a2808f7fbabc",
   "metadata": {},
   "source": [
    "- 124M参数GPT-2模型的配置细节包括："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5ed66875-1f24-445d-add6-006aae3c5707",
   "metadata": {},
   "outputs": [],
   "source": [
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,  # 词表大小\n",
    "    \"ctx_len\": 1024,      # 上下文长度\n",
    "    \"emb_dim\": 768,       # 嵌入维度\n",
    "    \"n_heads\": 12,        # 注意力头（attention heads）的数量\n",
    "    \"n_layers\": 12,       # 模型层数\n",
    "    \"drop_rate\": 0.1,     # Dropout rate\n",
    "    \"qkv_bias\": False     # Query-Key-Value bias\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12fcd28-d210-4c57-8be6-06cfcd5d73a4",
   "metadata": {},
   "source": [
    "- 我们使用简短的变量名以避免后续代码行的过长\n",
    "- \"vocab_size\" 是一个BPE tokenizer（分词器），词表大小为50257个词，这在第二章介绍过\n",
    "- \"ctx_len\" 表示模型支持输入的最大token数量，这数值由第二章中介绍的位置编码决定\n",
    "- \"emb_dim\" 是对输入token的嵌入维度，这里会将输入的每个token都嵌入成768维的向量\n",
    "- \"n_heads\" 是多头注意力机制中的注意力头数，这在第三章中实现过\n",
    "- \"n_layers\" 是模型中transformer blocks的数量，我们将在接下来的部分中实现它。\n",
    "- \"drop_rate\" 是第三章中讨论的dropout机制的强度；0.1表示在训练期间丢弃10％的隐藏神经元以缓解过拟合\n",
    "- \"qkv_bias\" 决定第三章中的多头注意力机制中的Linear层在计算Query（Q），Key（K）和Value（V）张量时是否应包含偏置向量（bias）；当代LLM通常不会启用这个选项，我们也不会；但在第六章中将OpenAI预训练的GPT-2权重加载到我们的实现的模型时，会再次讨论此选项。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4adce779-857b-4418-9501-12a7f3818d88",
   "metadata": {},
   "source": [
    "<img src=\"figures/chapter-steps.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "619c2eed-f8ea-4ff5-92c3-feda0f29b227",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class DummyGPTModel(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
    "        self.pos_emb = nn.Embedding(cfg[\"ctx_len\"], cfg[\"emb_dim\"])\n",
    "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
    "        \n",
    "        # 先用空白实现顶替下 TransformerBlock\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[DummyTransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
    "        \n",
    "        # 先用空白实现顶替下 LayerNorm\n",
    "        self.final_norm = DummyLayerNorm(cfg[\"emb_dim\"])\n",
    "        self.out_head = nn.Linear(\n",
    "            cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n",
    "        )\n",
    "\n",
    "    def forward(self, in_idx):\n",
    "        batch_size, seq_len = in_idx.shape\n",
    "        tok_embeds = self.tok_emb(in_idx)\n",
    "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
    "        x = tok_embeds + pos_embeds\n",
    "        x = self.drop_emb(x)\n",
    "        x = self.trf_blocks(x)\n",
    "        x = self.final_norm(x)\n",
    "        logits = self.out_head(x)\n",
    "        return logits\n",
    "\n",
    "\n",
    "class DummyTransformerBlock(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        # 略\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 先啥也别干，原样返回\n",
    "        return x\n",
    "\n",
    "\n",
    "class DummyLayerNorm(nn.Module):\n",
    "    def __init__(self, normalized_shape, eps=1e-5):\n",
    "        super().__init__()\n",
    "        # 这里的参数只是为了模拟 LayerNorm 接口。\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 先啥也别干，原样返回\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9665e8ab-20ca-4100-b9b9-50d9bdee33be",
   "metadata": {},
   "source": [
    "<img src=\"figures/gpt-in-out.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "794b6b6c-d36f-411e-a7db-8ac566a87fee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[6109, 3626, 6100,  345],\n",
      "        [6109, 1110, 6622,  257]])\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "import torch\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "batch = []\n",
    "\n",
    "txt1 = \"Every effort moves you\"\n",
    "txt2 = \"Every day holds a\"\n",
    "\n",
    "batch.append(torch.tensor(tokenizer.encode(txt1)))\n",
    "batch.append(torch.tensor(tokenizer.encode(txt2)))\n",
    "batch = torch.stack(batch, dim=0)\n",
    "print(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "009238cd-0160-4834-979c-309710986bb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output shape: torch.Size([2, 4, 50257])\n",
      "tensor([[[-1.2034,  0.3201, -0.7130,  ..., -1.5548, -0.2390, -0.4667],\n",
      "         [-0.1192,  0.4539, -0.4432,  ...,  0.2392,  1.3469,  1.2430],\n",
      "         [ 0.5307,  1.6720, -0.4695,  ...,  1.1966,  0.0111,  0.5835],\n",
      "         [ 0.0139,  1.6755, -0.3388,  ...,  1.1586, -0.0435, -1.0400]],\n",
      "\n",
      "        [[-1.0908,  0.1798, -0.9484,  ..., -1.6047,  0.2439, -0.4530],\n",
      "         [-0.7860,  0.5581, -0.0610,  ...,  0.4835, -0.0077,  1.6621],\n",
      "         [ 0.3567,  1.2698, -0.6398,  ..., -0.0162, -0.1296,  0.3717],\n",
      "         [-0.2407, -0.7349, -0.5102,  ...,  2.0057, -0.3694,  0.1814]]],\n",
      "       grad_fn=<UnsafeViewBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = DummyGPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "logits = model(batch)\n",
    "print(\"Output shape:\", logits.shape)\n",
    "print(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8332a00-98da-4eb4-b882-922776a89917",
   "metadata": {},
   "source": [
    "## 4.2 对激活进行层归一化"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066cfb81-d59b-4d95-afe3-e43cf095f292",
   "metadata": {},
   "source": [
    "- 层归一化（Layer normalization），也叫 LayerNorm ([Ba et al. 2016](https://arxiv.org/abs/1607.06450))，会将神经网络层的激活值规范到均值为0，并将其方差归一化为1。\n",
    "- 这稳定了训练过程，并提高了模型的收敛速度。。\n",
    "- Transformer block中多头注意力模块的输入和输出都会应用LayerNorm，一会会实现它；同时，在最终输出层之前也会应用LayerNorm。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "314ac47a-69cc-4597-beeb-65bed3b5910f",
   "metadata": {},
   "source": [
    "<img src=\"figures/layernorm.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab49940-6b35-4397-a80e-df8d092770a7",
   "metadata": {},
   "source": [
    "- 咱们用一个简单的网络，输入一个样本看看LayerNorm是怎么工作的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "79e1b463-dc3f-44ac-9cdb-9d5b6f64eb9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2260, 0.3470, 0.0000, 0.2216, 0.0000, 0.0000],\n",
      "        [0.2133, 0.2394, 0.0000, 0.5198, 0.3297, 0.0000]],\n",
      "       grad_fn=<ReluBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "# 创建两个训练样例，每个样例有5个维度（特征）\n",
    "batch_example = torch.randn(2, 5) \n",
    "\n",
    "layer = nn.Sequential(nn.Linear(5, 6), nn.ReLU())\n",
    "out = layer(batch_example)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fccc29e-71fc-4c16-898c-6137c6ea5d2e",
   "metadata": {},
   "source": [
    "- 计算上面两个输入的均值和方差："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9888f79e-8e69-44aa-8a19-cd34292adbf5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean:\n",
      " tensor([[0.1324],\n",
      "        [0.2170]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[0.0231],\n",
      "        [0.0398]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "mean = out.mean(dim=-1, keepdim=True)\n",
    "var = out.var(dim=-1, keepdim=True)\n",
    "\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052eda3e-b395-48c4-acd4-eb8083bab958",
   "metadata": {},
   "source": [
    "- LayerNorm 会对输入样本分别归一化（下图中的行）; 使用`dim=-1`是在最后一个维度（特征维度）而不是行维度（样本数）上进行计算"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "570db83a-205c-4f6f-b219-1f6195dde1a7",
   "metadata": {},
   "source": [
    "<img src=\"figures/layernorm2.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f8ecbc7-eb14-4fa1-b5d0-7e1ff9694f99",
   "metadata": {},
   "source": [
    "- 减去均值并除以方差的平方根（标准差）会使输入在列（特征）维度上的均值为0，方差为1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9a1d1bb9-3341-4c9a-bc2a-d2489bf89cda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized layer outputs:\n",
      " tensor([[ 0.6159,  1.4126, -0.8719,  0.5872, -0.8719, -0.8719],\n",
      "        [-0.0189,  0.1121, -1.0876,  1.5173,  0.5647, -1.0876]],\n",
      "       grad_fn=<DivBackward0>)\n",
      "Mean:\n",
      " tensor([[    0.0000],\n",
      "        [    0.0000]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[1.],\n",
      "        [1.]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "out_norm = (out - mean) / torch.sqrt(var)\n",
    "print(\"Normalized layer outputs:\\n\", out_norm)\n",
    "\n",
    "mean = out_norm.mean(dim=-1, keepdim=True)\n",
    "var = out_norm.var(dim=-1, keepdim=True)\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac62b90c-7156-4979-9a79-ce1fb92969c1",
   "metadata": {},
   "source": [
    "- 每个输入的均值都为0，方差都为1；为了提高可读性，我们可以关闭PyTorch的科学计数法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3e06c34b-c68a-4b36-afbe-b30eda4eca39",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean:\n",
      " tensor([[    0.0000],\n",
      "        [    0.0000]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[1.],\n",
      "        [1.]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.set_printoptions(sci_mode=False)\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "944fb958-d4ed-43cc-858d-00052bb6b31a",
   "metadata": {},
   "source": [
    "- 在上面，我们对每个输入的特征进行了归一化\n",
    "- 现在，用相同的思路，我们可以实现一个`LayerNorm`类："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "3333a305-aa3d-460a-bcce-b80662d464d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNorm(nn.Module):\n",
    "    def __init__(self, emb_dim):\n",
    "        super().__init__()\n",
    "        self.eps = 1e-5\n",
    "        self.scale = nn.Parameter(torch.ones(emb_dim))\n",
    "        self.shift = nn.Parameter(torch.zeros(emb_dim))\n",
    "\n",
    "    def forward(self, x):\n",
    "        mean = x.mean(dim=-1, keepdim=True)\n",
    "        var = x.var(dim=-1, keepdim=True, unbiased=False)\n",
    "        norm_x = (x - mean) / torch.sqrt(var + self.eps)\n",
    "        return self.scale * norm_x + self.shift"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e56c3908-7544-4808-b8cb-5d0a55bcca72",
   "metadata": {},
   "source": [
    "**缩放和偏移**\n",
    "- 注意，除了通过减去均值并除以方差执行归一化之外，我们还添加了两个可训练参数，一个是 `scale`，另一个是 `shift`。\n",
    "- 初始的 scale（乘以1）和 shift（加0）值没有任何效果；然而，scale 和 shift 是可训练的参数，如果确定这样做可以改善模型在训练任务上的性能，LLM 在训练过程中会自动调整它们。\n",
    "- 这使得模型能够学习适合其处理数据的适当缩放和偏移。\n",
    "- 注意，在计算方差的平方根之前，我们还添加了一个较小的值（eps）；这是为了避免在方差为0时发生分母为0的问题。\n",
    "\n",
    "**有偏方差**\n",
    "- 在上面的方差计算中，设置 `unbiased=False` 意味着用 $\\frac{\\sum_i (x_i - \\bar{x})^2}{n}$ 来计算方差，其中 n 是样本大小（在这里是特征或列数）；这个公式不包括 Bessel 修正（分母是 n-1），因此得到的方差是有偏估计。\n",
    "- 因为LLM的嵌入维度很高，所以使用 n 或 n-1 （有偏或无偏）的区别不大。\n",
    "- 但 GPT-2 在LayerNorm中使用了有偏方差进行训练，为了在后续章节能加载现有的预训练权重，咱需要`unbiased`这个变量做兼容。\n",
    "\n",
    "- 下面手动实现下 LayerNorm："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "23b1000a-e613-4b43-bd90-e54deed8d292",
   "metadata": {},
   "outputs": [],
   "source": [
    "ln = LayerNorm(emb_dim=5)\n",
    "out_ln = ln(batch_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "94c12de2-1cab-46e0-a099-e2e470353bff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean:\n",
      " tensor([[    -0.0000],\n",
      "        [     0.0000]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[1.0000],\n",
      "        [1.0000]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "mean = out_ln.mean(dim=-1, keepdim=True)\n",
    "var = out_ln.var(dim=-1, unbiased=False, keepdim=True)\n",
    "\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e136cfc4-7c89-492e-b120-758c272bca8c",
   "metadata": {},
   "source": [
    "<img src=\"figures/overview-after-ln.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11190e7d-8c29-4115-824a-e03702f9dd54",
   "metadata": {},
   "source": [
    "## 4.3 使用GELU激活函数实现前馈神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0585dfb-f21e-40e5-973f-2f63ad5cb169",
   "metadata": {},
   "source": [
    "- 在这一节中，我们将实现一个网络子模块，该模块将作为LLM中Transformer block的一部分\n",
    "- 我们从激活函数开始\n",
    "- 在深度学习中，由于ReLU（Rectified Linear Unit）激活函数在各种神经网络架构中的简单性和有效性，它们经常被使用\n",
    "- 在LLM中，除了ReLU之外，还使用了其他类型的激活函数；其中两个值得注意的例子是GELU（Gaussian Error Linear Unit）和SwiGLU（Sigmoid-Weighted Linear Unit）\n",
    "- GELU和SwiGLU是更复杂的、平滑的激活函数，它们分别结合了高斯和Sigmoid门控线性单元，为深度学习模型提供了更好的性能，与ReLU的简单分段线性函数不同"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d482ce7-e493-4bfc-a820-3ea99f564ebc",
   "metadata": {},
   "source": [
    "- GELU ([Hendrycks and Gimpel 2016](https://arxiv.org/abs/1606.08415))用多种实现；其精确版本定义为$GELU(x)=x\\cdot \\phi(x)$，其中$\\phi(x)$是标准高斯分布的累积分布函数。\n",
    "- 在实际应用中，常常采用计算成本较低的近似形式：$\\text{GELU}(x) \\approx 0.5 \\cdot x \\cdot \\left(1 + \\tanh\\left[\\sqrt{\\frac{2}{\\pi}} \\cdot \\left(x + 0.044715 \\cdot x^3\\right)\\right]\\right)$（原始的GPT-2模型也是使用这个近似形式进行训练的）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f84694b7-95f3-4323-b6d6-0a73df278e82",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GELU(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return 0.5 * x * (1 + torch.tanh(\n",
    "            torch.sqrt(torch.tensor(2.0 / torch.pi)) * \n",
    "            (x + 0.044715 * torch.pow(x, 3))\n",
    "        ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "fc5487d2-2576-4118-80a7-56c4caac2e71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAEiCAYAAABkykQ1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABn2klEQVR4nO3deVhUZfsH8O8My7AJiiAoICIqigsqpKG5lYpbRSnZoqKmqWHlkiX+SjPfpDK33K2UJM19KTMTTVJzB1HRJBcQFzZllWUYZs7vD2QSAWXYzpnh+7muud53zpzlvmdyHu55zvM8MkEQBBAREREREVWBXOwAiIiIiIhI/7GwICIiIiKiKmNhQUREREREVcbCgoiIiIiIqoyFBRERERERVRkLCyIiIiIiqjIWFkREREREVGUsLIiIiIiIqMpYWBARERERUZWxsCAqw2effQaZTCbKtUNDQyGTyRAfH1/r1y4sLMRHH30EFxcXyOVy+Pv713oMFSHme0REddvo0aPRrFkzUa4tZtv04MEDjBs3Do6OjpDJZJgyZYoocTyNmO8RsbCok+Li4jB58mS0atUKFhYWsLCwgKenJ4KCgnDhwoUS+xb/Ay3vkZSUBACIj4+HTCbDN998U+51mzVrhiFDhpT52tmzZyGTyRAaGlpteT5Nbm4uPvvsM0RERNTaNR81f/587N69W5Rrl2fdunVYsGABhg0bhh9//BFTp04VNR4pvkdEhqy4aC9+GBsbw8nJCaNHj8adO3cqdc6IiAjIZDJs37693H1kMhkmT55c5mvbt2+HTCar1e/qu3fv4rPPPkN0dHStXbOY2G1TeebPn4/Q0FBMmjQJYWFhGDlypGixSPU9IsBY7ACodu3duxfDhw+HsbEx3nrrLXh5eUEul+PKlSvYuXMnVq1ahbi4OLi6upY4btWqVbCysip1vvr169dS5NUvNzcXc+fOBQD07t27xGuffPIJZs6cWaPXnz9/PoYNG1aqV2DkyJF4/fXXoVAoavT6Zfnzzz/h5OSExYsX1/q1yyLF94ioLvj888/h5uaG/Px8nDx5EqGhoTh27BhiYmJgZmYmdng17u7du5g7dy6aNWuGjh07lnjtu+++g0ajqbFri902lefPP//Es88+izlz5ohy/UdJ9T0iFhZ1yvXr1/H666/D1dUVhw4dQuPGjUu8/tVXX2HlypWQy0t3ZA0bNgx2dna1FarojI2NYWwszj8PIyMjGBkZiXLtlJQUvSgWxXyPiOqCgQMHwsfHBwAwbtw42NnZ4auvvsIvv/yC1157TeToxGViYiLatcVsm1JSUuDp6SnKtXUh5ntEvBWqTvn666+Rk5OD9evXlyoqgKJ/jO+//z5cXFxEiK5i0tLS8OGHH6J9+/awsrKCtbU1Bg4ciPPnz5faNz8/H5999hlatWoFMzMzNG7cGK+++iquX7+O+Ph42NvbAwDmzp2r7fb/7LPPAJS+R7Ndu3bo06dPqWtoNBo4OTlh2LBh2m3ffPMNunXrhoYNG8Lc3Bze3t6lbgGQyWTIycnBjz/+qL326NGjAZQ/fmDlypVo27YtFAoFmjRpgqCgIGRkZJTYp3fv3mjXrh0uX76MPn36wMLCAk5OTvj666+f+L4W38p2+PBhXLp0SRtTRESE9jaGx7uci4959Pa10aNHw8rKCnfu3IG/vz+srKxgb2+PDz/8EGq1utR7t3TpUrRv3x5mZmawt7fHgAEDcPbsWUm+R0R1WY8ePQAU/UD1qCtXrmDYsGGwtbWFmZkZfHx88Msvv4gRIm7evIl3330XHh4eMDc3R8OGDREQEFDmWKyMjAxMnToVzZo1g0KhgLOzM0aNGoV79+4hIiICzzzzDABgzJgx2u+f4u+6R8dYqFQq2NraYsyYMaWukZWVBTMzM3z44YcAgIKCAsyePRve3t6wsbGBpaUlevTogcOHD2uP0bVtAorGxs2bNw/u7u5QKBRo1qwZZs2aBaVSWWK/4tuRjx07hi5dusDMzAzNmzfHhg0bnvi+FrcBcXFx+O2337QxxcfHl/tdXFa7oct3b3W237XxHtF/WFjUIXv37kWLFi3QtWtXnY9NS0vDvXv3Sjwe/4OtNty4cQO7d+/GkCFDsGjRIsyYMQMXL15Er169cPfuXe1+arUaQ4YMwdy5c+Ht7Y2FCxfigw8+QGZmJmJiYmBvb49Vq1YBAF555RWEhYUhLCwMr776apnXHT58OI4cOaIdU1Ls2LFjuHv3Ll5//XXttqVLl6JTp074/PPPMX/+fBgbGyMgIAC//fabdp+wsDAoFAr06NFDe+0JEyaUm/dnn32GoKAgNGnSBAsXLsTQoUOxZs0a9O/fHyqVqsS+6enpGDBgALy8vLBw4UK0bt0aH3/8MX7//fdyz29vb4+wsDC0bt0azs7O2pjatGlT7jHlUavV8PPzQ8OGDfHNN9+gV69eWLhwIdauXVtiv7fffhtTpkyBi4sLvvrqK8ycORNmZmY4efKkJN8jorqs+A/HBg0aaLddunQJzz77LP755x/MnDkTCxcuhKWlJfz9/bFr165aj/HMmTM4fvw4Xn/9dXz77beYOHEiDh06hN69eyM3N1e734MHD9CjRw8sW7YM/fv3x9KlSzFx4kRcuXIFt2/fRps2bfD5558DAN555x3t90/Pnj1LXdPExASvvPIKdu/ejYKCghKv7d69G0qlUts+ZGVl4fvvv0fv3r3x1Vdf4bPPPkNqair8/Py0Yzl0bZuAoh6l2bNno3Pnzli8eDF69eqFkJCQEu1SsWvXrmHYsGHo168fFi5ciAYNGmD06NG4dOlSuedv06YNwsLCYGdnh44dO2pjKv7jXhcV+e6t7va7Nt4jeoRAdUJmZqYAQPD39y/1Wnp6upCamqp95Obmal+bM2eOAKDMh4eHh3a/uLg4AYCwYMGCcmNwdXUVBg8eXOZrZ86cEQAI69evf2Ie+fn5glqtLrEtLi5OUCgUwueff67dtm7dOgGAsGjRolLn0Gg0giAIQmpqqgBAmDNnTql9ivMuFhsbKwAQli1bVmK/d999V7Cysirxnj36/wVBEAoKCoR27doJzz//fIntlpaWQmBgYKlrr1+/XgAgxMXFCYIgCCkpKYKpqanQv3//ErkvX75cACCsW7dOu61Xr14CAGHDhg3abUqlUnB0dBSGDh1a6lqP69Wrl9C2bdsS2w4fPiwAEA4fPlxie/Fn/uhnFhgYKAAo8VkIgiB06tRJ8Pb21j7/888/BQDC+++/XyqG4s9HEKT5HhEZsuJ/WwcPHhRSU1OFW7duCdu3bxfs7e0FhUIh3Lp1S7vvCy+8ILRv317Iz8/XbtNoNEK3bt2Eli1barcVf4ds27at3OsCEIKCgsp8bdu2bWV+Bz3u8e9eQRCEEydOlPr3Pnv2bAGAsHPnzlL7F3//PKlNCgwMFFxdXbXP//jjDwGA8Ouvv5bYb9CgQULz5s21zwsLCwWlUllin/T0dMHBwUEYO3asdpsubVN0dLQAQBg3blyJ/T788EMBgPDnn39qt7m6ugoAhCNHjmi3paSkCAqFQpg+fXqpaz2urDb88e/iYmW1GxX97q3u9rs23yMSBPZY1BFZWVkAUOYA7N69e8Pe3l77WLFiRal9duzYgfDw8BKP9evX13jcj1MoFNoxIGq1Gvfv34eVlRU8PDwQFRVVIl47Ozu89957pc5RmWnoWrVqhY4dO2LLli3abWq1Gtu3b8eLL74Ic3Nz7fZH/396ejoyMzPRo0ePEvHp4uDBgygoKMCUKVNKjH8ZP348rK2tS/SEAEWf8YgRI7TPTU1N0aVLF9y4caNS16+MiRMnlnjeo0ePEtffsWMHZDJZmYMAK/P56ON7RCRlffv2hb29PVxcXDBs2DBYWlril19+gbOzM4CiXuw///wTr732GrKzs7U92ffv34efnx+uXr1a6VmkKuvR716VSoX79++jRYsWqF+/fqn2wcvLC6+88kqpc1Tm++f555+HnZ1difYhPT0d4eHhGD58uHabkZERTE1NARTdCpqWlobCwkL4+PhUun3Yt28fAGDatGkltk+fPh0ASn33eXp6am9rA4p6SDw8PGrtu68i373V3X7r23uk7zi6pY6oV68egKIu4MetWbMG2dnZSE5OLvEP/lE9e/aslcHbT/vSKL4vf+XKlYiLiytx337Dhg21///69evw8PCo1gFcw4cPx6xZs3Dnzh04OTkhIiICKSkpJRoOoOiWs//973+Ijo4ucf9mZefVvnnzJgDAw8OjxHZTU1M0b95c+3oxZ2fnUtdq0KBBqamEa0rxeInHr5+enq59fv36dTRp0gS2trbVck19e4+IpG7FihVo1aoVMjMzsW7dOhw5cqTELGzXrl2DIAj49NNP8emnn5Z5jpSUFDg5OVVbTE/7Ds3Ly0NISAjWr1+PO3fuQBAE7WuZmZna/3/9+nUMHTq02uIyNjbG0KFDsWnTJiiVSigUCuzcuRMqlapU+/Djjz9i4cKFuHLlSolbNN3c3Cp17Zs3b0Iul6NFixYltjs6OqJ+/fqlvvuaNm1a6hyPfz/XpIp891Z3+61v75G+Y2FRR9jY2KBx48aIiYkp9VrxmIuaXmzMzMwMeXl5Zb5WfP/r06YxnD9/Pj799FOMHTsW8+bNg62tLeRyOaZMmVKj0/8BRYVFcHAwtm3bhilTpmDr1q2wsbHBgAEDtPscPXoUL730Enr27ImVK1eicePGMDExwfr167Fp06Yaja9YebMlPdrI6qK8xvzxwdhPu76UVPd7RGRounTpop0Vyt/fH8899xzefPNNxMbGwsrKSvt9++GHH8LPz6/Mczz+h9yTKBSKKrcP7733HtavX48pU6bA19cXNjY2kMlkeP3112u8fXj99dexZs0a/P777/D398fWrVvRunVreHl5aff56aefMHr0aPj7+2PGjBlo1KgRjIyMEBISUmpQvK4q+sOVVNuH2vjuFes9qmtYWNQhgwcPxvfff4/Tp0+jS5cutX59V1dXXL58uczXYmNjtfs8yfbt29GnTx/88MMPJbZnZGSU6FFxd3fHqVOnoFKpyp0aUNceBDc3N3Tp0gVbtmzB5MmTsXPnTvj7+5f4FW/Hjh0wMzPDH3/8UWJ7WbeNVfT6xe9JbGwsmjdvrt1eUFCAuLg49O3bV6c8dFU8WPPxwfqP/8qjC3d3d/zxxx9IS0t7Yq+FvrxHRIas+I/fPn36YPny5Zg5c6b235mJiUm1/PtydXXVtgOP06V9CAwMxMKFC7Xb8vPzS313ubu7l/kj26N0bR969uyJxo0bY8uWLXjuuefw559/4v/+7/9Kxde8eXPs3LmzxPkfvyVUl2u7urpCo9Hg6tWrJSbbSE5ORkZGxlPfs6qqqfahOttvsd+juoZjLOqQjz76CBYWFhg7diySk5NLvV7T1figQYNw+/btUispK5VKfP/992jUqBE6d+78xHMYGRmVinPbtm2l7uUdOnQo7t27h+XLl5c6R/HxFhYWAEp/IT7J8OHDcfLkSaxbtw737t0r1c1tZGQEmUxW4tea+Pj4MlePtrS0rNC1+/btC1NTU3z77bclcv/hhx+QmZmJwYMHVzj+ynB1dYWRkRGOHDlSYvvKlSsrfc6hQ4dCEATtAkePejRHfXmPiAxd79690aVLFyxZsgT5+flo1KgRevfujTVr1iAxMbHU/qmpqTqdf9CgQTh58iQiIyNLbM/IyMDGjRvRsWNHODo6PvEcZbUPy5YtK/Xr+dChQ3H+/PkyZ64qPt7S0lJ7/YqQy+UYNmwYfv31V4SFhaGwsLDM9uHRawDAqVOncOLEiRL76dI2DRo0CACwZMmSEtsXLVoEADX+3efu7g4AJdoHtVpdahZAXVR3+y32e1TXsMeiDmnZsiU2bdqEN954Ax4eHtqVtwVBQFxcHDZt2gS5XK4dnPeo7du3lznwu1+/fnBwcNA+P3ToEPLz80vt5+/vj3feeQfr1q1DQEAAxo4di06dOuH+/fvYsmULYmJisGHDBu3AtvIMGTIEn3/+OcaMGYNu3brh4sWL2LhxY4lfqQFg1KhR2LBhA6ZNm4bTp0+jR48eyMnJwcGDB/Huu+/i5Zdfhrm5OTw9PbFlyxa0atUKtra2aNeuHdq1a1fu9V977TV8+OGH+PDDD2Fra1vql7rBgwdj0aJFGDBgAN58802kpKRgxYoVaNGiRan79729vXHw4EEsWrQITZo0gZubW5lTAdvb2yM4OBhz587FgAED8NJLLyE2NhYrV67EM888U+64mOpiY2ODgIAALFu2DDKZDO7u7ti7dy9SUlIqfc4+ffpg5MiR+Pbbb3H16lUMGDAAGo0GR48eRZ8+fTB58mQA+vMeEdUFM2bMQEBAAEJDQzFx4kSsWLECzz33HNq3b4/x48ejefPmSE5OxokTJ3D79u1S6wvt2LEDV65cKXXewMBAzJw5E9u2bUPPnj0xYcIEtG7dGnfv3kVoaCgSExMrNFnIkCFDEBYWBhsbG3h6euLEiRM4ePBgifF3xXls375d2xZ5e3sjLS0Nv/zyC1avXg0vLy+4u7ujfv36WL16NerVqwdLS0t07dr1iWMhhg8fjmXLlmHOnDlo3759qem6hwwZgp07d+KVV17B4MGDERcXh9WrV8PT07PE+Edd2iYvLy8EBgZi7dq1yMjIQK9evXD69Gn8+OOP8Pf3L3P9perUtm1bPPvsswgODtb2QG/evBmFhYWVPmd1t99iv0d1Ti3PQkUScO3aNWHSpElCixYtBDMzM8Hc3Fxo3bq1MHHiRCE6OrrEvk+abhaPTCVXPPVoeY+wsDBBEIqm1ps6darg5uYmmJiYCNbW1kKfPn2E33//vUKx5+fnC9OnTxcaN24smJubC927dxdOnDgh9OrVS+jVq1eJfXNzc4X/+7//017L0dFRGDZsmHD9+nXtPsePHxe8vb0FU1PTElPXPT5d3aO6d+9e5tR1xX744QehZcuWgkKhEFq3bi2sX7++zPNduXJF6Nmzp2Bubi4A0E6rWt70fcuXLxdat24tmJiYCA4ODsKkSZOE9PT0EvuUNV2sIJSeHrE85R2fmpoqDB06VLCwsBAaNGggTJgwQYiJiSlzullLS8tSx5eVf2FhobBgwQKhdevWgqmpqWBvby8MHDhQiIyM1O4jxfeIyJAV/9s6c+ZMqdfUarXg7u4uuLu7C4WFhYIgCML169eFUaNGCY6OjoKJiYng5OQkDBkyRNi+fbv2uOKpR8t7HD16VBAEQbh9+7Ywbtw4wcnJSTA2NhZsbW2FIUOGCCdPnqxQ7Onp6cKYMWMEOzs7wcrKSvDz8xOuXLkiuLq6lpq2+v79+8LkyZMFJycnwdTUVHB2dhYCAwOFe/fuaffZs2eP4OnpKRgbG5f4rivvu0Kj0QguLi4CAOF///tfma/Pnz9fcHV1FRQKhdCpUydh7969ZZ5Pl7ZJpVIJc+fO1bZ1Li4uQnBwcIlpgAWh/Cnfy2o/y1Le8devXxf69u0rKBQKwcHBQZg1a5YQHh5e5nSzFf3ure72u7beIxIEmSBwNAoREREREVUNx1gQEREREVGVsbAgIiIiIqIqY2FBRERERERVxsKCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqsjq3QJ5Go8Hdu3dRr149nZaEJyIyZIIgIDs7G02aNIFcXnd/c2IbQURUki7tQ50rLO7evQsXFxexwyAikqRbt27B2dlZ7DBEwzaCiKhsFWkf6lxhUa9ePQBFb461tbVOx6pUKhw4cAD9+/eHiYlJTYRXKwwhD+YgHYaQhyHkAFQtj6ysLLi4uGi/I+uqut5GMAfpMIQ8DCEHwDDyqK32oc4VFsVd29bW1pVqNCwsLGBtba23/2EBhpEHc5AOQ8jDEHIAqiePun77T11vI5iDdBhCHoaQA2AYedRW+1B3b6QlIiIiIqJqw8KCiIiIiIiqTNTCYtWqVejQoYO2y9nX1xe///77E4/Ztm0bWrduDTMzM7Rv3x779u2rpWiJiKi2sH0gItI/ohYWzs7O+PLLLxEZGYmzZ8/i+eefx8svv4xLly6Vuf/x48fxxhtv4O2338a5c+fg7+8Pf39/xMTE1HLkRERUk9g+EBHpH1ELixdffBGDBg1Cy5Yt0apVK3zxxRewsrLCyZMny9x/6dKlGDBgAGbMmIE2bdpg3rx56Ny5M5YvX17LkRMRUU1i+0BEpH8kMyuUWq3Gtm3bkJOTA19f3zL3OXHiBKZNm1Zim5+fH3bv3l3ueZVKJZRKpfZ5VlYWgKLR8SqVSqcYi/fX9TipMYQ8mIN0GEIeBpGDWoPP915GK3Xl8pBy7jXVPhAR1RVHr97Dn3dlGCgINXod0QuLixcvwtfXF/n5+bCyssKuXbvg6elZ5r5JSUlwcHAosc3BwQFJSUnlnj8kJARz584ttf3AgQOwsLCoVMzh4eGVOk5qDCEP5iAdhpCHPuew9YYcfyfL0VBhBBvTcBjr2B+dm5tbM4FVQU23DwB/fHocc5AOQ8jDEHIA9D+Pm2m5mLL1ArLyjeBzJgGvd3HV6Xhd8ha9sPDw8EB0dDQyMzOxfft2BAYG4q+//iq38dBVcHBwiV+xihf56N+/f6XmKA8PD0e/fv30dh5jwDDyYA7SYQh56HsOP51KwN8nrkAG4JVmGgz00z2P4j+opaSm2weAPz6VhzlIhyHkYQg5APqZh1INLI4xQla+DK5WAixSLmHfvrLHqpVHlx+eRC8sTE1N0aJFCwCAt7c3zpw5g6VLl2LNmjWl9nV0dERycnKJbcnJyXB0dCz3/AqFAgqFotR2ExOTSv8BUZVjpcQQ8mAO0mEIeehjDkevpuJ/+2IBANP7tYTLg38qlYcU867p9gHgj0+PYw7SYQh5GEIOgP7mIQgCpmy9gMTcZDS0NMXYVrk1/sOT6IXF4zQaTYlu6Uf5+vri0KFDmDJlinZbeHh4uffcEhEZshupDxC0MQpqjYBXOzvhnR7N8Pvv/4gdVo2pifaBPz6VjTlIhyHkYQg5APqXx+q/rmNfTDKM5TIsf8MLKZdO1PgPT6IWFsHBwRg4cCCaNm2K7OxsbNq0CREREfjjjz8AAKNGjYKTkxNCQkIAAB988AF69eqFhQsXYvDgwdi8eTPOnj2LtWvXipkGEVGty8xVYdyPZ5GVX4jOTetj/ivtIYNG7LCqDdsHIqLKO/JvKr7efwUAMOeltvBxbQAd74CqFFELi5SUFIwaNQqJiYmwsbFBhw4d8Mcff6Bfv34AgISEBMjl/41A7NatGzZt2oRPPvkEs2bNQsuWLbF79260a9dOrBSIiGpdoVqDyT9H4ca9HDSxMcOakT4wMzGCSmU4hQXbByKiykm4n4v3fj4HjQAEeDtjRNemKCwsrJVri1pY/PDDD098PSIiotS2gIAABAQE1FBERETS97/f/sHRq/dgbmKE7wJ9YF+v9K08+o7tAxGR7nILCvFO2Flk5qng5VIf8/zbQSaT1dr1RV0gj4iIdLPpVAJCj8cDABYP90LbJjbiBkRERJIgCAI+3nERV5KyYWdlitUjOsPMxKhWY2BhQUSkJ05cv4/Ze2IAANP7tcKAdo1FjoiIiKTi+6Nx+PX8XRjLZVj5ljca25jXegwsLIiI9EDC/VxM2hiJQo2AF72aYPLzLcQOiYiIJOLY1XsIeTgr4KdDPNHFzVaUOFhYEBFJXHa+CuM2nEFGrgodnG2wYFiHWr1nloiIpOtWWi4m/xwFjQAM83bGKF/dVtauTiwsiIgkTK0RMGVzNP5NfgAHawW+G+VT6/fMEhGRNOUVqDEhLFL7w9P/anmw9uNYWBARSdiCP2Jx6EoKFMZyrB3pAwdrM7FDIiIiCRAEATN3XsDlxCw0tDTF6hHeov/wxMKCiEiidkbdxuq/rgMAvh7WAV4u9cUNiIiIJOOHY3HYE30XRnIZVrzVGU3q1/5g7cexsCAikqBzCemYufMiACCojzte7ugkckRERCQVx6/dQ8jvRStrfzK4DZ5t3lDkiIqwsCAikpjEzDy8ExaJgkIN+nk6YHo/D7FDIiIiibidnovJP5+DWiPg1c5OGN2tmdghabGwICKSkHyVGu9siERqthKtHethyfCOkMs5AxQRERW1ERPCIpGWU4B2TtaY/0p7Sc0SyMKCiEgiBEHAjO0XcPFOJmwtTfHdKB9YKozFDouIiCRAEATM2nkRl+5mwVYig7Ufx8KCiEgiVkZcf2TV1M5wsbUQOyQiIpKI0OPx2HnuDozkMix/sxOcG0ivjWBhQUQkAeGXk/HNgVgAwNyX20pmIB4REYnv5I37+N9vRStrzxrUBt3c7USOqGwsLIiIRBablI0pm89BEIBRvq54q6t4q6YSEZG03MnIQ9DGKKg1Avw7NsHY7s3EDqlcLCyIiESUnlOAcRvOIKdADd/mDfHpEE+xQyIiIonIV6kx6adI3M8pgGdja4S82kFSg7Ufx8KCiEgkKrUG726Mwq20PLjYmmPlW51hYsSvZSIiKhqs/X+7YnDhdiYaWJhgzUhvmJtKa7D249iCERGJ5H97L+PEjfuwNDXC96OeQQNLU7FDIiIiidhw4iZ2RN2GXAYsf1M/JvRgYUFEJIKfTyfgxxM3AQCLh3eEh2M9kSMiIiKpOHXjPubtvQwACB7YBt1bSHOw9uNELSxCQkLwzDPPoF69emjUqBH8/f0RGxv7xGNCQ0Mhk8lKPMzMzGopYiKiqjsTn4bZe2IAAB/2b4X+bR1FjoiIiKQiMTMPQZuiUKgR8JJXE4zr4SZ2SBUmamHx119/ISgoCCdPnkR4eDhUKhX69++PnJycJx5nbW2NxMRE7ePmzZu1FDERUdXcycjDxLBIqNQCBndojKA+LcQOiYiIJCJfpcbEsEjce1CANo2t8dVQaQ/WfpyohcX+/fsxevRotG3bFl5eXggNDUVCQgIiIyOfeJxMJoOjo6P24eDgUEsRExFVXl6BGhPCzmpn91gwTL8ajNrEHm0iqmsEQcCnu2Nw/nYmbMxNsGaE9AdrP05SYywyMzMBALa2tk/c78GDB3B1dYWLiwtefvllXLp0qTbCIyKqNEEQ8PGOC4i5kwVbS1OsHeUNC1NjscOSLPZoE1Fd89OpBGyLLB6s3QlNG0p/sPbjJNOqaTQaTJkyBd27d0e7du3K3c/DwwPr1q1Dhw4dkJmZiW+++QbdunXDpUuX4OzsXGp/pVIJpVKpfZ6VlQUAUKlUUKlUOsVYvL+ux0mNIeTBHKTDEPKojRzWHo3DL+fvwlguw7fDO8DByqTar1eVPKT2+e3fv7/E89DQUDRq1AiRkZHo2bNnuccV92gTEemTM/FpmPtL0Q/lHw9ojR4t7UWOqHIkU1gEBQUhJiYGx44de+J+vr6+8PX11T7v1q0b2rRpgzVr1mDevHml9g8JCcHcuXNLbT9w4AAsLCpXCYaHh1fqOKkxhDyYg3QYQh41lcPldBnWXpEDkMHftRD3/zmJff/UyKUAVC6P3NzcGoik+ujao63RaNC5c2fMnz8fbdu2rY0QiYgqJTkrH+9uLBqsPbhDY7zTs7nYIVWaJAqLyZMnY+/evThy5EiZvQ5PYmJigk6dOuHatWtlvh4cHIxp06Zpn2dlZcHFxQX9+/eHtbW1TtdSqVQIDw9Hv379YGJiotOxUmIIeTAH6TCEPGoyh7h7OfhkzSkIKMRwH2fMe6lNjY2rqEoexb25UlRTPdoAe7UfxxykwxDyMIQcgJrNQ1mowYSws0jNVsLDwQpfvNQGhYWF1X6d2urRFrWwEAQB7733Hnbt2oWIiAi4uek+nZZarcbFixcxaNCgMl9XKBRQKBSltpuYmFT6D4iqHCslhpAHc5AOQ8ijunPIzldh0qZoZOcXwse1Aeb5t4epcc0PbatMHlL+7GqqRxtgr3Z5mIN0GEIehpADUDN5bL4uR3SKHBZGAl5rkoG/Dh2o9ms8qqZ7tEUtLIKCgrBp0ybs2bMH9erVQ1JSEgDAxsYG5ubmAIBRo0bByckJISEhAIDPP/8czz77LFq0aIGMjAwsWLAAN2/exLhx40TLg4jocRqNgKlbonE9NQeNbcywaoR3rRQVhqYme7QB9mo/jjlIhyHkYQg5ADWXx+Yzt3HixGXIZMDyt7zRo2XNLYJXWz3aohYWq1atAgD07t27xPb169dj9OjRAICEhATI5f81xunp6Rg/fjySkpLQoEEDeHt74/jx4/D09KytsImInmrxwX9x8J8UKIzlWDPSG/b1SvecUvlqo0cbYK92eZiDdBhCHoaQA1C9eUTeTMfnvxUNtpvh54HnPRtXy3mfpqZ7tEW/FeppIiIiSjxfvHgxFi9eXEMRERFV3e8XE7Hsz6JfyUNebY8OzvXFDUgPsUebiAxVclY+Jv1UtFDqoPaOmNTLXeyQqo0kBm8TERmKK0lZmL7tPADg7efc8Gpn3W7foSLs0SYiQ1RQqMGknyKRkq1EKwcrLBjmZVALpbKwICKqJhm5BXhnQyRyC9To5t4QwQNbix2S3mKPNhEZorm/XkJUQgaszYyxdqQPLBWG9ac4RxISEVUDtUbAez+fQ0JaLpwbmGP5m51hbMSvWCIiKrL5dAI2nkqATAYsfb0TmtlZih1StWOrR0RUDRb8EYujV+/BzESOtSN9YGtpKnZIREQkEVEJ6Zi9p2hl7Q/7e6BP60YiR1QzWFgQEVXR3gt3sfqv6wCABcO84NlEt2lKiYjIcKVkFw3WLlBrMKCtI97tbTiDtR/HwoKIqAr+SczCjG0XAAATejXHi15NRI6IiIikoqBQg6CNUUjOUqJlIyt885phDdZ+HAsLIqJKysgtwISwSOSp1OjR0g4f+XGwNhER/Wfe3ss4E5+OegpjrBnpDSsDG6z9OBYWRESVoNYIeH9zNBLScuFia45lb3SCkdxwf4UiIiLdbD1zC2EnbxYN1n6jI5rbW4kdUo1jYUFEVAkLD8TiyL+pMDORY80IH9S34GBtIiIqEn0rA5/sjgEATO3bCs+3dhA5otrBwoKISEe/X0zEyoiiwdpfDe3AwdpERKSVmq3ExLCiwdr9PR0wuU8LsUOqNSwsiIh0cDU5Gx8+XFl73HNueLmjk8gRERGRVKjURYO1k7Ly4W5viYWveUFeh26TZWFBRFRBWfkqTAiLRM7DlbVncmVtIiJ6xBe//YPT8WmwUhhj7Sgf1DMzETukWsXCgoioAjQaAdO2nMeNezlwql80WJsraxMRUbHtkbcRejweALB4eEe414HB2o9jq0hEVAHLD1/DwX+SYWosx6oRndHQSiF2SEREJBEXbmdg1q6LAIApfVuin2fdGKz9OBYWRERPcfhKChYf/BcA8D//dujgXF/cgIiISDLuPXg4WLtQg75tGuH951uKHZJoWFgQET3Bzfs5+GDzOQgC8FbXpnjNx0XskIiISCKKB2vfzcxHc3tLLBresU4N1n4cCwsionLkFagx8acoZOUXolPT+pj9oqfYIRERkYTM3/cPTsU9HKw90gfWdWyw9uNYWBARlUEQBMzadRH/JGbBzsoUq97yhsLYSOywiIhIInZG3cb6v+MBAAtf80KLRnVvsPbjWFgQEZVhw4mb2HXuDozkMix/szMcbczEDomIiCQi5k4mgncWDdZ+//kW8GvrKHJE0iBqYRESEoJnnnkG9erVQ6NGjeDv74/Y2NinHrdt2za0bt0aZmZmaN++Pfbt21cL0RJRXRF5Mw3z9l4GAAQPbI1nmzcUOSIiIpKK+w+UmBAWCWWhBi+0boQpfVuJHZJkiFpY/PXXXwgKCsLJkycRHh4OlUqF/v37Iycnp9xjjh8/jjfeeANvv/02zp07B39/f/j7+yMmJqYWIyciQ5WSnY93N0ahUCNgcIfGePs5N7FDIiIiiShUazB50zncyciDmx0Haz/OWMyL79+/v8Tz0NBQNGrUCJGRkejZs2eZxyxduhQDBgzAjBkzAADz5s1DeHg4li9fjtWrV9d4zERkuFQPG4zkLCVaNrLC10M7QCZjg0FEREVCfr+CEzfuw9LUCGtGesPGvG4P1n6cqIXF4zIzMwEAtra25e5z4sQJTJs2rcQ2Pz8/7N69u8z9lUollEql9nlWVhYAQKVSQaVS6RRf8f66Hic1hpAHc5AOQ8ijOPav98fidFwaLBVGWPa6F0zlgl7lVZXPQmp5hoSEYOfOnbhy5QrMzc3RrVs3fPXVV/Dw8Hjicdu2bcOnn36K+Ph4tGzZEl999RUGDRpUS1ETkSHbE30XPxyLA1A0WLuVQz2RI5IeyRQWGo0GU6ZMQffu3dGuXbty90tKSoKDQ8nVDB0cHJCUlFTm/iEhIZg7d26p7QcOHICFhUWlYg0PD6/UcVJjCHkwB+nQ9zzO3Zch9N9bAIDhrgWIPfMXnj7iS5oq81nk5ubWQCSVV3yr7DPPPIPCwkLMmjUL/fv3x+XLl2FpaVnmMcW3yoaEhGDIkCHYtGkT/P39ERUV9cR2hYjoaW7nAN/uKRp7N7lPCwxo11jkiKRJMoVFUFAQYmJicOzYsWo9b3BwcIkejqysLLi4uKB///6wtrbW6VwqlQrh4eHo168fTEz0t+vLEPJgDtJhCHnEJmbgo9WnAADjnmuGj/30cyBeVT6L4t5cqeCtskQkFWk5Bfgh1gjKQg16e9hjaj/9bCNqgyQKi8mTJ2Pv3r04cuQInJ2dn7ivo6MjkpOTS2xLTk6Go2PZ03wpFAooFIpS201MTCr9R1BVjpUSQ8iDOUiHvuaRoyzElG2XoNTI0KVZA8wc2AbGRvo9E3dlPgupf3Y1cassEdHTFKo1mLr1AtKUMjS1NcfS4Z1gxMHa5RK1sBAEAe+99x527dqFiIgIuLk9ffYVX19fHDp0CFOmTNFuCw8Ph6+vbw1GSkSGSBAEzNx5EddSc2BtImDJax30vqgwRDV1qyzAcXiPYw7SYQh5GEIOX+6PxfEbaTCVC1j2WjtYmOhnPrU1Bk/UwiIoKAibNm3Cnj17UK9ePe2Xv42NDczNzQEAo0aNgpOTE0JCQgAAH3zwAXr16oWFCxdi8ODB2Lx5M86ePYu1a9eKlgcR6acfj8fj1/N3YSyXYUyrQtjXK927SeKrqVtlAY7DKw9zkA5DyENfc4i6J8OPV40AAG+10CD+/AnEnxc5qCqq6TF4ohYWq1atAgD07t27xPb169dj9OjRAICEhATI5f/9gtitWzds2rQJn3zyCWbNmoWWLVti9+7dHJhHRDqJSkjHF/v+AQB85NcKDhmXRI6IylKTt8oCHIf3OOYgHYaQhz7n8E9iNj7+7hQADcZ1b4r2mht6mUex2hqDJ/qtUE8TERFRaltAQAACAgJqICIiqgvuP1AiaGMUVGoBg9s3xmjfpvj9dxYWUlJbt8pyHF7ZmIN0GEIe+pZDek4BgjZHI1+lQY+Wdviwvwf+2H9D7/IoS02PwZPE4G0iotqi1giYsiUaiZn5aG5viS+HtgfXwJMe3ipLRGIoVGvw/uZzuJWWh6a2Flj2Bgdr64KjFImoTll66CqOXr0HcxMjrB7hjXpm+v3rk6FatWoVMjMz0bt3bzRu3Fj72LJli3afhIQEJCYmap8X3yq7du1aeHl5Yfv27bxVloh0suBArLaNWDPSG/UtTMUOSa9UqsciLi4OR48exc2bN5Gbmwt7e3t06tQJvr6+MDMzq+4YiYiqRURsCpb9eRUAMP/Vdlw1VcJ4qywR1ba9F+5izV83AAALAjqgTWPdxlmRjoXFxo0bsXTpUpw9exYODg5o0qQJzM3NkZaWhuvXr8PMzAxvvfUWPv74Y7i6utZUzEREOruTkYcpW6IhCMBbXZvilU5PHghMRER1xz+JWZix7QIAYELP5hjSoYnIEemnChcWnTp1gqmpKUaPHo0dO3bAxcWlxOtKpRInTpzA5s2b4ePjg5UrV/JXIyKShIJCDd7dGIWMXBU6ONtg9oueYodk0NirTUT6JCO3ABPCIpGnUqNHSzt8NKC12CHprQoXFl9++SX8/PzKfV2hUKB3797o3bs3vvjiC8THx1dHfEREVTZ/3z84fysDNuYmWPFmZyiMjcQOySCxV5uI9I1aI+D9zdFISMuFcwNzfPs6B2tXRYULiycVFY9r2LAhGjZsWKmAiIiq028XEhF6PB4AsOg1L7jYVm7RM3oy9moTkT5aeCAWR/5NhZmJHGtGeqOBJQdrV0WlZoUKDQ0tc3thYSGCg4OrEg8RUbW5kfoAH+8oumd2Um93vNDGQeSIDNeXX36JU6dO4d133y1VVAD/9WqvXr0aV65cQfPmzUWIkojoP/suJmJlxHUAwFdDO6BtExuRI9J/lSos3n//fQQEBCA9PV27LTY2Fl27dsXPP/9cbcEREVVWXoEa726MwgNlIbq42WJ6v1Zih2TQdO3V9vb2rsFoiIieLDYpGx9uOw8AGN/DDS93dBI5IsNQqcLi3LlzuH37Ntq3b4/w8HCsWLECnTt3RuvWrXH+/PnqjpGISGdzfonBlaRs2FmZYvkbnWBsxGV7agt7tYlIyjJzVZgQdha5BWp0c2+IjzlYu9pUqqV1d3fH33//jVdffRUDBgzA1KlT8f3332Pjxo2wsWE3EhGJa9vZW9h69jbkMuDb1zuhkTVnIqpN7NUmIqlSawR8sOUc4u/nwqm+OZa/2Zk/PFWjSr+Tv/32GzZv3gxfX1/Ur18fP/zwA+7evVudsRER6Sw2KRuf7okBAEzt2wrdWtiJHFHdw15tIpKqxeH/IiI2FQrjosHathysXa0qVVhMmDABAQEB+Pjjj3H06FFcuHABpqamaN++PbZu3VrdMRIRVUiOshCTNkYiX6VBz1b2COrTQuyQ6iT2ahORFO2PScTyw9cAAF8ObY92Tvw+qm6VKiz+/vtvnDp1CtOnT4dMJoOjoyP27duHzz//HGPHjq3uGImInkoQBMzadRE3UnPgaG2GJcM7Qs65yEXDXm0ikpKrydmYvrWox3Rsdze80slZ5IgMU6UKi8jISHh5eZXaHhQUhMjIyCoHRUSkq59P38Ke6Lswksuw/M1O7N4WEXu1iUhKMvNUeCcsEjkFajzb3BbBgzhYu6ZUeIG8RykUinJf8/DwqHQwRESVEXMnE5/9egkA8JGfB3ya2YocUd1W3Ktd/ANUca/2ihUrMHbsWLz22msiR0hEdYVGI2DqlmjE3ctBExszrHizM0w4WLvGVPidHTBgAE6ePPnU/bKzs/HVV19hxYoVVQqMiKgisvNVmLwpCgWFGrzQuhHG9+DCa2JjrzYRScWSQ1fx55WUh4O1fdDQqvwfx6nqKtxjERAQgKFDh8LGxgYvvvgifHx80KRJE5iZmSE9PR2XL1/GsWPHsG/fPgwePBgLFiyoybiJiCAIAmbuvKidNnDha14cVyEB7NUmIin441ISvj10FQAw/5X2aO/Mwdo1rcI9Fm+//TZu3LiBWbNm4fLly3jnnXfQo0cPPPPMM/Dz88N3332Hpk2b4syZM9iyZQuaNm361HMeOXIEL774Ipo0aQKZTIbdu3c/cf+IiAjIZLJSj6SkpIqmQUQG5KeTN/HbhUQYy2VY9mYn1LfguAqxsFebiKTkWsp/g7VHd2uGod4crF0bdBpjoVAoMGLECIwYMQIAkJmZiby8PDRs2BAmJiY6XzwnJwdeXl4YO3YsXn311QofFxsbC2tra+3zRo0a6XxtItJvF29nYt7efwAAMwe2RuemDUSOqG5jrzYRSUVWftFg7QfKQnR1s8X/DW4jdkh1RqUGbxezsbGp0pzkAwcOxMCBA3U+rlGjRqhfv36lr0tE+i0rX4WgTVEoUGvQz9MBbz/nJnZIdd7bb7+NESNGYNu2bdiyZQvWrl2LzMxMAIBMJoOnpyf8/Pxw5swZtGnDRp6IaoZGI2DalmjcSM1BYxszrHiLg7Vrk06FxbffflvmdhsbG7Rq1Qq+vr7VEtTTdOzYEUqlEu3atcNnn32G7t27l7uvUqmEUqnUPs/KygIAqFQqqFQqna5bvL+ux0mNIeTBHKSjtvMQBAEfbbuAhLRcONU3Q4i/JwoLC6t0Tn4W1ZN7dfdqExHp6ts/r+LgPykwNZZj9Qhv2HGwdq3SqbBYvHhxmdszMjKQmZmJbt264ZdffoGtbc1M9di4cWOsXr0aPj4+UCqV+P7779G7d2+cOnUKnTt3LvOYkJAQzJ07t9T2AwcOwMLColJxhIeHV+o4qTGEPJiDdNRWHkeTZNgfZwQjmYDhzg/w9+Hqu25d/ixyc3OrPY6q9moTEeki/HIylhwsGqz9hX87eLnUFzegOkinwiIuLq7c127cuIERI0bgk08+wcqVK6scWFk8PDxKzCjSrVs3XL9+HYsXL0ZYWFiZxwQHB2PatGna51lZWXBxcUH//v1LjNOoCJVKhfDwcPTr10+vf30zhDyYg3TUZh6X7mbhw7WnAAj4eEBrjOnmWi3n5WfxX29uVVR3r/aRI0ewYMECREZGIjExEbt27YK/v3+5+0dERKBPnz6lticmJsLR0VGnaxORfrme+gDTtkQDAAJ9XRHg4yJuQHVUlcZYPKp58+b48ssvMXbs2Oo6ZYV06dIFx44dK/d1hUJR5tSHJiYmlf4DoirHSokh5MEcpKOm88jKV+GDrRegUgvo28YB43u6Qyar3qll6/JnUR15V3evNif4IKKKyM5X4Z0NZ5GtLESXZrb4ZIin2CHVWdVWWABA06ZNa33q1+joaDRu3LhWr0lEtUsQBATvuIibD9er+CagQ7UXFVR11d2rzQk+iOhpNBoB07eex/XUHDham2H5W504WFtE1VpYXLx4Ea6uFb814cGDB7h27Zr2eVxcHKKjo2Fra4umTZsiODgYd+7cwYYNGwAAS5YsgZubG9q2bYv8/Hx8//33+PPPP3HgwIHqTIOIJOanUwn47WLRehXLuV6FXqrNXm1dJvggIv224vA1HLicDFMjOVaP9EajemZih1Sn6VRYlHcPbmZmJiIjIzF9+nQEBgZW+Hxnz54tcT9s8ViIwMBAhIaGIjExEQkJCdrXCwoKMH36dNy5cwcWFhbo0KEDDh48WOY9tURkGGLuZGLer5cBAB8PaI1OXK9Cb9V0r3ZlJvjgzIElMQfpMIQ8ajqHw7GpWHTwXwDAZy+2QVtHyxq5Vl3/LHQ5RqfCon79+uXefiCTyTBu3DjMnDmzwufr3bs3BEEo9/XQ0NASzz/66CN89NFHFT4/Eem37HwVJj9cr+KF1o0wrgfXq9BnuvZq66oyE3xw5sCyMQfpMIQ8aiKHlDxg0UUjCIIM3R00sEw+j337zlf7dR5VVz8LXWYN1KmwOHz4cJnbra2t0bJlS5iZmSElJQVNmjTR5bRERKUIgoBZu2IQfz8XTWzM8E2AF8dVSFx192pXh6dN8MGZA0tiDtJhCHnUVA4PlIUIWHMKeeoceDetj7VjfGBqXHPjKur6Z6HLrIE6FRa9evV64uvnz59H586doVardTktEVEpP5++hV/P34WRXIZlb3ZCA0uOq5C66u7Vrg5Pm+CDMweWjTlIhyHkUZ05CIKA4M0XcC01Bw7WCqwa6Q1L89pZBK+ufha67F+tg7eJiKrDP4lZmPvrJQDADD8PeLvWzKKbVL2qu1ebE3wQ0eNWRlzH/ktJMDGSYdUIDtaWGhYWRCQpOcpCBG2KgrJQg94e9ninR3OxQ6IKqu5ebU7wQUSPOhybgm8OxAIA5r7UDp05mYfksLAgIskQBAGf7I7BjYfzkS96rSPkco6rqKs4wQcRFYu/l4MPfj4HQQDe6NIUb3ZtKnZIVAadCosLFy488fXY2NgqBUNEddu2s7ex69wdGMll+PaNTrDluAoiojovR1mICWGRyMovRKem9fHZS1xZW6p0Kiw6duwImUxW5i9Ixds5awsRVca/ydmY/UsMAGBav1bo4sZxFUREdZ0gCPho+wXEJmfDvp4Cq0d4Q2FsJHZYVA6dCou4uLiaioOI6rDcgkIEbYxCvkqDHi3tMKmXu9ghUSWwV5uIqtvqv27gt4uJRYO13+oMB2sO1pYynQqLmlzYiIjqrjl7LuFqygM0qqfA4uEcV6Gv2KtNRNXpr39T8fUfVwAAc15sC59m7MmWOp0Ki6+//hrvvfcezM3NAQB///03fHx8tHOAZ2dn4+OPP8bKlSurP1IiMkg7Im9jW+RtyGXA0tc7wc6qduYjp+rHXm0iqi437+fg/YeDtYf7uOAtDtbWCzoVFsHBwRg9erS2sBg4cCCio6PRvHnRdJC5ublYs2YNCwsiqpBrKdn4ZHfRuIopfVvB172hyBFRVbBXm4iqQ25B0WDtzDwVOrrUx+f+bdnbqSd0Wv/88e7tJ00DSET0JHkFagRtPIc8lRrdWzREUJ8WYodE1ejo0aMYMWIEfH19cefOHQBAWFgYjh07JnJkRCRlxYO1ryRlw85KgVUjOnOwth7RqbAgIqoun/1yCbHJRQ3HkuGdYMRxFQZjx44d8PPzg7m5Oc6dOwelUgkAyMzMxPz580WOjoik7LujN7D3QiKM5TKsGtEZjW3MxQ6JdMDCgohq3c6o29hy9hZkMuDb1zvCvh7HVRiS//3vf1i9ejW+++47mJiYaLd3794dUVFRIkZGRFJ27Oo9fPl78WBtTzzDwdp6R+eVt7///ntYWVkBAAoLCxEaGgo7OzsARYO3iYie5FpKNv5vV9G4ig9eaIluLexEjoiqW2xsLHr27Flqu42NDTIyMmo/ICKSvFtpuZj8cxQ0AvCajzNGPMsxW/pIp8KiadOm+O6777TPHR0dERYWVmofIqKyPDquopt7Q7z3fEuxQ6Ia4OjoiGvXrqFZs2Ylth87dkw72QcRUbG8AjXeCYtERq4KXs42+Pzldhysrad0Kizi4+NrKAwiqgvm/BLz37iK1ztyXIWBGj9+PD744AOsW7cOMpkMd+/exYkTJzB9+nTMnj1b7PCISEIEQcDHOy7gn8Qs2FmZYvVIb5iZcLC2vtKpsMjPz8fBgwcxZMgQAEXTzxYPygMAY2NjfP755zAz46qIRFTSjsjb2Hq2aL2Kb1/viEb1+D1hqGbOnAmNRoMXXngBubm56NmzJxQKBWbMmIFx48aJHR4RScgPx+Lwy/m7MJbLsOJNDtbWdzoN3g4NDcWaNWu0z5cvX47jx4/j3LlzOHfuHMLCwnRaw+LIkSN48cUX0aRJE8hkMuzevfupx0RERKBz585QKBRo0aIFQkNDdUmBiERwNfm/9So+eKEVx1UYOJlMhv/7v/9DWloaYmJicPLkSaSmpsLGxgZubm5ih0dEEnH82j3M3/cPAOCTwW3QtTnXMtJ3OhUWGzduxDvvvFNi26ZNm3D48GEcPnwYCxYswLZt2yp8vpycHHh5eWHFihUV2j8uLg6DBw9Gnz59EB0djSlTpmDcuHH4448/dEmDiGpRbkEh3t0YhTyVGs+1sMPk57lehaFSKpUIDg6Gj48Punfvjn379sHT0xOXLl2Ch4cHli5diqlTp4odJhFJwK20XARtKhqsPbSzMwK7NRM7JKoGOt0Kde3aNbRv31773MzMDHL5f7VJly5dEBQUVOHzDRw4EAMHDqzw/qtXr4abmxsWLlwIAGjTpg2OHTuGxYsXw8/Pr8LnIaLaIQgCPtkdg6spD2BfT4HFwzmuwpDNnj0ba9asQd++fXH8+HEEBARgzJgxOHnyJBYuXIiAgAAYGfHeaaK6Lq9AjQlhkUjPVaGDsw2+eIWDtQ2FToVFRkZGiTEVqampJV7XaDQlXq9uJ06cQN++fUts8/Pzw5QpU2rsmkRUedvO3sbOqDuQy4Blb3TiehUGbtu2bdiwYQNeeuklxMTEoEOHDigsLMT58+f5RwMRASj6wWnWrou4nJiFhpamWD2Cg7UNiU6FhbOzM2JiYuDh4VHm6xcuXICzs3O1BFaWpKQkODg4lNjm4OCArKws5OXlwdy89IAfpVJZotjJysoCAKhUKqhUKp2uX7y/rsdJjSHkwRyko7w8riRl49M9ReMqpr7QAt4u1pLN1dA/C12OrYrbt2/D29sbANCuXTsoFApMnTqVRQURaa37Ox67zt2BkVyG5W92RpP6HKxtSHQqLAYNGoTZs2dj8ODBpWZ+ysvLw9y5czF48OBqDbCqQkJCMHfu3FLbDxw4AAsLi0qdMzw8vKphSYIh5MEcpOPRPPLVwMILRlAWytCmvgbOD65g374rIkZXMYb4WVRUbm5ula+rVqthamqqfW5sbKxdUJWI6Pj1koO1fd05WNvQ6FRYzJo1C1u3boWHhwcmT56MVq1aAShaZXX58uUoLCzErFmzaiRQoGjRpeTk5BLbkpOTYW1tXWZvBVA0Je60adO0z7OysuDi4oL+/fvD2tpap+urVCqEh4ejX79+MDEx0T0BiTCEPJiDdDyehyAImLL1AlLyk+ForcCPk3zRwML06ScSkaF+Froo7s2tCkEQMHr0aCgURbe85efnY+LEibC0tCyx386dO6t8LSLSL3cy8jB50zmoNQJe7eSE0RysbZB0KiwcHBxw/PhxTJo0CTNnzoQgCACKphbs168fVq5cWepWperk6+uLffv2ldgWHh4OX1/fco9RKBTaRu5RJiYmlf4DoirHSokh5MEcpKM4j9C/47AvJhnGchlWjvBGIxvLpx8sEYb2Weh6TFUFBgaWeD5ixIgqne/IkSNYsGABIiMjkZiYiF27dsHf3/+Jx0RERGDatGm4dOkSXFxc8Mknn2D06NFVioOIqiZfpcaEsLNIyylAOydrzH+1PW+RNFA6FRYA4Obmhv379yMtLQ3Xrl0DALRo0QK2trY6X/zBgwfacwBF08lGR0fD1tYWTZs2RXBwMO7cuYMNGzYAACZOnIjly5fjo48+wtixY/Hnn39i69at+O2333S+NhFVv6iEdHzxsJt71qA26Ny0gcgRUW1av359tZ6veErysWPH4tVXX33q/sVTkk+cOBEbN27EoUOHMG7cODRu3JgzBxKJRBCA2b9cRsydLNhysLbB07mwKGZra4suXbpU6eJnz55Fnz59tM+Lb1kKDAxEaGgoEhMTkZCQoH3dzc0Nv/32G6ZOnYqlS5fC2dkZ33//PRsMIglIyynA5I1RUKkFDGrviDHdm4kdEuk5TklOpP+OJsmwKz7x4WDtTnBuULnxraQfKl1YVIfevXtrb6cqS1mravfu3Rvnzp2rwaiISFcaAZi+/SLuZubDzc4SXw3twG5uqnWVmZKcMweWxBykwxDyOH41Bbvii9Y7+9ivFZ5paqOX+RjCZ1FbswaKWlgQkWH447Ycx27fh5mJHKtGdEY9M/0fp0D6pzJTknPmwLIxB+nQ1zzSlcA3F4yggQzedho0Sr+EffsuiR1WlejrZ/Gomp41kIUFEVXJkav38Mftot6JkFfbo7WjbrOtEYmJMweWxBykQ5/zUKrUeOOHM3hQmAUnCwFrx/eGtYXZ0w+UKH3+LIrV1qyBLCyIqNJup+di+raLECDDm12c8Uqnmlsgk+hpKjMlOWcOLBtzkA59y0MQBMzafRkX72ShvrkJ3vbIg7WFmV7lUB59+yzKUtOzBsp1DYiICCiaPnDST1HIyFOhqaWAWQNbix0S1XG+vr44dOhQiW1Pm5KciKrXTydvYlvkbchlwJLhHdBQfzsqqBJYWBCRzgRBwOw9Mbh4JxMNLEwwxkMNhTG/Tqh6PXjwANHR0YiOjgbw35TkxbMFBgcHY9SoUdr9J06ciBs3buCjjz7ClStXsHLlSmzduhVTp04VI3yiOud0XBrm/noZADBzYGt058radQ7/EiAinW0+cwtbzz78Req1DrAtfScJUZWdPXsWnTp1QqdOnQAUTUneqVMnzJ49GwDKnZI8PDwcXl5eWLhwIackJ6oliZl5eHdjJAo1AoZ0aIzxPZqLHRKJgGMsiEgn5xLSMWdP0cweH/p5oJt7Q+yLFTkoMkickpxIP+Sr1Jj4UxTuPShAa8d6+HoYpxyvq9hjQUQVlpKdj0k/RaFArYFfWwdM6uUudkhERCQiQRAwZ88lnL+VARtzE6wd6QMLU/5uXVexsCCiCiko1CBoYxSSsvLhbm+JbwK8+IsUEVEdt/FUAracvQW5DFj2Ric0bciVtesyFhZEVCFf/HYZZ+LTYaUwxtpRPlwEj4iojjsbn4a5vxbdGvvRgNbo2cpe5IhIbCwsiOiptp69hR9P3AQALB7eEe72ViJHREREYkrKzMfEn6KgUgsY3L4xJvTkYG1iYUFETxGVkI5PdsUAAD54oSX6eTqIHBEREYlJWajGpI2RuPdACQ8HDtam/7CwIKJyJWflY2JYJArUGvT3dMAHL7QUOyQiIhLZZ79cwrmEDFibGWPNSG9YKjhYm4qwsCCiMuWr1HgnLBIp2Uq0crDCouEdIZfzFykiorps06kE/Hz6FmQy4Ns3OqGZnaXYIZGEsLAgolIEQUDwzova6QO/G+UDK/4iRURUp0XeTMecX4pujf2wvwd6ezQSOSKSGhYWRFTKqr+uY9e5OzCSy7Dyrc5wbchfpIiI6rLkrHxM+ikSKrWAge0c8W5vrmNEpbGwIKISDlxKwoI/ipbS/uxFT3RvYSdyREREJKaCQg0m/VR0a2zLRlZYwHWMqBwsLIhI6/LdLEzZEg1BAEY82xQjfZuJHRIREYls7q+XEJWQgXpmResY8dZYKg8LCyICUNTN/faPZ5BboEY394aY82JbsUMiIiKRbT6dgI2nEooGa7/eCW4crE1PIInCYsWKFWjWrBnMzMzQtWtXnD59utx9Q0NDIZPJSjzMzMxqMVoiw5NbUIhxP55FYmY+3O0tseotb5gYSeLrgYiIRBKVkI7Ze4pW1p7WtxX6tOZgbXoy0f9y2LJlC6ZNm4Y5c+YgKioKXl5e8PPzQ0pKSrnHWFtbIzExUfu4efNmLUZMZFg0GgFTt0Tj4p1M2FqaYt3oZ2BjYSJ2WEREJKKU7KLB2gVqDfzaOiCoTwuxQyI9IHphsWjRIowfPx5jxoyBp6cnVq9eDQsLC6xbt67cY2QyGRwdHbUPBweuBExUWV/s+wd/XEqGqZEca0d6cwYoIqI6rqBQg6CNUUjOUqJFIyssfI3rGFHFiDr6pqCgAJGRkQgODtZuk8vl6Nu3L06cOFHucQ8ePICrqys0Gg06d+6M+fPno23bsu8HVyqVUCqV2udZWVkAAJVKBZVKpVO8xfvrepzUGEIezKF6hJ64iR+OxQEAvny1Lbyc6tXJfxeGkANQtTz0PXciqj7z9l7Gmfh01FMYY+1Ibw7WpgoT9b+Ue/fuQa1Wl+pxcHBwwJUrV8o8xsPDA+vWrUOHDh2QmZmJb775Bt26dcOlS5fg7Oxcav+QkBDMnTu31PYDBw7AwsKiUnGHh4dX6jipMYQ8mEPlnb8vw/p/5QBkeKmpGka3z2Hf7XOVPh8/C+moTB65ubk1EAkR6ZutZ24h7GTRLeaLh3dEc3srkSMifaJ3Jaivry98fX21z7t164Y2bdpgzZo1mDdvXqn9g4ODMW3aNO3zrKwsuLi4oH///rC2ttbp2iqVCuHh4ejXrx9MTPT3HnRDyIM5VM3Zm+nYGBoJARq82cUZnw1pU+k5yflZSEdV8ijuzSWiuiv6VgY+2V20svbUvq3Q15O3mpNuRC0s7OzsYGRkhOTk5BLbk5OT4ejoWKFzmJiYoFOnTrh27VqZrysUCigUijKPq+wfEFU5VkoMIQ/moLvYpGxM+OkclIUa9G3TCJ+/3B7G1TADFD8L6ahMHoaQNxFVXmq2EhPDigZr9/N0wHvPc7A26U7Uwdumpqbw9vbGoUOHtNs0Gg0OHTpUolfiSdRqNS5evIjGjRvXVJhEBuN2ei5GrTuFrPxCeLs2wLI3OldLUUFERPpLpdYgaFMUkrLy0dzeEote8+JgbaoU0f+imDZtGr777jv8+OOP+OeffzBp0iTk5ORgzJgxAIBRo0aVGNz9+eef48CBA7hx4waioqIwYsQI3Lx5E+PGjRMrBSK9cP+BEqPWnUZylhItG1nhh0AfmJsaiR0W0RNxnSOimvfFb//gdFwarBTGWDvSB/XM2INJlSP6GIvhw4cjNTUVs2fPRlJSEjp27Ij9+/drB3QnJCRALv+v/klPT8f48eORlJSEBg0awNvbG8ePH4enp6dYKRBJXla+CqPWncaN1Bw0sTHDhre7oL6FqdhhET1R8TpHq1evRteuXbFkyRL4+fkhNjYWjRqVvVCXtbU1YmNjtc8rO3aIqK7YHnkbocfjARQN1m7RiIO1qfJELywAYPLkyZg8eXKZr0VERJR4vnjxYixevLgWoiIyDHkFarwdegaX7mahoaUpwsZ1RWMbc7HDInqqR9c5AoDVq1fjt99+w7p16zBz5swyjyle54iInu7i7UzM2nURAPDBCy3Rj4O1qYokUVgQUc1QFqox4afIovnIzYyx4e0ucOfUgaQHamOdI4BrHT2OOUhHTedxP6cA74SdRUGhBs972OPdns2q/Vr8LKSjttY5YmFBZKAKCjV496coHPk3FeYmRggd8wzaNrEROyyiCqmNdY4ArnVUHuYgHTWRh1oDrPxHjsQsORqZCehvnYj9+xOr/TrF+FlIR02vc8TCgsgAqdQaTN4UhUNXUqAwluOHQB94u9qKHRZRjdJ1nSOAax09jjlIR03m8cW+K7iWlQBLUyP8OL5rjY2r4GchHbW1zhELCyIDo1Jr8MHmczhwORmmxnJ8N8oH3VrYiR0WkU5qY50jgGsdlYc5SEd157Hr3G2EnkgAACx8rSPaODWotnOXh5+FdNT0OkeiTzdLRNWnoLCop2LfxSSYGsmxZqQ3erayFzssIp1xnSOi6hdzJxMzdxQN1p7cpwUGtONEB1S92GNBZCDyVWq8uzEKf15JgamxHKtHdEYfj7Kn5CTSB9OmTUNgYCB8fHzQpUsXLFmypNQ6R05OTggJCQFQtM7Rs88+ixYtWiAjIwMLFizgOkdED6XlFGBCWCSUhRr08bDH1H6txA6JDBALCyIDkFtQiAlhkTh69R7MTORYO9KHPRWk97jOEVH1KHw47u5ORh6aNbTAktc7wYgra1MNYGFBpOcycgswNvQMohIyYGFqhB8Cn4Gve0OxwyKqFlzniKjqvvz9Co5fvw8LUyOsHeUDG3P9HidA0sXCgkiPJWflY9QPpxGbnA1rM2OsH/MMZ38iIiKtPdF38P2xOADANwFeaOVQT+SIyJCxsCDSU9dTH2D0+tO4lZaHRvUUCHu7Kzwc2WAQEVGRS3cz8fGOCwCAd3u7Y1B7TmRANYuFBZEeOhOfhvEbziIjVwXXhhb46e2ucLGt3GJeRERkeNIfDtbOV2nQq5U9pvf3EDskqgNYWBDpmb0X7mLa1vMoKNSgo0t9fB/oAzur0vPwExFR3VSo1uC9n8/hdnoemtpa4FsO1qZawsKCSE9oNAKWHrqKpYeuAgD82jpgyfBOMDc1EjkyIiKSkgV/xOLYtXswNzHC2lHesLHgYG2qHSwsiPRAjrIQ07eex/5LSQCAsd3d8H+D2/AXKCIiKuGX83ex5sgNAMCCgA5o7WgtckRUl7CwIJK4+Hs5mPhTJK4kZcPESIYv/NvjtWdcxA6LiIgk5p/ELHy0/TwAYGIvdwzp0ETkiKiuYWFBJGH7YxIxY9sFZCsLYWelwJqRnTmdLBERlZKRW4B3ws4iX6VBj5Z2mOHHwdpU+1hYEEmQslCNr/fH4oeHc48/06wBlr3RGY42ZiJHRkREUqPWCHjv53O4lZYHF1tzDtYm0bCwIJKYaynZeP/naFxOzAIAvNOzOWb4ecDESC5yZEREJEUL/ojF0asPB2uP9EEDS1OxQ6I6ShJ/qaxYsQLNmjWDmZkZunbtitOnTz9x/23btqF169YwMzND+/btsW/fvlqKlKjmaDQCNpyIx+Bvj+FyYhYaWJhg7UhvzBrUhkUFERGV6bcLiVj913UAwFfDOqBNYw7WJvGI/tfKli1bMG3aNMyZMwdRUVHw8vKCn58fUlJSytz/+PHjeOONN/D222/j3Llz8Pf3h7+/P2JiYmo5cqLqE38vB298dxKz91yCsrDo/tg/pvRE/7aOYodGREQSdSUpCx9uKxqs/U7P5njJi4O1SVyiFxaLFi3C+PHjMWbMGHh6emL16tWwsLDAunXrytx/6dKlGDBgAGbMmIE2bdpg3rx56Ny5M5YvX17LkRNVnVoDfH8sHgOWHsGpuDSYmxhhzoue+HFMFzSy5ngKIiIqW2auChPCIpGnUuO5Fnb4iIO1SQJEHWNRUFCAyMhIBAcHa7fJ5XL07dsXJ06cKPOYEydOYNq0aSW2+fn5Yffu3WXur1QqoVQqtc+zsoruW1epVFCpVDrFuyPyFi6myJAfdQsKExMYyWUwlstgbCSDkVwGUyM5jOUymBjJHz5kMDGWw9RIDlNjORQPH8ZyGWQy8QZVFeeta/5SYgg5HP03BV9fMEJS3r8AgG7NbTHvZU80tbWAWl0ItVrkACvIED4LQ8gBqFoe+p47UV2i1gh4f/M53LyfC+cG5lj2RicY85ZZkgBRC4t79+5BrVbDwcGhxHYHBwdcuXKlzGOSkpLK3D8pKanM/UNCQjB37txS2w8cOAALCwud4p172gh5aiNsvP6PTsc9TgYBJnJoH6ZywNTo4f/KBSiMUPSQAwpjwMxIgJkRYGYEmBsB5sYCzI0AC2PA3LjouMrUKeHh4VXKQwr0MYfUPGDvLTmi78sByGBpLOAlVw262qcg5mQK9PWmPn38LB5nCDkAlcsjNze3BiIhopqwKDwWf/2bCjMTOdaM9OZgbZIMg58VKjg4uEQPR1ZWFlxcXNC/f39YW+s2wGlf5jkk3E1G/QYNIQAo1Ago1AhQawSo1AIK1Rqo1AJUag0KNUX/W1CoQcHD7cUEyFCgAQo0ZV1F9wrB1FiO+uYmqG9uggaWJrC1MIWtpSkaWprC1soUdpamsK+ngJ2VKRrVU8AIGoSHh6Nfv34wMTHR+XpSoFKp9C6Hew+UWH74BrZcuI1CjQC5DOjuoMHXI3vCzlq3IldK9PGzeJwh5ABULY/i3lwikrbfLyZixeGHg7WHdkDbJjYiR0T0H1ELCzs7OxgZGSE5ObnE9uTkZDg6lj1o1dHRUaf9FQoFFApFqe0mJiY6N7zL3+iEffv2YdCgZ3Q+VqMRUKDWQKnSQFmoRr5Kg/xCNfJVauQVqJGrUiO/QI2cAjXyCgqRU6BGjrIQD5SFyFEWIju/+KFCdn4hMvNUyMxToVAjoKBQg5RsJVKylU8PBIC1mTEsZEbYlnoBTeqbw9HGHE1szNCkvjma1DeHU31zmJsa6ZSfWCrzOda2xMw8fHckDj+fTkCequj+pl6t7DG9bwvEnTsKO2sLyedQEfrwWTyNIeQAVC4PQ8ibyND9m5yN6Q8Ha497zg0vd3QSOSKikkQtLExNTeHt7Y1Dhw7B398fAKDRaHDo0CFMnjy5zGN8fX1x6NAhTJkyRbstPDwcvr6+tRBx5cnlMpjJjWBmYgSgehpwQRCQW6BGem4BMnJVSM8tQFrOf497Dwpw74ES9x8okfpAiZQsJZSFGmTlFyILMiRdu1/uue2sTOHUwALODczR1NYCLg0s0NTWAq4NLdCkvjkX3qmAfxKzEPp3PHaeu63tseroUh8fD2gNX/eGUKlUiDsncpBERKQXMvNUeGfDWeQWqNHNvSFmDmwtdkhEpYh+K9S0adMQGBgIHx8fdOnSBUuWLEFOTg7GjBkDABg1ahScnJwQEhICAPjggw/Qq1cvLFy4EIMHD8bmzZtx9uxZrF27Vsw0RCGTyWCpMIalwhjODZ6+vyAIyFYW4s79B/j14FG4tumA1Acq3M3MR1JmPu5m5OFOeh6ylYUPi5ICnL+VUeo8JkYyuDQoKjKa2VnC7ZFHExtzyOtw0ZGvUiP8cjLCTt7E6bg07faubraY/HwLPNfCTtSB+0REpH/UGgFTNp9D/P1cONU3x/I3O3OwNkmS6IXF8OHDkZqaitmzZyMpKQkdO3bE/v37tQO0ExISIJf/94+nW7du2LRpEz755BPMmjULLVu2xO7du9GuXTuxUtAbMpkM1mYmMG9kBY/6AgZ1cirz9ofMPBVup+fiVlrew//Nxc20XCSk5eJ2Wh4K1BrcuJeDG/dygNjUEseaGsvh1tASze0fPuys4N7ICs3tLWFtZpi3Wqg1AqIS0rHr3B3sPX8XWfmFAAAjuQwD2jpi7HPN4O1qK3KURESkr5Yc/BeHY1OhMC4arG3LwdokUaIXFgAwefLkcm99ioiIKLUtICAAAQEBNRxV3WVjbgIbc5syB4SpNQKSsvJx814O4u7nIP5eDuLu5SLu3gMkpOWioFCD2ORsxCZnlzrWzkqB5vaWcH9YcLjZFRUfLrYWereydI6yEKfi7iP8cjLCL6fg3oP/xrc0tjHDMG9nvNXVFY42XIuCqCpWrFiBBQsWICkpCV5eXli2bBm6dOlS7v7btm3Dp59+ivj4eLRs2RJfffUVBg0aVIsRE1WvA5eTsezPawCAL4e2RzsnDtYm6ZJEYUH6w0gug9PDAd7dWtiVeK1QrcGdjDzcSM3B9dQHRb0aqQ9wIzUHKdlK3HtQ9Hj0FqHic7o0MEczO0s0a2gJ14ZFt1k1tbWEcwPzh+NSxJWWU4DoW+k4l5CBkzfu41xCBgo1/830Vc/MGP08HTCsszOebd6wTt8ORlRdtmzZgmnTpmH16tXo2rUrlixZAj8/P8TGxqJRo0al9j9+/DjeeOMNhISEYMiQIdi0aRP8/f0RFRXFXm3SS3dygBU7iiYhH9vdDa90chY5IqInY2FB1cbYSA7XhpZwbWiJPq1LNvrZ+SrE3cvBjdSHxcbD/x93Lwd5KjXi7+ci/n4ugNRS521UTwGnBkXFTJP65nC0NoOdpTFuZAE37+fCsYElLE2Nqjx2QaXWICkzH7fSc3E7PQ/XUx/gavID/JucjdvpeaX2b2prgZ6t7ODX1hFd3RrC1Fi/el2IpG7RokUYP368dszd6tWr8dtvv2HdunWYOXNmqf2XLl2KAQMGYMaMGQCAefPmITw8HMuXL8fq1atrNXaiqlAWqrHiz+tYcdEIakGNZ5vbYtYgDtYm6WNhQbWinpkJOjjXRwfn+iW2C4KA5Cwlbtx7gJv3cxH/8PaqhLQ8JNzPQU6BWjuV7rmEjMfOaoyll44BKBrbYfNwLY96ZsawMDWGhakRFCZGMJbLtLNYaTQC1IKAfJUauQ+n9M3IU+H+gwJk5j155WF3e0t0dGkAn2YN0N3dDk0b6u/aE0RSV1BQgMjISAQHB2u3yeVy9O3bFydOnCjzmBMnTpRYtwgA/Pz8sHv37nKvo1QqoVT+dytj8XoeKpVKp9XIj127j70X7uLOHTmO7LxYYmygPtFoNMxBAiJvpuPGvVwAMjznbouFAR0gaNRQadRih6aT4n9DuvxbkiJDyKMqOehyDAsLEpVMJoOjjRkcbczQzb3ka4IgIC2nAHcezlZ1JyMPdzPykZyVj8TMPNxMTkeuxgh5qqKFCFOzlUit4Foe5TE1lsO5vjmcGpijWUNLtHKwQkuHemjjaA0bC8McfE4kRffu3YNardZO5FHMwcEBV65cKfOYpKSkMvdPSkoq9zohISGYO3duqe0HDhyAhUXFfzyISJRhV7wRADmQkljh46SJOUhBPRMBrzbToFPDFJz866DY4VRJeHi42CFUC0PIozI55ObmVnhfFhYkWTKZDA2tFGhopSjV06FSqR4uVuiHAo0M6blFPQ6ZuSpkKwuRV6BGTkEhCgo12pXRAcBIDshlMpiZGMFSYQQLU2NYm5nAvp4pGloqYGNuwvERRHVIcHBwiV6OrKwsuLi4oH///rC2tq7weZxvZ8L1aiquXbuKFi1awkhPfylXazTMQQIsFcYY6GmH08ci0K9fP71dwFKlUiE8PFyvcwAMI4+q5FDck1sRLCxI7+mylgcR6Qc7OzsYGRkhOTm5xPbk5GQ4OjqWeYyjo6NO+wOAQqGAQqEotV3X1cu93ezQwdkG+/L+xaA+LfT6jw/mIA3Ft5/o+t+iFBlCDoBh5FGZHHTZXz9LeSIiMmimpqbw9vbGoUOHtNs0Gg0OHToEX1/fMo/x9fUtsT9Q1O1f3v5ERFS92GNBRESSNG3aNAQGBsLHxwddunTBkiVLkJOTo50latSoUXByckJISAgA4IMPPkCvXr2wcOFCDB48GJs3b8bZs2exdu1aMdMgIqozWFgQEZEkDR8+HKmpqZg9ezaSkpLQsWNH7N+/XztAOyEhocSsP926dcOmTZvwySefYNasWWjZsiV2797NNSyIiGoJCwsiIpKsyZMnY/LkyWW+FhERUWpbQEAAAgICajgqIiIqC8dYEBERERFRlbGwICIiIiKiKqtzt0IJQtF6BrrMyVtMpVIhNzcXWVlZej3dmCHkwRykwxDyMIQcgKrlUfydWPwdWVfV9TaCOUiHIeRhCDkAhpFHbbUPda6wyM7OBgC4uLiIHAkRkfRkZ2fDxsZG7DBEwzaCiKhsFWkfZEId+3lKo9Hg7t27qFevHmQy3VZYLl6R9datWzqtyCo1hpAHc5AOQ8jDEHIAqpaHIAjIzs5GkyZNSsy0VNfU9TaCOUiHIeRhCDkAhpFHbbUPda7HQi6Xw9nZuUrnsLa21tv/sB5lCHkwB+kwhDwMIQeg8nnU5Z6KYmwjijAH6TCEPAwhB8Aw8qjp9qHu/ixFRERERETVhoUFERERERFVGQsLHSgUCsyZMwcKhULsUKrEEPJgDtJhCHkYQg6A4eShrwzh/WcO0mEIeRhCDoBh5FFbOdS5wdtERERERFT92GNBRERERERVxsKCiIiIiIiqjIUFERERERFVGQuLSnrppZfQtGlTmJmZoXHjxhg5ciTu3r0rdlg6iY+Px9tvvw03NzeYm5vD3d0dc+bMQUFBgdih6eSLL75At27dYGFhgfr164sdToWtWLECzZo1g5mZGbp27YrTp0+LHZJOjhw5ghdffBFNmjSBTCbD7t27xQ5JZyEhIXjmmWdQr149NGrUCP7+/oiNjRU7LJ2sWrUKHTp00M5N7uvri99//13ssOo8fW8jDKV9APSzjWD7ID5DaB+A2m8jWFhUUp8+fbB161bExsZix44duH79OoYNGyZ2WDq5cuUKNBoN1qxZg0uXLmHx4sVYvXo1Zs2aJXZoOikoKEBAQAAmTZokdigVtmXLFkybNg1z5sxBVFQUvLy84Ofnh5SUFLFDq7CcnBx4eXlhxYoVYodSaX/99ReCgoJw8uRJhIeHQ6VSoX///sjJyRE7tApzdnbGl19+icjISJw9exbPP/88Xn75ZVy6dEns0Oo0fW8jDKV9APSvjWD7IA2G0D4AIrQRAlWLPXv2CDKZTCgoKBA7lCr5+uuvBTc3N7HDqJT169cLNjY2YodRIV26dBGCgoK0z9VqtdCkSRMhJCRExKgqD4Cwa9cuscOospSUFAGA8Ndff4kdSpU0aNBA+P7778UOgx5hCG2EPrcPgqA/bQTbB2kylPZBEGq2jWCPRTVIS0vDxo0b0a1bN5iYmIgdTpVkZmbC1tZW7DAMWkFBASIjI9G3b1/tNrlcjr59++LEiRMiRkaZmZkAoLf/BtRqNTZv3oycnBz4+vqKHQ49ZChtBNuHmsf2Qbr0vX0AaqeNYGFRBR9//DEsLS3RsGFDJCQkYM+ePWKHVCXXrl3DsmXLMGHCBLFDMWj37t2DWq2Gg4NDie0ODg5ISkoSKSrSaDSYMmUKunfvjnbt2okdjk4uXrwIKysrKBQKTJw4Ebt27YKnp6fYYdV5htRGsH2oHWwfpEmf2wegdtsIFhaPmDlzJmQy2RMfV65c0e4/Y8YMnDt3DgcOHICRkRFGjRoFQQLrDeqaBwDcuXMHAwYMQEBAAMaPHy9S5P+pTA5EVREUFISYmBhs3rxZ7FB05uHhgejoaJw6dQqTJk1CYGAgLl++LHZYBscQ2ghDaB8AthFUu/S5fQBqt43gytuPSE1Nxf3795+4T/PmzWFqalpq++3bt+Hi4oLjx4+LfguCrnncvXsXvXv3xrPPPovQ0FDI5eLXm5X5LEJDQzFlyhRkZGTUcHRVU1BQAAsLC2zfvh3+/v7a7YGBgcjIyNDLXzVlMhl27dpVIh99MnnyZOzZswdHjhyBm5ub2OFUWd++feHu7o41a9aIHYpBMYQ2whDaB8Bw2wi2D9JjaO0DULNthHG1n1GP2dvbw97evlLHajQaAIBSqazOkCpFlzzu3LmDPn36wNvbG+vXr5dMo1GVz0LqTE1N4e3tjUOHDmm/aDUaDQ4dOoTJkyeLG1wdIwgC3nvvPezatQsREREG02hoNBpJfBcZGkNoIwyhfQAMt41g+yAdhto+ADXbRrCwqIRTp07hzJkzeO6559CgQQNcv34dn376Kdzd3UXvrdDFnTt30Lt3b7i6uuKbb75Bamqq9jVHR0cRI9NNQkIC0tLSkJCQALVajejoaABAixYtYGVlJW5w5Zg2bRoCAwPh4+ODLl26YMmSJcjJycGYMWPEDq3CHjx4gGvXrmmfx8XFITo6Gra2tmjatKmIkVVcUFAQNm3ahD179qBevXrae5htbGxgbm4ucnQVExwcjIEDB6Jp06bIzs7Gpk2bEBERgT/++EPs0OosQ2gjDKV9APSvjWD7IA2G0D4AIrQRNTLXlIG7cOGC0KdPH8HW1lZQKBRCs2bNhIkTJwq3b98WOzSdrF+/XgBQ5kOfBAYGlpnD4cOHxQ7tiZYtWyY0bdpUMDU1Fbp06SKcPHlS7JB0cvjw4TLf98DAQLFDq7Dy/vtfv3692KFV2NixYwVXV1fB1NRUsLe3F1544QXhwIEDYodVpxlCG2Eo7YMg6GcbwfZBfIbQPghC7bcRHGNBRERERERVJp0bJomIiIiISG+xsCAiIiIioipjYUFERERERFXGwoKIiIiIiKqMhQUREREREVUZCwsiIiIiIqoyFhZERERERFRlLCyIiIiIiKjKWFgQEREREVGVsbAgIiIiIqIqY2FBRERERERVxsKCqJalpqbC0dER8+fP1247fvw4TE1NcejQIREjIyIiMbF9IH0nEwRBEDsIorpm37598Pf3x/Hjx+Hh4YGOHTvi5ZdfxqJFi8QOjYiIRMT2gfQZCwsikQQFBeHgwYPw8fHBxYsXcebMGSgUCrHDIiIikbF9IH3FwoJIJHl5eWjXrh1u3bqFyMhItG/fXuyQiIhIAtg+kL7iGAsikVy/fh13796FRqNBfHy82OEQEZFEsH0gfcUeCyIRFBQUoEuXLujYsSM8PDywZMkSXLx4EY0aNRI7NCIiEhHbB9JnLCyIRDBjxgxs374d58+fh5WVFXr16gUbGxvs3btX7NCIiEhEbB9In/FWKKJaFhERgSVLliAsLAzW1taQy+UICwvD0aNHsWrVKrHDIyIikbB9IH3HHgsiIiIiIqoy9lgQEREREVGVsbAgIiIiIqIqY2FBRERERERVxsKCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRlbGwICIiIiKiKmNhQUREREREVcbCgoiIiIiIquz/AaMPFqDL6bfHAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "gelu, relu = GELU(), nn.ReLU()\n",
    "\n",
    "# Some sample data\n",
    "x = torch.linspace(-3, 3, 100)\n",
    "y_gelu, y_relu = gelu(x), relu(x)\n",
    "\n",
    "plt.figure(figsize=(8, 3))\n",
    "for i, (y, label) in enumerate(zip([y_gelu, y_relu], [\"GELU\", \"ReLU\"]), 1):\n",
    "    plt.subplot(1, 2, i)\n",
    "    plt.plot(x, y)\n",
    "    plt.title(f\"{label} activation function\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(f\"{label}(x)\")\n",
    "    plt.grid(True)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cd01662-14cb-43fd-bffd-2d702813de2d",
   "metadata": {},
   "source": [
    "- 显然，ReLU是一个分段线性函数，如果输入是正值，它直接原样输出；否则，输出为零。\n",
    "- GELU是一个平滑的非线性函数，近似于ReLU，但输入为负值时，梯度不为0。\n",
    "- 接下来，让我们实现小型神经网络模块 FeedForward，稍后我们将在LLM的Transformer block中使用它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "9275c879-b148-4579-a107-86827ca14d4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.layers = nn.Sequential(\n",
    "            nn.Linear(cfg[\"emb_dim\"], 4 * cfg[\"emb_dim\"]),\n",
    "            GELU(),\n",
    "            nn.Linear(4 * cfg[\"emb_dim\"], cfg[\"emb_dim\"]),\n",
    "            nn.Dropout(cfg[\"drop_rate\"])\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.layers(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "7c4976e2-0261-418e-b042-c5be98c2ccaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "768\n"
     ]
    }
   ],
   "source": [
    "print(GPT_CONFIG_124M[\"emb_dim\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdcaacfa-3cfc-4c9e-b668-b71a2753145a",
   "metadata": {},
   "source": [
    "<img src=\"figures/ffn.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "928e7f7c-d0b1-499f-8d07-4cadb428a6f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 3, 768])\n"
     ]
    }
   ],
   "source": [
    "ffn = FeedForward(GPT_CONFIG_124M)\n",
    "\n",
    "# input shape: [batch_size, num_token, emb_size]\n",
    "x = torch.rand(2, 3, 768) \n",
    "out = ffn(x)\n",
    "print(out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8756c5-6b04-443b-93d0-e555a316c377",
   "metadata": {},
   "source": [
    "<img src=\"figures/mental-model-3.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ffcb905-53c7-4886-87d2-4464c5fecf89",
   "metadata": {},
   "source": [
    "## 4.4 添加Shortcut连接"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5161bf8c",
   "metadata": {},
   "source": [
    "- 接下来，我们将探讨shortcut连接，这也被称为跳跃连接或残差连接\n",
    "- 最初，shortcut连接在计算机视觉的深度神经网络(残差网络)中被提出，以缓解消失梯度问题\n",
    "- Shortcut连接为网络中传播的梯度提供了一条更短的路径\n",
    "- 这是通过将一个层的输出加到后面层的输出上来实现，通常跳过中间的一个或多个层\n",
    "- 让我们通过一个小的示例网络来说明这个思想：\n",
    "\n",
    "<img src=\"figures/shortcut-example.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14cfd241-a32e-4601-8790-784b82f2f23e",
   "metadata": {},
   "source": [
    "- 示例代码如下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05473938-799c-49fd-86d4-8ed65f94fee6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ExampleDeepNeuralNetwork(nn.Module):\n",
    "    def __init__(self, layer_sizes, use_shortcut):\n",
    "        super().__init__()\n",
    "        self.use_shortcut = use_shortcut\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Sequential(nn.Linear(layer_sizes[0], layer_sizes[1]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[1], layer_sizes[2]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[2], layer_sizes[3]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[3], layer_sizes[4]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[4], layer_sizes[5]), GELU())\n",
    "        ])\n",
    "\n",
    "    def forward(self, x):\n",
    "        for layer in self.layers:\n",
    "            # 计算当前层的输出\n",
    "            layer_output = layer(x)\n",
    "            # 检查是否可以使用shortcut\n",
    "            if self.use_shortcut and x.size() == layer_output.size():\n",
    "                x = x + layer_output\n",
    "            else:\n",
    "                x = layer_output\n",
    "        return x\n",
    "\n",
    "\n",
    "def print_gradients(model, x):\n",
    "    # 前向传播\n",
    "    output = model(x)\n",
    "    target = torch.tensor([[0.]])\n",
    "\n",
    "    # 根据输出和标签差距来计算损失\n",
    "    loss = nn.MSELoss()\n",
    "    loss = loss(output, target)\n",
    "    \n",
    "    # 反向传播计算梯度\n",
    "    loss.backward()\n",
    "\n",
    "    for name, param in model.named_parameters():\n",
    "        if 'weight' in name:\n",
    "            # 打印权重的平均绝对梯度\n",
    "            print(f\"{name} has gradient mean of {param.grad.abs().mean().item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b39bf277-b3db-4bb1-84ce-7a20caff1011",
   "metadata": {},
   "source": [
    "- 让我们先打印**不使用**shortcut连接的梯度值:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c75f43cc-6923-4018-b980-26023086572c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layers.0.0.weight has gradient mean of 0.00020173587836325169\n",
      "layers.1.0.weight has gradient mean of 0.0001201116101583466\n",
      "layers.2.0.weight has gradient mean of 0.0007152041653171182\n",
      "layers.3.0.weight has gradient mean of 0.001398873864673078\n",
      "layers.4.0.weight has gradient mean of 0.005049646366387606\n"
     ]
    }
   ],
   "source": [
    "layer_sizes = [3, 3, 3, 3, 3, 1]  \n",
    "\n",
    "sample_input = torch.tensor([[1., 0., -1.]])\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model_without_shortcut = ExampleDeepNeuralNetwork(\n",
    "    layer_sizes, use_shortcut=False\n",
    ")\n",
    "print_gradients(model_without_shortcut, sample_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "837fd5d4-7345-4663-97f5-38f19dfde621",
   "metadata": {},
   "source": [
    "- 接下来我们打印**使用**shortcut连接的梯度值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11b7c0c2-f9dd-4dd5-b096-a05c48c5f6d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layers.0.0.weight has gradient mean of 0.22169792652130127\n",
      "layers.1.0.weight has gradient mean of 0.20694105327129364\n",
      "layers.2.0.weight has gradient mean of 0.32896995544433594\n",
      "layers.3.0.weight has gradient mean of 0.2665732502937317\n",
      "layers.4.0.weight has gradient mean of 1.3258541822433472\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model_with_shortcut = ExampleDeepNeuralNetwork(\n",
    "    layer_sizes, use_shortcut=True\n",
    ")\n",
    "print_gradients(model_with_shortcut, sample_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b385c50b",
   "metadata": {},
   "source": [
    "- 从上述输出可以看出，shortcut连接可以防止梯度在浅层（靠近layer.0）中消失。\n",
    "- 接下来，我们将在实现Transformer块时应用shortcut连接。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd8a2072",
   "metadata": {},
   "source": [
    "## 4.5 在transformer块中连接注意力层和线性层"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc571b76",
   "metadata": {},
   "source": [
    "- 本节将前述概念融合，搭建transformer块。\n",
    "- Transformer块将前一章的因果多头注意力模块与线性层结合起来，即之前章节中我们实现的前馈神经网络\n",
    "- 此外，transformer块还使用了Dropout和shortcut连接。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "0e1e8176-e5e3-4152-b1aa-0bbd7891dfd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import MultiHeadAttention\n",
    "\n",
    "\n",
    "class TransformerBlock(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.att = MultiHeadAttention(\n",
    "            d_in=cfg[\"emb_dim\"],\n",
    "            d_out=cfg[\"emb_dim\"],\n",
    "            block_size=cfg[\"ctx_len\"],\n",
    "            num_heads=cfg[\"n_heads\"], \n",
    "            dropout=cfg[\"drop_rate\"],\n",
    "            qkv_bias=cfg[\"qkv_bias\"])\n",
    "        self.ff = FeedForward(cfg)\n",
    "        self.norm1 = LayerNorm(cfg[\"emb_dim\"])\n",
    "        self.norm2 = LayerNorm(cfg[\"emb_dim\"])\n",
    "        self.drop_resid = nn.Dropout(cfg[\"drop_rate\"])\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 注意力块中的Shortcut连接\n",
    "        shortcut = x\n",
    "        x = self.norm1(x)\n",
    "        x = self.att(x)  # Shape [batch_size, num_tokens, emb_size]\n",
    "        x = self.drop_resid(x)\n",
    "        x = x + shortcut  # 与原始输入块求和\n",
    "\n",
    "        # 前馈块中的Shortcut连接\n",
    "        shortcut = x\n",
    "        x = self.norm2(x)\n",
    "        x = self.ff(x)\n",
    "        x = self.drop_resid(x)\n",
    "        x = x + shortcut  # 与原始输入块求和\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36b64d16-94a6-4d13-8c85-9494c50478a9",
   "metadata": {},
   "source": [
    "<img src=\"figures/transformer-block.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31d3dd26",
   "metadata": {},
   "source": [
    "- 假设我们有2个输入样本，每个样本包含6个token，且每个token都是一个768维的embedding向量。此时，Transformer块会对输入进行自注意力计算，接着进行线性变换，得到一个与输入形状相同的输出。\n",
    "- 我们可以将这个输出视为前一章中所讨论的上下文向量的增强版本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "3fb45a63-b1f3-4b08-b525-dafbc8228405",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input shape: torch.Size([2, 4, 768])\n",
      "Output shape: torch.Size([2, 4, 768])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "x = torch.rand(2, 4, 768)  # Shape: [batch_size, num_tokens, emb_dim]\n",
    "block = TransformerBlock(GPT_CONFIG_124M)\n",
    "output = block(x)\n",
    "\n",
    "print(\"Input shape:\", x.shape)\n",
    "print(\"Output shape:\", output.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "01e737a6-fc99-42bb-9f7e-4da899168811",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input shape: torch.Size([2, 4, 768])\n",
      "Output shape: torch.Size([2, 4, 768])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "x = torch.rand(2, 4, 768)  # Shape: [batch_size, num_tokens, emb_dim]\n",
    "block = TransformerBlock(GPT_CONFIG_124M)\n",
    "output = block(x)\n",
    "\n",
    "print(\"Input shape:\", x.shape)\n",
    "print(\"Output shape:\", output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91f502e4-f3e4-40cb-8268-179eec002394",
   "metadata": {},
   "source": [
    "<img src=\"figures/mental-model-final.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46618527-15ac-4c32-ad85-6cfea83e006e",
   "metadata": {},
   "source": [
    "## 4.6 编写GPT模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8a75745",
   "metadata": {},
   "source": [
    "- 我们已经接近成功了：现在让我们将transformer块集成到我们在本章开头编写的架构中，以便获得功能强大的GPT架构\n",
    "- 请注意，transformer块被重复多次使用；在最小的124M GPT-2模型中，我们重复了12次："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7b362d-f8c5-48d2-8ebd-722480ac5073",
   "metadata": {},
   "source": [
    "<img src=\"figures/gpt.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "324e4b5d-ed89-4fdf-9a52-67deee0593bc",
   "metadata": {},
   "source": [
    "- 对应的代码实现，其中 `cfg[\"n_layers\"] = 12`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "c61de39c-d03c-4a32-8b57-f49ac3834857",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPTModel(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
    "        self.pos_emb = nn.Embedding(cfg[\"ctx_len\"], cfg[\"emb_dim\"])\n",
    "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
    "        \n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
    "        \n",
    "        self.final_norm = LayerNorm(cfg[\"emb_dim\"])\n",
    "        self.out_head = nn.Linear(\n",
    "            cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n",
    "        )\n",
    "\n",
    "    def forward(self, in_idx):\n",
    "        batch_size, seq_len = in_idx.shape\n",
    "        tok_embeds = self.tok_emb(in_idx)\n",
    "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
    "        x = tok_embeds + pos_embeds  # Shape [batch_size, num_tokens, emb_size]\n",
    "        x = self.trf_blocks(x)\n",
    "        x = self.final_norm(x)\n",
    "        logits = self.out_head(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86571328",
   "metadata": {},
   "source": [
    "- 我们现在可以按照如下方式，采用124M参数模型的配置，以随机初始化权重的方式实例化这个GPT模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "252b78c2-4404-483b-84fe-a412e55c16fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input batch:\n",
      " tensor([[6109, 3626, 6100,  345],\n",
      "        [6109, 1110, 6622,  257]])\n",
      "\n",
      "Output shape: torch.Size([2, 4, 50257])\n",
      "tensor([[[-0.0055,  0.3224,  0.2185,  ...,  0.2539,  0.4578, -0.4747],\n",
      "         [ 0.2663, -0.2975, -0.5040,  ..., -0.3903,  0.5328, -0.4224],\n",
      "         [ 1.1146, -0.0923,  0.1303,  ...,  0.1521, -0.4494,  0.0276],\n",
      "         [-0.8239,  0.1174, -0.2566,  ...,  1.1197,  0.1036, -0.3993]],\n",
      "\n",
      "        [[-0.1027,  0.1752, -0.1048,  ...,  0.2258,  0.1559, -0.8747],\n",
      "         [ 0.2230,  0.1246,  0.0492,  ...,  0.8573, -0.2933,  0.3036],\n",
      "         [ 0.9409,  1.3068, -0.1610,  ...,  0.8244,  0.1763,  0.0811],\n",
      "         [ 0.4395,  0.2753,  0.1540,  ...,  1.3410, -0.3709,  0.1643]]],\n",
      "       grad_fn=<UnsafeViewBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "out = model(batch)\n",
    "print(\"Input batch:\\n\", batch)\n",
    "print(\"\\nOutput shape:\", out.shape)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af09a24f",
   "metadata": {},
   "source": [
    "- 我们将在下一章对这个模型进行训练。\n",
    "- 这里对模型大小做一个快速说明：我们之前提到它是一个拥有124M参数的模型；可以按照以下方式核对这个数字："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "84fb8be4-9d3b-402b-b3da-86b663aac33a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parameters: 163,009,536\n"
     ]
    }
   ],
   "source": [
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "print(f\"Total number of parameters: {total_params:,}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1160952b",
   "metadata": {},
   "source": [
    "- 正如我们看到的，这个模型的参数量为163M个，而不是124M个；为什么呢？\n",
    "- 在原始的GPT-2论文中，研究人员使用了权重绑定，这意味着他们将token嵌入层（tok_emb）重复用作输出层，即设置`self.out_head.weight = self.tok_emb.weight`\n",
    "- token嵌入层将50,257维输入token的one-hot编码投影到768维的embedding表示中\n",
    "- 输出层将768维的embedding投影回到50,257维的表示中，以便我们可以将其转换回单词（更多关于此的信息请参见下一节）\n",
    "- 因此，embedding层和输出层有相同数量的权重参数，正如我们根据其权重矩阵的形状所看到的那样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e3b43233-e9b8-4f5a-b72b-a263ec686982",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token embedding layer shape: torch.Size([50257, 768])\n",
      "Output layer shape: torch.Size([50257, 768])\n"
     ]
    }
   ],
   "source": [
    "print(\"Token embedding layer shape:\", model.tok_emb.weight.shape)\n",
    "print(\"Output layer shape:\", model.out_head.weight.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "029a0dc9",
   "metadata": {},
   "source": [
    "- 在原始的GPT-2论文中，研究人员将标记嵌入矩阵重复用作输出矩阵\n",
    "- 因此，如果我们减去输出层的参数数量，就会得到一个124M参数的模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "95a22e02-50d3-48b3-a4e0-d9863343c164",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of trainable parameters considering weight tying: 124,412,160\n"
     ]
    }
   ],
   "source": [
    "total_params_gpt2 =  total_params - sum(p.numel() for p in model.out_head.parameters())\n",
    "print(f\"Number of trainable parameters considering weight tying: {total_params_gpt2:,}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db1e245d",
   "metadata": {},
   "source": [
    "- 在实践中，我发现在没有权重绑定时训练模型更容易，这就是为什么在这里我们没有实现它的原因。\n",
    "- 然而，在第六章加载预训练权重时，我们将重新审视并应用这个权重绑定的想法。\n",
    "- 最后，我们可以按以下方式计算模型的内存需求，这可以作为一个有用的参考点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "5131a752-fab8-4d70-a600-e29870b33528",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total size of the model: 621.83 MB\n"
     ]
    }
   ],
   "source": [
    "# 计算总字节大小（假设每个参数均为占用4个字节的float32类型） \n",
    "total_size_bytes = total_params * 4\n",
    "\n",
    "# 转换为兆字节（MB）\n",
    "total_size_mb = total_size_bytes / (1024 * 1024)\n",
    "\n",
    "print(f\"Total size of the model: {total_size_mb:.2f} MB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "309a3be4-c20a-4657-b4e0-77c97510b47c",
   "metadata": {},
   "source": [
    "- 练习：你可以尝试实现以下其他配置，这些配置也在 [GPT-2 论文](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)中提到.\n",
    "\n",
    "    - **GPT2-small** (我们已经实现的124M参数配置):\n",
    "        - \"emb_dim\" = 768\n",
    "        - \"n_layers\" = 12\n",
    "        - \"n_heads\" = 12\n",
    "\n",
    "    - **GPT2-medium:**\n",
    "        - \"emb_dim\" = 1024\n",
    "        - \"n_layers\" = 24\n",
    "        - \"n_heads\" = 16\n",
    "    \n",
    "    - **GPT2-large:**\n",
    "        - \"emb_dim\" = 1280\n",
    "        - \"n_layers\" = 36\n",
    "        - \"n_heads\" = 20\n",
    "    \n",
    "    - **GPT2-XL:**\n",
    "        - \"emb_dim\" = 1600\n",
    "        - \"n_layers\" = 48\n",
    "        - \"n_heads\" = 25"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da5d9bc0-95ab-45d4-9378-417628d86e35",
   "metadata": {},
   "source": [
    "## 4.7 生成文本"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48da5deb-6ee0-4b9b-8dd2-abed7ed65172",
   "metadata": {},
   "source": [
    "- LLMs（如我们上面实现的GPT模型）一次生成一个单词。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caade12a-fe97-480f-939c-87d24044edff",
   "metadata": {},
   "source": [
    "<img src=\"figures/iterative-gen.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d933457",
   "metadata": {},
   "source": [
    "- 下面的 `generate_text_simple` 函数实现了贪婪解码，这是一种简单快速的文本生成方法\n",
    "- 在贪婪解码中，模型在每一步都选择概率最高的单词（或 token）作为其下一个输出（最高的 logits 输出对应于最高的概率，所以我们甚至不需要显式地计算 softmax 函数）\n",
    "- 在下一章中，我们将实现一个更高级的 `generate_text` 函数\n",
    "- 下图描述了 GPT 模型如何在给定输入上下文的情况下生成下一个单词 token"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee0f32c-c18c-445e-b294-a879de2aa187",
   "metadata": {},
   "source": [
    "<img src=\"figures/generate-text.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "c9b428a9-8764-4b36-80cd-7d4e00595ba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_simple(model, idx, max_new_tokens, context_size):\n",
    "    # idx是当前上下文中的索引数组，形状为(B, T)\n",
    "    for _ in range(max_new_tokens):\n",
    "\n",
    "        # 如果当前上下文超过了支持的长度，就对当前上下文进行截断\n",
    "        # 例如，如果LLM只支持5个token，而上下文长度为10，\n",
    "        # 那么只有最后5个token会被用作上下文\n",
    "\n",
    "        idx_cond = idx[:, -context_size:]\n",
    "        \n",
    "        # 获取预测结果\n",
    "        with torch.no_grad():\n",
    "            logits = model(idx_cond)\n",
    "        \n",
    "        # 只关注最后一个时间步\n",
    "        # (batch, n_token, vocab_size)变为(batch, vocab_size)\n",
    "        logits = logits[:, -1, :]  \n",
    "\n",
    "        # 通过softmax函数获得对应的概率\n",
    "        probas = torch.softmax(logits, dim=-1)  # (batch, vocab_size)\n",
    "\n",
    "        # 获取概率值最高的单词索引\n",
    "        idx_next = torch.argmax(probas, dim=-1, keepdim=True)  # (batch, 1)\n",
    "\n",
    "        # 将采样到的索引添加到当前运行的上下文索引序列中\n",
    "        idx = torch.cat((idx, idx_next), dim=1)  # (batch, n_tokens+1)\n",
    "\n",
    "    return idx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6515f2c1-3cc7-421c-8d58-cc2f563b7030",
   "metadata": {},
   "source": [
    "- 上述的 `generate_text_simple` 函数实现了一次迭代过程，它一次生成一个token。\n",
    "\n",
    "<img src=\"figures/iterative-generate.webp\" width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f682eac4-f9bd-438b-9dec-6b1cc7bc05ce",
   "metadata": {},
   "source": [
    "- 让我们准备一个输入示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "bb3ffc8e-f95f-4a24-a978-939b8953ea3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.4929,  4.4812, -1.6093], grad_fn=<SliceBackward0>)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([    0.0000,     0.0012,     0.0000,  ...,     0.0000,     0.0000,\n",
       "            0.0000], grad_fn=<SoftmaxBackward0>)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = logits[0, -1, :]\n",
    "b[0] = -1.4929\n",
    "b[1] = 4.4812\n",
    "b[2] = -1.6093\n",
    "\n",
    "print(b[:3])\n",
    "torch.softmax(b, dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "3d7e3e94-df0f-4c0f-a6a1-423f500ac1d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoded: [15496, 11, 314, 716]\n",
      "encoded_tensor.shape: torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "start_context = \"Hello, I am\"\n",
    "\n",
    "encoded = tokenizer.encode(start_context)\n",
    "print(\"encoded:\", encoded)\n",
    "\n",
    "encoded_tensor = torch.tensor(encoded).unsqueeze(0)\n",
    "print(\"encoded_tensor.shape:\", encoded_tensor.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "a72a9b60-de66-44cf-b2f9-1e638934ada4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output: tensor([[15496,    11,   314,   716, 27018, 24086, 47843, 30961, 42348,  7267]])\n",
      "Output length: 10\n"
     ]
    }
   ],
   "source": [
    "model.eval() # 关闭 dropout\n",
    "\n",
    "out = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=encoded_tensor, \n",
    "    max_new_tokens=6, \n",
    "    context_size=GPT_CONFIG_124M[\"ctx_len\"]\n",
    ")\n",
    "\n",
    "print(\"Output:\", out)\n",
    "print(\"Output length:\", len(out[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d131c00-1787-44ba-bec3-7c145497b2c3",
   "metadata": {},
   "source": [
    "- 移除批次维度并转回文本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "053d99f6-5710-4446-8d52-117fb34ea9f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, I am Featureiman Byeswickattribute argue\n"
     ]
    }
   ],
   "source": [
    "decoded_text = tokenizer.decode(out.squeeze(0).tolist())\n",
    "print(decoded_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31806429",
   "metadata": {},
   "source": [
    "- 请注意，该模型尚未训练；因此上述文本是随机生成的\n",
    "- 我们将在下一章训练这个模型"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}