tips/RealNVP_mnist_en.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST X RealNVP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multivariate normal distribution $\\rightarrow$ MNIST\n",
    "\n",
    "| | Data Dimention|\n",
    "|----| ----|\n",
    "| Multivariate normal distribution (2) | n(2)|    \n",
    "|MNIST | 28 x 28 x 1 |\n",
    "|CIFER10| h x w x 3|\n",
    "| Text | Seq_len x Embedding_size ? |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RealNVP (Multi-Scale Architechture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Scale Architecture\n",
    "![](./multi-scale-arch.jpeg)\n",
    "\n",
    "RealNVP proposes above architecture for the efficient variable conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Squeezing\n",
    "![](squeezing.jpeg)\n",
    "\n",
    "||Dimention|\n",
    "|---|---|\n",
    "| Input | $h * w * c$ |\n",
    "| Output | $\\cfrac{h}{2} * \\cfrac{w}{2} * 4c$ |\n",
    "\n",
    "In Coupling Layer (RealNVP Bijector), Black is to be $x_a$ , White is to be $x_b$ ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Setting\n",
    "Multivariate normal distiribution $\\leftrightarrow$ MNIST dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Tensorflow\n",
    "\n",
    "## Implementation Plan\n",
    "0. Preprocess\n",
    "1. Create Dataset\n",
    "2. Build Single-Scale Model\n",
    "    1. RealNVP Bijector\n",
    "    2. BatchNormalization Bijector\n",
    "    3. Permutation Bijector\n",
    "    3. Squeeze Bijector\n",
    "    4. Single-Scale Model\n",
    "3. Build Multi-Scale Model\n",
    "    1. Blockwise Bijector\n",
    "    2. Multi-Scale Model\n",
    "4. Build Model\n",
    "    1. TransformDistribution\n",
    "    2. Loss, Optimizar\n",
    "    3. Training\n",
    "    4. Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow:  2.0.0-rc0\n",
      "tensorflow-probability:  0.8.0-rc0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import tensorflow_probability as tfp\n",
    "\n",
    "tfd = tfp.distributions\n",
    "tfb = tfp.bijectors\n",
    "\n",
    "print('tensorflow: ', tf.__version__)\n",
    "print('tensorflow-probability: ', tfp.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Target Distribution ($z$)\n",
    "In this part, use Multivariate normal distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow             : shape: (2, 2, 256) mean: -0.025957 sd: 1.030809\n",
      "Tensorflow Probability : shape: (2, 2, 256) mean: 0.007123 sd: 0.998965\n"
     ]
    }
   ],
   "source": [
    "# use Tensorflow's distribution\n",
    "z = tf.random.normal([2, 2, 256])\n",
    "print('Tensorflow             : shape: {} mean: {:.6f} sd: {:.6f}'.format(z.shape, tf.math.reduce_mean(z), tf.math.reduce_std(z)))\n",
    "# use Tensorflow Probability's distribution\n",
    "target_dist = tfd.Normal(loc=0., scale=1.)\n",
    "z = target_dist.sample([2, 2, 256])\n",
    "print('Tensorflow Probability : shape: {} mean: {:.6f} sd: {:.6f}'.format(z.shape, tf.math.reduce_mean(z), tf.math.reduce_std(z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Original Distribution (MNIST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# From tensorflow's BEGINNER TUTORIALS\n",
    "def _bytes_feature(value):\n",
    "  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n",
    "  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value.tobytes()]))\n",
    "\n",
    "def _float_feature(value):\n",
    "  \"\"\"Returns a float_list from a float / double.\"\"\"\n",
    "  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n",
    "\n",
    "def _int64_feature(value):\n",
    "  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n",
    "  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def _preprocess(x):\n",
    "    x = tf.pad(x, paddings=[[0, 0], [2, 2], [2, 2]], mode=\"CONSTANT\")\n",
    "    x = tf.expand_dims(x, axis=-1)\n",
    "    return x\n",
    "\n",
    "def create_mnist():\n",
    "    (train_x, train_y), (test_x, test_y) = tf.keras.datasets.mnist.load_data()\n",
    "    print('train_dataset: {} images'.format(train_x.shape[0]))\n",
    "    print('test_dataset : {} images'.format(test_x.shape[0]))\n",
    "    if not os.path.exists(\"mnists\"):\n",
    "        os.mkdir(\"mnists\")\n",
    "\n",
    "    def _serialize_example_pyfunction(img, label):\n",
    "        feature = {\n",
    "            \"img\": _bytes_feature(img.numpy()),\n",
    "            \"label\": _int64_feature(label.numpy()),\n",
    "        }\n",
    "        example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n",
    "        return example_proto.SerializeToString()\n",
    "    \n",
    "    @tf.function\n",
    "    def _gen_tf_serialize_example(img, label):\n",
    "        tf_string = tf.py_function(\n",
    "            _serialize_example_pyfunction, (img, label), tf.string\n",
    "        )\n",
    "        return tf.reshape(tf_string, ())\n",
    "\n",
    "    def _save_features_dataset(save_to, feature_dataset):\n",
    "        serialized_feature_dataset = train_features_dataset.map(\n",
    "            _gen_tf_serialize_example\n",
    "        )\n",
    "        writer = tf.data.experimental.TFRecordWriter(save_to)\n",
    "        writer.write(serialized_feature_dataset)\n",
    "\n",
    "    train_x = _preprocess(train_x)\n",
    "    train_features_dataset = tf.data.Dataset.from_tensor_slices((train_x, train_y))\n",
    "    _save_features_dataset(os.path.join('mnists', 'train.tfrecord'), train_features_dataset)\n",
    "    \n",
    "    test_x = _preprocess(test_x)\n",
    "    test_features_dataset = tf.data.Dataset.from_tensor_slices((test_x, test_y))\n",
    "    _save_features_dataset(os.path.join('mnists', 'test.tfrecord'), test_features_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_dataset: 60000 images\n",
      "test_dataset : 10000 images\n"
     ]
    }
   ],
   "source": [
    "create_mnist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_description  = {\n",
    "    'img': tf.io.FixedLenFeature([], tf.string),\n",
    "    'label': tf.io.FixedLenFeature([], tf.int64, default_value=0)\n",
    "}\n",
    "\n",
    "def _parse_function(example_proto):\n",
    "    feature = tf.io.parse_single_example(example_proto, feature_description)\n",
    "    img = tf.io.decode_raw(feature['img'], out_type=tf.uint8)\n",
    "    img = tf.reshape(img, [32, 32, 1])\n",
    "    img = tf.cast(img, dtype=tf.float32)\n",
    "    img = (img / (255.0 / 2)) - 1\n",
    "    feature['img'] = img\n",
    "    return feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<TFRecordDatasetV2 shapes: (), types: tf.string>\n",
      "<MapDataset shapes: {img: (32, 32, 1), label: ()}, types: {img: tf.float32, label: tf.int64}>\n"
     ]
    }
   ],
   "source": [
    "SHUFFLE_BUFFER_SIZE = 1000\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "filenames = [os.path.join('mnists', 'train.tfrecord')]\n",
    "raw_dataset = tf.data.TFRecordDataset(filenames)\n",
    "print(raw_dataset)\n",
    "parsed_dataset = raw_dataset.map(_parse_function) \n",
    "print(parsed_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAPvElEQVR4nO3dfYxUZZbH8e+xBV8QFZZa0iLaM2jcEF2BlOBGo+hk1DWjSLIxGONbjJiNyJpADEqysol/OLpqVIymUSJsFGURImzMOmgwhpgwFIotiKwvaRwIL23wbTVZFc/+UZdMQ+rprq6qW9Xt+X2STlc9p27dkwu/vlX3Vj3X3B0R+fU7ptUNiEhzKOwiQSjsIkEo7CJBKOwiQSjsIkEcW8/CZnYV8ATQBjzn7g/19fgxY8Z4R0dHPasUkT50d3fz5ZdfWqVazWE3szbgaeD3wG5gs5mtdfePUst0dHRQKpVqXaWI9KNYLCZr9byMnwp86u6fu/uPwMvAjDqeT0RyVE/YxwF/6XV/dzYmIoNQ7gfozGy2mZXMrNTT05P36kQkoZ6w7wHG97p/ejZ2BHfvdPeiuxcLhUIdqxORetQT9s3A2Wb2GzMbDswC1jamLRFptJqPxrv7z2Y2B3iD8qm3pe6+vWGdiUhD1XWe3d1fB15vUC8ikiN9gk4kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kCIVdJAiFXSQIhV0kiLquCGNm3cB3wCHgZ3dPXwleRFqqrrBnLnP3LxvwPCKSI72MFwmi3rA78Ccz22JmsxvRkIjko96X8Re7+x4z+1tgvZl97O7v9H5A9kdgNsAZZ5xR5+pEpFZ17dndfU/2+wCwBpha4TGd7l5092KhUKhndSJSh5rDbmYjzGzk4dvAFcC2RjUmIo1Vz8v4scAaMzv8PC+5+383pCsRabiaw+7unwPnN7AXEcmRTr2JBKGwiwShsIsEobCLBKGwiwTRiC/CyBBw6NChZO2bb75p+PoWL15ccfyHH35ILrNz585k7emnn07W5s+fn6ytWLGi4vjxxx+fXGbBggXJ2gMPPJCsDXbas4sEobCLBKGwiwShsIsEobCLBKGj8S30xRdfJGs//vhjsvbuu+8maxs3bqw4/vXXXyeXWbVqVbLWTOPHj0/W7r777mRtzZo1ydrIkSMrjp9/fvprHZdeemmyNpRpzy4ShMIuEoTCLhKEwi4ShMIuEoTCLhKETr3l7P3330/WLr/88mQtjy+nDAZtbW3J2oMPPpisjRgxIlm78cYbk7XTTjut4vioUaOSy5xzzjnJ2lCmPbtIEAq7SBAKu0gQCrtIEAq7SBAKu0gQ/Z56M7OlwB+AA+5+bjY2GngF6AC6gevd/av82hy6zjzzzGRtzJgxydpgOfU2bdq0ZK2v01cbNmyoOD58+PDkMjfddFP1jcmAVbNnfwG46qixBcBb7n428FZ2X0QGsX7Dnl1v/eBRwzOAZdntZcB1De5LRBqs1vfsY919b3Z7H+UruorIIFb3ATp3d8BTdTObbWYlMyv19PTUuzoRqVGtYd9vZu0A2e8DqQe6e6e7F929WCgUalydiNSr1rCvBW7Jbt8CvNaYdkQkL9WcelsBTAfGmNlu4AHgIWClmd0O7AKuz7PJoWz06NHJ2iOPPJKsrVu3LlmbPHlysjZ37tzqGutl0qRJydqbb76ZrPX1TbRt27ZVHH/yySerb0waqt+wu/sNidLvGtyLiORIn6ATCUJhFwlCYRcJQmEXCUJhFwlCE0620HXXpb9S0NdklKnrlwF0dXVVHH/uueeSy8yfPz9Z6+v0Wl/OPffciuOdnZ01PZ/UT3t2kSAUdpEgFHaRIBR2kSAUdpEgFHaRIHTqbZA6+eSTa1rulFNOGfAyfZ2WmzVrVrJ2zDHaVwwl+tcSCUJhFwlCYRcJQmEXCUJhFwlCR+N/ZRYtWlRxfMuWLcll3n777WStrznorrjiimrbkkFAe3aRIBR2kSAUdpEgFHaRIBR2kSAUdpEgqrn801LgD8ABdz83G1sE3AEcvizr/e7+el5NSvVSc8YtWbIkucyUKVOStTvuuCNZu+yyy5K1YrFYcfyuu+5KLmNmyZrUr5o9+wvAVRXGH3f3SdmPgi4yyPUbdnd/BzjYhF5EJEf1vGefY2ZdZrbUzEY1rCMRyUWtYX8GmABMAvYCj6YeaGazzaxkZqWenp7Uw0QkZzWF3d33u/shd/8FWAJM7eOxne5edPdioVCotU8RqVNNYTez9l53ZwLbGtOOiOSlmlNvK4DpwBgz2w08AEw3s0mAA93AnTn2KA0wYcKEZO2FF15I1m677bZkbfny5QOuff/998llbr755mStvb09WZPq9Bt2d7+hwvDzOfQiIjnSJ+hEglDYRYJQ2EWCUNhFglDYRYLQhJPCzJkzk7WzzjorWZs3b16ylpqo8r777ksus2vXrmRt4cKFydq4ceOSNfkr7dlFglDYRYJQ2EWCUNhFglDYRYJQ2EWC0Kk36dN5552XrK1cuTJZW7duXcXxW2+9NbnMs88+m6x98sknydr69euTNfkr7dlFglDYRYJQ2EWCUNhFglDYRYIwd2/ayorFopdKpaatTwaf4447Lln76aefkrVhw4Yla2+88UayNn369Kr6+rUoFouUSqWK19HSnl0kCIVdJAiFXSQIhV0kCIVdJAiFXSSIai7/NB5YDoylfLmnTnd/wsxGA68AHZQvAXW9u3+VX6vSCl1dXcnaqlWrkrXNmzdXHO/r9FpfJk6cmKxdcsklNT1nNNXs2X8G5rn7ROBC4C4zmwgsAN5y97OBt7L7IjJI9Rt2d9/r7u9lt78DdgDjgBnAsuxhy4Dr8mpSROo3oPfsZtYBTAY2AWPdfW9W2kf5Zb6IDFJVh93MTgJeBe5x929717z8mduKn7s1s9lmVjKzUk9PT13Nikjtqgq7mQ2jHPQX3X11NrzfzNqzejtwoNKy7t7p7kV3LxYKhUb0LCI16DfsZmaUr8e+w90f61VaC9yS3b4FeK3x7YlIo1QzB91FwE3Ah2a2NRu7H3gIWGlmtwO7gOvzaVEaYefOncnaU089laytXr06Wdu3b19dPR3t2GPT/x3b29uTtWOO0cdFqtFv2N19I1DxK3PA7xrbjojkRX8SRYJQ2EWCUNhFglDYRYJQ2EWC0OWfhqC+Tnm99NJLFccXL16cXKa7u7velqp2wQUXJGsLFy5M1q699to82glFe3aRIBR2kSAUdpEgFHaRIBR2kSAUdpEgdOqthfbv35+sbd++PVmbM2dOsvbxxx/X1dNATJs2LVm79957K47PmDEjuYy+vZYvbV2RIBR2kSAUdpEgFHaRIBR2kSB0NL4BDh48mKzdeeedydrWrVuTtc8++6yungbioosuStbmzZuXrF155ZXJ2gknnFBXT9J42rOLBKGwiwShsIsEobCLBKGwiwShsIsE0e+pNzMbDyynfElmBzrd/QkzWwTcARy+NOv97v56Xo02y6ZNm5K1hx9+uOL45s2bk8vs3r277p4G4sQTT6w4Pnfu3OQyfc39NmLEiLp7ksGhmvPsPwPz3P09MxsJbDGz9VntcXf/9/zaE5FGqeZab3uBvdnt78xsBzAu78ZEpLEG9J7dzDqAycDh17pzzKzLzJaa2agG9yYiDVR12M3sJOBV4B53/xZ4BpgATKK85380sdxsMyuZWamnp6fSQ0SkCaoKu5kNoxz0F919NYC773f3Q+7+C7AEmFppWXfvdPeiuxcLhUKj+haRAeo37GZmwPPADnd/rNd4e6+HzQS2Nb49EWmUao7GXwTcBHxoZoe/pnU/cIOZTaJ8Oq4bSH+9awhZs2ZNTbVaTJw4MVm75pprkrW2trZkbf78+RXHTz311Oobk1+lao7GbwSsQmnIn1MXiUSfoBMJQmEXCUJhFwlCYRcJQmEXCcLcvWkrKxaLXiqVmrY+kWiKxSKlUqnS2TPt2UWiUNhFglDYRYJQ2EWCUNhFglDYRYJQ2EWCUNhFglDYRYJQ2EWCUNhFglDYRYJQ2EWCUNhFglDYRYJQ2EWCUNhFglDYRYJQ2EWCqOZab8eb2Z/N7AMz225m/5aN/8bMNpnZp2b2ipkNz79dEalVNXv2/wMud/fzKV+e+SozuxD4I/C4u58FfAXcnl+bIlKvfsPuZf+b3R2W/ThwObAqG18GXJdLhyLSENVen70tu4LrAWA98Bnwtbv/nD1kNzAunxZFpBGqCru7H3L3ScDpwFTg76pdgZnNNrOSmZV6enpqbFNE6jWgo/Hu/jWwAfgH4FQzO3zJ59OBPYllOt296O7FQqFQV7MiUrtqjsYXzOzU7PYJwO+BHZRD/0/Zw24BXsurSRGp37H9P4R2YJmZtVH+47DS3f/LzD4CXjazB4H3gedz7FNE6tRv2N29C5hcYfxzyu/fRWQI0CfoRIJQ2EWCUNhFglDYRYJQ2EWCMHdv3srMeoBd2d0xwJdNW3ma+jiS+jjSUOvjTHev+Om1pob9iBWbldy92JKVqw/1EbAPvYwXCUJhFwmilWHvbOG6e1MfR1IfR/rV9NGy9+wi0lx6GS8SREvCbmZXmdnObLLKBa3oIeuj28w+NLOtZlZq4nqXmtkBM9vWa2y0ma03s0+y36Na1MciM9uTbZOtZnZ1E/oYb2YbzOyjbFLTf8nGm7pN+uijqdskt0le3b2pP0Ab5WmtfgsMBz4AJja7j6yXbmBMC9Z7CTAF2NZr7GFgQXZ7AfDHFvWxCJjf5O3RDkzJbo8E/geY2Oxt0kcfTd0mgAEnZbeHAZuAC4GVwKxs/FngnwfyvK3Ys08FPnX3z939R+BlYEYL+mgZd38HOHjU8AzKE3dCkybwTPTRdO6+193fy25/R3lylHE0eZv00UdTeVnDJ3ltRdjHAX/pdb+Vk1U68Ccz22Jms1vUw2Fj3X1vdnsfMLaFvcwxs67sZX7ubyd6M7MOyvMnbKKF2+SoPqDJ2ySPSV6jH6C72N2nAP8I3GVml7S6ISj/Zaf8h6gVngEmUL5GwF7g0Wat2MxOAl4F7nH3b3vXmrlNKvTR9G3idUzymtKKsO8Bxve6n5ysMm/uvif7fQBYQ2tn3tlvZu0A2e8DrWjC3fdn/9F+AZbQpG1iZsMoB+xFd1+dDTd9m1Tqo1XbJFv3gCd5TWlF2DcDZ2dHFocDs4C1zW7CzEaY2cjDt4ErgG19L5WrtZQn7oQWTuB5OFyZmTRhm5iZUZ7DcIe7P9ar1NRtkuqj2dskt0lem3WE8aijjVdTPtL5GbCwRT38lvKZgA+A7c3sA1hB+eXgT5Tfe90O/A3wFvAJ8CYwukV9/AfwIdBFOWztTejjYsov0buArdnP1c3eJn300dRtAvw95Ulcuyj/YfnXXv9n/wx8CvwncNxAnlefoBMJIvoBOpEwFHaRIBR2kSAUdpEgFHaRIBR2kSAUdpEgFHaRIP4fM1P6z1+fNGAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for image_features in parsed_dataset.take(1):\n",
    "    plt.imshow(tf.squeeze(image_features['img'], axis=-1), cmap='gray_r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Single-Scale Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RealNVP Bijector\n",
    "\n",
    "Formula\n",
    "1. forward function\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "x_a, x_b &=& split(x) \\\\\n",
    "(\\log{s}, t) &=& NN(x_b) \\\\\n",
    "s &=& exp(\\log{s}) \\\\\n",
    "y_a &=& s \\odot x_a + t \\\\\n",
    "y_b &=& x_b \\\\\n",
    "y &=& concat(y_a, y_b)\n",
    "\\end{eqnarray*}\n",
    "\n",
    "2. reverse function\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "y_a, y_b &=& split(y)\\\\\n",
    "(\\log{s}, t) &=& NN(y_b)\\\\\n",
    "s &=& exp(\\log{s})\\\\\n",
    "x_a&=& (y_a - t) / s\\\\\n",
    "x_b &=& y_b\\\\\n",
    "x &=&  concat(x_a, x_b)\n",
    "\\end{eqnarray*}\n",
    "\n",
    "3. log-determinant\n",
    "\n",
    "\\begin{eqnarray*}\n",
    "sum(\\log{|s|})\n",
    "\\end{eqnarray*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NN Layer\n",
    "ref. https://github.com/openai/glow/blob/master/model.py#L420-L426 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"nn_test\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         [(None, 16, 16, 2)]       0         \n",
      "_________________________________________________________________\n",
      "nn_1 (NN)                    ((None, 16, 16, 2), (None 79876     \n",
      "=================================================================\n",
      "Total params: 79,876\n",
      "Trainable params: 79,876\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Layer, BatchNormalization, ReLU, Conv2D\n",
    "from tensorflow.keras import Model\n",
    "\n",
    "\n",
    "class NN(Layer):\n",
    "    def __init__(\n",
    "        self,\n",
    "        input_shape,\n",
    "        n_hidden=[512, 512],\n",
    "        kernel_size=[[3, 3], [1, 1]],\n",
    "        strides=[[1, 1], [1, 1]],\n",
    "        activation=\"relu\",\n",
    "        name=None,\n",
    "    ):\n",
    "        if name:\n",
    "            super(NN, self).__init__(name=name)\n",
    "        else:\n",
    "            super(NN, self).__init__()\n",
    "        layer_list = []\n",
    "        for i, (hidden, kernel, stride) in enumerate(\n",
    "            zip(n_hidden, kernel_size, strides)\n",
    "        ):\n",
    "            layer_list.append(\n",
    "                Conv2D(\n",
    "                    hidden,\n",
    "                    kernel_size=kernel,\n",
    "                    strides=stride,\n",
    "                    activation=activation,\n",
    "                    padding='SAME',\n",
    "                    name=\"dense_{}_1\".format(i),\n",
    "                )\n",
    "            )\n",
    "        self.layer_list = layer_list\n",
    "        self.log_s_layer = Conv2D(\n",
    "            input_shape,\n",
    "            kernel_size=[3, 3],\n",
    "            strides=[1, 1],\n",
    "            padding='SAME',\n",
    "            kernel_initializer=\"zeros\",\n",
    "            activation=\"tanh\",\n",
    "            name=\"log_s\",\n",
    "        )\n",
    "        self.t_layer = Conv2D(\n",
    "            input_shape,\n",
    "            kernel_size=[3, 3],\n",
    "            strides=[1, 1],\n",
    "            padding='SAME',\n",
    "            kernel_initializer=\"zeros\",\n",
    "            name=\"t\",\n",
    "        )\n",
    "\n",
    "    def call(self, x):\n",
    "        y = x\n",
    "        for layer in self.layer_list:\n",
    "            y = layer(y)\n",
    "        log_s = self.log_s_layer(y)\n",
    "        t = self.t_layer(y)\n",
    "        return log_s, t\n",
    "\n",
    "\n",
    "def nn_test():\n",
    "    nn = NN(2, [256, 256])\n",
    "    x = tf.keras.Input([16, 16, 2])\n",
    "    log_s, t = nn(x)\n",
    "    # Non trainable params: -> Batch Normalization's params\n",
    "    tf.keras.Model(x, [log_s, t], name=\"nn_test\").summary()\n",
    "\n",
    "\n",
    "nn_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RealNVP Bijector\n",
    "ref. https://github.com/openai/glow/blob/master/model.py#L367-L383"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainable_variables : 8\n",
      "(64, 16, 16, 4) (64, 16, 16, 4) (64,)\n"
     ]
    }
   ],
   "source": [
    "class RealNVP(tfb.Bijector):\n",
    "    def __init__(\n",
    "        self,\n",
    "        input_shape,\n",
    "        # ??? this bijector do Tensor wise quantities. (I don't understand well...)\n",
    "        forward_min_event_ndims=3,\n",
    "        validate_args: bool = False,\n",
    "        name=\"real_nvp\",\n",
    "        n_hidden=[512, 512],\n",
    "        **kargs,\n",
    "    ):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            input_shape: \n",
    "                input_shape, \n",
    "                ex. [28, 28, 3] (image) [2] (x-y vector)\n",
    "            forward_min_event_ndims:\n",
    "                this bijector do \n",
    "                1. element-wize quantities => 0\n",
    "                2. vector-wize quantities => 1\n",
    "                3. matrix-wize quantities => 2\n",
    "                4. tensor-wize quantities => 3\n",
    "            n_hidden:\n",
    "                see. class NN\n",
    "            **kargs:\n",
    "                see. class NN\n",
    "                you can inuput NN's layers parameter here.\n",
    "        \"\"\"\n",
    "        super(RealNVP, self).__init__(\n",
    "            validate_args=validate_args,\n",
    "            forward_min_event_ndims=forward_min_event_ndims,\n",
    "            name=name,\n",
    "        )\n",
    "\n",
    "        assert input_shape[-1] % 2 == 0\n",
    "        self.input_shape = input_shape\n",
    "        nn_layer = NN(input_shape[-1] // 2, n_hidden)\n",
    "        nn_input_shape = input_shape.copy()\n",
    "        nn_input_shape[-1] = input_shape[-1] // 2\n",
    "        x = tf.keras.Input(nn_input_shape)\n",
    "        log_s, t = nn_layer(x)\n",
    "        self.nn = Model(x, [log_s, t], name=self.name + \"/nn\")\n",
    "\n",
    "    def _forward(self, x):\n",
    "        x_a, x_b = tf.split(x, 2, axis=-1)\n",
    "        y_b = x_b\n",
    "        log_s, t = self.nn(x_b)\n",
    "        s = tf.exp(log_s)\n",
    "        y_a = s * x_a + t\n",
    "        y = tf.concat([y_a, y_b], axis=-1)\n",
    "        return y\n",
    "\n",
    "    def _inverse(self, y):\n",
    "        y_a, y_b = tf.split(y, 2, axis=-1)\n",
    "        x_b = y_b\n",
    "        log_s, t = self.nn(y_b)\n",
    "        s = tf.exp(log_s)\n",
    "        x_a = (y_a - t) / s\n",
    "        x = tf.concat([x_a, x_b], axis=-1)\n",
    "        return x\n",
    "\n",
    "    def _forward_log_det_jacobian(self, x):\n",
    "        _, x_b = tf.split(x, 2, axis=-1)\n",
    "        log_s, t = self.nn(x_b)\n",
    "        return tf.reduce_sum(log_s)\n",
    "\n",
    "\n",
    "def realnvp_test():\n",
    "    realnvp = RealNVP(input_shape=[16, 16, 4], n_hidden=[256, 256])\n",
    "    x = tf.keras.Input([16, 16, 4])\n",
    "    y = realnvp.forward(x)\n",
    "    print(\"trainable_variables :\", len(realnvp.trainable_variables))\n",
    "\n",
    "    flow = tfd.TransformedDistribution(\n",
    "        event_shape=[16, 16, 4],\n",
    "        distribution=tfd.Normal(loc=0.0, scale=1.0),\n",
    "        bijector=realnvp,\n",
    "    )\n",
    "    x = flow.sample(64)\n",
    "    y = realnvp.inverse(x)\n",
    "    log_prob = flow.log_prob(y)\n",
    "    print(\n",
    "        x.shape,\n",
    "        y.shape,\n",
    "        log_prob.shape,\n",
    "        # -tf.reduce_mean(log_prob),\n",
    "        # -tf.reduce_mean(flow.distribution.log_prob(x)),\n",
    "        # -tf.reduce_mean(\n",
    "        #     flow.bijector.forward_log_det_jacobian(\n",
    "        #         x, event_ndims=flow._maybe_get_static_event_ndims()\n",
    "        #     )\n",
    "        # ),\n",
    "        # -tf.reduce_mean(flow._log_prob(x)),\n",
    "        # flow._finish_log_prob_for_one_fiber(\n",
    "        #     y,\n",
    "        #     x,\n",
    "        #     flow.bijector.forward_log_det_jacobian(\n",
    "        #         x, event_ndims=flow._maybe_get_static_event_ndims()\n",
    "        #     ),\n",
    "        #     flow._maybe_get_static_event_ndims(),\n",
    "        #     \n",
    "        # ),\n",
    "        # tf.reduce_sum(flow.distribution.log_prob(flow._maybe_rotate_dims(x, rotate_right=True)), axis=flow._reduce_event_indices)\n",
    "    )\n",
    "\n",
    "\n",
    "realnvp_test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BatchNormalization Layer\n",
    "\n",
    "ref. tensorflow_probability/bijector/BatchNormalization "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Permutation Bijector\n",
    "Reverse by Channel dims.\n",
    "In Glow, it improve this layer to Invertible 1x1 convolution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(-1.4117244, shape=(), dtype=float32)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=937, shape=(), dtype=float32, numpy=0.0>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class RevPermute(tfb.Bijector):\n",
    "    def __init__(\n",
    "        self,\n",
    "        axis=[-1],\n",
    "        forward_min_event_ndims=0,\n",
    "        validate_args=False,\n",
    "        name=\"RevPermute\",\n",
    "    ):\n",
    "        super(RevPermute, self).__init__(\n",
    "            forward_min_event_ndims=forward_min_event_ndims,\n",
    "            validate_args=validate_args,\n",
    "            name=name,\n",
    "            is_constant_jacobian=True\n",
    "        )\n",
    "        self._axis = axis\n",
    "\n",
    "    @property\n",
    "    def axis(self):\n",
    "        return self._axis\n",
    "\n",
    "    def _forward(self, x):\n",
    "        return tf.reverse(x, self.axis)\n",
    "\n",
    "    def _inverse(self, y):\n",
    "        return tf.reverse(y, self.axis)\n",
    "\n",
    "    def _forward_log_det_jacobian(self, x):\n",
    "        return tf.constant(0.0, dtype=x.dtype)\n",
    "\n",
    "    def _inverse_log_det_jacobian(self, y):\n",
    "        return tf.constant(0.0, dtype=y.dtype)\n",
    "\n",
    "\n",
    "def test_revPermute():\n",
    "    revPermute = RevPermute()\n",
    "    x = tf.keras.Input([16, 16, 4])\n",
    "    y = revPermute.forward(x)\n",
    "    # tf.keras.Model(x, y).summary()\n",
    "\n",
    "    x = tf.random.normal([2, 16, 16, 4])\n",
    "    y = revPermute.forward(x)\n",
    "    z = revPermute.inverse(y)\n",
    "    flow = tfd.TransformedDistribution(distribution=target_dist, bijector=revPermute)\n",
    "    print(tf.reduce_mean(flow.log_prob(tf.random.normal([2, 16, 16, 3]))))\n",
    "    return tf.reduce_sum(z - x)\n",
    "\n",
    "\n",
    "test_revPermute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Squeeze Bijector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(0, shape=(), dtype=int32)\n",
      "(64, 16, 16, 2) (64, 8, 8, 8) (64,)\n"
     ]
    }
   ],
   "source": [
    "class Squeeze3D(tfb.Bijector):\n",
    "    def __init__(\n",
    "        self,\n",
    "        factor=2,\n",
    "        forward_min_event_ndims=0,\n",
    "        inverse_min_event_ndims=0,\n",
    "        validate_args=False,\n",
    "        name=\"Squeeze\",\n",
    "    ):\n",
    "        self._factor = factor\n",
    "        super(Squeeze3D, self).__init__(\n",
    "            forward_min_event_ndims=forward_min_event_ndims,\n",
    "            inverse_min_event_ndims=inverse_min_event_ndims,\n",
    "            name=name,\n",
    "            is_constant_jacobian=True\n",
    "        )\n",
    "\n",
    "    @property\n",
    "    def factor(self):\n",
    "        return self._factor\n",
    "\n",
    "    def _forward(self, x):\n",
    "        (H, W, C) = x.shape[1:]\n",
    "        batch_size = tf.shape(x)[0:1]\n",
    "        tmp_shape = tf.concat(\n",
    "            [\n",
    "                batch_size,\n",
    "                (H // self.factor, self.factor, W // self.factor, self.factor, C),\n",
    "            ],\n",
    "            axis=0,\n",
    "        )\n",
    "        output_shape = tf.concat(\n",
    "            [batch_size, (H // self.factor, W // self.factor, C * self.factor ** 2)],\n",
    "            axis=0,\n",
    "        )\n",
    "        y = tf.reshape(x, tmp_shape)\n",
    "        y = tf.transpose(y, [0, 1, 3, 5, 2, 4])\n",
    "        y = tf.reshape(y, output_shape)\n",
    "        return y\n",
    "\n",
    "    def _inverse(self, y):\n",
    "        (H, W, C) = y.shape[1:]\n",
    "        batch_size = tf.shape(y)[0:1]\n",
    "        tmp_shape = tf.concat(\n",
    "            [batch_size, (H, W, C // self.factor ** 2, self.factor, self.factor)], axis=0\n",
    "        )\n",
    "        output_shape = tf.concat(\n",
    "            [batch_size, (H * self.factor, W * self.factor, C // self.factor ** 2)], axis=0\n",
    "        )\n",
    "        x = tf.reshape(y, tmp_shape)\n",
    "        x = tf.transpose(x, [0, 1, 4, 2, 5, 3])\n",
    "        x = tf.reshape(x, output_shape)\n",
    "        return x\n",
    "\n",
    "    def _forward_log_det_jacobian(self, x):\n",
    "        return tf.constant(0.0, dtype=x.dtype)\n",
    "\n",
    "\n",
    "def test_squeeze3D():\n",
    "    factor = 2\n",
    "    x = tf.Variable([[[1, 2, 5, 6], [3, 4, 7, 8], [9, 10, 13, 14], [11, 12, 15, 16]]])\n",
    "    x = tf.expand_dims(x, axis=-1)\n",
    "    squeeze3d = Squeeze3D()\n",
    "    y = squeeze3d.forward(x)\n",
    "    z = squeeze3d.inverse(y)\n",
    "    print(tf.reduce_sum(x - z))\n",
    "    \n",
    "    flow = tfd.TransformedDistribution(\n",
    "        event_shape=[16, 16, 2],\n",
    "        distribution=tfd.Normal(loc=0., scale=1.),\n",
    "        bijector=squeeze3d\n",
    "    )\n",
    "    x = tf.random.normal([64, 16, 16, 2])\n",
    "    y = flow.bijector.forward(x)\n",
    "    log_prob = flow.log_prob(y)\n",
    "    print(x.shape, y.shape, log_prob.shape)\n",
    "    \n",
    "\n",
    "test_squeeze3D()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single-Scale Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64,) tf.Tensor(1454.9274, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "def gen_flow_chain(level=3):\n",
    "    flow_chain_list = []\n",
    "    for i in range(level):\n",
    "        flow_chain_list.append(Squeeze3D())\n",
    "        flow_chain_list.append(tfb.BatchNormalization())\n",
    "        flow_chain_list.append(RevPermute()),\n",
    "        flow_chain_list.append(RealNVP(input_shape=[16, 16, 4], n_hidden=[256, 256])),\n",
    "        flow_chain_list.append(tfb.Invert(Squeeze3D()))\n",
    "    return tfb.Chain(list(reversed(flow_chain_list)))  \n",
    "    \n",
    "flow_bijector = gen_flow_chain()\n",
    "\n",
    "flow = tfd.TransformedDistribution(\n",
    "    event_shape=[32, 32, 1],\n",
    "    distribution=tfd.Normal(loc=0.0, scale=1.0),\n",
    "    bijector=tfb.Invert(flow_bijector),\n",
    ")\n",
    "\n",
    "x = tf.random.normal([64, 32, 32, 1])\n",
    "log_prob = flow.log_prob(x)\n",
    "print(log_prob.shape, -tf.reduce_mean(log_prob))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Multi-Scale Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Blockwise Bijector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=2670, shape=(), dtype=float32, numpy=0.0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ref: https://github.com/tensorflow/probability/blob/master/tensorflow_probability/python/bijectors/blockwise.py\n",
    "# TODO: ask is the event_ndim for forward/invert_log_determinants correctly.\n",
    "class Blockwise3D(tfb.Bijector):\n",
    "    def __init__(self, bijectors, block_sizes=None, validate_args=False, name=None):\n",
    "        if not name:\n",
    "            name = \"blockwise3D_of_\" + \"_and_\".join([b.name for b in bijectors])\n",
    "            name = name.replace(\"/\", \"\")\n",
    "        super(Blockwise3D, self).__init__(\n",
    "            # ???\n",
    "            forward_min_event_ndims=3,\n",
    "            validate_args=validate_args,\n",
    "            name=name,\n",
    "        )\n",
    "        self._bijectors = bijectors\n",
    "        self._block_sizes = block_sizes\n",
    "\n",
    "    @property\n",
    "    def bijectors(self):\n",
    "        return self._bijectors\n",
    "\n",
    "    @property\n",
    "    def block_sizes(self):\n",
    "        return self._block_sizes\n",
    "\n",
    "    def _forward(self, x):\n",
    "        split_x = (\n",
    "            tf.split(x, len(self.bijectors), axis=-1)\n",
    "            if self.block_sizes is None\n",
    "            else tf.split(x, self.block_sizes, axis=-1)\n",
    "        )\n",
    "        split_y = [b.forward(x_) for b, x_ in zip(self.bijectors, split_x)]\n",
    "        y = tf.concat(split_y, axis=-1)\n",
    "        return y\n",
    "\n",
    "    def _inverse(self, y):\n",
    "        split_y = (\n",
    "            tf.split(y, len(self.bijectors), axis=-1)\n",
    "            if self.block_sizes is None\n",
    "            else tf.split(y, self.block_sizes, axis=-1)\n",
    "        )\n",
    "        split_x = [b.inverse(y_) for b, y_ in zip(self.bijectors, split_y)]\n",
    "        x = tf.concat(split_x, axis=-1)\n",
    "        return x\n",
    "\n",
    "    def _forward_log_det_jacobian(self, x):\n",
    "        split_x = (\n",
    "            tf.split(x, len(self.bijectors), axis=-1)\n",
    "            if self.block_sizes is None\n",
    "            else tf.split(x, self.block_sizes, axis=-1)\n",
    "        )\n",
    "        fldjs = [\n",
    "            # ???\n",
    "            b.forward_log_det_jacobian(x_, event_ndims=3)\n",
    "            for b, x_ in zip(self.bijectors, split_x)\n",
    "        ]\n",
    "        return sum(fldjs)\n",
    "\n",
    "    def _inverse_log_det_jacobian(self, y):\n",
    "        split_y = (\n",
    "            tf.split(y, len(self.bijectors), axis=-1)\n",
    "            if self.block_sizes is None\n",
    "            else tf.split(y, self.block_sizes, axis=-1)\n",
    "        )\n",
    "        ildjs = [\n",
    "            b.inverse_log_det_jacobian(y_, event_ndims=3)\n",
    "            for b, y_ in zip(self.bijectors, split_y)\n",
    "        ]\n",
    "        return sum(ildjs)\n",
    "\n",
    "\n",
    "def test_blockwise3D():\n",
    "    blockwise3D = Blockwise3D(\n",
    "        bijectors=[\n",
    "            tfb.Identity(),\n",
    "            RealNVP(input_shape=[16, 16, 2], n_hidden=[256, 256]),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    x = tf.keras.Input([16, 16, 4])\n",
    "    y = blockwise3D.forward(x)\n",
    "    # tf.keras.Model(x, y).summary()\n",
    "\n",
    "    x = tf.random.normal([3, 16, 16, 4])\n",
    "    y = blockwise3D.forward(x)\n",
    "    z = blockwise3D.inverse(y)\n",
    "    return tf.reduce_sum(z - x)\n",
    "\n",
    "\n",
    "test_blockwise3D()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-Scale Model\n",
    "\n",
    "For simplicity, we use below model.\n",
    "0. Squeeze $[32, 32, 1] \\rightarrow [16, 16, 4]$ \n",
    "1. Flow-step\n",
    "2. if not last Flow-step    \n",
    "    BlockWise3D \n",
    "        1. Identity\n",
    "        2. 0 へ\n",
    "3. Unsqueeze $[16, 16, 4] \\rightarrow [32, 32, 1]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, indicates the transition of L=3.    \n",
    "In below code, export RealNVP's model architecture to png file.\n",
    "\n",
    "![](./multi-scale-arch.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=3223, shape=(), dtype=float32, numpy=0.0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def gen_flowSteps(\n",
    "    # for realnvp\n",
    "    input_shape: list,\n",
    "    n_hidden: list = [256, 256],\n",
    "    # for flowStep\n",
    "    k=2,\n",
    "    forward_min_event_ndims: int = 3,\n",
    "    validate_args: bool = False,\n",
    "    name: str = \"flow_step\",\n",
    "):\n",
    "    flow_step_list = []\n",
    "    for i in range(k):\n",
    "        flow_step_list.append(tfb.BatchNormalization(validate_args=validate_args))\n",
    "        flow_step_list.append(RevPermute(validate_args=validate_args)),\n",
    "        flow_step_list.append(\n",
    "            RealNVP(\n",
    "                input_shape=input_shape,\n",
    "                n_hidden=n_hidden,\n",
    "                validate_args=validate_args,\n",
    "                name=\"{}_{}/realnvp\".format(name, i),\n",
    "            )\n",
    "        )\n",
    "\n",
    "    flowSteps = tfb.Chain(\n",
    "        list(reversed(flow_step_list)), validate_args=validate_args, name=name\n",
    "    )\n",
    "    return flowSteps\n",
    "\n",
    "\n",
    "def test_gen_flowSteps():\n",
    "    flowSteps = gen_flowSteps(\n",
    "        k=2, input_shape=[16, 16, 4], forward_min_event_ndims=0, name=\"flowstep_0\"\n",
    "    )\n",
    "    x = tf.keras.Input([16, 16, 4])\n",
    "    y = flowSteps(x)\n",
    "    # tf.keras.Model(x, y).summary()\n",
    "\n",
    "    x = tf.random.normal([6, 16, 16, 4])\n",
    "    y = flowSteps.forward(x)\n",
    "    z = flowSteps.inverse(y)\n",
    "    return tf.reduce_sum(z - x)\n",
    "\n",
    "\n",
    "test_gen_flowSteps()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=5371, shape=(), dtype=float32, numpy=0.0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def gen_flow(input_shape, level=3, flow_step_args: dict = None):\n",
    "    def _gen_input_shapes(input_shape, level):\n",
    "        input_shape = input_shape\n",
    "        input_shapes = []\n",
    "        for i in range(level):\n",
    "            input_shape = [\n",
    "                input_shape[0] // 2,\n",
    "                input_shape[1] // 2,\n",
    "                input_shape[2] * 4 - i * 8,\n",
    "            ]\n",
    "            input_shapes.append(input_shape)\n",
    "        return input_shapes\n",
    "\n",
    "    input_shapes = _gen_input_shapes(input_shape, level)\n",
    "\n",
    "    def _add_flow(_input_shapes, flow_step_args):\n",
    "        flow_lists = []\n",
    "        flow_lists.append(\n",
    "            Squeeze3D(name=\"Squeeze_{}\".format(level - len(_input_shapes)))\n",
    "        )\n",
    "        flowSteps = gen_flowSteps(\n",
    "           k=2,\n",
    "           input_shape=_input_shapes[0],\n",
    "           name=\"Flowsteps_{}\".format(level - len(_input_shapes)),\n",
    "        )\n",
    "        flow_lists.append(flowSteps)\n",
    "        if len(_input_shapes) != 1:\n",
    "            flow_lists.append(\n",
    "                Blockwise3D(\n",
    "                    [\n",
    "                        tfb.Identity(),\n",
    "                        tfb.Chain(\n",
    "                            list(reversed(_add_flow(_input_shapes[1:], flow_step_args)))\n",
    "                        ),\n",
    "                    ],\n",
    "                    name=\"Blockwise3D_{}\".format(level - len(_input_shapes)),\n",
    "                )\n",
    "            )\n",
    "        flow_lists.append(\n",
    "            tfb.Invert(\n",
    "                Squeeze3D(name=\"Unsqueeze_{}\".format(level - len(_input_shapes)))\n",
    "            )\n",
    "        )\n",
    "        return flow_lists\n",
    "\n",
    "    return tfb.Chain(list(reversed(_add_flow(input_shapes, level))))\n",
    "\n",
    "\n",
    "def test_gen_flow():\n",
    "    flow = gen_flow([32, 32, 1])\n",
    "    print(len(flow.trainable_variables))\n",
    "    x = tf.keras.Input([32, 32, 1])\n",
    "    y = flow.forward(x)\n",
    "    # tf.keras.Model(x, y).summary()\n",
    "    tf.keras.utils.plot_model(\n",
    "        tf.keras.Model(x, y), show_shapes=True, to_file=\"realnvp.png\"\n",
    "    )\n",
    "    x = tf.random.normal([3, 32, 32, 1])\n",
    "    y = flow.forward(x)\n",
    "    z = flow.inverse(y) \n",
    "    return tf.reduce_sum(z - x)\n",
    "\n",
    "test_gen_flow()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TransformDistribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48\n",
      "trainable_variables:  48\n"
     ]
    }
   ],
   "source": [
    "flow_bijector = gen_flow([32, 32, 1])\n",
    "print(len(flow_bijector.trainable_variables))\n",
    "flow = tfd.TransformedDistribution(\n",
    "    event_shape =[32, 32, 1],\n",
    "    distribution=tfd.Normal(loc=0.0, scale=1.0),\n",
    "    bijector=flow_bijector\n",
    ")\n",
    "print('trainable_variables: ', len(flow.bijector.trainable_variables))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss, Optimizar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -r checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def loss():\n",
    "    return - tf.reduce_mean(flow.log_prob(targets['img']))\n",
    "\n",
    "\n",
    "optimizer = tf.optimizers.Adam(learning_rate=1e-4) \n",
    "log = tf.summary.create_file_writer('checkpoints')\n",
    "avg_loss = tf.keras.metrics.Mean(name='loss', dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/meguru/Github/tf-test/venv/lib/python3.7/site-packages/tensorflow_probability/python/bijectors/batch_normalization.py:207: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n",
      "tf.Tensor(823.6975, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE=64\n",
    "train_dataset = parsed_dataset.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)\n",
    "for target in train_dataset.take(1):\n",
    "    targets = target\n",
    "targets['img'].shape\n",
    "with tf.GradientTape() as tape:\n",
    "    log_prob_loss = loss()\n",
    "grads = tape.gradient(log_prob_loss, flow.trainable_variables)\n",
    "print(log_prob_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=1000> Loss -113284.187500\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=2000> Loss -113918.203125\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=3000> Loss -114220.031250\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=4000> Loss -114303.640625\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=5000> Loss -114461.773438\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=6000> Loss -114594.523438\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=7000> Loss -114703.226562\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=8000> Loss -114813.507812\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=9000> Loss -114831.562500\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=10000> Loss -114902.093750\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=11000> Loss -114933.117188\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=12000> Loss -114964.781250\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=13000> Loss -114969.093750\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=14000> Loss -115019.828125\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=15000> Loss -115067.046875\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=16000> Loss -115094.132812\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=17000> Loss -115019.656250\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=18000> Loss -115188.523438\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=19000> Loss -115286.343750\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=20000> Loss -115318.609375\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=21000> Loss -115341.648438\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=22000> Loss -115361.101562\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=23000> Loss -115377.367188\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=24000> Loss -115391.703125\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=25000> Loss -115402.632812\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=26000> Loss -115414.796875\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=27000> Loss -115425.859375\n",
      "Step <tf.Variable 'Adam/iter:0' shape=() dtype=int64, numpy=28000> Loss -115436.546875\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 30\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for targets in train_dataset:\n",
    "        with tf.GradientTape() as tape:\n",
    "            log_prob_loss = loss()\n",
    "        grads = tape.gradient(log_prob_loss, flow.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, flow.trainable_variables))\n",
    "        avg_loss.update_state(log_prob_loss)\n",
    "        if tf.equal(optimizer.iterations % 1000, 0):\n",
    "            print(\"Step {} Loss {:.6f}\".format(optimizer.iterations, avg_loss.result()))\n",
    "        if tf.equal(optimizer.iterations % 100, 0):\n",
    "            with log.as_default():\n",
    "                tf.summary.scalar(\"loss\", avg_loss.result(), step=optimizer.iterations)\n",
    "\n",
    "                avg_loss.reset_states()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "filenames = [os.path.join('mnists', 'test.tfrecord')]\n",
    "test_raw_dataset = tf.data.TFRecordDataset(filenames)\n",
    "test_parsed_dataset = test_raw_dataset.map(_parse_function) \n",
    "test_dataset = test_parsed_dataset.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log probability:  -2.0900693\n",
      "inv mean:  4.170579e-06  std:  1.530445\n",
      "re:trg mean:  -0.8117025  std:  0.53417826\n",
      "(1, 32, 32, 1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f0e2f047f90>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAHTCAYAAAD/OsuWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5CdVZX38d+SpHO/kHQnRBIJIAIpRMK0qDUol4HBYiYCVRYFVSNoWRNqFLzhDBflZRQLeN8a5Q8pRaaAQJUvjq+3RBRnIqZGmYFoC+GSQBIIQRJDukMSciMhCfv9o0+Ys9d5+jzn+uzT6e+nKpVe5zyXdZ6zyOacXns/FkIQAAAo1jtSJwAAwEjEAAwAQAIMwAAAJMAADABAAgzAAAAkwAAMAEACTQ3AZvZRM1ttZi+Y2fWtSgqohrpD0ag5tIM1Og/YzI6QtEbS+ZI2SPqDpMtDCKuG2qe7uzvMnTu3ofPh8LJ+/Xpt2bLF6t2PukMzGqm7Rmpu/PjxYcqUKU3l2m7+336z/MuSt0+7ny/qHK30+uuva8+ePZknGdXEcc+Q9EIIYZ0kmdkPJF0kaciinDt3rvr6+po4JQ4Xvb29je5K3aFhDdZd3TU3ZcoUfepTn3o7fuutt6Ln3/GO+MvHRgaePP4YPgd/vCOOOCL3fHmvwz/vj3nw4MEoHj16dBTv37+/6v6SdODAgSgeNWpUXc/7HPx1youl+t7PRYsWVez/9n5DPpPvaEmvlMUbSo8B7UTdoWjUHNqi7U1YZrbQzPrMrG9gYKDdpwMkUXcoXnnN7dmzJ3U6GAaaGYA3SppTFs8uPRYJIdwdQugNIfT29PQ0cTpAEnWH4tVdc+PHjy8sOQxfzQzAf5B0gpkda2Zdki6TtKQ1aQFDou5QNGoObdFwE1YI4YCZXS3p3yUdIeneEMLKlmUGZKDuULRGa668Ecc3NOXFeceT8rt7vbwGKa+W5qO8Y/jmpLwmq3379kXxuHHjKo7ptxkzZkwU7927N4p9E5a/Tl1dXVVzrKUZrd734u3catpqCCGEX0r6ZTPHAOpF3aFo1BzagZWwAABIgAEYAIAEGIABAEiAARgAgASaasICAOTz3cBeLesR5y37mHcO/3wjayT7rmWfg19aMu95vyyk71h+8803K3LwXcu+K9ofw+fg+RzyluiU8t+vvPfi7e1q2goAALQUAzAAAAkwAAMAkAADMAAACTAAAwCQAAMwAAAJMA0JANqs3hspZE1zqWWbavx0G7+/n64zduzYimP4KT/+GHk3X/A3Tli5Mr6nxcknnxzFW7durchhwoQJVXPyefucsqY2lfPXKWtKUd57kTf16e1j17QVAABoKQZgAAASYAAGACABBmAAABJgAAYAIAG6oAGgDap1KTfSBV3vzRf8Mf1NCvJy2rt3b8U2/mYKea/jwIEDUew7lOfOnRvF/jVNnDixIgffWe238Xn7c+a9Bn+dfFe0lH2DhnL+OgyFT8AAACTAAAwAQAIMwAAAJMAADABAAk01YZnZekk7JR2UdCCE0NuKpIBqqDsUjZpDO7SiC/qcEMKWFhwHqAd1h6LVVXPlnbC+KzZvreCsLtq8Lui8Y/puX98N7NdUnjp1asUxduzYEcV+XeY9e/ZE8e9+97so/trXvlb1+axzemvXro1ifx02bNgQxe9+97uj2HdR+45mf7y8bnOp8tqXx9U6ovkKGgCABJodgIOk/zCzP5rZwlYkBNSAukPRqDm0XLNfQZ8ZQthoZjMkLTWz50MIvy3foFSsCyXpXe96V5OnAyRRdyheXTU3efLkFDlimGnqE3AIYWPp735JP5V0RsY2d4cQekMIvT09Pc2cDpBE3aF49dbc+PHji04Rw1DDA7CZTTCzSYd+lvTXkp5tVWJAFuoORaPm0C7NfAU9U9JPS511oyT93xDCr1qS1WFuy5a4kTLvE9qaNWui+IQTTmh5TsMIddfBHnrooShesGBBFH/sYx+L4sWLF7c9pxZoqObqWQs6b01lKb8b1x8jr2vadwN7r7/+eu45+vv7o9h/8j/33HOj+IorrojiMWPGRPEll1wSxd3d3RU5TJo0KYp95/T8+fOj2P9763P261X761btfRxK+XrS1fZveAAOIayT9L5G9wcaQd2haNQc2oVpSAAAJMAADABAAgzAAAAkwAAMAEACrVgLGnXK64D07rvvvii+9dZbW54T0IhXX301im+++eYo9rU9UhdF8V3NviPZP5/1b8LBgwfrOkaevK5pv0ZyFr8W9PTp06P4qKOOiuJbbrkliv/pn/4pit///vdH8X/+539WnHPXrl1R7LuYn322+gyxefPmRbHv3M7rkpby14+u9b3gEzAAAAkwAAMAkAADMAAACTAAAwCQAAMwAAAJ0AU9DNxxxx1RPG7cuIptbrrppqLSAd42duzYKF6/fn3V7Xfv3t3GbDpLeSes75LNW6c5S94xytcfzno+73h+/zfffLNiH792s38/87q9fQfzl7/85Sj+6U9/GsXTpk2ryOHEE0+MYn/rxyOPPDKKR48eHcV+jes9e/ZEsX8Nfn+psiPdowsaAIAOxgAMAEACDMAAACTAAAwAQAIMwAAAJMAADABAAkxDSuCxxx6ra/t9+/ZF8UsvvdTKdICGvfzyy1Hsp67MnDkzim+88ca259SJ8qbn1CLvZgn+eR/7my/46TV++61bt1bkcMwxx0Sxn4bkp6X19fVFsf+3y09b81OMjj322IocfJ7+GG+88UYU+6lP/mYM/jX4my9kTefy13L//v1RXP7+VpuSxCdgAAASYAAGACABBmAAABJgAAYAIIHcAdjM7jWzfjN7tuyxaWa21MzWlv4+stoxgHpRd0iBukORaumCXiTpTkkPlD12vaRHQgi3m9n1pfi61qd3eOLGCTVZJOqu4/lF6X3Hp+/U9Yv5d6BFalHdlXfK1ro4f7Xt/c0S8o6Zd3MGfxMC3xU9a9asimPu3bs3in3H8fjx46s+39vbG8V/8zd/E8W+I9nPAJEq8/ad2d6OHTuq7u870ru6uqoeL4s/Rt7NGt7eL2+DEMJvJfl+9Isk3V/6+X5JF9d0NqBG1B1SoO5QpEZ/BzwzhLCp9POrkmZW2xhoEeoOKVB3aIumm7DC4PcgQ34XYmYLzazPzPoGBgaaPR0gibpDGtXqrrzm/NecQJZGB+DNZjZLkkp/9w+1YQjh7hBCbwiht6enp8HTAZKoO6RRU92V15z/XSiQpdEBeImkK0s/XylpcWvSAaqi7pACdYe2yO2CNrMHJZ0tqdvMNki6WdLtkn5oZp+W9LKkS9uZJEYe6i7mu0kl6dZbb43iq6++Oor9OszN2rRpU8Vjn/3sZ6PYr4l75513RvGcOXNamlOrtavufMeyX0vYdyjXsla07zD31953LPsO9AkTJkSxXwO5v7/yg75///wn/XXr1kXxz372syg+66yzovihhx6K4vPOOy+Ks+r+zDPPjOIVK1ZEse96PuWUU6LYvxd5vy7w61tL+d3/tXa95w7AIYTLh3jqr2o6A9AA6g4pUHcoEithAQCQAAMwAAAJMAADAJAAAzAAAAnUshY0muS79LZv317X/r5b0a+nisPfXXfdVfHY9773vSi+/PK4f6jVXdC/+MUvKh574oknotivo3vBBRe0NIfhpLwT1nc1+y5Z/7zviq5lG99ZndW9W27r1njFTX/86dOnV+zjz+Ffx0knnRTFX/rSl6LYr/Xst/f/Nma9hs2bN0exXwvaL7yza9euqjn7mq133W6psiM96/3LwidgAAASYAAGACABBmAAABJgAAYAIAEGYAAAEqALugAPPPBAFL/88st17f/1r389ij/zmc80nRM62/Lly6P4wQcfrNhmy5YtUfzII49E8bx585rKwa+p69ftlSrXH77wwgujeNy4cU3lcLjI66zNWys6i+9a9ms5++5fP5vCvzc+zurk3blzZxS/733vi+Lbb789iufOnRvFs2fPjmK/XvXZZ58dxX7dZ0m67777othfO/+6R42Khzl/3XwN+45mv+5z1jn9taq1k5pPwAAAJMAADABAAgzAAAAkwAAMAEACDMAAACTAAAwAQAJMQ2qDZcuWRfGiRYuaOt6CBQua2h/Dz7p166K4r6+vYpvJkydH8VlnndXSHFauXBnFS5Ysyd3npptuamkOw1n5VBQ/rcjHfhqLnzojVU6X8fv46TO+Pjx/Dn+jhBkzZlTsM378+Ch+7rnnovgDH/hAFPspPP4GEG+88UYUr1+/Por9a5akWbNmRbGfGjV69OgozrvxxZgxY6rmnCXr/RnqHNWmlPEJGACABBiAAQBIgAEYAIAEGIABAEggdwA2s3vNrN/Mni177J/NbKOZrSj9ubDaMYB6UXcoGjWHotXSBb1I0p2SHnCP3xFC+JeWZ3QY8N2E27dvT5TJsLZII6juHn744Sj2i9r7zk5JuuGGG6L41FNPbWlOt9xyS+42l112WRS/5z3vaWkOBVukNtVc3uL8tXTe+q5nXxP+ed9963Pw9fK73/0uirdt21aRw1FHHRXFvgP5mGOOieI9e/ZE8bnnnhvFV111VRSffvrpUfy9732vIodTTjklit///vdHsb+W/iYVY8eOjWL/73WtN1Io5699eVzteLmfgEMIv5W0NW87oJWoOxSNmkPRmvkd8NVm9nTpa5sjW5YRUB11h6JRc2iLRgfg70o6XtJpkjZJ+uZQG5rZQjPrM7O+gYGBBk8HSKLuULyGas5/9QpkaWgADiFsDiEcDCG8JelfJZ1RZdu7Qwi9IYTenp6eRvMEqDsUrtGa8ytGAVkaGoDNrHwtsEskPTvUtkCrUHcoGjWHdsrtgjazByWdLanbzDZIulnS2WZ2mqQgab2kq4Y8ANCAw73uNmzYEMVf/OIXo3j16tVRPHHixIpjXHfddS3N6bHHHoviX//611Gc1c3p15/OynO4aHXNla8H7K+d71D26zhnrTXs9/Hb+G5ff8y9e/dG8QsvvBDFxx13XBRn/ermmWeeiWK/trNfy9kfc8WKFVF88sknR7Ffl/mKK66oyMGvH7158+aKbcr56/bmm29W3d5fV9/hXIvyY1RbCzp3AA4hXJ7x8D11ZwTUgbpD0ag5FI2VsAAASIABGACABBiAAQBIgAEYAIAEalkLGgXz6wDPnTs3TSJomO8OffLJJ6P4mmuuieI1a9ZEse+c3LdvX8U5Lrrooii+5JJLonjOnDlR7DtMzzzzzCj2dee7ak866aSKHC699NKKxzCovPM5rwvaP5+1NrTvxvU1NmnSpCj277fvUPfnWL58eRT/3d/9XUUO69ati+IFCxZE8f333x/Fjz/+eBR3dXVF8X/9139FsV9L3K/bLEkbN26M4unTp0exv7aTJ0+OYt9F7bvFvfJu9qH4dblrWdtb4hMwAABJMAADAJAAAzAAAAkwAAMAkAADMAAACdAF3QK+W/RXv/pVU8ebMmVKFGetC4vO5ruezz///KaO52tMkn7+859XjT3fqXniiSdG8dq1a6vun7WG7v79+6vuM5KVdzbnddL657M6c313r+9y9nzHsa8h3x18wQUXRPHKlSsrjunvLPazn/0sio866qgo/sxnPhPFfi3pefPmRbG/i1TWv32+G9xfO7/mtY/z9vcd6VlroPv3wnc9Z+2ThU/AAAAkwAAMAEACDMAAACTAAAwAQAIMwAAAJMAADABAAsxvaQE/PeM73/lOokzQKfwUjnPOOSeK/SL2CxcujOLVq1dHcdbUNr8I/bhx46J4w4YNUfyb3/wmih955JGKY5abMWNGFH/1q1/NzQGDQghVp6LkTXXJmrbkpxH5KTsTJkyIYn9jA3+zBX8DkN27dw+Z7yHd3d1RvHjx4ig++uijo/iEE06I4lWrVkXxRz/60Sj2U6u2bdtWkcPrr79edR9/7fzzfn+/vZ9ilPVe+KlM1d7PanXAJ2AAABJgAAYAIAEGYAAAEmAABgAggdwB2MzmmNkyM1tlZivN7POlx6eZ2VIzW1v6+8j2p4uRgrpD0ag5FK2WLugDkq4NITxhZpMk/dHMlkr6pKRHQgi3m9n1kq6XdF37Uj18XXzxxVH8yU9+Mk0inWVY193nPve5qnGe+fPnV40bcdZZZ0VxXhf0bbfdFsWf+tSnms6hw7Ws5swsuqGC74TNuyGAfz7rGP4GAP39/VH82muvRfHYsWOj+F3veldFzuVeeumlihzmzJkTxR/+8Iej2HfF+3P6my+8853vjGLf/Z91U4qpU6dGsb8u/nXs27cvivNuYlGLvJsxlN9Ewm9bLvcTcAhhUwjhidLPOyU9J+loSRdJur+02f2SLs4+AlA/6g5Fo+ZQtLp+B2xmcyXNl7Rc0swQwqbSU69KmtnSzIAS6g5Fo+ZQhJoHYDObKOnHkr4QQthR/lwY/A4gc7axmS00sz4z6xsYGGgqWYw81B2K1oqa27NnTwGZYriraQA2s9EaLMjvhxB+Unp4s5nNKj0/S1J/1r4hhLtDCL0hhF5/M2egGuoORWtVzflVqoAstXRBm6R7JD0XQvhW2VNLJF1Z+vlKSYv9vkCjqDsUjZpD0Wrpgv5LSZ+Q9IyZrSg9dqOk2yX90Mw+LellSZe2J8XDX1dXVxS3okvvMEDdNcl3lH7jG9+I4pkz419lXnPNNVF8xRVXtCexztWymgshVHTGlstbfzirC9qvOe+7gX3H8f79+6N4586dVc/hvymaPHlyRQ7vfe97o/iZZ56JYr8e9Y9+9KMoPuWUU6L4ggsuqLp/1jrK27dvj+Ijj4xnhfk1rf2/r3v37o3i0aNHV5wj73l/Lb3yf8OrdUHnDsAhhEclDXWEv8rbH2gEdYeiUXMoGithAQCQAAMwAAAJMAADAJAAAzAAAAnU0gWNHF/5ylfq2n7KlClRfN11HbeUMYahAwcORPHixfFsmYcffjiKfUfqVVddFcVZ6/CiNmZW0ensn68mq4Padzn7bXzXs3/ed037taP9msnHHHNMRQ7f/va3o7i7uzuK/frTf/u3fxvFfk3zdevWVZyjXNY1zOtq9vv4/y7yrmPe85I0bdq0qufI6mLPwidgAAASYAAGACABBmAAABJgAAYAIAEGYAAAEqALugUef/zxurb3a4v6zsDh4Nprr6147Jvf/GaCTHDIo48+GsW33XZbFM+aNSuKfUfr9OnT25PYCFXe6ew7aX2Hue+azer+9Z3TvuvZrxXt11H2Ofi1n8eNGxfF//iP/1iRw1NPPRXFf/rTn6L4vvvui+Lzzjsviv36474L+uSTT66as1T5Ov2a1W+88UYU++vmj+nr3nc033zzzRU5fPnLX45i3zntO8qHwidgAAASYAAGACABBmAAABJgAAYAIAEGYAAAEqALGg2h47nz3HLLLVH8+uuvR/Gpp54axWeffXa7UxrRyruQ6+169h3MWcfw3bpjxoyJ4j179lTNb9u2bVVzevDBByv2+eAHPxjFJ554YhT7Gjz99NOj2Hduz5s3r+rzWd3EEyZMqLpP3rX0z7/yyitRPH78+Cj+6le/WpHD7t27o7jatc56L9/OZchnAABA2zAAAwCQAAMwAAAJMAADAJAAAzAAAAnkDsBmNsfMlpnZKjNbaWafLz3+z2a20cxWlP5c2P50MVJQdygaNYei1TIN6YCka0MIT5jZJEl/NLOlpefuCCH8S/vSwwhG3eXYsmVLFG/YsKHq9n5hfFRoac2V3wTAT0XxNwiotu9Qx/DxqFHxP+ezZ8+OYn8TAj8NbenSpVE8adKkihy+9KUvRfHHP/7xKH7nO98Zxf7mDB/5yEei2N8IYdWqVVHspxxJldOl/FQl/zr9zRv8zXD8zRz89K5GlB+j2jSk3AE4hLBJ0qbSzzvN7DlJRzedIVAFdYeiUXMoWl2/AzazuZLmS1peeuhqM3vazO41syOH2GehmfWZWd/AwEBTyWJkou5QtGZrLm8RDECqYwA2s4mSfizpCyGEHZK+K+l4Sadp8P8aM5dGCiHcHULoDSH0+vtPAnmoOxStFTXnV1MCstQ0AJvZaA0W5PdDCD+RpBDC5hDCwRDCW5L+VdIZ7UsTIxF1h6JRcyhSLV3QJukeSc+FEL5V9visss0ukfRs69PDSEXdoWjUHIpWSxf0X0r6hKRnzGxF6bEbJV1uZqdJCpLWS7qqLRkOA4899ljqFA5H1F2O7u7uKF69enWiTA4bLa25at2vXt7NGaTKmw54vvvX9z74mzksW7as6jn7+voqznHDDTdE8RtvvFE1h7//+7+P4h07dkTxxo0bo7irqyuKd+3aVZGD72L219m/Tt/Nndfl7K+Dv6lJVg7VjlGt472WLuhHJWUd4Zd5+wKNou5QNGoORWMlLAAAEmAABgAgAQZgAAASYAAGACCBWrqgAQB1Ku/O9Z26vtM2b51nqbJ7N68Td+LEiVHsO5anTJkSxdu2bYtiv4ayJD3//PNRfPzxx1c9hl+QZOfOnVHsu5xnzpwZxVkrivmuYr8Gtu8W96/DX9t6u6qlyvciq2u9FnwCBgAgAQZgAAASYAAGACABBmAAABJgAAYAIAG6oAGgDco7Y/PWhfZdtFnrB+d1PXu+g9gfs7+/v+rxs87nO6d3794dxf51bN++PYrHjBlT9Xi+U9uvDS1Vvg6//rTvWq5nTe4sWe+F77xu9Bx8AgYAIAEGYAAAEmAABgAgAQZgAAASYAAGACABuqABoA3eeuutIZ/L6qyt93i+E9fbt29fFOd1OfvuYb+Os1TZWZ21XnS57u7uKPZd0f46+Jx8V3TWNnlrN/tz+HWcvVreG3+OvE7sofAJGACABBiAAQBIgAEYAIAEGIABAEiAARgAgARyB2AzG2tmvzezp8xspZl9rfT4sWa23MxeMLN/M7PKRTuBBlF3KBo1h6LVMg1pn6RzQwi7zGy0pEfN7GFJX5J0RwjhB2Z2l6RPS/puG3PFyELdoWgtrbl6bsbgp75kbe+nHeVNS/I3PvD8/v5GCq+99lrFPmPHjo3i/fv3Vz3Gzp07q+bopwTlTUvKOke16V5S5bXMu461TEOq95hDyf0EHAbtKoWjS3+CpHMl/aj0+P2SLq7pjEANqDsUjZpD0Wr6HbCZHWFmKyT1S1oq6UVJ20MIh/73ZYOko4fYd6GZ9ZlZ38DAQCtyxghB3aForao5v2AFkKWmATiEcDCEcJqk2ZLOkHRSrScIIdwdQugNIfT29PQ0mCZGIuoORWtVzWWtIgV4dXVBhxC2S1om6UOSpprZoS++Z0va2OLcAEnUHYpHzaEItXRB95jZ1NLP4ySdL+k5DRbnx0ubXSlpcbuSxMhD3aFo1ByKVksX9CxJ95vZERocsH8YQnjIzFZJ+oGZfUPSk5LuaWOeGHmoOxStpTVX3ilbb6dtVhe03yevGzjvnPV2E0uV3b6+S3nv3r1R7H8XPmHChNxzlPM5ZsnrIM/rQM+Ttb9/rFoO1c6fOwCHEJ6WND/j8XUa/B0J0HLUHYpGzaForIQFAEACDMAAACTAAAwAQAIMwAAAJGDNdojVdTKzAUkvS+qWtKWwEzeGHFtjqByPCSEUskIGdddywznHQuqOmmu54ZzjkDVX6AD89knN+kIIvYWfuA7k2BqdlGMn5TIUcmyNTsmxU/Kohhxbo5Ec+QoaAIAEGIABAEgg1QB8d6Lz1oMcW6OTcuykXIZCjq3RKTl2Sh7VkGNr1J1jkt8BAwAw0vEVNAAACRQ+AJvZR81stZm9YGbXF33+LGZ2r5n1m9mzZY9NM7OlZra29PeRCfObY2bLzGyVma00s893Wo6lfMaa2e/N7KlSnl8rPX6smS0vvef/ZmZdBedFzTWWY8fXXafWXCkH6q6xHEdO3YUQCvsj6QhJL0o6TlKXpKckzSsyhyHy+oik0yU9W/bY/5F0fenn6yX974T5zZJ0eunnSZLWSJrXSTmWcjBJE0s/j5a0XNIHJf1Q0mWlx++S9A/UXGfX3HCpu06sOeqOuqu17opO+kOS/r0svkHSDSnf7LJc5rqiXC1pVllBrE6dY1luizV4r9JOznG8pCckfUCDk9NHZdVAAXlQc63Lt6PrrlNqLuuc1B11l/Wn6K+gj5b0Slm8ofRYJ5oZQthU+vlVSTNTJnOImc3V4C3TlqsDczSzI8xshaR+SUs1+ClgewjhQGmTot9zaq4FOrnuOrDmJOquJQ73uqMJqwZh8H9nkreLm9lEST+W9IUQwo7y5zolxxDCwRDCaZJma/AeqiclTmlY6pT3U+r8uqPmWqcT3s9DRkLdFT0Ab5Q0pyyeXXqsE202s1mSVPq7P2UyZjZag8X4/RDCT0oPd1SO5UII2yUt0+DXMFPNbFTpqaLfc2quCcOp7jqo5iTqrikjpe6KHoD/IOmEUqdYl6TLJC0pOIdaLZF0ZennKzX4e4gkzMwk3SPpuRDCt8qe6pgcJcnMesxsaunncRr8vc1zGizOj5c2KzpPaq5Bw6HuOrTmJOquYSOq7hL8wvpCDXa1vSjpK6l/gV7K6UFJmyTt1+D39p+WNF3SI5LWSvq1pGkJ8ztTg1+3PC1pRenPhZ2UYynPUyU9WcrzWUn/q/T4cZJ+L+kFSf9P0hhqrrNrbrjUXafWHHVH3dVSd6yEBQBAAjRhAQCQAAMwAAAJMAADAJAAAzAAAAkwAAMAkAADMAAACTAAAwCQAAMwAAAJMAADAJAAAzAAAAkwAAMAkAADMAAACTAAAwCQAAMwAAAJMAADAJAAAzAAAAkwAAMAkAADMAAACTAAAwCQAAMwAAAJMAADAJAAAzAAAAk0NQCb2UfNbLWZvWBm17cqKaAa6g5Fo+bQDhZCaGxHsyMkrZF0vqQNkv4g6fIQwqqh9unu7g5z585t6Hw4vKxfv15btmyxevej7tCMRuqukZqbNm1amD17dlO5Qsoan8zq/mcjqQ0bNmjr1q2ZSY9q4rhnSHohhLBOkszsB5IukjRkUc6dO1d9fX1NnBKHi97e3kZ3pe7QsAbrru6amz17tpYsWfJ23ImDhs+p0Q9j1eQd8x3viL+E9du/9dZbFfv4vPOO4Z/POma1/Zt97xYsWDDkc818BX20pFfK4g2lxz6ccwkAABy+SURBVIB2ou5QNGoObdH2JiwzW2hmfWbWNzAw0O7TAZKoOxSvvOZee+211OlgGGhmAN4oaU5ZPLv0WCSEcHcIoTeE0NvT09PE6QBJ1B2KV3fNTZ8+vbDkMHw1MwD/QdIJZnasmXVJukzSkpx9gGZRdygaNYe2aLgJK4RwwMyulvTvko6QdG8IYWXLMgMyUHcoWqM155t/3DGj2DcGZe27f//+KB49enRdxzhw4EDV/Q8ePBjFXV1dFTns2bMniseNGxfF+/bti+JRo+Ihxjc0+XO++eabuTn4a+djf52OOOKIus5RS9NVXqNXeQ7VGtGa6YJWCOGXkn7ZzDGAelF3KBo1h3ZgJSwAABJgAAYAIAEGYAAAEmAABgAggaaasAAA2epZ2rFax/Qhvmu53mP4jmSfn+8W9t3EkjRmzJiq5/A5+th3Yo8dO7bq9j5nKb/r2R/Td4f7zu08tbyP/hzl17JaVzWfgAEASIABGACABBiAAQBIgAEYAIAEGIABAEiAARgAgASYhgQAbeZvOuCn1/gbBGRNOco7Rt50mbz9/fNZx/Pb+Ok3/nX4qU1vvPFGFE+aNKnq81lTeLZt2xbFfvqVz8FPO9q7d28UT548uWoOWdOW8m6MUSs+AQMAkAADMAAACTAAAwCQAAMwAAAJMAADAJAAXdAA0Ga+G9h3GNfSRZt3jHr39x3NvuO42k0EDhk/fnwU+xsh+JsvvOtd74riffv2RXFXV1cUv/rqqxXn3LlzZ9Vz+Jx8t7fP0V8H/3zWdc7rQK/l2kl8AgYAIAkGYAAAEmAABgAgAQZgAAASaKoJy8zWS9op6aCkAyGE3lYkBVRD3aFo1BzaoRVd0OeEELa04DhAPag7FK2umivvhPVrJvv1i33sO3Olyg5h30Hsu5z9OX3nru8e9sfL4vPcunVr1RyPO+64KP7DH/4Qxf51+g7nrC5ov3ZzT09PFB9//PFVj+nXgvY5e1nXxXc5++tCFzQAAB2s2QE4SPoPM/ujmS1sRUJADag7FI2aQ8s1+xX0mSGEjWY2Q9JSM3s+hPDb8g1KxbpQqpyEDTSIukPR6qq5o48+OkWOGGaa+gQcQthY+rtf0k8lnZGxzd0hhN4QQq//rh5oBHWHotVbc9OmTSs6RQxDDQ/AZjbBzCYd+lnSX0t6tlWJAVmoOxSNmkO7NPMV9ExJPy11e42S9H9DCL9qSVaHuS1b4kbKvE9oa9asieITTjih5TkNI9RdB3vooYeieMGCBVH8sY99LIoXL17c9pxaoKGaK18f2HfF+rWDs7qevTfffDOK/TF917Pnu579/n4NZN9VnZWn32fMmDFR/Pzzz0fx17/+9Sju7++P4k984hNR7LuLJWncuHFRPHHixCj262r7nPy1913R/rpkXQfPH7PWdbobHoBDCOskva/R/YFGUHcoGjWHdmEaEgAACTAAAwCQAAMwAAAJMAADAJBAK9aCRp18l13euqH33XdfFN96660tzwlohF+r9+abb45iX9sjdVEU3z3s12X2z2f9m+C7mH23b97az76L2q+B3Egntt9n06ZNUezrw8/4mD9/fhT/+c9/rrq9JO3evTuK165dG8X+dfm1oI866qgonjJlShS/8cYbUZz1XvjrUEundBY+AQMAkAADMAAACTAAAwCQAAMwAAAJMAADAJAAXdDDwB133BHFfi1USbrpppuKSgd4m18LeP369VW39x2sh7Py9YB9l6xfK7iWmRG+69nz6ybndUX7HPz+teTgX5dfd3nGjBlR7Nex98dbt25dFI8fP74ih+OOO65q7LuafY779++P4kbW5fbXqta1nyuO09BeAACgKQzAAAAkwAAMAEACDMAAACTAAAwAQAIMwAAAJMA0pAQee+yxurbft29fFL/00kutTAdo2MsvvxzFfpH6mTNnRvGNN97Y9pw6Rd5NVurlpxX56TX+Zg1+2pG/yUDWdMZyfkqRJE2aNCmKX3nllSju7u6O4u3bt0fxBz/4wSj204ymTZsWxY8//nhuXn5q29y5c6seM+8mFRMnTozivXv3VuSQd/OFPXv2vP2zf9/K8QkYAIAEGIABAEiAARgAgAQYgAEASCB3ADaze82s38yeLXtsmpktNbO1pb+PbG+aGGmoO6RA3aFItXRBL5J0p6QHyh67XtIjIYTbzez6Unxd69M7PHHjhJosEnXX8fzC9X5R+rzF+jvQIiWou1o6pvNuAOCvtT+m73r2NyXwN0bwsy8kadeuXVHs33/fib158+Yo9jdK8OfcuXNnFGfdjMF3Kb/3ve+N4qlTp0axv07+dfvrltX17FXrbJbim5T4961c7ifgEMJvJW11D18k6f7Sz/dLujjvOEA9qDukQN2hSI3+DnhmCGFT6edXJc2stjHQItQdUqDu0BZNN2GFwc/3Q94M0cwWmlmfmfUNDAw0ezpAEnWHNKrVXXnNbd3qP0QDlRodgDeb2SxJKv3dP9SGIYS7Qwi9IYTenp6eBk8HSKLukEZNdVdec371JSBLowPwEklXln6+UtLi1qQDVEXdIQXqDm2R2wVtZg9KOltSt5ltkHSzpNsl/dDMPi3pZUmXtjNJjDzUXcyv4ytJt956axRfffXVUezXYW7Wpk2bKh777Gc/G8W+w/TOO++M4jlz5rQ0p1ZrV935bmHf/eu7an3nrlTZTeu7nP05/PZ+zeTyTl1J2rJlSxRPmDChIgffMTx58uQoLl8DWarsSPZ8Xfscs9ZcnjFjRhT7rmV/XXxntn/d/rr5taGzcvAd4r7DvPx1VeuYzh2AQwiXD/HUX+XtCzSKukMK1B2KxEpYAAAkwAAMAEACDMAAACTAAAwAQAK1rAWNJq1YsSKKt2/fXtf+vhuxt7e36ZwwvNx1110Vj33ve9+L4ssvj/uHWt0F/Ytf/KLisSeeeCKKu7q6oviCCy5oaQ7Dle+kzepyLpe1frDfx3f7jhoV/3Puu2+rdepKlR3N/nhZefmc/L9VGzZsiOL+/ngK9e9+97sofvnll6P4+OOPr8jBP+bXi/brVfsu6LzXkPW6Pd9J7a91y9aCBgAArccADABAAgzAAAAkwAAMAEACDMAAACRAF3QBHnjggSj2nX55vv71r0fxZz7zmaZzQmdbvnx5FD/44IMV2/i1ex955JEonjdvXlM57NixI4ofeuihim382s8XXnhhFPvO25GkvEs5b51m30Xr14qWKq+176zOW4f59ddfj+LXXnutak7d3d0VOcyaNSuKfSf1qlWroviPf/xjFL/44otR/NJLL0XxscceG8WnnXZaRQ5+Lei1a9dGsa853909ZcqUKPZd035//9+BVNlZ7bv/y9+rah3vfAIGACABBmAAABJgAAYAIAEGYAAAEmAABgAgAQZgAAASYBpSGyxbtiyKFy1a1NTxFixY0NT+GH7WrVsXxX19fRXb+OkVZ511VktzWLlyZRQvWbIkd5+bbrqppTkMZ+XTT/wUH38jBf+8n7YkVU478tuMGTMmiv1UGb+/n6bkTZw4seIxPyXnmWeeieLf/va3Ufyzn/0siv2UHH+zhnPOOSeKP/ShD1Xk4Kds+WlJ/nX6a+1vhuOP56+b31+qnCZW7YYOWfsfwidgAAASYAAGACABBmAAABJgAAYAIIHcAdjM7jWzfjN7tuyxfzazjWa2ovTnwmrHAOpF3aFo1ByKVksX9CJJd0p6wD1+RwjhX1qe0WFg9+7dUey77lCTRRpBdffwww9H8e233x7FWYvz33DDDVF86qmntjSnW265JXebyy67LIrf8573tDSHgi1SC2uuWverf8530fquaKnyZgr+pgL+Zg07d+6sGo8dOzaKfffvn/70p4ocnn/++SjevHlzFP/iF7+I4qeffrriGOVOOOGEqjm9+uqrFfvMnj07iv1sAH+DCN/lXN6hLElvvvlmFPubWkyaNKkiB//++XOUd6g3dTOGEMJvJW3N2w5oJeoORaPmULRmfgd8tZk9Xfra5siWZQRUR92haNQc2qLRAfi7ko6XdJqkTZK+OdSGZrbQzPrMrG9gYKDB0wGSqDsUr6Ga27qVD9LI19AAHELYHEI4GEJ4S9K/SjqjyrZ3hxB6Qwi9PT09jeYJUHcoXKM1N23atOKSxLDV0ABsZrPKwkskPTvUtkCrUHcoGjWHdsrtgjazByWdLanbzDZIulnS2WZ2mqQgab2kq9qYI0agw73u/Bq4X/ziF6N49erVUZy1Lu91113X0pwee+yxKP71r38dxVndnH796aw8h4tW11y17tdqzw31vO/29XynvF8r2j+ft1b0Sy+9VHEO3/X8q1/9Kor9Gua+q/nd7353FJ988slR/OEPfziKp0+fXpGDz9N3d/vX6TuWfbe4P96ECROiOGtd7jzlXezVuuFzB+AQwuUZD99Td0ZAHag7FI2aQ9FYCQsAgAQYgAEASIABGACABBiAAQBIoJa1oFEwvw7w3Llz0ySChvmFGJ588skovuaaa6J4zZo1Uew7J/ft21dxjosuuiiKL7nkkiieM2dOFI8ZMyaKzzzzzCj2dee7ZE866aSKHC699NKKx1Apb63namsJD7WN7+b1+/g1kX0Nvfbaa1Hc1dUVxX5Ne6myY9jX0GmnnRbF73//+6vm6GtsxowZUbxt27aKHPy8fn8tfez/W/JrQfvr6q+DXys6ax9/zvJzVOuC5hMwAAAJMAADAJAAAzAAAAkwAAMAkAADMAAACdAF3QK+k8+vj1qvKVOmRLHv2kPn813P559/flPH8zUmST//+c+rxp5fI/fEE0+M4rVr11bdP6sb1Hfi4n+Udz77Tti8rmjfbZy1T94xp06dWvWYfp1m/+/McccdV5HD3r17o/jPf/5zFPsO4r/4i7+I4qeeeiqK/ev2XdJHHll5+2Xf3e3lXSffwez5Os9al9vn7ZX/91pt3W8+AQMAkAADMAAACTAAAwCQAAMwAAAJMAADAJAAAzAAAAkwv6UFfNv6d77znUSZoFOsXLkyis8555woXrBgQRQvXLgwilevXh3FWVPbpk+fHsXjxo2L4g0bNkTxb37zmyh+5JFHKo5Zzi+M/9WvfjU3B/yP8ukveVNj/NQaf+MMSdqzZ08U+2ll/hj+Zgp+eqOfMuRvzvDiiy9W5OBv6PD8889H8Uc+8pEoXr9+fRTv2rUriv20Jv9vqZ8qJUkTJkyI4smTJ0exnxrnr9OOHTui2E/X8u9VtWlEh/j3s/x1MA0JAIAOwwAMAEACDMAAACTAAAwAQAK5A7CZzTGzZWa2ysxWmtnnS49PM7OlZra29Hflop1Ag6g7FI2aQ9Fq6YI+IOnaEMITZjZJ0h/NbKmkT0p6JIRwu5ldL+l6Sde1L9XD18UXXxzFn/zkJ9Mk0lmGdd197nOfqxrnmT9/ftW4EWeddVYU53VB33bbbVH8qU99qukcOlxLa66emzH4ruesmzH4bl9/DN/t6zuK/Y0Otm/fHsW+w9l31UvSnDlzonjFihVR3N/fXzV+97vfHcU9PT1R7DuYs7rBfTe375T2r9Pz19Fv798rH2fxN3gov7FFtf1zPwGHEDaFEJ4o/bxT0nOSjpZ0kaT7S5vdL+ni7CMA9aPuUDRqDkWr63fAZjZX0nxJyyXNDCFsKj31qqSZLc0MKKHuUDRqDkWoeQA2s4mSfizpCyGEaCZzGPwuJHO2sZktNLM+M+sbGBhoKlmMPNQditaKmvOLWgBZahqAzWy0Bgvy+yGEn5Qe3mxms0rPz5LUn7VvCOHuEEJvCKHXf98PVEPdoWitqjlWCEMtaumCNkn3SHouhPCtsqeWSLqy9POVkha3Pj2MVNQdikbNoWi1dEH/paRPSHrGzA61vN0o6XZJPzSzT0t6WdKl7Unx8OfXZM3q/BuBqLsm+fWkv/GNb0TxzJnxrzKvueaaKL7iiivak1jnamnNveMd//P55sCBA0M+J1V20Wbxx5g0aVIU+7WgfRez75pet25dFPsO5G3btlXk8Oqrr0axX5/6v//7v6PYdyifccYZUew7kH2H8vjx4ytyOPLIeBaYX0/a/3vqX1d5h7JUeV29rLWcfce53ybvmG/nkrdBCOFRSUP1Uf9VTWcB6kTdoWjUHIrGSlgAACTAAAwAQAIMwAAAJMAADABAArV0QSPHV77ylbq292uZXnddxy1ljGHId14uXhzPlnn44Yej+JRTToniq666Koqz1iNG7co7Y33Xs+efz+q8zeve9Z25fjaFXxu6u7u76jl9N7FUuX74xo0bo9jXzHvf+94onj17dhT71+DXTfad3ZK0Y8eOisfqUUvHebms9y4v77z3++3t6soEAAC0BAMwAAAJMAADAJAAAzAAAAkwAAMAkABd0C3w+OOP17W971acP39+K9MpxLXXXlvx2De/+c0EmeCQRx99NIpvu+22KJ41a1YUf/vb345i7uDTWuWdsb7z1nfN+vWKszrQfZeyP4Zfl9kfw5/Dd+pu3rw5infv3l2Rg6+h97znPVG8Zs2aKPbrVfu1nz3f4ezXs5YqO6P9etH+dfpubt/BvG/fvij23eb+ukqV/6349ajLr21WR/vb2w35DAAAaBsGYAAAEmAABgAgAQZgAAASYAAGACABuqDREDqeO88tt9wSxa+//noUn3rqqVF89tlntzulEa2etaB9p25W56zvevZrO/vZFf799zls3bq16vGnTp1akYPvMPYdxTNmzIhiv+69397zHc1Z18Fv41+Xv5b+dfnucN/17OOs6+C7uf0+1c5fjk/AAAAkwAAMAEACDMAAACTAAAwAQAIMwAAAJJA7AJvZHDNbZmarzGylmX2+9Pg/m9lGM1tR+nNh+9PFSEHdoWjUHIpWyzSkA5KuDSE8YWaTJP3RzJaWnrsjhPAv7UsPIxh1l2PLli1RvGHDhqrbn3feee1M53DQtprzU4T8NBY/lcZPMco6hudvCOCnv/h62b59exSPGTMmiidOnFhxju7u7ij202/8NCR/Ewp/IwX/mvw5s6Yh+WlE/mYK/vm8KWB+apS/Dv66Zp1z7NixVXMYSu4AHELYJGlT6eedZvacpKNrOjrQIOoORaPmULS6fgdsZnMlzZe0vPTQ1Wb2tJnda2ZHtjg3QBJ1h+JRcyhCzQOwmU2U9GNJXwgh7JD0XUnHSzpNg//XmLk0kpktNLM+M+sbGBhoQcoYSag7FK0VNedXmQKy1DQAm9loDRbk90MIP5GkEMLmEMLBEMJbkv5V0hlZ+4YQ7g4h9IYQent6elqVN0YA6g5Fa1XNTZs2rbikMWzV0gVtku6R9FwI4Vtlj88q2+wSSc+2Pj2MVNQdikbNoWi1dEH/paRPSHrGzFaUHrtR0uVmdpqkIGm9pKvakuEw8Nhjj6VO4XBE3eXwHamrV69OlMlho201529i4DuUfSdu1gL+eTcyyLupwKxZs6LY14/vSB43blzFOfxjvsvZdwz7LmZ/Hfzze/bsqTin57ua/TH8tcvb3ncs+xyzuqjzbhqR1b2dpZYu6EclZd3O4Zc1nQFoAHWHolFzKBorYQEAkAADMAAACTAAAwCQAAMwAAAJ1NIFDQBogl/7Oa/z1q9HLFWuP5zXkZy3DrPv7vVd0D4nqbIT2+fk+dfpr4M/p3/dWZ3f/pxdXV1Vn/fXKW8dbt897q9j1mP+dZZf62od0XwCBgAgAQZgAAASYAAGACABBmAAABJgAAYAIAG6oAGgzXyXrO+M9Z23vlNXquwYzlsb2nf3Tpgwoeo5/PGycvCvw5+j3jWufae17y7O6iDO66z21zLvOnlZ3d95fA7lndlZ63ofwidgAAASYAAGACABBmAAABJgAAYAIAEGYAAAEqALGgDaoLz7tVonrFTZRZu1ve/m9esm+45h36G8d+/eKK62frFUWzfw5MmTo3j37t1R7DuSfZzXNe3XeZay12Yu56+DP0feOs4+zjqfP0e19Z6r4RMwAAAJMAADAJAAAzAAAAkwAAMAkAADMAAACeQOwGY21sx+b2ZPmdlKM/ta6fFjzWy5mb1gZv9mZpXtakCDqDsUjZpD0WqZhrRP0rkhhF1mNlrSo2b2sKQvSbojhPADM7tL0qclfbeNuWJkoe5QtJbWXLXpMn5qjI8buQmB38efP28K0Lhx46LYT3OqJc+JEydWzTHvdfupU7VMAcqb4uWPkTctKW+aUy1qnZaU+wk4DNpVCkeX/gRJ50r6Uenx+yVdXH+aQDbqDkWj5lC0mn4HbGZHmNkKSf2Slkp6UdL2EMKhWdMbJB3dnhQxUlF3KBo1hyLVNACHEA6GEE6TNFvSGZJOqvUEZrbQzPrMrG9gYKDBNDESUXcoWqtq7rXXXmtbjjh81NUFHULYLmmZpA9Jmmpmh36pMFvSxiH2uTuE0BtC6O3p6WkqWYxM1B2K1mzNTZ8+vaBMMZzV0gXdY2ZTSz+Pk3S+pOc0WJwfL212paTF7UoSIw91h6JRcyhaLV3QsyTdb2ZHaHDA/mEI4SEzWyXpB2b2DUlPSrqnjXli5KHuULSW1lx513K9nbi+GziL77T1Xc7e6NGjozjvZgtZ3cA+r7wuZ5+Tv+GD77z2nd7+5gxZ56hXvTfG8Dll5ZW1TS1yB+AQwtOS5mc8vk6DvyMBWo66Q9GoORSNlbAAAEiAARgAgAQYgAEASIABGACABKzWNStbcjKzAUkvS+qWtKWwEzeGHFtjqByPCSEUMkGXumu54ZxjIXVHzbXccM5xyJordAB++6RmfSGE3sJPXAdybI1OyrGTchkKObZGp+TYKXlUQ46t0UiOfAUNAEACDMAAACSQagC+O9F560GOrdFJOXZSLkMhx9bolBw7JY9qyLE16s4xye+AAQAY6fgKGgCABAofgM3so2a22sxeMLPriz5/FjO718z6zezZssemmdlSM1tb+vvIhPnNMbNlZrbKzFaa2ec7LcdSPmPN7Pdm9lQpz6+VHj/WzJaX3vN/M7OugvOi5hrLsePrrlNrrpQDdddYjiOn7kIIhf2RdISkFyUdJ6lL0lOS5hWZwxB5fUTS6ZKeLXvs/0i6vvTz9ZL+d8L8Zkk6vfTzJElrJM3rpBxLOZikiaWfR0taLumDkn4o6bLS43dJ+gdqrrNrbrjUXSfWHHVH3dVad0Un/SFJ/14W3yDphpRvdlkuc11RrpY0q6wgVqfOsSy3xRq8V2kn5zhe0hOSPqDByemjsmqggDyoudbl29F11yk1l3VO6o66y/pT9FfQR0t6pSzeUHqsE80MIWwq/fyqpJkpkznEzOZq8JZpy9WBOZrZEWa2QlK/pKUa/BSwPYRw6AaaRb/n1FwLdHLddWDNSdRdSxzudUcTVg3C4P/OJG8XN7OJkn4s6QshhB3lz3VKjiGEgyGE0yTN1uA9VE9KnNKw1Cnvp9T5dUfNtU4nvJ+HjIS6K3oA3ihpTlk8u/RYJ9psZrMkqfR3f8pkzGy0Bovx+yGEn5Qe7qgcy4UQtktapsGvYaaa2ajSU0W/59RcE4ZT3XVQzUnUXVNGSt0VPQD/QdIJpU6xLkmXSVpScA61WiLpytLPV2rw9xBJmJlJukfScyGEb5U91TE5SpKZ9ZjZ1NLP4zT4e5vnNFicHy9tVnSe1FyDhkPddWjNSdRdw0ZU3SX4hfWFGuxqe1HSV1L/Ar2U04OSNknar8Hv7T8tabqkRyStlfRrSdMS5nemBr9ueVrSitKfCzspx1Kep0p6spTns5L+V+nx4yT9XtILkv6fpDHUXGfX3HCpu06tOeqOuqul7lgJCwCABGjCAgAgAQZgAAASYAAGACABBmAAABJgAAYAIAEGYAAAEmAABgAgAQZgAAAS+P8f6wCgNtcuWwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x576 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(8, 8))\n",
    "for targets in test_dataset.take(1):\n",
    "    trg = targets[\"img\"]\n",
    "    inv = flow.bijector.inverse(trg)\n",
    "    re_trg = flow.bijector.forward(inv)\n",
    "\n",
    "print('log probability: ', tf.reduce_mean(tfd.Normal(0., 1.).log_prob(inv)).numpy())\n",
    "print(\"inv mean: \", tf.reduce_mean(inv).numpy(), \" std: \", tf.math.reduce_std(inv).numpy())\n",
    "print(\"re:trg mean: \", tf.reduce_mean(re_trg).numpy(), \" std: \", tf.math.reduce_std(re_trg).numpy())\n",
    "\n",
    "re_inv = np.array([inv[0] - inv[1] / 2])\n",
    "print(re_inv.shape)\n",
    "re_re_trg = flow.bijector.forward(re_inv)\n",
    "\n",
    "ax = fig.add_subplot(2, 3, 1)\n",
    "ax.imshow(tf.squeeze(trg[0], axis=-1), aspect=\"auto\", cmap=\"gray_r\")\n",
    "\n",
    "ax = fig.add_subplot(2, 3, 4)\n",
    "ax.imshow(tf.squeeze(re_trg[0], axis=-1), aspect=\"auto\", cmap=\"gray_r\")\n",
    "\n",
    "ax = fig.add_subplot(2, 3, 2)\n",
    "ax.imshow(tf.squeeze(trg[1], axis=-1), aspect=\"auto\", cmap=\"gray_r\")\n",
    "\n",
    "ax = fig.add_subplot(2, 3, 5)\n",
    "ax.imshow(tf.squeeze(re_trg[1], axis=-1), aspect=\"auto\", cmap=\"gray_r\")\n",
    "\n",
    "ax = fig.add_subplot(2, 3, 3)\n",
    "ax.imshow(tf.squeeze(re_inv[0], axis=-1), aspect=\"auto\", cmap=\"gray_r\")\n",
    "\n",
    "ax = fig.add_subplot(2, 3, 6)\n",
    "ax.imshow(tf.squeeze(re_re_trg[0], axis=-1), aspect=\"auto\", cmap=\"gray_r\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}