C1_W3_Lab_2_custom-dense-layer (1).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungraded Lab: Building a Custom Dense Layer\n",
    "\n",
    "In this lab, we'll walk through how to create a custom layer that inherits the [Layer](https://keras.io/api/layers/base_layer/#layer-class) class. Unlike simple Lambda layers you did previously, the custom layer here will contain weights that can be updated during training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DpioxwFXD9Is"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Layer with weights\n",
    "\n",
    "To make custom layer that is trainable, we need to define a class that inherits the [Layer](https://keras.io/api/layers/base_layer/#layer-class) base class from Keras. The Python syntax is shown below in the class declaration. This class requires three functions: `__init__()`, `build()` and `call()`. These ensure that our custom layer has a *state* and *computation* that can be accessed during training or inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jnVrzRT6BPWl"
   },
   "outputs": [],
   "source": [
    "# inherit from this base class\n",
    "from tensorflow.keras.layers import Layer\n",
    "\n",
    "class SimpleDense(Layer):\n",
    "\n",
    "    def __init__(self, units=32):\n",
    "        '''Initializes the instance attributes'''\n",
    "        super(SimpleDense, self).__init__()\n",
    "        self.units = units\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        '''Create the state of the layer (weights)'''\n",
    "        # initialize the weights\n",
    "        w_init = tf.random_normal_initializer()\n",
    "        self.w = tf.Variable(name=\"kernel\",\n",
    "            initial_value=w_init(shape=(input_shape[-1], self.units),\n",
    "                                 dtype='float32'),\n",
    "            trainable=True)\n",
    "\n",
    "        # initialize the biases\n",
    "        b_init = tf.zeros_initializer()\n",
    "        self.b = tf.Variable(name=\"bias\",\n",
    "            initial_value=b_init(shape=(self.units,), dtype='float32'),\n",
    "            trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        '''Defines the computation from inputs to outputs'''\n",
    "        return tf.matmul(inputs, self.w) + self.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use our custom layer like below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Variable 'simple_dense_10/kernel:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[-0.00811401,  0.00676195],\n",
      "       [ 0.10125976, -0.02970896]], dtype=float32)>, <tf.Variable 'simple_dense_10/bias:0' shape=(2,) dtype=float32, numpy=array([0., 0.], dtype=float32)>]\n"
     ]
    }
   ],
   "source": [
    "# declare an instance of the class\n",
    "my_dense = SimpleDense(units=2)\n",
    "\n",
    "# define an input and feed into the layer\n",
    "x = tf.ones((1, 2))\n",
    "y = my_dense(x)\n",
    "\n",
    "# parameters of the base Layer class like `variables` can be used\n",
    "print(my_dense.variables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's then try using it in simple network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 2), dtype=float32, numpy=array([[1., 1.]], dtype=float32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.concatenate?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.,  0.],\n",
       "       [ 1.,  2.],\n",
       "       [ 3.,  4.],\n",
       "       [-2., -2.],\n",
       "       [-2., -2.],\n",
       "       [-2., -2.]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define the dataset\n",
    "xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
    "xs2 = np.array([-2.0, -2.0, -2.0, -2.0, -2.0, -2.0], dtype=float)\n",
    "x = np.concatenate( (xs, xs2), axis=0).reshape((-1,2)) \n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-3.],\n",
       "       [-1.],\n",
       "       [ 1.],\n",
       "       [ 3.],\n",
       "       [ 5.],\n",
       "       [ 7.]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)\n",
    "ys = np.expand_dims(ys, axis=1)\n",
    "ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WwTPJT4DkTyW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 6 samples\n",
      "Epoch 1/500\n",
      "6/6 [==============================] - 0s 23ms/sample - loss: 16.2816\n",
      "Epoch 2/500\n",
      "6/6 [==============================] - 0s 182us/sample - loss: 14.4972\n",
      "Epoch 3/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 13.2626\n",
      "Epoch 4/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 12.3991\n",
      "Epoch 5/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 11.7867\n",
      "Epoch 6/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 11.3443\n",
      "Epoch 7/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 11.0176\n",
      "Epoch 8/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 10.7699\n",
      "Epoch 9/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 10.5764\n",
      "Epoch 10/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 10.4205\n",
      "Epoch 11/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 10.2908\n",
      "Epoch 12/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 10.1799\n",
      "Epoch 13/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 10.0824\n",
      "Epoch 14/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 9.9948\n",
      "Epoch 15/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 9.9149\n",
      "Epoch 16/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 9.8408\n",
      "Epoch 17/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 9.7715\n",
      "Epoch 18/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 9.7061\n",
      "Epoch 19/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 9.6440\n",
      "Epoch 20/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 9.5849\n",
      "Epoch 21/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 9.5285\n",
      "Epoch 22/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 9.4744\n",
      "Epoch 23/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 9.4225\n",
      "Epoch 24/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 9.3727\n",
      "Epoch 25/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 9.3247\n",
      "Epoch 26/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 9.2786\n",
      "Epoch 27/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 9.2342\n",
      "Epoch 28/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 9.1914\n",
      "Epoch 29/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 9.1501\n",
      "Epoch 30/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 9.1103\n",
      "Epoch 31/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 9.0719\n",
      "Epoch 32/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 9.0348\n",
      "Epoch 33/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 8.9990\n",
      "Epoch 34/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 8.9644\n",
      "Epoch 35/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.9310\n",
      "Epoch 36/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 8.8986\n",
      "Epoch 37/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 8.8674\n",
      "Epoch 38/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 8.8371\n",
      "Epoch 39/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 8.8078\n",
      "Epoch 40/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 8.7794\n",
      "Epoch 41/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 8.7519\n",
      "Epoch 42/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 8.7252\n",
      "Epoch 43/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 8.6993\n",
      "Epoch 44/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 8.6742\n",
      "Epoch 45/500\n",
      "6/6 [==============================] - 0s 720us/sample - loss: 8.6499\n",
      "Epoch 46/500\n",
      "6/6 [==============================] - 0s 154us/sample - loss: 8.6262\n",
      "Epoch 47/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.6032\n",
      "Epoch 48/500\n",
      "6/6 [==============================] - 0s 163us/sample - loss: 8.5808\n",
      "Epoch 49/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 8.5591\n",
      "Epoch 50/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 8.5379\n",
      "Epoch 51/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 8.5173\n",
      "Epoch 52/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 8.4973\n",
      "Epoch 53/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 8.4777\n",
      "Epoch 54/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 8.4586\n",
      "Epoch 55/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 8.4400\n",
      "Epoch 56/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 8.4218\n",
      "Epoch 57/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 8.4041\n",
      "Epoch 58/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.3868\n",
      "Epoch 59/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 8.3698\n",
      "Epoch 60/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 8.3533\n",
      "Epoch 61/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 8.3371\n",
      "Epoch 62/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 8.3212\n",
      "Epoch 63/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.3056\n",
      "Epoch 64/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.2904\n",
      "Epoch 65/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 8.2755\n",
      "Epoch 66/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 8.2608\n",
      "Epoch 67/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.2464\n",
      "Epoch 68/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 8.2323\n",
      "Epoch 69/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 8.2185\n",
      "Epoch 70/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 8.2048\n",
      "Epoch 71/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.1914\n",
      "Epoch 72/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 8.1782\n",
      "Epoch 73/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 8.1653\n",
      "Epoch 74/500\n",
      "6/6 [==============================] - 0s 162us/sample - loss: 8.1525\n",
      "Epoch 75/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.1399\n",
      "Epoch 76/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.1275\n",
      "Epoch 77/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 8.1153\n",
      "Epoch 78/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.1033\n",
      "Epoch 79/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 8.0914\n",
      "Epoch 80/500\n",
      "6/6 [==============================] - 0s 132us/sample - loss: 8.0796\n",
      "Epoch 81/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 8.0680\n",
      "Epoch 82/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 8.0566\n",
      "Epoch 83/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 8.0453\n",
      "Epoch 84/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.0341\n",
      "Epoch 85/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 8.0231\n",
      "Epoch 86/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.0121\n",
      "Epoch 87/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 8.0013\n",
      "Epoch 88/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.9906\n",
      "Epoch 89/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.9800\n",
      "Epoch 90/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 7.9695\n",
      "Epoch 91/500\n",
      "6/6 [==============================] - 0s 132us/sample - loss: 7.9591\n",
      "Epoch 92/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 7.9488\n",
      "Epoch 93/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.9386\n",
      "Epoch 94/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 7.9284\n",
      "Epoch 95/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 7.9184\n",
      "Epoch 96/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.9084\n",
      "Epoch 97/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.8985\n",
      "Epoch 98/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 7.8887\n",
      "Epoch 99/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 7.8790\n",
      "Epoch 100/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.8693\n",
      "Epoch 101/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 7.8597\n",
      "Epoch 102/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.8501\n",
      "Epoch 103/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.8406\n",
      "Epoch 104/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.8312\n",
      "Epoch 105/500\n",
      "6/6 [==============================] - 0s 269us/sample - loss: 7.8218\n",
      "Epoch 106/500\n",
      "6/6 [==============================] - 0s 280us/sample - loss: 7.8125\n",
      "Epoch 107/500\n",
      "6/6 [==============================] - 0s 164us/sample - loss: 7.8032\n",
      "Epoch 108/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 7.7940\n",
      "Epoch 109/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 7.7848\n",
      "Epoch 110/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.7757\n",
      "Epoch 111/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.7666\n",
      "Epoch 112/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.7576\n",
      "Epoch 113/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.7486\n",
      "Epoch 114/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 7.7396\n",
      "Epoch 115/500\n",
      "6/6 [==============================] - 0s 154us/sample - loss: 7.7307\n",
      "Epoch 116/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.7218\n",
      "Epoch 117/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.7130\n",
      "Epoch 118/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.7042\n",
      "Epoch 119/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.6954\n",
      "Epoch 120/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 7.6867\n",
      "Epoch 121/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.6780\n",
      "Epoch 122/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 7.6693\n",
      "Epoch 123/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 7.6607\n",
      "Epoch 124/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.6521\n",
      "Epoch 125/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 7.6435\n",
      "Epoch 126/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.6349\n",
      "Epoch 127/500\n",
      "6/6 [==============================] - 0s 131us/sample - loss: 7.6264\n",
      "Epoch 128/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.6179\n",
      "Epoch 129/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 7.6094\n",
      "Epoch 130/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.6010\n",
      "Epoch 131/500\n",
      "6/6 [==============================] - 0s 132us/sample - loss: 7.5925\n",
      "Epoch 132/500\n",
      "6/6 [==============================] - 0s 128us/sample - loss: 7.5841\n",
      "Epoch 133/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.5757\n",
      "Epoch 134/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.5674\n",
      "Epoch 135/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.5590\n",
      "Epoch 136/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.5507\n",
      "Epoch 137/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.5424\n",
      "Epoch 138/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.5342\n",
      "Epoch 139/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.5259\n",
      "Epoch 140/500\n",
      "6/6 [==============================] - 0s 132us/sample - loss: 7.5177\n",
      "Epoch 141/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 7.5095\n",
      "Epoch 142/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 7.5013\n",
      "Epoch 143/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 7.4931\n",
      "Epoch 144/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.4849\n",
      "Epoch 145/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 7.4768\n",
      "Epoch 146/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 7.4686\n",
      "Epoch 147/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 7.4605\n",
      "Epoch 148/500\n",
      "6/6 [==============================] - 0s 174us/sample - loss: 7.4524\n",
      "Epoch 149/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 7.4444\n",
      "Epoch 150/500\n",
      "6/6 [==============================] - 0s 130us/sample - loss: 7.4363\n",
      "Epoch 151/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.4283\n",
      "Epoch 152/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.4202\n",
      "Epoch 153/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.4122\n",
      "Epoch 154/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.4042\n",
      "Epoch 155/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 7.3962\n",
      "Epoch 156/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 7.3883\n",
      "Epoch 157/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.3803\n",
      "Epoch 158/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.3724\n",
      "Epoch 159/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.3644\n",
      "Epoch 160/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 7.3565\n",
      "Epoch 161/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 7.3486\n",
      "Epoch 162/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 7.3407\n",
      "Epoch 163/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.3329\n",
      "Epoch 164/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 7.3250\n",
      "Epoch 165/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.3172\n",
      "Epoch 166/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 7.3093\n",
      "Epoch 167/500\n",
      "6/6 [==============================] - 0s 242us/sample - loss: 7.3015\n",
      "Epoch 168/500\n",
      "6/6 [==============================] - 0s 182us/sample - loss: 7.2937\n",
      "Epoch 169/500\n",
      "6/6 [==============================] - 0s 167us/sample - loss: 7.2859\n",
      "Epoch 170/500\n",
      "6/6 [==============================] - 0s 186us/sample - loss: 7.2781\n",
      "Epoch 171/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 7.2703\n",
      "Epoch 172/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 7.2626\n",
      "Epoch 173/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 7.2548\n",
      "Epoch 174/500\n",
      "6/6 [==============================] - 0s 157us/sample - loss: 7.2471\n",
      "Epoch 175/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 7.2394\n",
      "Epoch 176/500\n",
      "6/6 [==============================] - 0s 170us/sample - loss: 7.2317\n",
      "Epoch 177/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 7.2240\n",
      "Epoch 178/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 7.2163\n",
      "Epoch 179/500\n",
      "6/6 [==============================] - 0s 161us/sample - loss: 7.2086\n",
      "Epoch 180/500\n",
      "6/6 [==============================] - 0s 155us/sample - loss: 7.2009\n",
      "Epoch 181/500\n",
      "6/6 [==============================] - 0s 157us/sample - loss: 7.1933\n",
      "Epoch 182/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 7.1856\n",
      "Epoch 183/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 7.1780\n",
      "Epoch 184/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 7.1704\n",
      "Epoch 185/500\n",
      "6/6 [==============================] - 0s 175us/sample - loss: 7.1628\n",
      "Epoch 186/500\n",
      "6/6 [==============================] - 0s 170us/sample - loss: 7.1552\n",
      "Epoch 187/500\n",
      "6/6 [==============================] - 0s 158us/sample - loss: 7.1476\n",
      "Epoch 188/500\n",
      "6/6 [==============================] - 0s 167us/sample - loss: 7.1400\n",
      "Epoch 189/500\n",
      "6/6 [==============================] - 0s 156us/sample - loss: 7.1324\n",
      "Epoch 190/500\n",
      "6/6 [==============================] - 0s 177us/sample - loss: 7.1249\n",
      "Epoch 191/500\n",
      "6/6 [==============================] - 0s 154us/sample - loss: 7.1173\n",
      "Epoch 192/500\n",
      "6/6 [==============================] - 0s 189us/sample - loss: 7.1098\n",
      "Epoch 193/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 7.1023\n",
      "Epoch 194/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 7.0948\n",
      "Epoch 195/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 7.0873\n",
      "Epoch 196/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 7.0798\n",
      "Epoch 197/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 7.0723\n",
      "Epoch 198/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 7.0648\n",
      "Epoch 199/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 7.0573\n",
      "Epoch 200/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 7.0499\n",
      "Epoch 201/500\n",
      "6/6 [==============================] - 0s 161us/sample - loss: 7.0424\n",
      "Epoch 202/500\n",
      "6/6 [==============================] - 0s 155us/sample - loss: 7.0350\n",
      "Epoch 203/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 7.0276\n",
      "Epoch 204/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 7.0202\n",
      "Epoch 205/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 7.0128\n",
      "Epoch 206/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 7.0054\n",
      "Epoch 207/500\n",
      "6/6 [==============================] - 0s 167us/sample - loss: 6.9980\n",
      "Epoch 208/500\n",
      "6/6 [==============================] - 0s 173us/sample - loss: 6.9906\n",
      "Epoch 209/500\n",
      "6/6 [==============================] - 0s 183us/sample - loss: 6.9833\n",
      "Epoch 210/500\n",
      "6/6 [==============================] - 0s 130us/sample - loss: 6.9759\n",
      "Epoch 211/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 6.9686\n",
      "Epoch 212/500\n",
      "6/6 [==============================] - 0s 161us/sample - loss: 6.9612\n",
      "Epoch 213/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 6.9539\n",
      "Epoch 214/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 6.9466\n",
      "Epoch 215/500\n",
      "6/6 [==============================] - 0s 247us/sample - loss: 6.9393\n",
      "Epoch 216/500\n",
      "6/6 [==============================] - 0s 156us/sample - loss: 6.9320\n",
      "Epoch 217/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 6.9247\n",
      "Epoch 218/500\n",
      "6/6 [==============================] - 0s 180us/sample - loss: 6.9175\n",
      "Epoch 219/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 6.9102\n",
      "Epoch 220/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 6.9029\n",
      "Epoch 221/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.8957\n",
      "Epoch 222/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.8885\n",
      "Epoch 223/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.8812\n",
      "Epoch 224/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.8740\n",
      "Epoch 225/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 6.8668\n",
      "Epoch 226/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.8596\n",
      "Epoch 227/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.8524\n",
      "Epoch 228/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.8453\n",
      "Epoch 229/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 6.8381\n",
      "Epoch 230/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 6.8309\n",
      "Epoch 231/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.8238\n",
      "Epoch 232/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.8166\n",
      "Epoch 233/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 6.8095\n",
      "Epoch 234/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 6.8024\n",
      "Epoch 235/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 6.7953\n",
      "Epoch 236/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.7882\n",
      "Epoch 237/500\n",
      "6/6 [==============================] - 0s 134us/sample - loss: 6.7811\n",
      "Epoch 238/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 6.7740\n",
      "Epoch 239/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.7669\n",
      "Epoch 240/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 6.7599\n",
      "Epoch 241/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 6.7528\n",
      "Epoch 242/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 6.7458\n",
      "Epoch 243/500\n",
      "6/6 [==============================] - 0s 171us/sample - loss: 6.7387\n",
      "Epoch 244/500\n",
      "6/6 [==============================] - 0s 178us/sample - loss: 6.7317\n",
      "Epoch 245/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 6.7247\n",
      "Epoch 246/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.7177\n",
      "Epoch 247/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 6.7107\n",
      "Epoch 248/500\n",
      "6/6 [==============================] - 0s 132us/sample - loss: 6.7037\n",
      "Epoch 249/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.6967\n",
      "Epoch 250/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 6.6897\n",
      "Epoch 251/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 6.6828\n",
      "Epoch 252/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.6758\n",
      "Epoch 253/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.6689\n",
      "Epoch 254/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.6619\n",
      "Epoch 255/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.6550\n",
      "Epoch 256/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 6.6481\n",
      "Epoch 257/500\n",
      "6/6 [==============================] - 0s 335us/sample - loss: 6.6412\n",
      "Epoch 258/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 6.6343\n",
      "Epoch 259/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.6274\n",
      "Epoch 260/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 6.6205\n",
      "Epoch 261/500\n",
      "6/6 [==============================] - 0s 156us/sample - loss: 6.6136\n",
      "Epoch 262/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.6067\n",
      "Epoch 263/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 6.5999\n",
      "Epoch 264/500\n",
      "6/6 [==============================] - 0s 159us/sample - loss: 6.5930\n",
      "Epoch 265/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.5862\n",
      "Epoch 266/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 6.5794\n",
      "Epoch 267/500\n",
      "6/6 [==============================] - 0s 159us/sample - loss: 6.5726\n",
      "Epoch 268/500\n",
      "6/6 [==============================] - 0s 157us/sample - loss: 6.5658\n",
      "Epoch 269/500\n",
      "6/6 [==============================] - 0s 165us/sample - loss: 6.5589\n",
      "Epoch 270/500\n",
      "6/6 [==============================] - 0s 329us/sample - loss: 6.5522\n",
      "Epoch 271/500\n",
      "6/6 [==============================] - 0s 300us/sample - loss: 6.5454\n",
      "Epoch 272/500\n",
      "6/6 [==============================] - 0s 217us/sample - loss: 6.5386\n",
      "Epoch 273/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 6.5318\n",
      "Epoch 274/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 6.5251\n",
      "Epoch 275/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 6.5183\n",
      "Epoch 276/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 6.5116\n",
      "Epoch 277/500\n",
      "6/6 [==============================] - 0s 226us/sample - loss: 6.5048\n",
      "Epoch 278/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.4981\n",
      "Epoch 279/500\n",
      "6/6 [==============================] - 0s 128us/sample - loss: 6.4914\n",
      "Epoch 280/500\n",
      "6/6 [==============================] - 0s 155us/sample - loss: 6.4847\n",
      "Epoch 281/500\n",
      "6/6 [==============================] - 0s 176us/sample - loss: 6.4780\n",
      "Epoch 282/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 6.4713\n",
      "Epoch 283/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.4646\n",
      "Epoch 284/500\n",
      "6/6 [==============================] - 0s 175us/sample - loss: 6.4580\n",
      "Epoch 285/500\n",
      "6/6 [==============================] - 0s 185us/sample - loss: 6.4513\n",
      "Epoch 286/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 6.4447\n",
      "Epoch 287/500\n",
      "6/6 [==============================] - 0s 156us/sample - loss: 6.4380\n",
      "Epoch 288/500\n",
      "6/6 [==============================] - 0s 156us/sample - loss: 6.4314\n",
      "Epoch 289/500\n",
      "6/6 [==============================] - 0s 176us/sample - loss: 6.4248\n",
      "Epoch 290/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 6.4181\n",
      "Epoch 291/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 6.4115\n",
      "Epoch 292/500\n",
      "6/6 [==============================] - 0s 163us/sample - loss: 6.4049\n",
      "Epoch 293/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 6.3983\n",
      "Epoch 294/500\n",
      "6/6 [==============================] - 0s 247us/sample - loss: 6.3917\n",
      "Epoch 295/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 6.3852\n",
      "Epoch 296/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 6.3786\n",
      "Epoch 297/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 6.3720\n",
      "Epoch 298/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 6.3655\n",
      "Epoch 299/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 6.3590\n",
      "Epoch 300/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 6.3524\n",
      "Epoch 301/500\n",
      "6/6 [==============================] - 0s 181us/sample - loss: 6.3459\n",
      "Epoch 302/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 6.3394\n",
      "Epoch 303/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 6.3329\n",
      "Epoch 304/500\n",
      "6/6 [==============================] - 0s 156us/sample - loss: 6.3264\n",
      "Epoch 305/500\n",
      "6/6 [==============================] - 0s 197us/sample - loss: 6.3199\n",
      "Epoch 306/500\n",
      "6/6 [==============================] - 0s 176us/sample - loss: 6.3134\n",
      "Epoch 307/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 6.3069\n",
      "Epoch 308/500\n",
      "6/6 [==============================] - 0s 174us/sample - loss: 6.3005\n",
      "Epoch 309/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 6.2940\n",
      "Epoch 310/500\n",
      "6/6 [==============================] - 0s 179us/sample - loss: 6.2876\n",
      "Epoch 311/500\n",
      "6/6 [==============================] - 0s 188us/sample - loss: 6.2811\n",
      "Epoch 312/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.2747\n",
      "Epoch 313/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 6.2683\n",
      "Epoch 314/500\n",
      "6/6 [==============================] - 0s 176us/sample - loss: 6.2618\n",
      "Epoch 315/500\n",
      "6/6 [==============================] - 0s 178us/sample - loss: 6.2554\n",
      "Epoch 316/500\n",
      "6/6 [==============================] - 0s 305us/sample - loss: 6.2490\n",
      "Epoch 317/500\n",
      "6/6 [==============================] - 0s 340us/sample - loss: 6.2427\n",
      "Epoch 318/500\n",
      "6/6 [==============================] - 0s 204us/sample - loss: 6.2363\n",
      "Epoch 319/500\n",
      "6/6 [==============================] - 0s 343us/sample - loss: 6.2299\n",
      "Epoch 320/500\n",
      "6/6 [==============================] - 0s 172us/sample - loss: 6.2235\n",
      "Epoch 321/500\n",
      "6/6 [==============================] - 0s 192us/sample - loss: 6.2172\n",
      "Epoch 322/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 6.2108\n",
      "Epoch 323/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.2045\n",
      "Epoch 324/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 6.1982\n",
      "Epoch 325/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 6.1918\n",
      "Epoch 326/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.1855\n",
      "Epoch 327/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 6.1792\n",
      "Epoch 328/500\n",
      "6/6 [==============================] - 0s 177us/sample - loss: 6.1729\n",
      "Epoch 329/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 6.1666\n",
      "Epoch 330/500\n",
      "6/6 [==============================] - 0s 182us/sample - loss: 6.1603\n",
      "Epoch 331/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 6.1541\n",
      "Epoch 332/500\n",
      "6/6 [==============================] - 0s 132us/sample - loss: 6.1478\n",
      "Epoch 333/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 6.1415\n",
      "Epoch 334/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.1353\n",
      "Epoch 335/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 6.1290\n",
      "Epoch 336/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 6.1228\n",
      "Epoch 337/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 6.1166\n",
      "Epoch 338/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.1104\n",
      "Epoch 339/500\n",
      "6/6 [==============================] - 0s 183us/sample - loss: 6.1042\n",
      "Epoch 340/500\n",
      "6/6 [==============================] - 0s 182us/sample - loss: 6.0980\n",
      "Epoch 341/500\n",
      "6/6 [==============================] - 0s 161us/sample - loss: 6.0918\n",
      "Epoch 342/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 6.0856\n",
      "Epoch 343/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 6.0794\n",
      "Epoch 344/500\n",
      "6/6 [==============================] - 0s 162us/sample - loss: 6.0732\n",
      "Epoch 345/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 6.0671\n",
      "Epoch 346/500\n",
      "6/6 [==============================] - 0s 131us/sample - loss: 6.0609\n",
      "Epoch 347/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 6.0548\n",
      "Epoch 348/500\n",
      "6/6 [==============================] - 0s 224us/sample - loss: 6.0486\n",
      "Epoch 349/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 6.0425\n",
      "Epoch 350/500\n",
      "6/6 [==============================] - 0s 203us/sample - loss: 6.0364\n",
      "Epoch 351/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 6.0303\n",
      "Epoch 352/500\n",
      "6/6 [==============================] - 0s 159us/sample - loss: 6.0242\n",
      "Epoch 353/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 6.0181\n",
      "Epoch 354/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 6.0120\n",
      "Epoch 355/500\n",
      "6/6 [==============================] - 0s 157us/sample - loss: 6.0059\n",
      "Epoch 356/500\n",
      "6/6 [==============================] - 0s 158us/sample - loss: 5.9998\n",
      "Epoch 357/500\n",
      "6/6 [==============================] - 0s 226us/sample - loss: 5.9938\n",
      "Epoch 358/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.9877\n",
      "Epoch 359/500\n",
      "6/6 [==============================] - 0s 185us/sample - loss: 5.9816\n",
      "Epoch 360/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 5.9756\n",
      "Epoch 361/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 5.9696\n",
      "Epoch 362/500\n",
      "6/6 [==============================] - 0s 336us/sample - loss: 5.9635\n",
      "Epoch 363/500\n",
      "6/6 [==============================] - 0s 268us/sample - loss: 5.9575\n",
      "Epoch 364/500\n",
      "6/6 [==============================] - 0s 213us/sample - loss: 5.9515\n",
      "Epoch 365/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.9455\n",
      "Epoch 366/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 5.9395\n",
      "Epoch 367/500\n",
      "6/6 [==============================] - 0s 257us/sample - loss: 5.9335\n",
      "Epoch 368/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 5.9275\n",
      "Epoch 369/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 5.9216\n",
      "Epoch 370/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 5.9156\n",
      "Epoch 371/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 5.9097\n",
      "Epoch 372/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 5.9037\n",
      "Epoch 373/500\n",
      "6/6 [==============================] - 0s 157us/sample - loss: 5.8978\n",
      "Epoch 374/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.8918\n",
      "Epoch 375/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 5.8859\n",
      "Epoch 376/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.8800\n",
      "Epoch 377/500\n",
      "6/6 [==============================] - 0s 150us/sample - loss: 5.8741\n",
      "Epoch 378/500\n",
      "6/6 [==============================] - 0s 154us/sample - loss: 5.8682\n",
      "Epoch 379/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 5.8623\n",
      "Epoch 380/500\n",
      "6/6 [==============================] - 0s 184us/sample - loss: 5.8564\n",
      "Epoch 381/500\n",
      "6/6 [==============================] - 0s 127us/sample - loss: 5.8505\n",
      "Epoch 382/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 5.8446\n",
      "Epoch 383/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.8388\n",
      "Epoch 384/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 5.8329\n",
      "Epoch 385/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.8271\n",
      "Epoch 386/500\n",
      "6/6 [==============================] - 0s 187us/sample - loss: 5.8212\n",
      "Epoch 387/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 5.8154\n",
      "Epoch 388/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.8096\n",
      "Epoch 389/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.8037\n",
      "Epoch 390/500\n",
      "6/6 [==============================] - 0s 189us/sample - loss: 5.7979\n",
      "Epoch 391/500\n",
      "6/6 [==============================] - 0s 128us/sample - loss: 5.7921\n",
      "Epoch 392/500\n",
      "6/6 [==============================] - 0s 196us/sample - loss: 5.7863\n",
      "Epoch 393/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 5.7806\n",
      "Epoch 394/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.7748\n",
      "Epoch 395/500\n",
      "6/6 [==============================] - 0s 166us/sample - loss: 5.7690\n",
      "Epoch 396/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 5.7632\n",
      "Epoch 397/500\n",
      "6/6 [==============================] - 0s 161us/sample - loss: 5.7575\n",
      "Epoch 398/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 5.7517\n",
      "Epoch 399/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.7460\n",
      "Epoch 400/500\n",
      "6/6 [==============================] - 0s 160us/sample - loss: 5.7402\n",
      "Epoch 401/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.7345\n",
      "Epoch 402/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 5.7288\n",
      "Epoch 403/500\n",
      "6/6 [==============================] - 0s 130us/sample - loss: 5.7231\n",
      "Epoch 404/500\n",
      "6/6 [==============================] - 0s 154us/sample - loss: 5.7174\n",
      "Epoch 405/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.7117\n",
      "Epoch 406/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 5.7060\n",
      "Epoch 407/500\n",
      "6/6 [==============================] - 0s 296us/sample - loss: 5.7003\n",
      "Epoch 408/500\n",
      "6/6 [==============================] - 0s 178us/sample - loss: 5.6946\n",
      "Epoch 409/500\n",
      "6/6 [==============================] - 0s 209us/sample - loss: 5.6890\n",
      "Epoch 410/500\n",
      "6/6 [==============================] - 0s 236us/sample - loss: 5.6833\n",
      "Epoch 411/500\n",
      "6/6 [==============================] - 0s 201us/sample - loss: 5.6776\n",
      "Epoch 412/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.6720\n",
      "Epoch 413/500\n",
      "6/6 [==============================] - 0s 155us/sample - loss: 5.6663\n",
      "Epoch 414/500\n",
      "6/6 [==============================] - 0s 165us/sample - loss: 5.6607\n",
      "Epoch 415/500\n",
      "6/6 [==============================] - 0s 167us/sample - loss: 5.6551\n",
      "Epoch 416/500\n",
      "6/6 [==============================] - 0s 174us/sample - loss: 5.6495\n",
      "Epoch 417/500\n",
      "6/6 [==============================] - 0s 165us/sample - loss: 5.6439\n",
      "Epoch 418/500\n",
      "6/6 [==============================] - 0s 197us/sample - loss: 5.6383\n",
      "Epoch 419/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.6327\n",
      "Epoch 420/500\n",
      "6/6 [==============================] - 0s 140us/sample - loss: 5.6271\n",
      "Epoch 421/500\n",
      "6/6 [==============================] - 0s 178us/sample - loss: 5.6215\n",
      "Epoch 422/500\n",
      "6/6 [==============================] - 0s 157us/sample - loss: 5.6159\n",
      "Epoch 423/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.6104\n",
      "Epoch 424/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 5.6048\n",
      "Epoch 425/500\n",
      "6/6 [==============================] - 0s 159us/sample - loss: 5.5992\n",
      "Epoch 426/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.5937\n",
      "Epoch 427/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 5.5882\n",
      "Epoch 428/500\n",
      "6/6 [==============================] - 0s 136us/sample - loss: 5.5826\n",
      "Epoch 429/500\n",
      "6/6 [==============================] - 0s 189us/sample - loss: 5.5771\n",
      "Epoch 430/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 5.5716\n",
      "Epoch 431/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 5.5661\n",
      "Epoch 432/500\n",
      "6/6 [==============================] - 0s 162us/sample - loss: 5.5606\n",
      "Epoch 433/500\n",
      "6/6 [==============================] - 0s 159us/sample - loss: 5.5551\n",
      "Epoch 434/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 5.5496\n",
      "Epoch 435/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.5441\n",
      "Epoch 436/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.5386\n",
      "Epoch 437/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 5.5332\n",
      "Epoch 438/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 5.5277\n",
      "Epoch 439/500\n",
      "6/6 [==============================] - 0s 153us/sample - loss: 5.5223\n",
      "Epoch 440/500\n",
      "6/6 [==============================] - 0s 166us/sample - loss: 5.5168\n",
      "Epoch 441/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.5114\n",
      "Epoch 442/500\n",
      "6/6 [==============================] - 0s 285us/sample - loss: 5.5059\n",
      "Epoch 443/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 5.5005\n",
      "Epoch 444/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 5.4951\n",
      "Epoch 445/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 5.4897\n",
      "Epoch 446/500\n",
      "6/6 [==============================] - 0s 133us/sample - loss: 5.4843\n",
      "Epoch 447/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 5.4789\n",
      "Epoch 448/500\n",
      "6/6 [==============================] - 0s 176us/sample - loss: 5.4735\n",
      "Epoch 449/500\n",
      "6/6 [==============================] - 0s 137us/sample - loss: 5.4681\n",
      "Epoch 450/500\n",
      "6/6 [==============================] - 0s 130us/sample - loss: 5.4628\n",
      "Epoch 451/500\n",
      "6/6 [==============================] - 0s 524us/sample - loss: 5.4574\n",
      "Epoch 452/500\n",
      "6/6 [==============================] - 0s 208us/sample - loss: 5.4520\n",
      "Epoch 453/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.4467\n",
      "Epoch 454/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.4413\n",
      "Epoch 455/500\n",
      "6/6 [==============================] - 0s 182us/sample - loss: 5.4360\n",
      "Epoch 456/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.4306\n",
      "Epoch 457/500\n",
      "6/6 [==============================] - 0s 168us/sample - loss: 5.4253\n",
      "Epoch 458/500\n",
      "6/6 [==============================] - 0s 135us/sample - loss: 5.4200\n",
      "Epoch 459/500\n",
      "6/6 [==============================] - 0s 146us/sample - loss: 5.4147\n",
      "Epoch 460/500\n",
      "6/6 [==============================] - 0s 183us/sample - loss: 5.4094\n",
      "Epoch 461/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 5.4041\n",
      "Epoch 462/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 5.3988\n",
      "Epoch 463/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 5.3935\n",
      "Epoch 464/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.3882\n",
      "Epoch 465/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 5.3830\n",
      "Epoch 466/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 5.3777\n",
      "Epoch 467/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.3724\n",
      "Epoch 468/500\n",
      "6/6 [==============================] - 0s 304us/sample - loss: 5.3672\n",
      "Epoch 469/500\n",
      "6/6 [==============================] - 0s 158us/sample - loss: 5.3619\n",
      "Epoch 470/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.3567\n",
      "Epoch 471/500\n",
      "6/6 [==============================] - 0s 173us/sample - loss: 5.3515\n",
      "Epoch 472/500\n",
      "6/6 [==============================] - 0s 155us/sample - loss: 5.3463\n",
      "Epoch 473/500\n",
      "6/6 [==============================] - 0s 184us/sample - loss: 5.3410\n",
      "Epoch 474/500\n",
      "6/6 [==============================] - 0s 147us/sample - loss: 5.3358\n",
      "Epoch 475/500\n",
      "6/6 [==============================] - 0s 144us/sample - loss: 5.3306\n",
      "Epoch 476/500\n",
      "6/6 [==============================] - 0s 139us/sample - loss: 5.3254\n",
      "Epoch 477/500\n",
      "6/6 [==============================] - 0s 138us/sample - loss: 5.3202\n",
      "Epoch 478/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 5.3151\n",
      "Epoch 479/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.3099\n",
      "Epoch 480/500\n",
      "6/6 [==============================] - 0s 168us/sample - loss: 5.3047\n",
      "Epoch 481/500\n",
      "6/6 [==============================] - 0s 152us/sample - loss: 5.2996\n",
      "Epoch 482/500\n",
      "6/6 [==============================] - 0s 143us/sample - loss: 5.2944\n",
      "Epoch 483/500\n",
      "6/6 [==============================] - 0s 163us/sample - loss: 5.2892\n",
      "Epoch 484/500\n",
      "6/6 [==============================] - 0s 149us/sample - loss: 5.2841\n",
      "Epoch 485/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 5.2790\n",
      "Epoch 486/500\n",
      "6/6 [==============================] - 0s 145us/sample - loss: 5.2738\n",
      "Epoch 487/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 5.2687\n",
      "Epoch 488/500\n",
      "6/6 [==============================] - 0s 168us/sample - loss: 5.2636\n",
      "Epoch 489/500\n",
      "6/6 [==============================] - 0s 148us/sample - loss: 5.2585\n",
      "Epoch 490/500\n",
      "6/6 [==============================] - 0s 167us/sample - loss: 5.2534\n",
      "Epoch 491/500\n",
      "6/6 [==============================] - 0s 176us/sample - loss: 5.2483\n",
      "Epoch 492/500\n",
      "6/6 [==============================] - 0s 184us/sample - loss: 5.2432\n",
      "Epoch 493/500\n",
      "6/6 [==============================] - 0s 166us/sample - loss: 5.2381\n",
      "Epoch 494/500\n",
      "6/6 [==============================] - 0s 172us/sample - loss: 5.2330\n",
      "Epoch 495/500\n",
      "6/6 [==============================] - 0s 181us/sample - loss: 5.2280\n",
      "Epoch 496/500\n",
      "6/6 [==============================] - 0s 151us/sample - loss: 5.2229\n",
      "Epoch 497/500\n",
      "6/6 [==============================] - 0s 177us/sample - loss: 5.2179\n",
      "Epoch 498/500\n",
      "6/6 [==============================] - 0s 142us/sample - loss: 5.2128\n",
      "Epoch 499/500\n",
      "6/6 [==============================] - 0s 141us/sample - loss: 5.2078\n",
      "Epoch 500/500\n",
      "6/6 [==============================] - 0s 278us/sample - loss: 5.2027\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f516c1e0950>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define the dataset\n",
    "xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)\n",
    "xs2 = np.array([-2.0, -2.0, -2.0, -2.0, -2.0, -2.0], dtype=float)\n",
    "x = np.concatenate( (xs, xs2), axis=0).reshape((-1,2)) \n",
    "\n",
    "ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)\n",
    "ys = np.expand_dims(ys, axis=1)\n",
    "\n",
    "\n",
    "# use the Sequential API to build a model with our custom layer\n",
    "my_layer = SimpleDense(units=1)\n",
    "model = tf.keras.Sequential([my_layer])\n",
    "\n",
    "# configure and train the model\n",
    "model.compile(optimizer='sgd', loss='mean_squared_error')\n",
    "model.fit( x, ys, epochs=500,verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_11\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "simple_dense_17 (SimpleDense multiple                  3         \n",
      "=================================================================\n",
      "Total params: 3\n",
      "Trainable params: 3\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[14.204527]]\n"
     ]
    }
   ],
   "source": [
    "# perform inference\n",
    "print(model.predict([[10.0, 4.0]]))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<tf.Variable 'sequential_11/simple_dense_17/kernel:0' shape=(2, 1) dtype=float32, numpy=\n",
      "array([[ 2.1627598],\n",
      "       [-2.593192 ]], dtype=float32)>, <tf.Variable 'sequential_11/simple_dense_17/bias:0' shape=(1,) dtype=float32, numpy=array([2.949698], dtype=float32)>]\n"
     ]
    }
   ],
   "source": [
    "# see the updated state of the variables\n",
    "print(my_layer.variables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14.204527999999998"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "10*2.1627598 + 4*(-2.593192) + 2.949698"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyMm2GqP/jiEhAWgS1uK97vE",
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "CustomDenseLayer.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}