13. 텐서플로를 사용하여 신경망 훈련.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "00bef3f9",
   "metadata": {},
   "source": [
    "#### 텐서플로 설치"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b2543578",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorflow in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (2.5.0)\n",
      "Requirement already satisfied: absl-py~=0.10 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (0.13.0)\n",
      "Requirement already satisfied: opt-einsum~=3.3.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (3.3.0)\n",
      "Requirement already satisfied: wheel~=0.35 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (0.36.2)\n",
      "Requirement already satisfied: tensorboard~=2.5 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (2.5.0)\n",
      "Requirement already satisfied: termcolor~=1.1.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.1.0)\n",
      "Requirement already satisfied: typing-extensions~=3.7.4 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (3.7.4.3)\n",
      "Requirement already satisfied: flatbuffers~=1.12.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.12)\n",
      "Requirement already satisfied: astunparse~=1.6.3 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.6.3)\n",
      "Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.1.2)\n",
      "Requirement already satisfied: six~=1.15.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.15.0)\n",
      "Requirement already satisfied: wrapt~=1.12.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.12.1)\n",
      "Requirement already satisfied: numpy~=1.19.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.19.5)\n",
      "Requirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0rc0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (2.5.0)\n",
      "Requirement already satisfied: google-pasta~=0.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (0.2.0)\n",
      "Requirement already satisfied: protobuf>=3.9.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (3.17.3)\n",
      "Requirement already satisfied: h5py~=3.1.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (3.1.0)\n",
      "Requirement already satisfied: grpcio~=1.34.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (1.34.1)\n",
      "Requirement already satisfied: keras-nightly~=2.5.0.dev in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (2.5.0.dev2021032900)\n",
      "Requirement already satisfied: gast==0.4.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow) (0.4.0)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (1.32.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (1.8.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (2.0.1)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (0.6.1)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (52.0.0.post20210125)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (2.25.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (3.3.4)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow) (0.4.4)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (4.7.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (0.2.8)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (4.2.2)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow) (1.3.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow) (0.4.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (2021.5.30)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (4.0.0)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (1.26.6)\n",
      "Requirement already satisfied: idna<3,>=2.5 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow) (2.10)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow) (3.1.1)\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "79a063eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow-gpu\n",
      "  Downloading tensorflow_gpu-2.5.0-cp39-cp39-win_amd64.whl (422.6 MB)\n",
      "Requirement already satisfied: absl-py~=0.10 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (0.13.0)\n",
      "Requirement already satisfied: gast==0.4.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (0.4.0)\n",
      "Requirement already satisfied: typing-extensions~=3.7.4 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (3.7.4.3)\n",
      "Requirement already satisfied: google-pasta~=0.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (0.2.0)\n",
      "Requirement already satisfied: wheel~=0.35 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (0.36.2)\n",
      "Requirement already satisfied: protobuf>=3.9.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (3.17.3)\n",
      "Requirement already satisfied: keras-nightly~=2.5.0.dev in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (2.5.0.dev2021032900)\n",
      "Requirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0rc0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (2.5.0)\n",
      "Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.1.2)\n",
      "Requirement already satisfied: six~=1.15.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.15.0)\n",
      "Requirement already satisfied: flatbuffers~=1.12.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.12)\n",
      "Requirement already satisfied: tensorboard~=2.5 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (2.5.0)"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Could not install packages due to an OSError: [WinError 5] 액세스가 거부되었습니다: 'C:\\\\Anaconda3\\\\envs\\\\dsfs\\\\Lib\\\\site-packages\\\\tensorflow\\\\lite\\\\experimental\\\\microfrontend\\\\python\\\\ops\\\\_audio_microfrontend_op.so'\n",
      "Consider using the `--user` option or check the permissions.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Requirement already satisfied: numpy~=1.19.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.19.5)\n",
      "Requirement already satisfied: wrapt~=1.12.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.12.1)\n",
      "Requirement already satisfied: grpcio~=1.34.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.34.1)\n",
      "Requirement already satisfied: opt-einsum~=3.3.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (3.3.0)\n",
      "Requirement already satisfied: h5py~=3.1.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (3.1.0)\n",
      "Requirement already satisfied: termcolor~=1.1.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.1.0)\n",
      "Requirement already satisfied: astunparse~=1.6.3 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorflow-gpu) (1.6.3)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (0.4.4)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (2.0.1)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (52.0.0.post20210125)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (1.32.1)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (2.25.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (1.8.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (0.6.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from tensorboard~=2.5->tensorflow-gpu) (3.3.4)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (4.7.2)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (4.2.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (0.2.8)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow-gpu) (1.3.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (0.4.8)\n",
      "Requirement already satisfied: idna<3,>=2.5 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (2.10)\n",
      "Requirement already satisfied: chardet<5,>=3.0.2 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (4.0.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (2021.5.30)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (1.26.6)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in c:\\anaconda3\\envs\\dsfs\\lib\\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow-gpu) (3.1.1)\n",
      "Installing collected packages: tensorflow-gpu\n"
     ]
    }
   ],
   "source": [
    "#GPU 용 텐서플로 설치\n",
    "!pip install tensorflow-gpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ac218077",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.5.0\n"
     ]
    }
   ],
   "source": [
    "#텐서플로 버전 확인 \n",
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e00d7315",
   "metadata": {},
   "source": [
    "### 텐서플로 1.x 방식의 저수준 API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2b8d60c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x= 1.0 --> z= 2.7\n",
      "x= 0.6 --> z= 1.9\n",
      "x=-1.8 --> z=-2.9\n"
     ]
    }
   ],
   "source": [
    "##그래프 생성\n",
    "g = tf.Graph()\n",
    "\n",
    "with g.as_default():\n",
    "    x = tf.compat.v1.placeholder(dtype=tf.float32,\n",
    "                                shape = (None), name = 'x')#1차원 데이터셋 x\n",
    "    w = tf.Variable(2.0, name = 'weight')#가중치 w\n",
    "    b = tf.Variable(0.7, name = 'bias')#절편 b\n",
    "    \n",
    "    z = w*x+b#최종입력 계산 z\n",
    "    init = tf.compat.v1.global_variables_initializer()\n",
    "\n",
    "##세션을 만들고 그래프 g를 전달함\n",
    "with tf.compat.v1.Session(graph=g) as sess:#세션을 만듦, 그래프 g를 전달\n",
    "    ##w와 b를 초기화\n",
    "    sess.run(init)\n",
    "    \n",
    "    #z를 평가\n",
    "    for t in [1.0,0.6,-1.8]:\n",
    "        print(\"x=%4.1f --> z=%4.1f\"%(t,sess.run(z,feed_dict = {x:t})))#z평가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0d929ce6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.7 4.7 6.7]\n"
     ]
    }
   ],
   "source": [
    "#원소 하나씩 차례대로 모델에 주입\n",
    "with tf.compat.v1.Session(graph=g) as sess:\n",
    "    sess.run(init)\n",
    "    print(sess.run(z, feed_dict = {x:[1.,2.,3.]}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a36ba587",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor(\"add:0\", dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "#텐서 z 출력\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9e8377",
   "metadata": {},
   "source": [
    "### 텐서플로 2.x 방식"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c3208e47",
   "metadata": {},
   "outputs": [],
   "source": [
    "w = tf.Variable(2.0, name = 'weight')#가중치 w=2.0\n",
    "b = tf.Variable(0.7, name = 'bias')#절편 b= 0.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c990c4c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x =  1.0 --> z=  2.7\n",
      "x =  0.6 --> z=  1.9\n",
      "x = -1.8 --> z= -2.9\n"
     ]
    }
   ],
   "source": [
    "#z를 평가\n",
    "for x in [1.0,0.6,-1.8]:\n",
    "    z = w*x + b\n",
    "    print('x = %4.1f --> z= %4.1f'%(x,z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "50739ad9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(-2.8999999, shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(z)#마지막 실행 결과인 -2.9 출력"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4cd77aab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.7 4.7 6.7]\n"
     ]
    }
   ],
   "source": [
    "z = w*[1.,2.,3.]+b#리스트 데이터 한 번에 계산 가능\n",
    "print(z.numpy())#넘파이 배열로 출력"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7309ff0",
   "metadata": {},
   "source": [
    "### 배열 구조 다루기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "749386b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 크기:  (3, 2, 3)\n",
      "\n",
      "크기가 변경된 입력:\n",
      " [[ 0  1  2  3  4  5]\n",
      " [ 6  7  8  9 10 11]\n",
      " [12 13 14 15 16 17]]\n",
      "\n",
      "열의 합:\n",
      " [18 21 24 27 30 33]\n",
      "\n",
      "열의 평균:\n",
      " [ 6  7  8  9 10 11]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "x_array = np.arange(18).reshape(3,2,3)\n",
    "x2 = tf.reshape(x_array, shape = (-1, 6))#턴서 크기를 바꿈 tf.reshape()\n",
    "\n",
    "#각 열의 합을 계산\n",
    "xsum = tf.reduce_sum(x2,axis = 0)#열 column의 합(axis = 0)\n",
    "\n",
    "#각 열의 평균을 계산\n",
    "xmean = tf.reduce_mean(x2, axis = 0)#열 column의 평균(axis = 0)\n",
    "\n",
    "print(\"입력 크기: \", x_array.shape)\n",
    "print(\"\\n크기가 변경된 입력:\\n\",x2.numpy())\n",
    "print(\"\\n열의 합:\\n\",xsum.numpy())\n",
    "print(\"\\n열의 평균:\\n\",xmean.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "892387c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 크기:  (3, 2, 3)\n",
      "\n",
      "크기가 변경된 입력:\n",
      " [[ 0  1  2  3  4  5]\n",
      " [ 6  7  8  9 10 11]\n",
      " [12 13 14 15 16 17]]\n",
      "\n",
      "행의 합:\n",
      " [15 51 87]\n",
      "\n",
      "행의 평균:\n",
      " [ 2  8 14]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "x_array = np.arange(18).reshape(3,2,3)\n",
    "x2 = tf.reshape(x_array, shape = (-1, 6))#턴서 크기를 바꿈 tf.reshape()\n",
    "\n",
    "#각 열의 합을 계산\n",
    "ysum = tf.reduce_sum(x2,axis = 1)#열 column의 합(axis = 0)\n",
    "\n",
    "#각 열의 평균을 계산\n",
    "ymean = tf.reduce_mean(x2, axis = 1)#열 column의 평균(axis = 0)\n",
    "\n",
    "print(\"입력 크기: \", x_array.shape)\n",
    "print(\"\\n크기가 변경된 입력:\\n\",x2.numpy())\n",
    "print(\"\\n행의 합:\\n\",ysum.numpy())\n",
    "print(\"\\n행의 평균:\\n\",ymean.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab642aaf",
   "metadata": {},
   "source": [
    "### 텐서플로 저수준 API로 간단한 모델 개발"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2a647c25",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = np.arange(10).reshape((10,1))#10개의 훈련 샘플\n",
    "y_train = np.array([1.0,1.3,3.1,\n",
    "                   2.0,5.0,6.3,\n",
    "                   6.6,7.4,8.0,\n",
    "                   9.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "243adf28",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TfLinreg(object):#선형 회귀 모델\n",
    "    \n",
    "    def __init__(self, learning_rate = 0.01):\n",
    "        #가중치와 절편 정의\n",
    "        self.w = tf.Variable(tf.zeros(shape=(1)))\n",
    "        self.b = tf.Variable(tf.zeros(shape = (1)))\n",
    "        \n",
    "        #경사하강법 옵티마이저 설정\n",
    "        self.optimizer = tf.keras.optimizers.SGD(lr = learning_rate)\n",
    "    \n",
    "    def fit(self,X,y,num_epochs = 10):#훈련\n",
    "        ##비용함수의 값을 저장하기 위한 리스트 정의\n",
    "        training_costs = []#비용 함수 값 저장 리스트\n",
    "        for step in range(num_epochs):#에포크마다\n",
    "            ##자동 미분을 위해 연산 과정 기록\n",
    "            with tf.GradientTape() as tape:\n",
    "                z_net = self.w *X + self.b\n",
    "                z_net = tf.reshape(z_net, [-1])\n",
    "                sqr_errors = tf.square(y-z_net)#오차행렬\n",
    "                mean_cost = tf.reduce_mean(sqr_errors)#평균 제곱 오차 MSE\n",
    "            \n",
    "            #비용함수에 대한 가중치의 그래디언트를 계산\n",
    "            grads = tape.gradient(mean_cost, [self.w, self.b])#(미분대상, 그래디언트를 구하려는 변수 리스트)\n",
    "            \n",
    "            #옵티마이저에 그래디언트를 반영\n",
    "            self.optimizer.apply_gradients(zip(grads, [self.w, self.b]))#(그래디언트, 변수 튜플)\n",
    "            \n",
    "            #비용함수의 값을 저장\n",
    "            training_costs.append(mean_cost.numpy())\n",
    "                \n",
    "        return training_costs#비용함수 반환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ff538987",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\optimizer_v2\\optimizer_v2.py:374: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "lrmodel = TfLinreg()#인스턴스 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "33e9d8ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_costs = \n",
    "lrmodel.fit(X_train,y_train)#모델 훈련"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e31b79cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEmCAYAAAAOb7UzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhDElEQVR4nO3de5hcdZ3n8fe3qrvTt6QqpDsh6SpoboIJSTXSIBDXB1F8WGcWRNcLDoijK46CwsrO6szuzujMrsP6jKKOeAly8zJeVkDA24iIOFwEGukOlwS5BRJy6wST7nTs7nTXd/+o07GTdHcqTVX96vJ5PU89VXWq6tQn5SOfPuf8zu+YuyMiIlItYqEDiIiIFJKKTUREqoqKTUREqoqKTUREqoqKTUREqkpd6AD5aGtr887OztAxRESkjDzyyCPb3L19/+UVUWydnZ309PSEjiEiImXEzF6Yarl2RYqISFVRsYmISFVRsYmISFVRsYmISFVRsYmISFVRsYmISFVRsYmISFVRsYmISFWpmWIbHcvyx9Hx0DFERKTIaqLYtu8a4cRP/Rs/6FkfOoqIiBRZTRTbYS0NJJvq6Vu/I3QUEREpspooNjMjk07St2FH6CgiIlJkNVFsAF3pJM/2DzEwvCd0FBERKaKaKbYVqQQAj23YGTiJiIgUU+0UW0cSgF4dZxMRqWo1U2yJ5nqObmvRABIRkSpXM8UGaACJiEgNqK1iSyXYMjDC5p3DoaOIiEiR1FaxpZOAjrOJiFSzmiq2Vy+eR33cWK3dkSIiVaumiq2xPs6rF8/TcTYRkSpWU8UGufPZVq/fSTbroaOIiEgR1FyxZVJJBkfGeG7bUOgoIiJSBDVXbF3RABKdzyYiUp2KVmxm1mhmD5lZn5k9YWafjpYfZWYPmtkzZvZ9M2soVoapHN3eSuucOh1nExGpUsXcYhsBznL3DNAFnGNmpwH/F7ja3Y8F/gB8oIgZDhCPGcs7EtpiExGpUkUrNs/ZFT2tj24OnAX8MFp+E/DWYmWYTiad5MlNA4yM6YraIiLVpqjH2Mwsbma9wFbgTuBZYIe7j0Vv2QB0TPPZS8ysx8x6+vv7C5qrK51gz7izdtNgQdcrIiLhFbXY3H3c3buAFHAqcMIhfHaVu3e7e3d7e3tBc03MQKLjbCIi1ackoyLdfQdwN3A6kDSzuuilFPBSKTJMdvi8RtrnztHUWiIiVaiYoyLbzSwZPW4CzgbWkCu4/xy97WLgtmJlmCEbmVRSA0hERKpQMbfYFgN3m9lq4GHgTnf/MfAJ4ONm9gywALiuiBmm1ZVO8Gz/EAPDe0J8vYiIFEndwd8yO+6+GjhpiuXPkTveFtTEcbbHNuxk5bFtYcOIiEjB1NzMIxNWdCQBXcJGRKTa1GyxJZrrObqtRcfZRESqTM0WG+R2R67esDN0DBERKaCaLrYVqQSbB4bZvHM4dBQRESmQmi42nagtIlJ9arrYli6eR13MdJxNRKSK1HSxNdbHefXiedpiExGpIjVdbACZdILV63eSzXroKCIiUgAqtlSSwZExnts2FDqKiIgUQM0XW9fEABIdZxMRqQo1X2xHt7fSOqeO1TrOJiJSFWq+2OIx48SOefTqRG0RkapQ88UGufPZ1mwcYGRsPHQUERF5hVRsQFcqyeh4lrWbBkNHERGRV0jFhmYgERGpJio2YHGikfa5c3QJGxGRKqBiA8yMTCqpIf8iIlVAxRbpSid4tn+IgeE9oaOIiMgroGKLTBxne1zD/kVEKpqKLbKiIwlArwaQiIhUNBVbJNFcz1FtLTrOJiJS4VRsk2RSCfrWa1ekiEglU7FNkkkn2TwwzOadw6GjiIjILKnYJtGJ2iIilU/FNsnSxfOoi5mOs4mIVDAV2ySN9XFevXietthERCpY0YrNzNJmdreZPWlmT5jZ5dHyT5nZS2bWG93eUqwMs5FJJ1i9YSfZrIeOIiIis1DMLbYx4Ep3XwqcBlxqZkuj1652967o9tMiZjhkK1JJBofHeH77UOgoIiIyC0UrNnff5O6/ix4PAmuAjmJ9X6F0TQwg0XE2EZGKVJJjbGbWCZwEPBgtuszMVpvZ9WY2f5rPXGJmPWbW09/fX4qYABzT3kpLQ1zFJiJSoYpebGbWCtwMXOHuA8BXgWOALmAT8LmpPufuq9y9292729vbix1zr3jMWJ5K0Ks5I0VEKlJRi83M6smV2nfc/RYAd9/i7uPungWuBU4tZobZyKSTrNk4wMjYeOgoIiJyiIo5KtKA64A17v75ScsXT3rb+cDjxcowW12pJKPjWdZuGgwdRUREDlFdEde9ErgIeMzMeqNlfwtcYGZdgAPrgA8VMcOsTJ6BZOKxiIhUhqIVm7vfC9gUL5XV8P6pLE400tY6Jzch8umh04iIyKHQzCNTMDO60gnNQCIiUoFUbNPIpJI827+LgeE9oaOIiMghULFNI5NO4g6Pa9i/iEhFUbFNY0UqAUCvdkeKiFQUFds0ks0NHNXWohlIREQqjIptBplUIjcyUkREKoaKbQaZdJLNA8Ns3jkcOoqIiORJxTaDFakkgIb9i4hUEBXbDJYtmUddzFitYhMRqRgqthk01sc5YfFcHWcTEakgKraDyKSS9G3YQTbroaOIiEgeVGwHkUknGRwe4/ntQ6GjiIhIHlRsB9E1MdO/zmcTEakIKraDOKa9lZaGuIpNRKRCqNgOIh4zlqcS9GrOSBGRiqBiy0MmlWTNxgFGx7Kho4iIyEGo2PKQSScZHc+ydvNA6CgiInIQKrY8ZDSARESkYqjY8rAk0Uhb6xx6daK2iEjZU7HlwczoSic0Z6SISAVQseUpk0rybP8uBob3hI4iIiIzULHlKZNO4g6Pa9i/iEhZU7HlaUUqAUCvdkeKiJQ1FVueks0NdC5o1shIEZEyd9BiM7PL81lWCzLpJKu1K1JEpKzls8V28RTL3lfgHBUhk0qyaecwWwaGQ0cREZFp1E33gpldALwHOMrMbp/00jzg5WIHK0eTT9R+87LDw4YREZEpTVtswP3AJqAN+Nyk5YPA6oOt2MzSwDeBRYADq9z9i2Z2GPB9oBNYB7zT3f8wm/CltmzJPOpiRt8GFZuISLmadleku7/g7r8G3gT8u7vfQ67oUoDlse4x4Ep3XwqcBlxqZkuBTwJ3uftxwF3R84rQWB/nhMVz6dMMJCIiZSufY2y/ARrNrAP4BXARcOPBPuTum9z9d9HjQWAN0AGcB9wUve0m4K2HnDqgTCpJ34YdZLMeOoqIiEwhn2Izd98NvA34iru/A1h2KF9iZp3AScCDwCJ33xS9tJncrsqpPnOJmfWYWU9/f/+hfF1RZVJJBofHeH77UOgoIiIyhbyKzcxOB/4C+Em0LJ7vF5hZK3AzcIW773PdF3d3csffDuDuq9y9292729vb8/26otNM/yIi5S2fYrsC+BvgVnd/wsyOBu7OZ+VmVk+u1L7j7rdEi7eY2eLo9cXA1kNOHdCxC1tpbojrfDYRkTJ10GJz93vc/VzgGjNrdffn3P1jB/ucmRlwHbDG3T8/6aXb+dO5cRcDt80idzDxmLG8I0GvtthERMpSPjOPLDezR4EngCfN7BEzy+cY20pyA03OMrPe6PYW4CrgbDN7mtyIy6teQf4gutJJntw4wOhYNnQUERHZz0znsU34OvBxd78bwMzOBK4FzpjpQ+5+L9OfFvDG/COWn0w6yeh4lrWbB1iRSoaOIyIik+RzjK1lotQAonPbWoqWqAJoAImISPnKp9ieM7P/ZWad0e1/As8VO1g5W5JopK11Dr06UVtEpOzkU2zvB9qBW8iNcGyLltUsMyOTStCna7OJiJSdmSZBbgTmuns/8LFJyxcCfyxBtrKWSSf51VNbGRzew9zG+tBxREQkMtMW25eA/zDF8pXA1cWJUzky6STu8NhL2h0pIlJOZiq2kyedVL2Xu98KvL54kSpDJpUA0ITIIiJlZqZia57l52pCsrmBzgXNGhkpIlJmZiqorWZ26v4LzewUoHxmJQ4ok05qAImISJmZ6QTtvwZ+YGY3Ao9Ey7qB9wLvLnKuipBJJbmtdyNbBoZZNK8xdBwREWHmC40+BJxKbvaQ90U3A17r7g+WIly504naIiLlZ8Yptdx9K/D3JcpScZYtmUddzOjbsIM3Lzs8dBwREUGDQF6Rxvo4xx8+VyMjRUTKiIrtFcqkk6zesINsdsrrpYqISImp2F6hrlSSgeEx1m0fCh1FRETI47I1ZnYHsP/myE6gB/i6uw8XI1il2DuAZMMOjm5vDRtGRETym90f2EXuGmzXAgPAIPCq6HlNO3ZhK80NcR1nExEpE/lcaPQMdz9l0vM7zOxhdz/FzJ4oVrBKEY8ZyzsS9GrIv4hIWchni63VzI6YeBI9ntjnNlqUVBWmK53kyY0DjI5lQ0cREal5+WyxXQnca2bPkjtB+yjgI2bWAtxUzHCVYkUqyeh4lrWbB1iRSoaOIyJS0w5abO7+UzM7DjghWvTUpAEjXyhWsEqSSU/M9L9DxSYiEli+w/1PBpYBGeCdZvbe4kWqPB3JJtpaG+jboAEkIiKh5TPc/1vAMUAvMB4tduCbxYtVWcyMTCqpOSNFRMpAPsfYuoGl7q6pNWaQSSf51VNbGRzew9zG+tBxRERqVj67Ih8HNMPvQWTSSdzhsZe0O1JEJKR8ttjagCfN7CFgZGKhu59btFQVKJOaGECykzOOaQucRkSkduVTbJ8qdohqkGxu4MgFzTrOJiISWD7D/e+ZzYrN7Hrgz4Gt7n5itOxTwAeB/uhtf+vuP53N+stRJpXk4XUvh44hIlLTpj3GZmb3RveDZjYw6TZoZgN5rPtG4Jwpll/t7l3RrWpKDXLH2TbtHGbrQE3PCy0iEtS0xebur4vu57r7vEm3ue4+72ArdvffADW1+dI1caK2zmcTEQkmrxO0zSxuZkvM7IiJ2yv4zsvMbLWZXW9m82f4zkvMrMfMevr7+6d7W1lZtiRBPGY6ziYiEtBBi83MPgpsAe4EfhLdfjzL7/squZO9u4BNwOeme6O7r3L3bnfvbm9vn+XXlVZjfZwTDp9L34YdoaOIiNSsfEZFXg4c7+7bX+mXufuWicdmdi2zL8iylUkn+XHfRrJZJxaz0HFERGpOPrsi15O7YvYrZmaLJz09n9zJ31WlK5VkYHiMdduHQkcREalJ+WyxPQf82sx+wr4naH9+pg+Z2XeBM4E2M9sA/D1wppl1kZtrch3woVmlLmMr9g4g2cHR7a0HebeIiBRaPsX2YnRriG55cfcLplh8Xb6fr1THLZxLc0OcvvU7Of+kVOg4IiI1J58TtD9diiDVIh4zTuxI0KuRkSIiQUxbbGb2BXe/wszuILfrcB+aK3J6XekkN96/jtGxLA11+V7yTkRECmGmLbZvRff/XIog1SSTSjI6luWpzYMsjyZHFhGR0pi22Nz9keh+VnNF1rJMNICkd8MOFZuISInlc4L2cWb2QzN70syem7iVIlyl6kg20dbaoBlIREQCyOcA0A3kZgwZA94AfBP4djFDVTozI5NKqthERALIp9ia3P0uwNz9BXf/FPBnxY1V+VakkjzTv4vB4T2ho4iI1JR8im3EzGLA02Z2mZmdD+jM44PIpBO4w2MvaaZ/EZFSyqfYLgeagY8BJwMXAhcXM1Q1yKSSAPStV7GJiJTSjCdom1kceJe7/zdgF/CXJUlVBea3NHDkgmZWa6Z/EZGSmukK2nXuPg68roR5qooGkIiIlN5MuyIfiu4fNbPbzewiM3vbxK0U4SpdJp1k485htg4Mh44iIlIz8pkEuRHYDpxFbmoti+5vKWKuqtC1d6b/nZy9tDFwGhGR2jBTsS00s4+Tu2baRKFNOGDuSDnQsiUJ4jGjb/0Ozl66KHQcEZGaMFOxxckN65/qMtAqtjw01sc5ftFc+jSARESkZGYqtk3u/g8lS1KlMukkP1m9kWzWicWm+htBREQKaabBI/qvcAF0pRMMDI+xbvtQ6CgiIjVhpmJ7Y8lSVLFMOgnA6g06UVtEpBSmLTZ3f7mUQarVcQvn0twQ1xW1RURKRJd3LrJ4zDixI6EBJCIiJaJiK4GudJInNg4wOpYNHUVEpOqp2EpgRSrB6FiWpzYPho4iIlL1VGwlMDHTf692R4qIFJ2KrQRS85tY0NKgCZFFREpAxVYCZkYmrZn+RURKQcVWIplUkmf6d7FrZCx0FBGRqqZiK5FMOoE7PKYTtUVEiqpoxWZm15vZVjN7fNKyw8zsTjN7OrqfX6zvLzcTA0h0PpuISHEVc4vtRuCc/ZZ9ErjL3Y8D7oqe14T5LQ0cuaBZx9lERIqsaMXm7r8B9p+W6zzgpujxTcBbi/X95WhFSgNIRESKrdTH2Ba5+6bo8WZg2qtvmtklZtZjZj39/f2lSVdkmVSCjTuH2TowHDqKiEjVCjZ4xN2dGS5Y6u6r3L3b3bvb29tLmKx4uqKZ/vs0gEREpGhKXWxbzGwxQHS/tcTfH9SyJQniMdPuSBGRIip1sd0OXBw9vhi4rcTfH1RTQ5zjF83VyEgRkSIq5nD/7wIPAMeb2QYz+wBwFXC2mT0NvCl6XlMmZiDJ7YkVEZFCqyvWit39gmlequkrc3elE3z3oRdZt303R7W1hI4jIlJ1NPNIiWUmBpDoOJuISFGo2Ers2PZWmurj9KrYRESKQsVWYnXxGMs7EhpAIiJSJCq2ADLpBE9sHGB0LBs6iohI1VGxBZBJJxkdy/LU5sHQUUREqo6KLQDN9C8iUjwqtgBS85tY0NKgkZEiIkWgYgvAzHInamuLTUSk4FRsgWRSSZ7euotdI2Oho4iIVBUVWyAr0gnc4THN9C8iUlAqtkA0gEREpDhUbIEc1tLAEYc1awCJiEiBqdgCmpjpX0RECkfFFlAmlWDjzmG2Dg6HjiIiUjVUbAF1RTP9r16vASQiIoWiYgto2ZIE8ZhpAImISAGp2AJqaojzqkVzdQkbEZECUrEF1pVO0Ld+B+4eOoqISFVQsQWWSSUZGB5j3fbdoaOIiFQFFVtgmWgAiYb9i4gUhootsOMWttJUH9dxNhGRAlGxBVYXj7G8I8GjL/5Bx9lERApAxVYGzjyhnb4NO7ny//UxvGc8dBwRkYpWFzqAwF+9/hj2jDlX//L3PL9tiK9feDIL5zWGjiUiUpG0xVYGYjHj8jcdx9cufA1rNw1y7pfvY7VO2hYRmRUVWxk558TF3PzhM4jHjHd87QFu630pdCQRkYoTpNjMbJ2ZPWZmvWbWEyJDuVq6ZB63XbaSTCrJ5d/r5bM/X0s2q0ElIiL5CrnF9gZ373L37oAZylJb6xy+/V9eywWnpvnKr5/lkm/1MDi8J3QsEZGKoF2RZaqhLsZnzl/OP5y3jLuf6uftX72fF7YPhY4lIlL2QhWbA78ws0fM7JKp3mBml5hZj5n19Pf3lzheeTAz3nt6J998/6lsGRjhvGvu4/5ntoWOJSJS1kIV2+vc/TXAfwQuNbPX7/8Gd1/l7t3u3t3e3l76hGVk5bFt3H7ZStpb53DR9Q/xzQfW6WRuEZFpBCk2d38put8K3AqcGiJHJTlyQQu3fOQMznxVO3932xP8jx89zuhYNnQsEZGyU/JiM7MWM5s78Rh4M/B4qXNUormN9ax6bzcfPvMY/vXBF7nwugfZvmskdCwRkbISYottEXCvmfUBDwE/cfefB8hRkeIx4xPnnMAX391F3/odnPvl+1izaSB0LBGRslHyYnP359w9E92Wufv/KXWGanBeVwc/+NDpjGWzvP2r9/PzxzeFjiQiUhY03L+CZdJJ7rjsdbxq0Vz+6tu/44u/fFqDSkSk5qnYKtzCeY1875LTeNtJHVz9y99z2b8+yu7RsdCxRESC0ez+VaCxPs7n3pnhhMVz+aefreX5bUNce3E3Hcmm0NFEREpOW2xVwsy45PXHcP3Fp7D+5d2c+y/38vC6l0PHEhEpORVblXnDCQu59dKVzGuq5z3X/pbvP/xi6EgiIiWlYqtCxy5s5UcfWclpRy/gEzc/xqfveIKxcZ3MLSK1QcVWpRLN9dzwvlN4/8qjuOG+dbzvhofZsXs0dCwRkaJTsVWxuniMv/tPS/ns21fw4PPbees19/HM1sHQsUREikrFVgPeeUqa737wNHaNjPHWa+7n7rVbQ0cSESkaFVuN6O48jNsuex1HHNbM+296mK/f86xO5haRqqRiqyEdySZ++OHTecuJi/mnn63l4z/oY3jPeOhYIiIFpWKrMc0NdXz5PSdx5dmv4tZHX+Jdq37LloHh0LFERApGxVaDzIyPvvE4vnbhyTy9ZZBzv3wvfet3hI4lIlIQKrYads6Jh3Pzh8+gLhbjHV9/gB89+lLoSCIir5iKrca9evE8br9sJV3pJFd8v5erfraW8awGlYhI5VKxCQta5/DtD7yW97z2CL52z7N88Js9DA7vCR1LRGRWVGwCQENdjM+cv5x/PG8Z9/y+n/O/cj/rtg2FjiUicshUbLKPi07v5FvvP5Vtu0Y475r7uO+ZbaEjiYgcEhWbHOCMY9u4/dLXsWjeHN57/UPccN/zOu4mIhXDKmH2ie7ubu/p6Qkdo+bsGhnjiu/18ss1W6iPG+nDmulc0JK7tTVz5IIWjlrQwpJkI3Vx/Y0kIqVlZo+4e/f+y3UFbZlW65w6Vl10Mnes3sjazYOs2zbEuu27+e1z29k9+qcZS+rjRnp+M0cuiMqurYUjF+RKMDW/SaUnIiWlYpMZxWLGeV0dnDdpmbvTPzjCuu27o7Ib4oXtu3l+2xAPPf8yQ5NKry5mpOY3HVB4nW250qtX6YlIganY5JCZGQvnNbJwXiOnHnXYPq+5O9t2jbJu+9De0lu3fTcvbB/ikRf+wK6Rsb3vjceMjmQTnW0tdO7d2svdp+c301Cn0hORQ6dik4IyM9rnzqF97hxO6Tyw9LYPjfLC9iGe35Yru4mtvkdf+AODk0ovZtAxv4nOBZO28qJje+nDmplTFy/1P01EKoSKTUrGzGhrnUNb6xxOPvLA0nt5aHTv1t3E8bwXtg9xe+9GBobHJq0HliSa6GzLFd7CuY001MWojxtz6mLUx6NbXYyGeIyGOqM+nnv8p2UT7zMaomX10fK6mGFmpf55RKRAVGxSFsyMBa1zWNA6h5OPnL/Pa+7Ojt17ot2aQ6zbtnvvLs4fr97Ezj8WfpaUP5Wf7S28hr2FafsUYcMMRVoXjxGzXBkbRswAM6I7DIvuc8czmWK5GcRs4rXJn809n3Gd0QPb/7OT3jfV/xb7PJ/i99n/c7bfu6Zc70HWMdW7ZvP3xWz+JJnNHzKF/tOn0H9LFXx9BfwXL0k2cfzhcwu2vv2p2KTsmRnzWxqY39LASUfMP+D1sfEse8ad0fEse8azjI7l7veMZxkZy722ZzzLnrEsI9F97v3j7BnLfW7yZ0bHsoxOfGbv8+gzY+N71zcylmXXyNik7/RJ7/3T92TdcXIFnbsv+U8oUlbe89oj+Mz5y4u2/iDFZmbnAF8E4sA33P2qEDmkOtTFY9TFoYnKOe7m7rhzQOFlo9bLvbbve7IOTLHciT63d9m+78lmp1/n/h17YOke2ML7v+fg68h970Hfc8B6D/0vgFL90VDo75nNv3XG9RU8X2EtaGko8Br3VfJiM7M4cA1wNrABeNjMbnf3J0udRSQUs8m7AXU8T6SQQoynPhV4xt2fc/dR4Huwz2lSIiIisxai2DqA9ZOeb4iW7cPMLjGzHjPr6e/vL1k4ERGpbGV7Bqy7r3L3bnfvbm9vDx1HREQqRIhiewlIT3qeipaJiIi8YiGK7WHgODM7yswagHcDtwfIISIiVajkoyLdfczMLgP+jdxw/+vd/YlS5xARkeoU5Dw2d/8p8NMQ3y0iItWtbAePiIiIzIaKTUREqop5BUxcZ2b9wAuhcxRZG7AtdIgKpd9u9vTbzZ5+u9kr1G93pLsfcD5YRRRbLTCzHnfvDp2jEum3mz39drOn3272iv3baVekiIhUFRWbiIhUFRVb+VgVOkAF0283e/rtZk+/3ewV9bfTMTYREakq2mITEZGqomITEZGqomILzMzSZna3mT1pZk+Y2eWhM1USM4ub2aNm9uPQWSqNmSXN7IdmttbM1pjZ6aEzVQIz+6/R/1cfN7Pvmllj6EzlzMyuN7OtZvb4pGWHmdmdZvZ0dD+/kN+pYgtvDLjS3ZcCpwGXmtnSwJkqyeXAmtAhKtQXgZ+7+wlABv2OB2VmHcDHgG53P5HcRO7vDpuq7N0InLPfsk8Cd7n7ccBd0fOCUbEF5u6b3P130eNBcv9xOeCK4nIgM0sBfwZ8I3SWSmNmCeD1wHUA7j7q7juChqocdUCTmdUBzcDGwHnKmrv/Bnh5v8XnATdFj28C3lrI71SxlREz6wROAh4MHKVSfAH470A2cI5KdBTQD9wQ7cr9hpm1hA5V7tz9JeCfgReBTcBOd/9F2FQVaZG7b4oebwYWFXLlKrYyYWatwM3AFe4+EDpPuTOzPwe2uvsjobNUqDrgNcBX3f0kYIgC7w6qRtGxoPPI/WGwBGgxswvDpqpsnjvnrKDnnanYyoCZ1ZMrte+4+y2h81SIlcC5ZrYO+B5wlpl9O2ykirIB2ODuE3sHfkiu6GRmbwKed/d+d98D3AKcEThTJdpiZosBovuthVy5ii0wMzNyxznWuPvnQ+epFO7+N+6ecvdOcgfvf+Xu+ss5T+6+GVhvZsdHi94IPBkwUqV4ETjNzJqj/+++EQ26mY3bgYujxxcDtxVy5Sq28FYCF5Hb4uiNbm8JHUpqwkeB75jZaqAL+EzYOOUv2sL9IfA74DFy/w3V1FozMLPvAg8Ax5vZBjP7AHAVcLaZPU1uK/iqgn6nptQSEZFqoi02ERGpKio2ERGpKio2ERGpKio2ERGpKio2ERGpKio2kQDMbHzS6R29ZlawWT/MrHPyTOoitaYudACRGvVHd+8KHUKkGmmLTaSMmNk6M/usmT1mZg+Z2bHR8k4z+5WZrTazu8zsiGj5IjO71cz6otvE9E5xM7s2um7YL8ysKdg/SqTEVGwiYTTttyvyXZNe2+nuy4Evk7uCAcC/ADe5+wrgO8CXouVfAu5x9wy5uR6fiJYfB1zj7suAHcDbi/qvESkjmnlEJAAz2+XurVMsXwec5e7PRZNjb3b3BWa2DVjs7nui5Zvcvc3M+oGUu49MWkcncGd0EUfM7BNAvbv/7xL800SC0xabSPnxaR4fipFJj8fR8XSpISo2kfLzrkn3D0SP7yd3FQOAvwD+PXp8F/BhADOLR1fGFqlp+itOJIwmM+ud9Pzn7j4x5H9+NOP+CHBBtOyj5K52/dfkrnz9l9Hyy4FV0Yzp4+RKbhMiNUzH2ETKSHSMrdvdt4XOIlKptCtSRESqirbYRESkqmiLTUREqoqKTUREqoqKTUREqoqKTUREqoqKTUREqsr/B6NbM4FARTepAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#열번의 에포크동안 계산된 훈련 비용 시각화\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(range(1,len(training_costs)+1),training_costs)\n",
    "plt.tight_layout()\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Training Cost')\n",
    "plt.show()#수렴함"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7174f667",
   "metadata": {},
   "source": [
    "예측 메서드 추가"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "72712a8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TfLinreg(object):#선형 회귀 모델\n",
    "    \n",
    "    def __init__(self, learning_rate = 0.01):\n",
    "        #가중치와 절편 정의\n",
    "        self.w = tf.Variable(tf.zeros(shape=(1)))\n",
    "        self.b = tf.Variable(tf.zeros(shape = (1)))\n",
    "        \n",
    "        #경사하강법 옵티마이저 설정\n",
    "        self.optimizer = tf.keras.optimizers.SGD(lr = learning_rate)\n",
    "    \n",
    "    def fit(self,X,y,num_epochs = 10):#훈련\n",
    "        ##비용함수의 값을 저장하기 위한 리스트 정의\n",
    "        training_costs = []#비용 함수 값 저장 리스트\n",
    "        for step in range(num_epochs):#에포크마다\n",
    "            ##자동 미분을 위해 연산 과정 기록\n",
    "            with tf.GradientTape() as tape:\n",
    "                z_net = self.w *X + self.b\n",
    "                z_net = tf.reshape(z_net, [-1])\n",
    "                sqr_errors = tf.square(y-z_net)#오차행렬\n",
    "                mean_cost = tf.reduce_mean(sqr_errors)#평균 제곱 오차 MSE\n",
    "            \n",
    "            #비용함수에 대한 가중치의 그래디언트를 계산\n",
    "            grads = tape.gradient(mean_cost, [self.w, self.b])#(미분대상, 그래디언트를 구하려는 변수 리스트)\n",
    "            \n",
    "            #옵티마이저에 그래디언트를 반영\n",
    "            self.optimizer.apply_gradients(zip(grads, [self.w, self.b]))#(그래디언트, 변수 튜플)\n",
    "            \n",
    "            #비용함수의 값을 저장\n",
    "            training_costs.append(mean_cost.numpy())\n",
    "                \n",
    "        return training_costs#비용함수 반환\n",
    "    \n",
    "    def predict(self,X):#예측 매서드 추가\n",
    "        return self.w*X + self.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7b7c5c11",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\optimizer_v2\\optimizer_v2.py:374: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "lrmodel = TfLinreg()#인스턴스 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "84c3e640",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_costs = lrmodel.fit(X_train,y_train)#모델 훈련"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "001b1978",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsYklEQVR4nO3de1hU950/8PeXYbjjiKBiQAU14g25iKIiFxnuQ7ubPkl38+y2uTzZ1DZtuumvyXb7PG2S3fye36+72W5z26Zp2ly22fy2abNtdIY7EokEFS9xVLwFUAGvBEHuDHx/f6ATjlxEYOacmXm//pLPOcz5ZKK8OWfO53yFlBJERERa46V2A0RERONhQBERkSYxoIiISJMYUEREpEkMKCIi0iRvtRsYLSwsTEZFRandBhEROdHBgwevSSnn317XVEBFRUWhrq5O7TaIiMiJhBDnxqvzEh8REWkSA4qIiDSJAUVERJqkqc+gxjM4OIjm5mb09fWp3Qrd5Ofnh8jISOj1erVbISI3pvmAam5uRnBwMKKioiCEULsdjyelRFtbG5qbmxEdHa12O0TkxjR/ia+vrw+hoaEMJ40QQiA0NJRntETkcJo/gwLAcNIY/v8g8mxWqxUVFRXo6OiAwWCA0WhEbGzsrB/HJQKKiIi0wWq1YufOnRgcHAQAdHR0YOfOnQAw6yGl+Ut8WhAUFDSm9vrrr+Pdd9+d9PuqqqpgMBgQHx+PVatW4Yc//OGs9FNVVQUhBN5880177ciRIxBC4MUXX5zy6zQ1NWHdunUz3oeIPEd5ebk9nG4ZHBxERUXFrB/L7c6gnHXquWPHjintl5qail27dqG3txcJCQm47777kJKSMuPjr1u3Dr///e/x2GOPAQDef/99xMXFzfh1iYjGI6XEoUOH0NnZOe72jo6OWT+mW51B3Tr1vPVG3Tr1tFqts36s5557zn62kpGRgX/4h3/Apk2bsHLlSlRXV4/Z39/fH/Hx8WhpaQEAlJaWYsuWLUhMTMQDDzyArq4uAIDFYsGqVauwYcMGPPnkkygsLBz3+EuXLkVfXx8uX74MKSWKi4uRn59v337kyBFs3rwZ69evx3333Yf29nYAwMGDBxEXF4e4uDi89tpr9v2Hhobw9NNPY+PGjVi/fj1+9atfzc4bRUQu79KlS/jtb3+LXbt2TbiPwWCY9eO61BnU888/f9ffMzg4iA8//BAffvjhpPs9++yz020LAGCz2bB//35YLBY8//zzKC8vV2xvb2/HmTNnkJaWhmvXruGFF15AeXk5AgMD8bOf/Qw///nP8cwzz+Bb3/oW9uzZg+joaDz44IOTHvP+++/HBx98gISEBCQmJsLX19e+7Zvf/CZeeeUVpKen46c//Smef/55/OIXv8AjjzyCV199FWlpaXj66aft+//mN7+BwWDAgQMH0N/fj5SUFOTk5PCGCCIP1t/fj6qqKuzbtw9Sygn30+v1MBqNs358tzqDUtPXvvY1AMCGDRvQ1NRkr1dXVyMuLg4RERHIzc1FeHg4amtrceLECaSkpCA+Ph7vvPMOzp07h5MnT2LZsmX2+aI7BdTXv/51fPDBB3j//fcV+3Z0dOD69etIT08HADz00EPYs2cPrl+/juvXryMtLQ0A8I1vfMP+PaWlpXj33XcRHx+P5ORktLW14cyZM7Py3hCRa5FS4sSJE3jttddQW1trDycvLy+kpqbiq1/9qv2MyWAw4Ctf+Qrv4tOyW2cvOp0ONpvNXr/1GVRjYyM2b96Mr3/965BSIjs7G++//77iNY4cOXJXxwwPD4der0dZWRleeukl1NTUTLt/KSVeeeUV5ObmKuqjw5aI3F97ezssFgvOnj2rqEdFRcFkMiEsLAwAkJCQ4PBeXCqg7nQZ7vbbH4GRU09HpfvdiI6Oxo9+9CP87Gc/w8svv4wnnngCZ8+exYoVK9Dd3Y2WlhbExMSgoaEBTU1NiIqKwn//93/f8XX/6Z/+CVeuXIFOp7PXDAYDQkJCUF1djdTUVPznf/4n0tPTMXfuXMydOxeffPIJtm3bhvfee8/+Pbm5ufjlL3+JzMxM6PV6nD59GhEREQ55L4hIe2w2G2pqalBdXa34JTswMBA5OTmIjY11+iV/lwqoO7kVQrN9F19PTw8iIyPtX//gBz+Y1uvs2LEDL774Irq7u/H222/jwQcfRH9/PwDghRdewMqVK/Ef//EfyMvLQ2BgIDZu3HjH19y6deu49XfeeQc7duxAT08Pli1bhrfeegsA8NZbb+HRRx+FEAI5OTn2/R977DE0NTUhMTERUkrMnz8ff/rTn6b130lErqWxsRFmsxltbW2KelJSEjIzM+Hv7w8AWPtsMbr7hyZ8nUBfHY4/nzdrfYnJPvhytqSkJHn7goX19fVYvXq1Sh05X1dXF4KCgiClxBNPPIF7770XTz31lNptjeFp/1+I3FFXVxfKyspw9OhRRT08PByFhYVjrqJE/ch8x9ds+r+mu+5DCHFQSpl0e92tzqDcwa9//Wu88847GBgYQEJCAr71rW+p3RIRuRkpJQ4ePIiKigrFczV9fHyQmZmJjRs3wstL/XvoGFAa89RTT2nyjImI3MPFixdhNpvtM5m3rF27Frm5uQgODlaps7EYUEREHqC/vx+7d+/G/v37FTNNISEhMJlMWL58uYrdjY8BRUTkxm7NNJWUlODGjRv2uk6nQ0pKCrZt26bZxUcZUEREbuqLL76AxWLB559/rqgvW7YMBQUFCA0NVamzqWFAERG5GZvNhr1796K6uhpDQ1/eFh4UFITc3FysXbvWJR5jpv5tGhrW1taG+Ph4xMfHIzw8HBEREfavBwYGJv3euro6PPnkk3c8xkRzTHfr1tIeCQkJiImJQVpa2qQPdhz9fTN5AgURaUtDQwNef/11VFVVKcJp48aNeOKJJ7Bu3bpph1Ogr25G2++W25xBOWKALDQ01P74oeeeew5BQUGKNZ1sNhu8vcd/C5OSkpCUNOa2/jFmMxxuPVYJGHls0l/+5V/C399/0oc4VlVVISgoaNaCkojU0dXVhdLS0jGrNyxatAiFhYW45557ZnyM2RzCnQq3OYOaLJymsn2qHn74YezYsQPJycl45plnsH//fmzZsgUJCQnYunUrTp06BWDkB/+tpTKee+45PProo8jIyMCyZcvw8ssv21/v1mKIVVVVyMjIwP33349Vq1bhb/7mb+x32kx1CY7R4uPj8dOf/hSvvvoqAGDnzp1ITk5GQkICsrKycPnyZTQ1NeH111/Hv//7vyM+Ph7V1dXj7kdE2jU8PIz9+/fj1VdfVYSTr68v8vPz8dhjj81KOKnBbc6gnKm5uRk1NTXQ6XTo7OxEdXU1vL29UV5ejh//+Mf44x//OOZ7Tp48id27d+PGjRuIiYnBt7/97TF3zhw+fBjHjx/HPffcg5SUFOzduxdJSUl3tQTHaImJifjXf/1XAMC2bdtQW1trX4n3X/7lX/Bv//Zv2LFjh+LMsL29fdz9iEh7WltbYTab0draqqivW7cOOTk5mpppmg4G1DQ88MAD9oezdnR04KGHHsKZM2cghBizFPItJpMJvr6+8PX1xYIFC3D58mXF8/0AYNOmTfZafHw8mpqaEBQUNGYJjjfeeGNKfY6edWhubsZf/dVf4eLFixgYGLC/3u2muh8Rqaevrw+VlZWoq6tT/DufN28eTCYTli1bpmJ3s8dtLvE5U2BgoP3PP/nJT7B9+3YcO3YMO3fuVDw2ZLTRiwneviTH3exzNw4fPmx/Xt73vvc9fPe734XVasWvfvWrCfuc6n5E5HxSShw7dgyvvfYaDhw4YA8nnU6HjIwMfPvb33abcAJ4BjVjHR0d9gcqvv3227P++tNZggMAjh49in/+53/Gm2++OabPd955x75fcHAwOjs77V9PtB8RqautrQ0WiwUNDQ2K+vLly1FQUIB58+ap1JnjMKBm6JlnnsFDDz2EF154ASbT3T/F9078/f2nvARHdXU1EhIS0NPTgwULFuDll1+238H33HPP4YEHHkBISAgyMzPR2NgIAPjKV76C+++/H3/+85/xyiuvTLgfETmf1WpFeXm54pfIW4KCgpCXl4c1a9a4xEzTdLjNchuOegy8FmhxCQ4ut0HkWFarFX/+858Vs0y3JCcnY/v27YqPBVzZRMttuM1nUM4eIHOmX//614iPj8fatWvR0dHBJTiI3NyNGzewc+fOccPp1pmTu4TTZNzmEp+zB8iciUtwEHmG4eFhHDhwAJWVlRPeEdzV1eXkrtTjEgElpXTba6yuSEuXhYncRUtLC8xmMy5evDjpfgaDwUkdqU/zAeXn54e2tjaEhoYypDRASom2tjb4+fmp3QqRW+jr60NFRQVu//w9MDAQfX19ist8er1+0keXuRvNB1RkZCSam5tx9epVtVuhm/z8/MYMGRPR3ZFSwmq1orS0FN3d3fa6t7c3UlNTsXXrVtTX16OiogIdHR0wGAwwGo2IjY1VsWvn0nxA6fV6Ps2AiNzKtWvXYLFYxoxxrFixAgUFBQgJCQEAxMbGelQg3U7zAUVE5C4GBwdRXV2NmpoaxaW74OBg5OXlYfXq1RBCOGR1Blfk0IASQjwF4DEAEoAVwCNSSj47h4g8ztmzZ2GxWNDe3m6vCSGQnJyMjIwMxW3jzlqdQescFlBCiAgATwJYI6XsFUL8HsBfA3jbUcckItKazs5OlJSU4MSJE4p6ZGQkTCYTwsPDVepM+xx9ic8bgL8QYhBAAIDWO+xPROQWbq3TtHv3bsUK3H5+fsjKykJiYiLvTL4DhwWUlLJFCPEigPMAegGUSilLb99PCPE4gMcBYMmSJY5qh4jIaZqbm2E2m3Hp0iVFPS4uDtnZ2YoVEWhijrzEFwLgLwBEA7gO4AMhxN9KKX83ej8p5RsA3gBGnsXnqH6IiBytt7cXFRUVOHjwoKIeFhYGk8mEqKgodRpzUY68xJcFoFFKeRUAhBAfAtgK4HeTfhcRkYuRUuLo0aMoLS1FT0+Pve7t7Y309HRs2bLFvsgpTZ0jA+o8gM1CiACMXOIzAqib/FuIiFzL1atXYbFY0NTUpKivXLkSeXl59pkmunuO/AxqnxDiDwAOAbABOIybl/KIiFzd4OAg9uzZg5qaGgwPD9vrc+bMQX5+PmJiYqZ9E0Sgr+6Oc1CeQPPrQRERac3p06dRVFSE69ev22tCCGzevBkZGRnw8fFRrzkXNNF6UHySBBHRFHV2dqK4uBj19fWK+uLFi2EymbBw4UKVOnNPDCgiojsYHh7Gvn37sHv3bsU6Tf7+/sjKykJCQgJnmhyAAUVENIkLFy7AbDbj8uXLinp8fDyys7MREBCgUmfujwFFRDSOnp4elJeX4/Dhw4r6/PnzYTKZsHTpUpU68xwMKCLySBM/MVxiha4NG/XN8BM2e1Wv1yM9PR2bN2/mTJOTMKCIyCONF05zRS+26M8hXNelqMfExCAvLw9z5851UncEMKCIyMNF69qQ5N2MQDFy88Poex0MBoN9pomcjwFFRB4rWteGVH0TdEI5DzosgWO2cLz/nUc406QiL7UbICJSQ6Dox7ZxwgkA+qQeB22RDCeV8QyKiDzK0NAQamtrcZ/vcXiPE04A4C8Gx62TczGgiMhjnD9/HmazGVeuXIF+krnabskzJy1gQBGR2+vp6UFZWRmOHDmiqHcN6+EnbIozKZv0Qp0twskd0ngYUETktqSUOHz4MMrLy9Hb22uv6/V6HOhfhCP9YYjStSPJuwWBYgDd0gd1tgg0DoV6zBPDtYwBRURu6fLlyzCbzbhw4YKivmrVKuTl5cFgMKjUGU0VA4qI3MrAwACqqqpQW1uL0csJzZ07F/n5+Vi5cqWK3dHdYEARkVuQUuLUqVMoKipCZ2enve7l5YWtW7ciLS0Ner1exQ7pbjGgiMjlXb9+HUVFRTh9+rSivnTpUphMJsyfP1+lzmgmGFBE5LKGhobw6aef4uOPP4bN9uWDXQMCApCTk4P169dznSYXxoAiIpfU1NQEi8WCq1evKuobNmyA0WiEv7+/Sp3RbGFAEZFL6e7uRllZGT777DNFfeHChSgsLERkZKRKndFsY0ARkUuQUuLQoUMoLy9HX1+fve7j44Pt27dj06ZN8PLi40XdCQOKiDTv0qVLMJvNaG5uVtTXrFmD3NxczJkzR6XOyJEYUESkWf39/aiqqsK+ffsUM00hISHIz8/Hvffeq2J35GgMKCJymImXVR8R6KvD8efzxtSllKivr0dxcTFu3Lhhr3t5eSElJQWpqamcafIADCgicpjJwmmi7e3t7SgqKsKZM2cU9aioKJhMJoSFhc1qj6RdDCgi0gSbzYaamhpUV1crZpoCAwORk5OD2NhYzjR5GAYUEamusbERZrMZbW1tinpSUhIyMzM50+ShGFBEpBo/DOJ//ud/cPToUUU9PDwchYWFiIjgukyejAFFRCqQiNFdxQZ9C44e/fJzKB8fH2RmZmLjxo2caSIGFBE5XrSuzb4oYK/0hg1emOM1oNhn7dq1yM3NRXBwsEpdktYwoIhc3HRv5XaWaF0btunPwVsMAwAChE2xPSQkBCaTCcuXL1ejPdIwBhSRi5vOrdzOEuirQxJa7OE0mpTACUTgv77zMLy9+aOIxuJFXiJymOrvb0LQbZfybhEC+P1zjzGcaEL8m0FEs85ms2Hv3r2orq6ecB+DweDEjsgVMaCIaFY1NDTAYrGMmWkaTa/Xw2g0OrErckUMKCKaFV1dXSgpKcGxY8cU9UWLFmHVqlU4dOgQOjo6YDAYYDQaERsbq1Kn5CoYUEQ0I8PDw6irq0NlZSX6+/vtdV9fX2RmZiIpKQleXl5IS0tTsUtyRQwoIpq21tZWmM1mtLa2Kurr1q1DTk4OZ5poRhwaUEKIuQDeBLAOgATwqJTyU0cek8jTBPrq7jgHNdv6+vpQWVmJuro6xTpNoaGhKCgowLJly2b9mOR5HH0G9RKAYinl/UIIHwABDj4ekcdx5hCulBLHjx9HSUkJurq67HWdTofU1FSkpKTwtnGaNQ77mySEMABIA/AwAEgpBwCMPxBBRJrX1tYGi8WChoYGRX358uUoKCjAvHnzVOqM3JUjf9WJBnAVwFtCiDgABwF8X0rZPXonIcTjAB4HgCVLljiwHSKaDpvNhk8++QSffPIJhoa+vJQYFBSEvLw8rFmzhus0kUOI0dePZ/WFhUgCUAsgRUq5TwjxEoBOKeVPJvqepKQkWVdX55B+iOjuff7557BYLPjiiy/sNSEENm3ahO3bt8PX11fF7shdCCEOSimTbq878gyqGUCzlHLfza//AOBHDjweEc2SGzduoKSkBMePH1fUIyIiYDKZsGjRIpU6I0/isICSUl4SQlwQQsRIKU8BMAI44ajjEdHMDQ8P48CBA6isrMTAwJcfGfv5+cFoNCIxMZHrNJHTOPp2m+8BeO/mHXwNAB5x8PGIaJpaWlpgNptx8eJFRX39+vXIzs5GUFCQSp2Rp3JoQEkpjwAYc12RiLSjr68PFRUVuP3z39DQUJhMJkRHR6vUGXk6DiwQeSgpJaxWK0pLS9Hd/eXNtd7e3khNTcXWrVs500Sq4t8+Ig907do1WCwWNDY2KuorVqxAQUEBQkJCVOqM6EsMKCIPMjg4iOrqatTU1ChmmoKDg5GXl4fVq1dzpok0gwFF5CHOnj0Li8WC9vZ2e00IgeTkZGRkZHCmiTSHAUXk5jo7O1FSUoITJ5RTHpGRkTCZTAgPD1epM6LJMaCI3NTw8DD279+P3bt3j5lpysrKQmJiIi/nkaYxoIjchNVqRUVFBTo6OhAYGAidTofOzk7FPnFxccjOzkZgYKBKXRJNHQOKyA1YrVbs3LkTg4ODAKC4bRwAwsLCYDKZEBUVpUJ3RNPDgCJyAxUVFfZwup3RaMSWLVug083+woVEjsSAInJxV69eRUdHx4Tbt23b5sRuiGYPA4rIRQ0ODmLPnj2oqamZcB+DweDEjohmFwOKyAWdPn0aRUVFuH79+oT76PV6GI1G5zVFNMsYUEQupKOjAyUlJaivr1fUFy9ejJUrV6Kurg4dHR0wGAwwGo2IjY1VqVOimWNAEU3R2meL0d0/NOH2QF8djj+f55BjDw0NYd++faiqqlLcDOHv74+srCwkJCRACMHPm8itMKCIpmiycJrK9um6cOECdu3ahStXrijq8fHxyM7ORkBAgEOOS6Q2BhSRRvX09KC8vByHDx9W1OfPn4/CwkIsWbJEpc6InIMBRaQxUkp89tlnKCsrQ09Pj72u1+uRnp6OzZs3c6aJPAIDikhDrly5ArPZjPPnzyvqMTExyMvLw9y5c9VpjEgFDCgiDRgYGMCePXvw6aefYnh42F43GAzIz89HTEyMit0RqYMBRaSyU6dOoaioSPE0CC8vL2zZsgVpaWnw8fFRsTsi9dwxoIQQ3wPwOyll+532JaKp6+joQFFREU6dOqWoL1myBCaTCQsWLFCpMyJtmMoZ1EIAB4QQhwD8FkCJlFI6ti0i7Qn01d1xDmoqhoaGUFtbi48//lgx0xQQEIDs7GzExcVxnSYiAGIqWSNG/rXkAHgEQBKA3wP4jZTy89lsJikpSdbV1c3mSxJpyvnz52E2m8fMNCUkJCArK4szTeSRhBAHpZRJt9en9BmUlFIKIS4BuATABiAEwB+EEGVSymdmt1Ui99PT04OysjIcOXJEUV+wYAEKCwuxePFidRoj0rCpfAb1fQDfBHANwJsAnpZSDgohvACcAcCAIpqAlBKHDx9GeXk5ent77XW9Xo+MjAwkJydzpoloAlM5g5oH4GtSynOji1LKYSFEoWPaInJ9ly9fhtlsxoULFxT11atXIzc3l0thEN3BHQNKSvnsJNvqJ9pG5KkGBgZQVVWF2tpajP6Md+7cucjPz8fKlStV7I7IdXAOimiWSClx8uRJFBcXo7Oz01738vLC1q1bkZaWBr1er2KHRK6FAUU0C65fv46ioiKcPn1aUV+6dClMJhPmz5+vUmdErosBRTQDQ0NDqKmpwZ49e2Cz2ez1gIAA5OTkYP369ZxpIpomBhTRXbJaraioqEBHRwe8vLwUz84DgA0bNsBoNMLf31+lDoncAwOK6C5YrVZ89NFH9rOl0eEUHh4Ok8mEyMhItdojcisMKKIpklKiqKhIcSnvFj8/P/zd3/0dvLy8VOiMyD0xoIim4NKlSzCbzYph29H6+voYTkSzjAFFNIn+/n5UVVVh3759mOy5lRy6JZp9DCiicUgpUV9fj+LiYty4cUOx7fYbI/R6PYxGo7NbJHJ7DCii27S3t8NiseDs2bOKenR0NAoKCnDx4kX7XXwGgwFGoxGxsbEqdUvkvhweUEIIHYA6AC1SSj67jzTLZrOhpqYG1dXVihshAgMDkZOTg9jYWAghEBYWxkAicgJnnEF9H0A9gDlOOBbRtDQ2NsJsNqOtrU1RT0pKgtFohJ+fn0qdEXkuhwaUECISgAnA/wbwA0cei2g6urq6UFZWhqNHjyrqixYtgslkQkREhEqdEZGjz6B+gZH1ooIn2kEI8TiAxwFgyZIlDm6HaMTw8DAOHjyIyspK9PX12es+Pj7IzMzExo0beds4kcocFlA314q6IqU8KITImGg/KeUbAN4ARpZ8d1Q/RLdcvHgRZrMZLS0tivratWuRm5uL4OAJf58iIidy5BlUCoCvCiEKAPgBmCOE+J2U8m8deEyiCfX396OyshIHDhxQzDTNmzcPBQUFWL58uYrdEdHtHBZQUsp/BPCPAHDzDOqHDCdSg5QSJ06cQHFxMbq6uux1nU6Hbdu2Ydu2bfD25sQFkdbwXyW5tS+++AIWiwWff/65or5s2TIUFBQgNDRUpc6I6E6cElBSyioAVc44FhEwMtO0d+9eVFdXY2hoyF4PCgpCbm4u1q5dy3WaiDSOZ1DkdhoaGmCxWBQzTUIIJCUlITMzkzNNRC6CAUVuo6urCyUlJTh27Jiifs8998BkMuGee+5RqTMimg4GFLm84eFh1NXVobKyEv39/fa6r68vMjMzkZSUxJkmIhfEgCKX1traCrPZjNbWVkU9NjYWOTk5CAoKUqkzIpopBhS5pL6+PlRWVqKurk4x0xQaGoqCggIsW7ZMxe6IaDYwoMilSClx7NgxlJaWjplpSk1NRUpKCmeaiNwE/yWTy2hra4PFYkFDQ4Oivnz5chQUFGDevHkqdUZEjsCAIs2z2Wyorq7G3r17x8w05eXlYc2aNZxpInJDDCjSJKvVal+19vYl1oUQ2LRpE7Zv3w5fX18VuyQiR2JAkeZYrVZ89NFH9lVtR4dTREQETCYTFi1apFZ7ROQkDCjSlOHhYVgsFsWS67f4+/vj0Ucf5UwTkYdgQHmgtc8Wo7t/aMLtgb46HH8+z4kdjWhpaYHZbFYsIDhab28vw4nIgzCgPNBk4TSV7bOtr68PFRUVqKurm3Q/g8HgpI6ISAsYUKQaKSWsVitKS0vR3d1trwshIIRQfPak1+thNBrVaJOIVMKAIlVcu3YNFosFjY2NivqKFStQUFCA5uZm+118BoMBRqMRsbGxKnVLRGpgQJFTDQ4O2meaRp8hBQcHIy8vD6tXr4YQAiEhIQwkIg/HgCKnOXPmDIqKitDe3m6vCSGQnJyMjIwMzjQRkQIDihyus7MTJSUlOHHihKIeGRkJk8mE8PBwlTojIi1jQJHDDA8PY9++faiqqsLAwIC97ufnh6ysLCQmJvIRRUQ0IQaUBwr01d1xDmqmmpubsWvXLly+fFlRj4uLQ3Z2NgIDA2d8DCJybwwoD+TIIdze3l6Ul5fj0KFDinpYWBhMJhOioqIcdmwici8MKJoVUkocPXoUpaWl6Onpsde9vb2Rnp6OLVu2QKeb+ZkZEXkOBhTN2NWrV2E2m3Hu3DlFfeXKlcjPz8fcuXPVaYyIXBoDiqZtcHAQe/bsQU1NjWKmac6cOcjPz0dMTAxvgiCiaWNA0bScPn0aRUVFuH79ur0mhMDmzZuRkZEBHx8f9ZojIrfAgKK70tHRgeLiYpw8eVJRX7x4MUwmExYuXKhSZ0TkbhhQNCVDQ0P2mabBwUF73d/fH1lZWUhISODlPCKaVQwouqMLFy5g165duHLliqIeHx+P7OxsBAQEqNQZEbkzBhRNqKenB+Xl5Th8+LCiPn/+fBQWFmLJkiUqdUZEnoABRWNIKXHkyBGUlZWht7fXXtfr9UhPT8fmzZs500REDseAIoUrV67AbDbj/PnzinpMTAzy8vI400RETsOAIgDAwMAAPv74Y9TW1ipmmgwGg32miYjImRhQHsxqtdpXrRVCQEpp3+bl5YUtW7YgLS2NM01EpAoGlIeyWq346KOPYLPZAEARTkuWLIHJZMKCBQvUao+IiAHliYaGhmCxWOzhNJq/vz8efvhhzjQRkeoYUB7m3LlzMJvN6OvrG3d7b28vw4mINIEB5SF6enpQVlaGI0eOTLqfwWBwTkMTWPts8R0XU3TkelZEpB0OCyghxGIA7wJYCEACeENK+ZKjjkfjk1Li8OHDKC8vV8w0eXl5AYDijj29Xg+j0ej0HkebLJymsp2I3Icjz6BsAP6XlPKQECIYwEEhRJmU8oQDj0mjXL58GWazGRcuXFDUV61ahby8PJw/f95+F5/BYIDRaERsbKxK3RIRKTksoKSUFwFcvPnnG0KIegARABhQDjYwMICqqirU1tYq7s6bO3cu8vPzsXLlSgBAbGwsA4mINMspn0EJIaIAJADYN862xwE8DoDPdpshKSVOnjyJ4uJidHZ22uteXl7YunUr0tLSoNfrVeyQiGjqHB5QQoggAH8E8PdSys7bt0sp3wDwBgAkJSXJ27fT1LS3t6OoqAhnzpxR1JcuXQqTyYT58+er1BkR0fQ4NKCEEHqMhNN7UsoPHXksTzU0NISamhrs2bNHMdcUEBCAnJwcrF+/nreNE5FLcuRdfALAbwDUSyl/7qjjeLKmpiaYzWZcu3ZNUU9MTERWVhb8/f1V6oyIaOYceQaVAuAbAKxCiCM3az+WUloceEyP0N3djbKyMnz22WeK+sKFC1FYWIjIyEiVOpu5QF/dHeegiMgzOPIuvk8A8NrSLJJS4tChQygvL1c8CcLHxwcZGRlITk62zze5Kg7hEtEtfJKEi7h06RLMZjOam5sV9TVr1iA3Nxdz5sxRqTMiIsdgQGlcf38/qqqqsG/fvjEzTQUFBbj33ntV7I6IyHEYUBolpUR9fT2Ki4tx48YNe93LywspKSlITU3lTBMRuTUGlAa1t7fDYrHg7NmzinpUVBRMJhPCwsJU6oyIyHkYUBpis9lQU1OD6upqxUxTYGAgcnJyEBsby5kmIvIYDCiNaGxshNlsRltbm6KelJSEzMxMzjQRkcdhQKmsq6sLZWVlOHr0qKIeHh6OwsJCREREqNQZEZG6GFAONNniewISsb5tSAm4NGamKTMzExs3bnT5mSYioplgQDnQROEUKrqxxec85nt1Y/TK62vXrkVubi6Cg4Od1CERkXYxoJwgWteGJO8WBIoBDMIL3hiG16h7HUJCQmAymbB8+XL1miQi0hgGlINF69qwTd8EbzEyZOuDL5dYH5IC29NTsW3bNs40ERHdhh9yONhG72Z7OI1mkwJ/6l+L7du3M5yIiMbBgHIQm82GOO9WBIjBcbfrINEp/ZzcFRGR63D5S3yT3SkHjCzP4OwnZDc0NMBsNiNR/8WE+3RLHyd2RETkelw+oCYLp6lsn01dXV0oKSnBsWPHFHUpgdEPgLBJL9TZON9ERDQZlw8oLRgeHkZdXR0qKyvR399vrw9IHQ4ORmAAXtjg3YpAMYBu6YM6WwQah0K5+B4R0SQYUDPU2toKs9mM1tZWRX3dunXIycnhTBMR0TQxoKapr68PlZWVOHDggKI+b948mEwmLFu2TKXOiIjcAwPqLkkpcezYMZSWlqKrq8te1+l0SE1NRUpKCry9+bYSEc0Uf5Lehba2NlgsFjQ0NCjqy5cvR0FBAebNm6dSZ0RE7ocBNQU2mw3V1dXYu3cvhoa+vCswKCgIeXl5WLNmDddpIiKaZS4fUIG+ujvOQc3E559/DovFgi+++HKmSQiBjRs3IjMzE76+vjN6fSIiGp/LB5SjhnBv3LiBkpISHD9+XFGPiIiAyWTCokWLHHJcIiIa4fIBNduGh4dx4MABVFZWYmBgwF739fVFVlYWEhMTuU4TEZETMKBGaWlpgdlsxsWLFxX19evXIzs7G0FBQSp1RkTkeRhQGJlpqqioQF1dnaIeGhoKk8mE6OholTojIvJcHh1QUkpYrVaUlpaiu7vbXvf29kZqaiq2bt3KmSYiIpV47E/fa9euwWKxoLGxUVFfsWIFCgoKEBISolJnREQEeGBADQ4Oorq6GjU1NYqZpuDgYOTl5WH16tWcaSIi0gCPCqizZ8/CYrGgvb3dXhNCIDk5GRkZGZxpIiLSEI8IqM7OTpSUlODEiROKemRkJEwmE8LDw1XqjIiIJuLWATU8PIz9+/dj9+7dipkmPz8/+0wTL+cREWmT2wZUc3MzzGYzLl26pKjHxcUhOzsbgYGBKnVGRERT4TYBZbVaUVFRgY6ODvj4+CjOmAAgLCwMJpMJUVFR6jRIRER3xS0Cymq1YufOnRgcHAQARTh5e3sjPT0dW7ZsgU7HJdaJiFyFWwRURUWFPZxG8/b2xne+8x3ONBERuSC3eOppR0fHuHWbzcZwIiJyUQ4NKCFEnhDilBDirBDiR446jsFguKs6ERFpn8MCSgihA/AagHwAawA8KIRY44hjGY1G6PV6RU2v18NoNDricERE5ASO/AxqE4CzUsoGABBC/D8AfwHgxKTfNQ2xsbEAYL+Lz2AwwGg02utEROR6HBlQEQAujPq6GUCyow4WGxvLQCIiciOq3yQhhHhcCFEnhKi7evWq2u0QEZFGODKgWgAsHvV15M2agpTyDSllkpQyaf78+Q5sh4iIXIkjA+oAgHuFENFCCB8Afw3gIwcej4iI3IjDPoOSUtqEEN8FUAJAB+C3UsrjjjoeERG5F4c+SUJKaQFgceQxiIjIPal+kwQREdF4hJRS7R7shBBXAZyb4cuEAbg2C+14Er5n08P37e7xPZsed3/flkopx9wlp6mAmg1CiDopZZLafbgSvmfTw/ft7vE9mx5Pfd94iY+IiDSJAUVERJrkjgH1htoNuCC+Z9PD9+3u8T2bHo9839zuMygiInIP7ngGRUREboABRUREmuQ2AeWs1XvdiRBisRBitxDihBDiuBDi+2r35CqEEDohxGEhxC61e3EVQoi5Qog/CCFOCiHqhRBb1O5J64QQT938t3lMCPG+EMJP7Z6cyS0Cypmr97oZG4D/JaVcA2AzgCf4vk3Z9wHUq92Ei3kJQLGUchWAOPD9m5QQIgLAkwCSpJTrMPJM079WtyvncouAwqjVe6WUAwBurd5Lk5BSXpRSHrr55xsY+YERoW5X2ieEiARgAvCm2r24CiGEAUAagN8AgJRyQEp5XdWmXIM3AH8hhDeAAACtKvfjVO4SUOOt3ssftHdBCBEFIAHAPpVbcQW/APAMgGGV+3Al0QCuAnjr5qXRN4UQgWo3pWVSyhYALwI4D+AigA4pZam6XTmXuwQUzYAQIgjAHwH8vZSyU+1+tEwIUQjgipTyoNq9uBhvAIkAfimlTADQDYCfFU9CCBGCkStB0QDuARAohPhbdbtyLncJqCmt3ktjCSH0GAmn96SUH6rdjwtIAfBVIUQTRi4lZwohfqduSy6hGUCzlPLWGfofMBJYNLEsAI1SyqtSykEAHwLYqnJPTuUuAcXVe6dBCCEw8plAvZTy52r34wqklP8opYyUUkZh5O9ZpZTSo36rnQ4p5SUAF4QQMTdLRgAnVGzJFZwHsFkIEXDz36oRHnZjiUMXLHQWrt47bSkAvgHAKoQ4crP245sLTRLNtu8BeO/mL5ENAB5RuR9Nk1LuE0L8AcAhjNxxexge9sgjPuqIiIg0yV0u8RERkZthQBERkSYxoIiISJMYUEREpEkMKCIi0iQGFBERaRIDioiINIkBRaQSIcRGIcRRIYSfECLw5ro/69Tui0grOKhLpCIhxAsA/AD4Y+RZdf9H5ZaINIMBRaSim4/9OQCgD8BWKeWQyi0RaQYv8RGpKxRAEIBgjJxJEdFNPIMiUpEQ4iOMLNsRDWCRlPK7KrdEpBlu8TRzIlckhPgmgEEp5X8JIXQAaoQQmVLKSrV7I9ICnkEREZEm8TMoIiLSJAYUERFpEgOKiIg0iQFFRESaxIAiIiJNYkAREZEmMaCIiEiT/j8J6jdD/IhnWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#훈련데아터와 학습된 선형 회귀 곡선 시각화\n",
    "plt.scatter(X_train, y_train,\n",
    "           marker ='s',\n",
    "           s=50,\n",
    "           label = 'Training Data')\n",
    "\n",
    "plt.plot(range(X_train.shape[0]),\n",
    "        lrmodel.predict(X_train),\n",
    "        color = 'gray',marker='o',\n",
    "        markersize=6, linewidth=3,\n",
    "        label = 'LinReg Model')#선형회귀곡선\n",
    "\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "plt.legend()\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2336a73",
   "metadata": {},
   "source": [
    "훈련 데이터셋과 테스트 데이터셋 로드"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3351c91a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#서버 액세스 안되서 그냥 불러옴\n",
    "from tensorflow.keras.datasets import mnist\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e710dcce",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = X_train.reshape(-1,784)\n",
    "X_test = X_test.reshape(-1,784)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "5698907c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "행: 60000, 열: 784\n",
      "행: 10000, 열: 784\n"
     ]
    }
   ],
   "source": [
    "print(\"행: %d, 열: %d\"%(X_train.shape[0],X_train.shape[1]))\n",
    "print(\"행: %d, 열: %d\"%(X_test.shape[0],X_test.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "526f4ba2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784) (60000,)\n",
      "(10000, 784) (10000,)\n"
     ]
    }
   ],
   "source": [
    "#평균을 0으로 만들고 표준편차로 나눔(정규화)\n",
    "mean_vals = np.mean(X_train, axis = 0)\n",
    "std_val = np.std(X_train)\n",
    "\n",
    "X_train_centered = (X_train-mean_vals)/std_val\n",
    "X_test_centered = (X_test-mean_vals)/std_val\n",
    "\n",
    "del X_train,X_test\n",
    "\n",
    "print(X_train_centered.shape,y_train.shape)\n",
    "print(X_test_centered.shape,y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "7f918733",
   "metadata": {},
   "outputs": [],
   "source": [
    "#넘파이 난수 초깃값 생성\n",
    "np.random.seed(123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ee38d9b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "처음 3개 레이블: [5 0 4]\n",
      "\n",
      " 처음 3개 레이블 (원-핫):\n",
      " [[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "#클래스 레이블 원-핫 인코딩\n",
    "y_train_onehot  = tf.keras.utils.to_categorical(y_train)\n",
    "print(\"처음 3개 레이블:\", y_train[:3])\n",
    "print(\"\\n 처음 3개 레이블 (원-핫):\\n\",y_train_onehot[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60263e03",
   "metadata": {},
   "source": [
    "###  tf.keras API를 이용한 신경망 구성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "753d060b",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.models.Sequential()\n",
    "\n",
    "model.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,\n",
    "        input_dim = X_train_centered.shape[1],#input_dim이 훈련세트에 있는 특성(열) 개수와 일치해야 함\n",
    "        kernel_initializer = 'glorot_uniform',#새로은 가중치 행렬 초기화 알고리즘(심층 신경망을 안정적으로 초기화하는 방법)\n",
    "        bias_initializer = 'zeros',\n",
    "        activation='tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = y_train_onehot.shape[1],\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer='zeros',\n",
    "        activation = 'softmax'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "9c565e8f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                39250     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 50)                2550      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                510       \n",
      "=================================================================\n",
      "Total params: 42,310\n",
      "Trainable params: 42,310\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30d831a0",
   "metadata": {},
   "source": [
    "#### 피드포워드 신경망 훈련"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "793764b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\optimizer_v2\\optimizer_v2.py:374: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "sgd_optimizer = tf.keras.optimizers.SGD(lr=0.001, decay = 1e-7, momentum = .9)#경사하강법 옵티마이저"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "8104fa8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#모델 컴파일\n",
    "model.compile(optimizer=sgd_optimizer,\n",
    "             loss = 'categorical_crossentropy')#범주형 크로스엔트로피 손실 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "7a1f4fa5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.7227 - val_loss: 0.3627\n",
      "Epoch 2/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.3707 - val_loss: 0.2756\n",
      "Epoch 3/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.3052 - val_loss: 0.2360\n",
      "Epoch 4/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.2688 - val_loss: 0.2119\n",
      "Epoch 5/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.2428 - val_loss: 0.1953\n",
      "Epoch 6/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.2230 - val_loss: 0.1823\n",
      "Epoch 7/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.2065 - val_loss: 0.1724\n",
      "Epoch 8/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1928 - val_loss: 0.1629\n",
      "Epoch 9/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1810 - val_loss: 0.1556\n",
      "Epoch 10/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1708 - val_loss: 0.1502\n",
      "Epoch 11/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1617 - val_loss: 0.1444\n",
      "Epoch 12/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1535 - val_loss: 0.1400\n",
      "Epoch 13/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1461 - val_loss: 0.1356\n",
      "Epoch 14/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1392 - val_loss: 0.1326\n",
      "Epoch 15/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1333 - val_loss: 0.1290\n",
      "Epoch 16/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1274 - val_loss: 0.1266\n",
      "Epoch 17/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1222 - val_loss: 0.1245\n",
      "Epoch 18/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1173 - val_loss: 0.1212\n",
      "Epoch 19/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.1129 - val_loss: 0.1195\n",
      "Epoch 20/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.1086 - val_loss: 0.1174\n",
      "Epoch 21/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.1046 - val_loss: 0.1154\n",
      "Epoch 22/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1009 - val_loss: 0.1148\n",
      "Epoch 23/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0973 - val_loss: 0.1129\n",
      "Epoch 24/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0941 - val_loss: 0.1118\n",
      "Epoch 25/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.0908 - val_loss: 0.1106\n",
      "Epoch 26/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0879 - val_loss: 0.1101\n",
      "Epoch 27/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0849 - val_loss: 0.1084\n",
      "Epoch 28/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0823 - val_loss: 0.1076\n",
      "Epoch 29/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0796 - val_loss: 0.1077\n",
      "Epoch 30/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0773 - val_loss: 0.1066\n",
      "Epoch 31/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0749 - val_loss: 0.1065\n",
      "Epoch 32/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.0726 - val_loss: 0.1061\n",
      "Epoch 33/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0705 - val_loss: 0.1053\n",
      "Epoch 34/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0683 - val_loss: 0.1047\n",
      "Epoch 35/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0663 - val_loss: 0.1047\n",
      "Epoch 36/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.0645 - val_loss: 0.1044\n",
      "Epoch 37/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0626 - val_loss: 0.1040\n",
      "Epoch 38/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0609 - val_loss: 0.1040\n",
      "Epoch 39/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0592 - val_loss: 0.1041\n",
      "Epoch 40/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0575 - val_loss: 0.1041\n",
      "Epoch 41/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0559 - val_loss: 0.1045\n",
      "Epoch 42/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0544 - val_loss: 0.1049\n",
      "Epoch 43/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0529 - val_loss: 0.1040\n",
      "Epoch 44/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0515 - val_loss: 0.1044\n",
      "Epoch 45/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0500 - val_loss: 0.1041\n",
      "Epoch 46/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0487 - val_loss: 0.1047\n",
      "Epoch 47/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0475 - val_loss: 0.1043\n",
      "Epoch 48/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0463 - val_loss: 0.1039\n",
      "Epoch 49/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0451 - val_loss: 0.1042\n",
      "Epoch 50/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0437 - val_loss: 0.1046\n"
     ]
    }
   ],
   "source": [
    "#훈련\n",
    "history = model.fit(X_train_centered, y_train_onehot,\n",
    "                   batch_size=64, epochs=50,\n",
    "                   verbose = 1,#훈련동안 비용함수 따라가기\n",
    "                    validation_split=0.1)#훈련 데이터의 10%를 검증 데이터로 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "50588450",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\engine\\sequential.py:455: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n",
      "  warnings.warn('`model.predict_classes()` is deprecated and '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "처음 3개 예측:  [5 0 4]\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = model.predict_classes(X_train_centered, verbose = 0)#정수로 된 클래스 레이블 예측\n",
    "print(\"처음 3개 예측: \",y_train_pred[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "cc6b2c61",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\engine\\sequential.py:455: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n",
      "  warnings.warn('`model.predict_classes()` is deprecated and '\n"
     ]
    }
   ],
   "source": [
    "#훈련 세트와 테스트 세트에서 모델 정확도 출력\n",
    "y_train_pred = model.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "6e480f58",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "훈련 정확도: 98.99%\n"
     ]
    }
   ],
   "source": [
    "print(\"훈련 정확도: %.2f%%\"%(train_acc*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "76894f50",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\engine\\sequential.py:455: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n",
      "  warnings.warn('`model.predict_classes()` is deprecated and '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "테스트 정확도: 96.36%\n"
     ]
    }
   ],
   "source": [
    "y_test_pred = model.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"테스트 정확도: %.2f%%\"%(test_acc*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52783605",
   "metadata": {},
   "source": [
    "#### 신경망 층 늘리면 성능이 개선될까?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "e597496f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\optimizer_v2\\optimizer_v2.py:374: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.7953 - val_loss: 0.3707\n",
      "Epoch 2/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.3690 - val_loss: 0.2746\n",
      "Epoch 3/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.2965 - val_loss: 0.2347\n",
      "Epoch 4/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.2585 - val_loss: 0.2109\n",
      "Epoch 5/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.2327 - val_loss: 0.1934\n",
      "Epoch 6/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.2124 - val_loss: 0.1807\n",
      "Epoch 7/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1960 - val_loss: 0.1691\n",
      "Epoch 8/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1820 - val_loss: 0.1626\n",
      "Epoch 9/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1698 - val_loss: 0.1544\n",
      "Epoch 10/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1591 - val_loss: 0.1493\n",
      "Epoch 11/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1494 - val_loss: 0.1444\n",
      "Epoch 12/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1409 - val_loss: 0.1402\n",
      "Epoch 13/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1331 - val_loss: 0.1352\n",
      "Epoch 14/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1262 - val_loss: 0.1328\n",
      "Epoch 15/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1197 - val_loss: 0.1292\n",
      "Epoch 16/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1139 - val_loss: 0.1276\n",
      "Epoch 17/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1083 - val_loss: 0.1259\n",
      "Epoch 18/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1034 - val_loss: 0.1248\n",
      "Epoch 19/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0989 - val_loss: 0.1227\n",
      "Epoch 20/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0944 - val_loss: 0.1219\n",
      "Epoch 21/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0905 - val_loss: 0.1202\n",
      "Epoch 22/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0865 - val_loss: 0.1198\n",
      "Epoch 23/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0831 - val_loss: 0.1180\n",
      "Epoch 24/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0795 - val_loss: 0.1156\n",
      "Epoch 25/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0763 - val_loss: 0.1167\n",
      "Epoch 26/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0731 - val_loss: 0.1146\n",
      "Epoch 27/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0701 - val_loss: 0.1145\n",
      "Epoch 28/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0674 - val_loss: 0.1139\n",
      "Epoch 29/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0648 - val_loss: 0.1139\n",
      "Epoch 30/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0621 - val_loss: 0.1145\n",
      "Epoch 31/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0597 - val_loss: 0.1133\n",
      "Epoch 32/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0574 - val_loss: 0.1145\n",
      "Epoch 33/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0550 - val_loss: 0.1134\n",
      "Epoch 34/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0531 - val_loss: 0.1141\n",
      "Epoch 35/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0511 - val_loss: 0.1153\n",
      "Epoch 36/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0492 - val_loss: 0.1132\n",
      "Epoch 37/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0472 - val_loss: 0.1138\n",
      "Epoch 38/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0455 - val_loss: 0.1144\n",
      "Epoch 39/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0437 - val_loss: 0.1140\n",
      "Epoch 40/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0421 - val_loss: 0.1162\n",
      "Epoch 41/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0405 - val_loss: 0.1158\n",
      "Epoch 42/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0390 - val_loss: 0.1146\n",
      "Epoch 43/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0376 - val_loss: 0.1166\n",
      "Epoch 44/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0363 - val_loss: 0.1171\n",
      "Epoch 45/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0350 - val_loss: 0.1165\n",
      "Epoch 46/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0337 - val_loss: 0.1178\n",
      "Epoch 47/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0326 - val_loss: 0.1176\n",
      "Epoch 48/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0314 - val_loss: 0.1190\n",
      "Epoch 49/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0303 - val_loss: 0.1181\n",
      "Epoch 50/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0292 - val_loss: 0.1183\n"
     ]
    }
   ],
   "source": [
    "model0 = tf.keras.models.Sequential()\n",
    "\n",
    "model0.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,\n",
    "        input_dim = X_train_centered.shape[1],#input_dim이 훈련세트에 있는 특성(열) 개수와 일치해야 함\n",
    "        kernel_initializer = 'glorot_uniform',#새로은 가중치 행렬 초기화 알고리즘(심층 신경망을 안정적으로 초기화하는 방법)\n",
    "        bias_initializer = 'zeros',\n",
    "        activation='tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model0.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "###층만 추가함\n",
    "model0.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "model0.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = y_train_onehot.shape[1],\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer='zeros',\n",
    "        activation = 'softmax'\n",
    "    )\n",
    ")\n",
    "sgd_optimizer = tf.keras.optimizers.SGD(lr=0.001, decay = 1e-7, momentum = .9)#경사하강법 옵티마이저\n",
    "\n",
    "#모델 컴파일\n",
    "model0.compile(optimizer=sgd_optimizer,\n",
    "             loss = 'categorical_crossentropy')#범주형 크로스엔트로피 손실 함수\n",
    "\n",
    "#훈련\n",
    "history = model0.fit(X_train_centered, y_train_onehot,\n",
    "                   batch_size=64, epochs=50,\n",
    "                   verbose = 1,#훈련동안 비용함수 따라가기\n",
    "                    validation_split=0.1)#훈련 데이터의 10%를 검증 데이터로 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "e6a1cdf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "기본 훈련 정확도: 98.99%\n",
      "기본 테스트 정확도: 96.36%\n",
      "model0 훈련 정확도: 99.32%\n",
      "model0 테스트 정확도: 95.98%\n"
     ]
    }
   ],
   "source": [
    "#훈련 세트와 테스트 세트에서 모델 정확도 출력\n",
    "y_train_pred = model.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"기본 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"기본 테스트 정확도: %.2f%%\"%(test_acc*100))\n",
    "\n",
    "y_train_pred = model0.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"model0 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model0.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"model0 테스트 정확도: %.2f%%\"%(test_acc*100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4d0a68",
   "metadata": {},
   "source": [
    "층을 추가했더니 성능이 감소함"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "84bf5c85",
   "metadata": {},
   "outputs": [],
   "source": [
    "model1 = tf.keras.models.Sequential()\n",
    "\n",
    "model1.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,\n",
    "        input_dim = X_train_centered.shape[1],#input_dim이 훈련세트에 있는 특성(열) 개수와 일치해야 함\n",
    "        kernel_initializer = 'glorot_uniform',#새로은 가중치 행렬 초기화 알고리즘(심층 신경망을 안정적으로 초기화하는 방법)\n",
    "        bias_initializer = 'zeros',\n",
    "        activation='tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model1.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "################추가된 층\n",
    "model1.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 100,#units == input_dim\n",
    "        input_dim = 100,\n",
    "        kernel_initializer = 'normal',#그냥 해봄\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'relu'#얘도 궁금해서 사용해봄\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "model1.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = y_train_onehot.shape[1],\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer='zeros',\n",
    "        activation = 'softmax'\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "79df0311",
   "metadata": {},
   "outputs": [],
   "source": [
    "adam_optimizer = tf.keras.optimizers.Adam(\n",
    "    learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False,\n",
    "    name='Adam'\n",
    ")#궁금해서 해봄"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "c1e9394c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#모델 컴파일\n",
    "model1.compile(optimizer=adam_optimizer,\n",
    "             loss = 'categorical_crossentropy')#범주형 크로스엔트로피 손실 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "1803a72a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.3199 - val_loss: 0.1299\n",
      "Epoch 2/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1350 - val_loss: 0.1167\n",
      "Epoch 3/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0981 - val_loss: 0.1069\n",
      "Epoch 4/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0775 - val_loss: 0.1146\n",
      "Epoch 5/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0637 - val_loss: 0.1153\n",
      "Epoch 6/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0529 - val_loss: 0.1067\n",
      "Epoch 7/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0439 - val_loss: 0.1209\n",
      "Epoch 8/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0381 - val_loss: 0.1303\n",
      "Epoch 9/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0316 - val_loss: 0.1352\n",
      "Epoch 10/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0270 - val_loss: 0.1337\n",
      "Epoch 11/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0270 - val_loss: 0.1332\n",
      "Epoch 12/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0245 - val_loss: 0.1471\n",
      "Epoch 13/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0196 - val_loss: 0.1440\n",
      "Epoch 14/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0197 - val_loss: 0.1560\n",
      "Epoch 15/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0209 - val_loss: 0.1530\n",
      "Epoch 16/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0164 - val_loss: 0.1489\n",
      "Epoch 17/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0156 - val_loss: 0.1589\n",
      "Epoch 18/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0166 - val_loss: 0.1723\n",
      "Epoch 19/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0159 - val_loss: 0.1784\n",
      "Epoch 20/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0142 - val_loss: 0.1684\n",
      "Epoch 21/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0126 - val_loss: 0.1776\n",
      "Epoch 22/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0141 - val_loss: 0.1698\n",
      "Epoch 23/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0129 - val_loss: 0.1837\n",
      "Epoch 24/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0130 - val_loss: 0.1809\n",
      "Epoch 25/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0126 - val_loss: 0.1838\n",
      "Epoch 26/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0081 - val_loss: 0.1748\n",
      "Epoch 27/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0106 - val_loss: 0.1910\n",
      "Epoch 28/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0142 - val_loss: 0.1826\n",
      "Epoch 29/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0132 - val_loss: 0.1899\n",
      "Epoch 30/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0092 - val_loss: 0.1930 ET\n",
      "Epoch 31/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0064 - val_loss: 0.1928\n",
      "Epoch 32/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0126 - val_loss: 0.1872\n",
      "Epoch 33/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0164 - val_loss: 0.1999\n",
      "Epoch 34/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0109 - val_loss: 0.1997\n",
      "Epoch 35/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0080 - val_loss: 0.1938\n",
      "Epoch 36/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0060 - val_loss: 0.2002\n",
      "Epoch 37/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0144 - val_loss: 0.2139\n",
      "Epoch 38/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0116 - val_loss: 0.1952\n",
      "Epoch 39/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0081 - val_loss: 0.1988\n",
      "Epoch 40/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0083 - val_loss: 0.2185\n",
      "Epoch 41/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0097 - val_loss: 0.2214\n",
      "Epoch 42/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0114 - val_loss: 0.2152\n",
      "Epoch 43/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0101 - val_loss: 0.2034\n",
      "Epoch 44/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0066 - val_loss: 0.1934\n",
      "Epoch 45/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0077 - val_loss: 0.2161\n",
      "Epoch 46/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0112 - val_loss: 0.2253\n",
      "Epoch 47/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0143 - val_loss: 0.2037\n",
      "Epoch 48/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0052 - val_loss: 0.2158\n",
      "Epoch 49/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0041 - val_loss: 0.2188\n",
      "Epoch 50/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0078 - val_loss: 0.2505\n"
     ]
    }
   ],
   "source": [
    "#훈련\n",
    "history = model1.fit(X_train_centered, y_train_onehot,\n",
    "                   batch_size=64, epochs=50,\n",
    "                   verbose = 1,#훈련동안 비용함수 따라가기\n",
    "                    validation_split=0.1)#훈련 데이터의 10%를 검증 데이터로 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "989a687c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "기본 훈련 정확도: 98.99%\n",
      "기본 테스트 정확도: 96.36%\n",
      "model1 훈련 정확도: 99.25%\n",
      "model1 테스트 정확도: 95.95%\n"
     ]
    }
   ],
   "source": [
    "#훈련 세트와 테스트 세트에서 모델 정확도 출력\n",
    "y_train_pred = model.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"기본 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"기본 테스트 정확도: %.2f%%\"%(test_acc*100))\n",
    "\n",
    "y_train_pred = model1.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"model1 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model1.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"model1 테스트 정확도: %.2f%%\"%(test_acc*100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "030b71d9",
   "metadata": {},
   "source": [
    "테스트 정확도 기본 모델보다 더 높여보고 싶다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "c8fc82bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.3413 - val_loss: 0.1503\n",
      "Epoch 2/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1534 - val_loss: 0.1263\n",
      "Epoch 3/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.1146 - val_loss: 0.1170\n",
      "Epoch 4/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0928 - val_loss: 0.1155\n",
      "Epoch 5/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0753 - val_loss: 0.1196\n",
      "Epoch 6/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0626 - val_loss: 0.1211\n",
      "Epoch 7/50\n",
      "844/844 [==============================] - 3s 3ms/step - loss: 0.0531 - val_loss: 0.1201\n",
      "Epoch 8/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0437 - val_loss: 0.1362\n",
      "Epoch 9/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0377 - val_loss: 0.1283\n",
      "Epoch 10/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0321 - val_loss: 0.1375\n",
      "Epoch 11/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0282 - val_loss: 0.1457\n",
      "Epoch 12/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0240 - val_loss: 0.1370\n",
      "Epoch 13/50\n",
      "844/844 [==============================] - 2s 3ms/step - loss: 0.0222 - val_loss: 0.1617\n",
      "Epoch 14/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0181 - val_loss: 0.1628\n",
      "Epoch 15/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0163 - val_loss: 0.1623\n",
      "Epoch 16/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0148 - val_loss: 0.1712\n",
      "Epoch 17/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0136 - val_loss: 0.1810\n",
      "Epoch 18/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0115 - val_loss: 0.1808\n",
      "Epoch 19/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0122 - val_loss: 0.1832\n",
      "Epoch 20/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0100 - val_loss: 0.2148\n",
      "Epoch 21/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0085 - val_loss: 0.2068\n",
      "Epoch 22/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0095 - val_loss: 0.2127\n",
      "Epoch 23/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0090 - val_loss: 0.2158\n",
      "Epoch 24/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0072 - val_loss: 0.2251\n",
      "Epoch 25/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0078 - val_loss: 0.2268\n",
      "Epoch 26/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0070 - val_loss: 0.2406\n",
      "Epoch 27/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0072 - val_loss: 0.2582\n",
      "Epoch 28/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0073 - val_loss: 0.2529\n",
      "Epoch 29/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0072 - val_loss: 0.2550\n",
      "Epoch 30/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0070 - val_loss: 0.2653\n",
      "Epoch 31/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0060 - val_loss: 0.2644\n",
      "Epoch 32/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0059 - val_loss: 0.2588\n",
      "Epoch 33/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0059 - val_loss: 0.2758\n",
      "Epoch 34/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0054 - val_loss: 0.2612\n",
      "Epoch 35/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0056 - val_loss: 0.2917\n",
      "Epoch 36/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0045 - val_loss: 0.3038\n",
      "Epoch 37/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0053 - val_loss: 0.2957\n",
      "Epoch 38/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0056 - val_loss: 0.2916\n",
      "Epoch 39/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0044 - val_loss: 0.2745\n",
      "Epoch 40/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0040 - val_loss: 0.2924\n",
      "Epoch 41/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0043 - val_loss: 0.3174\n",
      "Epoch 42/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0050 - val_loss: 0.3184\n",
      "Epoch 43/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0042 - val_loss: 0.3346\n",
      "Epoch 44/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0041 - val_loss: 0.3242\n",
      "Epoch 45/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0043 - val_loss: 0.3171\n",
      "Epoch 46/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0042 - val_loss: 0.3255\n",
      "Epoch 47/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0045 - val_loss: 0.3366\n",
      "Epoch 48/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0033 - val_loss: 0.3227\n",
      "Epoch 49/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0046 - val_loss: 0.3208\n",
      "Epoch 50/50\n",
      "844/844 [==============================] - 2s 2ms/step - loss: 0.0040 - val_loss: 0.3544\n"
     ]
    }
   ],
   "source": [
    "\n",
    "model2 = tf.keras.models.Sequential()\n",
    "\n",
    "model2.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,\n",
    "        input_dim = X_train_centered.shape[1],#input_dim이 훈련세트에 있는 특성(열) 개수와 일치해야 함\n",
    "        kernel_initializer = 'glorot_uniform',#새로은 가중치 행렬 초기화 알고리즘(심층 신경망을 안정적으로 초기화하는 방법)\n",
    "        bias_initializer = 'zeros',\n",
    "        activation='tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model2.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "###층 추가함\n",
    "model2.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 70,#units == input_dim\n",
    "        input_dim = 70,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'relu'\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "model2.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = y_train_onehot.shape[1],\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'normal',\n",
    "        bias_initializer='zeros',\n",
    "        activation = 'softmax'\n",
    "    )\n",
    ")\n",
    "\n",
    "rms_optimizer = tf.keras.optimizers.RMSprop(\n",
    "    learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False)#새로운 옵티마이저\n",
    "\n",
    "#모델 컴파일\n",
    "model2.compile(optimizer=rms_optimizer,\n",
    "             loss = 'categorical_crossentropy')#범주형 크로스엔트로피 손실 함수\n",
    "\n",
    "#훈련\n",
    "history = model2.fit(X_train_centered, y_train_onehot,\n",
    "                   batch_size=64, epochs=50,\n",
    "                   verbose = 1,#훈련동안 비용함수 따라가기\n",
    "                    validation_split=0.1)#훈련 데이터의 10%를 검증 데이터로 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "2396eab0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "기본 훈련 정확도: 98.99%\n",
      "기본 테스트 정확도: 96.36%\n",
      "model2 훈련 정확도: 99.58%\n",
      "model2 테스트 정확도: 96.01%\n"
     ]
    }
   ],
   "source": [
    "#훈련 세트와 테스트 세트에서 모델 정확도 출력\n",
    "y_train_pred = model.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"기본 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"기본 테스트 정확도: %.2f%%\"%(test_acc*100))\n",
    "\n",
    "y_train_pred = model2.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"model2 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model2.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"model2 테스트 정확도: %.2f%%\"%(test_acc*100))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "70ee5111",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Nadam\n",
    "model3 = tf.keras.models.Sequential()\n",
    "\n",
    "model3.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,\n",
    "        input_dim = X_train_centered.shape[1],#input_dim이 훈련세트에 있는 특성(열) 개수와 일치해야 함\n",
    "        kernel_initializer = 'glorot_uniform',#새로은 가중치 행렬 초기화 알고리즘(심층 신경망을 안정적으로 초기화하는 방법)\n",
    "        bias_initializer = 'zeros',\n",
    "        activation='tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model3.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "###층 추가함\n",
    "model3.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 70,#units == input_dim\n",
    "        input_dim = 70,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'relu'\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "model3.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = y_train_onehot.shape[1],\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'normal',\n",
    "        bias_initializer='zeros',\n",
    "        activation = 'softmax'\n",
    "    )\n",
    ")\n",
    "\n",
    "nadam_optimizer = tf.keras.optimizers.Nadam(\n",
    "    learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07,\n",
    ")\n",
    "\n",
    "#모델 컴파일\n",
    "model3.compile(optimizer=nadam_optimizer,\n",
    "             loss = 'categorical_crossentropy')#범주형 크로스엔트로피 손실 함수\n",
    "\n",
    "#훈련\n",
    "history = model3.fit(X_train_centered, y_train_onehot,\n",
    "                   batch_size=64, epochs=50,\n",
    "                   verbose = 0,#훈련동안 비용함수 따라가기\n",
    "                    validation_split=0.1)#훈련 데이터의 10%를 검증 데이터로 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "21fe560e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "기본 훈련 정확도: 98.99%\n",
      "기본 테스트 정확도: 96.36%\n",
      "model3 훈련 정확도: 99.52%\n",
      "model3 테스트 정확도: 96.17%\n"
     ]
    }
   ],
   "source": [
    "#훈련 세트와 테스트 세트에서 모델 정확도 출력\n",
    "y_train_pred = model.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"기본 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"기본 테스트 정확도: %.2f%%\"%(test_acc*100))\n",
    "\n",
    "y_train_pred = model3.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"model3 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model3.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"model3 테스트 정확도: %.2f%%\"%(test_acc*100))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "eb748d09",
   "metadata": {},
   "outputs": [],
   "source": [
    "#adagrad\n",
    "model4 = tf.keras.models.Sequential()\n",
    "\n",
    "model4.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,\n",
    "        input_dim = X_train_centered.shape[1],#input_dim이 훈련세트에 있는 특성(열) 개수와 일치해야 함\n",
    "        kernel_initializer = 'glorot_uniform',#새로은 가중치 행렬 초기화 알고리즘(심층 신경망을 안정적으로 초기화하는 방법)\n",
    "        bias_initializer = 'zeros',\n",
    "        activation='tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "model4.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 50,#units == input_dim\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'tanh'#하이퍼볼릭 탄젠트\n",
    "    )\n",
    ")\n",
    "\n",
    "###층 추가함\n",
    "model4.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = 70,#units == input_dim\n",
    "        input_dim = 70,\n",
    "        kernel_initializer = 'glorot_uniform',\n",
    "        bias_initializer = 'zeros',\n",
    "        activation = 'relu'\n",
    "    )\n",
    ")\n",
    "\n",
    "\n",
    "model4.add(\n",
    "    tf.keras.layers.Dense(\n",
    "        units = y_train_onehot.shape[1],\n",
    "        input_dim = 50,\n",
    "        kernel_initializer = 'normal',\n",
    "        bias_initializer='zeros',\n",
    "        activation = 'softmax'\n",
    "    )\n",
    ")\n",
    "\n",
    "ftrl_optimizer = tf.keras.optimizers.Adadelta(\n",
    "    learning_rate=0.001, rho=0.95, epsilon=1e-07\n",
    ")\n",
    "\n",
    "\n",
    "#모델 컴파일\n",
    "model4.compile(optimizer=ftrl_optimizer,\n",
    "             loss = 'categorical_crossentropy')#범주형 크로스엔트로피 손실 함수\n",
    "\n",
    "#훈련\n",
    "history = model4.fit(X_train_centered, y_train_onehot,\n",
    "                   batch_size=64, epochs=50,\n",
    "                   verbose = 0,#훈련동안 비용함수 따라가기\n",
    "                    validation_split=0.1)#훈련 데이터의 10%를 검증 데이터로 할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "aa8a76fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Anaconda3\\envs\\dsfs\\lib\\site-packages\\tensorflow\\python\\keras\\engine\\sequential.py:455: UserWarning: `model.predict_classes()` is deprecated and will be removed after 2021-01-01. Please use instead:* `np.argmax(model.predict(x), axis=-1)`,   if your model does multi-class classification   (e.g. if it uses a `softmax` last-layer activation).* `(model.predict(x) > 0.5).astype(\"int32\")`,   if your model does binary classification   (e.g. if it uses a `sigmoid` last-layer activation).\n",
      "  warnings.warn('`model.predict_classes()` is deprecated and '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "기본 훈련 정확도: 98.99%\n",
      "기본 테스트 정확도: 96.36%\n",
      "model4 훈련 정확도: 83.04%\n",
      "model4 테스트 정확도: 83.81%\n"
     ]
    }
   ],
   "source": [
    "#훈련 세트와 테스트 세트에서 모델 정확도 출력\n",
    "y_train_pred = model.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"기본 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"기본 테스트 정확도: %.2f%%\"%(test_acc*100))\n",
    "\n",
    "y_train_pred = model4.predict_classes(X_train_centered, verbose =0)\n",
    "correct_preds = np.sum(y_train == y_train_pred, axis = 0)#훈련세트 예측 중 맞은 것 개수\n",
    "train_acc = correct_preds/y_train.shape[0]#훈련 정확도\n",
    "print(\"model4 훈련 정확도: %.2f%%\"%(train_acc*100))\n",
    "\n",
    "y_test_pred = model4.predict_classes(X_test_centered, verbose = 0)#테스트세트 예측\n",
    "correct_preds = np.sum(y_test == y_test_pred, axis = 0)#테스트 세트 예측 중 맞은 것 개수\n",
    "test_acc = correct_preds/y_test.shape[0]#테스트 정확도\n",
    "print(\"model4 테스트 정확도: %.2f%%\"%(test_acc*100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48f1adf4",
   "metadata": {},
   "source": [
    "#### 로지스틱 함수 요약\n",
    "\n",
    "$$ \\Phi_{logistic} (z) = \\frac{1}{1+e^{-z}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "8a25f49f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P(y=1|x) = 0.888\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "X = np.array([1,1.4,2.5])\n",
    "w = np.array([0.4,0.3,0.5])\n",
    "\n",
    "def net_input(X,w):#최종 입력\n",
    "    return np.dot(X,w)\n",
    "\n",
    "def logistic(z):\n",
    "    return 1.0/(1.0 + np.exp(-z))\n",
    "\n",
    "def logistic_activation(X,w):\n",
    "    z = net_input(X,w)\n",
    "    return logistic(z)\n",
    "\n",
    "print(\"P(y=1|x) = %.3f\"%(logistic_activation(X,w)))#샘플 x가 양성 클래스에 속할 비율"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "2b46d1c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "최종입력:\n",
      " [1.78 0.76 1.65]\n",
      "출력 유닛:\n",
      " [0.85569687 0.68135373 0.83889105]\n",
      "예측 클래스 레이블: 0\n"
     ]
    }
   ],
   "source": [
    "W = np.array([[1.1,1.2,0.8,0.4],\n",
    "             [0.2,0.4,1.0,0.2],\n",
    "             [0.6,1.5,1.2,0.7]])\n",
    "A =np.array([[1,0.1,0.4,0.6]])\n",
    "Z = np.dot(W,A[0])\n",
    "y_probas = logistic(Z)\n",
    "print(\"최종입력:\\n\",Z)\n",
    "print(\"출력 유닛:\\n\",y_probas)#이해하기 어려운 로지스틱 함수\n",
    "\n",
    "y_class = np.argmax(Z,axis=0)#클래스 레이블 예측하는 방법: 가장 큰 값 선택\n",
    "print(\"예측 클래스 레이블: %d\"%y_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34a15b0b",
   "metadata": {},
   "source": [
    "### 소프트맥스 함수를 사용하여 다중 클래스 확률 예측\n",
    "하나의 클래스 인덱스를 찾는 대신 각 클래서의 확률을 반환"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6580b16b",
   "metadata": {},
   "source": [
    "$$P(y = i|z) = \\phi(z) \\frac{e^{Z_i}}{\\sum\\nolimits_{j=1}^{M} e^{Z_j}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "3516637b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(z):\n",
    "    return np.exp(z) / np.sum(np.exp(z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "c75ef627",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "확률:\n",
      " [0.44668973 0.16107406 0.39223621]\n"
     ]
    }
   ],
   "source": [
    "y_probas = softmax(Z)\n",
    "print(\"확률:\\n\",y_probas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "aa868cd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(y_probas)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9858d82",
   "metadata": {},
   "source": [
    "### 하이퍼볼릭 탄젠트로 출력 범위 넓히기\n",
    "$$\\phi_{logistic} = \\frac{1}{1+e^{-z}}$$\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\\phi_tanh(z) = 2x\\phi_{logistic}(2z) - 1 = \\frac{e^z - e^{-z}}{e^z + e^{-z}}$$\n",
    "\n",
    "하이퍼볼릭 탄젠트: <u>스케일이 조정된 로지스틱 함수</u>\n",
    "<br>출력 범위를 (-1,1) 사이로 넓힘"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "97db7d42",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def tanh(z):#하이퍼볼릭 탄젠트\n",
    "    e_p = np.exp(z)\n",
    "    e_m = np.exp(-z)\n",
    "    \n",
    "    return (e_p - e_m)/(e_p + e_m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "2aafa60b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8Z0lEQVR4nO3de1xUdfrA8c/DHQTF+920stSsNC9d7HIqK2lb26It3Gw3qqXdrM3d9ddmF92y3dxddje6WK3VUmtbdqHSrTBRwxQt0FDxQiqioKKiggICA/P9/XHGEQQEFJgBnvfrNa+Z55zvOfMwjjycc77n+xVjDEoppZS38fF0AkoppVRttEAppZTySlqglFJKeSUtUEoppbySFiillFJeSQuUUkopr+RVBUpE3hKR/SKSUcd6S0QKRSTd9ZjR0jkqpZRqGX6eTuAk8cDLwDunaPONMeaWlklHKaWUp3jVEZQxZjlwyNN5KKWU8jxvO4JqiMtFZB2wB5hmjNlYWyMRiQFiADp06DBqyJAhLZiiUkqphlqzZk2+Mab7yctbW4FaC5xljCkSkZuBT4HBtTU0xvwL+BfA6NGjTVpaWoslqZRSquFEZGdty73qFF99jDFHjDFFrtdfAP4i0s3DaSmllGoGrapAiUgvERHX67HY+R/0bFZKKaWag1ed4hOR9wAL6CYiucBMwB/AGPMacAfwaxGpAI4BUUaHY1dKqTbJqwqUMWZSPetfxu6GrpRSqo1rVaf4lFJKtR9aoJRSSnklLVBKKaW8khYopZRSXkkLlFJKKa+kBUoppZRX0gKllFLKK2mBUkop5ZW0QCmllPJKWqCUUkp5JS1QSimlvJIWKKWUUl5JC5RSSimvpAVKKaWUV9ICpZRSyitpgVJKKeWVtEAppZTySlqglFJKeSUtUEoppbySFiillFJeSQuUUkopr6QFSimllFfSAqWUUsoraYFSSinllbRAKaWU8kpaoJRSSnklLVBKKaW8khYopZRSXkkLlFJKKa/kVQVKRN4Skf0iklHHehGRF0Vkm4isF5FLWjpHpZRSLcPP0wmcJB54GXinjvURwGDX41LgVdezUkp5NWMMFU5D5fGHMRgDGDAYnMZuE+DnQ1iQf7VtC485KC6rwLj2Y4y93Ol67V4OdAr2p1toYLXt9xQc40ipo0ou1PoaoEfHwBrb78gvprisotaf65zuoQQH+Dbik2g4rzqCMsYsBw6dosmtwDvGthoIF5He9e03MzOT+Ph4ABwOB5ZlMW/ePABKSkqwLIv58+cDUFhYiGVZJCQkAJCfn49lWSxcuBCAvLw8LMsiMTERgJycHCzLIikpCYCsrCwsyyI5Odn93pZlkZKSAkBGRgaWZZGamgpAeno6lmWRnp4OQGpqKpZlkZFhH0SmpKRgWRaZmZkAJCcnY1kWWVlZACQlJWFZFjk5OQAkJiZiWRZ5eXkALFy4EMuyyM/PByAhIQHLsigsLARg/vz5WJZFSUkJAPPmzcOyLBwO+8scHx+PZVnuz3Lu3LmMHz/eHc+ZM4eIiAh3HBcXx8SJE91xbGwskZGR7nj27NlERUW541mzZjF58mR3PGPGDKKjo93x9OnTiYmJccfTpk1jypQp7njq1KlMnTrVHU+ZMoVp06a545iYGKZPn+6Oo6OjmTFjhjuePHkys2bNcsdRUVHMnj3bHUdGRhIbG+uOJ06cSFxcnDuOiIhgzpw57nj8+PHMnTvXHVuWpd89L/ru3RUVxeHicnYeLGbqH2P58X2/ZemWfXyWvpuoJ1/i+il/5r/f7gKqf/dStudz+WNvM3paPHe+vopbX1nJRY+9z/DHPuCavy1j3OylDHnsI4b8IYEH/5MGVP/uvZ68nUHTv2Dwk18y5OlELpi5iIv++BUXP/MVFz/7FSOeXcwlsxYz6rkknvjE/vyrfvdeWrKVK2YvZdzspVz5l2Vc9Vf7cc3fvsaK/ZprY7/mur8nc/3fk3l9WSaUFTFx/JV88MYLcCiLtxI+5/dx7/BE3Js88+Jr/OWll3jhpX/w6kt/4d+vzOLdV2bywStPkvDK42xPmEXxlzN5+97z2fraZFj0JBvfeJC1r97P+ld/weZXJ7P1tUlkv3Ynua9FUjHvpxz67Mkz+u7VxduOoOrTF8ipEue6lu09uaGIxAAxAIGBgSevVkq1IY5KJ4UVfhSF9GbJ5n1cP7RntfUrtubzduEQjp11ASNnLXYtHQo9hnJffJorPhvC4K2VO/jZpQOqbb//SBl7fbqBD+TvcP0N7RMGQNHBElccDMDhEgcn8/WROjI3hFBGR4rpKCV0opgLju6AdZnc1CWX88tTYNmfGZ+zi0F++wmRUkIoI4RSOlR5HSJlBFNGIA781jhhDSy4EsjdAC/O5CmAhv4azLIfvxgI5C2EPLgF6q4Wu6DYOBu488YRc/LxnYeJyEDgf8aY4bWs+x8w2xizwhUvAf5gjEk7uW1Vo0ePNmlpp2yilGoF8ovK+CHvKNsOFLF9fxHbDxSz/UAReUdK3aeqOgX7s27mjdW2S80+xE9fW9Wg9+jdKYhV06+vtuzLDXv59btr69nS0JFiru4rvDyxPxQfsB8lB9m0dTtZO7PpJMWEu4pRGMWEUYK/VDb0x/deg2+Cuz847c1FZI0xZvTJy1vbEdRuoH+VuJ9rmVKqDdt+oIh73viWPYWl9bYtPOagqKyC0MATv966dAhwvw4L8qNTsD+hgX72I8iPDoF+hAbYr7uGBtTY5yV9g5l3ew9Cy/bRoWwfIaX7CDq2j8CSvfgX5+FbvA/fYwcRpwMOAv+uvv0wYFjzXKaphYB/MPgGgF8Q+AW6no/HrmW+gdXX+fiDrz/4+Lme/cHXr8ryk+Mq7cJ61p/WaWhtBWoB8LCIvI/dOaLQGFPj9J5SqvVxOg3puQWs3JrPA1edXe3Ce59Owew7WlbntiLQIyyQPuHB9AkPptRRWa1AndUlhNXTr6dzB38C/WqpFMZA0T44tAMO74Cl8+3nQzvgcDY9S/Jpnl/BgF8wBIdDUCcIOv7cEQLDIKAD+HewnwNCICAU/ENccahrmauNf7D98PGzP5A2wKsKlIi8B1hANxHJBWYC/gDGmNeAL4CbgW1ACRBd+56UUq1BRaWTb3cc4suMvSzauI8DriI0YkA4Vw3u7m4XHODL4B6h7MgvZkjvjgzuEco53UM5p3sHzu0RSr/OIQT41d3ny8/Xh16dgqDSAQe2w4EtcCATDmy2nw9uh4pjZ/4DBYRCh27Qobv9COl64nWHbhDc5aRi1NE+ilG18qoCZYyZVM96A0w5VRullPfbkV/M/NQcPlqTS35RzSOjldsOVitQAPHRY+kWGoCfbwM7H5ccgr3rTjwObIH8reCs2YmhXuILYb2hYx/Xo2/112G97CIUENL4fas6eVWBUkq1XcYYlmXuZ+7yHazKOlhrm64dArh2SA8uO7tLjXW9OgXVvfOyo5CbBnvWwp50uyAV7GxcgkHh0GUQdB4InQe5Xrvijn3Ap8UuIikXLVBKqRbx0tJt/GPxDzWWdw8L5ObhvZgwvDdjB3U5RZfsKgp3Q85q2OV67MuAhnZ17tgXup8P3Ye6nodAt8EQUrMoKs/SAqWUahG3jexL3JKtVDoNPgLXDelB1JgBWOd3r/+0XdF+yEqGrK9hx3Io3FX/G/oGQI9h0Pti6DMCel5oF6Sgjk3x46gWoAVKKdXksg4U0b9LCP5VCk//LiHcObo/IQG+3H/lIPqEB9e9A8cxyF4JWcvsorSv1uE5qxDoeQH0GwN9RtoFqftQu/u0arW0QCmlmkxZRSWvfr2dV5Zt47c3nMdD1rnV1j9/+4V1b1y0H35YBJlfwvalp+5V5x8CfUfBgMthwKV2YQrq1EQ/hfIWWqCUUk1i894j/Oa979m6vwiAuKStRAzvzaBuHereKH8bbF5gF6XcVOxhT2vh4w/9x8LZlv3oM9K+SVS1aVqglFJnxBjDvG93Met/myivONFRYWjvjjhrG0qtYBds/AQ2fAR56+vecbfz4Nwb7IJ01hUQGNr0ySuvpgVKKXXaSsor+L8P1/P5hhMDugT7+/LYhPP5+eUDT/TIKzoAGxMg42PI+bb2nYmPfcru/Jvh/Ajoek4L/ATKm2mBUkqdlrzCUu5/O5WNe464lw3pFcbLP7uEc3uEQmUF/LAE1r4DPySCs5b5hHwD7KOkYRNh8I3a1VtVowVKKdVoGbsLuf/tVPYdOTEKxN2XDuDpW4YRdCQbkmIh/T0oyqu5sfjap+0uvAOG/Eg7N6g6aYFSSjVKWUUlMe+kuYuTn48w68dDmBS+Cf77hH2fUm36jYWLo2DYrfa4dErVQwuUUqpRAv18iZs0kp+/+R3dfY/yzsgfGLj6MSjMqdm4Qw8YMQlGTIbu57V8sqpV0wKllGq0MUF7+Pq8D+mevRCf708a7FV84LwJMPIeGHyDdgdXp00LlFKqXsYYBCB7Bax8AbYl1ZwfKaQrXPILGH0fhPevsQ+lGksLlFLqlDL3FpLw39f4XciXBO77vmaD3iPg0gfhgtvB/xQjjivVSFqglFK1q3RQuPodghbHMp09cLTqSrG7hl/+sD3MUBuZwVV5Fy1QSqnqKh2w7j2cyX+jU+EuqnYCd/oG4jNiElzxG72RVjU7LVBKKVtlBax/H5b/DQ5nU3UCjCMmhMILfk7/iN9BWI2rT0o1Cy1QSrV3lRWw4QNI/isc3lFt1SETytyKWxgw4REmXTXcQwmq9koLlFLtlTGQ+QUsngEHt1VbddiEMrfiR7xdeSO3XXa+FiflEVqglGqP9q6DRU9C9jfVFjuDwnndcTMvF19PMcGMGdiZmT++wENJqvZOC5RS7cmRvbB0FqT/l2pzLwV2wnn5wzy0fQyJW0sA6Bziz4uTRlabFVeplqQFSqn2oLwYUl62b7J1lJxYLr4w5gG45g98uKmYxK0b3Kv+cecIenc6xbTsSjUzLVBKtWVOJ6yfD0uehaN7qq87LwJueNY9Rt6PLurE+txC3v12FzFXn821Q3p4IGGlTtACpVRblb0CFj1hX2+qqudwuPE5OOfaaotDA/34020X8uOL+zByQHjL5alUHbRAKdXWHNxu98zb8r/qyzv0gOufhhF3g49vnZtfdnbXZk5QqYbRAqVUW3HsMCT/Db77FzgdJ5b7BcEVj8C4RyEwrNomFZVO/LQThPJSWqCUau0qHZD6JiTPtotUVRfdBdfPgE79amxmjOFX89bQOSSAxyOG0DU0sIUSVqphtEAp1VoZA5lfwuKna9xoy4DL4aY/Qd9RdW6+eNM+kjbvByBp8z6+nnYtnUJ07iblPbzq2F5EJohIpohsE5HHa1l/r4gcEJF01+MBT+SplMftXQ9v/xjen1S9OHUeCHe+A9FfnrI4lZRX8MzCTe444sLeWpyU1/GaIygR8QVeAW4AcoFUEVlgjNl0UtP5xpiHWzxBpbzBkb2w7Dn4/l1OvtGWa/4PxsaAX/2n6l5auo3dBccA6NIhgMduOr+ZElbq9HnTEdRYYJsxJssYUw68D9zaFDvOzMwkPj4eAIfDgWVZzJs3D4CSkhIsy2L+/PkAFBYWYlkWCQkJAOTn52NZFgsXLgQgLy8Py7JITEwEICcnB8uySEpKAiArKwvLskhOTna/t2VZpKSkAJCRkYFlWaSmpgKQnp6OZVmkp6cDkJqaimVZZGRkAJCSkoJlWWRmZgKQnJyMZVlkZWUBkJSUhGVZ5OTkAJCYmIhlWeTl5QGwcOFCLMsiPz8fgISEBCzLorCwEID58+djWRYlJfbNm/PmzcOyLBwO+yJ7fHw8lmW5P8u5c+cyfvx4dzxnzhwiIiLccVxcHBMnTnTHsbGxREZGuuPZs2cTFRXljmfNmsXkyZPd8YwZM4iOjnbH06dPJyYmxh1PmzaNKVOmuOOpU6cydepUdzxlyhSmTZvmjmNiYpg+fbo7jo6OZsaMGe548uTJzJo1yx1HRUUxe/ZsdxwZGUlsbKw7njhxInFxce44IiKCOXPmuOPx48czd+5cd2xZVtN898pLOPr50xz72zD4fh7Hi1OlgZzeEfCb78nq9SOs8TfV+91bsGwVb3yT5c7x7gtCCA8J0O+efvc89nuvLt5UoPoCOVXiXNeyk0WKyHoR+UhE6pxXWkRiRCRNRNKOf+GVam0EQ7+DK+ClUYSlvkiwr9O97lj/q4lOHUrmOQ9Ah4Z3Df9vRjGOSrvABR7dzbWDQpo8b6Waghhj6m/VAkTkDmCCMeYBV3wPcGnV03ki0hUoMsaUiciDwF3GmOvq2/fo0aNNWlpac6WuVPPIXum60Ta9+vIeF8BNz8E59X71a1iz8xCRr65yx588dAUjB3Q+w0SVOjMissYYM/rk5V5zDQrYDVQ9IurnWuZmjDlYJXwD+GsL5KVUyzq4HZJmwuaF1Zd36AHXPQUjJ5/yRtu6GGN47vPN7tgeMUKLk/Je3lSgUoHBIjIIuzBFAT+r2kBEehtj9rrCicBmlGorjh2G5bHw7es1b7S9/GG4cmqNG20b44sNeXy/qwCAAF8f7RihvF6jC5SIdABKjTGVTZmIMaZCRB4GFgG+wFvGmI0i8iyQZoxZAPxGRCYCFcAh4N6mzEEpj6goh7Q3IfkvNW+0vfBO+0bb8DovtzbYV5vy3K/vHTeQ/l302pPybvVegxIRH+yjmbuBMUAZEAjkA58DrxtjttW9B8/Ta1DKKxkDmxfA4pk1plqn/2Vw05+hX933MjX+7QyJGXm8tjyLd6LH6n1PymucyTWoZUASMB3IMMY4XTvsAlwL/EVEPjHGzGvKhJVq03JS4asnIefb6svDz4IbnoFhPwGRJn1LESHiwt5EXNi7SferVHNpSIEab4xxuHrZuWczM8YcAj4GPhYR/VNMqYY4lAVJz8CmT6svDwqHax6zJw9swI22SrUH9RYoY8zxq7X/wb4HafLx608iEm2M+XeVNkqp2hQfhG/+XnOkcR9/uPRBuOr3ENKlyd/WUenEVwQfn6Y9GlOqJTTmRt0tQDLVj5geafqUlGpDSo/Asuch7mJY/Ur14nTBbfBwqj2oazMUJ4C3Vuzg5he/YdHGPLzlnkelGqoxvfiMMeY1ESkBFojI7YD+WaZUbRzH4Lu5sOKfcOxQ9XX9L7NntO0/pllTKCqr4LXk7RwucfDgf9bw4qSRTLy4T7O+p1JNqTEF6jCAMeYdV5H6HNB+qkpVVVEO379j3890dG/1dd2H2DfaDrmlyTtA1OY/q3ZyuMQ+YuvXOZgJF/Rq9vdUqik1uEAZY66v8vojESkF4psjKaVanYpyWP++XZgKdlZfF34WXPsEXPjT0xoB4nSUOip5c8WJAWEfue5cAvy8aehNpepXb4ESETG1nLw2xvwP6HaqNkq1eY5S+P4/sOIFOJJbfV1oL3sKjJE/B7+AFk3rg7Qc8ovKAejdKYjbRtacUVcpb9eg+6BE5GPgM2PMruMLRSQAuBL4Bfa9UvHNkqFS3qi8GNL+DSkvQtG+6uuCO8OVv4Uxv4SAlj8L7qh08nryiaOnmKvP1qMn1So1pEBNAO4D3nONk1cABGEPR/QV8IIx5vtmy1Apb1JyCFLfhG9fhZKD1deFdIMrHrbvZTqDMfPO1IL0PdUmI4waM8BjuSh1JhpyH1QpMAeY4+pe3g04ZowpaObclPIe+dvsbuLp70HFserrQnvBuEdh1L0eOWKqyuk0vJq83R1HXzGQ4ICWue6lVFNr7GCxAhQYY47V21Kp1s4Y2JkCq16GzC+pNsU6QKf+9gjjIyaDf5AnMqxh8eZ9bNtfBEBooB8/v3ygZxNS6gw0uECJyKPADKBURI4ArxhjXm62zJTylLIi2PAhpL0Feetrru91IVz+iH2jbQt3fqjP61WOnu6+bIAOCKtatYb04osD1gKPAkONMftFpDvwjIjMMsY83dxJKtUi9m20i9K6+VB+tOb6wTfZ15gGXtUi9zGdjn/eNYLXl2excN0e7r9ykKfTUeqMNGS6jZ8AlwBTgf3AEWA99sCxvwLGGmMO17W9N9DpNlSdyo7aM9eueRtyVtdc7xcEF90Fl0+B7q1ngr+S8gpCArxpPlKl6nba020YYz4FPhWRy4DfAnuBi4CLgS7AUhHpaIw5p2lTVqqZOCthx3JY955dnBwlNdt0HQyj74MRk+xu462MFifVFjTmWzwF+ABIxz56GgpsMMZYrnuilPJexsC+DNjwEaz/AI7uqdnGxw+G/tguTF58Gk+p9qIxQx1tFZFLgRuwj57WA4+51pU3T3pKnQFjYO86e+6lTZ/ZczHVpvtQ+0jpoigI69miKTaFwmMO3knJZtKlA+gWqnNJqbajUecBXIXoc9dDKe9TWQG539ndwjd9VnNcvONCutlj410cBb0vbtVHS+9/t4u/L/6Bl5Zt4zfXncvD1w32dEpKNQk9Ua1av+J82LoYtn4F25dAaWHt7QLC4PwJMDwSzh0Pvq2/C3ZFpZO3U7IBKK9w0iPMO+7HUqopaIFSrU95CeR8C9nfQNbXsHstNW6iPS6wI5x/Mwy7Fc65zmtuqG0qiRvz2FNYCkDXDgFMHKHzPam2ozE36gYCkcDAqtsZY55t+rSUqsJxDHJTYcc3dlHKTas+M+3JOvaFwTfYhelsC/za7nWZN1fscL+++7KzCPLXYY1U29GYI6jPgEJgDVDWPOmods8YuzNDbppdlHanQd4GcFbUvY34Qv9L7aI0+EboeUGrvqbUUGt3Heb7XQUABPj6cM9lZ3k2IaWaWGMKVD9jzIRmy0S1P85KOLQD9m2AvAy7x93uNDjWgPu+uw+xu4IPugoGXd0q71U6U29VOXqaOKIP3cPa7pGiap8aU6BSRORCY8yGZstGtU3GwNE8OLgVDmTaR0T7MmD/5tpvkq1N18Ew8Eq7IA28CkJ7NG/OXm53wTG+zMhzx/eN02GNVNvTmAJ1JXCviOzAPsUngDHGXNQsmanWxRgoLYCCXXBwmz09xcGtkL8VDm6vfWy7ugSFQ7/R0G+M/dx3VLs8QjqVd1KyqXTaHUMuP7srw/p09HBGSjW9xhSoiGbLQnm/ygoo3g9H90JBjl2ICl3Px+PGFKHjOvSAXsOh53B7lPA+l0DXc9rFNaTTVVxWwXvfuSe31kFhVZvVmJEkdorIxcBVrkXfGGPWNU9aqkVUlNnXe0oOwbFDUHwAju6zpzAv2mefljv+XHKQOrtyN0RgJ+h2rn2qruewEwWpnZ+qOx07D5bQKcSfI6UVDOwawnVD9DNUbVNj54P6JZDgWjRPRP5ljHmpWTJTp2aMff2mrMgekbv8qP1cVgTlRVB25MS6Y4ftAnS8EB0rsF87ips2J/8QexK/LoOg67nQbbBdkLoNhg7d9aioiQzr05Gvp13L4k37EAEfH/1cVdvUmFN89wOXGmOKAUTkL8AqoMkKlIhMAOIAX+ANY8zsk9YHAu8Ao4CDwF3GmOymev8Gczrt+3AqHVBZbneBrvN1uR272zvsI5eKY+Aobfzz8aJUfhSMs2V/7pBuENbLLkLh/SF8gOv1APsR0lWLUAvx9REmDO/l6TSUalaNKVACVFaJK13LmoSI+AKvYA9GmwukisgCY8ymKs3uBw4bY84VkSjgL8BdTZVDDctj4bu5NYuNqax/29bAx8/ufBDcxX4O6WoPlhra68RzaA+7KHXo3iaGBlJKtR4+jWj7b+BbEfmjiPwRWA282YS5jAW2GWOyXIPSvg/celKbW4G3Xa8/Aq4Xqf9P9szMTOLj4wFwOBxYlsW8efMAKCkpwbIs5s+fD0BhYSGWZZGQkGAfrRTl2afFyo5ARal3FSe/YMr8O5FbEkh516Fw1jj2h19C0r7OlAy7C654hC29I4nb2o8jN/4TJiew9NwZRK0eRuFDG+HpfOYP+BPWRyGUTPoEJv2XeQWjsP74JY6L7obzJxC/eD3WxJ+5i9PcuXMZP368O4U5c+YQEXGi/0xcXBwTJ050x7GxsURGRrrj2bNnExUV5Y5nzZrF5MmT3fGMGTOIjo52x9OnTycmJsYdT5s2jSlTprjjqVOnMnXqVHc8ZcoUpk2b5o5jYmKYPn26O46OjmbGjBnuePLkycyaNcsdR0VFMXv2iQP3yMhIYmNj3fHEiROJi4tzxxEREcyZM8cdjx8/nrlz57pjy7JO77sH5OfnY1kWCxcuBCBn9x4syyIxMdGOc3KwLIukpCQAsrKysCyL5ORkwP7eW5ZFSkoKABkZGViWRWpqKgDp6elYlkV6ejoAqampWJZFRkYGACkpKViWRWZmJgDJyclYlkVWlj0qfFJSEpZlkZOTA0BiYiKWZZGXZ3d/X7hwIZZlkZ+fD0BCQgKWZVFYaI+VOH/+fCzLoqTEvtVg3rx5WJaFw2GPEhIfH49lWe7PUr97nvvu5eXlNet3ry4NLlDGmH8A9wGHXI9oY8wLDd2+AfoCOVXiXNeyWtsYYyqwR7boWtvORCRGRNJEJO34F77RfOo+YnCKL/iH4AzoyOFyP0oDukD4AMrDBrCjOIijoYOgzyWUdLuY7w+HUtB1JAy+iSN9rmbp/nAO9LsJxjzA/nPv4p3snuwZ+gDc9Geyhz/K85sHsOvSZ2HS+2wc9WceXjuY7JvegSnfsfryN7hlxYVkTU6Fp/L45tK3mPzdMPb95EOI/oK1Q/7Ac5sHcuTqZ+HG59ja5zY+2d2d8vMmwrnXUxAyiLzSQAgM09Nxrcxfluxi77BJfH/AuLuYK9WW1Tvle0sRkTuACcaYB1zxPdjXvB6u0ibD1SbXFW93tck/1b5Pe8r30kIoLwbfAPt0mG+AfSTh46e/3FWLOnC0jHGzl1JeaV93/OShKxg5QO8NU23DaU/5LiIrjDFXishRqvczPn6jblPdIbgb6F8l7udaVlubXBHxAzphd5ZoHkGd7IdSHvbutzvdxWlE/3AtTqpdqPcUnzHmStdzmDGmY5VHWBMWJ4BUYLCIDHJNIR8FLDipzQLgF67XdwBLjbccAirVTEodlcxbfWLiRb0xV7UXDb4G5epWXu+y0+W6pvQwsAjYDHxgjNkoIs+KyPErn28CXUVkG/A74PGmen+lvNWCdXvILyoHoHenIO1ertqNxnQzvwH4w0nLImpZdtqMMV8AX5y0bEaV16XAT5vq/ZTydsaYaqOW/+KKgfj7NqbzrVKtV0OuQf0aeAg4W0TWV1kVBqQ0V2JKKUjZfpAtefYYh8H+vkwaM8DDGSnVchpyBPVf4EvgeaqfUjtqjDnULFkppQB445ss9+s7R/ejU4jeLK3aj3oLlDGmEPt+o0ki0hkYDAQBiAjGmOXNm6JS7dO2/UUsyzwA2Hc1ROucT6qdacxgsQ8Aj2J3/04HLsMei++6ZslMqXbu3ytPXHu6fkhPBnbr4MFslGp5jbna+igwBthpjLkWGAkUNEdSSin4+eUDuWt0fwL8fHjgKj16Uu1PY3rxlRpjSkUEEQk0xmwRkfObLTOl2rnze4Xxlzsu4g8RQ+is155UO9SYApUrIuHAp8BiETkM7DzlFkqpM9alQ4CnU1DKIxozo+5trpd/FJFl2MMMJTZLVkoppdq9xowk8TsR6QtgjEk2xixwTYuhlGoixhj+++0uCkr0v5ZSjekkEQZ8JSLfiMjDItKzuZJSqr1anXWIJz7ZwGXPL+G5/22qfwOl2rDGzAf1jDHmAmAK0BtIFpGkZstMqXbo9eXbASh1OCku96LJMZXygNMZ1Gs/kIc9zUWPpk1HqfZr054jfF3lxtyYq8/2cEZKeVZjrkE9JCJfA0uwZ7H9pTHmouZKTKn25vjRE0DE8F4M0htzVTvXmG7m/YGpxpj0ZspFqXYr51AJC9ftcce/uuYcD2ajlHdoTDfz6c2ZiFLt2dxvsnC6pt4cd25XLuoX7tF8lPIG3jTlu1LtUn5RGfNTc9zxr68514PZKOU9GjKauXvK9+ZPR6n25+2UbMoqnAAM79uRced29XBGSnkHr5nyXan2qKisgndWnRgx7NfXnIuIeDAjpbxHY7qZ31DLsoimSkSp9ijnUAnhroFgz+oawoThvTyckVLeozFTvp+jU74r1bSG9u7Ikt9dw6fpewgN9MPXR4+elDpOp3xXysP8fH24Y1Q/T6ehlNep9xSfMabQGJMNlAOFxpidxpidgBGRt5o7QaWUUu1TY65BXWSMKTgeGGMOY8+qq5RqpD0FxzDG1N9QqXasMQXKR0Q6Hw9EpAuNG4lCKYXdc+9HL37Dna+vYuW2fC1UStWhMQXq78AqEZklIs9hd5D4a/OkpVTbFb9yB4dLHKRmH+YPH6+nwqkFSqnaNGaoo3dEJA24zrXodmOMTlijVCMcKi7nteQsd/zIdefi73s6kwoo1fY19n/GXuA7YD3QTUSubvqUlGq7Xl66jaKyCgDO6d6ByEu0955SdWnwEZSIPAA8CvQD0oHLgFWcOKJSSp1CzqES/rM62x0/NmEIfnr0pFSdGvO/41FgDLDTGHMtdg++guZISqm26B+Lf8BRaV9vumRAODcO6+nhjJTybo0pUKXGmFIAEQk0xmwBzm+KJESki4gsFpGtrufOdbSrFJF012NBU7y3Ui0hY3chn6bvdsfTbx6qY+4pVY/GFKhcEQkHPgUWi8hnwM5TbtFwjwNLjDGDsWfsfbyOdseMMSNcj4lN9N5KNStjDM8u3MTx3uTjh/ZkzMAunk1KqVagwQXKGHObMabAGPNH4GngTeAnTZTHrcDbrtdvN+F+AcjMzCQ+Ph4Ah8OBZVnMmzcPgJKSEizLYv78+QAUFhZiWRYJCQkA5OfnY1kWCxcuBCAvLw/LskhMTAQgJycHy7JISkoCICsrC8uySE5Odr+3ZVmkpNjDFmZkZGBZFqmpqQCkp6djWRbp6ekApKamYlkWGRkZAKSkpGBZFpmZmQAkJydjWRZZWXZPsKSkJCzLIifHnk8oMTERy7LIy8sDYOHChViWRX5+PgAJCQlYlkVhYSEA8+fPx7IsSkpKAJg3bx6WZeFwOACIj4/Hsiz3Zzl37lzGjx/vjufMmUNExIkxg+Pi4pg48cTfDrGxsURGRrrj2bNnExUV5Y5nzZrF5MmT3fGMGTOIjo52x9OnTycmJsYdT5s2jSlTprjjqVOnMnXqVHc8ZcoUpk2b5o5jYmKYPv3EXJvR0dHMmDHDHU+ePJlZs2a546ioKGbPnu2OIyMjiY2NdccTJ04kLi7OHUdERDBnzhx3PH78eObOneuOLcti2kvv8122a1QwZyUXVPwA6HdPv3vN/91rLb/36nJaV2iNMcnGmAXGmPLT2b4WPY0xe12v84C6Ts4HiUiaiKwWkZ+caociEuNqm3b8C6+UJxQ4fPBzDQLbMW8t3QOdHs5IqdZBWuoudhFJAmqbS+BJ4G1jTHiVtoeNMTWuQ4lIX2PMbhE5G1gKXG+M2V7fe48ePdqkpaWdfvJKnaGt+47y4tJt/Om24XQM8vd0Okp5FRFZY4wZffLyFhuqyBgzvq51IrJPRHobY/aKSG9gfx372O16zhKRr7F7EtZboJTytME9w3hpkg5dqVRjeMtNGAuAX7he/wL47OQGItJZRAJdr7sB4wAdyUIppdoobylQs4EbRGQrMN4VIyKjReQNV5uhQJqIrAOWAbN1qCXlrZZl7ifnUImn01CqVfOK0ciNMQeB62tZngY84HqdAlzYwqkp1Wj7jpTym/e+p6LS8PsbzyN63CCdKVep0+AtR1BKtQnGGJ7+NIOjpRUcc1Qyb/VOHJXaa0+p06EFSqkm9OGaXL7atM8d//n2Cwny9/VgRkq1XlqglGoi2fnFPLNgozu++9IBXHFONw9mpFTrpgVKqSbgqHQydX46xeWVAJzdvQNP/WiYh7NSqnXTAqVUE/jH4h9IzykAwM9HiLtrJMEBempPqTOhBUqpM5SYkcerX5+4X/x3N57Hhf06eTAjpdoGLVBKnYFt+4uY9uE6d3zNed158OpzPJiRUm2HFiilzkB6TgEl5fYU7v27BBMXNULveVKqiXjFjbpKtVZ3jOpHn/Ag/u/D9bx69yjCQwI8nZJSbYYWKKXO0BXndGPZNIsAPz0hoVRT0v9RSjWCMYYjpTXnF9PipFTT0/9VSjXCC0lbueXFFewuOObpVJRq87RAKdVALy7ZStySrew6VMJdr69ib6EWKaWak16DUqoexhj+tiiTOVXudTqneyidtUOEUs1KC5RSp+CodPL4xxv4eG2ue9lVg7vx+j2jdBBYpZqZFiil6lBY4uCR979n+Q8H3MuuH9KDl392iRYnpVqAFiilarF57xEe/M8adlWZFfeu0f35023D8fPVS7dKtQQtUEqd5KM1uTz16QZKHScmGvzN9YP57fjBiOgoEUq1FC1QSp0kv6jMXZw6BPjy9zsvZsLw3h7OSqn2RwuUUif55VVns2TzPg4Vl/P6PaM4t0eYp1NSql3SAqXatYzdhYjABX1OTI/h6yO88rNLCAvy1zmdlPIgLVCqXdq2/ygvJG3l8w17Gd6nE59OGVdtFPIeHYM8mJ1SCrRAqXZmXU4Bb67YwcL1ezDGXrZhdyHvp+7i7kvP8mxySqlqtECpNq/UUcmijXm8nZLN2l0FNdaPH9qTy8/u2vKJKaVOSQuUarPScwr477c7+XJDHkfLKmqsv/b87vz2hvO4qF94yyenlKqXFijVZqVlH+KDtNxqywJ8fbjl4t7ce8VALUxKeTktUKrV2lt4jLTsw6zZeZiM3YW8H3NZtVEefnxxH/70xWaMgbO6hhB5ST8mjR1A97BAD2atlGooLVDK6x0tdbDzYAlb8o6yZe8R+znvCPlF5dXabck7yvC+J7qL9+wYxMxbhnFx/3BG9A/XUSCUamW0QCmPKimvIP9oOQeKSukRFkT/LiHV1t82ZyXf19KxoTZrdh6uVqAA7h03qKlSVUq1MK8oUCLyU+CPwFBgrDEmrY52E4A4wBd4wxgzu8WSVBhjKKtwUuZwUlpRSamjkrIKJ6WOSkodTsJD/DmvZ/VRFxZv2sfqrIMcOebgaGkFR0rt58JjDg4WlVFcXulu+383nc+Ua8+ttn2nYP868wn29+Xi/p0YM7ALo87qzCVndW7aH1gp5VFeUaCADOB24PW6GoiIL/AKcAOQC6SKyAJjzKbmTKyi0smTn2RgMO77ZgxgDBhOLHC94tlbLyAs6MQv1UPF5cxcsBFj3K1d7avszxX7+ghz7h5V7f23Hyjiedd1FPt9TbUcTrw2dO0QwAtRI6tt/92OQ7yQ9ANOY3A6ocLppNJpqHAaKqs8KpyGC/p05NXJ1d//P6uyee7zze42p3LLRb15+WeXVFuWsj2ff6/MPuV2xx04WlZj2YAuIQT4+dC/czCDe4Rxfq8whvYOY0ivjgzoEoKPj562U6qt8op5A4wxm40xmfU0GwtsM8ZkGWPKgfeBWxuy/8zMTOLj4wFwOBxYlsW8efMAKCkpwbIs5s+fD0BhYSGWZZGQkABAfn4+89Ny+CAtlw/X2I+P1uTy8dpcEtbuth/f7+YT1+OH7TuwLIvk5GQANm75gYXr9vC/9Xv5/Phjw16+2JDHlxn2I3FjHos27mPRxn2kpqZiWRYZGRkArPh2LUmb97Nky36WbtnPsswDLMs8wNeZB0j+wX4s/+EA32zN59sdh0hMTMSyLPLy8gBYlLySlO0HWZ11iO+yD7F2VwHrcgvZuMe+lrN1fxFZ+cXsOlTC3sJS5s2bh2VZOBwOAFatXk1ZhbPe4gRQVuEkLi6OiRMnupdlrE095TY+OAmsKObifp3o1SmIGTNmEB0d7V5/bNV7XLP/E5b83uK1e0axe9G/WPjqcwzs1gEfH2Hq1KlMnTrV3X7KlClMmzbNHcfExDB9+nR3HB0dzYwZM9zx5MmTmTVrljuOiopi9uwTB+aRkZHExsa644kTJxIXF+eOIyIimDNnjjseP348c+fOdceWZZ3Rd8+yLBYuXAhAXl4elmWRmJgIQE5ODpZlkZSUBEBWVla1715mZiaWZZGSkgJARkYGlmWRmmr/m6Snp2NZFunp6QA1vnspKSlYlkVmpv1fMzk5GcuyyMrKAiApKQnLssjJyQGo8d1buHAhlmWRn58PQEJCApZlUVhYCMD8+fOxLIuSEntKk5O/e/Hx8ViW5f4s586dy/jx493xnDlziIiIcMcnf/diY2OJjIx0x7NnzyYqKsodz5o1i8mTJ7vjk79706dPJyYmxh1PmzaNKVOmuGP97jXdd68u3nIE1RB9gZwqcS5waV2NRSQGiAEIDDyDXlsevrB+pm/fmM2dpmYROnl7XzEYRxldO3ciyN+HkiOFFB8pYPjQ8zmneygcrt5+QEAxe3ft4OEH7ycsyJ8vPvuYnVs3E/f3v9CtQyBxsbP54YdM5sXa/3FmLKm+vZ84G/UzKOVpDoeD3Nxc7rnnHnx9fdm8eTMA9913H35+fu74wQcfrBY/9NBDBAQEuONHH32UwMBAd/z73/+eoKAgd/z4448THBzsjp966ilCQkLc8cyZMwkNDWXz5s0YY6rFTqeTmTNnEhYWVi3u2LEjmzdvprKysta4U6dObN68mYqKCmbOnEl4eHi1+Hh7h8PBzJkz3flUVlYyY8YM/Pz83H+ANISYWn4pNQcRSQJ61bLqSWPMZ642XwPTarsGJSJ3ABOMMQ+44nuAS40xD9f33qNHjzZpabVe1qqX02n4IC3HlQPI8V+XYv/yPt4zTFzrb76wd7XZVo+VV/LVprzjP4P7l+3xfcmJ3SEiTBhe/SMqLHHwXfahau9xYhuhSjoE+vly+TnVR0TILyojM+8ogj0Iqp+v4Ovjg69IlVjwFSHI35denaqPQeeotE8J+vnY7bQnnFKntmPHDsLCwujatav+f6nCGMPBgwc5evQogwZV77wkImuMMaNP3qbFjqCMMePrb3VKu4H+VeJ+rmXNysdHiBo74LS3Dw7w5dYRfU97+04h/twwrOdpb98tNJBu557+EaS/rw86u7lSDVdaWsrAgQO1OJ1EROjatSsHDhxo8DZecQ2qgVKBwSIySEQCgChggYdzUkqpGrQ41a6xn4tXFCgRuU1EcoHLgc9FZJFreR8R+QLAGFMBPAwsAjYDHxhjNnoqZ6WUUs3LKwqUMeYTY0w/Y0ygMaanMeYm1/I9xpibq7T7whhznjHmHGPMnzyXsVJKeaeCgoJqvfsay7IsTveafVPzigKllFKqaZxpgfImrambuVJKtTr/XPwDcUu2NqjtpLH9ef72i6otm56wnve+O3GHzaPXD+a3N5xX5z4ef/xxtm/fzogRI7j22mtZv349hw8fxuFw8Nxzz3HrrbeSnZ1NREQEV155JSkpKfTt25fPPvuM4OBgAD788EMeeughCgoKePPNN7nqqqtO4yc/c1qglFKqDZk9ezYZGRmkp6dTUVFBSUkJHTt2JD8/n8suu8x9M/PWrVt57733mDt3LnfeeScff/yx+8bliooKvvvuO7744gueeeYZ9w25LU0LlFJKtVHGGJ544gmWL1+Oj48Pu3fvZt++fQAMGjSIESNGADBq1Ciys7Pd291+++21Lm9pWqCUUqoZ/faG8055Sq4+z99+UY3Tfg317rvvcuDAAdasWYO/vz8DBw6ktLQUqD7Cjq+vL8eOHXPHx9f5+vpSUVFzNuqWop0klFKqDQkLC+Po0aOAPcZejx498Pf3Z9myZezcudPD2TWOHkEppVQb0rVrV8aNG8fw4cMZM2YMW7Zs4cILL2T06NEMGTLE0+k1SouNxedJZzIWn1JKNcbmzZsZOnSop9PwWrV9PnWNxaen+JRSSnklLVBKKaW8khYopZRSXkkLlFJKKa+kBUoppZRX0gKllFLKK2mBUkqpNiY0NPS0t33ggQfYtGlTnevj4+PZs2dPg9ufCb1RVymllNsbb7xxyvXx8fEMHz6cPn36NKj9mdACpZRSzeWPnZpx34X1NjHG8Nhjj/Hll18iIjz11FPcddddOJ1OHn74YZYuXUr//v3x9/fnvvvu44477sCyLGJjYxk5ciT3338/aWlpiAj33Xcf/fv3Jy0tjbvvvpvg4GBWrVpFREQEsbGxjB49msTERJ544gkqKyvp1q0bS5YsOaMfUQuUUkq1UQkJCaSnp7Nu3Try8/MZM2YMV199NStXriQ7O5tNmzaxf/9+hg4dyn333Vdt2/T0dHbv3k1GRgZgT4QYHh7Oyy+/7C5IVR04cIBf/vKXLF++nEGDBnHo0KEzzl+vQSmlVBu1YsUKJk2ahK+vLz179uSaa64hNTWVFStW8NOf/hQfHx969erFtddeW2Pbs88+m6ysLB555BESExPp2LHjKd9r9erVXH311QwaNAiALl26nHH+egSllFLNpQGn4bxV586dWbduHYsWLeK1117jgw8+4K233mrRHPQISiml2qirrrqK+fPnU1lZyYEDB1i+fDljx45l3LhxfPzxxzidTvbt28fXX39dY9v8/HycTieRkZE899xzrF27Fqg+nUdVl112GcuXL2fHjh0ATXKKT4+glFKqjbrttttYtWoVF198MSLCX//6V3r16kVkZCRLlixh2LBh9O/fn0suuYROnap36Ni9ezfR0dE4nU4Ann/+eQDuvfdefvWrX7k7SRzXvXt3/vWvf3H77bfjdDrp0aMHixcvPqP8dboNpZRqQq1luo2ioiJCQ0M5ePAgY8eOZeXKlfTq1avZ37cx023oEZRSSrVDt9xyCwUFBZSXl/P000+3SHFqLC1QSinVDtV23cnbaCcJpZRqYu3h0snpaOznogVKKaWaUFBQEAcPHtQidRJjDAcPHiQoKKjB2+gpPqWUakL9+vUjNzeXAwcOeDoVrxMUFES/fv0a3F4LlFJKNSF/f3/3aArqzHjFKT4R+amIbBQRp4jU6GpYpV22iGwQkXQR0X7jSinVhnnLEVQGcDvwegPaXmuMyW/mfJRSSnmYVxQoY8xmABHxdCpKKaW8hFcUqEYwwFciYoDXjTH/qquhiMQAMa6wSEQyWyLBZtQNaO9HjvoZ2PRzsOnnYGsLn8NZtS1ssQIlIklAbbcqP2mM+ayBu7nSGLNbRHoAi0VkizFmeW0NXcWrzgLW2ohIWm1DgbQn+hnY9HOw6edga8ufQ4sVKGPM+CbYx27X834R+QQYC9RaoJRSSrVuXtGLryFEpIOIhB1/DdyI3blCKaVUG+QVBUpEbhORXOBy4HMRWeRa3kdEvnA16wmsEJF1wHfA58aYRM9k7BFt5nTlGdDPwKafg00/B1ub/RzaxXQbSimlWh+vOIJSSimlTqYFSimllFfSAtUKicjvRcSISDdP59LSRORvIrJFRNaLyCciEu7pnFqSiEwQkUwR2SYij3s6n5YmIv1FZJmIbHINj/aop3PyJBHxFZHvReR/ns6lOWiBamVEpD92D8Zdns7FQxYDw40xFwE/ANM9nE+LERFf4BUgAhgGTBKRYZ7NqsVVAL83xgwDLgOmtMPPoKpHgc2eTqK5aIFqff4JPIY9qka7Y4z5yhhT4QpXAw0fu7/1GwtsM8ZkGWPKgfeBWz2cU4syxuw1xqx1vT6K/cu5r2ez8gwR6Qf8CHjD07k0Fy1QrYiI3ArsNsas83QuXuI+4EtPJ9GC+gI5VeJc2ukvZwARGQiMBL71cCqe8gL2H6tOD+fRbFrbWHxt3qmGhAKewD6916Y1ZFgsEXkS+3TPuy2Zm/IOIhIKfAxMNcYc8XQ+LU1EbgH2G2PWiIjl4XSajRYoL1PXkFAiciEwCFjnGvW9H7BWRMYaY/JaMMVmV9+wWCJyL3ALcL1pXzfy7Qb6V4n7uZa1KyLij12c3jXGJHg6Hw8ZB0wUkZuBIKCjiMwzxkz2cF5NSm/UbaVEJBsY3d7mxhKRCcA/gGuMMe1qTm0R8cPuGHI9dmFKBX5mjNno0cRakNh/nb0NHDLGTPVwOl7BdQQ1zRhzi4dTaXJ6DUq1Ni8DYdij2aeLyGueTqiluDqHPAwswu4c8EF7Kk4u44B7gOtc//7prqMI1QbpEZRSSimvpEdQSimlvJIWKKWUUl5JC5RSSimvpAVKKaWUV9ICpZRSyitpgVJKKeWVtEAppZTySlqglPIAEQkXkYdOsT6lpd9TKW+jBUopzwgH6iwWxpgrWvo9lfI2WqCUagIiMlBENovIXNdMr1+JSLBr3WQR+c41LM/rrokHZwPnuJb9rZb9FZ1qv67lW0TkXdf6j0QkpMo2GVX2NU1E/tiA91xaZfigUhG5s1k+LKUaSAuUUk1nMPCKMeYCoACIFJGhwF3AOGPMCKASuBt4HNhujBlhjPm/xu7Xtfx8YI4xZihwhPqPjk75nsaY61w5vg4swB4xXCmP0QKlVNPZYYxJd71eAwzEHnl8FJAqIumu+Owm2C9AjjFmpev1PODK00m6KhH5OfaU8ncbYyrPdH9KnQmdD0qpplNW5XUlEAwI8LYxZnrVhq7ZYM9kvwAnj/R8PK6g+h+fQQ15ExH5KfbR3a3GGEcj8lOqWegRlFLNawlwh4j0ABCRLiJyFnAUe9qQMzFARC53vf4ZsML1eh/QQ0S6ikgg9uSOnOo9XTO0PgTcbowpPcO8lGoSWqCUakbGmE3AU8BXIrIeWAz0NsYcBFaKSEZtHRYaKBOYIiKbgc7Aq673dADPAt+53m+La/mp3vNt7Bl6V7o6Sdx/mjkp1WR0PiilWiHXKcL/GWOGezoXpZqLHkEppZTySnoEpZRSyivpEZRSSimvpAVKKaWUV9ICpZRSyitpgVJKKeWVtEAppZTySlqglFJKeSUtUEoppbzS/wMO1abiEV6QNQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#로지스틱 함수와 하이퍼볼릭 탄젠트 함수 비교\n",
    "z = np.arange(-5,5,0.005)\n",
    "log_act = logistic(z)#로지스틱 함수\n",
    "tanh_act = tanh(z)#하이퍼볼릭 탄젠트\n",
    "\n",
    "plt.ylim([-1.5,1.5])\n",
    "plt.xlabel('net input $z$')\n",
    "plt.ylabel('activation $\\phi(z)$')\n",
    "plt.axhline(1,color = 'black', linestyle = ':')\n",
    "plt.axhline(0.5,color = 'black', linestyle = ':')\n",
    "plt.axhline(0,color = 'black', linestyle = ':')\n",
    "plt.axhline(-0.5,color = 'black', linestyle = ':')\n",
    "plt.axhline(-1,color = 'black', linestyle = ':')\n",
    "\n",
    "plt.plot(z, tanh_act,\n",
    "        linewidth = 3,\n",
    "        linestyle = '--',\n",
    "        label = 'tanh')\n",
    "plt.plot(z, log_act, \n",
    "        linewidth = 3,\n",
    "        label = 'logistic')\n",
    "\n",
    "plt.legend(loc = 'lower right')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "a7d2dc8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#넘파이 tanh 함수\n",
    "tanh_act = np.tanh(z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "5d22e5d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#사이파이 로지스틱 함수\n",
    "from scipy.special import expit\n",
    "log_act = expit(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca12877",
   "metadata": {},
   "source": [
    "### 렐루 활성화 함수(Rectified Linear Unit, ReLU)\n",
    "$$\\phi(z) = max(0,z)$$\n",
    "\n",
    "- 하이퍼볼릭 탄젠트와 로지스틱 활성화 함수의 그래디언트 소실 문제(vanishing gradient problem)을 개선\n",
    "- 그래디언트 소실 문제: 그래디언트가 0에 아주 가까워지기 때문에 훈련 과정 동안 가중치가 매우 느리게 학습됨\n",
    "- 입력 값이 양수면 입력에 대한 렐루의 도함수는 항상 1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}