10. Intro to Deep Learning/exercise/02. exercise-deep-neural-networks.ipynb

{"metadata":{"jupytext":{"cell_metadata_filter":"-all","formats":"ipynb"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Intro to Deep Learning](https://www.kaggle.com/learn/intro-to-deep-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/deep-neural-networks).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"# Introduction #\n\nIn the tutorial, we saw how to build deep neural networks by stacking layers inside a `Sequential` model. By adding an *activation function* after the hidden layers, we gave the network the ability to learn more complex (non-linear) relationships in the data.\n\nIn these exercises, you'll build a neural network with several hidden layers and then explore some activation functions beyond ReLU. Run this next cell to set everything up!","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\n\n# Setup plotting\nimport matplotlib.pyplot as plt\n\nplt.style.use('seaborn-whitegrid')\n# Set Matplotlib defaults\nplt.rc('figure', autolayout=True)\nplt.rc('axes', labelweight='bold', labelsize='large',\n       titleweight='bold', titlesize=18, titlepad=10)\n\n# Setup feedback system\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.deep_learning_intro.ex2 import *","metadata":{"execution":{"iopub.status.busy":"2022-07-16T02:09:32.219202Z","iopub.execute_input":"2022-07-16T02:09:32.220662Z","iopub.status.idle":"2022-07-16T02:09:32.230909Z","shell.execute_reply.started":"2022-07-16T02:09:32.220601Z","shell.execute_reply":"2022-07-16T02:09:32.229826Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"markdown","source":"In the *Concrete* dataset, your task is to predict the compressive strength of concrete manufactured according to various recipes.\n\nRun the next code cell without changes to load the dataset.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n\nconcrete = pd.read_csv('../input/dl-course-data/concrete.csv')\nconcrete.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-16T02:09:37.679197Z","iopub.execute_input":"2022-07-16T02:09:37.679623Z","iopub.status.idle":"2022-07-16T02:09:37.734202Z","shell.execute_reply.started":"2022-07-16T02:09:37.679590Z","shell.execute_reply":"2022-07-16T02:09:37.732819Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# 1) Input Shape #\n\nThe target for this task is the column `'CompressiveStrength'`. The remaining columns are the features we'll use as inputs.\n\nWhat would be the input shape for this dataset?","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE\ninput_shape = [8]\n\n# Check your answer\nq_1.check()","metadata":{"lines_to_next_cell":2,"execution":{"iopub.status.busy":"2022-07-16T02:09:40.240351Z","iopub.execute_input":"2022-07-16T02:09:40.241163Z","iopub.status.idle":"2022-07-16T02:09:40.250398Z","shell.execute_reply.started":"2022-07-16T02:09:40.241112Z","shell.execute_reply":"2022-07-16T02:09:40.248601Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_1.hint()\n#q_1.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2) Define a Model with Hidden Layers #\n\nNow create a model with three hidden layers, each having 512 units and the ReLU activation.  Be sure to include an output layer of one unit and no activation, and also `input_shape` as an argument to the first layer.","metadata":{}},{"cell_type":"code","source":"from tensorflow import keras\nfrom tensorflow.keras import layers\n\n# YOUR CODE HERE\nmodel = keras.Sequential([\n    layers.Dense(units=512, activation=\"relu\", input_shape=input_shape),\n    layers.Dense(units=512, activation=\"relu\"),\n    layers.Dense(units=512, activation=\"relu\"),\n    layers.Dense(units=1),\n])\n\n# Check your answer\nq_2.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-16T02:09:43.600413Z","iopub.execute_input":"2022-07-16T02:09:43.600952Z","iopub.status.idle":"2022-07-16T02:09:45.428338Z","shell.execute_reply.started":"2022-07-16T02:09:43.600812Z","shell.execute_reply":"2022-07-16T02:09:45.426984Z"},"trusted":true},"execution_count":5,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_2.hint()\n#q_2.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-16T00:48:19.362293Z","iopub.execute_input":"2022-07-16T00:48:19.362683Z","iopub.status.idle":"2022-07-16T00:48:19.368416Z","shell.execute_reply.started":"2022-07-16T00:48:19.362644Z","shell.execute_reply":"2022-07-16T00:48:19.367131Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"markdown","source":"# 3) Activation Layers #\n\nLet's explore activations functions some.\n\nThe usual way of attaching an activation function to a `Dense` layer is to include it as part of the definition with the `activation` argument. Sometimes though you'll want to put some other layer between the `Dense` layer and its activation function. (We'll see an example of this in Lesson 5 with *batch normalization*.) In this case, we can define the activation in its own `Activation` layer, like so:\n\n```\nlayers.Dense(units=8),\nlayers.Activation('relu')\n```\n\nThis is completely equivalent to the ordinary way: `layers.Dense(units=8, activation='relu')`.\n\nRewrite the following model so that each activation is in its own `Activation` layer.","metadata":{}},{"cell_type":"code","source":"### YOUR CODE HERE: rewrite this to use activation layers\nmodel = keras.Sequential([\n    layers.Dense(32, input_shape=[8]),\n    layers.Activation('relu'),\n    layers.Dense(32),\n    layers.Activation('relu'),\n    layers.Dense(1),\n])\n\n# Check your answer\nq_3.check()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-16T02:10:51.921227Z","iopub.execute_input":"2022-07-16T02:10:51.921751Z","iopub.status.idle":"2022-07-16T02:10:51.972510Z","shell.execute_reply.started":"2022-07-16T02:10:51.921707Z","shell.execute_reply":"2022-07-16T02:10:51.971010Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q_3.hint()\n#q_3.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-16T02:10:54.988827Z","iopub.execute_input":"2022-07-16T02:10:54.989232Z","iopub.status.idle":"2022-07-16T02:10:54.993981Z","shell.execute_reply.started":"2022-07-16T02:10:54.989201Z","shell.execute_reply":"2022-07-16T02:10:54.992759Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"markdown","source":"# Optional: Alternatives to ReLU #\n\nThere is a whole family of variants of the `'relu'` activation -- `'elu'`, `'selu'`, and `'swish'`, among others -- all of which you can use in Keras. Sometimes one activation will perform better than another on a given task, so you could consider experimenting with activations as you develop a model. The ReLU activation tends to do well on most problems, so it's a good one to start with.\n\nLet's look at the graphs of some of these. Change the activation from `'relu'` to one of the others named above. Then run the cell to see the graph. (Check out the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/activations) for more ideas.)","metadata":{}},{"cell_type":"code","source":"# YOUR CODE HERE: Change 'relu' to 'elu', 'selu', 'swish'... or something else\nactivation_layer = layers.Activation('relu')\n\nx = tf.linspace(-3.0, 3.0, 100)\ny = activation_layer(x) # once created, a layer is callable just like a function\n\nplt.figure(dpi=100)\nplt.plot(x, y)\nplt.xlim(-3, 3)\nplt.xlabel(\"Input\")\nplt.ylabel(\"Output\")\nplt.show()","metadata":{"lines_to_next_cell":0,"execution":{"iopub.status.busy":"2022-07-16T02:10:57.989930Z","iopub.execute_input":"2022-07-16T02:10:57.990320Z","iopub.status.idle":"2022-07-16T02:10:58.397873Z","shell.execute_reply.started":"2022-07-16T02:10:57.990287Z","shell.execute_reply":"2022-07-16T02:10:58.396727Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"markdown","source":"# Keep Going #\n\nNow move on to Lesson 3 and [**learn how to train neural networks**](https://www.kaggle.com/ryanholbrook/stochastic-gradient-descent) with stochastic gradient descent.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intro-to-deep-learning/discussion) to chat with other learners.*","metadata":{}}]}