notebook/ml_engine_presentation_bash.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scaling ML using Cloud ML Engine\n",
    "\n",
    "- Documentation [enryh.github.io/gcp_ml_engine_talk](https://enryh.github.io/gcp_ml_engine_talk/)\n",
    "- Repository with notebooks [github.com/enryh/gcp_ml_engine_talk](https://github.com/enryh/gcp_ml_engine_talk)\n",
    "- orignal project [github.com/tarrade/proj_DL_models_and_pipelines_with_GCP](https://github.com/tarrade/proj_DL_models_and_pipelines_with_GCP)\n",
    "\n",
    "Me on Twitter: [@Henrywebel](https://twitter.com/Henrywebel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Contents\n",
    "\n",
    "2. Data Science Workflow\n",
    "3. Developing code\n",
    "4. Hands-ON Tutorial: Running MNIST on ML-Engine\n",
    "5. Setup Runtime for Notebook\n",
    "6. Load Data from BQ\n",
    "7. Package Model\n",
    "8. Train using ML-Engine\n",
    "9. Deployment\n",
    "10. Predictions\n",
    "11. Recap\n",
    "12. Appendix: Jupyter Slides"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Shortcut: Run first cells and jump to any part in the notebook\n",
    "\n",
    "> Will only work after initial setup (see below) !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Fragment to initalize working with this notebook on the CLOUD\n",
    "# check working directory\n",
    "from utils import chdir_\n",
    "pwd = chdir_()\n",
    "## Import Tensorflow\n",
    "try:\n",
    "    import tensorflow as tf\n",
    "except ModuleNotFoundError:\n",
    "    raise ModuleNotFoundError(\"Install Tensorflow\")\n",
    "tf.__version__ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## import config:\n",
    "import yaml\n",
    "from pprint import pprint\n",
    "with open(\"config.yaml\", \"r\", encoding = \"utf8\") as f:\n",
    "    config = yaml.safe_load(f)\n",
    "pprint(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "## setup env-variables \n",
    "import os\n",
    "import platform\n",
    "PROJECT = config['project-id'] \n",
    "REGION = config['region'] # Choose an available region for Cloud MLE from https://cloud.google.com/ml-engine/docs/regions.\n",
    "BUCKET = config['bucket'] # REPLACE WITH YOUR BUCKET NAME. Use a regional bucket in the region you selected.\n",
    "PKG_NAME = config['pkg-name']\n",
    "TEST_DATA_JSON = config['testdatafile']\n",
    "\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['REGION'] = REGION \n",
    "os.environ['TFVERSION'] = str(config['tf-version'])  # Tensorflow version 1.4 before\n",
    "os.environ['PKG_NAME'] = PKG_NAME\n",
    "os.environ['TEST_DATA_JSON'] = TEST_DATA_JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Set new OUTPUT and DATA directory on GS\n",
    "OUTDIR = '/'.join(['gs:/', BUCKET, PKG_NAME, 'trained'])\n",
    "DATA = '/'.join(['gs:/', BUCKET, PKG_NAME, 'data', 'mnist.npz'])\n",
    "%env OUTDIR $OUTDIR\n",
    "%env DATA $DATA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "local_python = sys.executable\n",
    "%env PYTHON_LOCAL $local_python "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION\n",
    "gcloud config set ml_engine/local_python $PYTHON_LOCAL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud info "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data Science Workflow (DSP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Goal is to standardise the development of models\n",
    "     - Checklist of necessary technical steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> Vision: Achieve an first end-to-end model in production within a *productincrement* of 10 weeks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> Scale out: Scale without having to rewrite your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data Science Pipeline (DSP) - Checklist\n",
    "\n",
    "- further [documentation](https://confluence.axa.com/confluence/pages/viewpage.action?pageId=112334644)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### [Scaling Michelangelo](https://eng.uber.com/scaling-michelangelo/) - Data Science Process at Uber\n",
    "![Data Science Process at Uber](Images/uber_michelangelo_at_scale.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%javascript // some javascript to render markdown tables properly\n",
    "\n",
    "// from https://github.com/jupyter/notebook/issues/3024#issuecomment-435630413\n",
    "var marked = require('components/marked/lib/marked');\n",
    "\n",
    "if (marked.Renderer.name !== 'NonExtensibleTableRenderer') {\n",
    "    function tablecell(content, flags) {\n",
    "        var type = flags.header ? 'th' : 'td';\n",
    "        var style = flags.align == null ? '' : ' style=\"text-align: ' + flags.align + '\"';\n",
    "        var start_tag = '<' + type + style + '>';\n",
    "        var end_tag = '</' + type + '>\\n';\n",
    "        return start_tag + content + end_tag;\n",
    "    }\n",
    "\n",
    "    var DefaultRenderer = marked.Renderer;\n",
    "    function NonExtensibleTableRenderer(options) {\n",
    "        DefaultRenderer.call(this, options);\n",
    "        Object.defineProperty(this, 'tablecell', {\n",
    "            get: function () { return tablecell; },\n",
    "            set: function () { } // No-op, sorry for this hack but we must prevent it from being redefined\n",
    "        });\n",
    "    }\n",
    "    NonExtensibleTableRenderer.prototype = Object.create(DefaultRenderer.prototype);\n",
    "    NonExtensibleTableRenderer.prototype.constructor = NonExtensibleTableRenderer;\n",
    "\n",
    "    marked.setOptions({\n",
    "        renderer: new NonExtensibleTableRenderer()\n",
    "    });\n",
    "    // Look away... it has to be done as newer versions of the notebook build a custom\n",
    "    // renderer rather than extending the default.\n",
    "    marked.Renderer = NonExtensibleTableRenderer;\n",
    "}\n",
    "\n",
    "var Jupyter = require('base/js/namespace');\n",
    "Jupyter.notebook.get_cells()\n",
    "   .filter(cell => cell.cell_type === 'markdown' && cell.rendered)\n",
    "   .forEach(mdcell => {\n",
    "       mdcell.unrender();\n",
    "       mdcell.render();\n",
    "   });"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "|   Step 1: Preparation                   |      Step 2: Data exploration and model building                   |    Step 3: Model deployment                    \n",
    "|   :------------------                   |      :-------------------------------------------                  |   :-----------------------------------------  \n",
    "| 1.1  Define business and project goal   | 2.1  Define and setup ML project infrastructure                    | 3.1  Model industralization                             \n",
    "| 1.2  Quick data exploration             | 2.2  Data exploration and visualizaiton                            | 3.2  Gather and analyze insightbalancing ...)     \n",
    "| 1.3  ML models strategy                 | 2.3  Build and evaluate a model                                    | -\n",
    "|         -                              | 2.4  Interpretability of ML model                                  | -\n",
    "|        -                                | 2.5  Productionize and deploy the ML models                        | -                                          \n",
    "> steps 1 and 2 can be done *only* locally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Developing code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Using your own laptop:\n",
    "- Cloud SDK on your laptop (CLI)\n",
    "- your IDE (e.g. PyCharme)\n",
    "- Juypter Notebook\n",
    "- your conda env\n",
    "- `gcloud ml-engine local` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Simple Cloud setup using\n",
    "- [Google Console](https://console.cloud.google.com/) -Compute Engine with 5 GB storage\n",
    "- Cloud Editor\n",
    "- datalab, [Deep Learning VM](https://cloud.google.com/deep-learning-vm/)\n",
    "- env ([runtime](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list)) by google\n",
    "- `gcloud ml-engine` (`local`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "|Cloud SDK on Laptop | Google Console |\n",
    "|---------------------|----------------|\n",
    "| your machine | Tiny Compute Engine with 5 GB storage |\n",
    "| Your IDE | Code Editor|\n",
    "| Jupyter Notebook | Datalab, [Deep Learning VM](https://cloud.google.com/deep-learning-vm/) |\n",
    "| your conda env | env ([runtime](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list)) by google |\n",
    "| `gcloud ml-engine local` | `gcloud ml-engine`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Your laptop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![laptop-icon-24](Images/laptop-icon-24.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Call your python script (module) in your conda env\n",
    "2. Use `gcloud ml-engine local train`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## AI Platform Notebooks: Deep Learning VM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![Deep Learning VM](Images/deep-learning-overview_2x.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Preconfigured (Deep Learning) VMs for ML prottyping\n",
    "    - only CPUs possible\n",
    "- you use a preconfigured runtime compatible to ML Engine runtimes for deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## A cluster of machines using ML-Engine service"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![CloudMachineLearning.png](Images/CloudMachineLearning.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- runs a script \"autonomously\" on the cloud and stops afterwards\n",
    "- offers to run different type of clusters \n",
    "- invoked by `gcloud ml-engine train`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> develop on your laptop if you are comfortable with setting up your environements\n",
    "\n",
    "> otherwise develop on a preconfigured Notebook instance without too many compute attached to it\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    ">Migrate to ML-Engine Cluster on GCP to \n",
    ">   - distribute learning on several machines\n",
    ">   - serve model 24/7\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hands-ON Tutorial: Running MNIST on ML-Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"Images/gcp_training_options-Modelling.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- deep dive into step 2 and 3 of proposed Data Science process\n",
    "- data exploration is omitted since a curated dataset is used\n",
    "- Some title reference to previously described Data Science Process, e.g. DSP 2.3 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Adapted from [Notebook](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/cloudmle/cloudmle.ipynb) of Google Coursera Course [Serverless Machine Learning with Tensorflow on Google Cloud Platform](https://www.coursera.org/learn/serverless-machine-learning-gcp/). The current code respository is [github/tarrade/tarrade/proj_DL_models_and_pipelines_with_GCP/](https://github.com/tarrade/proj_DL_models_and_pipelines_with_GCP/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](https://ml4a.github.io/images/figures/mnist-input.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- black and white images are numeric vectors (Feat 1- 784)\n",
    "- ten labels (Figures 0-9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- recognise hand-written digits (e.g. on a postal card) \n",
    "- standardise inputs to 0 - 1 range (e.g. using BEAM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## GCP services used in Tutorial\n",
    "We will look today at following GCP Services-\n",
    "- [BigQuery](https://console.cloud.google.com/bigquery) (BQ)\n",
    "- [Cloudstorage](https://console.cloud.google.com/storage) (Buckets)\n",
    "- [ML Engine](https://console.cloud.google.com/mlengine/)\n",
    "- If time allows: Dataflow using Apache Beam"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from src.utils.mnist_utils import plot_mnist_testdata \n",
    "plot_mnist_testdata()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "ToDo: export in readable yaml format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# DSP 2.1: Setup\n",
    "\n",
    "1. ML Engine Runtimes\n",
    "2. Repository Structure\n",
    "3. Configuration Variables\n",
    "    - Environment variables to set\n",
    "    - How to add them to your runtime\n",
    "4. Setup `gcloud` runtime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    ">Create conda environment\n",
    ">  ```\n",
    ">  conda env create -f environment.yml -n gcp_dl\n",
    ">  conda activate gcp_dl\n",
    ">  jupyter notebook \n",
    ">  ```\n",
    "> Starts notebook-server with all packages in your current path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Change working directory**\n",
    "\n",
    "- In order to import from `src` functionality later in this notebook, it is necessary to change to the root directory of the notebooks directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "center",
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# check working directory\n",
    "import os\n",
    "WORKINGDIR = os.path.normpath(os.getcwd())\n",
    "print(\"Current Working direcotory:\\t{}\".format(WORKINGDIR))\n",
    "folders = WORKINGDIR.split(os.sep)\n",
    "if folders.pop() in ['notebook', 'src', 'talks']:\n",
    "  WORKINGDIR = os.sep.join(folders)\n",
    "  print(\"Changed to New working directory:\\t{dir}\".format(dir=WORKINGDIR))\n",
    "  os.chdir(WORKINGDIR) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## ML Engine Runtimes\n",
    "Default ML-Engine Runtimes depend on the Tensorflow Version\n",
    "- [list of runtimes](https://cloud.google.com/ml-engine/docs/tensorflow/runtime-version-list)\n",
    "- Current Version: `1.13`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#!conda install tensorflow=1.13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "TF_VERSION=$(python3 -c 'import tensorflow as tf; print(tf.__version__)')\n",
    "if $TF_VERSION != \"1.13.1\"\n",
    "then\n",
    "    echo \"Install Tensorflow using pip: pip install tensorflow==1.13\"\n",
    "fi\n",
    "    echo \"Found Tensorflow: $TF_VERSION\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- current version of gcp datalab\n",
    "- will be different on Windows machines\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Repository structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ls | grep \"DIR\\|yaml\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Key Directories containing information\n",
    "```\n",
    ".\n",
    "+-- data\n",
    "+-- src\n",
    "|  +-- models\n",
    "|  +-- packages\n",
    "config.yaml\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In the next step the contents of [`config.yaml`](config.yaml) will be important"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## GCP Environment Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- `PROJECT_ID`: unique ID that identifies your project, e.g. **ml-productive-pipeline-12345**\n",
    "- `BUCKET`: BLOB-store ID. Each project has per default an bucket named by the `PROJECT_ID`\n",
    "- `REGION`: Which data center to use\n",
    "\n",
    "> All Cloud-ML-Engine Services are only available in [`europe-west1`](https://cloud.google.com/ml-engine/docs/tensorflow/regions)\n",
    "\n",
    "- all products per Region in europe: [link](https://cloud.google.com/about/locations/?region=europe#region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# #Create config manually and save as yaml:\n",
    "config = {}\n",
    "config['project-id'] = 'ml-productive-pipeline-12345'  # # REPLACE WITH YOUR PROJECT ID\n",
    "config['region'] = 'europe-west1' # Choose an available region for Cloud MLE from https://cloud.google.com/ml-engine/docs/regions.\n",
    "config['bucket'] = 'ml-productive-pipeline-12345'  # REPLACE WITH YOUR BUCKET NAME. Use a regional bucket in the region you selected.\n",
    "config['pkg-name'] = 'pkg_mnist_fnn'\n",
    "config['tf-version'] = '1.13'\n",
    "config['env-name'] = 'gcp_dl'\n",
    "with open(\"config.yaml\", 'w', encoding= 'utf8') as f:\n",
    "      yaml.dump(config, stream=f,  default_flow_style=False) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**ML-Engine Environment Variables**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Additional Environment Variables needed for ML-Engine\n",
    "- `PKG_NAME`: Package Name which will contain your model\n",
    "- `TF_VERSION`: Tensorflow Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import yaml\n",
    "from pprint import pprint\n",
    "with open(\"config.yaml\", \"r\", encoding = \"utf8\") as f:\n",
    "    config = yaml.load(f)\n",
    "pprint(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Environment variables for project and bucket\n",
    "\n",
    "Note that:\n",
    "1. Your project id is the *unique* string that identifies your project (not the project name). You can find this from the GCP Console dashboard's Home page. My dashboard reads:  \n",
    "     \n",
    "     - Project ID: ml-productive-pipeline-12345\n",
    "     \n",
    "2. Cloud training often involves saving and restoring model files. If you don't have a bucket already, I suggest that you create one from the GCP console (because it will dynamically check whether the bucket name you want is available). A common pattern is to prefix the bucket name by the project id, so that it is unique. Also, for cost reasons, you might want to use a single region bucket.\n",
    "\n",
    "\n",
    "Add all detail in to [config.yaml](../config.yaml) file in main directory. Missing in public repository.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adding Environment Variables to your runtime\n",
    "- add variables **persistently**  to the runtime of your kernel from jupyter (or datalab)\n",
    "- use `os.environ` dictionary\n",
    "- behind a proxy, configure globally\n",
    "  - `REQUESTS_CA_BUNDLE`: optional, filepath to your SLL-certificate (works for `request`-package)\n",
    "  - `HTTPS_PROXY`: optional, link to your proxy, possibly includign authentification or ports\n",
    "- possiblity to set `environment variables` for user permanently"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "## setup env-variables \n",
    "import os\n",
    "import platform\n",
    "PROJECT = config['project-id'] \n",
    "REGION = config['region'] # Choose an available region for Cloud MLE\n",
    "BUCKET = config['bucket'] \n",
    "PKG_NAME = config['pkg-name']\n",
    "#TEST_DATA_JSON = config['testdatafile'] # added later\n",
    "\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['REGION'] = REGION\n",
    "os.environ['TFVERSION'] = str(config['tf-version']) \n",
    "os.environ['PKG_NAME'] = PKG_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Access Environment Variables**\n",
    "- Now, you can access the environement variable in the terminal where your jupyter, datalab or ipython."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!echo \"Using Tensorflow Version: $TFVERSION\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Setup gcloud runtime\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "local_python = sys.executable\n",
    "%env PYTHON_LOCAL $local_python "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud config set project $PROJECT\n",
    "gcloud config set compute/region $REGION\n",
    "## ensure we predict locally with our current Python environment\n",
    "gcloud config set ml_engine/local_python $PYTHON_LOCAL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Access Control \n",
    "\n",
    "- sign in and let clients pick up credentials from GCloud SDK (this stores a json with your credentials on your machine)\n",
    "    ```\n",
    "    gcloud auth application-default login\n",
    "    ```\n",
    "\n",
    "- Service Accounts ([Creating and Managing Service Accounts](https://cloud.google.com/iam/docs/creating-managing-service-accounts))\n",
    "  - need be assigned read/write permission to `BUCKET`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Load Data: Bigquery Client (DSP 2.2 )\n",
    "\n",
    "There are several python clients available, see list. Here we use `bigquery` to load some data.\n",
    "\n",
    "Picks up  PROXY_HTTPS, REQUESTS_CA_BUNDLE, PROJECT_ID from environment\n",
    "\n",
    "- set all relevant variables as user environment variables\n",
    "    1. search \"env\" in windows search bar (press windows button)\n",
    "    2. select \"Edit environment variables for your account\"\n",
    "    3. select \"new\" and add the PROXY_HTTPS, REQUESTS_CA_BUNDLE, PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Example: Download from public dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# pip install --upgrade google-cloud-bigquery\n",
    "from google.cloud import bigquery\n",
    "import os\n",
    "\n",
    "PROJECT_ID = os.environ['PROJECT']\n",
    "print(\"# Current project in use: {}\\n\".format(PROJECT_ID))\n",
    "client = bigquery.Client(project=PROJECT_ID)\n",
    "\n",
    "sql = \"\"\"\n",
    "    SELECT *\n",
    "    FROM `bigquery-public-data.usa_names.usa_1910_current`\n",
    "    WHERE state = 'TX'\n",
    "    LIMIT 10\n",
    "\"\"\"\n",
    "df = client.query(sql).to_dataframe()\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Download from project table\n",
    "- use `test` Dataset with table `DATA` of project (has to be created)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sql = \"\"\"\n",
    "    SELECT *\n",
    "    FROM `{project}.test.DATA`\n",
    "    LIMIT 15\n",
    "\"\"\".format(project=PROJECT)\n",
    "df = client.query(sql).to_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sql = \"\"\"\n",
    "    SELECT COUNT(label) as count\n",
    "    FROM `{project}.test.DATA`\n",
    "    GROUP BY label\n",
    "\"\"\".format(project=PROJECT)\n",
    "df = client.query(sql).to_dataframe()\n",
    "df.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Downloading the entire table to pandas\n",
    "- BQ Query Default limit of128MB maximum reponse size, see [quotas](https://cloud.google.com/bigquery/quotas), does not allow to download entire Table\n",
    "- [`bigquery_storage`](https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas#install_the_client_libraries) client has to be used to download large datasets"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "!pip install --upgrade google-cloud-bigquery[pandas]\n",
    "!pip install --upgrade google-cloud-bigquery-storage[fastavro,pandas]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import google.auth\n",
    "from google.cloud import bigquery\n",
    "from google.cloud import bigquery_storage_v1beta1\n",
    "\n",
    "# Explicitly create a credentials object. This allows you to use the same\n",
    "# credentials for both the BigQuery and BigQuery Storage clients, avoiding\n",
    "# unnecessary API calls to fetch duplicate authentication tokens.\n",
    "credentials, _ = google.auth.default(\n",
    "    scopes=[\"https://www.googleapis.com/auth/cloud-platform\"]\n",
    ")\n",
    "print(\"Credentials: {}\".format(credentials))\n",
    "print(\"PROJECT: {}\".format(PROJECT))\n",
    "\n",
    "# Make clients.\n",
    "client = bigquery.Client(\n",
    "    credentials=credentials,\n",
    "    project=PROJECT\n",
    ")\n",
    "bqstorageclient = bigquery_storage_v1beta1.BigQueryStorageClient(\n",
    "    credentials=credentials\n",
    ") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Download to pandas dataframe\n",
    "- can take very long"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Download a table.\n",
    "table = bigquery.TableReference.from_string(\n",
    "    \"{project}.test.DATA\".format(project=PROJECT)\n",
    ")\n",
    "rows = client.list_rows(\n",
    "    table,\n",
    "    #selected_fields=[ \n",
    "    #    bigquery.SchemaField(\"label\", \"INTEGER\")\n",
    "    #],\n",
    ")\n",
    "df = rows.to_dataframe(bqstorage_client=bqstorageclient)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.save(file='data/mnist/raw/mnist_all', allow_pickle=True, arr=df.to_numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Model: Packaging model (DSP 2.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Take your code and put into a standard Python package structure, see  [recommended package structure](https://cloud.google.com/ml-engine/docs/tensorflow/packaging-trainer#project-structure)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Key-Idea: \n",
    " - define entry point which can be called\n",
    " - write all tasks as a function (callable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    " \n",
    "Why a package?\n",
    " - can be called from other scripts `import model`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `model.py`\n",
    "\n",
    "load most recent version, if needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%load src/pkg_mnist_fnn/model.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%writefile src/pkg_mnist_fnn/model.py\n",
    "# First try to start Cloud ML uing MNIST example.\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "from .utils import load_data\n",
    "##########################################################################\n",
    "#Factor into config:\n",
    "IMAGE_SHAPE = (28,28)\n",
    "N_PIXEL = 28 * 28\n",
    "NUM_LABELS = 10\n",
    "\n",
    "BATCH_SIZE = 128\n",
    "EPOCHS = 5\n",
    "##########################################################################\n",
    "def parse_images(x):\n",
    "    return x.reshape(len(x), -1).astype('float32')\n",
    "\n",
    "\n",
    "def parse_labels(y):\n",
    "    return y.astype('int32')\n",
    "\n",
    "\n",
    "def numpy_input_fn(images: np.ndarray,\n",
    "                   labels: np.ndarray,\n",
    "                   mode=tf.estimator.ModeKeys.EVAL,\n",
    "                   epochs=EPOCHS,\n",
    "                   batch_size=BATCH_SIZE):\n",
    "    \"\"\"\n",
    "    Return depending on the `mode`-key an Interator which can be use to\n",
    "    feed into the Estimator-Model. \n",
    "\n",
    "    Alternative if a `tf.data.Dataset` named `dataset` would be created:\n",
    "    `dataset.make_one_shot_iterator().get_next()`\n",
    "    \"\"\"\n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        _epochs = epochs\n",
    "        _shuffle = True\n",
    "        _num_threads = 1 # This leads to doubling the number of epochs\n",
    "    else:\n",
    "        _epochs = 1\n",
    "        _shuffle = False\n",
    "        _num_threads = 1\n",
    "\n",
    "    return tf.estimator.inputs.numpy_input_fn(\n",
    "        {'x': images},\n",
    "        y=labels,\n",
    "        batch_size=batch_size,\n",
    "        num_epochs=_epochs,                             \n",
    "        shuffle=_shuffle, # Boolean, if True shuffles the queue. Avoid shuffle at prediction time.\n",
    "        queue_capacity=1000, # Integer, number of threads used for reading \n",
    "        # and enqueueing. To have predicted order of reading and enqueueing,\n",
    "        # such as in prediction and evaluation mode, num_threads should be 1.\n",
    "        num_threads=_num_threads\n",
    "    )\n",
    "\n",
    "\n",
    "def serving_input_fn():\n",
    "    feature_placeholders = {\n",
    "        'x': tf.placeholder(tf.float32, shape=[None, N_PIXEL])\n",
    "    }\n",
    "    features = feature_placeholders\n",
    "    return tf.estimator.export.ServingInputReceiver(\n",
    "         features=features, \n",
    "         receiver_tensors=feature_placeholders,\n",
    "         receiver_tensors_alternatives=None\n",
    "         )\n",
    "\n",
    "\n",
    "def train_and_evaluate(args):\n",
    "    \"\"\"\n",
    "    Utility function for distributed training on ML-Engine\n",
    "    www.tensorflow.org/api_docs/python/tf/estimator/train_and_evaluate \n",
    "    \"\"\"\n",
    "    ##########################################\n",
    "    # Load Data in Memoery\n",
    "    # ToDo: replace numpy-arrays\n",
    "    print('## load data, specified path to try: {}'.format(args['data_path']))\n",
    "    (x_train, y_train), (x_test, y_test) = load_data(\n",
    "        path=args['data_path'])\n",
    "    \n",
    "    x_train = parse_images(x_train)\n",
    "    x_test = parse_images(x_test)\n",
    "\n",
    "    y_train = parse_labels(y_train)\n",
    "    y_test = parse_labels(y_test)\n",
    "\n",
    "    model = tf.estimator.DNNClassifier(\n",
    "        hidden_units= args['hidden_units'],  #[256, 128, 64],\n",
    "        feature_columns=[tf.feature_column.numeric_column(\n",
    "            'x', shape=[N_PIXEL, ])],\n",
    "        model_dir=args['output_dir'],\n",
    "        n_classes=NUM_LABELS,\n",
    "        optimizer=tf.train.AdamOptimizer(learning_rate=args['learning_rate']),\n",
    "        # activation_fn=,\n",
    "        dropout=0.2,\n",
    "        batch_norm=False,\n",
    "        loss_reduction='weighted_sum',\n",
    "        warm_start_from=None,\n",
    "        config=tf.estimator.RunConfig(# save_summary_steps=200,\n",
    "                                      save_checkpoints_steps=400,\n",
    "                                      keep_checkpoint_max=5, \n",
    "                                      keep_checkpoint_every_n_hours=1,\n",
    "                                      #log_step_count_steps=100,\n",
    "                                      train_distribute=None,\n",
    "                                      )\n",
    "    )\n",
    "    train_spec = tf.estimator.TrainSpec(\n",
    "        input_fn=numpy_input_fn(\n",
    "            x_train, y_train, mode=tf.estimator.ModeKeys.TRAIN, \n",
    "            batch_size = args['train_batch_size']),    \n",
    "        max_steps=args['train_steps'],\n",
    "        # hooks = None\n",
    "    )\n",
    "    # use `LatestExporter` for regular model exports:\n",
    "    exporter = tf.estimator.LatestExporter('exporter', serving_input_fn)\n",
    "    eval_spec = tf.estimator.EvalSpec(\n",
    "        input_fn=numpy_input_fn(\n",
    "            x_test, y_test, mode=tf.estimator.ModeKeys.EVAL),\n",
    "        # steps=100,\n",
    "        start_delay_secs=args['eval_delay_secs'],\n",
    "        throttle_secs=args['min_eval_frequency'],\n",
    "        exporters=exporter\n",
    "    )\n",
    "    print(\n",
    "        \"## start training and evaluation\\n\"\n",
    "        \"### save model, ckpts, etc. to: {}\".format(args['output_dir'])\n",
    "    \n",
    "    )\n",
    "\n",
    "    tf.estimator.train_and_evaluate(\n",
    "        estimator=model, train_spec=train_spec, eval_spec=eval_spec)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Imports, Helper Functions\n",
    "```python\n",
    "%%writefile src/pkg_mnist_fnn/model.py\n",
    "# First try to start Cloud ML uing MNIST example.\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "from .utils import load_data\n",
    "##########################################################################\n",
    "#Factor into config:\n",
    "IMAGE_SHAPE = (28,28)\n",
    "N_PIXEL = 28 * 28\n",
    "NUM_LABELS = 10\n",
    "\n",
    "BATCH_SIZE = 128\n",
    "EPOCHS = 5\n",
    "##########################################################################\n",
    "def parse_images(x):\n",
    "    return x.reshape(len(x), -1).astype('float32')\n",
    "\n",
    "\n",
    "def parse_labels(y):\n",
    "    return y.astype('int32')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Input-Function used when Model is trained\n",
    "```python\n",
    "def numpy_input_fn(images: np.ndarray,\n",
    "                   labels: np.ndarray,\n",
    "                   mode=tf.estimator.ModeKeys.EVAL,\n",
    "                   epochs=EPOCHS,\n",
    "                   batch_size=BATCH_SIZE):\n",
    "    \"\"\"\n",
    "    Return depending on the `mode`-key an Interator which can be use to\n",
    "    feed into the Estimator-Model. \n",
    "\n",
    "    Alternative if a `tf.data.Dataset` named `dataset` would be created:\n",
    "    `dataset.make_one_shot_iterator().get_next()`\n",
    "    \"\"\"\n",
    "    if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        _epochs = epochs\n",
    "        _shuffle = True\n",
    "        _num_threads = 1 # This leads to doubling the number of epochs\n",
    "    else:\n",
    "        _epochs = 1\n",
    "        _shuffle = False\n",
    "        _num_threads = 1\n",
    "\n",
    "    return tf.estimator.inputs.numpy_input_fn(\n",
    "        {'x': images},\n",
    "        y=labels,\n",
    "        batch_size=batch_size,\n",
    "        num_epochs=_epochs,                             \n",
    "        shuffle=_shuffle, # Boolean, if True shuffles the queue. Avoid shuffle at prediction time.\n",
    "        queue_capacity=1000, # Integer, number of threads used for reading \n",
    "        # and enqueueing. To have predicted order of reading and enqueueing,\n",
    "        # such as in prediction and evaluation mode, num_threads should be 1.\n",
    "        num_threads=_num_threads\n",
    "    )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Input-Function used when Model is served\n",
    "```python\n",
    "def serving_input_fn():\n",
    "    feature_placeholders = {\n",
    "        'x': tf.placeholder(tf.float32, shape=[None, N_PIXEL])\n",
    "    }\n",
    "    features = feature_placeholders\n",
    "    return tf.estimator.export.ServingInputReceiver(\n",
    "         features=features, \n",
    "         receiver_tensors=feature_placeholders,\n",
    "         receiver_tensors_alternatives=None\n",
    "         )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Entrypoint (main function)\n",
    "```python\n",
    "def train_and_evaluate(args):\n",
    "    \"\"\"\n",
    "    Utility function for distributed training on ML-Engine\n",
    "    www.tensorflow.org/api_docs/python/tf/estimator/train_and_evaluate \n",
    "    \"\"\"\n",
    "    ##########################################\n",
    "    print('## load data, specified path to try: {}'.format(args['data_path']))\n",
    "    (x_train, y_train), (x_test, y_test) = load_data(\n",
    "        path=args['data_path'])\n",
    "    \n",
    "    x_train = parse_images(x_train)\n",
    "    x_test = parse_images(x_test)\n",
    "\n",
    "    y_train = parse_labels(y_train)\n",
    "    y_test = parse_labels(y_test)\n",
    "\n",
    "    model = tf.estimator.DNNClassifier(\n",
    "        hidden_units= args['hidden_units'],  #[256, 128, 64],\n",
    "        feature_columns=[tf.feature_column.numeric_column(\n",
    "            'x', shape=[N_PIXEL, ])],\n",
    "        model_dir=args['output_dir'],\n",
    "        n_classes=NUM_LABELS,\n",
    "        optimizer=tf.train.AdamOptimizer(learning_rate=args['learning_rate']),\n",
    "        # activation_fn=,\n",
    "        dropout=0.2,\n",
    "        batch_norm=False,\n",
    "        loss_reduction='weighted_sum',\n",
    "        warm_start_from=None,\n",
    "        config=tf.estimator.RunConfig(save_checkpoints_steps=400,\n",
    "                                      keep_checkpoint_max=5, \n",
    "                                      keep_checkpoint_every_n_hours=1,\n",
    "                                      train_distribute=None)\n",
    "    )\n",
    "    ## to cont.\n",
    "```  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "    ## to cont.\n",
    "    train_spec = tf.estimator.TrainSpec(\n",
    "        input_fn=numpy_input_fn(\n",
    "            x_train, y_train, mode=tf.estimator.ModeKeys.TRAIN, \n",
    "            batch_size = args['train_batch_size']),    \n",
    "        max_steps=args['train_steps'],\n",
    "        # hooks = None\n",
    "    )\n",
    "    # use `LatestExporter` for regular model exports:\n",
    "    exporter = tf.estimator.LatestExporter('exporter', serving_input_fn)\n",
    "    eval_spec = tf.estimator.EvalSpec(\n",
    "        input_fn=numpy_input_fn(\n",
    "            x_test, y_test, mode=tf.estimator.ModeKeys.EVAL),\n",
    "        # steps=100,\n",
    "        start_delay_secs=args['eval_delay_secs'],\n",
    "        throttle_secs=args['min_eval_frequency'],\n",
    "        exporters=exporter\n",
    "    )\n",
    "    print(\"## start training and evaluation\\n\"\n",
    "          \"### save model, ckpts, etc. to: {}\".format(args['output_dir']))\n",
    "    \n",
    "    tf.estimator.train_and_evaluate(\n",
    "        estimator=model, train_spec=train_spec, eval_spec=eval_spec)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## `task.py`\n",
    "\n",
    "write contents to file:\n",
    "\n",
    "```python\n",
    "%%writefile src/pkg_mnist_fnn/task.py\n",
    "# Parse arguments and call main function\n",
    "import os\n",
    "import json\n",
    "import argparse\n",
    "import shutil\n",
    "from pprint import pprint \n",
    "\n",
    "from .model import train_and_evaluate\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\n",
    "        '--data_path',\n",
    "        help='GCS or local path to training data',\n",
    "        required=True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--output_dir',\n",
    "        help='GCS location to write checkpoints and export models',\n",
    "        required=True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--train_batch_size',\n",
    "        help='Batch size for training steps',\n",
    "        type=int,\n",
    "        default='128'\n",
    "    )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "    parser.add_argument(\n",
    "        '--train_steps',\n",
    "        help='Steps to run the training job for',\n",
    "        type=int,\n",
    "        default='200'\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--learning_rate',\n",
    "        help='Learning Rate used for Adam',\n",
    "        type=float,\n",
    "        default='0.001'\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--hidden_units',\n",
    "        help = 'Hidden layer sizes to use for DNN feature columns -- provide space-separated layers',\n",
    "        type = str,\n",
    "        default = \"256 128 64\"\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--job_dir',\n",
    "        help='this model ignores this field, but it is required by gcloud',\n",
    "        default='junk'\n",
    "    )\n",
    "    # Eval arguments\n",
    "    parser.add_argument(\n",
    "        '--eval_delay_secs',\n",
    "        help='How long to wait before running first evaluation',\n",
    "        default=1,\n",
    "        type=int\n",
    "    )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```python\n",
    "    parser.add_argument(\n",
    "        '--min_eval_frequency',\n",
    "        help='Seconds between evaluations',\n",
    "        default=5,\n",
    "        type=int\n",
    "    )\n",
    "    \n",
    "    args = parser.parse_args().__dict__\n",
    "    pprint(\"Arguments:\\n{}\".format(args)) \n",
    "    args['hidden_units'] = [int(x) for x in args['hidden_units'].split(' ')]\n",
    "    pprint(\"Arguments:\\n{}\".format(args)) \n",
    "    \n",
    "    output_dir = args['output_dir']\n",
    "    # Append trial_id to path if we are doing hptuning\n",
    "    # This code can be removed if you are not using hyperparameter tuning\n",
    "    args['output_dir'] = os.path.join(\n",
    "        output_dir,\n",
    "        json.loads(\n",
    "            os.environ.get('TF_CONFIG', '{}')\n",
    "        ).get('task', {}).get('trial', '')\n",
    "    )\n",
    "    print(\"Save output to: {}\".format(args['output_dir']))\n",
    "    # #######################################\n",
    "    # # Train and Evaluate (use TensorBoard to visualize)\n",
    "    train_and_evaluate(args)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%%writefile src/pkg_mnist_fnn/task.py\n",
    "# Parse arguments and call main function\n",
    "import os\n",
    "import json\n",
    "import argparse\n",
    "import shutil\n",
    "from pprint import pprint \n",
    "\n",
    "from .model import train_and_evaluate\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\n",
    "        '--data_path',\n",
    "        help='GCS or local path to training data',\n",
    "        required=True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--output_dir',\n",
    "        help='GCS location to write checkpoints and export models',\n",
    "        required=True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--train_batch_size',\n",
    "        help='Batch size for training steps',\n",
    "        type=int,\n",
    "        default='128'\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--train_steps',\n",
    "        help='Steps to run the training job for',\n",
    "        type=int,\n",
    "        default='200'\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--learning_rate',\n",
    "        help='Learning Rate used for Adam',\n",
    "        type=float,\n",
    "        default='0.001'\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--hidden_units',\n",
    "        help = 'Hidden layer sizes to use for DNN feature columns -- provide space-separated layers',\n",
    "        type = str,\n",
    "        default = \"256 128 64\"\n",
    "    )   \n",
    "    parser.add_argument(\n",
    "        '--job_dir',\n",
    "        help='this model ignores this field, but it is required by gcloud',\n",
    "        default='junk'\n",
    "    )\n",
    "    # Eval arguments\n",
    "    parser.add_argument(\n",
    "        '--eval_delay_secs',\n",
    "        help='How long to wait before running first evaluation',\n",
    "        default=1,\n",
    "        type=int\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        '--min_eval_frequency',\n",
    "        help='Seconds between evaluations',\n",
    "        default=5,\n",
    "        type=int\n",
    "    )\n",
    "\n",
    "    args = parser.parse_args().__dict__\n",
    "    pprint(\"Arguments:\\n{}\".format(args)) \n",
    "    args['hidden_units'] = [int(x) for x in args['hidden_units'].split(' ')]\n",
    "    pprint(\"Arguments:\\n{}\".format(args)) \n",
    "    \n",
    "    output_dir = args['output_dir']\n",
    "    # Append trial_id to path if we are doing hptuning\n",
    "    # This code can be removed if you are not using hyperparameter tuning\n",
    "    args['output_dir'] = os.path.join(\n",
    "        output_dir,\n",
    "        json.loads(\n",
    "            os.environ.get('TF_CONFIG', '{}')\n",
    "        ).get('task', {}).get('trial', '')\n",
    "    )\n",
    "    print(\"Save output to: {}\".format(args['output_dir']))\n",
    "    # #######################################\n",
    "    # # Train and Evaluate (use TensorBoard to visualize)\n",
    "    train_and_evaluate(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Add empty `__init__.py` to create package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%writefile src/pkg_mnist_fnn/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Add function to load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```python\n",
    "%%writefile src/pkg_mnist_fnn/utils.py\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "from io import BytesIO\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.python.lib.io import file_io \n",
    "\n",
    "def load_data(path='./data/'):\n",
    "    \"\"\"\n",
    "    Load data in memory from local source, from data-repository\n",
    "    or bucket (ToDo)\n",
    "\n",
    "    Return\n",
    "    -----\n",
    "    x_train: numpy.array\n",
    "        Shape: (60000, 28, 28)\n",
    "    y_train: numpy.array\n",
    "        Shape: (10000, )\n",
    "    x_test: numpy.array\n",
    "        s\n",
    "    y_test: numpy.array\n",
    "    \"\"\"\n",
    "    try:\n",
    "        _path = os.path.normpath(path)\n",
    "        with np.load(_path) as f:\n",
    "            x_train, y_train = f['x_train'], f['y_train']\n",
    "            x_test, y_test = f['x_test'], f['y_test']\n",
    "            print(\"Loaded data from {}\".format(_path))\n",
    "        return (x_train, y_train), (x_test, y_test)\n",
    "    except Exception\n",
    "       # to cont.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "```python\n",
    "        try:\n",
    "            f = BytesIO(file_io.read_file_to_string(\n",
    "                filename=path,\n",
    "                binary_mode=True\n",
    "            ))\n",
    "            data = np.load(f)\n",
    "            with data as f:\n",
    "                x_train, y_train = f['x_train'], f['y_train']\n",
    "                x_test, y_test = f['x_test'], f['y_test']\n",
    "                print(\"Loaded data from {}\".format(path))\n",
    "            return (x_train, y_train), (x_test, y_test)\n",
    "        except Exception:\n",
    "            try:\n",
    "                from tensorflow.keras.datasets import mnist\n",
    "                (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "                return (x_train, y_train), (x_test, y_test)\n",
    "            except Exception:\n",
    "                raise Exception(\"Not Connection to Server:\"\n",
    "                                \"Download manually to ./data/ from {}\".format(\n",
    "                    \"https://storage.googleapis.com/tensorflow/\"\n",
    "                        \"tf-keras-datasets/mnist.npz\"\n",
    "                ))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Train using ML-Engine\n",
    "\n",
    "Section shows:\n",
    "  1. Conceptual Workflow on GCP with ML-Engine\n",
    "  2. Executing the model in different environments (local or on cluster)\n",
    "  3. Optimize hyperparameters using ML-Engine in cluster\n",
    "  4. Query ML-Engine Endpoint API to extract results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Modeling and ML-Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![gcp_training_options-overview.png](Images/gcp_training_options-overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Environment Variables with absolut paths to relevant folders: \n",
    "    - `PKG_NAME`: Self-Contained Package to be exported into `site-packages` in `venv`\n",
    "    - `DATA`, `OUTDIR`: Datafolder and where to store store checkpoints (logs,  weights, graph)\n",
    "    - `PWD`: where your project folder lies\n",
    "    - `JOBNAME`: ID for ML-Engine\n",
    "    - `BUCKET`: ID of Bucket\n",
    "    - `TIER`: Type of Cluster\n",
    "     \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Adding Code snippets\n",
    "\n",
    "![gcp_training_options-overview.png](Images/gcp_training_options-overview-code.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Schematic Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![GCP for Data Scientists](Images/gcp_scheme_parts.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Contents ML-Engine Section\n",
    "\n",
    "- Training\n",
    "    - local (on your machine)\n",
    "    - on cluster (submitting a job)\n",
    "- Hyperparameter search (on cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training on your local maschine with your *python env*\n",
    "\n",
    "- Set local folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data_local = os.path.join(os.getcwd(),'data', 'mnist', 'raw', 'mnist.npz')\n",
    "OUTDIR_local = os.path.join(os.getcwd(),'trained', PKG_NAME)\n",
    "os.environ['OUTDIR_LOCAL'] = OUTDIR_local\n",
    "os.environ['DATA_LOCAL'] = data_local\n",
    "\n",
    "print(\"Local Data Directory:\\t {}\".format(os.environ['DATA_LOCAL']))\n",
    "print(\"Local Output Dir:\\t {}\".format(os.environ['OUTDIR_LOCAL']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree(OUTDIR_local, ignore_errors=True)\n",
    "os.makedirs(name= OUTDIR_local, exist_ok=True)\n",
    "os.listdir(OUTDIR_local)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Running the Python `module` without gcp ml-engine\n",
    "\n",
    "- Entry point is defined in `task.py`\n",
    "  - parses command line arguments \n",
    "- conda env has to be active"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python3 -m src.${PKG_NAME}.task \\\n",
    "   --data_path=$DATA_LOCAL \\\n",
    "   --output_dir=$OUTDIR_LOCAL \\\n",
    "   --train_steps=1000 \\\n",
    "   --job_dir=tmp\n",
    "echo \"Saved Model, ckpts, exported model to: $OUTDIR_LOCAL\"\n",
    "ls $OUTDIR_LOCAL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Exported models \n",
    "!ls $OUTDIR_LOCAL/export/exporter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Call hidden units parameter\n",
    "- change model architecture\n",
    "- here previous model is deleted -> later several model will be compared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree(OUTDIR_local, ignore_errors=True)\n",
    "os.makedirs(name= OUTDIR_local, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "center",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python3 -m src.${PKG_NAME}.task \\\n",
    "   --data_path    $DATA_LOCAL \\\n",
    "   --output_dir   $OUTDIR_LOCAL \\\n",
    "   --train_steps  1000 \\\n",
    "   --job_dir      tmp  \\\n",
    "   --train_batch_size   128 \\\n",
    "   --learning_rate 0.01 \\\n",
    "   --hidden_units \"256 128 64\"\n",
    "echo \"Saved Model, ckpts, exported model to: $OUTDIR_LOCAL\"\n",
    "ls $OUTDIR_LOCAL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Saved Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "split",
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Some previous versions might exist, take latest:\n",
    "os.listdir(os.path.normpath(\"{}/export/exporter\".format(OUTDIR_local)))[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**And we would be ready to deploy**\n",
    "\n",
    "... but of course not without looking at performance metrics or predictions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Training using `gcloud ml-engine local train`\n",
    "\n",
    "- continue training using `ml-engine local`\n",
    "- needs full-paths for out-dir: Add `$PWD`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree(OUTDIR_local, ignore_errors=True)\n",
    "os.makedirs(name= OUTDIR_local, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud ml-engine local train \\\n",
    "   --module-name=${PKG_NAME}.task \\\n",
    "   --package-path=src/${PKG_NAME} \\\n",
    "   -- \\\n",
    "   --data_path=$DATA_LOCAL \\\n",
    "   --output_dir=$OUTDIR_LOCAL \\\n",
    "   --train_steps=5500 \\\n",
    "   --job_dir=./tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ml-engine local train  --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training Cloud using `gcloud ml-engine train`\n",
    "\n",
    "- a copy of the data is in [Google Storage](https://console.cloud.google.com/storage) (buckets)\n",
    "- `gcloud ml-engine` output is saved to `OUTDIR`in Google Storage \n",
    "  - checkpoints (logs)\n",
    "  - model graph and weights\n",
    "- data is copied to Google Storage (see [console](https://console.cloud.google.com/))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# 10 epochs in global steps using 128 Images per Mini-Batch:\n",
    "steps = 10000\n",
    "batch_size = 128\n",
    "epochs = 10\n",
    "n_train = 60000\n",
    "print(\"Number of epochs after {} steps: {:.1f}\".format(steps, steps * batch_size / n_train))\n",
    "steps = int(60000 / batch_size * epochs) + 1\n",
    "print(\"For ten epochs specify {} steps\".format(steps))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "##### Check: Does this account for parallel processes in input fct?\n",
    "- If global steps is done on two processes, the number doubles!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#Set JOBNAME\n",
    "import datetime\n",
    "JOBNAME = 'mnist_' + datetime.datetime.now().strftime(\"%y%m%d_%H%M%S\")\n",
    "%env JOBNAME {JOBNAME}\n",
    "# Set new OUTPUT and DATA directory in GS\n",
    "OUTDIR = '/'.join(['gs:/', BUCKET, JOBNAME])\n",
    "DATA = '/'.join(['gs:/', BUCKET, PKG_NAME, 'data', 'mnist.npz'])\n",
    "%env OUTDIR $OUTDIR\n",
    "%env DATA $DATA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "print(\"OUTDIR on GS: {}\".format(OUTDIR))\n",
    "print(\"DATA on GS: {}\".format(DATA))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Data Transfer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gsutil -m cp ${PWD}/data/mnist/raw/mnist.npz ${DATA}\n",
    "gsutil ls ${DATA}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### ml-engine on cluster\n",
    "- set `JOBNAME` and decide which [tier](https://cloud.google.com/ml-engine/docs/tensorflow/machine-types#scale_tiers) to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%env TIER BASIC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo $OUTDIR $DATA $REGION $JOBNAME\n",
    "gsutil -m rm -rf $OUTDIR\n",
    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=$PKG_NAME.task \\\n",
    "   --package-path=${PWD}/src/$PKG_NAME \\\n",
    "   --staging-bucket=gs://$BUCKET \\\n",
    "   --scale-tier=$TIER \\\n",
    "   --python-version 3.5 \\\n",
    "   --runtime-version=$TFVERSION \\\n",
    "   -- \\\n",
    "   --data_path=$DATA \\\n",
    "   --output_dir=$OUTDIR \\\n",
    "   --train_steps=5000 \\\n",
    "   --job_dir=$OUTDIR/jobs "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Fetch logs from ml-engine job\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo $JOBNAME\n",
    "gcloud ml-engine jobs describe $JOBNAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud ml-engine jobs stream-logs $JOBNAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Don't be concerned if the notebook appears stalled (with a blue progress bar) or returns with an error about being unable to refresh auth tokens. This is a long-lived Cloud job and work is going on in the cloud. \n",
    "\n",
    "**Use the Cloud Console link to monitor the job and do NOT proceed until the job is done.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Check out results in the logs, see [example](https://cloud.google.com/solutions/machine-learning/recommendation-system-tensorflow-train-cloud-ml-engine#results_of_tuning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## ML-Engine with Hyperparameter search\n",
    "- Bayesian approach to find optimal hyperparameters, see \n",
    "    > Golovin et.al (2017): Google Vizier: A Service for Black-Box Optimization\n",
    "- consecutive search, here\n",
    "    - 2 trials in parallel\n",
    "    - a total of 30 trials\n",
    "- see `hyperp_config.yaml`:\n",
    "    - `train_batch_size`\n",
    "    - `hidden_units`\n",
    "\n",
    "- Pick an [algorithm](https://cloud.google.com/ml-engine/docs/tensorflow/hyperparameter-tuning-overview#search_algorithms) to search Hyperparameter space\n",
    "  1. `ALGORITHM_UNSPECIFIED`: Bayesian Search\n",
    "  2. `GRID_SEARCH`\n",
    "  3. `RANDOM_SEARCH`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Configure Search in  `hyperp_config.yaml`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%writefile hyperp_config.yaml\n",
    "trainingInput:\n",
    "  hyperparameters:\n",
    "    goal: MAXIMIZE\n",
    "    hyperparameterMetricTag: accuracy\n",
    "    maxTrials: 30\n",
    "    maxParallelTrials: 4\n",
    "    algorithm: ALGORITHM_UNSPECIFIED\n",
    "    params:\n",
    "      - parameterName: train_batch_size\n",
    "        type: INTEGER\n",
    "        minValue: 64\n",
    "        maxValue: 512\n",
    "        scaleType: UNIT_LINEAR_SCALE\n",
    "      - parameterName: hidden_units\n",
    "        type: CATEGORICAL\n",
    "        categoricalValues: [\"256 128 64\", \"128 64 32\", \"512 256 128 64\", \"256 128 64 32\"]\n",
    "      - parameterName: learning_rate\n",
    "        type: DOUBLE\n",
    "        minValue: 0.0001\n",
    "        maxValue: 0.1\n",
    "        scaleType: UNIT_LOG_SCALE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%load hyperp_config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Create unique jobname: `JOBNAME_HYPER`\n",
    "\n",
    "- decide which [`TIER`](https://cloud.google.com/ml-engine/docs/tensorflow/machine-types#scale_tiers) to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Set JOBNAME environment variable\n",
    "import datetime\n",
    "JOBNAME_HYPER = \"mnist_{}_hyper\".format(datetime.datetime.now().strftime(\"%y%m%d_%H%M%S\"))\n",
    "%env JOBNAME_HYPER {JOBNAME_HYPER}\n",
    "# Set new OUTPUT and DATA directory in GS\n",
    "OUTDIR_HYPER = '/'.join(['gs:/', BUCKET, JOBNAME_HYPER])\n",
    "DATA = '/'.join(['gs:/', BUCKET, PKG_NAME, 'data', 'mnist.npz'])\n",
    "%env OUTDIR_HYPER $OUTDIR_HYPER\n",
    "%env DATA $DATA\n",
    "%env TIER STANDARD_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Start Bayesian Hyperparameter Search:\n",
    "\n",
    "- add `config` parameter with `hyperp_config.yaml` as argument:\n",
    "- one can add other parameter to `hyperp_config.yaml`, see [docs on submitting](https://cloud.google.com/ml-engine/docs/tensorflow/training-jobs#formatting_your_configuration_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo $OUTDIR_HYPER $DATA $REGION $JOBNAME_HYPER\n",
    "gcloud ml-engine jobs submit training $JOBNAME_HYPER \\\n",
    "   --region $REGION \\\n",
    "   --module-name $PKG_NAME.task \\\n",
    "   --package-path ${PWD}/src/$PKG_NAME \\\n",
    "   --staging-bucket gs://$BUCKET \\\n",
    "   --scale-tier $TIER \\\n",
    "   --python-version 3.5 \\\n",
    "   --runtime-version $TFVERSION \\\n",
    "   --config hyperp_config.yaml \\\n",
    "   -- \\\n",
    "   --data_path $DATA \\\n",
    "   --output_dir $OUTDIR_HYPER \\\n",
    "   --train_steps 5000 \\\n",
    "   --job_dir $OUTDIR/jobs "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ml-engine jobs submit training --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ml-engine jobs describe $JOBNAME_HYPER"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Get results from job using API\n",
    "\n",
    "See [client documentation on ml-engine](https://cloud.google.com/ml-engine/docs/tensorflow/python-client-library#putting_it_all_together) and [` ml.projects().jobs().get()`](https://cloud.google.com/ml-engine/reference/rest/v1/projects.jobs/get) method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Using `requests-package`\n",
    "- can work behind a proxy if `HTTPS_PROXY` is defined and/or a specific SSL certificte is needed `REQUESTS_CA_BUNDLE`:\n",
    "  - `REQUESTS_CA_BUNDLE`: optional, filepath to your SLL-certificate \n",
    "  - `HTTPS_PROXY`: optional, link to your proxy, possibly includign authentification or ports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import subprocess\n",
    "import requests\n",
    "import json\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# all jobs\n",
    "url = 'https://ml.googleapis.com/v1/projects/{project}/jobs'.format(project=PROJECT)\n",
    "headers = {\n",
    "   'Content-Type': 'application/json',\n",
    "   'Authorization':  'Bearer {}'.format(subprocess.run('gcloud auth print-access-token', shell=True, check=True, \n",
    "                                                       stdout=subprocess.PIPE).stdout.decode().replace(os.linesep, ''))\n",
    "}\n",
    "json_response = requests.get(url=url, headers=headers)\n",
    "json.loads(json_response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# current Hyperparameter Training Job\n",
    "jobname = os.environ['JOBNAME_HYPER']\n",
    "url = 'https://ml.googleapis.com/v1/projects/{project}/jobs/{jobname}'.format(project=PROJECT, jobname=jobname)\n",
    "json_response = requests.get(url=url, headers=headers)\n",
    "json.loads(json_response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Using `googleapiclient.discorvery`\n",
    "\n",
    "- does most likely won't work if you are behind a proxy\n",
    "    - would need configuration of `httplib2.HTTP` instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from googleapiclient import discovery\n",
    "ml = discovery.build('ml', 'v1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ml.projects().jobs().list(parent='projects/{}'.format(PROJECT)).execute() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "jobname = os.environ['JOBNAME'] \n",
    "request = ml.projects().jobs().get(\n",
    "    name='projects/{project}/jobs/{jobname}'.format(\n",
    "        project=PROJECT, jobname=jobname))\n",
    "request.execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from googleapiclient import discovery\n",
    "from googleapiclient import errors\n",
    "from pprint import pprint\n",
    "\n",
    "def get_job_results(jobname, project=PROJECT):\n",
    "    \"\"\"\n",
    "    Builds a discovery client with GCMLE endpoint.\n",
    "    \"\"\"\n",
    "    ml = discovery.build('ml', 'v1')\n",
    "    endpoint = 'projects/{project}/jobs/{jobname}'.format(\n",
    "        project=project, jobname=jobname)\n",
    "    print(\"API endpoint: {}\".format(endpoint))\n",
    "    request = ml.projects().jobs().get(name=endpoint)\n",
    "    try: # Make the call.\n",
    "        response_dict = request.execute()\n",
    "        pprint(response_dict)\n",
    "    except errors.HttpError as err:\n",
    "        print('There was an error creating the model. Check the details:')\n",
    "        print(err._get_reason())\n",
    "        raise\n",
    "    return response_dict\n",
    "\n",
    "get_job_results(os.environ['JOBNAME_HYPER'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "job_details = get_job_results(jobname=os.environ['JOBNAME_HYPER'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "job_details.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Excursus: Check Results in TensorBoard\n",
    "\n",
    "- metrics and variables are inspected from the logs, called checkpoints (`ckpt`)\n",
    "- Dashboard on localhost: `TensorBoard`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Inspect Model trained on your machine by starting a local tensorboard server:\n",
    "- `tensorboard --logdir trained/pkg_mnist_fnn/`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Screenshot of Tensorboard\n",
    "![screenshot Tensorboard](Images/tensorboard_screenshot.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Excursus: Load data from the bucket\n",
    "\n",
    "- Binary Object has to be read by `BytesIO` module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from google.cloud import storage\n",
    "from io import BytesIO\n",
    "import numpy as np\n",
    "\n",
    "storage_client = storage.Client(project=PROJECT) # use current gcloud PROJECT_ID\n",
    "bucket = storage_client.get_bucket(BUCKET)\n",
    "blob = bucket.blob(\"pkg_mnist_fnn/data/mnist.npz\")\n",
    "\n",
    "data = blob.download_as_string()\n",
    "data = BytesIO(data)\n",
    "data = np.load(data)\n",
    "with data as f:\n",
    "    x_train, y_train = f['x_train'], f['y_train']\n",
    "    x_test, y_test = f['x_test'], f['y_test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "#ToDo# Does only work under linux-> version?\n",
    "from io import BytesIO\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow.python.lib.io import file_io\n",
    "\n",
    "f = BytesIO(file_io.read_file_to_string(\n",
    "    filename=DATA, #'gs://BUCKET/PKG_NAME/data/mnist.npz', \n",
    "    binary_mode=True\n",
    "))\n",
    "data = np.load(f)\n",
    "with data as f:\n",
    "    x_train, y_train = f['x_train'], f['y_train']\n",
    "    x_test, y_test = f['x_test'], f['y_test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deploy model - from any previous step (DSP 2.5)\n",
    "\n",
    "- `tf.estimator.LatestExporter`is used to store a model for deployment in the cloud\n",
    "- See also:  `tf.estimator.export`, `tf.saved_model`\n",
    "\n",
    "[Link to Console](https://console.cloud.google.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Check that a model has been saved on your Bucket:\n",
    "\n",
    "get best model found in from Hyperparameter Tuning\n",
    "- `get_job_results` is defined before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#%env JOBNAME_HYPER mnist_190427_132255_hyper  # uncomment and set\n",
    "job_details = get_job_results(os.environ[\"JOBNAME_HYPER\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "best_run = job_details['trainingOutput']['trials'][0]\n",
    "print(\"Run with best performance on chosen metrics:\\n{}\".format(best_run))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%env TRIAL_ID {best_run['trialId']}\n",
    "models = !gsutil ls gs://$PROJECT/$JOBNAME_HYPER/$TRIAL_ID/export/exporter/\n",
    "models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Use best  model from Hyper-Parameter Tuning Job (Query is shown before)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%env MODEL_LOCATION={models[-1]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Deploy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Identifier for deployed model:\n",
    "- `MODEL_NAME`\n",
    "- `MODEL_VERSION`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%env MODEL_NAME MNIST_MLENGINE\n",
    "%env MODEL_VERSION v1 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Create: A model (Dataset) has different versions (Tables)  - (comp. to BQ Datasets and Tables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud ml-engine models   create ${MODEL_NAME} --regions $REGION\n",
    "gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} \\\n",
    "     --origin ${MODEL_LOCATION} \\\n",
    "     --runtime-version $TFVERSION \\\n",
    "     --python-version 3.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ml-engine versions create --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. Using the Model saved by Python Module\n",
    "2. Using Model saved by `ml-engine local`\n",
    "3. Using Model trained online"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Tools get predictions:\n",
    "- Command Line Interfaces\n",
    "    - `gcloud ml-engine local predict`\n",
    "    - `gcloud ml-engine predict`\n",
    "- Python Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Create an test-image in numpy format\n",
    "1. Add filename to config-file\n",
    "2. Create file containing `N` examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# add filename to config-file\n",
    "import yaml\n",
    "N=4\n",
    "testdatafile = \"data/mnist/json/ml_engine_testdatafile_N{}.json\".format(N)\n",
    "with open(\"config.yaml\", \"r\", encoding = \"utf8\") as f:\n",
    "    config = yaml.load(f)\n",
    "with open(\"config.yaml\", \"w\", encoding = \"utf8\") as f:\n",
    "    config['testdatafile'] = testdatafile\n",
    "    yaml.dump(config, stream=f,  default_flow_style=False)\n",
    "TEST_DATA_JSON = testdatafile\n",
    "%env TEST_DATA_JSON $testdatafile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Create a file with 4 test images\n",
    "import numpy as np\n",
    "import json\n",
    "from src.pkg_mnist_fnn.utils import load_data\n",
    "from src.pkg_mnist_fnn.model import parse_images\n",
    "(_,_), (x_test, y_test) = load_data(path='data/mnist/raw/mnist.npz')\n",
    "test_indices = np.random.randint(low=0, high=len(y_test), size=N)\n",
    "x_test, y_test = x_test[test_indices], y_test[test_indices]\n",
    "x_test = parse_images(x_test).tolist()\n",
    "\n",
    "#eol = os.linesep\n",
    "#print(eol)\n",
    "n_lines = len(y_test)\n",
    "with open(testdatafile, \"w\") as f:\n",
    "    for image, label in zip(x_test, y_test):\n",
    "        _dict = {\"x\": image} #, \"y\": int(label)}\n",
    "        f.write(json.dumps(_dict)+ \"\\n\")\n",
    "print(\"Wrote to {}\".format(testdatafile))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Let's look at our four examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from src.utils.mnist_utils import plot_mnist_testdata\n",
    "plot_mnist_testdata(TEST_DATA_JSON)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## ML-Engine: `ml-engine local predict`\n",
    "- Using Model saved\n",
    "  - Python module\n",
    "  - `ml-engine local`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### `ML-Engine local` using Python 3 ...\n",
    "you still have to remove manually some compiled python files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "#/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/\n",
    "folder = \"C:\\\\eplatform\\\\tools\\\\google-cloud-sdk\\\\lib\\\\googlecloudsdk\\\\command_lib\\\\ml_engine\\\\\"\n",
    "folder = \"/usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/\"\n",
    "files = os.listdir(folder)\n",
    "files = [x for x in files if \".pyc\" in x]\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "for x in files:\n",
    "    assert \".pyc\" in x\n",
    "    path_ = os.path.join(folder, x)\n",
    "    os.remove(path_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "folder = \"C:\\\\eplatform\\\\tools\\\\google-cloud-sdk\\\\lib\\\\googlecloudsdk\\\\command_lib\\\\ml_engine\\\\\"\n",
    "files = os.listdir(folder)\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "In order to work with `Python 3`, delete the `*.pyc` files, see [post](https://stackoverflow.com/questions/48824381/gcloud-ml-engine-local-predict-runtimeerror-bad-magic-number-in-pyc-file)\n",
    "\n",
    "Default Datalab\n",
    "```\n",
    "rm /tools/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/*.pyc\n",
    "```\n",
    "Default UNIX:\n",
    "```\n",
    "sudo rm /usr/lib/google-cloud-sdk/lib/googlecloudsdk/command_lib/ml_engine/*.pyc\n",
    "```\n",
    "\n",
    "> Process running Datalab or Jupyter Notebook needs admin rights. This is not always given for locally run notebooks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "model_dir = os.listdir(\"{}/export/exporter\".format(OUTDIR_local))[-1]\n",
    "%env model_dir=$model_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "model_dir=$(ls $OUTDIR_LOCAL/export/exporter/ | tail -1)\n",
    "echo \"Selected Model:  $model_dir\" \n",
    "gcloud ml-engine local predict \\\n",
    "    --model-dir=${PWD}/$OUTDIR_LOCAL/export/exporter/${model_dir} \\\n",
    "    --json-instances=$TEST_DATA_JSON \\\n",
    "    --verbosity debug > data/test_predictions\n",
    "cat data/test_predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ml-engine local predict --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Online Prediction - Command Line\n",
    "\n",
    "- same output format as before,  check Console: [link](https://console.cloud.google.com/mlengine/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud ml-engine predict --model=MNIST_MLENGINE --version=v1 --json-instances=$TEST_DATA_JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from src.utils.mnist_utils import plot_mnist_testdata\n",
    "plot_mnist_testdata()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Online Predictions - Batch \n",
    "\n",
    "- cp [example](https://cloud.google.com/ml-engine/docs/tensorflow/batch-predict)\n",
    "- `data_format`= `'text'` for JSON-Format\n",
    "- `output-path`: GS folder where results will be saved \n",
    "- `input-paths`: File-Location (can be folder with several files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "os.path.split(OUTDIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "JOBNAME_BATCH_PRED = 'BATCH_' + datetime.datetime.now().strftime(\"%y%m%d_%H%M%S\")\n",
    "%env JOBNAME_BATCH_PRED {JOBNAME_BATCH_PRED}\n",
    "%env DATA_FORMAT text\n",
    "%env OUTPUT_PATH {'/'.join([os.path.split(OUTDIR)[0], \"batch_pred/\"])}\n",
    "%env TEST_DATA_GS {'/'.join([os.path.split(DATA)[0], os.path.split(TEST_DATA_JSON)[1]])}_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Copy files**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!gsutil cp data/mnist/json/ml_engine_testdatafile_N4.json $TEST_DATA_GS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Submit job using `gcloud` functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "!gcloud ml-engine jobs submit prediction $JOBNAME_BATCH_PRED  --model=MNIST_MLENGINE --version=v1 --input-paths=$TEST_DATA_GS --output-path $OUTPUT_PATH  --region $REGION --data-format $DATA_FORMAT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Retrieve results from batch and parse them "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "files = !gsutil ls $OUTPUT_PATH\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from google.cloud import storage\n",
    "import json\n",
    "mybucket= storage.Client(project=PROJECT).get_bucket('{}'.format(BUCKET))\n",
    "file = files[1].split(\"{}\".format(BUCKET + \"/\"))[1]\n",
    "print(\"Get file {}\".format(file))\n",
    "blob= mybucket.blob(file)\n",
    "result = blob.download_as_string()\n",
    "\n",
    "result = [json.loads(x) for x in (result.decode().split(\"\\n\"))[:-1]]\n",
    "print(result[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Online Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Get predictions using the [Python-Client-Library, see Tutorial](https://cloud.google.com/ml-engine/docs/tensorflow/python-client-library). \n",
    "\n",
    "- [API-Reference](https://cloud.google.com/ml-engine/reference/rest/)\n",
    "\n",
    "-  service account authentification:  [link](https://cloud.google.com/iam/docs/creating-managing-service-accounts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "MODEL_NAME = 'MNIST_MLENGINE' \n",
    "VERSION = 'v1'\n",
    "print(PROJECT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Load data** into python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "instances = []\n",
    "with open(TEST_DATA_JSON, \"r\") as f:\n",
    "    data = f.readlines()\n",
    "instances = [json.loads(x) for x in data]   # for discovery-client\n",
    "data = [image['x'] for  image in instances] # for requests-package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "###  Using `requests`-package \n",
    "\n",
    "- see hints above on possiblity to configure proxy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import subprocess\n",
    "import requests\n",
    "import os\n",
    "from pprint import pprint\n",
    "url = 'https://ml.googleapis.com/v1/projects/{project}/'\n",
    "      'models/{model}/versions/{version}:predict'.format(\n",
    "        project=PROJECT, model=MODEL_NAME, version=VERSION)\n",
    "headers = {\n",
    "   'Content-Type': 'application/json',\n",
    "   'Authorization':  'Bearer {}'.format(\n",
    "       subprocess.run('gcloud auth print-access-token', \n",
    "                       shell=True, check=True, \n",
    "                       stdout=subprocess.PIPE).stdout.decode().replace(\n",
    "                                                       os.linesep, '')\n",
    "                      ) \n",
    "}\n",
    "request_data = {\"instances\":\n",
    "    data\n",
    "}\n",
    "print(headers)\n",
    "json_response = requests.post(url=url, data=json.dumps(request_data), headers=headers)\n",
    "pprint(json.loads(json_response.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Using `googleapiclient.discovery` \n",
    "- fails behind proxy due to SSL verification (which could not be deactivated)\n",
    "#### Authentification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from oauth2client.client import GoogleCredentials\n",
    "#import json\n",
    "#import google.auth\n",
    "#cred, project = google.auth.default()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from googleapiclient import discovery\n",
    "api = discovery.build(serviceName='ml', version='v1',\n",
    "                      discoveryServiceUrl='https://www.googleapis.com/discovery/'\n",
    "                                          'v1/apis/{api}/{apiVersion}/rest'\n",
    "                     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Picks up per default GCLOUD SDK authentification:\n",
    "```\n",
    "UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. **If your application continues to use end user credentials from Cloud SDK, you might receive a \"quota exceeded\" or \"API not enabled\" error**. For more information about service accounts, see https://cloud.google.com/docs/authentication/\n",
    "warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Get predictions for samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "project_id = 'projects/{project}/models/{model}/versions/{version}'.format( \n",
    "              project=PROJECT, model=MODEL_NAME, version=VERSION)\n",
    "print(\"Endpoint to use: {}\\n\".format(project_id))\n",
    "request_data = {\"instances\":\n",
    "    instances\n",
    "}\n",
    "request = api.projects().predict(body=request_data, name=project_id).execute()\n",
    "pprint(request) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for i, pred in enumerate(request['predictions']):\n",
    "    print(\"Predicted class: {}, True Class:\\t{}\".format(\n",
    "        pred['classes'][0], \n",
    "        y_test[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "%pdoc discovery.build"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```\n",
    "Signature: discovery.build(serviceName, version, http=None, discoveryServiceUrl='https://www.googleapis.com/discovery/v1/apis/{api}/{apiVersion}/rest', developerKey=None, model=None, requestBuilder=<class 'googleapiclient.http.HttpRequest'>, credentials=None, cache_discovery=True, cache=None)\n",
    "Docstring:\n",
    "Construct a Resource for interacting with an API.\n",
    "\n",
    "Construct a Resource object for interacting with an API. The serviceName and\n",
    "version are the names from the Discovery service.\n",
    "\n",
    "Args:\n",
    "serviceName: string, name of the service.\n",
    "version: string, the version of the service.\n",
    "http: httplib2.Http, An instance of httplib2.Http or something that acts\n",
    "like it that HTTP requests will be made through.\n",
    "discoveryServiceUrl: string, a URI Template that points to the location of\n",
    "the discovery service. It should have two parameters {api} and\n",
    "{apiVersion} that when filled in produce an absolute URI to the discovery\n",
    "document for that service.\n",
    "developerKey: string, key obtained from\n",
    "https://code.google.com/apis/console.\n",
    "model: googleapiclient.Model, converts to and from the wire format.\n",
    "requestBuilder: googleapiclient.http.HttpRequest, encapsulator for an HTTP\n",
    "request.\n",
    "credentials: oauth2client.Credentials or\n",
    "google.auth.credentials.Credentials, credentials to be used for\n",
    "authentication.\n",
    "cache_discovery: Boolean, whether or not to cache the discovery doc.\n",
    "cache: googleapiclient.discovery_cache.base.CacheBase, an optional\n",
    "cache object for the discovery documents.\n",
    "\n",
    "Returns:\n",
    "A Resource object with methods for interacting with the service.\n",
    "File: /usr/local/envs/py3env/lib/python3.5/site-packages/googleapiclient/discovery.py\n",
    "Type: function\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![gcp_training_options-gcp_services.png](Images/gcp_training_options-gcp_services.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Outlook\n",
    "- Add different models types\n",
    "    - different layers of abstraction in tensorflow\n",
    "    - sklearn\n",
    "- Show how to use `ml-engine` in SQL in BigQuery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Notes on **Jupyter Slides**\n",
    "- Activate: View -> Cell Toolbar -> Slideshow\n",
    "- Install [nbextensions](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/install.html) into `base` conda environment\n",
    "   - activate [split cells vertically](https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/splitcell/readme.html)\n",
    "   - Code folding\n",
    "   - Table of Contents\n",
    "- [RISE](https://rise.readthedocs.io/en/5.4.1/installation.html) for interactive presentations\n",
    "  - using conda: `conda install -c conda-forge rise`\n",
    "  - activte scrolling in Notebook-Metadata, see [link](https://rise.readthedocs.io/en/5.4.1/customize.html#config-right-scroll) \n",
    "  - adapt width and height of your slides to your machine and needs. [link](https://rise.readthedocs.io/en/5.4.1/customize.html#change-the-width-and-height-of-slides)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "livereveal": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "598.438px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}