examples/training/00_introduction_training_example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12b45b0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2024 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8790d7ed",
   "metadata": {},
   "source": [
    "# AutoMLOps - Introduction Training Example\n",
    "\n",
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/automlops/blob/main/examples/training/00_introduction_training_example.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://github.com/GoogleCloudPlatform/automlops/blob/main/examples/training/00_introduction_training_example.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      View on GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/automlops/examples/training/00_introduction_training_example.ipynb\">\n",
    "        <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      Open in Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>\n",
    "<br/><br/><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f938540",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "In this tutorial, you will build two [Vertex AI](https://cloud.google.com/vertex-ai) pipelines, complete with an integrated CI/CD pipeline. This tutorial will walk you through how to use AutoMLOps to define, create and run pipelines, as well as monitoring deployed models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22881a6a",
   "metadata": {},
   "source": [
    "# Objective\n",
    "In this tutorial, you will learn how to create and run MLOps pipelines integrated with CI/CD. This tutorial goes through an example kubeflow pipeline that is defined using AutoMLOps. The example pipeline builds and deploys a classification model; the pipeline go through a very basic workflow:\n",
    "1. create_dataset: A custom component that will export the dataset from BQ to GCS as a csv.\n",
    "2. train_model: A custom component that will train a decision tree classifier on the training data.\n",
    "3. deploy_model: A custom component that will upload the saved_model to Vertex AI Model Registry and deploy it to an endpoint.\n",
    "\n",
    "# Prerequisites\n",
    "\n",
    "In order to use AutoMLOps, the following are required:\n",
    "\n",
    "- Python 3.7 - 3.10\n",
    "- [Google Cloud SDK 407.0.0](https://cloud.google.com/sdk/gcloud/reference)\n",
    "- [beta 2022.10.21](https://cloud.google.com/sdk/gcloud/reference/beta)\n",
    "- `git` installed\n",
    "- `git` logged-in:\n",
    "```\n",
    "  git config --global user.email \"you@example.com\"\n",
    "  git config --global user.name \"Your Name\"\n",
    "```\n",
    "- [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/provide-credentials-adc) are setup. This can be done through the following commands:\n",
    "```\n",
    "gcloud auth application-default login\n",
    "gcloud config set account <account@example.com>\n",
    "```\n",
    "\n",
    "# APIs & IAM\n",
    "Based on the user options selection, AutoMLOps will enable up to the following APIs during the provision step:\n",
    "- [aiplatform.googleapis.com](https://cloud.google.com/vertex-ai/docs/reference/rest)\n",
    "- [artifactregistry.googleapis.com](https://cloud.google.com/artifact-registry/docs/reference/rest)\n",
    "- [cloudbuild.googleapis.com](https://cloud.google.com/build/docs/api/reference/rest)\n",
    "- [cloudfunctions.googleapis.com](https://cloud.google.com/functions/docs/reference/rest)\n",
    "- [cloudresourcemanager.googleapis.com](https://cloud.google.com/resource-manager/reference/rest)\n",
    "- [cloudscheduler.googleapis.com](https://cloud.google.com/scheduler/docs/reference/rest)\n",
    "- [compute.googleapis.com](https://cloud.google.com/compute/docs/reference/rest/v1)\n",
    "- [iam.googleapis.com](https://cloud.google.com/iam/docs/reference/rest)\n",
    "- [iamcredentials.googleapis.com](https://cloud.google.com/iam/docs/reference/credentials/rest)\n",
    "- [logging.googleapis.com](https://cloud.google.com/logging/docs/reference/v2/rest)\n",
    "- [pubsub.googleapis.com](https://cloud.google.com/pubsub/docs/reference/rest)\n",
    "- [run.googleapis.com](https://cloud.google.com/run/docs/reference/rest)\n",
    "- [storage.googleapis.com](https://cloud.google.com/storage/docs/apis)\n",
    "\n",
    "AutoMLOps will create the following service account and update [IAM permissions](https://cloud.google.com/iam/docs/understanding-roles) during the provision step:\n",
    "1. Pipeline Runner Service Account (defaults to: vertex-pipelines@PROJECT_ID.iam.gserviceaccount.com). Roles added:\n",
    "- roles/aiplatform.user\n",
    "- roles/artifactregistry.reader\n",
    "- roles/bigquery.user\n",
    "- roles/bigquery.dataEditor\n",
    "- roles/iam.serviceAccountUser\n",
    "- roles/storage.admin\n",
    "- roles/cloudfunctions.admin\n",
    "\n",
    "# User Guide\n",
    "\n",
    "For a user-guide, please view these [slides](../../AutoMLOps_User_Guide.pdf).\n",
    "\n",
    "# Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "- Vertex AI\n",
    "- Artifact Registry\n",
    "- Cloud Storage\n",
    "- Cloud Build\n",
    "- Cloud Run\n",
    "- Cloud Scheduler\n",
    "- Cloud Pub/Sub\n",
    "\n",
    "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n",
    "\n",
    "# Ground-rules for using AutoMLOps\n",
    "1. Do not use variables, functions, code, etc. not defined within the scope of a custom component. These custom components will become containers and will have no reference to the out of scope code.\n",
    "2. Import statements and helper functions must be added inside the function. Provide parameter type hints.\n",
    "3. Test each of your components for accuracy and correctness before running them using AutoMLOps. We cannot fix bugs automatically; bugs are much more difficult to fix once they are made into pipelines.\n",
    "4. If you are using Kubeflow, be sure to define all the requirements needed to run the custom component - it can be easy to leave out packages which will cause the container to fail when running within a pipeline. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12381413",
   "metadata": {},
   "source": [
    "# Dataset\n",
    "For training data, we are using the [dry beans dataset](https://archive.ics.uci.edu/ml/datasets/dry+bean+dataset) which contains metadata on images of seven different types of dry beans taken with a high-resolution camera. The raw dataset can be found [here](https://github.com/GoogleCloudPlatform/automlops/blob/main/example/data/Dry_Beans_Dataset.csv)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a231b629",
   "metadata": {},
   "source": [
    "# Setup Git\n",
    "Set up your git configuration below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92f90b44",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git config --global user.email 'you@example.com'\n",
    "!git config --global user.name 'Your Name'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e4d190",
   "metadata": {},
   "source": [
    "# Install AutoMLOps\n",
    "\n",
    "Install AutoMLOps from [PyPI](https://pypi.org/project/google-cloud-automlops/), or locally by cloning the repo and running `pip install .`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94451868",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install google-cloud-automlops --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17db55d5",
   "metadata": {},
   "source": [
    "# Restart the kernel\n",
    "Once you've installed the AutoMLOps package, you need to restart the notebook kernel so it can find the package.\n",
    "\n",
    "**Note: Once this cell has finished running, continue on. You do not need to re-run any of the cells above.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "57c53b86",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv('IS_TESTING'):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "250d511b",
   "metadata": {},
   "source": [
    "# Set your project ID\n",
    "Set your project ID below. If you don't know your project ID, leave the field blank and the following cells may be able to find it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "931ff517",
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = '[your-project-id]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "be0be295",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID: automlops-sandbox\n"
     ]
    }
   ],
   "source": [
    "if PROJECT_ID == '' or PROJECT_ID is None or PROJECT_ID == '[your-project-id]':\n",
    "    # Get your GCP project id from gcloud\n",
    "    shell_output = !gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print('Project ID:', PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2c36482d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51e4c34d",
   "metadata": {},
   "source": [
    "Set your Model_ID below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "419b63b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_ID = 'dry-beans-dt'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26184c76",
   "metadata": {},
   "source": [
    "Miscellaneous constants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "10e1123f",
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATASET = f'{PROJECT_ID}.test_dataset.dry_beans'\n",
    "TARGET_COLUMN = 'Class'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "503f4a85-c357-4ec4-8ea5-0eeb2c8afd39",
   "metadata": {},
   "source": [
    "# Upload Data\n",
    "This will create a BQ table and upload the Dry Beans csv. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "28557b2d-598c-466b-8116-bc1e34ba090d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset automlops-sandbox.test_dataset already exists\n",
      "Table test_dataset.dry_beans already exists\n"
     ]
    }
   ],
   "source": [
    "!python3 -m data.load_data_to_bq --project $PROJECT_ID --file data/Dry_Beans_Dataset.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3fe994",
   "metadata": {},
   "source": [
    "# Example Workflow\n",
    "This workflow will define and generate a pipeline using AutoMLOps. AutoMLOps provides 2 functions for defining MLOps pipelines:\n",
    "\n",
    "- `AutoMLOps.component(...)`: Defines a component, which is a containerized python function.\n",
    "- `AutoMLOps.pipeline(...)`: Defines a pipeline, which is a series of components.\n",
    "\n",
    "AutoMLOps provides 6 functions for building and maintaining MLOps pipelines:\n",
    "\n",
    "- `AutoMLOps.generate(...)`: Generates the MLOps codebase. Users can specify the tooling and technologies they would like to use in their MLOps pipeline.\n",
    "- `AutoMLOps.provision(...)`: Runs provisioning scripts to create and maintain necessary infra for MLOps.\n",
    "- `AutoMLOps.deprovision(...)`: Runs deprovisioning scripts to tear down MLOps infra created using AutoMLOps.\n",
    "- `AutoMLOps.deploy(...)`: Builds and pushes component container, then triggers the pipeline job.\n",
    "- `AutoMLOps.launchAll(...)`: Runs `generate()`, `provision()`, and `deploy()` all in succession.\n",
    "- `AutoMLOps.monitor(...)`: Creates model monitoring jobs on deployed endpoints.\n",
    "\n",
    "Please see the [readme](https://github.com/GoogleCloudPlatform/automlops/blob/main/README.md) for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2219ee67",
   "metadata": {},
   "source": [
    "## Import AutoMLOps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f6209360",
   "metadata": {},
   "outputs": [],
   "source": [
    "from google_cloud_automlops import AutoMLOps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ac0c712-acdd-4830-a9bd-44f41eaa590c",
   "metadata": {},
   "source": [
    "## Data Loading\n",
    "Define a custom component for loading and creating a dataset using `@AutoMLOps.component`. Import statements and helper functions must be added inside the function. Provide parameter type hints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "94a96074-761f-40f0-84bb-adfcb9bf6911",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.component(\n",
    "    packages_to_install=[\n",
    "        'google-cloud-bigquery', \n",
    "        'pandas',\n",
    "        'pyarrow',\n",
    "        'db_dtypes',\n",
    "        'fsspec',\n",
    "        'gcsfs'\n",
    "    ]\n",
    ")\n",
    "def create_dataset(\n",
    "    bq_table: str,\n",
    "    data_path: str,\n",
    "    project_id: str\n",
    "):\n",
    "    \"\"\"Custom component that takes in a BQ table and writes it to GCS.\n",
    "\n",
    "    Args:\n",
    "        bq_table: The source biquery table.\n",
    "        data_path: The gcs location to write the csv.\n",
    "        project_id: The project ID.\n",
    "    \"\"\"\n",
    "    from google.cloud import bigquery\n",
    "    import pandas as pd\n",
    "    from sklearn import preprocessing\n",
    "    \n",
    "    bq_client = bigquery.Client(project=project_id)\n",
    "\n",
    "    def get_query(bq_input_table: str) -> str:\n",
    "        \"\"\"Generates BQ Query to read data.\n",
    "\n",
    "        Args:\n",
    "            bq_input_table: The full name of the bq input table to be read into\n",
    "                the dataframe (e.g. <project>.<dataset>.<table>)\n",
    "        Returns: A BQ query string.\n",
    "        \"\"\"\n",
    "        return f'''\n",
    "        SELECT *\n",
    "        FROM `{bq_input_table}`\n",
    "        '''\n",
    "\n",
    "    def load_bq_data(query: str, client: bigquery.Client) -> pd.DataFrame:\n",
    "        \"\"\"Loads data from bq into a Pandas Dataframe for EDA.\n",
    "        Args:\n",
    "            query: BQ Query to generate data.\n",
    "            client: BQ Client used to execute query.\n",
    "        Returns:\n",
    "            pd.DataFrame: A dataframe with the requested data.\n",
    "        \"\"\"\n",
    "        df = client.query(query).to_dataframe()\n",
    "        return df\n",
    "\n",
    "    dataframe = load_bq_data(get_query(bq_table), bq_client)\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    dataframe['Class'] = le.fit_transform(dataframe['Class'])\n",
    "    dataframe.to_csv(data_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4476cb4-91c5-42ff-a500-8cc275fedbd1",
   "metadata": {},
   "source": [
    "## Model Training\n",
    "Define a custom component for training a model using `@AutoMLOps.component`. Import statements and helper functions must be added inside the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5371be73-db3f-4d79-bde7-94fcd5ea13b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.component(\n",
    "    packages_to_install=[\n",
    "        'scikit-learn==1.3.2',\n",
    "        'pandas',\n",
    "        'joblib',\n",
    "        'tensorflow'\n",
    "    ]\n",
    ")\n",
    "def train_model(\n",
    "    data_path: str,\n",
    "    model_directory: str\n",
    "):\n",
    "    \"\"\"Custom component that trains a decision tree on the training data.\n",
    "\n",
    "    Args:\n",
    "        data_path: GS location of the training data.\n",
    "        model_directory: GS location of saved model.\n",
    "    \"\"\"\n",
    "    from sklearn.tree import DecisionTreeClassifier\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    import pandas as pd\n",
    "    import tensorflow as tf\n",
    "    import pickle\n",
    "    import os\n",
    "\n",
    "    def save_model(model, uri):\n",
    "        \"\"\"Saves a model to uri.\"\"\"\n",
    "        with tf.io.gfile.GFile(uri, 'w') as f:\n",
    "            pickle.dump(model, f)\n",
    "\n",
    "    df = pd.read_csv(data_path)\n",
    "    labels = df.pop('Class').tolist()\n",
    "    data = df.values.tolist()\n",
    "    x_train, x_test, y_train, y_test = train_test_split(data, labels)\n",
    "    skmodel = DecisionTreeClassifier()\n",
    "    skmodel.fit(x_train,y_train)\n",
    "    score = skmodel.score(x_test,y_test)\n",
    "    print('accuracy is:',score)\n",
    "\n",
    "    output_uri = os.path.join(model_directory, 'model.pkl')\n",
    "    save_model(skmodel, output_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5ed1eaa",
   "metadata": {},
   "source": [
    "## Uploading & Deploying the Model\n",
    "Define a custom component for uploading and deploying a model in Vertex AI, using `@AutoMLOps.component`. Import statements and helper functions must be added inside the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "47377d7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.component(\n",
    "    packages_to_install=[\n",
    "        'google-cloud-aiplatform'\n",
    "    ]\n",
    ")\n",
    "def deploy_model(\n",
    "    model_directory: str,\n",
    "    project_id: str,\n",
    "    region: str\n",
    "):\n",
    "    \"\"\"Custom component that uploads a saved model from GCS to Vertex Model Registry\n",
    "       and deploys the model to an endpoint for online prediction.\n",
    "\n",
    "    Args:\n",
    "        model_directory: GS location of saved model.\n",
    "        project_id: Project_id.\n",
    "        region: Region.\n",
    "    \"\"\"\n",
    "    import pprint as pp\n",
    "    import random\n",
    "\n",
    "    from google.cloud import aiplatform\n",
    "\n",
    "    aiplatform.init(project=project_id, location=region)\n",
    "    # Check if model exists\n",
    "    models = aiplatform.Model.list()\n",
    "    model_name = 'beans-model'\n",
    "    if 'beans-model' in (m.name for m in models):\n",
    "        parent_model = model_name\n",
    "        model_id = None\n",
    "        is_default_version=False\n",
    "        version_aliases=['experimental', 'challenger', 'custom-training', 'decision-tree']\n",
    "        version_description='challenger version'\n",
    "    else:\n",
    "        parent_model = None\n",
    "        model_id = model_name\n",
    "        is_default_version=True\n",
    "        version_aliases=['champion', 'custom-training', 'decision-tree']\n",
    "        version_description='first version'\n",
    "\n",
    "    serving_container = 'us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.1-3:latest'\n",
    "    uploaded_model = aiplatform.Model.upload(\n",
    "        artifact_uri=model_directory,\n",
    "        model_id=model_id,\n",
    "        display_name=model_name,\n",
    "        parent_model=parent_model,\n",
    "        is_default_version=is_default_version,\n",
    "        version_aliases=version_aliases,\n",
    "        version_description=version_description,\n",
    "        serving_container_image_uri=serving_container,\n",
    "        serving_container_ports=[8080],\n",
    "        labels={'created_by': 'automlops-team'},\n",
    "    )\n",
    "\n",
    "    endpoint = uploaded_model.deploy(\n",
    "        machine_type='n1-standard-4',\n",
    "        deployed_model_display_name='deployed-beans-model')\n",
    "\n",
    "    sample_input = [[random.uniform(0, 300) for x in range(16)]]\n",
    "\n",
    "    # Test endpoint predictions\n",
    "    print('running prediction test...')\n",
    "    try:\n",
    "        resp = endpoint.predict(instances=sample_input)\n",
    "        pp.pprint(resp)\n",
    "    except Exception as ex:\n",
    "        print('prediction request failed', ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73d96dcb-a020-4bab-b0e3-1e32f6b2aecf",
   "metadata": {},
   "source": [
    "## Define the Pipeline\n",
    "Define your pipeline using `@AutoMLOps.pipeline`. You can optionally give the pipeline a name and description. Define the structure by listing the components to be called in your pipeline; use `.after` to specify the order of execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a36e0154",
   "metadata": {},
   "outputs": [],
   "source": [
    "@AutoMLOps.pipeline #(name='automlops-pipeline', description='This is an optional description')\n",
    "def pipeline(\n",
    "    bq_table: str,\n",
    "    model_directory: str,\n",
    "    data_path: str,\n",
    "    project_id: str,\n",
    "    region: str):\n",
    "\n",
    "    create_dataset_task = create_dataset(\n",
    "        bq_table=bq_table,\n",
    "        data_path=data_path,\n",
    "        project_id=project_id)\n",
    "\n",
    "    train_model_task = train_model(\n",
    "        model_directory=model_directory,\n",
    "        data_path=data_path).after(create_dataset_task)\n",
    "\n",
    "    deploy_model_task = deploy_model(\n",
    "        model_directory=model_directory,\n",
    "        project_id=project_id,\n",
    "        region=region).after(train_model_task)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3874f4f6",
   "metadata": {},
   "source": [
    "## Define the Pipeline Arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4cb3786b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "pipeline_params = {\n",
    "    'bq_table': TRAINING_DATASET,\n",
    "    'model_directory': f'gs://{PROJECT_ID}-{MODEL_ID}-bucket/trained_models/{datetime.datetime.now()}',\n",
    "    'data_path': f'gs://{PROJECT_ID}-{MODEL_ID}-bucket/data.csv',\n",
    "    'project_id': PROJECT_ID,\n",
    "    'region': 'us-central1'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1e0aa05",
   "metadata": {},
   "source": [
    "## Generate and Run the pipeline\n",
    "`AutoMLOps.generate(...)` generates the MLOps codebase. Users can specify the tooling and technologies they would like to use in their MLOps pipeline. If you are interested in integrating with Github and Github Actions, please follow the setup steps in [this doc](../../docs/Using%20Github%20With%20AMO.md) and uncomment the relevant code block below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "db51a8d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing directories under AutoMLOps/\n",
      "Writing configurations to AutoMLOps/configs/defaults.yaml\n",
      "Writing kubeflow pipelines code to AutoMLOps/pipelines\n",
      "Writing kubeflow components code to AutoMLOps/components\n",
      "     -- Writing create_dataset\n",
      "     -- Writing train_model\n",
      "     -- Writing deploy_model\n",
      "Writing submission service code to AutoMLOps/services\n",
      "Writing gcloud provisioning code to AutoMLOps/provision\n",
      "Writing cloud build config to AutoMLOps/cloudbuild.yaml\n",
      "Code Generation Complete.\n"
     ]
    }
   ],
   "source": [
    "# Setup using local scripts and cloudbuild:\n",
    "AutoMLOps.generate(project_id=PROJECT_ID,\n",
    "                   pipeline_params=pipeline_params,\n",
    "                   use_ci=False,\n",
    "                   naming_prefix=MODEL_ID,\n",
    "                   deployment_framework='cloud-build'\n",
    ")\n",
    "\n",
    "# # Setup using Github, Github Actions, and Terraform:\n",
    "# AutoMLOps.generate(project_id=PROJECT_ID,\n",
    "#                    pipeline_params=pipeline_params,\n",
    "#                    naming_prefix=MODEL_ID,\n",
    "#                    schedule_pattern='59 11 * * 0', # retrain every Sunday at Midnight\n",
    "#                    setup_model_monitoring=True,    # use this if you would like to use Vertex Model Monitoring\n",
    "#                    use_ci=True,\n",
    "#                    deployment_framework='github-actions',\n",
    "#                    provisioning_framework='terraform',   \n",
    "#                    source_repo_type='github',\n",
    "#                    project_number='<project_number>',\n",
    "#                    source_repo_name='<source/repo/string>',\n",
    "#                    workload_identity_pool='<identity_pool_string>',\n",
    "#                    workload_identity_provider='<identity_provider_string>',\n",
    "#                    workload_identity_service_account='<workload_identity_sa>'\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb9e4f08",
   "metadata": {},
   "source": [
    "`AutoMLOps.provision(...)` runs provisioning scripts to create and maintain necessary infra for MLOps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a7e1efd9",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Provisioning requires these permissions:\n",
      "-serviceusage.services.enable\n",
      "-serviceusage.services.use\n",
      "-resourcemanager.projects.setIamPolicy\n",
      "-iam.serviceAccounts.list\n",
      "-iam.serviceAccounts.create\n",
      "-iam.serviceAccounts.actAs\n",
      "-storage.buckets.get\n",
      "-storage.buckets.create\n",
      "-artifactregistry.repositories.list\n",
      "-artifactregistry.repositories.create\n",
      "-pubsub.topics.list\n",
      "-pubsub.topics.create\n",
      "-pubsub.subscriptions.list\n",
      "-pubsub.subscriptions.create\n",
      "-cloudbuild.builds.list\n",
      "-cloudbuild.builds.create\n",
      "-cloudscheduler.jobs.list\n",
      "-cloudscheduler.jobs.create\n",
      "-cloudfunctions.functions.get\n",
      "-cloudfunctions.functions.create\n",
      "-source.repos.list\n",
      "-source.repos.create\n",
      "\n",
      "You are currently using: srastatter@google.com. Please check your account permissions.\n",
      "The following are the recommended roles for provisioning:\n",
      "-roles/serviceusage.serviceUsageAdmin\n",
      "-roles/resourcemanager.projectIamAdmin\n",
      "-roles/iam.serviceAccountAdmin\n",
      "-roles/iam.serviceAccountUser\n",
      "-roles/storage.admin\n",
      "-roles/artifactregistry.admin\n",
      "-roles/pubsub.editor\n",
      "-roles/cloudbuild.builds.editor\n",
      "-roles/cloudscheduler.admin\n",
      "-roles/cloudfunctions.admin\n",
      "-roles/source.admin\n",
      "\n",
      "\u001b[0;32m Setting up API services in project automlops-sandbox \u001b[0m\n",
      "Operation \"operations/acat.p2-45373616427-45388980-7b68-4bb7-8cbb-016158191e09\" finished successfully.\n",
      "\u001b[0;32m Setting up Artifact Registry in project automlops-sandbox \u001b[0m\n",
      "Listing items under project automlops-sandbox, location us-central1.\n",
      "\n",
      "dry-beans-dt-artifact-registry              DOCKER  STANDARD_REPOSITORY  Artifact Registry dry-beans-dt-artifact-registry in us-central1.                            us-central1                                  Google-managed key  2023-09-05T11:25:48  2024-01-24T13:11:04  23235.943\n",
      "Artifact Registry: dry-beans-dt-artifact-registry already exists in project automlops-sandbox\n",
      "\u001b[0;32m Setting up Storage Bucket in project automlops-sandbox \u001b[0m\n",
      "gs://automlops-sandbox-dry-beans-dt-bucket/\n",
      "GS Bucket: automlops-sandbox-dry-beans-dt-bucket already exists in project automlops-sandbox\n",
      "\u001b[0;32m Setting up Pipeline Job Runner Service Account in project automlops-sandbox \u001b[0m\n",
      "Pipeline Runner Service Account         vertex-pipelines@automlops-sandbox.iam.gserviceaccount.com           False\n",
      "Service Account: vertex-pipelines already exists in project automlops-sandbox\n",
      "\u001b[0;32m Setting up IAM roles for Pipeline Job Runner Service Account in project automlops-sandbox \u001b[0m\n",
      "\u001b[0;32m Setting up Cloud Source Repository in project automlops-sandbox \u001b[0m\n",
      "dry-beans-dt-repository              automlops-sandbox  https://source.developers.google.com/p/automlops-sandbox/r/dry-beans-dt-repository\n",
      "Cloud Source Repository: dry-beans-dt-repository already exists in project automlops-sandbox\n",
      "\u001b[0;32m Setting up Queueing Service in project automlops-sandbox \u001b[0m\n",
      "name: projects/automlops-sandbox/topics/dry-beans-dt-queueing-svc\n",
      "Pub/Sub Topic: dry-beans-dt-queueing-svc already exists in project automlops-sandbox\n",
      "\u001b[0;32m Deploying Cloud Functions: dry-beans-dt-job-submission-svc in project automlops-sandbox \u001b[0m\n",
      "Deploying function (may take a while - up to 2 minutes)...\n",
      "..\n",
      "For Cloud Build Logs, visit: https://console.cloud.google.com/cloud-build/builds;region=us-central1/21ce8664-3fb0-419e-bc56-9d284b7907d9?project=45373616427\n",
      ".....................................................done.\n",
      "automaticUpdatePolicy: {}\n",
      "availableMemoryMb: 512\n",
      "buildId: 21ce8664-3fb0-419e-bc56-9d284b7907d9\n",
      "buildName: projects/45373616427/locations/us-central1/builds/21ce8664-3fb0-419e-bc56-9d284b7907d9\n",
      "dockerRegistry: ARTIFACT_REGISTRY\n",
      "dockerRepository: projects/automlops-sandbox/locations/us-central1/repositories/dry-beans-dt-artifact-registry\n",
      "entryPoint: process_request\n",
      "eventTrigger:\n",
      "  eventType: google.pubsub.topic.publish\n",
      "  failurePolicy: {}\n",
      "  resource: projects/automlops-sandbox/topics/dry-beans-dt-queueing-svc\n",
      "  service: pubsub.googleapis.com\n",
      "ingressSettings: ALLOW_ALL\n",
      "labels:\n",
      "  deployment-tool: cli-gcloud\n",
      "maxInstances: 3000\n",
      "name: projects/automlops-sandbox/locations/us-central1/functions/dry-beans-dt-job-submission-svc\n",
      "runtime: python39\n",
      "serviceAccountEmail: vertex-pipelines@automlops-sandbox.iam.gserviceaccount.com\n",
      "sourceUploadUrl: https://storage.googleapis.com/uploads-961973632599.us-central1.cloudfunctions.appspot.com/a3809ffc-667c-467b-908c-4dbf5a0bf567.zip\n",
      "status: ACTIVE\n",
      "timeout: 540s\n",
      "updateTime: '2024-01-25T16:56:04.129Z'\n",
      "versionId: '19'\n",
      "\u001b[0;32m Setting up Cloud Build Trigger in project automlops-sandbox \u001b[0m\n",
      "name: dry-beans-dt-build-trigger\n",
      "Cloudbuild Trigger already exists in project automlops-sandbox for repo dry-beans-dt-repository\n",
      "\u001b[0;32m Setting up Cloud Scheduler Job in project automlops-sandbox \u001b[0m\n",
      "dry-beans-dt-schedule  us-central1  59 11 * * 0 (Etc/UTC)  Pub/Sub      ENABLED\n",
      "Cloud Scheduler Job: dry-beans-dt-schedule already exists in project automlops-sandbox\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.provision(hide_warnings=False)           # hide_warnings is optional, defaults to True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f9f8df2",
   "metadata": {},
   "source": [
    "`AutoMLOps.deploy(...)` builds and pushes component container, then triggers the pipeline job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c193febc",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Running precheck for deploying requires these permissions:\n",
      "-serviceusage.services.get\n",
      "-resourcemanager.projects.getIamPolicy\n",
      "-storage.buckets.update\n",
      "-iam.serviceAccounts.get\n",
      "-artifactregistry.repositories.get\n",
      "-pubsub.topics.get\n",
      "-pubsub.subscriptions.get\n",
      "-cloudbuild.builds.get\n",
      "-cloudfunctions.functions.get\n",
      "-source.repos.update\n",
      "\n",
      "You are currently using: srastatter@google.com. Please check your account permissions.\n",
      "The following are the recommended roles for deploying with precheck:\n",
      "-roles/serviceusage.serviceUsageViewer\n",
      "-roles/iam.roleViewer\n",
      "-roles/storage.admin\n",
      "-roles/iam.serviceAccountUser\n",
      "-roles/artifactregistry.reader\n",
      "-roles/pubsub.viewer\n",
      "-roles/cloudbuild.builds.editor\n",
      "-roles/cloudfunctions.viewer\n",
      "-roles/source.writer\n",
      "\n",
      "Checking for required API services in project automlops-sandbox...\n",
      "Checking for Artifact Registry in project automlops-sandbox...\n",
      "Checking for Storage Bucket in project automlops-sandbox...\n",
      "Checking for Pipeline Runner Service Account in project automlops-sandbox...\n",
      "Checking for IAM roles on Pipeline Runner Service Account in project automlops-sandbox...\n",
      "Checking for Cloud Source Repo in project automlops-sandbox...\n",
      "Checking for Pub/Sub Topic in project automlops-sandbox...\n",
      "Checking for Pub/Sub Subscription in project automlops-sandbox...\n",
      "Checking for Cloud Functions Pipeline Job Submission Service in project automlops-sandbox...\n",
      "Checking for Cloud Build Trigger in project automlops-sandbox...\n",
      "Precheck successfully completed, continuing to deployment.\n",
      "\n",
      "[automlops 0a90a9e] Run AutoMLOps\n",
      " 2 files changed, 2 insertions(+), 2 deletions(-)\n",
      "remote: Waiting for private key checker: 1/1 objects left        \n",
      "To https://source.developers.google.com/p/automlops-sandbox/r/dry-beans-dt-repository\n",
      "   e6d1067..0a90a9e  automlops -> automlops\n",
      "Pushing code to automlops branch, triggering build...\n",
      "Cloud Build job running at: https://console.cloud.google.com/cloud-build/builds;region=us-central1\n",
      "Please wait for this build job to complete.\n",
      "\n",
      "#################################################################\n",
      "#                                                               #\n",
      "#                       RESOURCES MANIFEST                      #\n",
      "#---------------------------------------------------------------#\n",
      "#     Generated resources can be found at the following urls    #\n",
      "#                                                               #\n",
      "#################################################################\n",
      "\n",
      "Google Cloud Storage Bucket: https://console.cloud.google.com/storage/automlops-sandbox-dry-beans-dt-bucket\n",
      "Artifact Registry: https://console.cloud.google.com/artifacts/docker/automlops-sandbox/us-central1/dry-beans-dt-artifact-registry\n",
      "Service Accounts: https://console.cloud.google.com/iam-admin/serviceaccounts?project=automlops-sandbox\n",
      "APIs: https://console.cloud.google.com/apis\n",
      "Cloud Build Jobs: https://console.cloud.google.com/cloud-build/builds;region=us-central1\n",
      "Vertex AI Pipeline Runs: https://console.cloud.google.com/vertex-ai/pipelines/runs\n",
      "Cloud Source Repository: https://source.cloud.google.com/automlops-sandbox/dry-beans-dt-repository/+/automlops:\n",
      "Cloud Build Trigger: https://console.cloud.google.com/cloud-build/triggers;region=us-central1\n",
      "Pipeline Job Submission Service (Cloud Functions): https://console.cloud.google.com/functions/details/us-central1/dry-beans-dt-job-submission-svc\n",
      "Pub/Sub Queueing Service Topic: https://console.cloud.google.com/cloudpubsub/topic/detail/dry-beans-dt-queueing-svc\n",
      "Pub/Sub Queueing Service Subscriptions: https://console.cloud.google.com/cloudpubsub/subscription/list\n",
      "Cloud Scheduler Job: https://console.cloud.google.com/cloudscheduler\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.deploy(precheck=True,                     # precheck is optional, defaults to True\n",
    "                 hide_warnings=False)               # hide_warnings is optional, defaults to True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3294b18",
   "metadata": {},
   "source": [
    "## Create Monitoring Jobs (Optional): can only be used with `use_ci=True` setting\n",
    "Set up the monitoring job by first getting the most recent deployed beans-model endpoint.\n",
    "\n",
    "**Note: Only run this step after the PipelineJob above has completed successfully**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "369d215d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'projects/45373616427/locations/us-central1/endpoints/6902586664619606016'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from google.cloud import aiplatform\n",
    "\n",
    "aiplatform.init(project=PROJECT_ID)\n",
    "beans_endpoints = aiplatform.Endpoint.list(filter=f'display_name=\"beans-model_endpoint\"')\n",
    "\n",
    "# Grab the most recent beans-model deployment\n",
    "endpoint_name = beans_endpoints[0].resource_name\n",
    "endpoint_name"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "236ef763",
   "metadata": {},
   "source": [
    "Install the requirements for creating model monitoring jobs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec4a3a87",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install -r AutoMLOps/model_monitoring/requirements.txt --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3492aab1",
   "metadata": {},
   "source": [
    "`AutoMLOps.monitor(...)` Creates model monitoring jobs on deployed endpoints. Users can specify the drift and skew thresholds, as well as other parameters to configure the monitoring job. Specifying `alert_emails` will send anomaly alerts to the listed emails. Specifying `auto_retraining_params` will enable automatic re-running of the above pipeline if an anomaly is detected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "96c5b999",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating ModelDeploymentMonitoringJob\n",
      "ModelDeploymentMonitoringJob created. Resource name: projects/45373616427/locations/us-central1/modelDeploymentMonitoringJobs/5631206457295765504\n",
      "To use this ModelDeploymentMonitoringJob in another session:\n",
      "mdm_job = aiplatform.ModelDeploymentMonitoringJob('projects/45373616427/locations/us-central1/modelDeploymentMonitoringJobs/5631206457295765504')\n",
      "View Model Deployment Monitoring Job:\n",
      "https://console.cloud.google.com/ai/platform/locations/us-central1/model-deployment-monitoring/5631206457295765504?project=45373616427\n",
      "Updated Anomaly Log Sink dry-beans-dt-model-monitoring-log-sink.\n",
      "\n",
      "All anomaly logs for this model monitoring job are being routed to pub/sub topic dry-beans-dt-queueing-svc for automatic retraining.\n",
      "Retraining will use the following parameters located at gs://automlops-sandbox-dry-beans-dt-bucket/pipeline_root/dry-beans-dt/automatic_retraining_parameters.json: \n",
      "\n",
      "{'bq_table': 'automlops-sandbox.test_dataset.dry_beans',\n",
      " 'data_path': 'gs://automlops-sandbox-dry-beans-dt-bucket/data.csv',\n",
      " 'gs_pipeline_spec_path': 'gs://automlops-sandbox-dry-beans-dt-bucket/pipeline_root/dry-beans-dt/pipeline_job.json',\n",
      " 'model_directory': 'gs://automlops-sandbox-dry-beans-dt-bucket/trained_models/2024-01-25 '\n",
      "                    '13:08:46.155929',\n",
      " 'project_id': 'automlops-sandbox',\n",
      " 'region': 'us-central1'}\n",
      "\n",
      "Updating cloud-logs@system.gserviceaccount.com with roles/pubsub.publisher\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.monitor(\n",
    "    alert_emails=[], # update if you would like to receive email alerts\n",
    "    target_field=TARGET_COLUMN,\n",
    "    model_endpoint=endpoint_name,\n",
    "    monitoring_interval=1,\n",
    "    # auto_retraining_params=pipeline_params, # only works if use_ci=True\n",
    "    drift_thresholds={'Area': 0.000001, 'Perimeter': 0.000001},\n",
    "    skew_thresholds={'Area': 0.000001, 'Perimeter': 0.000001},\n",
    "    training_dataset=f'bq://{TRAINING_DATASET}',\n",
    "    hide_warnings=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf672b9c",
   "metadata": {},
   "source": [
    "### Test the monitoring job by sending some sample requests\n",
    "The below code will send a request for predicting 5000 instances. Based on the above configuration, Vertex Model monitoring will run a monitoring job every hour at the top of the hour, compile skew and drift statistics, and compare to the thresholds specified. Thus, the below prediction code should produce a series of alerts in a few hours, and trigger a retraining of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "587a0bb2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0\n"
     ]
    }
   ],
   "source": [
    "from google.cloud import bigquery\n",
    "import pandas as pd\n",
    "\n",
    "def get_query(bq_input_table: str) -> str:\n",
    "    \"\"\"Generates BQ Query to read data.\n",
    "\n",
    "    Args:\n",
    "        bq_input_table: The full name of the bq input table to be read into\n",
    "        the dataframe (e.g. <project>.<dataset>.<table>)\n",
    "\n",
    "    Returns: A BQ query string.\n",
    "    \"\"\"\n",
    "    return f'''SELECT * FROM `{bq_input_table}`'''\n",
    "\n",
    "def load_bq_data(query: str, client: bigquery.Client) -> pd.DataFrame:\n",
    "    \"\"\"Loads data from bq into a Pandas Dataframe for EDA.\n",
    "\n",
    "    Args:\n",
    "        query: BQ Query to generate data.\n",
    "        client: BQ Client used to execute query.\n",
    "\n",
    "    Returns:\n",
    "        pd.DataFrame: A dataframe with the requested data.\n",
    "    \"\"\"\n",
    "    df = client.query(query).to_dataframe()\n",
    "    return df\n",
    "\n",
    "bq_client = bigquery.Client(project=PROJECT_ID)    \n",
    "\n",
    "# Get samples\n",
    "df = load_bq_data(get_query(TRAINING_DATASET), bq_client)\n",
    "X_sample = df.iloc[:,:-1][:5000].values.tolist()\n",
    "\n",
    "endpoint = aiplatform.Endpoint(endpoint_name)\n",
    "response = endpoint.predict(instances=X_sample)\n",
    "prediction = response[0]\n",
    "# print the first prediction\n",
    "print(prediction[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf929353",
   "metadata": {},
   "source": [
    "# Train using a GPU\n",
    "Use the `custom_training_job_specs` parameter to specify custom resources for any custom component in the pipeline. The example below uses a GPU for accelerated training.\n",
    "See [Machine types](https://cloud.google.com/vertex-ai/docs/training/configure-compute#machine-types) and [GPUs](https://cloud.google.com/vertex-ai/docs/training/configure-compute#specifying_gpus)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "89f42645",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing directories under AutoMLOps/\n",
      "Writing configurations to AutoMLOps/configs/defaults.yaml\n",
      "Writing README.md to AutoMLOps/README.md\n",
      "Writing kubeflow pipelines code to AutoMLOps/pipelines, AutoMLOps/components\n",
      "Writing scripts to AutoMLOps/scripts\n",
      "Writing submission service code to AutoMLOps/services\n",
      "Writing gcloud provisioning code to AutoMLOps/provision\n",
      "Writing cloud build config to AutoMLOps/cloudbuild.yaml\n",
      "Code Generation Complete.\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.generate(project_id=PROJECT_ID, \n",
    "                   pipeline_params=pipeline_params, \n",
    "                   use_ci=False, \n",
    "                   naming_prefix=MODEL_ID,\n",
    "                   deployment_framework='cloud-build',\n",
    "                   base_image='us-docker.pkg.dev/vertex-ai/training/tf-gpu.2-11.py310:latest', # includes required cuda pacakges\n",
    "                   custom_training_job_specs = [{\n",
    "                       'component_spec': 'train_model',\n",
    "                       'display_name': 'train-model-accelerated',\n",
    "                       'machine_type': 'n1-standard-8',\n",
    "                       'accelerator_type': 'NVIDIA_TESLA_V100',\n",
    "                       'accelerator_count': 1\n",
    "                   }]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccca3907",
   "metadata": {},
   "source": [
    "## Default Run Settings\n",
    "Below are the default parameters for running `AutoMLOps`. Note there are only two required parameters:\n",
    "1. project_id\n",
    "2. pipeline_params\n",
    "\n",
    "The other parameters are optional. You can customize the output of `AutoMLOps` by specify the resources you'd like to use (or specifying the name of resources you'd like `AutoMLOps` to create if they don't currently exist). A description of the parameters is below:\n",
    "- `project_id`: The project ID.\n",
    "- `pipeline_params`: Dictionary containing runtime pipeline parameters.\n",
    "- `artifact_repo_location`: Region of the artifact repo (default use with Artifact Registry).\n",
    "- `artifact_repo_name`: Artifact repo name where components are stored (default use with Artifact Registry).\n",
    "- `artifact_repo_type`: The type of artifact repository to use (e.g. Artifact Registry, JFrog, etc.)        \n",
    "- `base_image`: The image to use in the component base dockerfile.\n",
    "- `build_trigger_location`: The location of the build trigger (for cloud build).\n",
    "- `build_trigger_name`: The name of the build trigger (for cloud build).\n",
    "- `custom_training_job_specs`: Specifies the specs to run the training job with.\n",
    "- `deployment_framework`: The CI tool to use (e.g. cloud build, github actions, etc.)\n",
    "- `naming_prefix`: Unique value used to differentiate pipelines and services across AutoMLOps runs.\n",
    "- `orchestration_framework`: The orchestration framework to use (e.g. kfp, tfx, etc.)\n",
    "- `pipeline_job_runner_service_account`: Service Account to run PipelineJobs (specify the full string).\n",
    "- `pipeline_job_submission_service_location`: The location of the cloud submission service.\n",
    "- `pipeline_job_submission_service_name`: The name of the cloud submission service.\n",
    "- `pipeline_job_submission_service_type`: The tool to host for the cloud submission service (e.g. cloud run, cloud functions).\n",
    "- `precheck`: Boolean used to specify whether to check for provisioned resources before deploying.\n",
    "- `project_number`: The project number.\n",
    "- `provision_credentials_key`: Either a path to or the contents of a service account key file in JSON format.\n",
    "- `provisioning_framework`: The IaC tool to use (e.g. Terraform, Pulumi, etc.)\n",
    "- `pubsub_topic_name`: The name of the pubsub topic to publish to.\n",
    "- `schedule_location`: The location of the scheduler resource.\n",
    "- `schedule_name`: The name of the scheduler resource.\n",
    "- `schedule_pattern`: Cron formatted value used to create a Scheduled retrain job.\n",
    "- `setup_model_monitoring`: Boolean parameter which specifies whether to set up a Vertex AI Model Monitoring Job.\n",
    "- `source_repo_branch`: The branch to use in the source repository.\n",
    "- `source_repo_name`: The name of the source repository to use.\n",
    "- `source_repo_type`: The type of source repository to use (e.g. gitlab, github, etc.)\n",
    "- `storage_bucket_location`: Region of the GS bucket.\n",
    "- `storage_bucket_name`: GS bucket name where pipeline run metadata is stored.\n",
    "- `hide_warnings`: Boolean used to specify whether to show provision/deploy permission warnings\n",
    "- `use_ci`: Flag that determines whether to use Cloud CI/CD.\n",
    "- `vpc_connector`: The name of the vpc connector to use.\n",
    "- `workload_identity_pool`: Pool for workload identity federation. \n",
    "- `workload_identity_provider`: Provider for workload identity federation.\n",
    "- `workload_identity_service_account`: Service account for workload identity federation (specify the full string).\n",
    "\n",
    "The `use_ci` parameter specifies whether to use the generated `scripts/run_all.sh` local script to submit the build job and PipelineJob. If this parameter is set to True, `AutoMLOps` will use the cloud [CI/CD workflow](https://github.com/GoogleCloudPlatform/automlops#deployment). The run above uses `use_ci=True`, and the run below uses `use_ci=False`, notice the differences in output (`use_ci=False` means you will not use the Source Repository to trigger build jobs on push). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c92f80a8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing directories under AutoMLOps/\n",
      "Writing configurations to AutoMLOps/configs/defaults.yaml\n",
      "Writing Kubeflow Pipelines code to AutoMLOps/pipelines, AutoMLOps/components, AutoMLOps/services\n",
      "Writing README.md to AutoMLOps/README.md\n",
      "Writing scripts to AutoMLOps/scripts\n",
      "Writing CloudBuild config to AutoMLOps/cloudbuild.yaml\n",
      "Code Generation Complete.\n"
     ]
    }
   ],
   "source": [
    "AutoMLOps.generate(project_id=PROJECT_ID, # required\n",
    "                   pipeline_params=pipeline_params, # required\n",
    "                   artifact_repo_location='us-central1', # default\n",
    "                   artifact_repo_name=None, # default\n",
    "                   artifact_repo_type='artifact-registry', # default\n",
    "                   base_image='python:3.9-slim', # default\n",
    "                   build_trigger_location='us-central1', # default\n",
    "                   build_trigger_name=None, # default\n",
    "                   custom_training_job_specs=None, # default\n",
    "                   deployment_framework='github-actions', # default\n",
    "                   naming_prefix='automlops-default-prefix', # default\n",
    "                   orchestration_framework='kfp', # default\n",
    "                   pipeline_job_runner_service_account=None, # default\n",
    "                   pipeline_job_submission_service_location='us-central1', # default\n",
    "                   pipeline_job_submission_service_name=None, # default\n",
    "                   pipeline_job_submission_service_type='cloud-functions', # default\n",
    "                   project_number=None, # default\n",
    "                   provision_credentials_key=None, # default\n",
    "                   provisioning_framework='gcloud', # default\n",
    "                   pubsub_topic_name=None, # default\n",
    "                   schedule_location='us-central1', # default\n",
    "                   schedule_name=None, # default\n",
    "                   schedule_pattern='No Schedule Specified', # default\n",
    "                   setup_model_monitoring=False, # default\n",
    "                   source_repo_branch='automlops', # default\n",
    "                   source_repo_name=None, # default\n",
    "                   source_repo_type='github', # default\n",
    "                   storage_bucket_location='us-central1', # default\n",
    "                   storage_bucket_name=None, # default\n",
    "                   use_ci=False, # default\n",
    "                   vpc_connector='No VPC Specified', # default\n",
    "                   workload_identity_pool=None, # default\n",
    "                   workload_identity_provider=None, # default\n",
    "                   workload_identity_service_account=None, # default\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "common-cpu.m102",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/base-cpu:m102"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}