notebook_examples/visualizing_data-exploring_data.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{visualizing_data-exploring_data.ipynb,\n",
    "    title: Visualizing Data,\n",
    "    summary: Perform common data visualization tasks and explore data with the ADS SDK. Plotting approaches include 3D plots, pie chart, GIS plots, and Seaborn pairplot graphs.,\n",
    "    developed_on: generalml_p38_cpu_v1,\n",
    "    keywords: data visualization, seaborn plot, charts,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2019, 2022 Oracle, Inc. All rights reserved. Licensed under the [Universal Permissive License v 1.0](https://oss.oracle.com/licenses/upl).\n",
    "\n",
    "---\n",
    "# <font color=\"red\">Visualizing Data</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=teal>Oracle Cloud Infrastructure Data Science Service Team</font></p>\n",
    "\n",
    "---\n",
    "    \n",
    "# Overview:\n",
    "\n",
    "Data visualization is an important component of data exploration and data analysis in modern data science practices. An efficient and flexible data visualization tool can provide more insight into the data for data scientists.\n",
    "\n",
    "This notebook provides an overview of the data visualizations that you can perform with ADS. It will focus on smart data visualization technology that uses the columns types and other settings to atomically great an intuitive plot for your data.\n",
    "\n",
    "Compatible with: [General Machine Learning](https://docs.oracle.com/en-us/iaas/data-science/using/conda-gml-fam.htm) for CPU on Python 3.8 (version 1.0)\n",
    "\n",
    "---\n",
    "    \n",
    "## Contents:\n",
    "\n",
    "- <a href='#data'>Dataset</a>\n",
    "- <a href='#eda'>Exploratory Data Analysis</a>\n",
    "    - <a href='#eda_target'>Plot Target Distribution</a>\n",
    "    - <a href='#eda_feature'>Plot Feature Distributions</a>\n",
    "    - <a href='#eda_target'>Automatic Feature Plotting</a>\n",
    "- <a href='#custom'>Custom Plotting</a>\n",
    "    - <a href='#custom_lambda'>Ploting with Lambdas</a>\n",
    "    - <a href='#custom_3d'>3D Ploting</a> \n",
    "    - <a href='#custom_pairplot'>Seaborn's `pairplot` Method</a>\n",
    "    - <a href='#custom_matplotlib'>Matplotlib</a>\n",
    "    - <a href='#custom_pie'>Pie Chart</a>\n",
    "    - <a href='#custom_gis'>GIS Plot</a> \n",
    "- <a href='#ref'>References </a> \n",
    " \n",
    "---\n",
    " \n",
    "Datasets are provided as a convenience.  Datasets are considered third-party content and are not considered materials \n",
    "under your agreement with Oracle.\n",
    "    \n",
    "You can access the `earthquake` dataset license [here](https://creativecommons.org/publicdomain/zero/1.0/).    \n",
    "\n",
    "You can access the `iris` dataset license [here](https://github.com/scikit-learn/scikit-learn/blob/master/COPYING).  \n",
    "    \n",
    "You can access the `oracle_classification_dataset1` dataset license [here](https://oss.oracle.com/licenses/upl). \n",
    "    \n",
    "You can access the `oracle_traffic_timeseries_dataset1.csv` dataset license [here](https://oss.oracle.com/licenses/upl). \n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "\n",
    "from ads.dataset.factory import DatasetFactory\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "from numpy.random import randn\n",
    "from os.path import join\n",
    "from sklearn.utils import Bunch\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "logging.basicConfig(format=\"%(levelname)s:%(message)s\", level=logging.ERROR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='data'></a>\n",
    "# Dataset\n",
    "\n",
    "You are working with the Oracle Classification Dataset. This has a set of features and a binary (`1`/`0`) target called `class`.\n",
    "\n",
    "The `oracle_classification_dataset1_150K.csv` file is stored here on Oracle ArtiFactory, but the source could be any number of locations, Oracle Storage, HDFS, Git etc. The format and additional options are inferred, however, there are many options to control how the `.open()` method works. It can also convert any local Pandas DataFrame to a Dataset.\n",
    "\n",
    "The data is downsampled down to 1,500 rows and 21 columns. The columns describe the different attributes of each row.\n",
    "\n",
    "If you don't yet know the target in your project, you can explore the data first and set the target later.\n",
    "\n",
    "In this notebook, you will be working with a CSV file that is stored on the network. The `DatasetFactory` class allows you to load from both the local and network file system. You can read many different file formats such as CSV, TSV, Parquet, libsvm, JSON, Excel, HDF5, SQL, XML, apache server log files (clf, log), and ARFF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = join(\n",
    "    \"/\",\n",
    "    \"opt\",\n",
    "    \"notebooks\",\n",
    "    \"ads-examples\",\n",
    "    \"oracle_data\",\n",
    "    \"oracle_classification_dataset1_150K.csv\",\n",
    ")\n",
    "ds_preview = DatasetFactory.from_dataframe(data_path, target=\"class\")\n",
    "ds_preview = ds_preview[\n",
    "    [\n",
    "        \"class\",\n",
    "        \"col01\",\n",
    "        \"col02\",\n",
    "        \"col03\",\n",
    "        \"col04\",\n",
    "        \"col05\",\n",
    "        \"col06\",\n",
    "        \"col07\",\n",
    "        \"col08\",\n",
    "        \"col09\",\n",
    "        \"col010\",\n",
    "        \"col011\",\n",
    "        \"col012\",\n",
    "        \"col013\",\n",
    "        \"col014\",\n",
    "        \"col015\",\n",
    "        \"col016\",\n",
    "        \"col017\",\n",
    "        \"col018\",\n",
    "        \"col019\",\n",
    "        \"col020\",\n",
    "    ]\n",
    "].sample(frac=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='eda'></a>\n",
    "# Exploratory Data Analysis\n",
    "\n",
    "<a id='eda_target'></a>\n",
    "## Plot Target Distribution\n",
    "\n",
    "Let's take a look at the distribution of the target column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_preview.target.show_in_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above cell, the target column `count` is a categorical value, therefore the smart data visualization tool selected a `count plot`. The above plot demonstrates that the count of class 1 is more than the count of class 0.\n",
    "\n",
    "<a id='eda_feature'></a>\n",
    "## Plot Feature Distributions\n",
    "\n",
    "The next cell will plot a set of features against the target feature by specifying a list of feature names in the parameter `feature_names` in the method of `show_in_notebook`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_preview.target.show_in_notebook(feature_names=[\"col01\", \"col02\", \"col03\", \"col09\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above cell demonstrates that given different types of features, the ADS SDK selected different plotting methods. When plotting `col01` (a continuous variable) against `class` (a categorical variable) a family of PDF curves was the most appropriate plot. Meanwhile, when plotting `col02` against `class`, in which both are categorical variables, a count plot was created.\n",
    "\n",
    "<a id='eda_target'></a>\n",
    "## Automatic Feature Plotting\n",
    "\n",
    "The `.plot()` method is an automatic plotting method. Users can pass in a variable for the x-axis and optionally a variable for y., and then call `show_in_notebook()` method to plot. Here are some examples using the Oracle Classification Synthetic dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_preview.plot(\"col02\").show_in_notebook(figsize=(4, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above cell, since you only passed the x variable `col02`, which is a categorical variable. ADS automatic plotting used `countplot`, which is a simple and straightforward visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_preview.plot(\"col02\", y=\"col01\").show_in_notebook(figsize=(4, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this above example, you are plotting `col02` against `col01`, with one being a categorical typed feature and one being a continuous typed feature, the best plotting method is violin plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_preview.plot(\"col01\").show_in_notebook(figsize=(4, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The automatic plotting routine used a histogram to plot `col01` as it was a continuous variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_preview.plot(\"col01\", y=\"col03\").show_in_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When plotting `col01` against `col03`, which are both continuous typed features, the ADS SDK used a Gaussian heatmap to visualize the data. It generates a scatter plot and assigns a color to each data point based on the local density (Gaussian kernel) of points.\n",
    "ADS SDK analyzes the data and selects an appropriate plot type. Here are some showcase examples using `oracle_traffic_timeseries` datasets.\n",
    "\n",
    "As you can probably tell at this point, our ADS SDK can pick the best plotting type based on different data types. Here are some showcase examples using the Oracle traffic time series dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = join(\n",
    "    \"/\",\n",
    "    \"opt\",\n",
    "    \"notebooks\",\n",
    "    \"ads-examples\",\n",
    "    \"oracle_data\",\n",
    "    \"oracle_traffic_timeseries_dataset1.csv\",\n",
    ")\n",
    "oracle_traffic_timeseries = DatasetFactory.from_dataframe(data_path)\n",
    "oracle_traffic_timeseries.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above cell visualizes the relationship between `date` and `cloud_coverage` using a scatter plot. It shows how the value of the ordinal variable `cloud_coverage` changes across different years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oracle_traffic_timeseries.plot(\"weather\", y=\"cloud_coverage\").show_in_notebook(\n",
    "    figsize=(4, 4)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By plotting `weather` against `cloud_coverage`, you can visualize the count of different kinds of weather that occurred in different cloud coverages.\n",
    "\n",
    "<a id='custom'></a>\n",
    "# Custom Plotting\n",
    "\n",
    "The `.call()` method allows users to have a more flexible way to plotting using their preferred plotting libraries/packages.\n",
    "\n",
    "<a id='custom_lambda'></a>\n",
    "## Ploting with Lambdas\n",
    "\n",
    "Here is an example of a matplotlib scatter plot with the custom lambda function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oracle_traffic_timeseries.call(\n",
    "    lambda df, x, y: plt.scatter(df[x], df[y]), x=\"cloud_coverage\", y=\"sensor4\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='custom_3d'></a>\n",
    "## 3D Plots\n",
    "\n",
    "This section showcases 3D plotting using the `iris` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_iris()\n",
    "iris_df = pd.DataFrame(data.data, columns=data.feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_3d_plot(df, figsize=None):\n",
    "    plt.subplots_adjust(left=0, bottom=0, right=1, top=1, wspace=1, hspace=1)\n",
    "    plt.style.use(\"seaborn-white\")\n",
    "\n",
    "    fig = plt.figure(figsize=figsize)\n",
    "    ax = fig.add_subplot(111, projection=\"3d\")\n",
    "\n",
    "    ax.scatter(df[\"sepal_length_(cm)\"], df[\"sepal_width_(cm)\"], df[\"petal_length_(cm)\"])\n",
    "\n",
    "    ax.set_xlabel(\"Sepal Length (cm)\")\n",
    "    ax.set_ylabel(\"Sepal Width (cm)\")\n",
    "    ax.set_zlabel(\"Petal Length\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = DatasetFactory.from_dataframe(iris_df)\n",
    "ds.call(my_3d_plot, figsize=(10, 10));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='custom_pairplot'></a>\n",
    "## Seaborn's `pairplot` Method\n",
    "\n",
    "The next cell demonstrates how the dataframe is passed directly to the Seaborn `.pairplot()` method. It plots a pairwise relationship for the dataset. This function will create a grid of Axes such that each variable in data will be shared in the `y-axis` across a single row and in the `x-axis` across a single column. The diagonal Axes are treated differently, drawing a plot to show the univariate distribution of the data for the variable in that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set(style=\"ticks\", color_codes=True)\n",
    "DatasetFactory.from_dataframe(iris_df).call(lambda df: sns.pairplot(df.dropna()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='custom_matplotlib'></a>\n",
    "## Matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(randn(1000, 4), columns=list(\"ABCD\"))\n",
    "\n",
    "\n",
    "def ts_plot(df, figsize):\n",
    "    ts = pd.Series(randn(1000), index=pd.date_range(\"1/1/2000\", periods=1000))\n",
    "    df.set_index(ts)\n",
    "    df = df.cumsum()\n",
    "    plt.figure()\n",
    "    df.plot(figsize=figsize)\n",
    "    plt.legend(loc=\"best\")\n",
    "\n",
    "\n",
    "ds = DatasetFactory.from_dataframe(df, target=\"A\")\n",
    "ds.call(ts_plot, figsize=(7, 7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='custom_pie'></a>\n",
    "## Pie Chart\n",
    "\n",
    "In this example, you make a customized pie-chart and show how to load data using the `DatasetFactory` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\"data\": [1109, 696, 353, 192, 168, 86, 74, 65, 53]}\n",
    "df = pd.DataFrame(\n",
    "    data,\n",
    "    index=[\n",
    "        \"20-50 km\",\n",
    "        \"50-75 km\",\n",
    "        \"10-20 km\",\n",
    "        \"75-100 km\",\n",
    "        \"3-5 km\",\n",
    "        \"7-10 km\",\n",
    "        \"5-7 km\",\n",
    "        \">100 km\",\n",
    "        \"2-3 km\",\n",
    "    ],\n",
    ")\n",
    "\n",
    "\n",
    "explode = (0, 0, 0, 0.1, 0.1, 0.2, 0.3, 0.4, 0.6)\n",
    "colors = [\n",
    "    \"#191970\",\n",
    "    \"#001CF0\",\n",
    "    \"#0038E2\",\n",
    "    \"#0055D4\",\n",
    "    \"#0071C6\",\n",
    "    \"#008DB8\",\n",
    "    \"#00AAAA\",\n",
    "    \"#00C69C\",\n",
    "    \"#00E28E\",\n",
    "    \"#00FF80\",\n",
    "]\n",
    "\n",
    "\n",
    "def bar_plot(df, figsize):\n",
    "    df[\"data\"].plot(kind=\"pie\", fontsize=17, colors=colors, explode=explode)\n",
    "    plt.axis(\"equal\")\n",
    "    plt.ylabel(\"\")\n",
    "    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.0)\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "ds = DatasetFactory.from_dataframe(df)\n",
    "ds.call(bar_plot, figsize=(7, 7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='custom_gis'></a>\n",
    "## GIS Plot\n",
    "\n",
    "Here are some examples where you can visualize geographical data using ADS SDK visualizations.\n",
    "\n",
    "For this example, the dataset used is the California earthquake data retrieved from the USGS earthquake catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = join(\n",
    "    \"/\", \"opt\", \"notebooks\", \"ads-examples\", \"3P_data\", \"earthquake_01.csv\"\n",
    ")\n",
    "earthquake = DatasetFactory.from_dataframe(data_path, target=\"depth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A brief overview to visual major places where earthquakes happened. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "earthquake.plot_gis_scatter(lon=\"longitude\", lat=\"latitude\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "earthquake.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell, you will do minor transformations using Pandas so that our earthquake dataset contains the column `location` which has the format of `\"(latitude, longitude)\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = earthquake.compute()\n",
    "earthquake_df = df.assign(location=[*zip(df.latitude, df.longitude)]).astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell, you plot the column `location` using `.plot`, which outputs an interactive map that gives you the flexibility to zoom in/out, identify outliers/inliers etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "earthquake_02 = DatasetFactory.from_dataframe(earthquake_df)\n",
    "earthquake_02.plot(\"location\").show_in_notebook()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}