Chapter 1 - Joint and Conditional Probability.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#-Joint-Probability\" data-toc-modified-id=\"-Joint-Probability-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Joint Probability</font></a></span><ul class=\"toc-item\"><li><span><a href=\"#-Example-1\" data-toc-modified-id=\"-Example-1-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 1</font></a></span></li><li><span><a href=\"#-Example-2\" data-toc-modified-id=\"-Example-2-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 2</font></a></span></li><li><span><a href=\"#-Example-3\" data-toc-modified-id=\"-Example-3-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 3</font></a></span></li></ul></li><li><span><a href=\"#-Conditional-Probability\" data-toc-modified-id=\"-Conditional-Probability-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Conditional Probability</font></a></span><ul class=\"toc-item\"><li><span><a href=\"#-Example-1\" data-toc-modified-id=\"-Example-1-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 1</font></a></span></li><li><span><a href=\"#-Example-2\" data-toc-modified-id=\"-Example-2-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 2</font></a></span></li><li><span><a href=\"#-Example-3\" data-toc-modified-id=\"-Example-3-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 3</font></a></span></li></ul></li><li><span><a href=\"#-Independence\" data-toc-modified-id=\"-Independence-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Independence</font></a></span><ul class=\"toc-item\"><li><span><a href=\"#-Independence-vs-Disjoint\" data-toc-modified-id=\"-Independence-vs-Disjoint-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Independence vs Disjoint</font></a></span></li></ul></li><li><span><a href=\"#Law-of-Total-Probability\" data-toc-modified-id=\"Law-of-Total-Probability-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\">Law of Total Probability</font></a></span><ul class=\"toc-item\"><li><span><a href=\"#-Example-1\" data-toc-modified-id=\"-Example-1-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 1</font></a></span></li><li><span><a href=\"#-Example-2\" data-toc-modified-id=\"-Example-2-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 2</font></a></span></li><li><span><a href=\"#-Example-3\" data-toc-modified-id=\"-Example-3-4.3\"><span class=\"toc-item-num\">4.3&nbsp;&nbsp;</span><font face=\"gotham\" color=\"purple\"> Example 3</font></a></span></li></ul></li></ul></div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Joint Probability</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Joint probability** and **conditional probability** are closely intertwined, however elusive for beginners, sometimes even experienced data scientists are hard to tell the subtlety. \n",
    "\n",
    "We will tackle the joint probability first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The joint probability is defined as\n",
    "\n",
    "$$\n",
    "P(A\\cap B) = P(A)P(B)\n",
    "$$\n",
    "\n",
    "when we talking about joint probability, we implicitly assume that\n",
    "\n",
    "1. $A$ and $B$ are independent events.\n",
    "2. $A$ and $B$ happen in the same time.\n",
    "\n",
    "For instance, I throw a dice (event A) and you also throw a dice (event B) in the mean time. The joint probability of obtaining two $1$'s is $\\frac{1}{6}\\times\\frac{1}{6}$. \n",
    "\n",
    "If $A$ and $B$ are not independent, then it is pointless to talk about joint probability, because they are affection each other's results. In other words, if $A$ and $B$ are dependent, when $A$ happens, the probability of $B$ happening is no longer $P(B)$ anymore, hence $P(AB) = P(A)P(B)$ won't hold in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 1</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the joint probability of rolling the number $6$ twice in a fair six-sided dice?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "Throwing twice dice at a time are independent events, <br>\n",
    "\n",
    "$\\text{Event A}$ = $\\text{Obtaining number 6}$\n",
    "\n",
    "$$\n",
    "P(A\\cap A)=\\frac{1}{6}\\frac{1}{6}= \\frac{1}{36}\n",
    "$$\n",
    "</div> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 2</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the joint probability of obtaining 3 times head followed by 3 times tail in a sequence of coin tosses?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event $A$}$ = $\\text{Head}$<br>\n",
    "$\\text{Event $B$}$ =$\\text{Tail}$\n",
    "\n",
    "$$\n",
    "P(BBBAAA)= .5^6\n",
    "$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.015625"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ".5**6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 3</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the joint probability that you pick a card which is  <b>larger</b> than 4 and <b>less</b> than $10$ and in the meanwhile a $diamond$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$ = $\\text{Getting card in (4<x<10)}$<br>\n",
    "$\\text{Event B}$ = $\\text{Getting a diamond}$.\n",
    "\n",
    "There are five cards in range $(4, 10)$, probability is $\\frac{5}{52}$, considering four suits $\\frac{5\\times 4}{52}$, and thirteen cards for each suit.\n",
    "\n",
    "$P(A\\cap B) = P(A)P(B) = \\frac{5\\times 4}{52}\\frac{13}{52}$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09615384615384616"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(5*4/52)*(13/52)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Conditional Probability</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conditional probability is more complicated, denoted as\n",
    "\n",
    "$$\n",
    "P(B|A)= \\frac{P(A\\cap B)}{P(A)}=\\frac{P(A\\cap B)}{P(A\\cap B)+P(A\\cap B^C)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which reads, the probability of $B$ given $A$. Here we treat $A$ as a fact, it either happens or assumed to happen. $B^C$ is the complement event of $A$, such that $P(B^C)+P(B) =1 $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 1</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the probability of drawing a card, given which is a $\\text{diamond}$, that is also <b>larger</b> than 4 and <b>less</b> than $10$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how we rephrase the question than Example 3 in last section. When we are talking about conditional probability, the sample space is usually different from joint probability, here the sample space changes from all deck to $\\text{diamonds}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$ = $\\text{Getting card in (4<x<10)}$<br>\n",
    "$\\text{Event B}$ = $\\text{Getting a diamond}$.<br><br>\n",
    "    \n",
    "There are $13$ cards of diamonds, this is the new sample space. And in this new sample space, we count how many numbers there are in $(4,\\ 10)$. There are 5 numbers. Thus\n",
    "\n",
    "$P(A|B) = \\frac{5}{13}$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.38461538461538464"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "5/13"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then what is the joint probability of $P(A\\cap B)$? Using multiplication rule $P(A\\cap B) = P(A|B)P(B)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.09615384615384616"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "5/13*1/4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The joint probability only makes sense is because drawing a $\\text{diamond}$ and in $(4 <x< 10)$ are independent events."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have reproduced the same result of the joint probability in Example 3 of last section. Note that, this is also a common way of calculating joint probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 2</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We roll a fair die twice and obtain two numbers $X_1= a$ and $X_2= b$. Given $a+b=7$, what is the probability that $a=4$ or $b=4$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "Our goal is to find out $P(a=4|a+b=7)$ and $P(b=4|a+b=7)$.\n",
    "\n",
    "$\\text{Event A}$ : $a = 4$ or $b = 4$<br>\n",
    "$\\text{Event B}$ : $a+b = 7$\n",
    "\n",
    "There are only two possibilities of obtaining a $7$ if either $a$ or $b$ equals $4$, that are $(3, 4)$ and $(4, 3)$. And two dice have $6^2 = 36$ combinations, thus $P(A\\cap B) = \\frac{2}{36}$.\n",
    "\n",
    "To obtain $7$ there are $6$ combination: $(1, 6), (2, 5), (3,4), (4,3), (5,2),(6, 1)$, this is actually the sample space, so we get\n",
    "\n",
    "$$\\frac{P(A\\cap B)}{P(B)}= \\frac{2}{36}\\Big/\\frac{6}{36}=\\frac{1}{3}$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 3</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a family that has two children. We are interested in the children's genders. The sample space of gender is $S=\\{(G,G),(G,B),(B,G),(B,B)\\}$. Also assume that all four possible outcomes are equally likely.\n",
    "\n",
    "1. What is the probability that both children are girls given that the first child is a girl?\n",
    "2. We ask the father: \"Do you have at least one daughter?\" He responds \"Yes!\". Given this extra information, what is the probability that both children are girls? In other words, what is the probability that both children are girls given that we know at least one of them is a girl?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "Since we have sample space explicitly presented, the conditional probability is simply counting events\n",
    "\n",
    "The first question's sample space is $\\{(G,G),(G,B)\\}$, the answer is $1/2$.\n",
    "\n",
    "The second question's sample space is $\\{(G,G),(G,B),(B,G)\\}$, the answer is $1/3$.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Independence</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let me define two events:\n",
    "\n",
    "$\\text{Event A}$: $\\text{Obtaining head when tossing a coin}$<br>\n",
    "$\\text{Event B}$: $\\text{Raining today}$<br>\n",
    "\n",
    "What is $P(A|B)$? Unless you believe weather will affect your winning odds of gambling, the sober answer is that it doesn't have a conditional probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "Apparently, for independent events, conditional probability reduces to $P(A|B) = P(A)$. Formally derived from\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "P(A | B) &=\\frac{P(A \\cap B)}{P(B)} \\\\\n",
    "&=\\frac{P(A) P(B)}{P(B)} \\\\\n",
    "&=P(A)\n",
    "\\end{aligned}\n",
    "$$\n",
    "That's exactly why we mentioned in the beginning that joint probability only makes sense in the context of probabilistic independence.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Independence vs Disjoint</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Independence and disjoint are entirely disparate concepts. **Disjoint events** mean if $A$ happens, then $B$ certainly can't happen, which exactly assures they are dependent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Formally, consider two events $A$ and $B$, with $P(A) \\neq 0$ and $P(B) \\neq 0$. \n",
    "\n",
    "We can show that if $A$ and $B$ are disjoint, we have\n",
    "\n",
    "$$\n",
    "P(A \\cap B)=\\varnothing  \\neq P(A) P(B)\n",
    "$$\n",
    "\n",
    "Thus, $A$ and $B$ are not independent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\">Law of Total Probability</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $B_{1}, B_{2}, B_{3}, \\cdots B_k$ are partitions of the sample space $S$, then for any event $A$ we have\n",
    "\n",
    "$$\n",
    "P(A)=\\sum_{i=1}^k P\\left(A \\cap B_{i}\\right)=\\sum_{i=1}^k  P\\left(A | B_{i}\\right) P\\left(B_{i}\\right)\n",
    "$$\n",
    "\n",
    "This is called **law of total probability**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 1</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A person has undertaken a mining job. The probabilities of completion of job on time with and without rain are $0.42$ and $0.90$ respectively. If the probability that it will rain is $0.45$, then determine the probability that the mining job will be completed on time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total probability to some extent means that you have to consider all scenarios, in this case raining $R$ and not raining $R^C$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event R}$ : $\\text{raining}$<br>\n",
    "\n",
    "$\\text{Event C}$ : $\\text{complete the job on time}$\n",
    "$$p(C) = p(C|R)p(R) + p(C|R^c)p(R^c) = .42\\times .45 + .9\\times .55$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of completing mining job on time is 68.400%.\n"
     ]
    }
   ],
   "source": [
    "P_C = .42*.45+.9*.55\n",
    "print('The probability of completing mining job on time is %.3f%%.'%(P_C*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 2</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We draw two cards from a well-shuffled deck without replacement. Find the probability of getting a queen in the second draw. So we calculate probabilities of two scenarios (first card queen and first card not queen) then sum up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event Q1}$ : $\\text{Drawing a queen in the first card}$<br>\n",
    "\n",
    "$\\text{Event Q2}$ : $\\text{Drawing a queen in the second card}$\n",
    "\n",
    "\n",
    "$$\n",
    "P(Q2) = P(Q2|Q1)P(Q1) + P(Q2|Q1^c)P(Q1^c) = \\frac{3}{51}\\frac{4}{52}+\\frac{4}{51}\\frac{48}{52}\n",
    "$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of drawing a queen in the second card is 7.692%.\n"
     ]
    }
   ],
   "source": [
    "PQ2 = 3/51*4/52+4/51*48/52\n",
    "print('The probability of drawing a queen in the second card is %.3f%%.'%(PQ2*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example 3</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a certain county<br>\n",
    "1. $60 \\%$ of registered voters are Republicans \n",
    "2. $30 \\%$ are Democrats\n",
    "3. $10 \\%$ are Independents\n",
    "\n",
    "When those voters were asked opinions on increasing military spending\n",
    "1. $40 \\%$ of Republicans opposed it \n",
    "2. $65 \\%$ of the Democrats opposed it \n",
    "3. $55 \\%$ of the Independents opposed it.\n",
    "\n",
    "What is the probability that a randomly selected voter in this county opposes increasing military spending? \n",
    "\n",
    "Three scenarios to be taken into consideration, being a Republican, a Democrat or an Independent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color:Lavender; color:DarkBlue; padding:30px;\">\n",
    "$\\text{Event A}$: $\\text{opposes increased military spending}$<br>\n",
    "$\\text{Event B}$: $\\text{Being Republican}$<br>\n",
    "$\\text{Event C}$: $\\text{Being Democrats}$<br>\n",
    "$\\text{Event D}$: $\\text{Being Independents}$\n",
    "\n",
    "\n",
    "$$P(A) = P(A|B)P(B)+P(A|C)P(C)+P(A|D)P(D) = .4\\times.6+ .65\\times.3+ .55\\times.1$$\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability that a randomly selected voter in this county opposes increased military spending is 49.000%.\n"
     ]
    }
   ],
   "source": [
    "PA = .4*.6+.65*.3+.55*.1\n",
    "print('The probability that a randomly selected voter in this county opposes increased military spending is %.3f%%.'%(PA*100))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}