Bayesian Inference and Base Rates

Probability Overview

Probability: the probability of an event $A$ is the chance that $A$ occurs. We denote it as $P(A)$.

• $P(A)\in[0,1]=\in[0\%,100\%]$

Probability Space: a probability space is a set of all possible outcomes of an experiment. We denote it as $S$.

• Sample space $S$ = total area. So, $P(S)=1=100\%$
• We use Venn Diagram to depict probabilities graphically.

Independent Events: two events $A$ and $B$ are independent if they are unrelated. Independent events satisfy the following formula for their joint probability: $P(A\cap B)=P(A)\cdot P(B).$

Conditional Probability: the probability of an event $A$ given that another event $B$ has occurred. We denote it as $P(A\mid B)$.

• To calculate conditional probability, we have the following formula: $P(A\mid B)=\dfrac{P(A\cap B)}{P(B)}.$
• The notation $\mid$ is called a pipe. It is read as "given that" or "conditional on".

Joint Probability: the probability of two events $A$ and $B$ occurring together. We denote it as $P(A\cap B)$.

• To calculate joint probability, we have the following formula: $\text{Joint Probability}=P(A\cap B)=P(A\mid B)\cdot P(B).$

• Law of Total Probability: If we have an event $B$ that takes on $n$ values and an event $A$ conditional on it, then $P(a)=\sum_{n}P(A\mid B_n)\cdot P(B_n).$
  • This is a simple-weighted average.

Flipping Conditional Probability

• Bayes' Rule: a formula that allows us to flip conditional probabilities: $P(B\mid A)=\dfrac{P(A\mid B)\cdot P(B)}{P(A)}$

Bayes' Rule

• Bayes' Rule and Binary Events.
  • Binary events: only two potential outcomes
  • When we have binary events, it is useful to re-write $P(B)$ as $P(B)=P(B\mid A)P(A)+P(B\mid\neg A)P(\neg A),$ where $\neg A$ means "not $A$".
  • Then, Bayes' Rule becomes $P(A\mid B)=\dfrac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid\neg A)P(\neg A)}$

Medical Testing

• Positive Test Results
  • False Positive Rate of 5%: If you don't have the disease, the test correctly return a negative result 95% of the time.
    • $P(-\mid\text{don't have disease})=95\%$
    • $P(+\mid\text{don't have disease})=5\%\quad\text{false positive}$
    • In statistics: Type I error
  • False Negative Rate of 5%: If you do not have the disease, the test correctly returns a positive result 5% of the time.
    • $P(+\mid\text{have disease})=95\%$
    • $P(-\mid\text{have disease})=5\%\quad\text{false negative}$
    • In statistics: Type II error
• Application:
  • Given $P(+\mid\text{have})$ and $P(+\mid\text{don't})$. Find $P(\text{have disease}\mid+)$
  • By Bayes' Rule, we have $P(\text{have}\mid+)=\dfrac{P(+\mid\text{have})\times P(\text{have})}{P(+\mid\text{have})\times P(\text{have})+P(+\mid\text{don't})\times P(\text{don't})}$
  • So, we will need to know $P(\text{have})$ and $P(\text{don't})$.

Information, Beliefs, Priors, and Posteriors

• Bayes' Theorem in Use:
  • Prior: our beliefs, such as $P(\text{have})$ for having the disease
  • Evidence: the new information we have, such as $P(+\mid\text{have})$, positive test result
  • Posterio: update our beliefs based on the evidence, such as $P(\text{have}\mid+)$
• Balancing prior and evidence
  • The new information seems damming, but the prior pushes in the other direction.
  • But now that we know that we are in one of these extremely unlikely scenarios.
  • We have to ask about the relative likelihood of each one.
• Bayes' Rates:
  • Sometimes, we will use the term bayes' rates to refer to the priors.
  • In order to compute the quantity of interest, $P(\mid A)$, we need to know the base rate of $A$, $P(A)$.
  • Lots of analysts draw incorrect inferences because they fail to consider the base rate.

Bayes' Theorem and Quantitative Analysis

• Bayes' Theorem and Quantitative Analysis
  • Most quantitative analysts conduct hypothesis test and compute p-values.
    • We've already discussed some reasons why the p-value might lead us astray.
    • But any given study is not all that we know about something.
  • When an analyst finds a low p-value and concludes that their finding must be true, they've made the same mistake as the mathematician and the prosecutor in People v. Collins.
    • They calculated $P(\text{estimate}\mid\text{no effect})$
      • What's the chance that I have an estimate of this size given there was no effect
      • The probability of bad luck driving my result
    • However, we really want to compute $P(\text{no effect}\mid\text{estimate})$
      • What's the probability that there is no effect given my estimate.
• Bayes' and Estimated Effect
  • Suppose we've analyzed some data and obtained a statistically significant effect at the .05 level. (i.e., p<.05)
  • What's the probability that the estimated effect is genuine? - Bayes' Theorem! $\begin{aligned} &P(\text{effect}\mid\text{estimate})\\ &=\frac{P(\text{estimate}\mid\text{effect})\times {P(\text{effect})}}{P(\text{estimate}\mid\text{effect})\times P(\text{effect})+P(\text{estimate}\mid\text{no effect})\times [1-P(\text{effect})]} \end{aligned}$
    • We know ${\color{green}{P(\text{estimate}\mid\text{no effect})}}$
      • The significance level used in our hypothesis test
      • If we would declare a statistically significant estimate if p<.05, then ${\color{green}{P(\text{estimate}\mid\text{no effect})}}=.05$
    • We don't know the others, but they can be calculated:
      • ${\color{navy}{P(\text{estimate}\mid\text{effect})}}$ is the statistical power of the test
        • Assuming the effect is genuine, how likely that we would obtain a sigficant estimate?
        • There are ways of estimating the statistical power once we know some things about the data.
      • ${\color{red}{P(\text{effect})}}$ is the prior probability that there is a genuine effect
        • Our belief that there is an effect before seeing any of the statistical evidence.
  • New Bayes' Rule: $P(\text{true effect}\mid\text{stat. sig. estimate})=\dfrac{\text{power}\times\text{prior}}{\text{power}\times\text{prior}+\text{significance}\times(1-\text{prior})}$