A Bayesian update consists of updating the probability distribution that represents our knowledge of a Random variable after an event, using Bayes theorem, , where:
- is the original probability we assigned to the random variable
- is the updated probability
- is the probability of the observed event IF we know the value of the Random variable .
- is the probability of the event.
Since the resulting value is normalised by dividing by , we can often simplify calculations by working with the likelihood .
Conjugate priors
A family of priors is conjugate to a likelihood if the posterior lies in the same family, so a Bayesian update reduces to updating the hyperparameters. The table lists the standard cases for independent observations , with and . Gamma distributions use the shape-rate parameterisation, density .
| Likelihood | Unknown parameter | Prior | Posterior |
|---|---|---|---|
| Bernoulli | Beta | Beta | |
| Binomial, known | Beta | Beta | |
| Geometric, number of failures before the first success | Beta | Beta | |
| Negative binomial, known, failures before the -th success | Beta | Beta | |
| Categorical or multinomial over classes | Dirichlet | Dirichlet, the count of class | |
| Poisson | Gamma | Gamma | |
| Exponential | rate | Gamma | Gamma |
| Gamma, shape known | rate | Gamma | Gamma |
| Normal, known | Normal | Normal with and | |
| Normal, known | precision | Gamma | Gamma |
| Normal | Normal-Gamma: Gamma, Normal given | Normal-Gamma | |
| Multivariate normal, known | Normal | Normal with and | |
| Uniform | Pareto | Pareto |
Reading the posteriors: the prior hyperparameters act as pseudo-observations. For the Beta prior, and count prior successes and failures; for the normal mean, the posterior precision is the sum of prior and data precisions, and the posterior mean is the precision-weighted average of and .
All the cases above except the uniform one are exponential families. If , the prior is conjugate, and observing updates
Relations between distributions
Independence is assumed throughout, and Gamma is again shape-rate.
| Relation | Statement |
|---|---|
| Bernoulli to binomial | Bernoulli Binomial |
| Binomial to Poisson | Binomial Poisson in distribution as with |
| Binomial to normal | Binomial Normal for large (de Moivre-Laplace, a case of the Central Limit Theorem) |
| Poisson to normal | Poisson Normal for large |
| Poisson additivity | Poisson + Poisson Poisson |
| Geometric to negative binomial | Geometric = NegBin, and a sum of Geometric is NegBin |
| Exponential to Gamma | Exponential = Gamma, and a sum of Exponential is Gamma (Erlang) |
| Gamma additivity | Gamma + Gamma Gamma |
| Minimum of exponentials | Exponential Exponential |
| Poisson process | Inter-arrival times of a Poisson process of rate are Exponential; counts in a window of length are Poisson |
| Normal to chi-squared | Normal gives , and = Gamma; in particular = Exponential |
| Student t | for Normal, ; Normal as , and is Cauchy |
| F distribution | for , ; |
| Cauchy | The ratio of two independent Normal is Cauchy |
| Gamma to Beta | Beta for Gamma, Gamma |
| Gamma to Dirichlet | Dirichlet for Gamma |
| Beta and uniform | Beta = Uniform, and the -th order statistic of Uniform is Beta |
| Uniform to exponential | Exponential for Uniform |
| Inverse transform | has cumulative distribution function for Uniform |
| Lognormal | is lognormal if and only if is normal |
| Normal additivity | Normal + Normal Normal |
It might also be useful to treat this as the construction of a Probability Monoid. The exponential family update above makes this precise: the pairs form a commutative Monoid under addition, with identity , and Bayesian updating is an action of this monoid on the hyperparameters. Updating on two batches of data in sequence or at once gives the same posterior, in either order.