A Bayesian update consists of updating the probability distribution that represents our knowledge of a Random variable after an event, using Bayes theorem, , where:

  • is the original probability we assigned to the random variable
  • is the updated probability
  • is the probability of the observed event IF we know the value of the Random variable .
  • is the probability of the event.

Since the resulting value is normalised by dividing by , we can often simplify calculations by working with the likelihood .

Conjugate priors

A family of priors is conjugate to a likelihood if the posterior lies in the same family, so a Bayesian update reduces to updating the hyperparameters. The table lists the standard cases for independent observations , with and . Gamma distributions use the shape-rate parameterisation, density .

LikelihoodUnknown parameterPriorPosterior
BernoulliBetaBeta
Binomial, knownBetaBeta
Geometric, number of failures before the first successBetaBeta
Negative binomial, known, failures before the -th successBetaBeta
Categorical or multinomial over classesDirichletDirichlet, the count of class
PoissonGammaGamma
Exponentialrate GammaGamma
Gamma, shape knownrate GammaGamma
Normal, knownNormalNormal with and
Normal, knownprecision GammaGamma
NormalNormal-Gamma: Gamma, Normal given Normal-Gamma
Multivariate normal, knownNormalNormal with and
UniformParetoPareto

Reading the posteriors: the prior hyperparameters act as pseudo-observations. For the Beta prior, and count prior successes and failures; for the normal mean, the posterior precision is the sum of prior and data precisions, and the posterior mean is the precision-weighted average of and .

All the cases above except the uniform one are exponential families. If , the prior is conjugate, and observing updates

Relations between distributions

Independence is assumed throughout, and Gamma is again shape-rate.

RelationStatement
Bernoulli to binomial Bernoulli Binomial
Binomial to PoissonBinomial Poisson in distribution as with
Binomial to normalBinomial Normal for large (de Moivre-Laplace, a case of the Central Limit Theorem)
Poisson to normalPoisson Normal for large
Poisson additivityPoisson + Poisson Poisson
Geometric to negative binomialGeometric = NegBin, and a sum of Geometric is NegBin
Exponential to GammaExponential = Gamma, and a sum of Exponential is Gamma (Erlang)
Gamma additivityGamma + Gamma Gamma
Minimum of exponentials Exponential Exponential
Poisson processInter-arrival times of a Poisson process of rate are Exponential; counts in a window of length are Poisson
Normal to chi-squared Normal gives , and = Gamma; in particular = Exponential
Student t for Normal, ; Normal as , and is Cauchy
F distribution for , ;
CauchyThe ratio of two independent Normal is Cauchy
Gamma to Beta Beta for Gamma, Gamma
Gamma to Dirichlet Dirichlet for Gamma
Beta and uniformBeta = Uniform, and the -th order statistic of Uniform is Beta
Uniform to exponential Exponential for Uniform
Inverse transform has cumulative distribution function for Uniform
Lognormal is lognormal if and only if is normal
Normal additivityNormal + Normal Normal

It might also be useful to treat this as the construction of a Probability Monoid. The exponential family update above makes this precise: the pairs form a commutative Monoid under addition, with identity , and Bayesian updating is an action of this monoid on the hyperparameters. Updating on two batches of data in sequence or at once gives the same posterior, in either order.