APPENDIX A

Basic Statistical Distributions

A.1 Discrete Distributions

A.1.1 Finite discrete distribution

Notation: X ∼ FDiscreten(x,p), x = (x1, . . . , xn)>, p = (p1, . . . , pn)> ∈Tn =̂ {(p1, . . . , pn): pi > 0,

∑ni=1 pi = 1}.

Density: Pr(X = xi) = pi, i = 1, . . . , n.

Moments: E(X) =∑n

i=1 xipi, Var(X) =∑n

i=1 x2i pi − (

∑ni=1 xipi)

2.

Note: The uniform discrete distribution is a special case of the finite

discrete distribution with pi = 1/n for all i.

A.1.2 Hypergeometric distribution

Notation: X ∼ Hgeometric(m,n, k), m,n, k are positive integers.

Density: Hgeometric(x|m,n, k) = (mx )( n

k − x )/(m + nk ),

where x = max(0, k − n), . . . ,min(m, k).

Moments: E(X) = km/N ′, Var(X) = kmn(N ′ − k)/[N ′2(N ′ − 1)],

where N ′ =̂m + n.

A.1.3 Poisson distribution

Notation: X ∼ Poisson(λ), λ > 0

Density: Poisson(x|λ) = λx e−λ/x!, x = 0, 1, 2, . . .

Moments: E(X) = λ, Var(X) = λ.

154 A. Basic Statistical Distributions

Properties: • If {Xi}ni=1

ind∼ Poisson(λi), then

∑ni=1 Xi ∼ Poisson(

∑ni=1 λi), and

(X1, . . . , Xn)|(∑ni=1 Xi = m) ∼ Multinomialn(m,p),

where p = (λ1, . . . , λn)>/∑n

i=1 λi;

• The Poisson and gamma distribution have relationship:

∑∞x=k Poisson(x|λ) =

∫ λ0 Gamma(y|k, 1) dy.

A.1.4 Binomial distribution

Notation: X ∼ Binomial(n, p), n is a positive integer, p ∈ (0, 1).

Density: Binomial(x|n, p) = (nx )px(1 − p)n−x, x = 0, 1, . . . , n.

Moments: E(X) = np, Var(X) = np(1 − p).

Properties: • If {Xi}di=1

ind∼ Binomial(ni, p), then

∑di=1 Xi ∼ Binomial(

∑di=1 ni, p);

• The binomial and beta distribution have relationship:

∑kx=0 Binomial(x|n, p) =

∫ 1−p0 Beta(x|n − k, k + 1) dx,

where 0 6 k 6 n.

Note: When n = 1, binomial distribution is called Bernoulli distribu-

tion.

A.1.5 Multinomial distribution

Notation: x = (X1, . . . , Xd)>∼ Multinomial(n; p1, . . . , pd) or

x = (X1, . . . , Xd)>∼ Multinomiald(n,p),

n is a positive integer, p = (p1, . . . , pd)> ∈ Td,

Density: Multinomiald(x|n,p) =( n

x1, . . . , xd

)

∏di=1 pxi

i ,

x = (x1, . . . , xd)>, xi > 0,

∑di=1 xi = n.

A.2 Continuous Distributions 155

Moments: E(Xi) = npi, Var(Xi) = npi(1 − pi), Cov(Xi, Xj) = −npipj .

Note: The binomial distribution is a special case of the multinomial

with d = 2.

A.2 Continuous Distributions

A.2.1 Uniform distribution

Notation: X ∼ U(a, b), a < b

Density: U(x|a, b) = 1/(b − a), x ∈ (a, b).

Moments: E(X) = (a + b)/2, Var(X) = (b − a)2/12.

Properties: If Y ∼ U(0, 1), then X = a + (b − a)Y ∼ U(a, b).

A.2.2 Beta distribution

Notation: X ∼ Beta(a, b), a > 0, b > 0.

Density: Beta(x|a, b) = xa−1(1 − x)b−1/B(a, b), 0 < x < 1.

Moments: E(X) = a/(a + b), E(X2) = a(a + 1)/[(a + b)(a + b + 1)],

Var(X) = ab/[(a + b)2(a + b + 1)].

Properties: If Y1 ∼ Gamma(a, 1), Y2 ∼ Gamma(b, 1), and Y1 ⊥⊥ Y2, then

Y1/(Y1 + Y2) ∼ Beta(a, b).

Note: When a = b = 1, Beta(1, 1) = U(0, 1).

A.2.3 Exponential distribution

Notation: X ∼ Exponential(β), rate parameter β > 0.

Density: Exponential(x|β) = β e−βx, x > 0.

Moments: E(X) = 1/β, Var(X) = 1/β2.

Properties: • If U ∼ U(0, 1), then − log Uβ ∼ Exponential(β);

• If {Xi}ni=1

iid∼ Exponential(β), then∑n

i=1 Xi ∼ Gamma(n, β).

156 A. Basic Statistical Distributions

A.2.4 Gamma distribution

Notation: X ∼ Gamma(α, β), shape parameter α > 0, rate parameter

β > 0.

Density: Gamma(x|α, β) = βα

Γ(α)xα−1 e−βx, x > 0.

Moments: E(X) = α/β, Var(X) = α/β2.

Properties: • If X ∼ Gamma(α, β) and c > 0, then cX ∼ Gamma(α, β/c);

• If {Xi}ni=1

ind∼ Gamma(αi, β), then∑

Xi ∼ Gamma(∑

αi, β);

• Γ(α + 1) = αΓ(α), Γ(1) = 1 and Γ(1/2) =√

π.

Note: Gamma(1, β) = Exponential(β). Gamma(ν/2, 1/2) = χ2(ν).

A.2.5 Chi-square distribution

Notation: X ∼ χ2(n) ≡ Gamma(n2 , 1

2), degree of freedom n > 0.

Density: χ2(x|n) = 2−n/2

Γ(n/2)xn/2−1 e−x/2, x > 0.

Moments: E(X) = n, Var(X) = 2n.

Properties: • If Y ∼ N(0, 1), then X = Y 2 ∼ χ2(1);

• If {Xj}mj=1

ind∼ χ2(nj), then∑m

j=1 Xj ∼ χ2(∑m

j=1 nj).

A.2.6 t- or Student’s t-distribution

Notation: X ∼ t(n), n is a positive integer.

Density: t(x|n) =Γ(n+1

2 )√πnΓ(n

2 )

[

1 +x2

n

]−n+1

2

, −∞ < x 6 ∞.

Moments: E(X) = 0 (if n > 1), Var(X) = nn−2 (if n > 2).

Properties: Let Z ∼ N(0, 1), Y ∼ χ2(n), and Z ⊥⊥ Y , then

Z√

Y/n∼ t(n).

A.2 Continuous Distributions 157

Note: When n = 1, t(n) = t(1) is called standard Cauchy distribution,

whose mean and variance do not exist.

A.2.7 F or Fisher’s F-distribution

Notation: X ∼ F (n1, n2), n1, n2 are positive integers.

Density: F (x|n1, n2) = (n1/n2)n1/2

B(n12

,n22

)x

n12−1(1 + n1x

n2)−

n1+n22 , x > 0.

Moments: E(X) = n2

n2−2 (if n2 > 2), Var(X) =2n2

2(n1+n2−2)

n1(n2−4)(n2−2)2(if n2 > 4).

Properties: Let Yi ∼ χ2(ni), i = 1, 2, and Y1 ⊥⊥ Y2, then

Y1/n1

Y2/n2∼ F (n1, n2).

A.2.8 Normal or Gaussian distribution

Notation: X ∼ N(µ, σ2), −∞ < µ < ∞, σ2 > 0.

Density: N(x|µ, σ2) = 1√2πσ

exp[− (x−µ)2

2σ2 ], −∞ < x < ∞.

Moments: E(X) = µ, Var(X) = σ2.

Properties: • If {Xi} ind∼ N(µi, σ2i ), then

∑

aiXi ∼ N(∑

aiµi,∑

a2i σ

2i );

• If X1|X2 ∼ N(X2, σ21) and X2 ∼ N(µ2, σ

22), then

X1 ∼ N(µ2, σ21 + σ2

2).

A.2.9 Multivariate Normal or Gaussian distribution

Notation: x = (X1, . . . , Xd)>∼ Nd(µ,Σ) or N(µ,Σ), µ ∈ R

d, Σ > 0.

Density: Nd(x|µ,Σ) = 1

(√

2π )d|Σ|12

exp{−12 (x−µ)>Σ−1(x−µ)}, x ∈ R

d.

Moments: E(x) = µ, Var(x) = Σ.