Foundations of Statistical Mechanics:

Random Walks and Distribution Functions

Dr M Shawa

Contents
1 Introduction to Random Walks and Stochastic Processes 2

2 The Binomial Distribution and its Properties 2

2.1 Moments of the Binomial Distribution . . . . . . . . . . . . . . . . . . . 2
2.1.1 Mean (Expected Value) . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.2 Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3 The Gaussian (Normal) Distribution 5

3.1 Derivation of the Gaussian Approximation . . . . . . . . . . . . . . . . . 5
3.2 Proof of the Central Limit Theorem (Sketch) . . . . . . . . . . . . . . . . 6

4 Connection to Statistical Mechanics: Macrostates and Entropy 7

5 Applications and Further Extensions 9

5.1 Derivation of the Poisson Distribution . . . . . . . . . . . . . . . . . . . . 9
5.2 Derivation of the Maxwell-Boltzmann Velocity Distribution . . . . . . . . 9

6 Physical Problems 10
6.1 Problem 1: The One-Dimensional Lattice Gas . . . . . . . . . . . . . . . 10
6.2 Problem 2: The Schottky Anomaly . . . . . . . . . . . . . . . . . . . . . 11

7 Summary of Key Concepts 12

1
Notes for a Theoretical Physicist Statistical Mechanics

1 Introduction to Random Walks and Stochastic Pro-

cesses
The study of statistical mechanics is fundamentally about understanding the collective
behavior of a vast number of microscopic constituents. This macroscopic behavior often
emerges from simple, random microscopic interactions. The random walk model, as
discussed in works like Bowley & Sánchez and Glazer & Wark, serves as a quintessential
example of this emergence. Further details on the one-dimensional random walk can be
found in resources such as the Wolfram MathWorld page on the topic.
A random walk is a mathematical formalization of a path that consists of a succession
of random steps. Its simplicity belies its power in describing a wide range of physical
phenomena, from the diffusion of a drop of ink in water to the Brownian motion of a
pollen grain, which was so crucial in confirming the atomic nature of matter.
Consider a simple one-dimensional random walk on a discrete lattice. A particle starts
at the origin (x = 0) and, at each time step, takes a step of length d either to the right or
to the left. Let the probability of moving to the right be p and the probability of moving
to the left be q = 1 − p. The steps are independent and identically distributed.
After N steps, let nR be the number of steps taken to the right and nL be the number
of steps taken to the left. The total number of steps is N = nR + nL . The final position
of the particle, XN , is given by:

XN = (nR − nL )d = (2nR − N )d. (1)

The probability of a specific sequence of nR right steps and nL left steps is pnR q nL . The
total number of such unique sequences that result in the same final position is given by
the binomial coefficient, which leads us to the binomial distribution.

2 The Binomial Distribution and its Properties

The binomial distribution governs the probability of obtaining a certain number of ”suc-
cesses” (e.g., a step to the right) in a sequence of N independent Bernoulli trials.
The probability, PN (nR ), that the particle takes exactly nR steps to the right out of
a total of N steps is given by:
 
N nR N!
PN (nR ) = p (1 − p)N −nR = pnR q N −nR . (2)
nR nR !(N − nR )!

This distribution is normalized, meaning PN (nR ) = 1. The sum can be confirmed

PN
nR =0
by the binomial theorem:
N  
X N nR N −nR
p q = (p + q)N = (p + (1 − p))N = 1N = 1. (3)
n =0
n R
R

2.1 Moments of the Binomial Distribution

The moments of the distribution, particularly the mean and variance, provide important
information about the expected behavior of the system.

2
Notes for a Theoretical Physicist Statistical Mechanics

2.1.1 Mean (Expected Value)

The mean number of right steps, hnR i, is calculated as:
N
X
hnR i = nR PN (nR ). (4)
nR =0

We will use a powerful technique involving differentiation. Consider the generating func-
tion for the binomial distribution, which is simply the binomial expansion:
N
X N!
G(p, q) = (p + q) = N
pnR q N −nR . (5)
nR
n !(N − nR )!
=0 R

The expected value of nR can be obtained by differentiating G(p, q) with respect to p and
then multiplying by p.
N
! N
∂G ∂ X N! X N! ∂
= nR N −nR
p q = . (6)
∂p ∂p n =0 nR !(N − nR )! n =0
n R !(N − n R )! ∂p
R R

Using the product rule, the derivative of the term inside the sum is:
∂
= nR pnR −1 (1 − p)N −nR − pnR (N − nR )(1 − p)N −nR −1 . (7)
∂p
Thisapproach can be  complicated. A simpler method is to use the property that hnR i =
nR =0 PN (nR ) .
∂
PN
p ∂p

N
!
∂ X N! ∂ N
hnR i = p pnR q N −nR =p . (8)
∂p nR
n
=0 R
!(N − nR )! ∂p

Using the chain rule, the derivative of the function itself is:

∂ N ∂
= N (p + q)N −1 = N (p + q)N −1 · 1. (9)
∂p ∂p
Since p + q = 1, we get:

hnR i = p N (p + q)N −1 = pN (1)N −1 = N p. (10)

 

This confirms that the expected number of right steps is simply the total number of steps
multiplied by the probability of a right step. For a symmetric random walk (p = 1/2),
the mean is N/2.

2.1.2 Variance
The variance, σ 2 , is defined as σ 2 = h(nR − hnR i)2 i = hn2R i − hnR i2 . We already have
hnR i = N p, so we need to find the second moment, hn2R i. We can obtain this using the
2
relationship hnR (nR − 1)i = p2 ∂∂pG2 .

∂ 2G ∂  ∂
N −1
= N (N − 1)(p + q)N −2 = N (N − 1)(p + q)N −2 . (11)

2
= N (p + q)
∂p ∂p ∂p

3
Notes for a Theoretical Physicist Statistical Mechanics

Again, since p + q = 1, this becomes N (N − 1)(1)N −2 = N (N − 1). Therefore, the second

moment is:
∂ 2G
hnR (nR − 1)i = p2 2 = p2 N (N − 1). (12)
∂p
We can express hn2R i in terms of this result:

n2R = hnR (nR − 1) + nR i = hnR (nR − 1)i + hnR i = N (N − 1)p2 + N p. (13)

Finally, we can calculate the variance:

σ 2 = n2R − hnR i2 = [N (N − 1)p2 + N p] − (N p)2 . (14)

σ 2 = (N 2 p2 − N p2 + N p) − N 2 p2 = N p − N p2 = N p(1 − p) = N pq. (15)

√
The standard deviation, σ = N pq, is a key measure of the expected √ magnitude of
fluctuations around the mean. For the symmetric random √ walk, σ = N /2. This implies
that the typical displacement from the origin grows as N .

4
Notes for a Theoretical Physicist Statistical Mechanics

3 The Gaussian (Normal) Distribution

While the binomial distribution is exact, it becomes computationally unwieldy for large
N . In statistical mechanics, we are often concerned with systems where N is of the order
of Avogadro’s number (NA ≈ 6.022 × 1023 ), making approximations essential. For large
N , the binomial distribution converges to the Gaussian distribution, a profound result of
the Central Limit Theorem.

3.1 Derivation of the Gaussian Approximation

Let’s approximate the binomial distribution for a large number of steps (N  1) and
assume that the number of right steps (nR ) is close to the mean (N p). We will work
with the natural logarithm of the probability PN (nR ), which is more convenient for large
numbers due to Stirling’s approximation.
 
N!
ln PN (nR ) = ln + nR ln p + (N − nR ) ln q. (16)
nR !(N − nR )!
Using the Stirling’s approximation, ln(x!) ≈ x ln x − x, we get:
ln PN (nR ) ≈ (N ln N − N ) − (nR ln nR − nR ) − ((N − nR ) ln(N − nR ) − (N − nR ))
+ nR ln p + (N − nR ) ln q
= N ln N − nR ln nR − (N − nR ) ln(N − nR ) + nR ln p + (N − nR ) ln q.
To proceed, we define the deviation from the mean as δ = nR − N p, where δ is small
compared to N . Thus, nR = N p + δ and N − nR = N q − δ. Let’s analyze the terms
containing ln nR and ln(N − nR ) using a Taylor series expansion around the mean value.
    
δ δ
ln nR = ln(N p + δ) = ln N p 1 + = ln(N p) + ln 1 + .
Np Np
    
δ δ
ln(N − nR ) = ln(N q − δ) = ln N q 1 − = ln(N q) + ln 1 − .
Nq Nq
2
For small , the Taylor series expansion of ln(1 + ) is ln(1 + ) = − 2 +O(3 ). Applying
this to our terms:
"  2 #
δ 1 δ
nR ln nR = (N p + δ) ln(N p) + − + O(δ 3 )
Np 2 Np
δ(N p) δ2
= N p ln(N p) + δ ln(N p) + − + O(δ 3 )
Np 2N p
δ2
= N p ln(N p) + δ ln(N p) + δ − + O(δ 3 ).
2N p
"  2 #
δ 1 δ
(N − nR ) ln(N − nR ) = (N q − δ) ln(N q) − − + O(δ 3 )
Nq 2 Nq
δ(N q) δ2
= N q ln(N q) − δ ln(N q) − − + O(δ 3 )
Nq 2N q
δ2
= N q ln(N q) − δ ln(N q) − δ − + O(δ 3 ).
2N q

5
Notes for a Theoretical Physicist Statistical Mechanics

Now, we substitute these back into the full expression for ln PN (nR ).
δ2
 
ln PN (nR ) ≈ N ln N − N p ln(N p) + δ ln(N p) + δ −
2N p
2
 
δ
− N q ln(N q) − δ ln(N q) − δ − + (N p + δ) ln p + (N q − δ) ln q
2N q
= (N ln N − N p ln(N p) − N q ln(N q))
+ δ(− ln(N p) + ln(N q) + ln p − ln q)
δ2 δ2
+ + + O(δ 3 ).
2N p 2N q
Let’s analyze each group of terms. The constant terms (independent of δ) are:
N ln N − N p ln(N p) − N q ln(N q) = N ln N − N p(ln N + ln p) − N q(ln N + ln q)
= (N − N p − N q) ln N − N p ln p − N q ln q
= (N − N (p + q)) ln N − N p ln p − N q ln q.
Since p + q = 1, this simplifies to 0 · ln N − N p ln p − N q ln q, which is a constant.
The terms linear in δ are:
δ(− ln(N p) + ln(N q) + ln p − ln q) = δ(− ln N − ln p + ln N + ln q + ln p − ln q)
= δ[0] = 0.
All the linear terms in δ have cancelled out, which is a key feature of an expansion around
a maximum. We are left with only the quadratic term:
δ2 δ2 q + p
   
1 1
ln PN (nR ) ≈ const − + = const − . (17)
2 Np Nq 2 N pq
Since p + q = 1, this simplifies to:
δ2
ln PN (nR ) ≈ const − . (18)
2N pq
Exponentiating both sides and including the normalization factor (which can be derived
from the integral of the Gaussian function), we obtain the final expression:
δ2 (nR − hnR i)2
   
1 1
PN (δ) = √ exp − =√ exp − . (19)
2πN pq 2N pq 2πσ 2 2σ 2
This is the Gaussian distribution with a mean of hnR i = N p and a variance of σ 2 = N pq.

3.2 Proof of the Central Limit Theorem (Sketch)

A more general and formal proof of the Central Limit Theorem (CLT) can be done using
the method of characteristic functions. The characteristic function of a random variable
X is defined as φX (k) = eikX . The key property is that the characteristic function of
a sum of independent random variables is the product of their individual characteristic
functions.
N
Y
φS (k) = e ikSN
= e ik(X1 +···+XN )
= eikXi = [φX (k)]N . (20)
i=1

6
Notes for a Theoretical Physicist Statistical Mechanics

We consider a standardized variable YN = SσN −N

√ µ , where µ = hXi i and σ 2 = h(Xi − µ)2 i.
X
X N
The characteristic function of YN is:
  
SN − N µ
φYN (k) = e ikYN
= exp ik √ (21)
σX N
* N
!+ N   
ik X Y ik(Xj − µ)
= exp √ (Xj − µ) = exp √ . (22)
σX N j=1 j=1
σX N

Let’s consider the Taylor expansion of the characteristic function of the standardized
individual variable:
* 2 +
√ E
   
D ik(X j − µ) 1 ik(X j − µ) 1
eik(Xj −µ)/σX N
=1+ √ + √ +O . (23)
σX N 2! σX N N 3/2

The first term is 1. The second term, hXj − µi, is zero by definition. The third term is
(ik)2 2 k2
2σ 2 N
h(Xj − µ)2 i = 2σ−k2 N σX
2
= − 2N . So, for large N :
X X

D √ E k2
ik(Xj −µ)/σX
e N
≈1− . (24)
2N
Then, the characteristic function of YN becomes:
N
k2

φYN (k) = 1 − . (25)
2N

In the limit as N → ∞, we use the identity limN →∞ (1 + x/N )N = ex .

 2
k
lim φYN (k) = exp − . (26)
N →∞ 2

This is the characteristic function of a standard normal distribution, and by the unique-
ness theorem for characteristic functions, this proves that YN approaches a standard
normal distribution.

4 Connection to Statistical Mechanics: Macrostates

and Entropy
The relationship between the binomial and Gaussian distributions is of immense impor-
tance in statistical mechanics. As discussed by Bowley and Sánchez (Chapter 3), the
macroscopic properties of a system are related to its statistical weight, Ω, which repre-
sents the number of accessible microstates corresponding to a given macrostate. For a
random walk, the macrostate is the final position, and the number of ways to achieve it
is the binomial coefficient.
The entropy, as defined by Boltzmann, is S = kB ln Ω. For a system with a large
number of particles or degrees of freedom, the statistical weight Ω is an extremely rapidly
increasing function of the most probable state. The Gaussian distribution shows that for
a large system, the distribution of microstates is overwhelmingly peaked around the most
√
probable configuration. The width of this peak, given by the standard deviation σ ∼ N ,

7
Notes for a Theoretical Physicist Statistical Mechanics

√
is a measure of the fluctuations around the mean. The relative width, σ/N ∼ 1/ N ,
vanishes for large N .
This means that for a macroscopic system, fluctuations away from the equilibrium
state are incredibly rare. The system effectively stays in its most probable configuration,
which corresponds to the state of maximum entropy. This provides the link between the
microscopic probabilistic description of the world and the deterministic laws of classical
thermodynamics.

8
Notes for a Theoretical Physicist Statistical Mechanics

5 Applications and Further Extensions

The concepts of random walks and Gaussian distributions permeate statistical mechanics.

5.1 Derivation of the Poisson Distribution

The Poisson distribution is a limiting case of the binomial distribution, typically arising
for events that are rare but have a very large number of trials. We consider the limit as
N → ∞ and p → 0 such that the mean value λ = N p remains finite. Let’s start with the
binomial probability:
N!
PN (nR ) = pnR (1 − p)N −nR . (27)
nR !(N − nR )!
First, let’s rewrite the factorials. For large N , we can use the approximation N!
(N −nR )!
≈
N nR .
N nR nR (N p)nR
PN (nR ) ≈ p (1 − p)N −nR = (1 − p)N −nR . (28)
nR ! nR !
Now, substitute N p = λ and p = λ/N :
N −nR
λnR

λ
PN (nR ) ≈ 1− . (29)
nR ! N
As N → ∞, the last term can be split:
 N −nR  N  −nR
λ λ λ
1− = 1− 1− . (30)
N N N
The first part, using the limit definition of the exponential function, approaches e−λ :
 N
λ
lim 1 − = e−λ . (31)
N →∞ N
For the second part, since nR is a finite integer (the number of ”successes”), as N →

−nR
∞, the term 1 − Nλ → (1)−nR = 1. Combining these results, we get the Poisson
distribution, where we replace nR with the more common notation k for the number of
events:
λk e−λ
P (k) = . (32)
k!

5.2 Derivation of the Maxwell-Boltzmann Velocity Distribution

The Maxwell-Boltzmann distribution describes the velocities of particles in a system in
thermal equilibrium. We can derive it from the canonical ensemble. The probability of
finding a particle in a state with energy  is given by the Boltzmann factor:
P () ∝ e−β , (33)
where β = 1/kB T . For a classical particle, the energy is purely kinetic,  = 12 m(vx2 +
vy2 + vz2 ). The probability density function f (~v ) for finding a particle with velocity ~v is
therefore proportional to the Boltzmann factor of its kinetic energy:
!
1
m(vx2 + vy2 + vz2 ) mvx2 mvy2 mvz2
     
2
f (~v ) = C exp − = C exp − exp − exp − .
kB T 2kB T 2kB T 2kB T
(34)

9
Notes for a Theoretical Physicist Statistical Mechanics

Here, C is a normalization constant. The distribution is separable into three independent

distributions for each velocity component. The probability density for a single velocity
component, say vx , is:
mvx2
 
0
f (vx ) = C exp − . (35)
2kB T
To find the normalization constant C 0 , we integrate over all possible velocities:
Z ∞
f (vx )dvx = 1. (36)
−∞
R∞
Using the Gaussian integral π/a, we have:
2
p
−∞
e−ax dx =
Z ∞ r
2
 
mv 2πkB T
C0 exp − x
dvx = C 0 = 1. (37)
−∞ 2kB T m
 1/2
So, C =
0 m
. Since the components are independent, the full distribution is the
2πkB T
product of the three normalized distributions:
3/2
mv 2
  
m
f (~v ) = exp − . (38)
2πkB T 2kB T
This is the Maxwell-Boltzmann velocity distribution.

6 Physical Problems
The following problems are designed to test your understanding of the concepts presented
in the document.

6.1 Problem 1: The One-Dimensional Lattice Gas

Consider a one-dimensional lattice of N sites. Each site can either be empty or occupied
by a single particle. There are a total of n particles on the lattice, where n < N . The
particles are indistinguishable.

1. Find the statistical weight (multiplicity) Ω(n, N ) for a given macrostate defined by
the number of occupied sites n.

2. Calculate the entropy S(n, N ) of the system.

3. Using Stirling’s approximation, find an expression for the entropy in the thermo-
dynamic limit (N, n → ∞ with n/N = x held constant).

Solution
1. The problem is equivalent to choosing n sites out of N to be occupied. Since the
particles are indistinguishable, the order does not matter. The number of ways to
do this is given by the binomial coefficient:
 
N N!
Ω(n, N ) = = . (39)
n n!(N − n)!

10
Notes for a Theoretical Physicist Statistical Mechanics

2. The entropy is related to the statistical weight by Boltzmann’s formula, S = kB ln Ω:

 
N!
S(n, N ) = kB ln . (40)
n!(N − n)!

3. For large N and n, we apply Stirling’s approximation ln x! ≈ x ln x − x:

S
= ln N ! − ln n! − ln(N − n)!
kB
≈ (N ln N − N ) − (n ln n − n) − ((N − n) ln(N − n) − (N − n))
= N ln N − n ln n − (N − n) ln(N − n).

Now, let x = n/N so n = xN . Then N − n = N (1 − x).

S
≈ N ln N − xN ln(xN ) − (N − xN ) ln(N − xN )
kB
= N ln N − xN (ln x + ln N ) − N (1 − x)(ln N + ln(1 − x))
= N ln N − xN ln x − xN ln N − N ln N + xN ln N − N (1 − x) ln(1 − x)
= −xN ln x − N (1 − x) ln(1 − x).

Finally, the entropy per site, s = S/N , is:

s
= −x ln x − (1 − x) ln(1 − x). (41)
kB
This is the well-known entropy of mixing for an ideal solution.

6.2 Problem 2: The Schottky Anomaly

Consider a system of N independent, distinguishable two-level systems. Each system has
two non-degenerate energy states, with energies 0 and .

1. Find the partition function Z for a single system.

2. Find the total partition function for all N systems.

3. Calculate the average internal energy hEi and the heat capacity CV of the system
as a function of temperature.

Solution
1. The partition function for a single two-level system is the sum over all possible
states, weighted by the Boltzmann factor:

Z1 = e−β(0) + e−β = 1 + e−β . (42)

2. Since the systems are independent and distinguishable, the total partition function
is the product of the individual partition functions:

Z = (Z1 )N = (1 + e−β )N . (43)

11
Notes for a Theoretical Physicist Statistical Mechanics

3. The average internal energy is given by hEi = − ∂β∂

ln Z.
ln Z = N ln 1 + e−β .

∂ 1 ∂
hEi = − [N ln 1 + e−β ] = −N


∂β 1 + e−β ∂β
1 −β e−β N
= −N (−e ) = N = .
1 + e−β 1 + e−β eβ + 1
The heat capacity at constant volume is CV = ∂hEi
∂T
= ∂hEi ∂β
∂β ∂T
. Since β = 1/kB T ,
we have ∂T
∂β
= −1/kB T 2 = −kB β 2 .
∂ hEi ∂ −1
= N = N (−1)(eβ + 1)−2 (eβ )
∂β ∂β
N 2 eβ N 2 e−β
= − β = − .
(e + 1)2 (1 + e−β )2
Therefore, the heat capacity is:
2
N 2 eβ 2 eβ e/kB T
  
2 2 
CV = − β (−kB β ) = N k B β = N k B .
(e + 1)2 (eβ + 1)2 kB T (e/kB T + 1)2
(44)
This is the famous Schottky anomaly, which shows a peak in the heat capacity at
a temperature T ≈ /kB .

7 Summary of Key Concepts

• Random Walk: A sequence of random steps that models stochastic phenomena
like diffusion.
• Binomial Distribution: The exact probability distribution for a finite number
of independent trials with two outcomes. For a random walk, it describes the
probability of being at a specific final position after a given number of steps.
• Gaussian Distribution: The limiting form of the binomial distribution for a large
number of trials. It is a continuous, bell-shaped curve that describes the distribution
of fluctuations around the mean.
• Central Limit Theorem: The fundamental reason why the Gaussian distribution
is so prevalent. It states that the sum of many independent random variables will
be approximately Gaussian.
• Connection to Thermodynamics: For macroscopic systems, the most probable
configuration (the one with the largest statistical weight) corresponds to the equi-
librium state of maximum entropy. Fluctuations around this state are described by
the Gaussian distribution, and their relative magnitude vanishes for large systems.
• Maxwell-Boltzmann Distribution: Derived from the canonical ensemble, this
distribution describes the velocities of particles in an ideal gas, showing that particle
velocities are distributed as a Gaussian.
• Poisson Distribution: A limiting case of the binomial distribution for rare events
in a large number of trials.

12