Adaptive Filters – Algorithms (Part 1)
Gerhard Schmidt
Christian-Albrechts-Universität zu Kiel
Faculty of Engineering
Electrical Engineering and Information Technology
Digital Signal Processing and System Theory
Contents of the Lecture
Today:
Exercises:
Topics for the Talks
Adaptive Algorithms:
Introductory Remarks
Recursive Least Squares (RLS) Algorithm
Least Mean Squares Algorithm (LMS Algorithm) – Part 1
Least Mean Squares Algorithm (LMS Algorithm) – Part 2
Affine projection Algorithm (AP Algorithm)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 2
Adaptive Filters – Talks
Possible Topics
Suggestions:
Hearing aids
GSM (source) coding
Localization and tracking
Active noise control (anti-noise)
Noise suppression
Bandwidth extension
Audio upmix of stereo signals
Adaptive beamforming
MPEG audio coding
Non-linear echo cancellation
Adaptation of neural networks
Feedback suppression
…
Your own topic suggestions are
welcome …
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 3
Adaptive Filters – Algorithms
Contents
Exercises:
Topics for the Talks
Adaptive Algorithms:
Introductory Remarks
Recursive Least Squares (RLS) Algorithm
Least Mean Squares Algorithm (LMS Algorithm) – Part 1
Least Mean Squares Algorithm (LMS Algorithm) – Part 2
Affine projection Algorithm (AP Algorithm)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 4
Introductory Remarks
Motivation
Why adaptive filters?
Signal properties are not known in advance or are time variant.
System properties are not known in advance or time variant.
Examples:
Speech signals
Mobile telephone channels
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 5
Introductory Remarks
Literature
Books:
E. Hänsler, G. Schmidt: Acoustic Echo and Noise Control, Wiley, 2004
S. Haykin: Adaptive Filter Theory, Prentice Hall, 2002
A. Sayed: Fundamentals of Adaptive Filtering, Wiley, 2004
E. Hänsler: Statistische Signale: Grundlagen und Anwendungen, Springer, 2001
(in German)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 6
Introductory Remarks
Two Hook-Ups of Adaptive Filters
Adaptive filter for channel equalization:
Transmission Adaptive
channel filter
Adaptive filter for system identification:
System
Adaptive
filter
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 7
Introductory Remarks
Application Examples – Part 1
Adaptive filter for cancellation of hybrid echoes:
Adaptive
Hybrid
filter
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 8
Introductory Remarks
Application Examples – Part 2
Adaptive filter for noise reduction with reference signal:
Noisy
Signal signal
source
Transmission
path 1
Noise Adaptive
source filter
Transmission
path 2
Reference
signal
Signal model
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 9
Introductory Remarks
Application Examples – Part 3
Antenna array:
Adaptive Adaptive Adaptive
filter 1 filter 2 filter N
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 10
Introductory Remarks
Application Examples – Part 4
Adaptive equalization without reference signal
Adaptive
filter
Decision
circuit
Assumptions:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 11
Introductory Remarks
Generic Setup
Desired
output
signal
Adaptive filter
Adaptive
algorithm
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 12
Introductory Remarks
Structure of an Adaptive FIR Filter
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 13
Introductory Remarks
Error Measures – Part 1
No local noise
Mean square (signal) error:
System distance:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 14
Introductory Remarks
Mean Square Error and System Distance
Relation of the normalized mean square (signal) error power and the system distance:
Let be white noise:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 15
Introductory Remarks
Adaptation
Local noise
Basic principle:
New = old + correction +
+
Properties:
„Correction“ depends on the input signal and the error signal .
Procedures differ by the functions and :
Step size
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 16
Introductory Remarks
Error Measures
Local noise
+
+
Three error measures control the adaptation:
Coefficient error
A priori error
A posteriori error:
Old data New filter
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 17
Adaptive Filters - Algorithms
Contents
Exercises:
Topics for the Talks
Adaptive Algorithms:
Introductory Remarks
Recursive Least Squares (RLS) Algorithm
Least Mean Squares Algorithm (LMS Algorithm) – Part 1
Least Mean Squares Algorithm (LMS Algorithm) – Part 2
Affine projection Algorithm (AP Algorithm)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 18
Recursive Least Squares (RLS) Algorithm
Algorithmic Properties
Attributes of the RLS algorithm:
No a priori knowledge of signal statistics is required.
Optimization criterion is the (weighted) sum of squared errors.
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 19
Recursive Least Squares (RLS) Algorithm
Error Criterion
Signal Filter
Filter
filter
Forgetting factor
Alternative:
Adaptation
adaptive
algorithm
algorithm
error:
Filter at time n
Signal at time l
error inserted:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 20
Recursive Least Squares (RLS) Algorithm
Derivation – Part 1
Cost function:
Differentiate with respect to the complex filter coefficients and setting the result to zero:
Definitions:
… Estimate for the auto correlation matrix
… Estimate for the cross correlation vector
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 21
Recursive Least Squares (RLS) Algorithm
Derivation – Part 2
From Simon Haykin, „Adaptive Filter Theory“, Prentice Hall, 2002:
or: The Matrix Cookbook [ http://matrixcookbook.com ]
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 22
Recursive Least Squares (RLS) Algorithm
Derivation – Part 3
Filter
filter
Adaptation
adaptive
algorithm
algorithm
Inserting the results leads to:
„Wiener solution“
… assuming that the auto correlation matrix is invertable
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 23
Recursive Least Squares (RLS) Algorithm
Recursion – Part 1
Recursion of the auto correlation matrix over time:
Recursion of the cross correlation vector over time:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 24
Recursive Least Squares (RLS) Algorithm
Recursion – Part 2
Recursion for the auto correlation matrix:
Matrix Inversion Lemma:
Inserting the Lemma in the recursion:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 25
Recursive Least Squares (RLS) Algorithm
Recursion – Part 3
Recursion for the auto correlation matrix:
Definition of a gain vector:
Inserting this definition leads to:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 26
Recursive Least Squares (RLS) Algorithm
Recursion – Part 4
Definition of a gain factor:
Multiplication by the denominator on the right hand side leads to:
Rewriting leads to:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 27
Recursive Least Squares (RLS) Algorithm
Recursion – Part 5
Recursion of the filter coefficient vector:
Step from n to n+1:
Reducing the right hand side:
Inserting the recursion of the cross correlation vector leads to:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 28
Recursive Least Squares (RLS) Algorithm
Recursion – Part 6
What we have so far:
If we insert the recursive computation of the inverse auto correlation matrix
we obtain:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 29
Recursive Least Squares (RLS) Algorithm
Recursion – Part 7
What we have so far:
Inserting according to
results in
Gain factor Error: old filter with new data
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 30
Recursive Least Squares (RLS) Algorithm
Adaptation Rule – Part 1
Inserting previous results:
Adaptation rule for the filter coefficients according to the RLS algorithm:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 31
Recursive Least Squares (RLS) Algorithm
Summary
Computing a preliminary gain vector (complexity prop. N²):
Update of the inverse auto correlation matrix (complexity prop. N²):
Computing the error signal (complexity prop. N):
Update of the filter vector (complexity prop. N):
Step size (0 … 1), will be treated later …
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 32
Adaptive Filters – Algorithms
Contents
Exercises:
Topics for the Talks
Adaptive Algorithms:
Introductory Remarks
Recursive Least Squares (RLS) Algorithm
Least Mean Squares Algorithm (LMS Algorithm) – Part 1
Least Mean Squares Algorithm (LMS Algorithm) – Part 2
Affine projection Algorithm (AP Algorithm)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 33
Least Mean Square (LMS) Algorithm
Basics – Part 1
Optimization criterion:
Minimizing the mean square error
Assumptions:
Real, stationary random processes
Structure:
Unknown system
Adaptive
filter
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 34
Least Mean Square (LMS) Algorithm
Basics – Part 2
Unknown system
Output signal of the adaptive filter:
Adaptive
filter
Error signal:
Mean square error:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 35
Least Mean Square (LMS) Algorithm
Basics – Part 3
Mean square error:
The filter coefficients are adjusted optimally in case of orthogonality:
Abbreviations:
(auto correlation matrix)
(cross correlation vector)
Solution (according to Wiener):
(assuming that the inverse exists)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 36
Least Mean Square (LMS) Algorithm
Basics – Part 4
Mean square error:
Optimal filter vector:
Minimum mean square error:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 37
Least Mean Square (LMS) Algorithm
Basics – Part 5
Mean square error:
Minimum mean square error :
Mean square error:
Quadratic form unique minimum
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 38
Least Mean Square (LMS) Algorithm
Derivation – Part 1
Derivation with respect to the coefficients of the adaptive filter:
… needed later on …
Inserting , results in:
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 39
Least Mean Square (LMS) Algorithm
Derivation – Part 2
What we have so far:
Resolving it to leads to:
With the introduction of a step size , the following adaptation rule can be
formulated:
Method according to Newton
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 40
Least Mean Square (LMS) Algorithm
Derivation – Part 3
Method according to Newton:
Method of steepest descent:
For practical approaches the expectation value is replaced by its instanteneous
value. This leads to the so-called least mean square algorithm:
LMS algorithm
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 41
Adaptive Filters – Algorithms
Summary and Outlook
This week:
Topics for the Talks
Introductory Remarks
Recursive Least Squares (RLS) Algorithm
Least Mean Squares Algorithm (LMS Algorithm) – Part 1
Next week:
Least Mean Squares Algorithm (LMS Algorithm) – Part 2
Affine projection Algorithm (AP Algorithm)
Digital Signal Processing and System Theory| Adaptive Filters | Algorithms – Part 1 Slide 42
