EE434 Biomedical Signal Processing Lecture # 7
EE434
Biomedical Signal Processing
Lecture # 7
I would like to thank
Professor Robi Polikar for using his lecture notes.
21.11.2024 Lecture # 7 1
EE434 Biomedical Sig. Proc. Lecture # 7 Optimal & Adaptive Filtering
Optimal & Adaptive
Filtering
21.11.2024 Lecture # 7 2
EE434 Biomedical Sig. Proc. Lecture # 7 Optimal & Adaptive Filtering
Optimal Filtering
21.11.2024 Lecture # 7 3
EE434 Biomedical Sig. Proc. Lecture # 7 Optimal Filtering
• Optimal Filters
• The case against standard FIR / IIR filters
• Wiener Filter
• Development of Wiener Filter
• Wiener-Hopf Equations
• Applications / Demos of Wiener Filter
• Optimal Filtering
• Noise Removal
21.11.2024 Lecture # 7 4
EE434 Biomedical Sig. Proc. Lecture # 7 The Case Against FIR/IIR Filters
• The many forms of FIR and IIR filter design described in DSP allows us to design an arbitrarily
specific frequency response with arbitrary precision.
• Is there anything that we cannot do with IIR/FIR filters?
• No, not really. You can design any filter with any accuracy
• Then what is the problem?
• The user needs to know the precise filter characteristics that need to be designed.
• Is that not usually the case…?
• Not really! Often, we specify the filter characteristics based on some past experience, and/or
trial and error.
• FIR/IIR design techniques can design any filter you specify – it does not tell you whether that is
the right filter for that specific signal.
• Is there a technique that can…?
• Yes! There are design techniques that ensures that filter specs they specify are optimal with
respect to some meaningful criterion
• Wiener filters
21.11.2024 Lecture # 7 5
EE434 Biomedical Sig. Proc. Lecture # 7 Problem Statement
The optimal filter design problem can be stated as follows:
• Design an optimal filter to remove noise from a signal, given that the signal and
noise processes are independent, (wide sense) stationary random processes.
We will assume that we have access to the “desired” or signal, or the ideal
characteristics of the uncorrupted signal to be known. We further assume that
the statistical properties of the noise process are also known.
• The optimality is understood to be in “minimum mean square error” sense.
21.11.2024 Lecture # 7 6
EE434 Biomedical Sig. Proc. Lecture # 7 Shaky Assumptions?
• What does it mean that we have access to the desired / clean signal? If we
have access to the desired signal, why do we need to filter in the first place?
• Of course, we typically do not have the “clean” version of the noisy signal – if we did, we
would not need to clean it.
• However, we often have access to a reference signal, or at least a model of such a signal,
that can reasonably estimate the true, uncorrupted signal.
• As we will also see later, we technically need the autocorrelation sequence of the input
signal, along with its cross correlation with the desired signal – both of which are normally
estimated from the model / reference signal.
• This is discussed in more detail later in this lecture.
• How about noise?
• Assuming that the noise is white with constant power is not an unreasonable one most
applications.
21.11.2024 Lecture # 7 7
EE434 Biomedical Sig. Proc. Lecture # 7 Wiener Filter
• Wiener filter requires no spectral specification of which frequencies should be
blocked and which ones should be passed.
• Instead, it takes the statistical properties of the signal and noise processes into
account, and determines an optimal set of filter coefficients that are optimized
with respect to a performance criterion – a figure of merit.
• The criterion function is typically the difference between the filter output and
the desired signal (reference /model).
• Wiener filter minimizes the average squared difference between the two.
• The output of this filter is then guaranteed to be the best under the stated
assumptions.
21.11.2024 Lecture # 7 8
EE434 Biomedical Sig. Proc. Lecture # 7 Wiener Filter
21.11.2024 Lecture # 7 9
EE434 Biomedical Sig. Proc. Lecture # 7 Some Filter Math…
• The Wiener filter is typically designed to be an FIR filter – hence its output is
the convolution of x and w vectors:
21.11.2024 Lecture # 7 10
EE434 Biomedical Sig. Proc. Lecture # 7 Some Filter Math…
• Note that the vector x(n) is time dependent – it includes the currently available
sample at time instant n and its preceding M-1 values.
• The estimation error is then
• And the error we wish to minimize is
21.11.2024 Lecture # 7 11
EE434 Biomedical Sig. Proc. Lecture # 7 Some Random Process Math…
• Considering all signals as random variables, we have the following scenario
• Note that w is fixed (filter weights) and hence is not a random variable.
21.11.2024 Lecture # 7 12
EE434 Biomedical Sig. Proc. Lecture # 7 Some Random Process Math…
• Under the (somewhat incorrect) assumption that x(n) and d(n) are jointly
stationary, we have the following interpretations
• Since d(n) can be normalized to have zero mean, E[d2(n)] is the variance of
d(n), denoted as σd2
• Θxd = Θ = E[x(n) d(n)] is the cross correlation of the input and the desired
signals.
Similarly, ΘT = E[d(n) xT(n)] = (E[x(n) d(n)])T
• Also note that Θ is a M–by-1 vector of values Θ = [ θ(0), θ(-1), …, θ(1-M)]T,
where θ(k) = θ(-k) = E[x(n-k) d(n)], k = 0, 1, …, M-1. The stationarity
assumption makes the covariance function independent of n.
• Φxx = Φ = E[x(n) xT(n)] is the autocorrelation function of the signal x.
21.11.2024 Lecture # 7 13
EE434 Biomedical Sig. Proc. Lecture # 7 Some Random Process Math…
• The autocorrelation matrix, Φxx = Φ = E[x(n) xT(n)] can be shown in the full M-by-M
matrix form as
• where row k, column i element is ϕ(k - i) = ϕ(i - k) = E[x(n-k) x(n - i)]
• This is a toeplitz matrix, and for a WSS process, it is completely defined by its single
row of values, ϕ = ϕ0,…, ϕM-1.
• In Matlab, the autocorrelation matrix can be obtained by Phi_matrix = toeplitz(phi);
• Putting it all together, then, the criterion function to be minimized is
J(w) = σd2 − wTΘ − Θ Tw + wTΦw
21.11.2024 Lecture # 7 14
EE434 Biomedical Sig. Proc. Lecture # 7 Some Random Process Math…
J(w) = σd2 − wTΘ − Θ Tw + wTΦw
• The optimal filter weights, w* are the ones that minimize this (error) function → we
set the derivative of J(w) wrt to w to zero and solve for w :
• The following derivatives apply:
• Then
• This is the Wiener-Hopf equation, also called normal equation, the solution of which gives
the optimal filter coefficients w*
21.11.2024 Lecture # 7 15
EE434 Biomedical Sig. Proc. Lecture # 7 Wiener-Hopf Equations
• In expanded form, the Wiener-Hopf equations can be written as
• or as
• Note that this equation defines the convolution of filter coefficients with the
autocorrelation function of the input signal, which means we have
• The frequency response of the optimal Wiener filter is then
21.11.2024 Lecture # 7 16
EE434 Biomedical Sig. Proc. Lecture # 7 Wiener-Hopf Equations
• Having obtained the optimal filter coefficients as w* = Φ-1Θ, we can now calculate the
minimum mean square error (MMSE) as
• We note that the calculation of optimal filter weights require knowledge of the signal’s
autocorrelation function (easy to obtain), and the crosscorrelation function of the
input with the desired output (not so easy to obtain).
• Typically, the cross correlation function is estimated from some prior knowledge about the
second order statistics of the desired signal, and / or from a reference or model signal.
21.11.2024 Lecture # 7 17
EE434 Biomedical Sig. Proc. Lecture # 7 Matlab Implementation
• It seems rather simple to obtain the optimum filter coefficients once the
autocorrelation and cross-correlation functions are known:
w* = Φ-1Θ
• There are two potential problems:
• The autocorrelation function can be near singular (it cannot be completely singular, why
not…?), in which case the inverse will be unreliable
• Even if the matrix is not near-singular, for large matrices, the inversion may take a very long
time
• These problems can usually be solved by
• Test the condition of the matrix with c = cond(Φ). If c > 104, Φ is near singular! Then, reduce
the number of coefficients in the Φ, which will result in a shorter, but a more stable filter
• Use Levinson – Durbin to solve the Wiener-Hopf equations.
21.11.2024 Lecture # 7 18
EE434 Biomedical Sig. Proc. Lecture # 7 Matlab Demo
21.11.2024 Lecture # 7 19
EE434 Biomedical Sig. Proc. Lecture # 7 Matlab Demo
21.11.2024 Lecture # 7 20
EE434 Biomedical Sig. Proc. Lecture # 7 Matlab Demo
21.11.2024 Lecture # 7 21
EE434 Biomedical Sig. Proc. Lecture # 7 System Identification Using Optimal Wiener Filter
• Wiener filters can be used for different types of problems, such as system
identification:
• System identification: There is an unknown system, and we would like to
determine the characteristics (e.g., frequency response) of this system from an
input / output signal pair.
• For example to model the lungs from respiratory signals, or to model response
of a prosthetic, etc.
21.11.2024 Lecture # 7 22
EE434 Biomedical Sig. Proc. Lecture # 7 System Identification
• This is an example of match filter, where the wiener filter parameters are
adjusted such that its output can match the desired output.
• In this case, the unknown system output to a known input constitutes the desired
(reference) signal.
• Wiener filter will work particularly well, if the unknown system is an all-zero type
system (that is, an MA type system, with no poles), since the Wiener filters are FIR
21.11.2024 filters. Lecture # 7 23
EE434 Biomedical Sig. Proc. Lecture # 7 System Identification
fs = 500; % Sampling frequency
N = 1024; % Number of points
L = 8; % Optimnal filter order
f = (1:N) * fs/N; % freq. vector for plotting
%Generate unknown system and noise input
b_unknown = [.5 .75 1.2]; % Define unknown process
xn = randn(1,N);
xd = conv(b_unknown,xn); % Generate unknown system output
xd = xd(3:N+2); % Truncate extra points
% Apply Weiner filter
b = wiener_hopf(xn,xd,L); % Compute matching filter coefficients
b = b/N; % Scale filter coefficients
% Calculate frequency characteristics
ps_wiener = (abs(fft(b,N))).^2; %Spectrum of the matched filter
ps_unknown = (abs(fft(b_unknown,N))).^2; % of the unknown system
ps_out = (abs(fft(xd))).^2;
% Plot frequency characteristic of unknown and identified processes
subplot(1,2,1);
plot(f(1:N/2),ps_unknown(1:N/2),'k'); % Plot filtered data
xlabel('Frequency (Hz)'); ylabel('|G(z)|');
axis([0 fs/2 0 1.2*max(ps_unknown)]); grid
title('Unknown Process');
subplot(1,2,2);
plot(f(1:N/2),ps_wiener(1:N/2),'k'); % Plot filtered data
xlabel('Frequency (Hz)'); ylabel('|W(z)|');
title(['Matching Process with order ', num2str(L)]); grid
21.11.2024 Lecture # 7 24
EE434 Biomedical Sig. Proc. Lecture # 7 Filtering Out Noise
• Consider the problem of separating the additive noise η(n) from the (desired)
clean signal d(n) in the signal x(n) = d(n) + η(n). In vector notation:
x(n) = d(n) + η(n)
• We are still looking for the optimal wiener filter: w* = Φ-1Θ
• Let’s look at these terms individually:
• The autocorrelation of the input signal:
Φ = E[x(n) xT(n)] = E[{d(n) + η(n)}{d(n) + η(n)}T]
• Making the reasonable assumption that noise and signal are statistically independent:
E[d(n) ηT(n)] = E[dT(n) η(n)] = 0 and
Φ = E[d(n) dT(n)] + E[η(n) ηT(n)] = Φd + Φη
21.11.2024 Lecture # 7 25
EE434 Biomedical Sig. Proc. Lecture # 7 Filtering Out Noise
M-by-1
• As for the cross-correlation function: autocorrelation vector
of the desired signal
Θ = E[x(n) d(n)] = E[{d(n) + η(n)} d(n)]
= E[d(n) d(n)] + E[η(n) d(n)] = E[d(n) d(n)] = φd
0
• Then we have the optimal Wiener filter as
w* = Φ-1Θ = (Φd + Φη)-1φd
• The frequency responses are then
This is because Θ = φd
21.11.2024 Lecture # 7 26
EE434 Biomedical Sig. Proc. Lecture # 7 Mini Project
• Acquire several cycles of ECG – make sure it is noisy with, say 60 Hz noise as
well as EMG noise.
• Also collect as clean of an ECG as possible to be used as reference.
• Clean the noisy ECG using the Wiener filter approach
• Use a canonical ECG (time synchronized) as in the figure (in the next page) as a
reference ECG. How well does this work? Try to replicate the result in the figure.
• Now, add random noise to the clean ECG as well as the noisy ECG and denoise
this signal using Wiener filter, using each of the references mentioned above
(clean ECG, canonical ECG, etc.)
• Clearly explain and discuss your findings. Include matlab code, plots, etc.
21.11.2024 Lecture # 7 27
EE434 Biomedical Sig. Proc. Lecture # 7 Mini Project
21.11.2024 Lecture # 7 28
EE434 Biomedical Sig. Proc. Lecture # 7 Adaptive Filtering
Adaptive Filtering
21.11.2024 Lecture # 7 29
EE434 Biomedical Sig. Proc. Lecture # 7 Adaptive Filtering
• Adaptive Filters
• LMS algorithm
• Gradient descent
21.11.2024 Lecture # 7 30
EE434 Biomedical Sig. Proc. Lecture # 7 The Case Against FIR/IIR Filters
The main disadvantage of FIR & IIR filters has already discussed :
• Despite techniques available to design very accurate filters with specific design
parameters (specifications), we need such specification in the first place to
design them
• We need to know the specific spectral properties of the information carrying signal, as well
as the spectral properties of the corrupting signal (noise) in order to design filters that can
effectively separate the two signals
• Such information is often not available for many applications
• In such cases we need mechanism to determine what the filter is supposed to do – or
what kind of a filter we need in the first place.
• Wiener filters provide such a solution:
• Wiener filter provides us with the (FIR type) filter coefficients that are optimal in the mean
square error sense;
• The error is computed with respect to a reference signal, which is assumed to be available.
21.11.2024 Lecture # 7 31
EE434 Biomedical Sig. Proc. Lecture # 7 Wiener Filter
21.11.2024 Lecture # 7 32
EE434 Biomedical Sig. Proc. Lecture # 7 The Case Against The Wiener Filter
• But, the Wiener filter also has a main problem:
• And no, it is not the requirement that we have access to a reference signal, as in most
application we usually have access to some kind of a reference/model signal
• Although the Wiener filter does not require any information regarding the
spectral properties of the signal or the noise, it is too fixed for any given
application.
• Just like classical filter design techniques, Wiener filters cannot respond to
changes in the signal and/or noise properties that may occur in time
• Once the Wiener filter is designed, its coefficients are fixed and remain unchanged in time
– whether the signal / noise properties do so.
21.11.2024 Lecture # 7 33
EE434 Biomedical Sig. Proc. Lecture # 7 Adaptive Filters
• Applications where the signal / noise characteristics change in time are
common in practice, in particular in biological signals:
• The spectral characteristics of fetus ECG that is mixed with mother’s ECG vary
often
• The EMG noise that corrupts the ECG also varies in time depending on how strong
the muscle contractions are
• Often, the signal’s spectral properties themselves change in time, as is often the
case in EEG signals
• In such cases, where the interference (noise) is non-stationary, a filter whose
characteristics can be changed to track the changes in the noise is needed.
Such a filter is called an adaptive filter.
21.11.2024 Lecture # 7 34
EE434 Biomedical Sig. Proc. Lecture # 7 Adaptive Filters
The problem statement:
• Design an optimal filter to remove a nonstationary interference from a
nonstationary signal. An additional channel of information related to the
interference is assumed to be available. The filter should then continuously
adapt to the changing characteristics of the signal and the interference.
• The filter should be adaptive; the two weights of the filter must vary with
time
• The filter should be optimal.
• Such an adaptive filter are also known as an adaptive noise canceller (ANC)
21.11.2024 Lecture # 7 35
EE434 Biomedical Sig. Proc. Lecture # 7 Adaptive Filter Structure
Adaptive filter model (to be analyzed first):
Alternative adaptive filter model where input is signal + noise (to be analyzed later):
21.11.2024 Lecture # 7 36
EE434 Biomedical Sig. Proc. Lecture # 7 How Does It Work?
• The filter taps, lets call them “b” parameters in reference to numerator coefficients
of an FIR filter, are continuously updated – with every time step – through a
feedback loop
• The update is intended to bring the filters output y(n) closer and closer to the desired signal
d(n), such that the error between them, e(n) is minimized.
• The approach used for this minimization is a well known technique in optimization, called the
steepest descent, the same approach used in determining the synaptic weights of a neural
network.
• The algorithm that uses the steepest descent for determining optimal filter coefficients is also
known as the LMS (least mean square) algorithm.
21.11.2024 Lecture # 7 37
EE434 Biomedical Sig. Proc. Lecture # 7 How Does It Work?
Similar to the Wiener case, we assume that the filter is of FIR type
• The inherent stability of FIR filters make them an attractive choice – in particular
when designing adaptive filters, as we do not want to deal with the additional
issue of stability while continuously updating the filter taps.
• The output signal y(n) is once again the convolution of input and filter taps, but
the filter coefficients are now themselves a function of time:
21.11.2024 Lecture # 7 38
EE434 Biomedical Sig. Proc. Lecture # 7 How Does It Work?
• Hence while in Wiener filters, the filter taps are all determined at once ( in a
block approach), in adaptive filters, they are updated continuously
• While the Wiener filter approach can be adapted for this case, the recursive
LMS algorithm is more attractive because it does not require the
computation of auto / cross correlation functions – but rather uses the
gradient method known as the steepest descent.
• In such an approach, the taps are determined iteratively, updating the
weights at each time step by an amount equal to the gradient
21.11.2024 Lecture # 7 39
EE434 Biomedical Sig. Proc. Lecture # 7 Gradient Descent
• In gradient descent the goal is to find the minimum of the error function. Since
in LMS type approaches, the error is “least mean square” type, it is a quadratic
function:
J(b) = E[e2(n)]
• Therefore, this is a quadratic function, and has a parabolar shape. If we consider
the error as a one dimensional function, then it is a parabola (quadratic). If we
consider it as a two dimensional function (two weights) it is a parabolic surface.
If we have more than two weights, it is a higher dimensional surface, called
parabolic hypersurface.
• In gradient descent type approaches, we are then iteratively trying to find the
minimum point of this parabolic hypersurface, one step at a time.
21.11.2024 Lecture # 7 40
EE434 Biomedical Sig. Proc. Lecture # 7 Gradient (Steepest) Descent
• We are again interested in minimizing an error function, much similar to the one
used for Wiener filter:
• In this case, we iteratively adjust the weights based on the negative gradient of
the error with respect to the filter coefficients bn(k)
• Note that d(n) is independent of bn(k), and the partial derivative of y(n) with
respect to bn(k) is just x(n-k). Then, the gradient can be written in terms of the
instantaneous product of error and the input:
21.11.2024 Lecture # 7 41
EE434 Biomedical Sig. Proc. Lecture # 7 Gradient Descent
• The update rule is then bn+1(k) = bn(k) + 𝛻 n(J(bn(k))) = bn(k) + η e(n) x(n-k)
where η is a constant, called the learning rate, which controls the rate of
descent.
• Large values of the learning rate allows the algorithm to take big steps towards the
solution, but risks missing the minimum error point. Small values of learning rate takes
smaller steps, and hence may causes slow convergence and hence unnecessary
computation time.
• A common rule of thumb is usually to choose the learning rate as where L is the filter
order, and Px is the power of the input signal:
• All confused…?
• Remember that the error is quadratic, hence we are dealing with a parabolic hypersurface,
and we are trying to find the “bottom” of this surface, one step at a time. The learning rate
21.11.2024 is then the step size. Lecture # 7 42
EE434 Biomedical Sig. Proc. Lecture # 7 Gradient Descent
21.11.2024 Lecture # 7 43
