Objective

Topic Outline
IIIB2b. Smoothing
Types
General Noise Removal Algorithms
Sliding Point Algorithms General
Simple n Point FFT Filtering
Weighted n Point Principles
Example Fourier Transforms
Problems Pros and Cons
End Effects Basic Filter Design
Low Pass Limitations
Curve Fitting Algorithms
Ideal Filter Design
Polynomial/Spline
Wavelet Analysis
Peak Fitting
Problems Sample Smoothed Curves
MTS 723 1
 Objective
The objective of smoothing is to increase the
S/N at each channel in a spectrum by utilizing
the trends in the signal over the entire spectrum.
Goal
A “noise free” spectrum where Schannel = Schannel
and therefore (S/N)channel -> infinity
In this process, we want to
be consistent in our approach
be accurate in our methodology
avoid altering the true character of
the spectrum

MTS 723 2
 General
Sliding Point Algorithms
Simple n Point Averaging
Weighted n Point Averaging
Curve Fitting Algorithms
Polynomial/Spline Curve Fitting
Peak Fitting (to be discussed separately)
Noise Reduction Algorithms
Fast Fourier Transform (FFT) Filtering
Wavelet Filtering

MTS 723 3
 Simple n Point Averaging
j + (p-1)/2
^S = Σ S / p
j i
j - (p-1)/2
where
^ the new value of signal at channel j
Sj
Si the original value of signal at channel i
p the number of smoothing points
(an odd number)
^
We must calculate the new value Sj based on the
old values Si, then substitute the new for the old.
MTS 723 4
 General
j + (p-1)/2
^S = Σ wi Si
j
j - (p-1)/2
where
wi the weighting factor at channel i
Σ wi cannot be greater than 1

The weighting factor has a defined functional form

that is designed to account for the significance of
signals in channel i as a function of their distance
from the central channel j.
MTS 723 5
 Types
Binomial 1
11
The values of wi form a binomial 121
distribution (Gaussian curve) derived 1331
from Pascal’s triangle. 14641
....
P. Marchard and L. Marmet, Rev. Sci. Instrum. 54(8)
(1983) 1034.

Savitzky-Golay
The values of wi are determined by a type of least
squares polynomial with relevance to chemistry.
H. Madden, Anal. Chem. 50 (1978) 1383.

MTS 723 6
 Example

j + (p-1)/2 Signal
^ =ΣS /n
S j i
j - (p-1)/2 Value will be
substituted for signal
at this point.
Value of simple 17
point smoothing
algorithm calculated at
central point.

channel number

MTS 723 7
 Artifacts
n point smoothing algorithms ALWAYS alter the
shape of peaks in a spectrum

increase peak half-width

decrease peak height
shift peak positions on assymetric peaks

The extent of these changes may be

insignificant, but they are NOT non-existent.

MTS 723 8
 Examples
1.0
Effects of Smoothing
7 Point Algorithms
0.8
binomial (Gaussian) Savitzky-Golay
0.6
Counts

box (simple)
0.4

0.2

0.0

0 2 4 6 8 10 12 14
Channel Number

MTS 723 9
 End Effects
We have to consider how to handle the values
that extend beyond the ends of a spectrum.
Method S-i S(n-1)+i
Bounce Si S(n-1)-i
Wrap S(n-1)-i Si
Zero 0 0
Fill So S(n-1)
j + (p-1)/2
^ =ΣS /p n total number of points in spectrum
S j i
j - (p-1)/2
i counting index (from 0 to n-1)

MTS 723 10
 Polynomial/Spline Curve
Fitting
The data is fit over a
given range (of n) by Signal
regression analysis to 17 point cubic spline
a polynomial curve of polynomial fit to data
a given order. around this point
The value of the fit
curve is substituted
for the original value.
We calculate the new
values without changing
channel number
the old ones.

MTS 723 11
 Peak Fitting

Peak fitting is a sophisticated form of

smoothing.
We will cover the details later.

MTS 723 12
 Problems

The spline fitting routines have the same

problems as the n point smoothing
algorithms.
They introduce artifacts into peak shapes.

MTS 723 13
 General
The assumption
behind noise
reduction algorithms Parabolic Spectrum + Noise (dots)
overlaying Original Spectrum (line)
is that the signal and
the noise can be
cleanly separated
over the entire
spectrum, for
example by simple
Noise
mathematical
subtraction.
0 50 100 150 200
Channel Number
MTS 723 14
 Principles
The objective of FFT filtering is to remove noise N(E)
from an ideal signal S*(E) by working in Fourier space.

S(E) = S*(E) + N(E) in real space

S(f) = S*(f) + N(f) in Fourier space

Our goal is to find a filter f(f) to multiply to the

signal in Fourier space such that

S*(f) = S(f) • f(f)

(multiplication in Fourier space is convolution in real space)

MTS 723 15
 General
1.0
Assume this
A Fourier transform 0.5
goes on forever
puts a signal measured
0.0

with time into an

amplitude versus -0.5

frequency domain. -1.0

0 50 100 150 200 250
Time [s]
Coefficient Magnitude

To make further 120

considerations easier, 100
The above spectrum
consider channel 80

60 contains only one

number as a measure 40
frequency.
of time. 20

0.0 0.1 0.2 0.3 0.4 0.5

Hz

MTS 723 16
 Samples

Coefficient Magnitude Coefficient Magnitude

1.0
Signal Amplitude

120

0.5 100
Signal
80
0.0
60 f = 1 cycle/256 s
-0.5 40

20
-1.0
0
0 100 200
Time [s] 0.0 0.1 0.2
Hz
0.3 0.4 0.5
Signal Amplitude

1.0
120
0.5 100 Signal + Noise
80
0.0
60
fnoise = 1 cycle/50 s
-0.5
40
Anoise = Asignal/10
-1.0 20

0
0 100 200
Time [s] 0.0 0.1 0.2 0.3 0.4 0.5
Hz

MTS 723 17
 Nyquist Sampling Theorem
With digital data, the maximum frequency spectrum
that can fit the data is limited by the sampling rate.
1.0 1.0

0.5 0.5

0.0 0.0

-0.5 -0.5

-1.0 -1.0

0 2 4 6 8 0 2 4 6 8
Time [s] Time [s]

These both show the maximum frequency sine wave that fits to the
measured signals. Doubling the sampling rate doubles this frequency.
MTS 723 18
 Frequency Limit

The maximum “frequency” in

Fourier space that will fit digital
r
data sampled at a rate r is fmax =
defined by the Nyquist limit. 2
r is number of points per “time”
(or number of points per unit interval)

The minimum “frequency” in Fourier space that will

fit digital data containing n points in real space is
just one half of a sine wave over all the points.

MTS 723 19
 Example
Sampling Sampling
Rate Rate
(points per fmax (points per fmax
second) (Hz) second) (Hz)
1/2 1/4 1 1/2
1.0 1.0

0.5 0.5

0.0 0.0

-0.5 -0.5

-1.0 -1.0

0 2 4 6 8 0 2 4 6 8
Time [s] Time [s]

In AES or XPS, you can consider sampling rate as the number of

“points per eV” or the inverse of step size between channels (in eV).

MTS 723 20
 Series Truncation
Signal
Some signals may require an
infinite number of coefficients
to be correctly represented in
Fourier space.
(square pulses are an example)

Channel Number

Use of a finite number of Fourier coefficients will

cause “ringing” or so-called “Gibb’s oscillations” at the
points of abrupt change in slope.

MTS 723 21
 Pros and Cons
Pros
We expect to be able to use Fourier space to remove
noise easily because the noise coefficients should
appear mostly at higher frequencies.
Cons
The signal coefficients are distributed throughout the
entire frequency range.
The noise coefficients are distributed throughout the
entire frequency range.
Cutting off some of the coefficients could lead to
artifacts (Gibb’s oscillations).
MTS 723 22
 Objective
The objective of FFT transform filtering is to remove
noise from a spectrum by selectively removing the
coefficients associated with noise.

Restrictions
FFT must be done on a spectrum with 2n points
(2, 4, 8, 16, ... 128, 256, 512, 1024 ...)
filter design must consider the “type” of coefficients
used to represent the FFT spectrum
(they are typically a real + imaginary pair or an
amplitude + phase pair)

MTS 723 23
 Basic Types
Low Pass High Pass Band Pass
All coefficients All coefficients All coefficients
below a cutoff above a cutoff between two
frequency are frequency are cutoff frequencies
used. The used. The are used. The
others are zero. others are zero. others are zero.
Ci Ci Ci
1 1 1

frequency frequency frequency

The low pass filter has the most utilty for removing noise from a signal.

MTS 723 24
 Low Pass Limitations
• we cannot filter out all the noise
• we loose some of the signal coefficients
• we produce Gibb’s oscillations (especially at the
endpoints of the spectrum)
• itis not the optimal filter when the shape of the
signal is known
The balance between loss of spectral features and
amount of noise removed is determined by the
placement of the cutoff frequency
Use of an FFT low pass filter does not distort peak parameters as
drastically as n-point smoothing algorithms do.

MTS 723 25
 Principle
The objective is to find the optimal filter f*(f) that can
separate the signal S*(f) from the noise N(f) in Fourier
space based on a priori knowledge about the shape
of the “ideal” signal in real space S*(E).
F(f) = f*(f) • S(f) when transformed F(f) -> S*(E)

Goal
Minimize the residual r2 between the filtered signal
F(E) and the “ideal” signal S*(E).

< r2 > = | F(E) - S*(E) |2 dE

MTS 723 26
 Formulation
The optimal filter (the Wiener filter) is designed
according to
| S*(f) |2
f*(f) =
| S*(f) |2 + < | N(f) |2 >
RMS noise
We must have
the shape of the ideal signal S*(E)
the magnitude of the RMS noise in Fourier space
The former is a guess, and the latter is measured
from the spectrum in some manner, typically by
looking at high frequency in the FFT coefficients.
MTS 723 27
 Samples
1.0

0.8 A) Gauss 10% FWHM

As the FWHM of
Coefficient Magnitude

a peak widens,
0.6
the filter narrows.
C) Lorentz 10% FWHM
A Lorentzian
filter is narrower
0.4
at low frequency
B) Gauss 30% FWHM than a Gaussian.
0.2

0.0
0 5 10 15 20
mHz

MTS 723 28
 Wavelet Filtering
Wavelet analysis applies FFT filtering to a
higher level of sophistication. It breaks the
spectrum into “time slices” and analyzes for the
frequencies that best fit within the given time
slice. In other words, it does some of the
separation of low and high frequency
information in the time domain before going to
the Fourier domain.
Wavelet analysis may be an option for
smoothing spectra (as well as images) at some
point in your future career.

MTS 723 29
Sample Smoothed Curves

MTS 723 30