Introduction sparsity and compressed sensing

Dr Priscilla Canizares

1
© Dr Priscilla Cañizares
What have we got in store for today

• Wavelets and wavelets transform

• Sparsity and compressibility
• Introduction to compress sensing
• Examples in MRI
 Where are the highest and lowest frequencies?

Speach signal

https://speechprocessingbook.aalto. /Representations/Waveform.html
fi
 STFT
f (⌧ )W (t ⌧ )e i!⌧
d⌧ (90)m n
2. The Fourier Transform of the segment s(t) is computed to g
here x½k" denotes a signal and g½k" denotes an L-point window function. From
aussian: The function F (t = t , !) reveals the spectral content of
.7), the STFT of x½k" can be interpreted as the Fourier transform of the product
0

k"g½k$m". Figure 7-1 illustrates interval ofSTFT

computing time beginning
by taking at tFourier
= t0 andtransforms
ending at t =of
t0 a
+T
STFT can handle one frequency at the time, given by the xed size of the window
indowed signal. (91)
Z a
3. The window W (t) is shifted by t so that it can be used
e FT, and ⌧ determines the i!⌧
= hf, gt,! i = f (⌧ )W (t ⌧ )e segment (90)
d⌧ s(t) = fx(t)
(t)W (t t0 t). As before, the FT is app
content of this new segment.
osen to be g(t)
a Gaussian:
)2 /a2 Time
, Mathematically,
(91) this involves: Time
a controls the size of the window “scaling”
···· Z

FT
FT
FT

Z
window for the FT, and ⌧ controls
determines the
1
F (t, !) = fg (t, !) = in time
translation
ˆ f (⌧ )e i!⌧ ⇤
g (⌧ t)dt = hf, gt,! i = f(
1

The function g(t) is the kernel and is often chosen to be a Gau

y

fi
 Compressible signals/images

•Most natural signals, such as images and audio, are highly compressible

•The inherent structure observed in natural data implies that the data admits a sparse
representation in an appropriate coordinate system: Fourier decomposition, wavelets

•Sparse signal representation: when our signal is written in an appropriate basis (e.g. FT, WT)
only a few modes are active thus reducing the number of values that must be stored for an
accurate representation.

•The Fourier modes and wavelets are generic or universal bases vs ,e.g, SVD
es the CWT’s sensitivity to long-time-scale events, and wavelet
Mother wavelet
nsitivity to short-time-scale events. The basic idea in wavelet a
nction, (t), known as the mother wavelet:
!
1 t b
a,b (t) = p ;
a a
family of scaled and translated versions of the function by choos

aar wavelet: 8
es the CWT’s sensitivity to long-time-scale events, and wavelet
Mother wavelet
nsitivity to short-time-scale events. The basic idea in wavelet a
Shift
nction, (t), known as the mother wavelet:
!
1 t b
a,b (t) = p ;
a a
family of scaled and translated versions of the function by choos
Scaling

aar wavelet: 8
the CWT’s sensitivity to long-time-scale events, and wavelet contraction
tivity to short-time-scale events. The basic idea in wavelet analysis is to
ion, How does it works?
(t), known as the mother wavelet:
!
1 t b
a,b (t) = p ; 2.5.(99)
WAVELETS AND MULTI-RESOLUTION ANALYSIS 87
a a
mily of scaled and translated versions of the function by choosing a and b.
Example: Haar wavelet 1

1,0 0

wavelet: 8 -1
0 0.25 0.5 0.75 1
>
>
>
> 1 0  t < 1/2
>
< 1
1
,0 0
(100)-1
2
(t) = 1 1/2  t < 1
>
>
>
>
>
:0 otherwise
0 0.25 0.5 0.75 1

1
1 1
,
elets provide an orthogonal and hierarchical basis for a signal. (explain0
2 2

-1

0 0.25 0.5 0.75 1

45 t

Figure 2.24: Three Haar wavelets for the first two levels of the multi-resolution
in Fig. 2.23 (d).

begin with the continuous wavelet transform (CWT), which is given by:
Z 1
W (f )(a, b) = hf, a,b i = f (t) ¯a,b (t) dt, (2.54)
1

where ¯a,b denotes the complex conjugate of a,b . This is only valid for func-
• The wavelet is translated, dilated, and contracted
depending on the scale of activity under study.

• Dilation increases the sensitivity to long-time-scale

events, and contraction increases its sensitivity to
short-time-scales

• Wavelets provide an orthogonal and hierarchical basis

for a signal.

9
 86 86 CHAPTER CHAPTER
2. FOURIER
2. AND WAVELET
FOURIER TRANSFORMS
AND WAVELET TRANSFORMS
(c) Spectogram (d) multi-resolution
(c)(a)
Spectogram
Time series
(a) Time series (b) (d) multi-resolution
Fourier transform
(b) Fourier transform

Frequency
86 CHAPTER 2. FOURIER AND WAVELET TRANSFORMS

Frequency
(a) Time series (b) Fourier transform

!
!
(c) Spectogram
(c) Spectogram (d) multi-resolution
(d) multi-resolution
Time t
Time t
Figure 2.23: Illustration of resolution limitations and uncertainty in time-
Frequency
Frequency

Figure 2.23: Illustration of resolution limitations and uncertainty in time-

frequency analysis.
frequency analysis.
STFT
(c) Spectogram (d) multi-resolutionWavelets

tions are orthogonal then the basis may be used for projection, as in the Fourier
Frequency

tions are orthogonal then the basis may be used for projection, as in the Fourier
! transform.
transform.!
The simplest and earliest example of a wavelet is the Haar wavelet, devel-
The simplest and earliest example of a wavelet is the Haar wavelet, devel-
t
t Time oped in 1910 [227]:
oped in 1910 [227]: Time
Figure 2.23: Illustration of resolution limitations and uncertainty 8 in time-
Figure 2.23: Illustration of resolution limitations and uncertainty in time-
frequency analysis. 8 < 1 0  t < 1/2
frequency analysis. < 1 0  t < 1/2
! (t) = 1 1/2  t < 1 (2.53)
(t) = 1 1/2  t < 1 (2.53)
:
: 0 otherwise.
t tions are orthogonal then the otherwise.
0 basis may be used for projection, as in the Fourier
Time
tions are orthogonal then the basis may be used for projection, as in the Fourier
Figure 2.23: transform.
Illustration of resolution The three
limitations and Haar wavelets,
uncertainty in 1,0 , 1/2,0 ,10
time- and 1/2,1/2 , are shown in Fig. 2.24, repre-
The transform.
three Haar wavelets,
The simplest , 1/2,0
1,0and , andexample
earliest , are
1/2,1/2of a shownis in
wavelet theFig.
Haar2.24, repre-
wavelet, devel-
frequency
The analysis.
simplest and earliest example of a
senting
wavelet
the
is
first
the
two
Haar
layers
wavelet,
of
devel-
the multi-resolution in Fig. 2.23 (d). Notice that
senting the first twoinlayers
oped of the multi-resolution in Fig. 2.23 (d). Notice that
1910 [227]:
 Signals at multiple scales

• Generalises the STFT

• Allows for the analysis of non-stationary signals at multiple scales.
• Exploits multi-resolution decomposition: Enables di erent time and frequency delities in
di erent frequency bands.

11
ff
ff
fi
 <latexit sha1_base64="e9Kyev5DcIVpkqyxB39D/LVIMZY=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhYhBSmJSHUjFAUR3FToC9pQJtNJO3QyCTMTSShduvFX3LhQxK2f4M6/cfpYaOuBC4dz7uXee7yIUals+9vILC2vrK5l13Mbm1vbO+buXl2GscCkhkMWiqaHJGGUk5qiipFmJAgKPEYa3uB67DceiJA05FWVRsQNUI9Tn2KktNQxD63qTQFaaQFewjblqpPAOys5SQu+lRRgN+mYebtoTwAXiTMjeTBDpWN+tbshjgPCFWZIypZjR8odIqEoZmSUa8eSRAgPUI+0NOUoINIdTh4ZwWOtdKEfCl1cwYn6e2KIAinTwNOdAVJ9Oe+Nxf+8Vqz8C3dIeRQrwvF0kR8zqEI4TgV2qSBYsVQThAXVt0LcRwJhpbPL6RCc+ZcXSf206JSKpfuzfPlqFkcWHIAjYAEHnIMyuAUVUAMYPIJn8ArejCfjxXg3PqatGWM2sw/+wPj8AeG6lsA=</latexit>

Z
Generic transform (T F )(y) = K(x, y)f (x)dx
x

i!t
<latexit sha1_base64="9Y6od6cPUOEADcsIg8WEZzXxTj8=">AAACCnicbVA9SwNBEN3z2/h1ammzGgQFCXciainaCDYRjAaSGPY2c8ni3u2xO6eGI7WNf8XGQhFbf4Gd/8ZNcoUaHww83pthZl6QSGHQ876csfGJyanpmdnC3PzC4pK7vHJpVKo5VLiSSlcDZkCKGCooUEI10cCiQMJVcHPS969uQRuh4gvsJtCIWDsWoeAMrdR018+27ne627SuRbuDTGt1RylcZ6KuImgzir2mW/RK3gB0lPg5KZIc5ab7WW8pnkYQI5fMmJrvJdjImEbBJfQK9dRAwvgNa0PN0phFYBrZ4JUe3bRKi4ZK24qRDtSfExmLjOlGge2MGHbMX68v/ufVUgwPG5mIkxQh5sNFYSopKtrPhbaEBo6yawnjWthbKe8wzTja9Ao2BP/vy6Pkcrfk75f2z/eKR8d5HDNkjWyQLeKTA3JETkmZVAgnD+SJvJBX59F5dt6c92HrmJPPrJJfcD6+AfxNmds=</latexit>

Fourier K(x, y) ! e

<latexit sha1_base64="AiwK0ftp4xOv64PWdORUN8wO2hQ=">AAAB/XicbVDJSgNBEO2JW4zbuNy8NAYhgoQZkegx6MVjBLNAZhx6Oj1Jk56F7hohDoO/4sWDIl79D2/+jZ3loIkPCh7vVVFVz08EV2BZ30ZhaXllda24XtrY3NreMXf3WipOJWVNGotYdnyimOARawIHwTqJZCT0BWv7w+ux335gUvE4uoNRwtyQ9CMecEpAS5554CSKexk59fP7zCEK8gqceGbZqloT4EViz0gZzdDwzC+nF9M0ZBFQQZTq2lYCbkYkcCpYXnJSxRJCh6TPuppGJGTKzSbX5/hYKz0cxFJXBHii/p7ISKjUKPR1Z0hgoOa9sfif100huHQzHiUpsIhOFwWpwBDjcRS4xyWjIEaaECq5vhXTAZGEgg6spEOw519eJK2zql2r1m7Py/WrWRxFdIiOUAXZ6ALV0Q1qoCai6BE9o1f0ZjwZL8a78TFtLRizmX30B8bnD1kTlSs=</latexit>

⇤
Wavelet a,b (t)
Continuous wavelet transform

Admisibility condition
ere j,k is a discrete family of wavelets:
!
Continuous wavelet transform1 t kb
j,k (t) = (105)
aj aj
orthogonal wavelets like Haar wavelets, it is possible to expand a function f (t) uniquely
his basis:For orthogonal wavelets, it is possible to expand a function uniquely in this bases

1
X
f (t) = hf (t), j,k (t)i j,k (t) (106)
j,k= 1

47
Signals are generally not sparse in the original (pixel) space but can be sparse after being
decomposed on a suitable set of functions.

Pixel domain Wavelet domain

Smooth signals/images and piecewise smooth signals; images are compressible in the wavelet domain family.
Wavelet types

• We want the sparsest representation N>>M; this is problem-dependent

 104 CHAPTER 3. SPARSITY AND COMPRESSED SENSING

Compressibility and sparsity SENSING

3.2. COMPRESSED y C
101 s

Compressibility of images is related to the high dimensionality

= of image space:

• For a 20 × 20 pixel black and white image, there are 2400 distinct possible images!

pixel space
Figure 3.4: Schematic of measurements in the compressed sensing framework.

y ⇥ s y ⇥ s

= =
Figure 3.3: Illustration of the vastness of image (pixel) space, with natural im-
ages occupying a vanishingly small fraction of the space.

3.2 Compressed sensing

(a) Sparse s (`1 ) (b) Least-squares s (`2 )
Despite the considerable success of compression in real-world applications, it
still relies on having access to full high-dimensional measurements. The recent
Figure 3.5: `1 and `2 minimum norm solutions to compressed sensing prob-
advent of compressed sensing [150, lem. 112, 111, 113,
The difference 115, 109,
in solutions for 39, 114, 40]are
this regression turns the
further considered in
compression paradigm upside down: Chapter 4. instead of collecting high-dimensional
data just to compress and discard most of the information, it is instead pos-
calculation since using compressed sensing for image reconstruction is compu-
 possible. Sparsity also promotes interpretability since it provides the minimum number of
terms to explain our data. It also
Compressible signals/images: Sparsityleads to sparse optimisation, to add robustness to outliers
and missing data that bias regression techniques such as SVD.

What is sparsity?—A signal/image is considered a vector in a finite-dimensional subspace

of R N
(pixel space) so:
⇥ ⇤
x = x[1], . . . , x[N ] (110)

is sparse if most of its entries are equal to zero. That is, if it’s support:

⇤ = 1  i  N | x[i] 6= 0 (111)

is of cardinality k ⌧ N . Thus, a k sparse signal is a signal for which exactly k entries

are non-zero. If a signal is not sparse, it might be sparsified on an appropriate basis, that
is, with an appropriate transform domain. Eg. a sinus signal vs its FT (which is 1-sparse),
or a piecewise constant image, away from edges of finite length that has a sparse gradient.
Compressible signals/images: Sparsity

We can model a signal, x, as a linear combination of A elementary function (also called waveforms or atoms):
A
X
x= ↵= ↵[i]'i (112)
i=1

Where ↵[i] is the (representation) coefficient of x in the dictionary = ['1 , . . . , 'A ], the
N ⇥ A matrix whose columns are the atoms 'i , in general normalized to a unit `2 norm:

N
X
2
8i 2 {1, . . . , A} , k'i k = 2
|'i [n]| = 1 (113)
n=1

Signals/images, x, that are sparse in are those that can be written exactly as a super-
position of a small fraction of the atoms in the family ('i ))k
 2. Compressible signals/images
Compressible signals/images: Sparsity
In general, signals and images are not sparse.
A signal is compressible or "weakly sparse" if the sorted magnitudes k↵[i]k or the represen-
T
tation coefficients ↵ = x decay quickly according to:

k↵[i]k  Ci 1/s
i = 1, . . . , N (114)

With an approximation error from the k largest coefficients:

r
2
kx xk k  C k 1/2 1/s
s<2 (115)
s 1
order k approximation

You can keep a small fraction of the coe cients without much loss
ffi
FFT keep 0.05%
 FFT-1

Despite the considerable success of compression in real-world applications, it still

relies on having access to full high-dimensional measurements
 JPEG and MP3 compression

• Images and audio signals are compressible in Fourier or wavelet bases: in their Fourier or wavelet
representations (wave atoms or local discrete cosine transforms to represent locally oscillating textures), most
coefficients are small and may be neglected with negligible loss of quality.

• Instead of transmitting or storing full data, we just need to transmit/store these remaining few active coefficients

• Signal/image can be recovered by reconstructing it using the inverse Fourier or wavelet transform

International Journal of Computer Science & Engineering Survey (IJCSES) Vol.5, No.2, April 2014

Figure 1.represents the encoding of image compression system

1.3 Performance Criteria in Image Compression

 Compressed Sensing

23rd European Signal Processing Conference (EUSIPCO)

Magnetic Resonance in Medicine 58:1182–1195 (2007)

Sparse MRI: The Application of Compressed Sensing

for Rapid MR Imaging
COMPRESSED SENSING AND RADIO INTERFEROMETRY
Michael Lustig,1∗ David Donoho,2 and John M. Pauly1
M. Jiang⇤ , J. N. Girard⇤ , J.-L. Starck⇤ , S. Corbel⇤ , C. Tasse†
The sparsity which is implicit in MR images is exploited to groups: (a) Methods generating artifacts that are incoher-
significantly undersample k -space. Some MR images such as ent or less visually apparent, at the expense of reduced GEPI
†
Service d’Astrophysique, CEA Saclay,
⇤
angiograms are already sparse in the pixel representation; apparent SNR (1–5); (b) Methods exploiting redundancy
Observatoire de Paris-Meudon
other, more complicated images have a sparse representation in k-space, such as partial-Fourier, parallel imaging, etc. Orme des Merisiers
in some transform domain–for example, in terms of spatial 5, rue place Jules Janssen
finite-differences or their wavelet coefficients. According to
(6–8); (c) Methods exploiting either spatial or temporal 91410 GIF-Sur-YVETTE, France
the recently developed mathematical theory of compressed-
redundancy or both (9–13). 92190 Meudon, France
sensing, images with a sparse representation can be recov- In this article we aim to exploit the sparsity which is
Compressed Sensing for Time-Frequency Gravitational Wave Data Analysis
ered from randomly undersampled k -space data, provided an implicit in MR images, and develop an approach combin-
appropriate nonlinear recovery scheme is used. Intuitively, arti- ing elements of approaches a and c. By implicit sparsity
facts due to random undersampling add as noise-like inter- we mean transform sparsity, i.e., the underlying object we ABSTRACT ation of the beam impacting polarization (corrected wi
ference. In the sparsePaolo
transformAddesso,
domain theMaurizio Longo,
significant coef- aimStefano Marano,
to recover happens Vincenzo
to have Matta
a sparse representation in a projection [5]). Moreover, due to the limited number of
ficients stand out above
Dept.the interference.
of ElectricalA nonlinear thresh- known
and Computer and fixed mathematical
Engineering and AppliedtransformRadio interferometric
domain. To begin
Mathematics, imaging constitutes a strong ill-posed
inverse lines, not all Fourier regions are sampled and one requir
olding scheme can recover the sparse coefficients, effectively
University of Salerno, with,
84084 consider the identity
Fisciano (SA), transform,
Italy so that problem.
the transformIn addition, the next generation radio tele-
recovering the image itself. In this article, practical incoher- domain is simply the image domain scopes, such sparsity
itself. Here as the Low Frequency Array (LOFAR) and the tegrated measurements (in time or frequency) to increas
ent undersampling schemes are developed and analyzed by
means that there are relatively few significant pixels withArray (SKA), come with an additional
Square Kilometre accuracy of the recovered sky. The problem of aperture
means of their aliasing interference. Incoherence is intro-
Maria Principe,nonzero Innocenzovalues. MFor Pinto
example, angiograms are extremely thesis comes down to process this incomplete Fourier
duced by pseudo-random variable-density undersampling of direction-dependent effects which impacts the image restora-
Waves Group,
phase-encodes. The reconstruction is performed by minimiz- sparse
University of in the
Sanniopixelatrepresentation.
Benevento, More complex medi- either by solving a deconvolution problem using the in
ing the ℓ1 norm of a transformed 82100
image,Benevento,
subject to data cal INFN,
Italy, images mayLVC notand
be sparse
KAGRA in thetion. In the compressed sensing framework, we used the
pixel representation,
analysis
since and
they synthesis formulation of the problem and we mental point spread function (PSF), such as CLEAN an
fidelity constraints. Examples demonstrate improved(Dated: spatial but
May they do exhibit transform sparsity,
16, 2016) have a
resolution and accelerated acquisition for multislice fast spin- sparse representation in terms of spatial solved it using
finite proximal algorithms. A simple version of our
differences, derivates (e.g. [6–8]) or by solving the inpainting pro
echo brain imaging and 3D contrast enhanced angiography. in terms of their wavelet coefficients, or in terms of other
The potential of compressed sensing for obtaining sparse time-frequency representations method has been forimplemented
gravi- within the LOFAR imager and by recovering missing information in the Fourier plan
Magn Reson Med 58:1182–1195, 2007. © 2007 Wiley-Liss, Inc. transforms.
has been validated that scope, many teams have addressed this issue withi
the notion on simulated and real LOFAR data. It
tational wave data analysis is illustrated by comparison with existing methods, as regards i) shedding
Key words: compressed sensing; compressive sampling; ran- Sparsity is a powerful constraint, generalizing
dom light on the
sampling; rapidfine-structure of noise
MRI; sparsity; sparse transients (glitches)
reconstruction; in preparation of theirdemonstrated its capability
classification, and ii) to super-resolve radio sources, to framework of the Compressed Sensing (CS) theory (e.g
of finite object support. It is well understood why support
nonlinear reconstruction
boosting the performance of waveform consistency testsin in
constraints the space
image detection of unmodeled
(i.e., small transient
provide correct photometry of point sources in a large field of
FOV or band-pass and references therein). In [9], we developed a 2D spar
 d i = m1 + m2 z i (21)
1.1 What is Compressive Sensing?
Compressed sensing paradigm-shift:
In many practical problems of science and technology, one encounters the task of
M = 2
inferring quantities of interest from measured information. For instance, in signal (2
Mand= 2 processing, one would like to reconstruct a signal from(22)
image measured data.
When the information acquisition process is linear, the problem reduces to solving
a linear system of equations. In mathematical terms, the observed data y ∈ Cm is
2006 CSconnected
paradigm the signal x ∈ C of interest via
to shift: N

Ax = y.N >M (1.1) (2

N >M
If the measurements are sampled (23)
m×N randomly (i.i.d. and uniformly distributed), e.g., m × N Gaussian or
The matrix A ∈ C models the linear measurement (information) pro-
Bernoullicess.
matrix
Then one triesalltos-sparse
sampling, recover thevectors
vector x ∈ CN can be reconstructed
by solving exactly by optimisation if:
the above linear
system. Traditional wisdom suggests that the number m of measurements, i.e.,
✓ ◆
the amount of measured data, must be at least as large as the signal length N
✓ ◆of components of x). This N
(the number
N m principle
s lnis the basis for most devices (2
m used
where sisln
in current
a constant
technology, such as analog-to-digital s
conversion, medical
(24) imaging,
radar, and smobile communication. Indeed, if m < N , then classical linear algebra
Given a forward operator K 2 R
indicates that the linear system (1.1) is we can look for a solution in [0, 1] by solving
underdetermined
n⇥n and that there are infinitely n
n⇥n many solutions (provided, of course, that there exists n at least one). In other words,
K 2 R we can look for a solution in [0, 1] by solving a
constrained minimisation
without problem
additional information, it is impossible to recover x from y in the case
m < N . This fact also relates to the Shannon sampling theorem, which states that
blem Candes, Romberg and Tao., IEET IT 2006
the sampling rate of a continuous-time signal must be twice its highest frequency in
order to ensure reconstruction.
blem
nd and givesone encounters the task of
technology,
em of equations. In mathematical terms, the observed data
sured information.
heory is highly For instance, in signal
nd technology,
actical problems
reconstruct
of the potential a one
of
signal encounters
science
from and the task
technology,
measured data. of
one encounters the task of
the signal x ∈ C of interest via
sured
uantities How
information.
of to
interest reconstruct
For
from instance,
measured
s is linear, the problem reduces to solving
N an
in image
signal
information. from
For a few
instance, in measurements
signal
atical terms, the observed data y ∈ C data.
reconstruct
processing, onea signal
would from
like to measured
reconstruct m aissignal from measured data.
nformation
vialinear,acquisition
s is
st the problem process is linear,
reduces the problem reduces to solving
to solving
tem ofterms,
equations. In mathematical ∈ C theisobserved data y ∈ C is
data yterms,
m
atical the observed m
Ax = y.
o the
st via
= y. signal x ∈ C N
of interest via (1.1)
Ax = y. (1.1)
=linear
y. measurement (information) pro-
(1.1)
ix A ∈ C
nters the task
ctor x ∈ C m×N
trix
tance, models the linear measurement (inform
N of m×N
by solving •the
∈ Csignal models the
A in
Operator
abovethat
linear
measurement
models the linear
linear
measurement (information) •
pro-Without additional information, it is impossible to
hat the
linear number
measurement
m of measurements,
(information) Ni.e.,
pro-
one tries to recover the vector x ∈ C by solving the ab
one tries to
measured
at
ctor least
aditional x∈C as
wisdom
duces to solving
recover the vector x ∈ C by solving the above linear
data.
N large as the signal length N
by suggests
solving the that above
the linearm of measurements, i.e.,
number
• Vector we want to recover •
recover N
x from y

The underlying assumption which makes the

st of
hat principle
the is
number them basis
of for most
measurements, devices i.e.,
ditional wisdom suggests that the number m of measurem
measured
data y ∈ C is
og-to-digital
eratofleast
m data, must
conversion,
as largeofasx).theThis
components
be at
medical
signal
least as
imaging,
length
principle
• Our measurements
large
is Nthe
as the signal length N
basis for most devices
reconstruction possible is sparsity
d, principle
if technology,
m < Nis, then suchclassical forlinear
mostalgebra
of measured data, must be at least as large as the signa
srent
nderdetermined
mobile communication.
og-to-digital
the
and
basis
conversion,
(1.1)
as analog-to-digital
that there
Indeed,
medicalare
if m
devices
infinitely
imaging,
< N
conversion, medical imaging,
, then
• Undermined system of equations.
classical linear algebra
Compressed sensing = Sparse recovery

of components of x). This principle is the basis for mo

td,
atthere
ions
the
possible
exists
linear
(provided,
to
at
system
of
least
recover
one).
(1.1)
course,
x
is
that
from
In
y
ormation) pro- and that there are infinitely
nderdetermined
other
underdetermined
if m < N , then classical linear algebra
there
in
words,
exists
the case at
and that
least
there
one). In
are infinitely
other words,

ent technology, such as analog-to-digital conversion, medic

ditional
he nnon
gnal
above
t there
his fact must
surements,
information,
sampling
linear
exists it
theorem, is impossible
which
at least one). In other words,
alsoberelates
twice
i.e.,
to
itsthe Shannon
highest
to
states
sampling
frequency
recover
that
in
x from y in the case
theorem, which states that
possible to recover x from y in the case
g rate of a continuous-time signal must be twice its highest frequency in
CS: Beyond Nyquist–Shannon sampling

•Instead of collecting a high-dimensional measurement and then compressing

it, acquire compressed measurements.

•Solve the inverse problem through optimisation: nd the sparsest high-

dimensional signal/vector s consistent with those measurements.

fi
 relatively simple to state mathematically, but until
CHAPTER 3. SPARSITY AND COMPRESSED SENSING
t vector consistent with measurements was a non-
102 CHAPTER 3. SPARSITY AND COMPRESSED SENSING
lem. The rapid adoption of compressed sensing
surement How
matrix
and applied sciences rests
to
3
C reconstruct
2 R p⇥n
represents
on the
an
solida image
set of p
mathemat-
from
linear a few
measurements measurements
3
tate
The x. The
measurement choice of measurement
matrix C 2
es conditions for when it is possible to reconstructR p⇥n
matrix
representsC is aof critical
set of p importance
linear measurements
ressed
on the sensing,
state x. and
The
bability using convex algorithms. is discussed
choice of in Sec.
measurement 3.4. Typically,
matrix C measurements
is of critical importance
sistcompressed
in of random
• CS projections
sensing,
exploits the andofofisthe
sparsity a state,
discussed
signal on in
a which
in casetoTypically,
Sec.basis
generic 3.4. the entries
achieve full of Creconstruction from a few measurements
measurements
signal
ssed sensing exploits the sparsity of a signal in a
ssian or Bernoulli
may consist of randomdistributed random
projections variables.
of the state, inItwhich
is alsocase
possible to
the entries of C
signal reconstruction from surprisingly few mea-
individual
are • It is entries
Gaussian of collect
or Bernoulli
possible to x (i.e., single
distributed
fewer pixels
randomly randomif x (compressed)
chosen is an image),
variables. inalso
It is which
measurements possible tosolve for the nonzero elements of s in the
and then
-sparse
onsists
measure
in ,
oftransformed then
random rows
individual
instead
of the
entries
of
of
measuring
identity x directly
matrix.pixels if x is an image), in which
x (i.e., single
coordinate system.
compressing,
knowledge
case C consists it
of the is possible
of sparse
randomvector to collect
rows sofit the dramatically
is possible
identityto reconstruct
matrix. the signal
104 CHAPTER 3. SPARSITY AND COMPRESSED SENSING
compressed
3.1). With measurements
Thus,knowledge
the goal ofofcompressed and then
the sparse sensing
vector solve tofor
is is
s it findthe
possible the tosparsest vectorthe signal
reconstruct Sparsifying basis
he transformed
consistent
x from (3.1).with coordinate
the
Thus, goal ofsystem.
measurements
the y: The sensing
compressed measure- is to find the sparsest vector
areisgiven
⌧snthat by with the measurements y:
consistent y A
C s
x is K-sparse
y = inC s = ⇥s. (3.3)
y =A Cx. y =A C s = ⇥s. (3.2) = (3.3)
em of equations in (3.3) is underdetermined since there are infinitely
mportant
nsistent
The system collaboration
solutions
of equations between
s. The Emmanuel
sparsest
in (3.3) solutionCandès and Ter-
ŝ satisfies
is underdetermined thesince
following
there opti-
are infinitely
nsing
many theconsistent
problem: odd properties of signal
solutions reconstruction
s. The at their kids’
sparsest solution ŝ satisfies the following opti-
Measurement matrix
mization Theproblem:
system of equations is underdetermined since there
are in ŝnitely
= argmin ksk0 subject
many consistent to s.y = C s,
solutions (3.4)
7 Brunton & Kutz. All
s
ŝ =Rights
argmin ksk0 subject to y = C s,
Reserved. (3.4)
s
· k0 denotes the `0 pseudo-norm, given by the number of nonzero en-
swhere
is alsokreferred
· k0 denotesto asthethe`cardinality
0 pseudo-norm, of s. given by the number of nonzero en-
fi
Before Compressed Sensing

• Classically, we cannot recover a single if it is sampled at a bigger rate than the Nyquist-Shannon limit.

• In CS theory, sparsity can be exploited to recover an image/signal from far fewer measurements than

is required by the Nyquist–Shannon sampling theorem.

Nyquist-Shannon Aliasing

The sampling frequency should be at least twice

the highest frequency contained in the signal
Search for the sparsest vector

Ax=b

•Underdetermined system M<N

• Convex optimisation problem, it can be
solved with e cient methods, e.g.

MxN greedy type methods like basis pursuit.

See CVXPY or libLBFGS libraries.

min kxk1
<latexit sha1_base64="QJPlWzVhYyx70X7AcQ9B8jizU3U=">AAAB9XicbVBNS8NAEJ34WetX1aOXxSJ4KolI9Vj04rGC/YAmhs120y7dbMLuRlvS/g8vHhTx6n/x5r9x2+agrQ8GHu/NMDMvSDhT2ra/rZXVtfWNzcJWcXtnd2+/dHDYVHEqCW2QmMeyHWBFORO0oZnmtJ1IiqOA01YwuJn6rUcqFYvFvR4l1ItwT7CQEayN9OBGTPhD5I6H7th3/FLZrtgzoGXi5KQMOep+6cvtxiSNqNCEY6U6jp1oL8NSM8LppOimiiaYDHCPdgwVOKLKy2ZXT9CpUboojKUpodFM/T2R4UipURSYzgjrvlr0puJ/XifV4ZWXMZGkmgoyXxSmHOkYTSNAXSYp0XxkCCaSmVsR6WOJiTZBFU0IzuLLy6R5XnGqlerdRbl2ncdRgGM4gTNw4BJqcAt1aAABCc/wCm/Wk/VivVsf89YVK585gj+wPn8AQ9qSYA==</latexit>

subject to Ax = b
<latexit sha1_base64="DqHz5KaE3QRpMGd31GWdDDDnlUQ=">AAAB7HicbVBNSwMxEJ3Ur1q/qh69BIvgqeyKVC9C1YvHCm5baJeSTbNtaDa7JFmxLP0NXjwo4tUf5M1/Y9ruQVsfDDzem2FmXpAIro3jfKPCyura+kZxs7S1vbO7V94/aOo4VZR5NBaxagdEM8El8ww3grUTxUgUCNYKRrdTv/XIlOaxfDDjhPkRGUgeckqMlbzrJ3wV9MoVp+rMgJeJm5MK5Gj0yl/dfkzTiElDBdG64zqJ8TOiDKeCTUrdVLOE0BEZsI6lkkRM+9ns2Ak+sUofh7GyJQ2eqb8nMhJpPY4C2xkRM9SL3lT8z+ukJrz0My6T1DBJ54vCVGAT4+nnuM8Vo0aMLSFUcXsrpkOiCDU2n5INwV18eZk0z6purVq7P6/Ub/I4inAEx3AKLlxAHe6gAR5Q4PAMr/CGJHpB7+hj3lpA+cwh/AH6/AEEfY4v</latexit>

31
ffi
sparsity K is unknown, the search is even broader. Because this search is com-
binatorial, solving (3.4) is possible
intractable for to moderately
even relax the optimization
large n in (3.4)
and K, to a convex `1 -minimization [112, 150]:
and
N ⇥N
2R
the prospect of solving larger problems does not improve with Moore’s law of
ŝ = argmin ksk1 subject to y = C s, (3.5)
Thisincreasing
exponentially search is combinatorial:
computational solving
power. this `0 is intractable for even moderately large n
Search for the sparsest vector l1-minimisation s
Fortunately,
and K, under certain conditions on the measurement matrix
where k · k1 is the `1 norm, given by
C, it is
possible to relax the optimization in (3.4) to a convex `1 -minimization [112, 150]:
Fortunately, under certain conditions on the measurement matrix X C,n it is possible to relax
ŝ = argmin ksk1 subject to y = A C s, ksk1 = (3.5) |sk |. (3.6)
the `0 optimisation intos a convex `1 minimisation. k=1

where k · kThe
1 is `
the1 minimum
` 1 norm, norm
given solution
by is sparse, while the ` is not.
The `1 norm is also known as the taxicab or Manhattan norm because it repre-
2

sentsntothe
Conditions for `1 -minimisation distancewith
converge a taxi would
high take between
probability to the two points
sparsest on a rectangular grid.
solution:
X
ksk1 = 3 In the
|skcompressed
|. (3.6)
sensing literature, the measurement matrix is often denoted ; instead,
(1) The measurement matrix weCAuse
must
k=1 C to be
be consistent
incoherentwith the
wrt output
the equation
sparsifying in control
basis theory.
—the is
rowsalso already used to
denote DMD modes in Chapter 7.
The `1 norm is of also
C known
A are as the taxicab
not correlated or Manhattan
with the columns of norm because it repre-
sents the distance a taxi would take between twoCopyright points on © a2017
rectangular
Brunton &grid.
Kutz. All Rights Reserved.
3 (2) The number of measurements p must be of the order of:
In the compressed sensing literature, the measurement matrix is often denoted ; instead,
we use C to be consistent with the output equation in control theory. is also already used to
denote DMD modes in Chapter 7. p ⇡ O(K log (N/K))k (K log (N/K)) (144)
1

Copyright © 2017 Brunton & Kutz. All Rights Reserved.

Roughly speaking, these two conditions guarantee that the matrix C acts as a unitary
transformation on K sparse vectors s, preserving relative distances between vectors and en-
 ess is linear, the problem reduces to solving
to
e reconstruct
processing, onea signal
would
matical terms, the observed data from
like to measured
reconstruct
Such y∈ C data.
aissignal is
regularisation
m from measured
often used
Choosing aN todata.
promote *sparse*
independently solutions.
B. singleat random
pxl cameragives the inte
i
information
ess
rest is
vialinear, acquisition
the problem process
reducesis linear,Given
the
to solving an
problem image
reduces
B. zto
single2 solving
R
pxl
This functional is well-defined for u 2 H (⌦), with H (⌦) denoting the Sobolev
and
camera its (inverse) wavelet transform:
m space of
1 1

matical Example:
ystem ofterms,
equations.
the In single-pixel
mathematical
observed
functions u for N
data y ∈ camera
terms,
C mtheisobservedy data
= y ∈=C AWx
Az is
which both u and ru are square integrable. Thus, this type of regularisation Given an image z 2 RN
and its (inverse
drest
xto=they. signal
via x ∈ C of interest via (1.1)
generally leads to smooth solutions. Given an image z 2 R N
and its (inverse) wavelet transform:
B. single pxl camera
B.K : `single
Consider a forward operator pxl
! ` and let camera 8
he linear Ax = y.
measurement (information) pro-
1 2 (1.1)
x = y.
unters the task of (1.1) z = Wx z
ector x ∈ C m×N N
by solving theJ(u) above linear
matrix A ∈ C
nstance, in signal models the linear2Motivations,
1.2 Applications, = 1
kKu measurement
f k 2
+ ↵kuk
` and ` (information)
.N
Extensions pro- where z 9= Wx is unitary and
that
he the
linear Given
number
measurement m an
of image Given
measurements,
(information) z 2an 2

Rimage
i.e.,
Npro- and
1
z 2 its
R and
(inverse) its (inverse)
wavelet wavelet transform:
transform: W 2 R N xN
wavelet x
transform 2 RN
n one tries
m measuredSuch to
data. recover
regularisation the is vector
often used xto ∈ Cwhere
N
promote by solving
*sparse*W 2 R
the
solutions.
N xN
above is unitary
linear and x 2 R N
is a sparse vector.
e
ectorat
Traditionalleast
x ∈ CasN large as the signal length N
wisdom by solving
suggests the
that above
the numberlinear m of where
measurements, W 2 R N xN
i.e., is unitary {aand
i , . . .x, a
2 m }
R N2 is
[ a
1, sparse
+1] vector.
educes to solving
his
that
nt principle
of the number
measured is data,thembasis of
must for
be most
measurements,
at least devices
{a
as i.e.,
, .
large . . ,
asa the } 2
signal [ 1, +1]
length N Choosing a independently at random g
d data y ∈ C is m i m {a i , . . . , a m } 2 [ 1, +1] i
elog-to-digital
ber atofleast as conversion,
components largeofasx).theThis medical
signal
principle imaging,
lengthis N the basis for most devices z = Wx y = Az = AWx
eed, if m <
B. single pxl camera
N
technology, , then
his principle is the basis for most devices
urrent such classical
as linear
analog-to-digital Choosing
algebra conversion, a i independently
Choosing
medical a independently
imaging,
z = Wx
i at random
at random givesgives the
the intensities
intensities y n y
=
underdetermined
mobile communication.
log-to-digital (1.1)
Given an and
conversion,
image that
z 2 Rthere
Indeed,
N
medical
and are
if
its m infinitely
imaging,
<
(inverse) N ,
waveletthen classical
transform: y = linear
Az = algebra
AWx N
where Wy 2=R Az is N xN
= unitary
AWx and x 2 R is a sparse vector.
at there
that
eed, the
if m exists
linear
< where
N at
system
, thenleast (1.1)one). InNlinear
other
xN words,
is underdetermined
classical algebra and that there are infinitely
utions
mpossible (provided,
to recover of W
course,
x 2
fromthatR {a
ytherein
, . . is
exists
the
. , a unitary
case
} at
2 least
[ 1, and
one).
+1] In x 2
other R N
words,is a sparse vector. 8
nformation) pro- and that there are infinitely
underdetermined i m
(25)
dditional
hannon
thethere
at A= information,
sampling
above linear
exists{a at
i , theorem,
.least
. . , it
a is
one).
m
impossible
}which
2In [ states
other
Choosing 1, to
z = Wx
recover
that
words,
+1]a i
x
independently
from y
at
in the
random
case
gives the intensities y 8
n = hz, an i
This fact
signal mustalsobe relates
twice to xNthe
its Shannon
highest frequencysampling in theorem, which states that
easurements,
mpossible to
where i.e.,
recover
W 2 R N
x is from
unitary and
y in
x 2 the
R N
iscase
a sparse vector.
ing rate of a{acontinuous-time signal
y = must
Az = be AWxtwice its highest frequency in
signal
hannon length Choosing
samplingi , . . .
N , a
theorem,
m } 2 [ Fig.
1, a
+1]1.5 independently
Schematic
i which states that representation at
of a random
single-pixel cameragives
(Image the
courtesyintensities
of Rice University)y = hz, a
n n
nsure reconstruction.
Choosing ai independently at random gives the intensities yn = hz, an i
for most
signal must devices
be twice its highest frequency in
yy
roduction to Compressive
= Az = =Az =
AWxSensing, AWx 1
medical
and H. imaging,
Rauhut, A Mathematical a small mirror
C.
Introduction being
Solving
to switched
Compressive inverse on orproblems
Sensing,off, it contributes
using or not to the light
data-driven
1 intensity
models
OI 10.1007/978-0-8176-4948-7 measured 1,
cal linear algebra
d Numerical Harmonic Analysis, DOI at the sensor. In this way, one 1,
10.1007/978-0-8176-4948-7 realizes in hardware the inner product ⟨z, b⟩
 Before Compressed Sensing
Nyquist
NyquistSampling
SamplingTheorem
Theorem
k-space Image space

High-resolution (full-FOV)
Full sampling

Reduce high freq Kmax Low-resolution (full-FOV)

y we need to sample in k-space (k max

)? High-resolution aliasing
Increase(Δk)?
es be in k-space (low-FOV)
4/8/19
4/8/19 | PAGE
| PAGE 1414
Gototo3rd
Go 3rdnotebook
notebook

| PAGE 9
Choosing ai independently at random gives the intensities yn = hz, an i
y = Az = AWx
Preliminaries
Preliminaries

(1) Wavelet transform: W

W = ( W1 , . . . , Wn ) 2 Cn⇥n wavelet transform (orthogonal).

Pn
x = Wz = i=1 zi Wi where zi = hx, Wi i

Remember we want tor recover z not x. Preliminaries vs.

(2) Fourier transform and sampling:

vs.
W
Sensing matrix: Ao = F ⇤ W = (a⇤1 , . . . , a⇤n )T . Thus, one measurement is
yk = hak , zi

Let ⌦ ✓ {1, . . . , n} and A⌦ = (a⇤i )i2⌦ the sampling matrix. Then:

y = A⌦ z vs.

• 4/8/19
Nonlinear L1 reconstruction with noise
10

4/8/19
W
 Undersampled:six fold CS reconstruction Original: Fully sampled

https://candes.su.domains/teaching/stats330/index.shtml