IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO.

3, SEPTEMBER 2018 311

Adaptive Sparse Image Sampling and Recovery

Ali Taimori and Farokh Marvasti , Senior Member, IEEE

Abstract—This paper presents an adaptive and intelligent specific domain can be recovered at a rate less than the tra-
sparse model for digital image sampling and recovery. In the pro- ditional Shannon-Nyquist rate theorem [4]–[7]. DCT, wavelet,
posed sampler, we adaptively determine the number of required and gradient spaces represent paradigms in which natural im-
samples for retrieving image based on space-frequency-gradient
information content of image patches. By leveraging texture ages are sparse or at least compressible. The later is not fully
in space, sparsity locations in DCT domain, and directional invertible.
decomposition of gradients, the sampler structure consists of
a combination of uniform, random, and nonuniform sampling
strategies. For reconstruction, we model the recovery problem as A. Considered Scenario and Related Arts
a two-state cellular automaton to iteratively restore image with In the literature, significant attempts have been done to com-
scalable windows from generation to generation. We demonstrate
the recovery algorithm quickly converges after a few generations
pressively sample data at a very low sampling rate [8]–[12]. Con-
for an image with arbitrary degree of texture. For a given number ventional CS-based sampling/recovery strategies only exploit
of measurements, extensive experiments on standard image-sets, the sparsity-prior of signals in an appropriate domain [13]–[15].
infra-red, and mega-pixel range imaging devices show that the On the other hand, theoretical compressive sampling analyses
proposed measurement matrix considerably increases the overall rely on using Gaussian or random sampling matrices to measure
recovery performance, or equivalently decreases the number
of sampled pixels for a specific recovery quality compared to
signals [16]. However, to design sensing matrix, recent scien-
random sampling matrix and Gaussian linear combinations tific studies discover considering signal model-prior in sampling
employed by the state-of-the-art compressive sensing methods. In phase can significantly improve recovery quality [9], [10], [12].
practice, the proposed measurement-adaptive sampling/recovery In this regard, the present paper concentrates on finding a way
framework includes various applications from intelligent com- to adaptively and sparsely sense image scene in order to simply
pressive imaging-based acquisition devices to computer vision and
graphics, and image processing technology. Simulation codes are
and efficiently retrieve the image.
available online for reproduction purposes. Ji et al. in [8] introduced an adaptive Bayesian CS (BCS) with
abilities such as determining sufficient number of measurements
Index Terms—Cellular automaton, compressive sensing, direc- and confidence of recovery. The researchers in [9] proposed
tional gradients, measurement-adaptive sampling, sparse recovery, a nonuniform CS-based image sampling/recovery method in
sparsity location, texture. which a Hidden Markov Tree (HMT) is utilized to consider the
I. INTRODUCTION correlation among sparse wavelet coefficients in both sampling
and reconstructing phases, so-called uHMT-NCS. In [10], a vari-
URRENT digital imaging devices at first acquire images
C and then separately compress them leading to big data-
related problems. Contrarily, the aim of emerging Compressive
able density sampling strategy was designed in the frequency
domain which exploits priori statistical distributions of images
in the wavelet space. Malloy and Nowak in [11] suggested an
Sensing (CS) theory is to merge sampling and compressing into adaptive compressed sensing algorithm that in comparison to
one step by introducing the concept of “compressing during standard random nonadaptive design matrices requires nearly
sensing” for reducing data-rate, acquisition time, power con- the same number of measurements but succeeds at lower SNR
sumption, and device manufacturing cost with demanding ap- levels. Yang et al. in [12] designed an efficient Mixed Adaptive-
plications to next-generation Infra-Red (IR), remote sensing and Random (MAR) sensing matrix which employs image edge
Magnetic Resonance Imaging (MRI) systems [1]–[3]. The the- information with a sensor array 16 times less than the ulti-
ory of CS states that signals with a sparse representation in a mate length of recovered image. The Iterative Method with
Adaptive Thresholding (IMAT) [15] and its modified version
Manuscript received June 27, 2017; revised November 19, 2017, February equipped with interpolation, IMATI [17], are extensions of the
25, 2018, and April 21, 2018; accepted April 22, 2018. Date of publication May Iterative Hard Thresholding (IHT) family for sparse recovery
7, 2018; date of current version August 13, 2018. This work was supported
by Iran National Science Foundation (INSF) and ACRI of Sharif University of [14]. Instead of a fixed thresholding, they adaptively thresh-
Technology under agreements 95/SAD/47585 and 7000/6642, respectively. The old coefficients in a transform domain such as Discrete Cosine
associate editor coordinating the review of this manuscript and approving it for Transform (DCT) during iterations. To recover sparse signals
publication was Prof. Jong Chul Ye. (Corresponding author: Ali Taimori.)
The authors are with the Department of Electrical Engineering, Sharif Univer- from Gaussian measurements, the authors in [18] presented a
sity of Technology, Tehran 14588-89694, Iran (e-mail:,[email protected]; general iterative framework based on proximal methods called
[email protected]). Iterative Sparsification-Projection (ISP). ISP family contains
This paper has supplementary downloadable material available at http://
ieeexplore.ieee.org. the well-known SL0 algorithm [19] as a special case. In [20], a
Digital Object Identifier 10.1109/TCI.2018.2833625 learning-based approach by jointly optimizing the sensing and

2333-9403 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 312 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

overcomplete dictionary matrices was introduced which per-

forms better than random matrices or the matrices that are op-
timized without learning dictionary. In [21], a variational BCS
in complex wavelet domain was suggested. This model, called
TSCW-GDP-HMT, considers sparsity and structural informa-
tion of images.

B. Motivations and Contributions

Although aforementioned efforts result in performance im-
provement somewhat, it seems that the role of Artificial Intelli-
gence (AI) in the context of CS-based sampling/recovery is yet Fig. 1. The proposed measurement-adaptive sparse sampler.
negligible. In this paper, we take a step towards considering AI
and present an adaptive and intelligent model for digital image
Briefly, the main contributions of the paper are
sampling and recovery. For a given number of measurements, r proposing a new measurement-adaptive image sampling
we show the proposed sensing matrix considerably increases
mechanism consisting of three uniform, random, and
the overall recovery performance, or equivalently decreases the
nonuniform samplers which, in a synergistically manner,
number of sampling points for a specific recovery quality com-
incorporates local space-frequency-gradient information
pared to Gaussian, random, and dynamic sampling matrices
content of the image,
employed by the state-of-the-art CS methods [8], [12], [13], r introducing a novel nonuniform sampler based on direc-
[15], [18], [21]. The number of required samples in the spa-
tional gradients,
tial domain to reconstruct image is adaptively determined based r suggesting a novel cellular automaton model for image
on space-frequency-gradient information content of image. To
recovery which its convergence is guaranteed after a few
this intent, a mixture of uniform, random, and nonuniform sam-
generations, and
pling strategies is suggested [2]. The devised intelligent sam- r exposing various examinations to demonstrate the effi-
pling mechanism donates the advantages of simple, fast, and
ciency of the proposed sampling/recovery approach under
efficient recovery compared to complicated methods [9], [18],
different settings and application areas.
[21]. For reconstruction, we model the recovery as a Cellular
The remainder of this paper is organized as follows. Section II
Automaton (CA) machine to iteratively restore the image with
describes the proposed sampler. In Section III, we explain the
scalable windows from generation to generation. To the best
CA-based recoverer. Section IV evaluates the effectiveness of
of the authors’ knowledge, this is the first work for modeling
our framework and compares it to modern approaches. The
of image reconstruction issue via a CA. The convergence of
paper is finally concluded in Section V.
the proposed CA-based recovery algorithm is guaranteed after
a few generations, thus making it suitable for reconstructing
II. INTELLIGENT SAMPLING
the present mega-pixel range images, whereas sophisticated ap-
proaches such as ISP [18], TSCW-GDP-HMT [21], and BCS Based on the content of local image patches, the proposed
[8] fail to process and store such high-dimensional signals with sampling method performs flexibly at various rates to capture
general-purpose computers. required informative samples for recovery step. Fig. 1 depicts
The adaptivity in signal measurements helps to automatically the block diagram of the suggested measurement-adaptive sam-
select enough sampling points based on local information con- pler. As shown, the system gets a gray-level image patch and
tent of the scene under view. For instance, lowpass images are gives the related binary mask. Due to psycho-visual considera-
sampled with low rates, whereas cluttered scenes are acquired tions, we assume that patches are square with the length b = 8.
with more samples for an accurate reconstruction. Generally, The final sampling mask is generated from the union of three
available methods in the literature do not have such a degree of uniform, random, and nonuniform patterns obtained from spa-
flexibility and for each signal are manually adjusted at a fixed tial, frequency, and gradient spaces, respectively. In the further
rate. This undesirable phenomenon may lead to data-redundancy sub-sections, we discuss about each sampling strategy in detail.
in smooth images or information-loss in textured ones. In prac-
tice, the proposed Measurement-Adaptive Sampling/Cellular A. Texture Measure
Automaton Recovery (MASCAR) scheme includes various ap- To adaptively sense the information content of each local
plications from intelligent compressive imaging-based acquisi- image patch, it may be required to measure a representative cri-
tion devices [22] to computer vision and graphics, and image terion of signal’s complexity [24]. Here, we utilized Shannon’s
processing technology, e.g. inpainting, remote sensing miss- joint entropy determined from Haralick’s co-occurrence matrix
ing recovery, denoising of impulsive noise, and mesh-based [25], [26] as a quantitative metric to measure image block tex-
image representation [23]. For reproduction purposes, simula- ture. We calculate the texture percentage η ∈ [0, 100] as
tion codes are available online1 .
 
100  
B B
1
η= p̂i,j log2 , (1)
1 http://ee.sharif.edu/∼imat/ Hm ax i=1 j =1 p̂i,j

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 313

and local correlation. We tried to experimentally design patterns

based on such constraints.

C. Random Sampler
Fig. 2. From left to right, patterns of the uniform sampler defined by matrices The intermediate layer of Fig. 1 depicts the proposed random
P UVL , P ULT , P UBT , P UHT , and P UVH .
sampler. Here, we first convert the data matrix R from the
spatial domain to the frequency space, Rf ∈ Rb×b , by using 2-D
where the gray-level co-occurrence matrix P  = [p̂i,j ] DCT transform as Rf = TRTT , where the matrix T represents
B ×B is the
joint Probability Mass Function (PMF) which is obtained by the the transformation kernel of DCT. In DCT domain, samples
quantized are decorrelated and sparse. After an adaptive quantization of
 s or
 scaled version of the input image patch, namely
Gs = gi,j . The joint probabilities can be defined in the DCT coefficients, we estimate the rate of random sampling.
b×b
horizontal, vertical, main diagonal, and off-diagonal directions. Accordingly, a uniformly distributed random mask is finally
We used the horizontal neighbor as defined in MATLAB by generated. In the following sections, we discuss about creating
s
p̂i,j = P gr,c s
= i, gr,c+1 = j , where 1 ≤ i, j ≤ B, 1 ≤ r ≤ this mask.
b, and 1 ≤ c ≤ b − 1. The variables r and c represent row and 1) Random Sampling Rate Measure: To measure sparsity,
column, respectively. We considered the number of distinct gray one way is to threshold insignificant coefficients in a transform
levels in Gs as B = 8, hence gi,j s
∈ [1, B], 1 ≤ i, j ≤ b. The domain [15]. However, such thresholding may be inaccurate due
 to varying information of the signal under process/acquisition.
normalizer Hm ax in (1) denotes the maximum entropy of P,
To overcome this problem, we suggest adaptive quantization of
which obtains from the uniform distribution, i.e. p̂i,j = B 2 , ∀
1
DCT coefficients inspired from the following observation.
1 ≤ i, j ≤ B. Thus, Hm ax = B i=1
B 1 2
j =1 B 2 log2 (B ), which Observation 1 (Adaptive quantization): In JPEG compres-
B = 8 ⇒ Hm ax = 6. sion standard where DCT transform is utilized, the results in
[24] reveal that the number of zero entries of quantized DCT
B. Uniform Sampler coefficients is inversely proportional to both compression qual-
In the proposed uniform sampling strategy, we punch local ity level, l, and the image block texture, η. Now, let the matrix
patches at certain regular locations based on their local texture R f = [r̃f ] ∈ Zb×b be the quantized DCT matrix for a b × b
i,j
content in space domain. In fact, the punch operation makes image patch, then  R  f  = b2 − Z(R  f ), where  ·  de-
0 p
appropriate regular patterns for a stable recovery. As shown in notes the p -norm and the operator Z(·) counts the number
Fig. 1, the uniform sampler at first gets the image patch R = of zero entries of a matrix/vector. From the above results, we
[ri,j ]b×b . Afterwards, the textureness of the signal is estimated. have both  R  f  ∝ 1 and  R  f  ∝ 1 . By assuming their
0 l 0 η
The number of punched points is proportional to the calculated equivalence, we have η ∝ l. This shows the texture percentage
texture percentage. By this intuition, whenever the textureness defined in (1) can be a good criterion for adaptive quantization of
is high, we decorate the image with denser regular points and information in the frequency domain to measure the necessary
vice versa. Therefore, the binary mask of uniform pattern is random sampling rate.
determined by the following fuzzy rule Details: Based on Observation 1, we calculate the quantized
⎧
⎪ PUVL , 0 ≤ η < 10 DCT matrix as
⎪
⎪
⎪
⎪
⎪
⎪ PULT , 10 ≤ η < 25  f = Rf ÷ Qη ,
⎪
⎨ R (3)
Pu = PUBT , 25 ≤ η < 45 , (2)
⎪
⎪ where the symbols ÷ and · denote the entry-by-entry division
⎪
⎪ η 
⎪
⎪ PUHT , 45 ≤ η < 70 and the nearest integer, respectively. The matrix Qη = qi,j ,
⎪
⎪ b×b
⎩ η
∀qi,j ∈ N, also represents the quantization table related to the
PUVH , 70 ≤ η ≤ 100
texture η. For determining Qη , we exploited the Quantization
in which the matrices PUVL , PULT , PUBT , PUHT , and PUVH are
Tables (QTs) introduced by Independent JPEG Group (IJG)
assigned for very-low, low, bandpass, high, and very-high tex-
[27]. Based on the value of η, the corresponding quantization
tures, respectively. Fig. 2 shows the regular lattices employed
matrix is chosen and applied (See Appendix.). Therefore, we
in the proposed uniform sampler. The number of live cells
adaptively derive sparsity number from the number of nonzero
(punched points) in the matrices PUVL , PULT , PUBT , PUHT ,  f  . Now,
entries of quantized DCT coefficients by k =  R
and PUVH are 20 , 21 , 22 , 23 , and 24 , respectively. This reveals 0
we suggest measuring the rate of random sampling as the fol-
the relation between the texture and the number of live cells is
lowing formula, which is consistent with the theory of CS, i.e.
nonlinear with an increasing exponential form. The configura-
c · k · log10 ( nk ),
tion of punched points affects the recovery performance. Sub-
optimal lattices should have minimum number of sample points 
c · k · log10 (d · nk ) , k=0
and maximum number of live cells in a predefined neighboring Rrs = , (4)
of missing samples. They should also consider patch boundaries 0, otherwise

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 314 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

in which n = b2 , and c and d are tunable factors. The param-

eter c changes the number of undersampled points, whereas
the coefficient d prevents from decaying sampling rate on well-
textured patches and saturates it at a relatively fixed rate to con-
trol unnecessary information. By setting c = 1.3 and d = 2.8,
the function in (4) is a non-decreasing curve vs k. It is im-
portant to note that, in practical situations, the number of
non-zero quantized DCT coefficients, k, may be a large value
even near the length of signal, n. This phenomenon occurs Fig. 3. A partitioning of 2-D frequency domain defined in [24], [29] into 6
regions.
especially in high quality level settings or textured regions.
Generally, high performance compressive sensing in such a
dense scenario is problematic because compressive sensing the- uncertainty and unambiguously recover the missing scattered
ory for real-world applications imposes the sampling rate to be samples. Hence, we suggest a sparsity location-aware algorithm
at most 10% [28]. We tried to consider this limitation for calcu- to capture horizontal, vertical, and diagonal edges in sampling
lating the rate of sampling in (4) by setting c = 1.3 and d = 2.8, phase.
which covers both sparse and dense cases. To implement the above idea, we address sparsity locations in
2) Graduate Randomization Procedure: Instead of blind different frequency regions. Fig. 3 depicts a partitioning of 2-D
random sampling adopted by researchers of compressive sens- frequency domain defined in [24], [29]. Except for the single
ing field, we introduce a Graduate Randomization Procedure DC mode D, the five subsets of AC modes exist, namely S1 to
(GRP). In this procedure, by receiving both the feedback of the S5 corresponding to low, horizontal, vertical, diagonal, and high
sampling rate in frequency domain and the generated uniform frequencies information, respectively. If nonzero quantized
mask, we add a certain randomness to the uniform pattern. In DCT coefficients are sparsely located in horizontal, vertical, or
other words, from low- to high- textured patches, GRP gradu- diagonal regions, the corresponding edge points are sampled
ally adds random nonuniformity to the uniform base lattice. It in space domain. For instance, in a block of well-structured
is important to note that random samples are selected and added vertical edges, the nonzero coefficients are only appeared in
to locations other than uniform marks. In order to generate the the partitions S1 and S2 . To detect the edge maps, we first
random pattern, the GRP algorithm is implemented as follows determine a D-directional decomposition of the image patch
r Find the number of zero elements in the uniform pattern,
gradient [30], [31]. For this purpose, we utilized Sobel operator
i.e. Z(Pu ), and their locations by storing in the set L  ∂r ∂r
to calculate the gradient of image block, g  [ ∂ ix, j , ∂ iy, j ]T ,
 i i Z(P u )
(p , q ) i=1 , where pi and q i represent the row and ∀i, j, and considered D = 8 with four cardinal directions N, E,
column of the ith location, respectively. S, W in addition to four ordinal directions NE, SE, SW, NW to
r Shuffle the location of zero entries and substitute the set L decompose the block. In order to preserve the image structure
with the result. in the boundary of blocks, we also acquired border pixels for
r Truncate (4) by d
computing the gradient. Let Fd γ = [fi,jγ ]b×b , ∀γ ∈ [1, D] ≡
 {N, NW, W, SW, S, SE, E, NE} ≡ {d1 , d2 , d3 , d4 , d5 , d6 , d7
Z(Pu ), Rrs > Z(Pu )
Rrs = . (5) d8 }, be D-directional gradient features. To project the gradient
Rrs , otherwise vector g in (x-y) plane onto the two adjacent directions
r Generate Rrs integer values randomly without replacement ⎧ ∂ ri,j ∂ ri,j
⎨ |(| ∂ x |−| ∂ y |)|
in the range [1, Z(Pu )], namely v = [v1 , v2 , . . . , vR ]T . g ,  g  2 = 0
α1 = , (6)
r Define the matrix of random pattern, Pr = [pr ] r,s and ⎩
2

i,j b×b 0, otherwise

initialize it with Pr = O, where O represents a b × b zero ⎧√
⎨ 2 m in(| ∂ ix, j |,| ∂ iy, j |)
∂r ∂r
matrix. ,  g  2 = 0
r For n = 1 to Rrs , at first, assign (i, j) ← L{vn }, and then α2 = g , (7)
⎩
2

pri,j ← 1. 0, otherwise

the following rules are utilized.

D. Nonuniform Sampler r If ∂ r i , j ≥ 0 ∧ | ∂ r i , j | ≤ | ∂ r i , j |, then assign f d 1 ← α1 ;
∂x ∂y ∂x i,j
∂ ri , j ∂ ri , j ∂ ri , j
It is very important to preserve the structure of image like else if ∂x <0∧| ∂y |≤| |, then fi,j
∂x
d5
← α1 ; else
edges in recovery side. Generally, edge preservation is a dif- ∂ ri , j ∂ ri , j ∂ ri , j
if ∂x ≥ 0 ∧ | ∂ y | > | ∂ x |, then fi,j ← α1 ; else if
d7
ficult task in interpolation even for sophisticated algorithms. ∂ ri , j ∂r ∂r
Due to fuzziness nature of image edges, an edge-prior sampler ∂x < 0 ∧ | ∂ iy, j | > | ∂ ix, j |, then fi,j
d3
← α1 .
can reduce uncertainty in reconstruction phase. Although, the r ∂r
If ∂ ix, j
∂ ri , j
≥ 0 ∧ ∂ y ≥ 0, then assign fi,j d8
← α2 ; else if
GRP carry this task somewhat, in well-structured horizontal, ∂ ri , j ∂ ri , j ∂r
∂x ≥ 0 ∧ ∂ y < 0, then fi,j ← α2 ; else if ∂ ix, j < 0 ∧
d2
vertical, and diagonal edges, or periodic configurations which ∂ ri , j ∂r ∂r
their recovery is perceptually important from subjective eval- ∂y < 0, then fi,j d4
← α2 ; else if ∂ ix, j < 0 ∧ ∂ iy, j ≥ 0,
uation viewpoint, more edge samples are required to reduce then fi,jd6
← α2 .

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 315

Algorithm 1: The proposed adaptive sampling algorithm.

1: Input: The image patch R.
2: Measure the texture η as (1).
3: Generate the uniform sampling mask Pu by (2).
4: Determine the quantization table Qη in terms of η.
5: Transform the patch R to 2-D DCT domain Rf .
6: Quantize DCT coefficients via R  f = Rf ÷ Qη .
7: Estimate the sparsity number k =  R f  .
0
8: Calculate the rate of random sampling as (4).
9: Perform the GRP routine to obtain the random
mask Pr .
10: Run the nonuniform sampler algorithm  to generate Pn .
11: Combine the created masks as M = i∈{u ,r,n } Pi .
Fig. 4. The gradient directions corresponding to cardinal and ordinal direc-
12: Obtain the scattered sample points by S = M  R.
tions, and the directional decomposition of the gradient vector g into two d1 13: Outputs: The mask M and the subsampled image S.
and d8 adjacent directions.

Afterwards, we apply a threshold to the magnitude of two di- chaotic behavior. For selecting the threshold τ in the nonuniform
rections with stronger gradients to suppress redundant uninfor- sampler, a trade-off between subjective quality enhancement and
mative samples. Fig. 4 illustrates the directional decomposition increased redundancy exists. If τ → 1, then M → Pu Pr .
of the gradient vector g into two adjacent directions. For this ex- Therefore, we empirically set τ = 0.9 to maintain sampling
d1 d2 d3 d4 d5
ample, we have fi,j = α1 , fi,j = 0, fi,j = 0, fi,j = 0, fi,j = 0, near an optimal sub-Shannon-Nyquist rate. The downsampled
d6 d7 d8 image patch is constructed from
fi,j = 0, fi,j = 0, and fi,j = α2 based on the specified rules.
The steps of nonuniform sampling algorithm are implemented S = M  R, (9)
as follows.
r Initialize D-directional gradient matrices, i.e. Fd = O, in which the symbol  denotes Hadamard product. Algorithm 1
γ
∀γ ∈ [1, D]. summarizes the proposed adaptive intelligent image sampling
r Extract D-directional gradient features as explained above strategy. For the previously captured image I of the dimension
for the vector g. h × w, the proposed block-wise algorithm can be repeated on all
r Obtain the normalized versions of Fd , ∀γ ∈ [1, D], be- nonoverlapping image patches to get the overall sampling mask
γ
tween 0 and 1. Ms = [msi,j ]h×w and the subsampled image Is = [isi,j ]h×w .
r Define the matrix of nonuniform pattern, Pn = [pn ] ,
i,j b×b
and initialize it as Pn = O. III. CELLULAR AUTOMATON-BASED IMAGE RECOVERY
r For determining the structure of edges, initialize a binary
As mentioned in the literature, different approaches exist for
string with the length of the number of AC frequency
image recovery [8]–[11], [13]–[15], [18]–[21]. Here, we tailor a
regions, Nfr = 5, such as a = “a1 a2 a3 a4 a5 ” = “00000”.
r For s = 1 to Nfr , obtain the set Bs  {r̃f |(i, j) ∈ Ss }. If novel technique to recover scattered samples obtained from the
i,j intelligent sampling stage. We model the reconstruction mecha-
 Bs  0 = 0, set as ← “1”.
r If the string a = “11000” ∨ “10100” ∨ “10010” ∨ nism by a dynamic CA. Cellular automata are simple intelligent
agents with fundamental characteristics of locality, uniformity,
“11010” ∨ “10110”, which respectively represent vertical,
and synchronicity [32]. They are composed of discrete cells
horizontal, diagonal, vertical-diagonal, or horizontal-
d equipped with special rules to be able to solve sophisticated
diagonal edges, at first, sort the set { bi=1 bj =1 fi,jγ }D γ =1 . problems. Various applications have been found for CA in im-
Afterwards, for the two directions having stronger gra- age processing tasks such as noise removal, edge detection, and
dients, calculate its gradient magnitude, and then forgery localization [33].
renormalize it between 0 and 1. Finally, for i = 1 to b and
j = 1 to b, if the resulting magnitude of gradient exceeds A. Cellular Automaton Modeling for Recovery
the predefined threshold 0 ≤ τ ≤ 1, set pni,j ← 1.
In modeling, we consider a 2-state CA machine, in which the
E. Measurement-Adaptive Sampling Algorithm states of dead and live cells represent the zero and one values in
the sampling mask Ms , respectively. Contrary to conventional
As shown in Fig. 1, the ultimate binary mask is generated by fixed-neighbor CA models, the proposed CA-based recoverer
the union of uniform, random, and nonuniform patterns as performs as an iterative method which applies variable-scale
 windows to the sampled image Is . The size of square win-
M= Pi . (8)
dow, Ω, increases at each generation of CA. In summary, we
i∈{u ,r,n }
define the elements of model as M(Ms , Is , Ω, σ, ζ, I, Mg ), in
The regularity, randomness, and structure of patterns in the which σ represents the standard deviation of a Gaussian kernel,
above combined form create naturally a sampling mask with ζ ≥ 1 is a coefficient for increasing σ during generations, and
Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 316 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

Mg and I denote the matrix at next generation of Ms and the Algorithm 2: The proposed CA-based recovery algorithm.
recovered image, respectively. The rule behind the suggested 1: Inputs: The mask Ms and the subsampled image Is .
recovery algorithm is simple yet efficient as follows. By con- 2: Initialize Ω = 3, σ = 1, ζ = 1.05, I = Is , Mg = Ms ,
sidering the correlation among adjacent samples, missing pixels (0)
and Ms = Ms .
are reconstructed via a Gaussian-weighted averaging. We find
3: Calculate the number of dead cells by Nd = Z(Mg ).
live cells of Is around the central dead cell msi,j and extract
4: while Nd = 0 do
their corresponding weights in (Ω − 1)-Moore neighborhood
5: ωh  Ω2 
[32], namely the dynamic-length vectors x = [x1 , . . . , xk ]T and
6: Obtain the Gaussian kernel Wσ = [wi,j σ
]Ω×Ω .
ω = [ω1 , . . . , ωk ]T , respectively. If at least one live cell exists,
7: for r ← 1, h do
then, the dead cell msi,j is replaced with the weighted mean of
8: for c ← 1, w do
live cells in the subsampled image and that dead cell will re-
9: if msr,c = 0 then
vive at next generation. At each generation of CA-based recov-
erer, the model elements are updated. Algorithm 2 presents the 10: ϑflag  0
pseudo-code of CA-based image recovery algorithm, in which 11: t1
the symbols · and · represent the floor and ceiling func- 12: for p ← −ωh , ωh do
tions, respectively. We utilized the replication rule to cope with 13: for q ← −ωh , ωh do
boundary conditions of border cells. 14: if msr +p,c+q = 1 then
After the last generation, we utilize a post processing stage 15: xt  isr +p,c+q
to alleviate possible blockiness artifacts due to the patch-based 16: ωt  wp+σ
Ω Ω
2 ,q + 2
nature of recovery. This phenomenon may be observed in plain 17: ϑflag = 1
regions. Based on the coding style of measurement-adaptive 18: t←t+1
sampler, the number of live cells within a window can be an 19: end if
appropriate criterion to discriminate plain regions from textured 20: end for
ones in the recovery phase. Therefore, we apply a conditional 21: end for
smoothing filter only for flat regions. To this intent, at first, 22: if ϑflag = 1 then
we initialize the post-processed result Ip = [ipi,j ]h×w as Ip = I.
îr,c = Σxk ωωk
T
23:
Afterwards, for all i ∈ [1, h] and j ∈ [1, w], we scan the im-
24: mgr,c = 1
age I = [ii,j ]h×w with a square window of the length Ωf = 3. 25: end if
(0)
If the number of live cells in the initial sampling mask Ms 26: Release the vectors x and ω.
inside the window are less than the threshold ρ · Ωf , the cor-
2
27: end if
responding pixel values of I in the window are smoothed via 28: end for
σ
the Gaussian kernel Wσ f = [wi,jf ]Ω f ×Ω f , for which 0 ≤ ρ ≤ 1. 29: end for
Then, the result is placed into ipi,j . In experiments, we set 30: Ω←Ω+2
the coefficient ρ = 0.3 and the standard deviation of smoother 31: σ ←ζ ·σ
σf = 1.5. 32: Is = I
33: Ms = Mg
34: Nd = Z(Mg )
B. Convergence Analysis 35: end while
36: Apply the post processing on I via Ms to obtain Ip .
(0)
As illustrated in Fig. 6, the CA-based recoverer first starts
with exposing high-textured dense missing samples at a fine 37: Output: The recovered post-processed image Ip .
scale. Then, by updating model parameters, this process con-
tinues and finally terminates by retrieving lowpass sparse sam-
ples at a coarse scale. The advised variable-scale windowing
in comparison to fixed window speeds up the convergence rate for all  hb ×  wb patches, we have η = 0 which results in
of Algorithm 2 without sacrificing recovery quality. Also, this Pu = PUVL , and the random and nonuniform patterns equal
mechanism controls better error propagation during recursion. Pr = Pn = O. Hence, M = Pu for all patches. Let the live
In Lemma 1, we prove the algorithm quickly converges after a and dead cells be represented by white- and black- colored cells,
few generations for an image with arbitrary degree of texture. respectively. For visualization, Fig. 5 illustrates generations of
Lemma 1 (Convergence guarantee): Let patches of the im- the sampling mask matrix Ms for a given flat DC image of the
age I = [ii,j ]h×w be square of the length b = 8. If Ng denotes dimension 32 × 32. It is important to note that, if h and w are
(κ)
the number of CA generations, then, Algorithm 2 for recovering not multiples of b, we can use zero padding. If the matrix Ms
the image I guarantees to converge at most at three iterations, denotes the sampling mask at generation κth, then it is shown
i.e. Ng  3. in Fig. 5 that the number of dead cells at the 3rd generation is
(3)
Proof: To prove, we consider the worst case, i.e. the im- Nd = Z(Ms ) = 0. This implies that N g = 3 for a completely
age under reconstruction is a fully DC image as the possible DC image because the distance between two live cells is 7 pixels
sparsest signal to determine the upper bound. In such a case, in both horizontal and vertical directions. For a given image with

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 317

Fig. 5. Applying Algorithm 2 on a 32 × 32 DC image. From left to right,

sampling matrices encompassed by the green-colored frame illustrate initial
state, and the evolution at 1st, 2nd, and 3rd generations, respectively.

a certain degree of texture, the density of live cells in Ms is al-

ways more than or equal to the worst case, thus demonstrating
Ng  3. 
Example 1 (Sampling and recovery): Fig. 6 shows the orig-
inal images of Baboon, Boat, and Lena for which the sampling
masks, subsampled images, the intermediate, and final results
of recovery step are visualized. These images have different tex-
ture averages sorted in descending order, for which the sampler
automatically extracted 35.23, 25.16 and 18.12 percent pixels,
and the recoverer estimated the images with PSNRs of 29.38,
29.43 and 32.19 decibels, respectively.

C. Practical Considerations
In order to implement the proposed sampling and recovery
algorithms in practice, we can consider a mechanism similar
to the structure of image acquisition using Rice single-pixel
camera [22]. However, one of the main differences is to uti-
lize a small array of sensors but not a single photo-diode. This
allows us to measure adaptively the local image content, sam-
ple in 2-D space, and recover the scene by the suggested CA-
based recovery approach. The solution is to focus the physical
scene on a Digital Micro-mirror Device (DMD) via a primary
lens. The DMD itself is partitioned into non-overlapping b × b
patches such as 8 × 8 blocks. In a sequential line-scan manner
and at each time, only one of the analog image patches is re-
flected to another lens by configuring mirror positions. Other
mirrors reflect the scene on a black absorbent surface. This sec-
ondary lens focuses the light of that single patch on an 8 × 8
CCD or CMOS sensor. After amplifying the signal and passing
through an Analog to Digital Converter (ADC), the digitized
sub-image, R, is obtained. Afterwards, the texture of the patch
is measured. Then, based on the proposed sampler, the sampling
mask, M, and the sub-sampled image, S, are determined. This
process is repeated until the generation of the binary mask as Fig. 6. From top to down, different images respectively show the original
images, sampling masks, subsampled images, recovered results after 1st, 2nd
well as the sub-sampled image pertaining to the last patch. The and 3rd generations, and the results of applying conditional smoother.
overall sampling mask, Ms , and corresponding sampled image,
Is , can be stored in a memory card. In the decoding phase, A. Experimental Settings
the proposed CA recoverer gets Ms and Is and simply restores
the image scene, Ip , whereas the single-pixel camera utilizes In order to evaluate the performance of our sampling/recovery
the sophisticated Basis Pursuit (BP) algorithm for recovery. framework and comparing it with other related approaches, we
utilized various databases having different statistical character-
istics. Image sets include the standard databases of NCID [34],
IV. EXPERIMENTS CCID [35], UCID [36], Microsoft Research Cambridge Object
This section provides extensive experiments to support our Recognition Image Database,2 and a collection of well-known
findings. Algorithms were implemented in MATLAB and run
on an Intel Core i7 2.2 GHz laptop with 8 GB RAM. 2 https://www.microsoft.com/en-us/download/details.aspx?id=52644

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 318 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

test images such as Baboon, Cameraman, Lena, etc. Beside, TABLE I

RESULTS OF THE PROPOSED SAMPLING/RECOVERY METHOD
we gathered a set of IR test images as well as a mega-pixel
range images database called RCID [37], to more precisely
investigate the efficiency of our algorithms. Some of these im-
ages are shown in Fig. 12. In order to evaluate recovery per-
formance, we employed standard objective criteria of PSNR,
SSIM [38], and Normalized Recovery Error (NRE) defined as
NRE   Ip − I  2 / I  2 as well as subjective evaluation.
We compared our sampling and recovery algorithms to related
approaches including the intelligent MAR sampler suggested in
[12], the scattered data recovery technique of spline interpola-
tion and the state-of-the-art methods of BCS [8], uHMT-NCS
AMP [9], IMAT [15], IMATI [17], ISP [18], and TSCW-GDP-
HMT [21].
We adjusted the set of parameters so that, in the suggested
sampler, the PSNR is maximum for a given average rate. For
tuning parameters, we used images of UCID database as val-
idation set. To generate uniform matrices in the uniform sam-
pler, different regular and periodic lattices with various punched
points may be used. We examined a set of such configurations
and finally selected the patterns introduced in Fig. 2, which
optimize the aforementioned objective criterion. We also set pa-
rameters of CA-based recoverer and its post-processing stage
to obtain a sub-optimal objective PSNR performance criterion
in dB. The influence of post-processing stage with appropriate
parameters of σf and ρ in the CA-based recovery algorithm is
to improve PSNR up to about 0.5 dB. The standard deviation
of smoother σf , meanwhile, has negligible impact on perfor-
mance. The number of punched points in the uniform sampler,
the coefficients c and d in the random sampler, and the thresh-
old τ in the nonuniform sampler control the sampling rate in the
suggested algorithm. On one hand, for natural image patches,
we have often η ≤ 45, thus considering short intervals for such
textures. This also means the number of punched points is often
less than or equal to 4. On the other hand, the main usage of
the coefficient d is to make the formula acceptable for dense
images, i.e. high values of k. Therefore, the influential factors
for increasing or decreasing the adaptive sampling rate are the
coefficient c and the threshold τ .

B. Performance on Standard Databases

Here, we evaluate the performance of our sampling/recovery
method on 21 well-known test images of the size 512 × 512
and four standard databases. The adaptive sampling rate mea-
Fig. 7. Average PSNR (dB) vs average sampling rate (%) on images listed in
sured by sampler and recovery quality on image processing test Table I.
images are tabulated in Table I. For an average 23.75% dy-
namic sampling rate, the mean of PSNR was 30.48 dB. For
variations 0 ≤ c ≤ 3 and 0 ≤ τ ≤ 1, the curves of the average the sensitivity of our algorithm is low for a wide variation of
PSNR (dB) vs the average sampling rate (%) on 21 well-known c and τ . Fig. 8 also shows average PSNR (dB) vs ρ. In this
images are depicted in Fig. 7. As seen in the red curve of this figure, the coefficient ρ = 0 means the recovery without apply-
figure, an increase in the coefficient c yields an increase in the ing post-processing step and the optimal value is ρ∗ = 0.3, thus
sampling rate and also an improvement in PSNR with a linear demonstrating about 0.5 dB gain.
relation between the sampling rate and PSNR. In the blue curve As an example, Fig. 9 illustrates recovery of Baboon image
of Fig. 7, decreasing the threshold τ causes an increase in the for compared state-of-the-art approaches. To be able to compare
sampling rate without meaningful improvement in the objective different algorithms of various complexities on the laptop, we
performance, i.e. a flat curve. The curves also demonstrate that reduced the image to the size 128 × 128. For all methods, the

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 319

Fig. 10. The run-time (s) of different methods in terms of sampling rate (%)
Fig. 8. The influence of the coefficient ρ on recovery performance for images on the image of Fig. 9.
listed in Table I.

also intra-database variability of textural information. However,

we tuned parameters of our sampler so that dynamic sampling
rate for each image doesn’t approximately exceed 50%, based
on the limitation imposed in compressive sensing theory for
real-world applications [28]. This leads to an acceptable aver-
age PSNR with a high variance, so that the range of PSNR (dB),
[min(PSNR), max(PSNR)], is [17.39, 48.48], [21.91, 41.25],
[18.41, 40.03], and [26.29, 44.35] for NCID, CCID, UCID, and
RCID databases, respectively. From compression viewpoint, re-
covery performance in comparison to sensed samples is promis-
ing for all databases.
Fig. 9. PSNR for different methods in dB. From top to down and left to right,
the original image, recovered images obtained by uHMT-NCS AMP (24.15), C. Test on Infra-Red Imaging Systems
ISP (19.46), BCS (18.58), IMAT (24.25), IMATI (24.95), TSCW-GDP-HMT
(26.72), and the proposed algorithm (27.33), respectively. Infra-red imaging has a wide range of applications from
surveillance and Intelligent Transportation Systems (ITSs) to
medical imaging. These images are naturally sparse and of in-
sampling rate was 36.62%, i.e. 6000 measurements. Based on terest in the compressive sensing community to fabricate their
PSNR, the proposed approach outperforms other methods. For low-cost CS-based imagers [39]. In this experiment, we evaluate
BCS, IMAT, IMATI, TSCW-GDP-HMT, and the proposed al- the performance of adaptive sampling/recovery framework on a
gorithm, the average PSNR (dB) on all images with the size of test-set including 20 images grabbed by various near IR cameras
128 × 128 was 18.52, 23.34, 24.26, 28.54, and 27.99, respec- for ITSs applications such as License Plate Recognition (LPR).
tively. Other methods such as ISP failed to compute the image Four representative IR images are shown in the first row of
restoration process. The average sampling rate determined by Fig. 12. Specifically, for recovering images 1 to 4 in this figure,
our algorithm was 34.3%. For a fair comparison, we set this PSNRs(sampling rates) in terms of dB (%) were 46.68 (2.08),
rate for competing algorithms, too. Although the average PSNR 35.89 (11.58), 42.04 (4.41) and 38.86 (6.57), respectively. Gen-
for TSCW-GDP-HMT algorithm is 0.55 dB more than the sug- erally, all IR images were recovered by an average PSNR of
gested method, its computational complexity is very high. To 39.28 dB for the average adaptive sampling rate 7.16%, thus
clarify this issue, Fig. 10 compares the run-time (s) of differ- demonstrating an excellent IR recovery quality of our scheme
ent methods in terms of sampling rate (%) on the 128 × 128 for the high sample reduction.
Baboon image. As seen, the proposed method is slightly better For BCS, IMAT, IMATI, TSCW-GDP-HMT, and the pro-
than IMATI. posed algorithm, the average PSNR (dB) on all images with the
Table II reports the performance of our approach on NCID, size of 128 × 128 was 28.44, 22.11, 26.34, 35.23, and 31.88,
CCID, UCID, and RCID databases, which are sorted in descend- respectively. The average sampling rate determined by our al-
ing order of textural information; i.e. RCID and NCID databases gorithm was 15.37%. For a fair comparison, we set this rate
have minimum and maximum texture averages, respectively. for competing algorithms. The performance of the proposed
The average sampling rate acquired by the proposed dynamic method is the second best. However, as illustrated in Fig. 10,
sampler confirms this subject. Images of each image-set have the main advantage of the proposed method in comparison to

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 320 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

TABLE II
PERFORMANCE OF THE PROPOSED IMAGE SAMPLING/RECOVERY FRAMEWORK ON FOUR STANDARD DATABASES

the sophisticated TSCW-GDP-HMT approach is its very low

computational burden which makes it practical for real-world
implementation.

D. Influence of Uniform and Nonuniform Samples

In this experiment, we considered two scenarios to evalu-
ate the role of the proposed uniform and nonuniform sam-
plers. In the first scenario, we omitted our proposed uniform
sampler from the scheme of Fig. 1 and compared the re-
sulting random/nonuniform sampler with the whole model.
This can be done by setting Mu = O. In this case, for a
fair comparison by preserving samples at an equal rate, we
can manually increase the coefficient c in the random sam-
pler or decrease τ in the nonuniform sampler. Here, we
only increased c from the resulting random/nonuniform sam-
pler. In this case, the average PSNR on 21 well-known
images listed in Table I was 29.78 dB. In comparison to
the result reported in Table I, a decrease of nearly 0.7 dB
in PSNR is seen which demonstrates the importance of existing
uniform sampler. It is noticeable that performance reduction in
low-pass and periodic images like Cameraman and Barbara is
much more than cluttered ones.
In the second scenario, the nonuniform sampler was neglected
by setting τ = 1. Similarly, we increased c to equalize the rate of
resulting uniform/random sampler with the complete scheme.
The average PSNR was 30.52 dB, that is 0.04 dB more than
the average of Table I. Although such an objective evalua-
tion shows a little performance improvement in the absence
of nonuniform sampler, different subjective evaluations narrate
a better reconstruction of image structure such as edges in the
presence of nonuniform sampler. For instance, Fig. 11 visual-
izes this phenomenon for four images in which reconstructed
edges are sharper in the entire solution than the second scenario.
Therefore, for approximately equal sampling rate and objective
PSNR metric, the combined strategy has subjective superiority
in performance.

E. Recovery Performance under Fixed Sampling Structures Fig. 11. From left to right and top to down, the reconstruction of Walkbridge,
Livingroom, Jetplane, and Lake images without and with nonuniform sampler,
This section compares the performance of different recovery respectively.
approaches under fixed sampling structures. To this intent, we
considered two sampling scenarios including conventional pure [15], [17], [18], [21], we applied the prominent scattered spline
random and our measurement-adaptive samplers. For recover- interpolator of spline to separately investigate the efficiency of
ing scattered samples obtained from the proposed intelligent our CA-based recoverer under the considered scenarios.
sampling stage as well as random sampling structure, various In this experiment, we address the problem of image re-
approaches such as scattered data interpolators can be employed. construction of different scales. To do this, we selected 4
Therefore, in addition to modern sparse recovery techniques [8], mega-pixel images from RCID database with the original size

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 321

TABLE III
ANALYZING THE PERFORMANCE OF DIFFERENT RECOVERY TECHNIQUES IN THE CASE OF TWO DISTINCT SAMPLING SCENARIOS ON IMAGES SHOWN IN THE
SECOND ROW OF FIG. 12

For each scenario, the values in bold type show the best performance.

Generally, state-of-the-art results were obtained for config-

urations of measurement-adaptive sampler+spline for low-
resolution and measurement-adaptive sampler+CA for high-
resolution levels. This demonstrates the power of measurement-
adaptive sampling scheme.
It is noticeable that IMAT and IMATI recovery methods re-
construct the whole image rather than block-wise processing.
The sub-optimal parameters for adaptive thresholding tech-
Fig. 12. From left to right, images in the first and the second rows indexed
by numbers 1 to 4, show representative infra-red and RCID mega-pixel images
nique of IMAT were α = 0.5, β = 50, and Niteration = 30. In
[37], respectively. ISP algorithm, DCT was utilized as sparsifying transform and
hard thresholding function. We also set its sub-optimal param-
eters as γ = 0.4, σnoise = 0, τf = 5e-6, c = 0.9, I = 3, and
min(h, w) × max(h, w) = 3456 × 5184 or vice versa. These maxiter = 300. TSCW-GDP-HMT method uses Generalized
images are shown in the second row of Fig 12. To evaluate the Double Pareto (GDP) distribution for modeling signal sparsity.
efficiency of the mentioned approaches, by using the bicubic in- An inherent limitation of this algorithm is that the input image
terpolation, we generated 6 down-sampled versions for each of should be square with a length of multiples of 8. To tackle this
which by a factor about ↓ 2 to finally construct a 7-level pyramid. problem and compare different approaches as fair as possible,
Table III compares the proposed CA-based recovery algorithm we first considered the nearest length more than or equal to di-
to modern reconstruction methods under random and adaptive mensions reported in Table III, and then rescaled the recovered
sampling scenarios for original and scaled images. From lowest image to the real size. BCS algorithm utilizes Daubechies1 2-D
to highest resolution, i.e. in the direction of increasing corre- wavelet at four decomposition levels. We also used MATLAB
lation, the average dynamic sampling rates (%) were 42.95, built-in function to implement spline interpolation.
38.63, 34.4, 30.21, 26.69, 22.37, and 15.59, respectively. For a
fair comparison, we set the random sampling rate equal to that of
F. Evaluation under Fixed Recovery Algorithms
the average dynamic rate determined by measurement-adaptive
sampler. The bolded values show the best performance. Except In this experiment, we evaluate the performance of three sam-
for very low resolution levels, ISP, TSCW-GDP-HMT, BCS, pling methods including the pure random, the state-of-the-art
and spline methods failed to yield viable results due to com- dynamic MAR sampling approach [12], and the suggested sam-
putational complexity of matrix operations or huge memory pler under fixed recovery algorithms. We utilized 21 well-known
requirement. Because of heavy blockiness effects, BCS algo- test images of the original size 512 × 512 for performance eval-
rithm fails to reconstruct images in both the random measure- uation. As known in Table III, only IMAT, IMATI and the
ment and the adaptive scheme especially in lowpass images. proposed CA-based recoveres can restore the images of such

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 322 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

Fig. 13. The comparison of recovery performance for different sampling ap-
proaches. 2
Fig. 14. Average PSNR (dB) vs the noise variance, σ AW G N , for the case of
noisy measurements.

dimensions. Hence, these recovery algorithms were considered

to be able to recover subsampled images. MAR sensing matrix is
basically generated from the low-resolution versions of images
with the size 128 × 128. For a fair comparison with this method,
we created subsampled images from Hadamard product of MAR
masks and original images in recovery side. Fig. 13 depicts av-
erage PSNR criterion in dB for different sampling/recovery con-
figurations. The sampling rate for all samplers is the same and
equal to 23.75%. The first and the second ranks are belonging Fig. 15. From top to down and left to right, test samples of Microsoft Database
to the proposed measurement-adaptive sampler+CA recoverer indexed by numbers 1 to 27.
and MAR sampler+CA recoverer, respectively. It is noticeable
that the performance of IMAT and IMATI recovery algorithms the performance of the proposed sampling/recovery with other
under the pure random sampling strategy are better than other state-of-the-art frameworks [8], [9], [15], [17], [18]. To do this,
state-of-the-art samplers due to their nature. we collected 27 randomly selected samples from Microsoft Ob-
ject Recognition Database as shown in Fig. 15, which were
G. Recovery Robustness in the Presence of Noise already employed in [9]. At first, the randomly chosen images
In order to evaluate the robustness of the proposed CA-based were cropped for highlighting salient objects in the scene, and
recovery algorithm in the presence of noise, we can model the then their gray-scale versions were resized to 128 × 128 di-
noisy subsampled image, Is , as mensions using the bicubic interpolation to be able to compare
different algorithms of various complexity. Fig. 16 plots NRE
Is = Is + (Ms  N), (10) vs image number as indexed by numbers 1 to 27 in Fig. 15
where matrices Is , Ms , and N denote the noise-free subsampled for competing methods. The average NREs are given in leg-
image, the binary sampling mask, and Additive White Gaussian end parentheses. As shown, the average NRE of the proposed
Noise (AWGN) with the distribution of N (0, σAWGN2
), respec- scheme is lower than other methods. BCS failed to recover the
tively. We then applied the normalized version of Is between smooth images 4 and 18, whereas our scheme retrieved them
0 and 1 to CA recoverer. Fig. 14 plots average PSNR in dB vs with minimum recovery error. For a fair comparison, we set the
2
the noise variance, σAWGN , for 21 well-known test images of sampling rate of competing methods the same as the average
the size 512 × 512. In this figure, σAWGN
2
= 0 means the noise- rate 32.44% obtained adaptively by our sampler.
free case. The suggested method shows better performance in
terms of the noise variance than IMAT and IMATI approaches. I. Sensitivity Analysis on Various Patch Sizes
2
All curves exponentially decay by increasing σAWGN . As men-
Different imaging modalities may employ various patch sizes
tioned before, other algorithms fail to recover these images due
such as scalable High Efficiency Video Coding (HEVC) with
to data dimensionality.
the normal patch sizes of 4 × 4, 8 × 8, 16 × 16, and 32 × 32
[40]. Here, we investigate the sensitivity of the proposed sam-
H. Performance of Joint Sampling/Recovery Frameworks
pler on these patch sizes. To this end, all designed matrices of the
In practice, design of sampling structure and recovery stage dimension 8 × 8 in Section II must be either down-sampled or
are closely related to each other. Here, we jointly compare up-sampled. Hence, for generating uniform patterns, frequency
Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 323

TABLE IV
A QUALITATIVE PERFORMANCE COMPARISON OF THE PROPOSED SAMPLING/RECOVERY FRAMEWORK TO BASELINE JPEG SCHEME

Fig. 16. Normalized Recovery Error (NRE) of different approaches for the Fig. 17. The robustness analysis of the proposed method under various sam-
images labeled in Fig. 15. pling patch sizes.

modes, and the reference quantization matrix of the dimension

algorithm. The considered performance metrics consist of sam-
4 × 4, we used the sample-and-hold technique by selecting the
pling rate, bit rate in bit per pixel (bpp), objective quality in terms
top-left entries of 2 × 2 partitions from the related 8 × 8 patches.
of PSNR (dB), and computational complexity. Table IV quali-
To create the uniform patterns for dimensions more than 8 × 8,
tatively compares our method to JPEG scheme. The main steps
we periodically repeated the base uniform samples of the size
of JPEG compression are 2-D DCT, quantization, and Huffman
8 × 8 depicted in Fig. 2. However, we utilized the same rule
encoding [41]. Therefore, their computational complexities are
behind HEVC standard by replicating a single entry of an 8 × 8
respectively of the order O(N 4 ) + O(N ) + O(N log2 N ) ≈
matrix to 2 × 2 and 4 × 4 squares of equal values to make fre-
O(N 4 ), where N = h × w denotes the number of image pix-
quency modes and quantization tables of dimensions 16 × 16
els. The decompression phase of JPEG has the same order as the
and 32 × 32, respectively [40]. Then, we performed sensitivity
encoder. For the suggested sampler, dominant term has O(N 4 ),
analysis on three image-sets by changing the input patch size
whereas recoverer exhibits a complexity about O(N log2 N )
(pixel) and then measuring the average PSNR (dB). The re-
due to the convolution operation. This means that the compu-
sults of this experiment are shown in Fig. 17 on 21 well-known
tational complexity of JPEG scheme is totally higher than the
test-images, 20 IR images, and 27 images of Microsoft Object
proposed approach. JPEG acquires an input image at the rate
Recognition Database with their original dimensions. For a fair
of Shannon-Nyquist, whereas the proposed sampler works at
comparison, we tuned the random sampling coefficient c, so that
sampling rates about 15 times of JPEG rate. However, as shown
the sampling rate of different patches is approximately equal.
in Table IV, because of employing quantizer and lossless coding
The relative flatness of curves, especially for the dimension
mechanisms in JPEG compression, our method cannot compete
more than 8 × 8, demonstrates the robustness of the suggested
with JPEG on bit rate and quality. Basically, bit rate reduction is
method to various patch sizes in the sampler. As seen, the opti-
out of the scope of this paper and we do not utilize quantizer and
mum patch dimension is 8 × 8 for all databases. In the figure,
lossless coder blocks in the suggested framework. Instead, the
the high performance gap between IR and visible-light images
main focus is on sample reduction to achieve some advantages
is also due to the sparsity nature of IR test images.
of compressive sensing technology such as sensor-size reduc-
tion [22]. For a fair evaluation in quality, we set both the quality
J. Comparison to JPEG Compression Standard
level of JPEG in (12) and the coefficient c of the proposed ap-
In this experiment, we compare the performance of the proach in (4) at a middle point, i.e. l = η = 45 and c = 1.3 on
proposed sampler/recoverer to baseline JPEG coder/decoder 21 well-known test images of Table I.

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
 324 IEEE TRANSACTIONS ON COMPUTATIONAL IMAGING, VOL. 4, NO. 3, SEPTEMBER 2018

V. CONCLUSION ACKNOWLEDGMENT
This paper suggested a measurement-adaptive sparse image The authors would like to thank Dr Z. Sadeghigol who pro-
sampling and recovery framework called MASCAR. The pro- vided TSCW-GDP-HMT codes for comparisonand also thank
posed sampler adaptively measures the required sampling rate Prof A. Amini, A. Esmaeili, and other researchers in Signal
and accordingly determines sample positions by a combined Processing and Multimedia Lab for their priceless comments.
uniform, random, and nonuniform sampling mechanism. Our
idea originated from the fact that natural images may have low-
pass, bandpass, highpass, or sparse components depending on REFERENCES
the scene under view. Therefore, a measurement-adaptive mech- [1] Y. C. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applica-
anism was advised to trap informative samples. Unlike Gaussian tions. Cambridge, U.K.: Cambridge Univ. Press, 2012.
measurement sensing scenario, the proposed sparse coding style [2] F. Marvasti, Nonuniform Sampling: Theory and Practice. Berlin,
Germany: Springer, 2012.
does not need all samples in advance. [3] J. Romberg, “Imaging via compressive sampling,” IEEE Signal Process.
In the recovery phase, we modeled the problem by a cellular Mag., vol. 25, no. 2, pp. 14–20, Mar. 2008.
automaton machine. The suggested algorithm converges always [4] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol. 52,
no. 4, pp. 1289–1306, Apr. 2006.
at a few generations. Low computational burden and memory [5] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Ex-
usage are two main advantages of this algorithm in compari- act signal reconstruction from highly incomplete frequency information,”
son to sophisticated techniques of ISP [18], TSCW-GDP-HMT IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006.
[6] E. J. Candès, “Compressive sampling,” in Proc. Int. Congr. Mathemati-
[21], and BCS [8], especially in reconstruction of mega-pixel cians, 2006, vol. 3, pp. 1433–1452.
range imaging such as remote sensing. In CA-based recovery [7] F. Marvast et al., “A unified approach to sparse signal processing,”
algorithm, updating rule for reviving dead cells is done based EURASIP J. Adv. Signal Process., vol. 2012, no. 1, pp. 1–45, 2012.
[8] S. Ji, Y. Xue, and L. Carin, “Bayesian compressive sensing,” IEEE Trans.
on a simple weighted averaging. As a future work, more precise Signal Process., vol. 56, no. 6, pp. 2346–2356, Jun. 2008.
predictors can be utilized for this purpose. Also, the suggested [9] B. Shahrasbi and N. Rahnavard, “Model-based nonuniform compressive
sampling/recovery pipeline can be generalized to other sparsify- sampling and recovery of natural images utilizing a wavelet-domain uni-
versal hidden Markov model,” IEEE Trans. Signal Process., vol. 65,
ing transform domains like wavelets. Extensive tests on standard no. 1, pp. 95–104, Jan. 2017.
image data-sets, infra-red, and mega-pixel imaging show the ca- [10] Z. Wang and G. R. Arce, “Variable density compressed image sampling,”
pabilities of proposed technique for practical implementations IEEE Trans. Image Process., vol. 19, no. 1, pp. 264–270, Jan. 2010.
[11] M. L. Malloy and R. D. Nowak, “Near-optimal adaptive compressed sens-
of compressively sampled imaging systems. ing,” IEEE Trans. Inf. Theory, vol. 60, no. 7, pp. 4001–4012, Jul. 2014.
[12] J. Yang, W. E. I. Sha, H. Chao, and Z. Jin, “High-quality image restoration
from partial mixed adaptive-random measurements,” Multimedia Tools
APPENDIX Appl., vol. 75, no. 11, pp. 6189–6205, 2016.
[13] J. A. Tropp and A. C. Gilbert, “Signal recovery from random measure-
Based on JPEG standard [27], the quantization matrix corre- ments via orthogonal matching pursuit,” IEEE Trans. Inf. Theory, vol. 53,
sponding to the texture η can be formulated as no. 12, pp. 4655–4666, Dec. 2007.
[14] T. Blumensath and M. E. Davies, “Normalized iterative hard threshold-
  ing: Guaranteed stability and performance,” IEEE J. Sel. Topics Signal
sη 1 Process., vol. 4, no. 2, pp. 298–309, Apr. 2010.
Qη = max Qr + J, J , (11) [15] F. Marvasti et al., “Sparse signal processing using iterative method with
1≤i,j ≤b 100 2
adaptive thresholding (IMAT),” in Proc. IEEE 19th Int. Conf. Telecom-
mun., 2012, pp. 1–6.
in which the matrix J denotes a b × b all-one matrix. Also, the [16] E. J. Candès and T. Tao, “Near-optimal signal recovery from random
projections: Universal encoding strategies?” IEEE Trans. Inf. Theory, vol.
scaling factor, sη , and the reference quantization table, Qr , are 52, no. 12, pp. 5406–5425, Dec. 2006.
determined by [17] A. I. Zayed and G. Schmeisser, New Perspectives on Approximation and
Sampling Theory. Berlin, Germany: Springer, 2014.
 5000 [18] M. Sadeghi and M. Babaie-Zadeh, “Iterative sparsification-projection:
η , 0+ ≤ η < 50 Fast and robust sparse signal approximation,” IEEE Trans. Signal Process.,
sη = , (12) vol. 64, no. 21, pp. 5536–5548, Nov. 2016.
2(100 − η), 50 ≤ η ≤ 100 [19] H. Mohimani, M. Babaie-Zadeh, and C. Jutten, “A fast approach for
overcomplete sparse decomposition based on smoothed 0 norm,” IEEE
Trans. Signal Process., vol. 57, no. 1, pp. 289–301, Jan. 2009.
and ⎛ ⎞ [20] J. M. Duarte-Carvajalino and G. Sapiro, “Learning to sense sparse signals:
16 11 10 16 24 40 51 61 Simultaneous sensing matrix and sparsifying dictionary optimization,”
⎜ ⎟
⎜ 12 12 14 19 26 58 60 55 ⎟ IEEE Trans. Image Process., vol. 18, no. 7, pp. 1395–1408, Jul. 2009.
⎜ ⎟ [21] Z. Sadeghigol, M. H. Kahaei, and F. Haddadi, “Model based variational
⎜ ⎟
⎜ 14 13 16 24 40 57 69 56 ⎟ Bayesian compressive sensing using heavy tailed sparse prior,” Signal
⎜ ⎟ Process., Image Commun., vol. 41, pp. 158–167, 2016.
⎜ ⎟
⎜ 14 17 22 29 51 87 80 62 ⎟ [22] M. F. Duarte et al., “Single-pixel imaging via compressive sampling,”
⎜ ⎟ IEEE Signal Process. Mag., vol. 25, no. 2, pp. 83–91, Mar. 2008.
Qr = ⎜
⎜ 18
⎟.
⎟ [23] X. Li, “Anisotropic mesh adaptation for image representation,” EURASIP
⎜ 22 37 56 68 109 103 77 ⎟ J. Image Video Process., vol. 2016, no. 1, pp. 1–16, 2016.
⎜ ⎟
⎜ 24 35 55 64 81 104 113 92 ⎟ [24] A. Taimori, F. Razzazi, A. Behrad, A. Ahmadi, and M. Babaie-Zadeh,
⎜ ⎟ “Quantization-unaware double JPEG compression detection,” J. Math.
⎜ ⎟
⎜ 49 64 78 87 103 121 120 101 ⎟ Imag. Vis., vol. 54, no. 3, pp. 269–286, 2016.
⎜ ⎟ [25] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural features
⎝ ⎠ for image classification,” IEEE Trans. Syst., Man, Cybern., vol. SMC-3,
72 92 95 98 112 100 103 99 no. 6, pp. 610–621, Nov. 1973.

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.
TAIMORI AND MARVASTI: ADAPTIVE SPARSE IMAGE SAMPLING AND RECOVERY 325

[26] R. Jain, R. Kasturi, and B. G. Schunck, Machine Vision. New York, NY, Farokh Marvasti (SM’83) received the B.Sc., the
USA: McGraw-Hill, 1995. M.Sc., and the Ph.D. degrees all from Rensselaer
[27] W. Luo, J. Huang, and G. Qiu, “JPEG error analysis and its applications Polytechnic Institute, Troy, NY, USA, in 1970, 1971,
to digital image forensics,” IEEE Trans. Inf. Forensics Security, vol. 5, and 1973, respectively. He has worked, consulted and
no. 3, pp. 480–491, Sep. 2010. taught in various industries and academic institutions
[28] M. Lustig, D. L. Donoho, J. M. Santos, and J. M. Pauly, “Compressed since 1972. Among which are Bell Labs, Univer-
sensing MRI,” IEEE Signal Process. Mag., vol. 25, no. 2, pp. 72–82, Mar. sity of California Davis, Illinois Institute of Technol-
2008. ogy, University of London, and King’s College Lon-
[29] R. Hasimoto-Beltrán, S. Baqai, and A. Khokhar, “Transform domain inter- don. He is currently a Professor with Sharif Univer-
block interleaving schemes for robust image and video transmission in sity of Technology, Tehran, Iran and the Director of
ATM networks,” J. Visual Commun. Image Representation, vol. 15, no. 4, Advanced Communications Research Institute and a
pp. 522–547, 2004. former Head of The Center of Excellence in Multiaccess Communication Sys-
[30] Z.-L. Bai and Q. Huo, “A study on the use of 8-directional features for tems. He spent his sabbatical leave at The Communications and Information
online handwritten chinese character recognition,” in Proc. IEEE 8th Int. Systems Group of University College London in 2013. He has about 100 jour-
Conf. Document Anal. Recognit., 2005, pp. 262–266. nal publications and has written several reference books; he has also several
[31] L.-L. Huang, A. Shimizu, Y. Hagihara, and H. Kobatake, “Gradient fea- international patents. His main research interests are signal and image process-
ture extraction for classification-based face detection,” Pattern Recognit., ing and multimedia systems. He was one of the Editors and Associate Editors of
vol. 36, no. 11, pp. 2501–2511, 2003. the IEEE TRANSACTIONS ON COMMUNICATIONS AND SIGNAL PROCESSING from
[32] J. L. Schiff, Cellular Automata: A Discrete View of the World, vol. 45, 1990–1997. His last book is on Nonuniform Sampling: Theory and Practice by
Hoboken, NJ, USA: Wiley, 2011. Springer in 2001. He was also a Guest Editor on Special Issue on Nonuniform
[33] P. Rosin, A. Adamatzky, and X. Sun, Cellular Automata in Image Pro- Sampling for the Sampling Theory and Signal and Image Processing journal,
cessing and Geometry, vol. 10, Berlin, Germany: Springer, 2014. May 2008. Besides being the co-founders of two international conferences (ICT
[34] Q. Liu, A. H. Sung, and M. Qiao, “Neighboring joint density-based JPEG and SampTA), he has been the organizer and special session Chairs of many
steganalysis,” ACM Trans. Intell. Syst. Technol., vol. 2, no. 2, 2011, Art IEEE conferences including International Conference on Acoustics, Speech,
no. 16. and Signal Processing. Recently, he was the Lead Editor on “Sparse Signal
[35] A. Olmos and F. A. A. Kingdom, “A biologically inspired algorithm Processing” for the Special Issue of EURASIP Journal on Advanced in Signal
for the recovery of shading and reflectance images,” Perception, vol. 33, Processing.
no. 12, pp. 1463–1473, 2004. He is the recipient of distinguished Award from the Academy of Sciences
[36] G. Schaefer and M. Stich, “UCID: An uncompressed color image of Iran, in 2014, and a five year term Chair position from Iran National Science
database,” in Proc.SPIE Storage Retrieval Methods Appl. Multimedia, foundation, in 2015.
vol. 5307, pp. 472–480, 2004.
[37] A. Taimori, F. Razzazi, A. Behrad, A. Ahmadi, and M. Babaie-Zadeh,
“A novel forensic image analysis tool for discovering double JPEG com-
pression clues,” Multimedia Tools Appl., vol. 76, no. 6, pp. 7749–7783,
2017.
[38] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality
assessment: From error visibility to structural similarity,” IEEE Trans.
Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[39] H. Chen et al., “Infrared camera using a single nano-photodetector,” IEEE
Sensors J., vol. 13, no. 3, pp. 949–958, Mar. 2013.
[40] M. Budagavi, High Efficiency Video Coding (HEVC): Algorithms and
Architectures. Berlin, Germany: Springer, 2016.
[41] P. T. Chiou, Y. Sun, and G. S. Young, “A complexity analysis of the JPEG
image compression algorithm,” in Proc. IEEE 9th Comput. Sci. Electron.
Eng., 2017, pp. 65–70.

Ali Taimori received the associate of arts degree in

electronics from Technical and Vocational Univer-
sity, Kermanshah, Iran, in 2000, the B.Sc. degree
in electrical engineering from Arak Branch, Islamic
Azad University, Arak, Iran, in 2004, the M.Sc. de-
gree from Shahed University, Tehran, Iran, in 2008,
and the Ph.D. degree from Science and Research
Branch, Islamic Azad University, Tehran, Iran, in
2015, all in electrical engineering. He has graduated
with the first, the second, and the first ranks in the
B.Sc., the M.Sc., and the Ph.D. degrees, respectively.
He is currently a Postdoctoral Researcher with the Department of Elec-
trical Engineering, Sharif University of Technology, Tehran, Iran. His current
research interests include information forensics and security, signal processing
for intelligent and communication systems, pattern analysis and recognition,
and machine vision.

Authorized licensed use limited to: Universitatea Tehnica din Cluj-Napoca. Downloaded on September 27,2023 at 09:33:40 UTC from IEEE Xplore. Restrictions apply.