FPA-CS: Focal Plane Array-based Compressive Imaging in Short-wave Infrared

Huaijin Chen,1 M. Salman Asif,1 Aswin C. Sankaranarayanan,2 and Ashok Veeraraghavan1

1 Department of Electrical and Computer Engineering, Rice University.
2 Department of Electrical and Computer Engineering, Carnegie Mellon University.

64 x 64
SWIR
sensor array

compressive low-res
scene frames from 64x64
DMD sensor
(as seen in a visible camera)

Total variation-based
reconstruction

Figure 2: Selected frames from reconstructed SWIR videos. Each frame in

reconstructed SWIR
mega-pixel image the moving car videos is reconstructed using 16 captured images; compres-
sion factor = 16, and a consequently 32-fps frame rate. Each frame in the
Figure 1: Focal plane array-based compressive sensing (FPA-CS) camera moving hand videos is reconstructed using 22 captured images; compres-
architecture: A 64 64 SWIR sensor array is equivalent to 4096 single sion factor = 11.6, and a consequently 21.8-fps frame rate. Both videos
pixel cameras (SPCs) operating in parallel. This results in vastly superior are reconstructed using 3D-TV prior. XT and YT slices for both videos are
spatio-temporal resolutions against what is achievable using the SPC or a shown to the right of the images.
traditional camera.

Cameras for imaging in short and mid-wave infrared spectra are signif- Natural images have been shown to have sparse gradients. We can view
icantly more expensive than their counterparts for visible imaging. For ex- a video signal as a 3D object that consists of a sequence of 2D images,
ample, a cellphone camera with a several megapixel sensor costs a few dol- and we expect pixels in each image to be similar to their neighbors along
lars, but a megapixel sensor for short-wave infrared (SWIR) imaging costs horizontal, vertical, and temporal directions. To exploit the spatio-temporal
tens of thousands dollars. As a result, high-resolution imaging beyond the similarity in a video signal, we can use priors for sparse spatio-temporal
visible spectrum remains out of reach for many consumers. gradients, and solve an optimization problem of the following form for
Over the last decade, compressive sensing (CS) [1] has emerged as reconstruction[4]:
a useuful technology for designing high-resolution imaging systems us-
ing low-resolution sensors. For instance, a single-pixel camera (SPC) uses x = arg min TV3D (x) subject to ky Axk2 ,
(TV) b
x
a single-pixel detector and a digital micromirror device (DMD) to record
coded measurements of a high-resolution image [3]. A computational re- where the term TV3D (x) refers to the 3D total-variation of x. TV3D can be
construction algorithm is then used to recover the high-resolution image defined as
from the coded measurements. Unfortunately, the measurement rate of an q
SPC is insufficient for imaging at high spatial and temporal resolutions [5]. TV3D (x) = (Du x(i))2 + (Dv x(i))2 + (Dt x(i))2 ,
In this paper, we present a focal plane array-based compressive sensing i
(FPA-CS) architecture that achieves high spatial and temporal resolutions
where Du x and Dv x are the spatial gradients along horizontal and vertical
using inexpensive, low-resolution sensors. Our proposed architecture can
dimensions of x, respectively, and Dt x represents gradient along the tempo-
be viewed as an array of SPCs working in parallel, thereby increasing the
ral dimension of x. We present some of our experimental results in Figure 2,
measurement rate, and consequently, the achievable spatio-temporal resolu-
where we used MFISTA [2] for the reconstruction of videos.
tion of CS-based cameras. We develop a proof-of-concept prototype SWIR
FPA-CS provides three advantages over conventional imaging. First,
video camera using a low-resolution sensor with 64 64 pixels; the proto-
our CS-inspired FPA-CS system provides an inexpensive alternative to achieve
type provides a 4096 increase in measurement rate compared to the SPC,
SWIR imaging in high spatiotemporal resolution . Second, compared to tra-
and for the first time, achieves megapixel resolution at video rate using CS
ditional single-pixel-based compressive cameras, FPA-CS simultaneously
techniques.
records data from 4096 parallel, compressive systems, thereby significantly
Our prototype FPA-CS camera is constructed using a low-resolution improves the measurement rate. As a consequence, the achieved spatio-
sensor array of 64 64 pixels, each observing a 16 16 patch of micromir- temporal resolution of our device is an order of magnitude better than the
rors. The DMD patterns and sensor readout timings are synchronized to SPC.
record modulated, low-resolution images at a frame rate Fs = 480 fps. The
sensor image at time t can be described as yt = At xt , where yt is a vec- [1] Richard Baraniuk. Compressive sensing. IEEE signal processing magazine, 24
tor with 4096 measurements, xt represents the high-resolution image at the (4), 2007.
DMD plane, and the matrix At encodes modulation of xt with the DMD [2] A. Beck and M. Teboulle. Fast gradient-based algorithms for constrained total
pattern and mapping onto the SWIR sensor pixels. To reconstruct video variation image denoising and deblurring problems. IEEE Transactions on Image
Processing, 18(11):24192434, 2009.
at a desired frame-rate, say Fr fps, we divide low-resolution sensor im-
[3] M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and
ages into sets of T = Fs /Fr measurements, all of which correspond to the R. G. Baraniuk. Single-pixel imaging via compressive sampling. IEEE Signal
same high-resolution image. Suppose the kth set correspond to yt = At xt for Processing Magazine, 25(2):8391, Mar. 2008.
t = (k 1)T +1, . . . , kT ; we assume that xt = xk and stack all the yt and At in [4] S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin. An iterative regularization
the kth set in yk and Ak , respectively. Our goal is to reconstruct the xk from method for total variation-based image restoration. Multiscale Modeling and
the noisy and possibly under-determined sets of linear equations yk = Ak xk . Simulation, 4(2):460489, 2005.
[5] A. C. Sankaranarayanan, C. Studer, and R. G. Baraniuk. CS-MUVI: Video com-
This is an extended abstract. The full paper is available at the Computer Vision Foundation pressive sensing for spatial-multiplexing cameras. In IEEE International Con-
webpage. ference on Computational Photography, 2012.