MIT Media Lab

Computational Photography: Advanced Topics

Camera Culture
Ramesh Raskar
Paul Debevec Jack Tumblin

Speaker: Jack Tumblin

Associate Professor of Computer Science at Northwestern Univ.

His Look Lab group pursues research on new methods to capture and manipulate the appearance of objects and surroundings, in the hope that hybrid optical/computer methods may give us new ways to see, explore, and interact with objects and people anywhere in the world. During his doctoral studies at Georgia Tech and post-doc at Cornell, he investigated tone-mapping methods to depict highcontrast scenes. His MS in Electrical Engineering (December 1990) and BSEE (1978), also from Georgia Tech, bracketed his work as co-founder of IVEX Corp., (>45 people as of 1990) where his flight simulator design work was granted 5 US Patents. He was an Associate Editor of ACM Transactions on Graphics (2000-2006), a member of the SIGGRAPH Papers Committee (2003, 2004), and in 2001 was a Guest Editor of IEEE Computer Graphics and Applications.

http://www.cs.northwestern.edu/~jet

Speaker: Paul Debevec

Research Associate Professor , University of Southern California and the Associate director of Graphics Research, USC's Institute for Creative Technologies.

Debevec's Ph.D. thesis (UC Berkeley, 1996) presented Faade, an image-based modeling and rendering system for creating photoreal architectural models from photographs. Pioneer in high dynamic range photography, he demonstrated new imagebased lighting techniques in his films Rendering with Natural Light (1998), Fiat Lux (1999), and The Parthenon (2004); he also led the design of HDR Shop, the first high dynamic range image editing program. At USC ICT, Debevec has led the development of a series of Light Stage devices used in Spider Man 2 and Superman Returns. He is the recipient of ACM SIGGRAPH's first Significant New Researcher Award and a co-author of the 2005 book High Dynamic Range Imaging from Morgan Kaufmann.

http://www.debevec.org

Speaker: Ramesh Raskar

Associate Professor, MIT Media Lab. Previously at MERL as a Senior Research Scientist. His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002 through 2005. He was the panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May 2005 and taught a graduate level class on Computational Photography at Northeastern University, Fall 2005. He is a member of the ACM and IEEE. http://raskar.info http://www.media.mit.edu/~raskar

Overview
Unlocking Photography
Not about the equipment but about the goal Capturing machine readable visual experience Goes beyond what you can see through the viewfinder Push the envelope with seemingly peripheral techniques and advances

Think beyond post-capture image processing

Computation well before image processing and editing Learn how to build your own camera-toys

Emphasis
Most recent work in graphics/vision (2006 and later) Research in other fields: Applied optics, novel sensors, materials Review of 50+ recent papers and projects

What we will not cover

Minimum discussion of graphics/vision papers before 2006 Epsilon photography (improving camera performance by bracketing) Film Cameras, Novel view rendering (IBR), Color issues, Traditional image processing/editing

Traditional Photography
Detector
Lens

Pixels

Image
Courtesy: Shree Nayar

Traditional Photography
Detector
Lens

Pixels

Image

Mimics Human Eye for a Single Snapshot: Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world

Computational Photography:
Optics, Sensors and Computations
Generalized Sensor

Computations
Ray Reconstruction Upto 4D Ray Sampler

Generalized Optics
4D Ray Bender

Picture Merged braketed photos, Coded sensing

Computational Photography
Novel Cameras
Generalized

Sensor
Generalized

Processing

Optics

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources

Sensor
Generalized

Processing

Optics

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources

Sensor
Generalized

Processing

Optics

Scene: 8D Ray Modulator

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources

Sensor
Generalized

Processing

Optics

Display

Recreate 4D Lightfield

Scene: 8D Ray Modulator

Computational Photography
Novel Cameras
Generalized

Novel Illumination
Light Sources Modulators Generalized Optics

Sensor
Generalized

Processing
Ray Reconstruction

Optics
4D Ray Bender

4D Incident Lighting

Upto 4D Ray Sampler

4D Light Field

Display

Recreate 4D Lightfield

Scene: 8D Ray Modulator

What is Computational Photography?

Create photo that could not have been taken by a traditional Camera (?)

Goal: Record a richer, multi-layered visual experience

1. Overcome limitations of todays cameras 2. Support better post-capture processing
Relightable photos, Focus/Depth of field, Fg/Bg, Shape boundaries

3. Enables new classes of recording the visual signal

Moment [Cohen05], Time-lapse, Unwrap mosaics, Cut-views

4. Synthesize impossible photos

Wrap-around views [Rademacher and Bishop 1998]), fusion of time-lapsed events [Raskar et al 2004], motion magnification [Liu et al 2005]), video textures and panoramas [Agarwala et al 2005].

5. Exploit previously exotic forms of scientific imaging

Coded aperture [Veeraraghavan 2007, Levin 2007], confocal imaging [Levoy 2004], tomography [Trifonov 2006]

Computational Photography
1. Epsilon Photography
Low-level vision: Pixels Multi-photos by perturbing camera parameters HDR, panorama, Ultimate camera

2. Coded Photography

Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Scene analysis : (Consumer software?)

3. Essence Photography

Beyond single view/illum Not mimic human eye New art form

Epsilon Photography
Dynamic range
Exposure bracketing
[Mann-Picard, Debevec]

Wider FoV
Stitching a panorama

Depth of field
Fusion of photos with limited DoF
[Agrawala04]

Noise
Flash/no-flash image pairs

Frame rate
Triggering multiple cameras
[Wilburn04]

Dynamic Range
Short Exposure

Goal: High Dynamic Range

Long Exposure

Epsilon Photography
Dynamic range
Exposure braketing
[Mann-Picard, Debevec]

Wider FoV
Stitching a panorama

Depth of field
Fusion of photos with limited DoF
[Agrawala04]

Noise
Flash/no-flash image pairs
[Petschnigg04, Eisemann04]

Frame rate
Triggering multiple cameras
[Wilburn05, Shechtman02]

Computational Photography
1.

Epsilon Photography

Low-level Vision: Pixels Multiphotos by perturbing camera parameters HDR, panorama Ultimate camera

2.

Coded Photography

Mid-Level Cues:

Single/few snapshot

Regions, Edges, Motion, Direct/global Reversible encoding of data

Additional sensors/optics/illum Scene analysis Not mimic human eye Beyond single view/illum New artform

3.

Essence Photography

 3D
Stereo of multiple cameras

Higher dimensional LF
Light Field Capture
lenslet array
[Adelson92, Ng05], [Veeraraghavan07]

3D lens

[Georgiev05],

heterodyne masks

Boundaries and Regions

Multi-flash camera with shadows Fg/bg matting [Chuang01,Sun06]
[Raskar08]

Deblurring
Engineered PSF Motion: Flutter shutter[Raskar06], Camera Motion [Levin08] Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding
[Cathey95]

Global vs direct illumination

High frequency illumination [Nayar06] Glare decomposition [Talvala07, Raskar08]

Coded Sensor
Gradient camera
[Tumblin05]

Digital Refocusing using Light Field Camera

125 square-sided microlenses

[Ng et al 2005]
Marc Levoy

 3D
Stereo of multiple cameras

Higher dimensional LF
Light Field Capture
lenslet array
[Adelson92, Ng05], [Veeraraghavan07]

3D lens

[Georgiev05],

heterodyne masks

Boundaries and Regions

Multi-flash camera with shadows Fg/bg matting [Chuang01,Sun06]
[Raskar08]

Deblurring
Engineered PSF Motion: Flutter shutter[Raskar06], Camera Motion [Levin08] Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding
[Cathey95]

Global vs direct illumination

High frequency illumination [Nayar06] Glare decomposition [Talvala07, Raskar08]

Coded Sensor
Gradient camera
[Tumblin05]

Left

Top

Right

Bottom

Depth Edges

Canny Edges

Depth Edges

 3D
Stereo of multiple cameras

Higher dimensional LF
Light Field Capture
lenslet array
[Adelson92, Ng05], [Veeraraghavan07]

3D lens

[Georgiev05],

heterodyne masks

Boundaries and Regions

Multi-flash camera with shadows Fg/bg matting [Chuang01,Sun06]
[Raskar08]

Deblurring
Engineered PSF Motion: Flutter shutter[Raskar06], Camera Motion [Levin08] Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding
[Cathey95]

Global vs direct illumination

High frequency illumination [Nayar06] Glare decomposition [Talvala07, Raskar08]

Coded Sensor
Gradient camera
[Tumblin05]

Flutter Shutter Camera

Raskar, Agrawal, Tumblin [Siggraph2006]

LCD opacity switched in coded sequence

Traditio nal

Coded Exposu re

Deblurred Image

Deblurred Image

Image of Static Object

 3D
Stereo of multiple cameras

Higher dimensional LF
Light Field Capture
lenslet array
[Adelson92, Ng05], [Veeraraghavan07]

3D lens

[Georgiev05],

heterodyne masks

Boundaries and Regions

Multi-flash camera with shadows Fg/bg matting [Chuang01,Sun06]
[Raskar08]

Deblurring
Engineered PSF Motion: Flutter shutter[Raskar06], Camera Motion [Levin08] Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding
[Cathey95]

Decomposition Problems
High frequency illumination, Global/direct illumination Glare decomposition [Talvala07, Raskar08]
[Nayar06]

Coded Sensor
Gradient camera
[Tumblin05]

"Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination," S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar, ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH), Jul, 2006.

Separating Reflectance Components with Polarization-Difference Imaging

cross-polarized subsurface component

normal image

polarization difference (primarily) specular component

Computational Photography
1. Epsilon Photography Multiphotos by varying camera parameters HDR, panorama Ultimate camera: (Photo-editor) Coded Photography Single/few snapshot Reversible encoding of data Additional sensors/optics/illum Scene analysis : (Next software?)

2.

3.

Essence Photography High-level understanding

New artform

Not mimic human eye Beyond single view/illum

Blind Camera

Sascha Pohflepp, U of the Art, Berlin, 2006

Capturing the Essence of Visual Experience

Exploiting online collections
Photo-tourism [Snavely2006] Scene Completion [Hays2007]

Multi-perspective Images
Multi-linear Perspective [Jingyi Yu, McMillan 2004] Unwrap Mosaics [Rav-Acha et al 2008] Video texture panoramas [Agrawal et al 2005]

Non-photorealistic synthesis
Motion magnification [Liu05]

Image Priors
Learned features and natural statistics Face Swapping: [Bitouk et al 2008] Data-driven enhancement of facial attractiveness [Leyvand et al 2008] Deblurring [Fergus et al 2006, 2008 papers

Scene Completion Using Millions of Photographs Hays and Efros, Siggraph 2007

Capturing the Essence of Visual Experience

Exploiting online collections
Photo-tourism [Snavely2006] Scene Completion [Hays2007]

Multi-perspective Images
Multi-linear Perspective [Jingyi Yu, McMillan 2004] Unwrap Mosaics [Rav-Acha et al 2008] Video texture panoramas [Agrawal et al 2005]

Non-photorealistic synthesis
Motion magnification [Liu05]

Image Priors
Learned features and natural statistics Face Swapping: [Bitouk et al 2008] Data-driven enhancement of facial attractiveness [Leyvand et al 2008] Deblurring [Fergus et al 2006, 2008 papers

Andrew Davidhazy

Unwrap Mosaics + Video Editing

Rav-Acha et al Siggraph 2008

Capturing the Essence of Visual Experience

Exploiting online collections
Photo-tourism [Snavely2006] Scene Completion [Hays2007]

Multi-perspective Images
Multi-linear Perspective [Jingyi Yu, McMillan 2004] Unwrap Mosaics [Rav-Acha et al 2008] Video texture panoramas [Agrawal et al 2005]

Non-photorealistic synthesis
Motion magnification [Liu05]

Image Priors
Learned features and natural statistics Face Swapping: [Bitouk et al 2008] Data-driven enhancement of facial attractiveness [Leyvand et al 2008] Deblurring [Fergus et al 2006, 2008 papers

Motion Magnification

Liu, Torralba, Freeman, Durand, Adelson Siggraph 2005

Motion Magnification

Liu, Torralba, Freeman, Durand, Adelson Siggraph 2005

Motion Magnification

Liu, Torralba, Freeman, Durand, Adelson Siggraph 2005

Capturing the Essence of Visual Experience

Exploiting online collections
Photo-tourism [Snavely2006] Scene Completion [Hays2007]

Multi-perspective Images
Multi-linear Perspective [Jingyi Yu, McMillan 2004] Unwrap Mosaics [Rav-Acha et al 2008] Video texture panoramas [Agrawal et al 2005]

Non-photorealistic synthesis
Motion magnification [Liu05]

Image Priors
Learned features and natural statistics Face Swapping: [Bitouk et al 2008] Data-driven enhancement of facial attractiveness [Leyvand et al 2008] Deblurring [Fergus et al 2006, 2007-2008 papers]

Face Swapping
Find Candidate face in DB and align Tune pose, lighting, color and blend Keep result with optimized matching cost

[Bitouk et al 2008]

Computational Photography
1. Epsilon Photography Low-level vision: Pixels Multi-photos by perturbing camera parameters HDR, panorama, Ultimate camera

2.

Coded Photography

Mid-Level Cues:

Single/few snapshot

Regions, Edges, Motion, Direct/global

Reversible encoding of data

Additional sensors/optics/illum Scene analysis

3.

Essence Photography High-level understanding

New artform

Not mimic human eye Beyond single view/illum

Submit your questions ..

1. Today What makes photography hard? What moments you are not able to capture?

2. Future
What do you expect in a camera or photo-software you buy in 2020?

Please submit by break at 3:30pm Panel Discussion at 5:10pm

Siggraph 2006 16 Computational Photography Papers

Hybrid Images Oliva et al (MIT) Coded Exposure Photography: Motion Deblurring Raskar et al (MERL) Photo Tourism: Exploring Photo Collections in 3D Snavely et al (Washington) AutoCollage Rother et al (Microsoft Research Cambridge) Photographing Long Scenes With Multi-Viewpoint Panoramas Agarwala et al (University of Washington) Projection Defocus Analysis for Scene Capture and Image Display Zhang et al (Columbia University) Multiview Radial Catadioptric Imaging for Scene Capture Kuthirummal et al (Columbia University) Light Field Microscopy (Project) Levoy et al (Stanford University) Fast Separation of Direct and Global Components of a Scene Using High Frequency Illumination Nayar et al (Columbia University)

Drag-and-Drop Pasting Jia et al (MSRA)

Two-scale Tone Management for Photographic Look Bae et al (MIT)

Interactive Local Adjustment of Tonal Values Lischinski et al (Tel Aviv)

Image-Based Material Editing Khan et al (Florida) Flash Matting Sun et al (Microsoft Research Asia) Natural Video Matting using Camera Arrays Joshi et al (UCSD / MERL) Removing Camera Shake From a Single Photograph Fergus (MIT)

Siggraph 2007 19 Computational Photography Papers

Image Analysis & Enhancement

Image Deblurring with Blurred/Noisy Image Pairs Photo Clip Art Scene Completion Using Millions of Photographs Soft Scissors: An Interactive Tool for Realtime High Quality Matting Seam Carving for Content-Aware Image Resizing Image Vectorization Using Optimized Gradient Meshes Detail-Preserving Shape Deformation in Image Editing Veiling Glare in High-Dynamic-Range Imaging Ldr2Hdr: On-the-Fly Reverse Tone Mapping of Legacy Video and Photographs Multiscale Shape and Detail Enhancement from Multi-light Image Collections Active Refocusing of Images and Videos Multi-Aperture Photography Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing Image and Depth from a Conventional Camera with a Coded Aperture Capturing and Viewing Gigapixel Images Efficient Gradient-Domain Compositing Using Quadtrees Factored Time-Lapse Video Computational Time-Lapse Video (project page) Real-Time Edge-Aware Image Processing With the Bilateral Grid

Image Slicing & Stretching

Light Field & High-Dynamic-Range Imaging

Appearance Capture & Editing

Big Images

Computational Cameras

Video Processing

Siggraph 2008 19 Computational Photography Papers

Computational Photography & Display
Programmable Aperture Photography: Multiplexed Light Field Acquisition Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses Light-Field Transfer: Global Illumination Between Real and Synthetic Objects
Motion Invariant Photography Single Image Dehazing High-Quality Motion Deblurring From a Single Image Progressive Inter-scale and intra-scale Non-blind Image Deconvolution

Deblurring & Dehazing

Faces & Reflectance

Data-driven enhancement of facial attractiveness Face Swapping: Automatic Face Replacement in Photographs (Project) AppProp: All-Pairs Appearance-Space Edit Propagation
Factoring Repeated Content Within and Among Images Finding Paths through the World's Photos Improved Seam Carving for Video Retargeting (Project) Unwrap Mosaics: A new representation for video editing (Project) A Perceptually Validated Model for Surface Depth Hallucination A Perception-based Color Space for Illumination-invariant Image Processing Self-Animating Images: Illusory Motion Using Repeated Asymmetric Patterns Edge-preserving decompositions for multi-scale tone and detail manipulation Light Mixture Estimation for Spatially Varying White Balance

Image Collections & Video

Perception & Hallucination

Tone & Color

 Ramesh Raskar and Jack Tumblin

Book Publishers: A K Peters Siggraph 2008 booth: 20% off Booth #821

 Articles
IEEE Computer, IEEE CG&A,

More ..

August 2006 Special Issue Bimber, Nayar, Levoy, Debevec, Cohen/Szeliski March 2007 Special issue Durand and Szeliski

Science News cover story

April 2007 Featuring : Levoy, Nayar, Georgiev, Debevec

American Scientist
February 2008

Siggraph 2008
19 papers HDRI, Mon/Tue 8:30am Principles of Appearance Acquisition and Representation Bilateral Filter course, Fri 8:30am Other courses .. (Citizen Journalism, Wedn 1:45pm)

First International Conf on Comp Photo, April 2009

Athale, Durand, Nayar (Papers due Oct 3nd)

Class: Computational Photography, Advanced Topics

Debevec, Raskar and Tumblin
Module 1: 105 minutes 1:45: A.1 Introduction and Overview 2:00: A.2 Concepts in Computational Photography 2:15: A.3 Optics: Computable Extensions 2:45: A.4 Sensor Innovations 3:15: Q & A 3:30: Break: 15 minutes Module 2: 105 minutes 3:45: B.1 Illumination As Computing (Debevec, 25 minutes) (Raskar, 15 minutes) (Tumblin, 15 minutes) (Raskar, 30 minutes) (Tumblin, 30 minutes) (15 minutes)

4:10: B.2 Scene and Performance Capture

4:30: B.3 Image Aggregation & Sensible Extensions 4:50: B.4 Community and Social Impact 5:10: B.4 Panel discussion

(Debevec, 20 minutes)
(Tumblin, 20 minutes) (Raskar, 20 minutes) (All, 20 minutes)

Class Page : http://ComputationalPhotography.org