CS 484/684

Computational Vision

Image Modalities
most slides are shamelessly stolen from
Steven Seitz, Aleosha Efros, and Terry Peters
CS 484/684
Computational Vision

Photo/Video data
• Lenses
• Pin-hole camera model – the basics
• Digital images and volumes
Medical Images and Volumes
• X-ray, MRI, CT, and Ultrasound

Extra Reading: Szeliski, Ch. 2,

Gonzalez & Woods, Ch. 1
Slide by Steve Seitz

How do we see the world?

Let’s design a camera

• Idea 1: put a piece of film in front of an object
• Do we get a reasonable image?
Slide by Steve Seitz

Pinhole camera

Add a barrier to block off most of the rays

• This reduces blurring
• The opening known as the aperture
• How does this transform the image?
Slide by Steve Seitz

Camera Obscura (a.k.a. darkroom)

The first camera

• Known to Aristotle
• Depth of the room is the focal length
• Pencil of rays – all rays through a point

3D world is projected on an image plane (2D)

Can we restore 3D points from their image?
• That is, can we measure distances (the lost 3rd dimension)?
Figure by David Forsyth

Depth is ambiguous
as distant objects appear smaller
e.g. compare the image of A and C

Eventually, we will learn methods for restoring depth

(prior knowledge about 3D world, geometric constraints, learning)
Slide by Aleosha Efros

How to record an image

Drawing from “The Great Art of Light and Shadow “

Jesuit Athanasius Kircher, 1646.

Camera Obscura
Slide by Aleosha Efros

Home-made pinhole camera

electronic CCD or CMOS

image sensors

http://www.debevec.org/Pinhole/

How does the aperture size affect the image?

Slide by Steve Seitz

Shrinking the aperture

Why not make the aperture as small as possible?

• Less light gets through
• Diffraction effects…
Slide by Steve Seitz

Shrinking the aperture

Slide by Steve Seitz

The reason for lenses

?
Slide by Steve Seitz

Adding a lens

“circle of
confusion”
object distance (depth) image distance

A lens focuses light onto the film

• There is a specific depth at which objects are “in focus”
– other points project to a “circle of confusion” in the image
• Changing image distance changes this depth (see a problem in HW0)
NOTE: depth-from-focus - estimate “depth map” by finding “sharp” image
regions while changing either image distance or lens’ focal length.
Slide by Steve Seitz

Adding a lens

more generally,
image plane

infinite (large enough) distance focal length

Lens’ focal length is image distance where objects at

infinity appear in focus.
• Focal length depends on lens’ construction (e.g. surface radius). Some lenses may
allow changing their focal length (typically, these are multi-lens constructions).
• To focus on closer objects, image distance should differ from lens’ focal length
(see a problem in HW0)
Basic lens camera
image
plane lens

point
far away
(at infinity)
assuming image assuming object
distance equals distance is
focal length f infinity
Basic lens camera
image
plane lens

the ray through lens center

is enough to find projected point

point
far away
(at infinity)
focal
assuming object
length f
distance is
infinity
Basic lens camera
image
plane lens

point
far away
(at infinity)
focal
assuming object
length f
distance is
infinity
Basic lens camera
image
plane lens

point
far away
(at infinity)
focal
assuming object
length f
distance is
infinity
Basic lens camera
image
plane lens

focal
assuming object
length f
distance is
infinity
 Basic lens camera
image
plane lens

focal
assuming object
length f
distance is
infinity
NOTE: - Rays from closer 3D points converge at a “circle of confusion” if image distance is lens’ focal length.
- In-focus points can be adjusted by changing image distance.
- We use simplified “pin hole camera model” ignoring “out-of-focus” issues assuming 3D points are far enough.
 Basic camera model: “pin hole”
image
plane lens
simplified
“pin hole” model

NOTE:
for pin hole camera model “focal length” (f) is defined as image distance (to the “hole”).
As mentioned earlier, focal length of a lens does not have to be equal to the image distance (to the lens).
Basic camera model: “pin hole”
image “equivalent”
plane lens image plane
simplified
“pin hole” model

f f
 Basic camera model: “pin hole”
(virtual)
lens image plane
simplified
“pin hole” model

camera’s
“optical center” C
or
“view point”

Simplified camera representation: image plane is drawn in front of the optical center.
We will use such “pin hole” camera model later in the course.
 Simple example of
projective geometry (from 3D point to 2D pixel)

Consider a simple example of so-called

camera-centered 3D world coordinate system (x,y,z):

for world points (3D)

y
v
camera-centered
z
world coordinate f for image pixels locations (2D)
system
u
C image
x coordinate Simplifying Assumptions
system (more general case in Topic 7)

- world coordinate system center (0,0,0) is at optical center C

- x -y plane is parallel to the image plane
- x and y axis parallel to u and v axis of the image coordinate system
- axis z (called optical axis) intersects image at its coordinate center (0,0)
 Simple example of
projective geometry (from 3D point to 2D pixel)

First, consider arbitrary world point

inside x-z plane u x
c f

y
v
camera-centered
z
world coordinate f (u,0) u v
system
(x,0,z)
u image-based
C image
coordinates
x of the projection
coordinate
point
system

It projects onto some image point/pixel (u,0) on axis u

(by construction, intersection of x-z plane with the image plane is axis u )
 Simple example of
projective geometry (from 3D point to 2D pixel)

For a general point (x,y,z) in 3D

y
v
camera-centered
z
world coordinate f (x,y,z) u v
(u,v)
system
u image-based
C image
coordinates
x of the projection
coordinate
point
system

Simple observation: size of any 3D object image is inversely proportional to

object’s distance from the camera (z-coordinate value)
We will further study projective camera models in topics 5 and 7
Slide by Aleosha Efros

The eye

The human eye is a camera!

• Iris - colored annulus with radial muscles
• Pupil - the hole (aperture) whose size is controlled by the iris
• What’s the “film”?
– photoreceptor cells (rods and cones) in the retina
Slide by Aleosha Efros

Cameras
Widely available
Figure by Gonzalez & Woods

Digital Image Formation

f(x,y) = reflectance(x,y) * illumination(x,y)

Reflectance in [0,1], illumination in [0,inf]
Figure by Gonzalez & Woods

Sampling and Quantization

intensity (color) depth

a.k.a. bit depth

spatial resolution
(number of pixels, pixel size)
Figure by Gonzalez & Woods

Sampling and Quantization

Slide by Aleosha Efros

What is an image?

We can think of an image as a function f(x,y)

from R2 to R:
• f( x, y ) gives the intensity at position ( x, y )
• Realistically, we expect the image only to be defined
over a rectangle, with a finite range:
– f : [a,b]x[c,d] → [0,1]
A color image is just three functions pasted
together. We can write this as a “vector-valued”
 r ( x, y ) 
function: f ( x, y ) =  g ( x, y ) 
 

 b ( x, y ) 

Slide by Aleosha Efros

Images as functions
Slide by Aleosha Efros

What is a digital image?

We usually operate on digital (discrete) images:
• Sample the 2D space (XY) on a regular grid
• Quantize each intensity sample (round to nearest integer)
If our samples are D apart, we can write this as:
f [i ,j] = Quantize{ f (i D, j D) }
The image can be represented as a matrix (2D array) of integers
quantized
intensities
Image Data “Tensors”
(multidimensional arrays)
We usually operate on digital (discrete) images:
• Sample the 2D space (XY) on a regular grid
• Quantize each intensity sample (round to nearest integer)
If our samples are D apart, we can write this as:
f [i ,j] = Quantize{ f (i D, j D) }
Color image can be stored as a 3D array (tensor) of integers

RGB
Image Data “Tensors”
(multidimensional arrays)
Video - 3D = X*Y*Time
Medical volumetric data (MRI, CT) - 3D = X*Y*Z

Combine multiple images (slices) into a volume

CS 484/684 Computational Vision

Image Modalities

PART II:
Medical images and volumes
• X-ray
• CT
• MRI
• Ultrasound
Slides from Terry Peters

In the beginning…..X-rays

Discovered in 1895
Mainstay of medical imaging till 1970’s
1971 – Computed Tomography (CAT, CT) scanning
1978 - Digital Radiography
………
1980 Magnetic Resonance Imaging
X-rays
Wilhelm Conrad Röntgen (1845-1923)
Nobel Prise in Physics, 1901

• “X” stands for “unknown”

• X-ray imaging is also known as
- radiograph
- Röntgen imaging
X-rays

Bertha Röntgen’s Hand 8 Nov, 1895 A modern radiograph of a hand

• Calcium in bones absorbs X-rays the most

• Fat and other soft tissues absorb less, and look gray
• Air absorbs the least, so lungs look black on a radiograph
X-rays
2D “projection” imaging 1895 - 1970’s
 From Projection Imaging
Towards True 3D Imaging
X-ray imaging
1895

Mathematical results: Development of CT

Radon transformation (computed tomography)
1917 1972
•Image reconstruction from
projection
Computers can perform •Also known as CAT
complex mathematics to (Computerized Axial Tomography)
reconstruct and process images •"tomos" means "slice" (Greek)

Late 1960’s:
Radon Transformation

•Mathematical transformation (related to Fourier)

•Reconstruction of the shape of object (distribution f(x,y)) from
the multitude of 2D projections g ( s,  )
Figure from www.imaginis.com/ct-scan/how_ct.asp

CT imaging
CT imaging, inventing (1972)
Sir Godfrey Hounsfield
Engineer for EMI PLC
1972
Nobel Prize 1979 (with
Alan Cormack)
CT imaging, availability (since 1975)
25 years later

1974

Original axial CT image from the Axial CT image of a normal brain using a
dedicated Siretom CT scanner circa 1975. state-of-the-art CT system and a 512 x
The EMI-Scanner
This image is a coarse 128 x 128 matrix; 512 matrix image. Note the two black
however, in 1975 physicians were "pea-shaped" ventricles in the middle of
fascinated by the ability to see the soft the brain and the subtle delineation of
tissue structures of the brain, gray and white matter
including the black ventricles for the first (Courtesy: Siemens)
time (enlarged in this patient)
Slides from Terry Peters

Clinical Acceptance of CT!?

Dr James Ambrose 1972
• Radiologist, Atkinson -
Morley’s Hospital London
• Recognised potential of EMI-
scanner

“Pretty pictures, but they

will never replace
radiographs” –
Neuroradiologist 1972
Slides from Terry Peters

Then ……………and Now

80 x 80 image 512 x 512 image
3 mm pixels <1mm slice thickness
<0.5mm pixels
13 mm thick slices
0.5 sec rotation
Two simultaneous
0.5 sec recon per slice
slices!!!
Isotropic resolution
80 sec scan time per
Spiral scanning - up to 16
slice
slices simultaneously
80 sec recon time
Slides from Terry Peters

30 Years of CT
Slides from Terry Peters

Birth of MRI

Paul Lautebur 1975

• Presented at Stanford CT
meeting
• “Zeugmatography”
Raymond Damadian 1977 –
Sir Peter Mansfield early
1980’s

Early Thorax Image

Nottingham
Slides from Terry Peters

Birth of MRI

Early Thorax Image

Nottingham
• Electro Marnetic signal emitted (in harmless radio frequensy)
is acquired in the time domain
• image has to be reconstructed (Fourier transform)
Birth of MRI

In 1978, Mansfield presented his first

image through the abdomen.

Lauterbur
and the
first
magnetic
resonance
images
(from
Nature)
Slides from Terry Peters

30 Years of MRI

First brain MR image Typical T2-weighted MR image

Slides from Terry Peters

MR Imaging

“Interesting images, but will never be as

useful as CT”

• (A different) neuroradiologist, 1982

Slides from Terry Peters

MR Imaging …more than T1 and T2

MRA - Magnetic resonance angiography
• images of vessels
MRS - Magnetic resonance spectroscopy
• images of chemistry of the brain and muscle metabolism
fMRI - functional magnetic resonance imaging
• image of brain function
PW MRI – Perfusion-weighted imaging
DW MRI – Diffusion-weighted MRI
• images of nerve pathways
Slides from Terry Peters

Magnetic Resonance Angiography

MR scanner tuned to
measure only moving
structures
“Sees” only blood - no static
structure
Generate 3-D image of
vasculature system
May be enhanced with
contrast agent (e.g. Gd-DTPA)
 Slides from Terry Peters

MR Angiography

GD-enhanced
Phase-contrast

GD-enhanced

In-flow
Slides from Terry Peters

Dynamic 3-D MRI of the thorax

1 www.atamai.com
Slides from Terry Peters

Diffusion-Weighted MRI

Image diffuse fluid motion in

brain
Construct “Tensor image” –
extent of diffusion in each
direction in each voxel in
image
Diffusion along nerve sheaths
defines nerve tracts.
Create images of nerve
connections/pathways
Slides from Terry Peters

Tractography

Data analysed after

scanning
Identify “streamlines”
of vectors
Connect to form fibre
tracts Internal Capsule
14 min scan time

- Dr. D Jones, NIH

Slides from Terry Peters

Tractography

Wernicke’s area
Broca’s area

Short fibres

Long fibres
Insula
fibres

Temporal fibres
“just like Gray’s Anatomy”!
Superior
Longitudinal Fasciculus
- Dr. D Jones, NIH USA
Slides from Terry Peters

Functional MRI (fMRI)

Active brain regions demand more fuel
(oxygen)
Extra oxygen in blood changes MRI signal
Activate brain regions with specific tasks
Oxygenated blood generates small (~1%) signal
change
Correlate signal intensity change with task
Represent changes on anatomical images
 Slides from Terry Peters

fMRI

Subject looks at Stimulus

flashing disk while
being scanned
“Activated” sites
detected and
merged with 3-D
MR image Activation
Slides from Terry Peters

fMRI in Neurosurgery Planning

Hand Activation

Tumour
Face Activation
Slides from Terry Peters

Ultrasound

Images courtesy GE Medical