AI for Computer Vision

Unit I: Introduction to Image Formation and Processing

This unit focuses on the foundational concepts that are critical for understanding how images are
formed, represented, and processed in computer vision. It covers both geometric and photometric
aspects of image formation, as well as basic image processing techniques.

1. Computer Vision

Computer vision is the field of artificial intelligence that enables computers to interpret and
understand the visual world. The goal is to develop algorithms that allow machines to process, analyze,
and make decisions based on visual data (images and video).

Example: Computer vision powers facial recognition systems, object detection, and self-driving cars.

2. Geometric Primitives and Transformations

Geometric primitives refer to basic shapes or structures used in image analysis, such as points, lines,
circles, and polygons. Transformations involve altering these shapes or coordinates within an image
through operations like scaling, translation, rotation, and affine transformations. These
transformations allow objects to be resized, rotated, or shifted in the image plane.

Example: A translation transformation moves all points in an image by a fixed distance along the x and
y axes.

3. Photometric Image Formation

Photometric image formation refers to the process of how light is captured by a camera sensor to form
an image. This process includes factors such as lighting conditions, reflectance properties of surfaces,
and camera parameters. Photometric techniques help in understanding how images are affected by
changes in lighting, shadows, and material properties.

Example: A change in lighting in an image may cause shadows or highlights, affecting the pixel values
(intensity) in the image.

4. Digital Camera

A digital camera captures images using an image sensor (CCD or CMOS) which converts light into
electrical signals. The camera settings, such as exposure time, white balance, and aperture, affect the
final image.

5. Point Operators

Point operators are image processing techniques that operate on each pixel independently. These
operators include operations like brightness adjustments, contrast enhancement, or color
transformations.

Example: A point operator can be used to adjust the brightness of an image by adding a constant value
to each pixel’s intensity.
6. Linear Filtering

Linear filtering involves the application of a filter (or kernel) to an image to modify its properties.
Common filters include smoothing filters (for blurring) and edge-detection filters (for detecting
boundaries).

Example: A Gaussian filter smooths an image, reducing noise.

7. More Neighborhood Operators

These operators consider a neighborhood of pixels around each pixel. Examples include median
filtering (which replaces a pixel with the median of its neighbors) and Sobel edge detection (which uses
the gradient of the image to find edges).

8. Fourier Transforms

The Fourier transform is a mathematical tool used to analyze the frequency components of an image.
It decomposes an image into its sinusoidal components (frequencies). This is useful for tasks like image
compression and filtering.

Example: The Fourier transform can be used to remove noise from an image by filtering out high-
frequency components.

9. Pyramids and Wavelets

Image pyramids are multi-scale representations of images where each level of the pyramid is a
downsampled version of the original image. Wavelets provide a way to represent an image at multiple
resolutions while capturing both spatial and frequency information.

10. Geometric Transformations

Geometric transformations refer to operations that manipulate the shape or position of objects in an
image. These include affine transformations (scaling, rotation, translation) and projective
transformations (perspective changes).

Example: A perspective transformation can change the view of a building in an image as if seen from a
different angle.

11. Global Optimization

Global optimization involves finding the best solution over a large space of possible solutions. In
computer vision, this might involve minimizing an error function across an entire image or video
sequence, like minimizing the difference between predicted and observed feature locations.

---

Unit II: Feature Detection, Matching, and Segmentation

This unit dives into techniques used to identify significant features in images, match these features
across different views, and segment an image into meaningful regions.

1. Points and Patches

Feature points are specific locations in an image that are distinctive and can be easily tracked across
frames or images. Patches are small regions around these points that can be used for further analysis,
such as matching between images.
Example: Corners detected using Harris corner detection are often used as feature points.

2. Edges

Edges represent significant changes in intensity in an image and are often used to detect boundaries
between objects. Edge detection algorithms like the Canny edge detector identify edges by looking for
areas with large gradients in pixel intensity.

3. Lines

Line detection is a key part of many vision tasks. Lines in images can be detected using methods like
the Hough Transform, which transforms points in the image space to a parameter space to detect lines.

4. Segmentation

Segmentation involves dividing an image into meaningful regions. This can be done by clustering pixels
based on color, texture, or other attributes. The goal is to simplify the representation of the image and
make analysis easier.

Example: Segmentation can be used to detect different objects in an image, such as segmenting a
person from the background.

5. Active Contours

Active contours (also known as snakes) are used to detect object boundaries in images. The contour is
initialized and then evolves under the influence of internal and external forces to fit the object’s
boundary.

6. Split and Merge

Split and merge algorithms recursively divide an image into smaller regions (split) and then combine
them (merge) based on certain criteria, such as homogeneity in color or texture.

7. Mean Shift and Mode Finding

Mean shift is a non-parametric clustering technique that can be used for image segmentation. It
iteratively moves a window to the average color in the region and finds modes in the data distribution,
typically used for tracking and segmentation.

8. Normalized Cuts

Normalized cuts is a graph-based segmentation algorithm that divides an image into segments by
minimizing the dissimilarity between different segments while maximizing the similarity within each
segment.

9. Graph Cuts and Energy-Based Methods

Graph cuts model the image segmentation problem as a graph where nodes represent image pixels,
and edges represent the relationship between neighboring pixels. Energy-based methods aim to
minimize an energy function that encodes segmentation costs.

---

Unit III: Feature-based Alignment & Motion Estimation

This unit deals with advanced methods for aligning features across images (particularly useful in tasks
like 3D reconstruction and motion tracking) and estimating the motion between images or across video
sequences.

1. 2D and 3D Feature-based Alignment

This technique involves matching and aligning features between images (2D) or across multiple views
of a 3D scene. For 2D, this might involve techniques like feature matching using descriptors (SIFT,
SURF). For 3D, this could involve techniques like matching points between stereo images to reconstruct
a 3D scene.

2. Pose Estimation

Pose estimation refers to the process of determining the position and orientation (pose) of a camera
or an object in 3D space, often based on feature correspondences in multiple images.

3. Geometric Intrinsic Calibration

Intrinsic calibration involves determining the internal parameters of the camera, such as the focal
length, optical center, and distortion coefficients, which are necessary for accurate 3D reconstruction.

4. Triangulation

Triangulation is a technique used in stereo vision to compute the 3D coordinates of a point by

observing it from two different camera positions and solving for the intersection of the rays.

5. Two-Frame Structure from Motion

This technique estimates the 3D structure of a scene and the relative motion between two frames. It
involves finding correspondences between points in two images and using them to recover camera
motion and scene geometry.

6. Factorization

Factorization methods, like the factorization method of Tomasi and Kanade, are used in structure-from-
motion (SfM) to decompose image data into camera motion and 3D structure without explicit feature
tracking.

7. Bundle Adjustment

Bundle adjustment is an optimization technique used in SfM to refine both the 3D structure and
camera poses simultaneously by minimizing reprojection errors across all views.

8. Constrained Structure and Motion

Constrained structure and motion refers to solving for 3D structure and camera motion under certain
constraints, such as known object shapes, camera motion models, or physical properties of the scene.

9. Translational Alignment

Translational alignment refers to aligning images or features by compensating for translational shifts,
typically in video sequences where the camera is moving across a scene.

10. Parametric Motion

Parametric motion refers to modeling the movement of objects or cameras using a set of parameters,
such as translation, rotation, or scaling, to describe the transformation over time. This method is often
used to describe rigid motion in both computer vision and robotics. The parameters of motion can be
expressed using transformation matrices or more specialized functions, depending on the application.

11. Spline-based Motion

Spline-based motion modeling involves fitting splines (piecewise polynomial functions) to the motion
of objects or camera trajectories. This method allows for smooth modeling of complex motions, such
as curvilinear trajectories or articulated movements. Spline-based motion is especially useful in
scenarios where a smooth, continuous representation of motion is required.

12. Optical Flow

Optical flow is a technique used to estimate the apparent motion of objects in an image sequence. By
analyzing pixel intensity changes over time, optical flow calculates the velocity of objects across the
image plane. This technique assumes that pixel brightness is conserved across frames and that the
motion between frames is small enough for linear approximations to hold.

Example: In a self-driving car system, optical flow can help estimate the movement of surrounding
vehicles, pedestrians, or other obstacles.

13. Layered Motion

Layered motion modeling refers to the concept of different parts of an image moving at different
speeds or in different directions. This is common in dynamic scenes where the foreground and
background may move independently. Layered motion can help to separate these components for
tasks like background subtraction or object tracking.

Example: In a video of a person walking in front of a moving car, layered motion can be used to track
the motion of the person separately from the motion of the car.

Real-World Applications of These Techniques

The methods covered in this course are widely applicable in many domains. Here are some specific
examples:

1. Self-Driving Cars:

Optical flow and motion estimation are used to track the motion of vehicles, pedestrians, and other
obstacles in real-time.

Feature detection and pose estimation help in identifying lanes, traffic signs, and pedestrians.

Segmentation is used to detect different regions in the road (e.g., road, sidewalk, other vehicles).

2. Augmented Reality (AR):

Pose estimation and feature alignment are essential for accurately aligning virtual objects with real-
world environments.

Geometric transformations are used to adjust the position and orientation of virtual objects in real-
time.
3. Medical Imaging:

Image segmentation plays a key role in segmenting and analyzing different parts of medical images
(e.g., detecting tumors in MRI scans).

Motion estimation is used to track the movement of organs or tissue in dynamic medical imaging, such
as video-based endoscopy or MRI scans.

4. Robotics:

Feature-based alignment and 3D reconstruction are crucial for robot vision systems to map
environments and navigate autonomously.

Motion estimation helps robots understand their movement and surroundings, enabling tasks like
object manipulation and navigation.

5. Video Surveillance:

Segmentation, motion tracking, and layered motion are used to detect and track moving objects in
surveillance video feeds.

Active contours and graph-based segmentation can detect and track people or vehicles, assisting in
security monitoring.

6. Computer Graphics:

Geometric transformations and pose estimation are used in rendering 3D scenes and animations.

Fourier transforms and wavelets are used in image compression, enhancement, and texture mapping.