COMPUTER VISION

MOTION & OPTICAL FLOW

Ngo Quoc Viet-2024

CONTENTS

1. Motion
2. Optical Flow
• Brightness Constancy constraint equation
• Lucas-Kanade algorithm
3. Project: Action Recognition

2
LECTURE OUTCOMES

• Understanding the concepts of motion estimation.

• Understanding and using optical flow algorithms such as Lucas-
Kanade to estimate motion.
• Using functions like cv.calcOpticalFlowPyrLK() to track feature
points in a video.
• Implementing simple action recognition based on optical flow and
others.

3
M O T I O N E S T I M AT I O N

• Motion refers to the change in position or appearance of objects

in a sequence of images or frames over time.
• Analyzing motion is a fundamental aspect of computer vision, and
it plays a crucial role in various applications, including video
surveillance, object tracking, activity recognition, and robotics.
• There are several key concepts related to motion: Optical Flow,
Motion Detection, Object Tracking, Activity Recognition
• It involves the analysis of consecutive frames in a video sequence
to determine the motion of objects or the camera itself.
4
W H Y M O T I O N E S T I M AT I O N

• Video Compression: Motion estimation helps in detecting changes

between consecutive frames, allowing for more effective
compression algorithms like MPEG standards.
• Data Transmission and Storage: It enables the transmission of only
the changes in a scene, reducing the amount of data that needs to
be transmitted or stored.
• Object Tracking: tracking the movement of objects within a scene
is used in surveillance systems, autonomous vehicles, and robotics
to follow the trajectory of objects or people.
• VR and AR: providing realistic and immersive experiences
5
H OW M O T I O N E S T I M AT I O N

• Block Matching: simple but may not handle complex motions

• Optical Flow: estimate the motion of each pixel by analyzing the
pattern of intensity changes between consecutive frames. The optical
flow field represents the motion vectors for each pixel in the image.
Various algorithms, such as Lucas-Kanade and Horn-Schunck.
• Gradient-Based Methods: computing the gradient of the image and
using it to derive motion information. Lucas-Kanade algorithm.
• Feature Tracking: Algorithms like the Kanade-Tomasi tracker or the
Shi-Tomasi
• Deep Learning Approaches.
6
MOTION FIELD

• Camera moves (rotation,

translation)
• Objects in scence move
rigidly
• Objects articulate (e.g:
hand moving)
From: CSE 152, Spring 2018
• Objects bend and deform https://michaelbach.de/ot/cog-hiddenBird/index.html
• Blowing smoke, clouds
• We estimate the motion field by computing the optical flow.
7
O P T I C A L F L OW

8
https://sandipanweb.wordpress.com/2018/02/25/implementing-lucas-kanade-optical-flow-algorithm-in-python/
O P T I C A L F L OW

• The most general of motion estimation is to compute an independent estimate of

motion at each pixel, which is generally known as optical (or optic) flow.
• Optical flow refers to the analysis of the apparent motion of objects or patterns
within a visual scene based on the pixel intensity changes between consecutive
frames of a video or image sequence.
• Optical flow is a vector field that shows the direction and magnitude of these
intensity changes from one image to the other. These vectors convey information
about the local motion patterns within the scene.
• The concept of optical flow is particularly valuable for tasks such as motion
analysis, object tracking, and scene understanding. By computing the optical flow,
computer vision systems can infer information about how objects are moving, their
speed, and their direction of motion.
• The goal of optical flow algorithms is to estimate the optical flow vector 𝒖, 𝒗 for
each pixel in the image, satisfying this brightness constancy constraint.
9
W H Y W E N E E D O P T I C A L F L OW

Images from: Sefidia Academy

Track the motion of cars/vehicles Recognize human actions

across frames to estimate its current
velocity and predict its position
10
O P T I C A L F L OW E Q UAT I O N

• Given an image 𝐼(𝑥, 𝑦, 𝑡) and move its pixels by (𝑑𝑥, 𝑑𝑦) over t time, we obtain the new
image 𝐼(𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡). Assume that the pixel intensities of an object are constant
between consecutive frame, then
𝐼 𝑥, 𝑦, 𝑡 = 𝐼 𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡
• Taking the Taylor Series Approximation 1 of the right-hand side of the above equation
𝜕𝐼 𝜕𝐼 𝜕𝐼
𝐼 𝑥 + 𝑑𝑥, 𝑦 + 𝑑𝑦, 𝑡 + 𝑑𝑡 = 𝐼 𝑥, 𝑦, 𝑡 + 𝑑𝑥 + 𝑑𝑦 + 𝑑𝑡 + ⋯
𝜕𝑥 𝜕𝑦 𝜕𝑡
• Rearranging terms and setting the approximation equal to the original expression, we get
𝜕𝐼 𝜕𝐼 𝜕𝐼
𝑑𝑥 + 𝑑𝑦 + 𝑑𝑡 = 0
𝜕𝑥 𝜕𝑦 𝜕𝑡
• Dividing by dt to get optical flow equation
𝜕𝐼 𝜕𝐼 𝜕𝐼 𝑑𝑥 𝑑𝑦
𝑢+ 𝑣+ = 𝑰𝒙 𝒖 + 𝑰𝒚 𝒗 + 𝑰𝒕 = 0, 𝑢 = ,𝑣 =
𝜕𝑥 𝜕𝑦 𝜕𝑡 𝑑𝑡 𝑑𝑡
𝜕𝐼 Τ𝜕𝑥, 𝜕𝐼 Τ𝜕𝑦 , and 𝜕𝐼 Τ𝜕𝑡 are the image gradients (derivative of intensity) along the
horizontal axis, the vertical axis, and time
11
https://brilliant.org/wiki/taylor-series-approximation/
O P T I C A L F L OW E Q UAT I O N

𝒅𝒙 𝒅𝒙
• The problem of optical flow, that is, solving 𝒖 = and 𝐯 = to
𝒅𝒕 𝒅𝒕
determine movement over time.
• We cannot directly solve the optical flow equation for u and v
since there is only one equation for two unknown variables.

• We will implement some methods such as the Lucas-Kanade

method to address this issue.

12
B R I G H T N E S S C O N S TA N C Y E Q UAT I O N

• The Brightness Constancy Equation is a fundamental assumption in

optical flow estimation, i.e the intensity (brightness) of a pixel remains
constant over time.
• Constancy Equation is expressed as
𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 + 𝐼𝑡 = 0
• 𝐼𝑥 and 𝐼𝑦 are the partial derivatives of image intensity with respect to the
coordinates x, y. 𝐼𝑡 is the temporal derivative of image intensity with respect to
time t.
• u, v are the horizontal and vertical components of the optical flow vector,
representing the motion of the pixel in the image.
• Optical flow constraint or brightness constancy constraint equation
(Horn and Schunck 1981)
13
O P T I C A L F L OW M E T H O D S

• The goal of optical flow algorithms is to estimate the optical flow vector
𝒖, 𝒗 for each pixel in the image, satisfying this brightness constancy
constraint.
• Optical flow works on several assumptions
• The pixel intensities of an object do not change between consecutive frames
• Neighbouring pixels have similar motion
• Sparse Optical Flow: This approach tracks specific features or interest
points in the image, such as corners or key points, and estimates the
motion vectors of these points.
• Dense Optical Flow: This method calculates motion vectors for every
pixel in an image, providing a dense field of motion information across
the entire image
14
S PA R S E O P T I C A L F L OW M E T H O D S

• Lucas-Kanade: This method assumes that the motion is essentially

constant in a local neighborhood. It solves a system of linear equations
for each pixel, considering the gradient of the image and the temporal
gradients.
• KLT (KLT Tracker): The Kanade-Lucas-Tomasi tracker is an extension
of the Lucas-Kanade method. It uses the eigenvalues of the gradient
matrix to determine the features that are best suited for tracking.
• Feature Tracking Algorithms: Various feature tracking algorithms, such
as Shi-Tomasi or Harris corner detectors, can be used to identify key
points in the image, and then the motion of these features is tracked
over time.
15
D E N S E O P T I C A L F L OW M E T H O D S

• Horn-Schunck: This method formulates optical flow as a global constraint

problem, assuming that the motion is smooth across the entire image. It
minimizes the global error between the observed flow and the estimated
flow.
• Farneback: Farneback's algorithm is a polynomial expansion method that
estimates dense optical flow by approximating the motion with a polynomial
function. It's known for being computationally efficient.
• Deep Learning-Based Methods: Convolutional neural networks (CNNs) and
other deep learning architectures have been employed for optical flow
estimation. Networks like FlowNet use convolutional layers to directly
predict optical flow from image pairs.
• Variational Methods: Variational methods formulate optical flow as an
optimization problem and use energy minimization techniques to estimate
the flow field.
16
D E N S E V S . S PA R S E M E T H O D S

Dense Optical Flow Sparse Optical Flow

Objective Estimate motion for every pixel in an Estimate motion for a selected set of
image or within specified regions keypoints or interest points in an image.
Output A dense motion field, where motion Motion vectors only for specific points or
vectors are calculated for all pixels. keypoints chosen in the image.
Applications Useful in applications requiring detailed Suitable for scenarios where motion
motion information across the entire information for specific features or
image, such as video analysis, frame locations is sufficient, such as object
interpolation, and dense motion tracking, feature tracking, and matching
segmentation
Algorithms Farnebäck Optical Flow, Horn-Schunck Lucas-Kanade, KLT (Kanade-Lucas-
Method (Adapted for Dense Flow), Brox Tomasi) Tracker, Good Features to Track
Optical Flow, PCA-Flow, DeepFlow, (GFTT), SURF (Speeded Up Robust
PWC-Net (Pyramidal Network for Features), ORB (Oriented FAST and
Optical Flow) Rotated BRIEF), Feature Matching with
SIFT or AKAZE 17
 L U C A S - K A N A D E O P T I C A L F L OW A L G O R I T H M

• LK algorithm is based on the idea of linearizing the image

intensity variations in both spatial and temporal dimensions to
estimate the local motion.
• LK algorithm involves finding the motion (u, v) that minimizes
the sum-squared error of the brightness constancy equations for
each pixel in a window.
• LK method formulates the following system of linear equations
for each pixel in a local window.

18
 L U C A S - K A N A D E O P T I C A L F L OW A L G O R I T H M

• Spatial Gradients: Compute the spatial gradients 𝐼𝑥 and 𝐼𝑦 by using

spatial filter (Prewitt, Sobel,… operators). For example, Prewitt kernel
−1 1 −1 −1
𝐻𝑥 = 𝐻𝑦 =
−1 1 1 1

𝐼𝑥 = 𝐻𝑥 ∗ I
• Temporal Gradient: compute the temporal gradient 𝐼𝑡 by taking the
pixel-wise difference between the two consecutive frames.
𝜕𝐼
𝐼𝑡 𝑥, 𝑦 = = 𝐼 𝑥, 𝑦, 𝑡 + 1 − 𝐼(𝑥, 𝑦, 𝑡)
𝜕𝑡
• Windowed Intensity Values: Choose a local window around a pixel
19
 L U C A S - K A N A D E O P T I C A L F L OW A L G O R I T H M

𝑢
• Lucas-Kanade Equation: 𝐼𝑥 𝑢 + 𝐼𝑦 𝑣 = −𝐼𝑡 ⇔ 𝑥 𝐼 𝐼𝑦 = −𝐼𝑡
𝑣
𝑢
𝐼𝑥 𝐼𝑦 𝑇 𝐼𝑥 𝐼𝑦 = − 𝐼𝑥 𝐼𝑦 𝑇 𝐼𝑡
𝑣
𝑢
= − 𝐼𝑥 𝐼𝑦 𝑇 𝐼𝑥 𝐼𝑦 −1 𝐼𝑥 𝐼𝑦 𝑇 𝐼𝑥 𝐼𝑦 𝐼𝑡
𝑣
• Consider n = 𝑘𝑥𝑘 window (n pixels) arounding the specific pixel
𝐼𝑥1 𝐼𝑦1 −𝐼𝑡1
𝐼𝑥2 𝐼𝑦2 𝑢 −𝐼𝑡2
=
⋮ ⋮ 𝑣 ⋮
𝐼𝑥𝑛 𝐼𝑦𝑛 −𝐼𝑡𝑛
𝑢 𝑇 𝑢 𝑇 𝑢
𝐴 = 𝐼𝑡 ⇒ 𝐴 𝐴 = 𝐴 𝐼𝑡 ⇒ = 𝐴𝑇 𝐴 −1 𝐴𝑇 𝐼𝑡
𝑣 𝑣 𝑣
• We solve a least-square problem to find the best flow vectors for the pixel
20
 L U C A S - K A N A D E O P T I C A L F L OW A L G O R I T H M

𝑢
• Instead of one equation 𝐼𝑥 𝐼𝑦 = −𝐼𝑡 with two unknow variables, we have n linear
𝑣
equations (local window with n pixels). The problem of over-determined.
𝐼𝑥 𝑝1 𝐼𝑦 𝑝1 −𝐼𝑡 𝑝1
𝐼𝑥 𝑝2 𝐼𝑦 𝑝2 𝑢 −𝐼𝑡 𝑝2
=
⋮ ⋮ 𝑣 ⋮
𝐼𝑥 𝑝𝑛 𝐼𝑥 𝑝𝑛 −𝐼𝑡 𝑝𝑛
• In such cases, a least squares approach is used to find the best-fitting solution.
• The least squares fit provides a way to solve this system of equations and find the motion
parameters that best satisfy the constraints imposed by the brightness constancy assumption.
• The least squares fit aims to minimize the sum of squared differences (residuals) between
the observed intensity changes (temporal gradient 𝐼𝑡 ) and the predicted changes based on
the motion parameters. This minimization process provides a robust estimation of the
motion parameters that best fits the observed data.
21
 L U C A S - K A N A D E O P T I C A L F L OW A L G O R I T H M

• To address the over-determined issue, we apply least squares fitting

to obtain the following two-equation-two-unknown problem

−1
2
෍ 𝐼𝑥 𝑝𝑖 ෍ 𝐼𝑥 𝑝𝑖 𝐼𝑦 𝑝𝑖
𝑢 𝑖 𝑖 − ෍ 𝐼𝑥 𝑝𝑖 𝐼𝑡 𝑝𝑖 − ෍ 𝐼𝑦 𝑝𝑖 𝐼𝑡 𝑝𝑖
=
𝑣 2
෍ 𝐼𝑦 𝑝𝑖 𝐼𝑥 𝑝𝑖 ෍ 𝐼𝑦 𝑝𝑖 𝑖 𝑖

𝑖 𝑖

Least squares fitting: https://mathworld.wolfram.com/LeastSquaresFitting.html;

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html . 22
 LUCAS-KANADE STEP BY STEP

• We want to estimate the motion vector at 1, 1 in 3x3 patch.

1 2 3 0 1 2
• Frame t: 4 5 6 , t+1: 3 4 5 .
7 8 9 6 7 8
1 1 1
• Using Iy, Ix = np.gradient(I1) to compute the spatial gradients: 𝐼𝑥 = 1 1 1 ,
3 3 3 −1 −1 −1 1 1 1
𝐼𝑦 = 3 3 3 , Temporal Gradient 𝐼𝑡 = 𝐼 = −1 −1 −1 ,
3 3 3 −1 −1 −1
• np.column_stack((Ix.flatten(), Iy.flatten())) to compute
1 1 1 1 1 1 1 1 1
𝐴𝑇 = , 𝑏 = −𝐼𝑡 = [1 1 1 1 1 1 1 1 1]
3 3 3 3 3 3 3 3 3
• motion_params = np.linalg.lstsq(A, b, rcond=None)[0] to get 𝑢, 𝑣 = 0.1, 0.3

23
LUCAS-KANADE PYTHON STEP BY STEP

24
 LUCAS-KANADE PYTHON WITH OPENCV

• Select features to track (SIFT, SURF, • ret, first_frame = cap.read()

or else).
• prev_gray = cv.cvtColor(first_frame,
• while (cap.isOpened()): cv.COLOR_BGR2GRAY)
• next, status, error =
cv.calcOpticalFlowPyrLK(prev_gray, • feature_params = dict(maxCorners = 300,
gray, prev, None, **lk_params) qualityLevel = 0.2, minDistance = 2,
• good_old = prev[status == 1] blockSize = 7)
• # Selects good feature points for • prev = cv.goodFeaturesToTrack(prev_gray,
next position
mask = None, **feature_params)
• good_new = next[status == 1]
• Draw lines (motion)

• lk_params = dict(winSize = (25,25), maxLevel = 2, criteria = (cv.TERM_CRITERIA_EPS |

cv.TERM_CRITERIA_COUNT, 10, 0.03))
• next, status, error = cv.calcOpticalFlowPyrLK(prev_gray, gray, prev, None, **lk_params) 25
P RO B L E M S O F L U C A S - K A N A D E

• Suppose 𝐴𝑇 𝐴 is easily invertible

• Suppose there is not much noise in the image
• Brightness constancy is not satisfied
• The motion is not small
• Coarse-to-fine optical flow estimation (Pyramid Lucas-Kanade, …)
• A point does not move like its neighbors (window size is too
large)

26
A C T I O N R E C O G N I T I O N W I T H O P T I C A L F L OW

• Optical can be used for motion analysis. It involves computing the flow
vectors that represent the apparent motion of pixels between consecutive
frames in a video sequence.
• Optical flow can be used to estimate the direction and speed of moving
objects.
• OpenCV provides functions like cv2.calcOpticalFlowFarneback(),
cv2.goodFeaturesToTrack() and cv2.calcOpticalFlowPyrLK() that can be
used for optical flow analysis, feature tracking.
• Extract pertinent information from the video frames such as optical flow, or
feature tracking to extract motion-related features from the video frames and
feed them into machine learning model (SVM, AdaBoost, etc).
• Lucas-Kanade itself is not typically used for action recognition, it can be a
component of a larger system for motion analysis.
27
A C T I O N R E C O G N I T I O N W I T H O P T I C A L F L OW

28
A C T I O N R E C O G N I T I O N W I T H O P T I C A L F L OW

29
S O M E C V 2 F U N C T I O N S F O R O P T I C F L OW

• calcOpticalFlowPyrLK (optical flow for a sparse feature set using

the iterative Lucas-Kanade method with pyramids)
• calcOpticalFlowFarneback (dense optical flow using the Gunnar
Farneback’s algorithm).
• calcOpticalFlowHS (optical flow for two images using Horn-
Schunck algorithm)

30
ACTIVITY RECOGNITION USING MEDIAPIPE

These demos are made by my student-Yen Thai Thi Kim.

31
ACTIVITY RECOGNITION USING MEDIAPIPE

• This problem involves recognizing human actions with 3 classes:

walking, playing cello and playing tennis. The data is collected in
the form of videos from the following sources:

• The video features are from Google MediaPipe,

https://github.com/google/mediapipe.
• Activity recognition is based from LSTM. For example:
https://github.com/nam157/human_activity_recognition-. 32
ACTIVITY RECOGNITION USING MEDIAPIPE

12 keypoints

• Each keypoint contains 4 parameters: x, y, z, visibility.

• With 12 keypoints per frame, a single frame captures 48 values.
• And there are 24 frames captured per second
33
S U M M A RY

• Optical flow algorithms such as Lucas-Kanade to estimate motion.

• Lucas-Kanade algorithm is based on Brightness Constancy
constraint equation and least squares method for neighbor of the
specific location.
• The OpenCV functions for optical flow: goodFeaturesToTrackm
calcOpticalFlowPyrLK, calcOpticalFlowFarneback,
calcOpticalFlowHS.

34