Digital Image Processing

Chapter 0

Dept. of Computer Engineering

DMS (Digital Media System) Lab.

Yung-Lyul Lee (이영렬)

http://dms.sejong.ac.kr
http://home.sejong.ac.kr/~yllee/
 “One picture is worth more than ten
thousand words.”
- anonymous –

백문이 불여일견

Spring : 이영렬 2
 Text

Spring : 이영렬 3
 References
Books
- Digital Image Processing Third Edition by Rafael C. Gonzalez & Richard E.
Woods, Addison Wesley
- Digital Image Processing by Gregory A. Baxes, John Wiley & Sons
- Fundamentals of Digital Image Processing by Anil K. Jain, International
edition
- Digital Image Processing Algorithms and Applications by I. Pitas, John Wiley
& Son
- Digital Video Processing by Tekalp
- …..

Journals
⚫ IEEE Transactions on Image Processing
⚫ IEEE Transactions on Circuits and Systems for Video Technology
⚫ Signal Processing: Image Communications
⚫ Multimedia Tools and Applications
⚫ MDPI Journals

Spring : 이영렬 4
 Cont.
한국 Journal
⚫ 한국방송· 미디어공학회
⚫ 대한전자공학회 SP (Signal Processing)
⚫ 한국멀티미디어학회
⚫ …

Spring : 이영렬 5
 영상처리 관련자료

http://scholar.google.com/ 을 참조하면 개인관련 많은 논문/특허

정보를 얻을 수 있음.
Use “google” when you search papers and patents

Homework
⚫ 숙제 공지, 1주일 후 수업 전 숙제제출.
⚫ 늦게 제출한 숙제: 절대 안 받음.

Grade
⚫ Mid-term exam: 30%, final exam: 40%, Homework + programming:
20%, attendance: 10%

6
 Why is Math needed?
+∞ +∞

𝐻(𝜔
ෞ1 , 𝜔
ෞ2 ) = ෍ ෍ ℎ(𝑛1 , 𝑛2 )𝑒 −𝑗𝜔ෞ1 𝑛1 𝑒 −𝑗𝜔ෞ2 𝑛2
2D Fourier Transform 𝑛1 =−∞ 𝑛2 =−∞
𝑀−1 𝑁−1
1 2𝜋𝑘𝑥 2𝜋𝑙𝑦
𝐻(𝑘, 𝑙) = ෍ ෍ ℎ(𝑥, 𝑦)𝑒 −𝑗 𝑀 𝑒 −𝑗 𝑁
𝑀𝑁
𝑥=0 𝑦=0

𝑥′ cos 𝜃 − sin 𝜃 𝑠𝑥 0 𝑥 𝑡𝑥
= +
Rotation & Translation 𝑦′ sin 𝜃 cos 𝜃 0 𝑠𝑦 𝑦 𝑡𝑦
𝑁
Entropy 𝐻(𝑆𝑖 ; 𝑖 = 1, . . . , 𝑁) ≡ 𝐸{− log 2 𝑃 (𝑆𝑖 )} = − ෍ 𝑃(𝑆𝑖 ) log 2 { 𝑃(𝑆𝑖 )}
𝑖=1

Convolution f(x,y)*h(x,y)  F(u,v)H(u,v)

𝑡𝐿+1 𝐿 𝑡𝑘+1

𝑒𝑟𝑟𝑜𝑟 = න መ 2 𝑝(𝑓)𝑑𝑓 = ෍ න (𝑓 − 𝑟𝑘 )2 𝑝(𝑓)𝑑𝑓

(𝑓 − 𝑓)
𝑡1 𝑘=1 𝑡𝑘
𝑡𝑘+1
Solution after minimizing 𝜕𝑒𝑟𝑟𝑜𝑟
=2න (𝑓 − 𝑟𝑘 )𝑝(𝑓)𝑑𝑓 = 0, ⥄ 𝑘 = 1,2, . . . , 𝐿
Mean Square error 𝜕𝑟𝑘
𝑡𝑘
𝜕𝑒𝑟𝑟𝑜𝑟
= (𝑡𝑘 − 𝑟𝑘−1 )2 𝑝(𝑡𝑘 ) − (𝑡𝑘 − 𝑟𝑘 )2 𝑝(𝑡𝑘 ) = 0, 𝑘
𝜕𝑡𝑘
= 2, . . . , 𝐿
Based on Math and Ideas → C (or C++) or other languages
Spring : 이영렬 7
 Why is DCT used in video coding?
DCT (Discrete Cosine Transform) II
𝑁−1 𝑁−1
(2𝑥 + 1)𝑢𝜋 (2𝑦 + 1)𝑣𝜋
𝐹(𝑢, 𝑣) = 𝛼(𝑢)𝛼(𝑣) ෍ ෍ 𝑓(𝑥, 𝑦) cos cos
2𝑁 2𝑁
𝑥=0 𝑦=0
𝑁−1 𝑁−1
(2𝑥 + 1)𝑢𝜋 (2𝑦 + 1)𝑣𝜋
𝑓(𝑥, 𝑦) = ෍ ෍ 𝛼(𝑢)𝛼(𝑣)𝐹(𝑢, 𝑣) cos cos
2𝑁 2𝑁
𝑢=0 𝑣=0

1 2
𝑤ℎ𝑒𝑟𝑒 𝛼(0) = , 𝛼(𝑘) = , 𝑘 = 1, . . . , 𝑁 − 1
𝑁 𝑁

x u
50

100

150

200

250
y v
300

350

400

450

500
100 200 300 400 500
Spring : 이영렬 image Transformed image 8
 And

Slide download
⚫ 집현시스템
Introduction to DMS(Digital Media System) Lab
⚫ http://dms.sejong.ac.kr or http://home.sejong.ac.kr/~yllee/
⚫ Patents: paper submission after patent application
⚫ H.264/MPEG-4 AVC and HEVC/H.265 standardization (contribution
documents) and implementation
✓ 45+ standard-related contribution documents
✓ patents
⚫ Versatile Video Coding (VVC/H.266)
✓ patents
⚫ CNN-based video coding
⚫ next video coding standard (after VVC/H.266)

9
 Digital signal 관련 Royalty 지급 현실

4G (LTE) 휴대폰의 경우 판매가의 5.25～5.75%

5G
PC의 경우 매출의 10%를 IBM, 마이크로소프트, TI사 등에
지불
반도체 매출의 12%를 TI, 인텔, IBM등에 로얄티로 지불
디지털TV 등 차세대 주력 상품의 경우에도 핵심 기술에
대한 특허를 외국 기업이 보유 (한국도 많이 보유)

10
 Royalty

MPEG-4 AVC/H.264
HEVC/H.265
⚫ standardized in May 2013
VVC (Versatile Video Coding)/H.266
⚫ standardized in July 2020
AV1/VP9
⚫ youtube video codec (Alliance for Open Media(AOM))

11
 JPEG 특허분쟁

Forgents Networks: Compression Labs인수

⚫ 1986년 10.27 특허신청 (Variable length coding)
✓ US 4,698,672 B1
✓ “Coding system for reducing redundancy” ．1987.10.6. 등록됨．46건의
클레임으로 구성됨. 2D Variable Length Coding
⚫ Forgent Networks 소유권 인수 (1997)
⚫ Sony: 1500만불 지급
⚫ 마쓰시타, 도시바, 삼성, LG, 현대: ?
무료 표준으로서 JPEG의 명예 실추
Forgents Networks는 Google 및 yahoo 등 특허침해제소

12
 Royalty의 무서움
인터디지털: paper company, 특허 괴물(Patent Troll)
⚫ 폐업한 회사나 개인 발명가, 특허경매 등을 통해 저평가된 특허를
사들인 후 대기업을 상대로 소송을 벌여 라이선스 수익을 창출하는
회사
⚫ 애플, 노키아와 삼성, LG 등 글로벌 IT업계의 강자들도 이 회사에
수억 달러에 달하는 라이선스비 지불 중
최근 매물로 나온 인터디지털 특허에 대해 애플과 퀄컴,
구글 등 IT공룡들이 눈독

13
 Chapter 1 Introduction
What is Digital Image Processing ?

(수강생을 제외하고 해당 강의자료의 배포 및 무단 복제 금지)

Yung-Lyul Lee (이영렬)

Dept. of Computer Engineering
DMS (Digital Media System) Lab.

http://dms.sejong.ac.kr
http://home.sejong.ac.kr/~yllee/
 DIP

Amplitude values of x, y, f : all finite, discrete quantities => digital image

Digital image processing refers to processing digital images by
means of a digital computer.

Spring : 이영렬 15
 Digital Image Processing
Processing of 2D (3D) picture by digital computer
Applications
⚫ image display, transmission and storage
⚫ video compression: HDTV (MPEG-2), iPhone/DMB/IPTV codec
(H.264/MPEG-4 AVC), iPhone and Galaxy (HEVC(High Efficiency Video
Codec)/H.265), Netflix UHD Streaming (HEVC), Google AV1 or VP9 of
“YouTube”, and so on.
⚫ medical processing (Chest X-ray; tomography, MRI)
⚫ remote sensing, deep-space-probe mission
⚫ radar, sonar, acoustic imaging (DSP)
⚫ robotics, computer vision
⚫ Deep learning (CNN, Convolutional Neural Network)

16
 Sample Images

Baboon Girl F-16

Moon surface Clock Peppers

Spring : 이영렬 17
 Image Processing to computer vision
Low-level processing: Image to Image (Image processing)
Mid-level or high-level processing: Image to symbol
⚫ Computer vision, Image understanding, Machine learning (AI)
Computer graphics Image processing

Spring : 이영렬 18
 Historical Backgrounds
1920s: the Bartlane cable picture transmission system
⚫ across the Atlantic ( ~ 3 Hours)
⚫ using a telegraph printer with 5 distinct gray levels

1940’s: the basis of modern digital computer by John Von Neumann

Von Neumann: memory, conditional branch
1960’s: powerful enough to carry out meaningful image processing
tasks , space program

1970’s: application of image processing in medical diagnosis (CT)

2000’s: HDTV & mobile (Now) & UHD
Applications: astronomy, biology, nuclear medicine, law
enforcement(법집행), military defense, and industrial applications
Spring : 이영렬 19
Spring : 이영렬 20
 Snapshot of Application (1)
Video Compression: Smart Phone, UDH/HD TV, DVD, Blu-ray Disc,
Set-Top box, Media Player, Multimedia….

Photography: digital camera, special effects(warping, blending,…)

Broadcasting/Multimedia: video archiving (compression), transmission

(compression), streaming, video conference(Webex, Zoom), digital
museum

Biological Research: cell analysis; DNA typing

Video/Film: visual effects, animation

Defense/Intelligence: automated interpretation of satellite images;

missile-guidance system; road detection

Noise removal: Convolutional Neural Network

Spring : 이영렬 21
 Snapshot of Application (2)
Factory Automation: automated quality inspection
Space Exploration/Astronomy: exploratory spacecraft
Law Enforcement: fingerprints recognition, DNA matching
Material Research: classification of impurities, grain size (Electron
microscopy)
Medical Diagnostics: (digital) X-ray, computed tomography
(CT,MRI,PET)
…

Spring : 이영렬
 Sample Images
Lena Soderberg (Swedish)

Lena in 1997

A good test image !!

Nice mixture of detail,
Texture,
Shading,
flat regions, and so on.
By W.K. Pratt (USC)

Spring : 이영렬 23
Spring : 이영렬 24
 Examples of fields that use Digital
Image Processing
Categorize images according to the source (energy)
⚫ 1. Gamma rays - include nuclear medicine and astronomical(천문학적)
observations
⚫ 2. X-rays - medical diagnostics, industry and other areas like astronomy
⚫ 3. Ultraviolet(자외선) - industrial inspection, microscopy, lasers, biological,
imaging and astronomical observations
⚫ 4. Visible(가시광선) & Infrared(적외선) - light microscopy, astronomy,
remote sensing (원격탐사) , industry, and law enforcement.
⚫ 5. Microwaves – radar
⚫ 6. Radio waves – medicine(MRI) and astronomy

Spring : 이영렬 25
Spring : 이영렬 26
 PET(Positive emission tomography)
Cygnus Loop (a superheated stationary gas cloud)

Spring : 이영렬 27
 lenz

Spring : 이영렬 28
 Radio wave

thyroid(갑상선)
상처

Spring : 이영렬 29
 Fundamental Steps in Digital Image
Processing

Spring : 이영렬 30
 Fundamental Steps in Digital Image
Processing
Image Enhancement
Wavelets
⚫ representing images in various degree of resolution
Image Restoration
⚫ Noisy removal
Image Understanding (Computer Vision)
⚫ Segmentation, Representation, and Recognition
Image Reconstruction from Projection
Image Coding (Image Data Compression)
Color image processing
Morphological processing
⚫ deal with tools for extracting image components
Segmentation
⚫ deal with tools for extracting image components that are useful in the
representation and description of shape

Spring : 이영렬 31
 Fundamental Steps in Digital Image
Processing
Representation and description
⚫ represented and described in a form suitable for computer processing
✓ boundary representation - external shape characteristics
✓ regional representation - internal properties (ex. Texture, DC)
description (called Feature selection: size, DC, moments, …)

Recognition
⚫ assign a label to an object based on the information provided by its
descriptors
✓ Decision function, loss function

Spring : 이영렬 32
 Image Enhancement
Goal
⚫ to accentuate certain image features for subsequent analysis or for image
display

Spring : 이영렬 33
 Image Restoration (1)
Goal
⚫ to remove or minimize known/unknown degradations in image

Input image Output image

Spring : 이영렬 34
 Image Restoration (2)
Removal or minimization of known degradation in an image
⚫ de-blurring
⚫ geometric correction
⚫ sensor non-linearity compensation

f(x,y) : signal g(x,y) : observed

H
Given H (impulse response), g(x,y)

n(x,y) : noise -> what is best estimate of f(x,y) ?

Modeling the degradation

⚫ statistical characterization of noise and signal. Observed image

(Jain, page.7)
35 이영렬
 Image Understanding
Goal
⚫ to interpret or describe the meaning contained in the image

Tank

Breast Cancer

Spring : 이영렬 36
 Image Reconstruction
Reconstruction from projection
Obtain cross-section image from projections
Computer-Aided Tomography (CAT)

Positron Emission Tomography (PET)

Magnetic Resonance Imaging (MRI)
⚫ In MRI, measured data is not only projection data.
⚫ Fourier transform can be used for reconstruction.
Spring : 이영렬 37
 Image(Video) Data Compression (1)
Goal
⚫ to reduce the amount of data required to represent images
Aspect of Image Storage and Image Transport
⚫ HDTV (1080i) : 1920 x (1080/2) x 1.5 x 60Hz  93 Mbytes/sec
Assuming that current MBC and KBS broadcasting bandwidths are
⚫ 10 Mbits/sec
Compression ratio = (93x8 Mbps)/(10Mbps) = 74.4
Input : image Output : bit-stream data

“010100101100110101001 . . . .”

Spring : 이영렬 38
 Image(Video) Data Compression (2)
Techniques
⚫ Lossless coding (Error-free coding) : medical image, Digital Film
⚫ Lossy compression
⚫ Image Compression Standard
✓ JPEG, H.263, MPEG-1,2,4, H.264/MPEG-4 AVC, HEVC (H.265)
✓ VVC(Versitile Video Coding) Applications
⚫ Transmission
✓ teleconferencing ,TV system, remote sensing via satellite
✓ Broadcasting
✓ Streaming (OTT, Over The Top)
⚫ Storage
✓ VOD (video on demand), DVD (digital video disk), medical imaging,
educational and business documents, Blu-ray Disc (50GB)
✓ Probably new one

Spring : 이영렬 39
 Video Coding Standard
Evolution of video coding standard
⚫ Storage, mobile and internet, broadcasting, streaming
H.261 H.263
(1988) (1996)

Joint
H.262/ H.264/AVC H.264/AVC VVC/H.266
ITU-T VCEG and H.265/HEVC
MPEG-2 Main Profile High Profile (2020)
ISO/IEC MPEG (2013)
(1994) (2003) (2005)
standard

MPEG-1 MPEG-4
(1991) (1999)

1990 1995 2000 2005 2010 2015 2020

Storage Capacity in (Mbytes)

⚫ Human brain 125,000,000
⚫ Blu-ray disk 50,000
Spring : 이영렬 40
 Components of an Image Processing
System
Image Acquisition
⚫ camera, scanner, CCD line-scan sensor, CCD area sensor
⚫ hardware realization, sampling problem, preprocessing, etc
Image Storage
⚫ frame buffer(short-term), disk(on-line storage)
⚫ traditionally, in film, video tapes,…
Image Processing
⚫ in software & hardware (optimum S/W and/or H/W co-deign problem)
⚫ dedicated vs. general purpose H/W issue
Image Communication
⚫ compression techniques
⚫ transport, delivery problem
Image Display
Communication
⚫ LCD, Printing,... channel

Imaging Sample/ Digital Digital Online Display

system Quantize Storage Computer buffer

Object observe digitize store process refresh/store output

Record
41 이영렬
 Components of an Image Processing
System

Spring : 이영렬 42