EECE 5642

Introduction to Visualization

Y. Raymond Fu
Professor
Electrical and Computer Engineering (ECE), COE
College of Computer and Information Science (CCIS)
Northeastern University
 SMILE Lab

What can a machine do?

Machine Learning

Computer Vision

2
Courtesy of Yu Yin
 Outline

• Introductions
– Who am I?
– Who are you? Major, background, expectation et al
• Course Overview
• What is Scientific Visualization?
 Research Lab
Synergetic Media Learning (SMILE) Lab [since 2012]
Northeastern University
• 1 Faculty
• Prof. Raymond Fu
• 14 Ph.D. Students

https://web.northeastern.edu/smilelab/
 Motivation 1: Smart Environment

Wikipedia.com: conceptually a physical world that is richly and invisibly interwoven with
sensors, actuators, displays, and computational elements, embedded seamlessly in the
everyday objects of our lives, and connected through a continuous network…
The image is from http://sedl.kaist.ac.kr/images/smart_architecture_spaces.jpg
 Motivation 2: Social Media in the Cloud

o How to model the multi-label, multi-instance, and multi-task characteristics?

o How to effectively infer meaningful user information from large scale visual data?
o How to provide targeted services through human-computer interactions?
Motivation 3: Visual Analytics
 Motivation 4: Computer Vision
• Images & Videos
High-dimensional data -> visual semantics
Sky
Cloud
Tree
…
Computer vision tasks: Big data
• Scene understanding
• Millions of images
• Object detection
• Saliency detection • Internet image/video sharing
• ... • Social media analytics
 Early work: Graph Embedded Subspace Learning

Machine Learning Framework

Subspace Learning

Demographic Recognition
Emotion/Expression Analysis
Age/Gender Estimation
Inference
Ethnic Group Recognition
Kinship Recognition
Occupation Recognition
Courtesy of Tamara Berg

Human-Centered Computing
 Subspace Learning Extensions
• Low-Rank, Sparse, and Deep Structure Modeling
– Learning Balanced and Unbalanced Graphs via Low-Rank Coding
– Robust Low-Rank Subspace Discovery
– Low-Rank Common Subspace for Multi-view Learning
– Low-Rank Transfer Subspace Learning
– Low-Rank Tensor Completion
– Deep Low-Rank Coding for Transfer Learning
– Deep Linear Coding for Fast Graph Clustering
– …
Low-Rank Multi-Source Data Mining
 It Is All About Data!
• Goal: Interpret given human images in terms of demographic
and behavioral attributes (Expression, Age, Gender, Occupation,
Kinship, Action, Pose, and Intention, etc.).

• Challenge
– Dimensionality redundancy
– Large scale (big data)
– Unknown distribution
– Large attributes variations
– Multimodality , multi-source, multi-label data
– Noise and outliers
 It Is All About Data!
• Goal: Interpret given human images in terms of demographic
and behavioral attributes (Expression, Age, Gender, Occupation,
Kinship, Action, Pose, and Intention, etc.).

• Challenge
– Dimensionality redundancy
– Large scale (big data)
– Unknown distribution
– Large attributes variations
– Multimodality , multi-source, multi-label data
– Noise and outliers
 Manifold Learning
Swiss Roll

Dimensionality
Reduction

Courtesy of Sam T. Roweis and Lawrence K. Saul, Sience 2002

Stabilized Manifold Learning
 Expression Manifold

Manifold visualization of 1,965 Frey’s face images by LEA using k = 6 nearest neighbors.
Yun Fu, et. al. “Locally Adaptive Subspace and Similarity Metric Learning for Visual Clustering and Retrieval”, CVIU,
Vol. 110, No. 3, pp: 390-402, 2008.
 Emotion State Manifold

Manifold visualization for 11,627 AAI sequence images of a male subject using LLE algorithm. (a) A video frame snapshot
and the 3D face tracking result. The yellow mesh visualizes the geometric motion of the face. (b) Manifold visualization with
k=5 nearest neighbors. (c) k=8 nearest neighbors. (d) k=15 nearest neighbors and labeling results.
 Application for Age Estimation
AS International, How Old Are You?, www.asmag.com Vol. 120, Page 40-41, Dec. 2008.
PhysOrg.com, Intelligent Computers See Your Human Traits, May 2008.
Roland Piquepaille's Technology Trends, Computers can now guess our age, Sep. 2008.
UIUC News Bureau, Step right up, let the computer look at your face and tell you your age, Sep. 2008.
ABC Science, Age recognition software has a human eye, Oct. 2008.
UPI.com, Age estimation software is created, Sep. 2008.
Eureka! Science News, Step right up, let the computer look at your face and tell you your age, 2008
Zdnet.com, Computers can now guess our age, Sep. 2008.
Webindia123.com, Age estimation software is created, Sep. 2008.
Newkerala.com, Now, a computer software that can tell age just by looking at your face!, 2008.
Hindustantimes.com, Computer that says how old you are, Sep. 2008.
TXonline.net, Age estimation software is created, Sep. 2008.
Topnews.in, Now, computer software that can tell age just by looking at your face, Oct. 2008.

Age estimation on Einstein’s faces.

The estimated ages below each face
might be a little bit older than the
true ages (unknown to us) but
reasonable. Our training data are all
Asian faces.

This might be a good example to echo

the phenomenon that Asian faces
often aesthetically look younger than
the Western.
Y. Fu, et. al., IEEE TPAMI, CVPR, ICCV, 2009, 2010, 2011.
 Kinship Recognition
Son
Father Mother

Family Album KinFace Database

Son Young Father Father

o Hypothesis: most of children look like their parents at young ages
o Utilizing transfer learning method to bridge the gap
 Family Tree

Y. Fu, et. al., IEEE Transactions on Multimedia (TMM), 2012.

 Hand Parts Recognition from Depth Image
Reconstruct skeleton model from Kinect Difficulties
• More flexible than body
• More sensitive to environment change
• Small volume and difficult to detect
• No obviously texture to distinguish
depth image classification reconstruction
Training data collection Labeling process
Manually
labeling

Denoising Background can

Clustering be changed to
achieve more
Glove Aided Labeling Labeled data training data
Classification and reconstruction
⁻ Using Microsoft’s framework as
a baseline [Jamie etc., CVPR 11]
⁻ Combine simple features
In processing…
⁻ add more manually labeled&synthetic data
⁻ design robust features Training and test with combined features
 Learning in SMILE Lab: Our Recent Focuses
Computational
Visual Intelligence

Learning for
Learning in Deep Learning with Less
Interpretability

• Residual dense network • Few/Zero shot learning • Supervised attention

• Deep sequential model • Domain adaptation • Graph learning/distilling
• Adversarial training • Incremental learning • Causal inference
• Deep multi-view learning • Multi-task learning • Ensemble learning

High-level vision
Applications in SMILE Lab Visual-text matching
Style transfer
Image restoration Video understanding

Low-level vision Object/face recognition

Image segmentation
 Why going Deep?
Deep Convolutional Neural Networks CVPR
• 2D convolutions + Pooling
• Deep layers of network (~100 layers)
• State-of-the-art performance

ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012

Deep Residual Learning for Image Recognition, CVPR 2015
 Deep Residual Learning: Summary
Image Synthesis Video Enhancement
• Multimodal image style transfer [ICCV’19] • Temporally-deformable alignment network [CVPR’20]
• Space-Time Video Super-Resolution [CVPR’20]

Image Super-Resolution Image Restoration

• Residual dense network [CVPR’18] • Non-local attention network [ICLR’19]
• Residual channel attention [ECCV’18] • Residual dense network [TPAMI’20]

Biomedical Image Enhancement

• Graph reasoning attention for MR image SR [ECCV’20 submission]
• Biomedical image deblurring [MICCAI’20 submission]
Video-based Human Activity Analysis

Video action prediction Multi-view video action recognition

 Why expect Interpretability?
Explanation

CNN ‘Cat’ Backprop

Input Image Attention Map

Method Top1
Localization Cue
Top2 Top3 Top4 Top5 Metho

Pop tags Graphic Design Illustration Photography Branding Art Direction Pop ta

Understanding how the CNN model recognize

Log2Intent the “cat” [1].
Photography
Digital
Photography Fine Arts Graphic Design
Performing
Arts
Log2In

User interests (Ground truth) Fine Arts, Photography, Digital Photography User in

Interpretability is the key in many fields

• How to explain deep model?
Method Top1 Top2 Top3 Top4 Top5 Metho

• How to interpret data via high-level semantics? Pop tags Graphic Design Illustration Photography Branding Art Direction Pop ta

Character
• How to do reasoning over low-level data? Log2Intent Illustration

User interests (Ground truth)

Digital Art Graphic Design Drawing

Illustration, Digital Art, Character Design

Design
Log2In

User in

Describe user behavior [2].

[1] Tell me where to look: Guided attention inference network, CVPR’18
[2] Log2Intent: Towards Interpretable User Modeling via Recurrent Semantics Memory Unit, KDD'19
Guided Attention Learning
• Attention map can describe which part of the image supports the network’s prediction.
Explanation

CNN ‘Cat’ Backprop

Input Image Attention Map Localization Cue

• Problems
Input image Attention Maps [2] Improved Attention Maps
• Only cover small and partial regions
• Wrong focus due to dataset bias Incomplete Self-
Exploration
End-to-end
training
Extra
Wrong focus Supervision
[1] Tell me where to look: Guided attention inference network, CVPR’18
[2] Guided Attention Inference Network, TPAMI, 2019
[3] Attention Bridging Network for Knowledge Transfer, ICCV’19
 Outline

• Introductions
– Who am I?
– Who are you? Major, background, expectation et al
• Course Overview
• What is Scientific Visualization?
 Outline

• Introductions
– Who am I?
– Who are you? Major, background, expectation et al
• Course Overview
• What is Scientific Visualization?
 Course Overview

Workload
• 4 homeworks
– HW1,HW2, and HW4 aim to solve related problems?
– HW3 is about paper reading and research
• 1 mid-term exam
• 1 final projects
– proposal and final presentations, final submission
Syllabus
 Outline

• Introductions
– Who am I?
– Who are you? Major, background, expectation et al
• Course Overview
• What is Scientific Visualization?
 Definition
• The power or process of forming a
mental picture or vision.
(Oxford English Dictionary)

• To form a mental image of. Envisage.

(Merriam-Webster Dictionary)

• To visualize the scene as it was described.

(thefreedictionary.com)

• To make visual or visible.

(dictionary.com)

• The purpose of visualization is to convey

information to people through graphical
means.
(Pat Hanrahan, Barbara Tverski, et al.)

WebAnalysts.Info
 Human Brain and Visualization

• The human brain has a powerful capacity,

often underutilized, to process visuals.

• A large portion of the brain processes

visuals, and visualization technology puts
that ability to work.

• The brain absorbs a lot more information

when it's presented in pictures rather
than in stacks of data from a computer.

• By Prof. Robert Jacob, Tufts University.

PhysOrg.com Google.com
 The Applications of Visualization

• Help thinking
• Provide solutions
• Uncover patterns
• Clarify
• Encourage creativity
• Communicate
• Cartogram
• Inspiration
• Entertainment
• Make money
• … Courtesy of Prof. Hanspeter Pfister,
Harvard University.
Help Thinking: Mind Map

Jamie Nast's book titled “Orbiting the Giant Hairball Courtesy of Prof. Hanspeter Pfister,
“- Idea Map by Megan Clark Harvard University.
Provide Solutions

Courtesy of Prof. Hanspeter Pfister,

Harvard University.
Provide Solutions

Panorama of the location Courtesy of Prof. Hanspeter Pfister,

Harvard University.
Cartogram
 Encourage Creativity
Dog anatomy from
memory
Uploaded on March 24,
2007 by Nad

Map of solar system

from memory
Uploaded on February 27,
2007 by Nad

Brazil map from memory

Uploaded on January 24, 2008 by Nad
All from flickr.com
Clarify
Clarify

Boston T Map
Communication

Facebook.com
 Inspiration
From ted.com Hans Rosling's TED 2007 talk (link)

Hans Rosling's TED 2006 talk (link)

This is the map that all

London estate agents give
to their employees the
first day on their job.

500 Years of Female Portraits in Western Art

From youtube.com Courtesy of Prof. Hanspeter Pfister,
Harvard University.
Entertainment
 Make Money

The Million Dollar Homepage’ by Alex Tew (2005) http://www.milliondollarhomepage.com/

 Visualization Goals

• Record - Preserve information

• Analyze - Reason about data
• Present - Convey information

Courtesy of Prof. Hanspeter Pfister,

Harvard University.
Record

Courtesy of Prof. Hanspeter Pfister,

Harvard University.
Analyze

weblog.infoworld.com/udell/
Analyze

Matthew Ericson/The New York Times, 2004-2005

Analyze

Courtesy of Prof. Hanspeter Pfister, Harvard University.

Photo Finishes: Information, Rich Roger von Oech
Present

www.oracle.com/
Present

Courtesy of Edward Tufte.

Present

Courtesy of Edward Tufte.