CGLecture 02 Interactive Graphics-1
CGLecture 02 Interactive Graphics-1
1/39
What is Computer Graphics? (1/2)
Computer graphics generally means creation, storage and manipulation
of models and images
Such models come from diverse and expanding set of fields including
physical, biological, mathematical, artistic, and conceptual/abstract
structures
Frame from animation by William Latham, shown at
SIGGRAPH 1992. Latham creates his artwork using
rules that govern patterns of natural forms.
2/39
What is Computer Graphics? (2/2)
William Fetter coined term “computer graphics” in
1960 to describe new design methods he was pursuing
at Boeing for cockpit ergonomics
Created a series of widely reproduced images on “pen
plotter” exploring cockpit design, using 3D model of
human body.
“Perhaps the best way to define computer graphics is to find out what it is not. It is not a machine. It is
not a computer, nor a group of computer programs. It is not the know-how of a graphic designer, a
programmer, a writer, a motion picture specialist, or a reproduction specialist.
Computer graphics is all these – a consciously managed and documented technology directed toward
communicating information accurately and descriptively.”
Computer Graphics, by William A. Fetter, 1966
3/39
What is Interactive* Computer Graphics? (1/2)
User controls content, structure, and appearance of objects and their displayed
images via rapid visual feedback
Basic components of an interactive graphics system
input (e.g., mouse, stylus, multi-touch, in-air fingers…)
processing (and storage of the underlying representation/model)
display/output (e.g., screen, paper-based printer,
video recorder…)
First truly interactive graphics system, Note CRT monitor, light
Sketchpad, pioneered by Ivan Sutherland 1963 Ph.D. thesis pen, and function-key
panels – the “organ
Sketchpad, A Man-Machine Graphical Communication System
console” showing bi-
manual operation
Used TX-2 transistorized “mainframe”
at MIT Lincoln Lab * Sometimes called real-time computer graphics, and in certain
4/39
contexts, real-time rendering
What is Interactive Computer Graphics? (2/2)
Almost all key elements of interactive graphics system are expressed in first
paragraph of Sutherland’s 1963 Ph.D. thesis
5/39
What is Batch Computer Graphics?
Today, we still use non-interactive batch mode (aka offline rendering) for final
production-quality video and film (special effects – FX). Rendering a single frame of
The Good Dinosaur (2015, a 24 fps movie) averaged 48 hours on a 30,000-core render
farm!
Statistics from
https://www.fxguide.com/featured/making-the-world-of-pixars-the-good 6/39
Enabling Modern Computer Graphics (1/5)
Hardware revolution
Moore’s Law: every 12-18 months, computer power
improves by factor of 2 in price / performance as size shrinks
Newest CPUs are 64-bit with 4, 8, 16, even up to 22 cores
Intel Kaby Lake – consumer processor with 4 cores, 8
threads, and a fully featured graphics chip built in to the
processor
Significant advances in commodity graphics chips every 6
months vs. several years for general purpose CPUs
NVIDIA GeForce GTX Titan Xp… 3840 cores, 12GB memory,
and 12 teraflops of processing power in a single chip
NVIDIA statistics from
https://www.nvidia.com/en-us/geforce/products/10series/titan-xp/ 7/39
Enabling Modern Computer Graphics (2/5)
Graphic subsystems
Offloads graphics processing from CPU to chip designed for doing graphics operations quickly
NVIDIA GeForce™, AMD Radeon™, and Intel HD and Iris Pro Graphics
GPUs originally designed to handle special-purpose graphics computations
Increasingly, GPUs used to parallelize other types of computation (known as GPGPU, or General-
Purpose Computing on the Graphics Processing Unit)
Hardware show and tell: NVIDIA GeForce GTX 1080 Ti (2017)
3584 cores, 1.58 GHz clock, 11GB memory, 139 billion pixels/second fill rate
Department machines (row 6): NVIDIA GeForce GTX 970 (2014)
1664 cores, 1.05 GHz clock, 4GB memory, 75 billion pixels/second fill rate
Department machines: NVIDIA GeForce GTX 460 (2010)
336 cores, 1.35 GHz clock, 1GB memory, 37.8 billion pixels/second fill rate
8/39
Enabling Modern Computer Graphics (3/5)
Input Devices
Mouse, tablet & stylus, multi-touch, force feedback, and other game
controllers (e.g., Wii U), scanner, digital camera (images, computer vision), etc.
Body as interaction device
http://youtu.be/zXghYjh6Gro
9/39
Enabling Modern Computer Graphics (4/5)
Many form factors
Smartphones/laptops/desktops/tablets
Smart watches Android Phones
Apple iPhone Tablets
Head-mounted displays (HMDs)
HTC Vive
Augmented Reality
Virtual Reality
AR vs VR: Different experiences! Microsoft’s first
Surface Apple Watch Android Wear
Brown’s old Cave Microsoft Hololens Vive Oculus Rift Google Cardboard
10/39
Digression: Augmented Reality
Easily accessible AR through smartphones
11/39
Digression: Cave Redesign
Old Cave:
4 1024 x 786 projectors on 8’ x 8’ walls (8-10 pixels per inch)
Too low resolution and brightness for many applications, and got worse
(brightness, contrast deteriorated over time)
New Cave:
69 projectors onto cylindrically
curved screen 8’ radius, floor, ceiling
140 million pixels
Powered by a ~69 gpu cluster
No right angles, up to 40 pixels per Brown’s new Cave, the YURT
inch (can’t see individual pixels at
normal viewing distance) 12/39
Enabling Modern Computer Graphics (5/5)
Software Improvements
Algorithms and data structures
Modeling of materials
Rendering of natural phenomena
“Acceleration data structures” for ray tracing and other renderers
Parallelization
Most operations are embarrassingly parallel: calculating value of one
pixel is often independent of other pixels
Distributed and Cloud computing
Send operations to the cloud, get back results, don’t care how
Rendering even available as internet service!
13/39
Environmental Evolution (1/5)
Character Displays (1960s – now)
Display: text plus alphamosaic pseudo-graphics (ASCII art)
Object and command specification: command-line typing
Control over appearance: coding for text formatting
(.p = paragraph, .i 5 = indent 5)
Application control: single task
14/39
Environmental Evolution (2/5)
Vector (Calligraphic, Line Drawing)
Displays (1963 – 1980s)
Display: line drawings and stroke text; 2D and 3D transformation hardware
Object and command specification: command-line typing, function keys, menus
Control over appearance: pseudo-WYSIWYG
Application control: single or multitasked, distributed computing pioneered at
Brown via mainframe host <-> minicomputer satellite
Term “vector” graphics survives as “scalable vector graphics” SVG library from
Adobe and W3C – shapes as transformable objects rather than just bitmaps
15/39
Environmental Evolution (3/5)
2D bitmap raster displays for PCs and workstations
(1972 at Xerox PARC - now)
Display: windows, icons, legible text, “flat earth” graphics
Above, a classic WIMP
Note: late 60’s saw first use of raster graphics, especially for flight interface. The technology,
simulators at its core, remains largely
the same today. Below, a
Minimal typing via WIMP GUI (Windows, Icons, Menus,
modern WIMP interface.
Pointer): point-and-click selection of menu items and objects,
direct manipulation (e.g., drag and drop), “messy desktop”
metaphor
Control over appearance: WYSIWYG (which is really WYSIAYG,
What You See Is All You Get – not pixel-accurate or controllable)
Application control: multi-tasking, networked client-server
computation and window management (even “X terminals”)
16/39
Environmental Evolution (4/5)
3D graphics workstations (1984 at SGI – now)
could cost up to $1M for high-end!
Display: real-time, pseudo-realistic images of 3D
scenes
Object and command specification: 2D, 3D and N-D
input devices (controlling 3+ degrees of freedom) and
force feedback haptic devices for point-and-click, Graphics workstations such as
widgets, and direct manipulation these have been replaced with
commodity hardware (CPU + GPU),
Control over appearance: WYSIWYG (still WYSIAYG) e.g., our MaxBuilts + NVIDIA cards
Application control: multi-tasking, networked
(client/server) computation and window
management
17/39
Environmental Evolution (5/5)
High-end PCs with hot graphics cards (NVIDIA GeForce™, AMD Radeon™)
have supplanted graphics workstations
Such PCs are clustered together over
high speed buses or LANs to provide
“scalable graphics” to drive tiled
PowerWalls, CAVEs, etc.
Also build GPU-clusters as number crunchers, e.g.,
protein folding, weather prediction
Now accessible to consumers via
technologies like NVIDIA’s
SLI (Scalable Link Interface) bridge
SLI Image from
18/39
http://www.overclock.net/t/1606562/official-nvidia-titan-x-pascal-o
Graphics Display Hardware
Vector (calligraphic, stroke, Raster (TV, bitmap, pixmap) used
random-scan) in displays and laser printers
Driven by display commands Driven by array of pixels (no semantics,
(move (x, y), char(“A”) , line(x, y)…) lowest form of representation)
Survives as “scalable vector graphics”
Note “jaggies” (aliasing errors) due to
discrete sampling of continuous primitives
Ideal Vector
Drawing Drawing Outline Filled
19/39
Conceptual Framework for Interactive Graphics
Graphics library/package is intermediary between application and display hardware
(Graphics System)
Application program maps application objects to views (images) of those objects by
calling on graphics library. Application model may contain lots of non-graphical data
(e.g., non-geometric object properties)
User interaction results in modification of image and/or model
This hardware and software framework is 5 decades old but is still useful
Software Hardware
Graphics
System/
Application Graphics GPU
Application
Library
Model / database program
20/39
Graphics Library
Examples: OpenGL™, DirectX™, Windows Presentation Foundation™
(WPF) accessed via XAML, RenderMan™, HTML5 + WebGL™
Primitives (characters, lines, polygons, meshes,…)
Attributes
Color, line style, material properties for 3D
Lights
Transformations
Immediate mode vs. retained mode
immediate mode: no stored representation, package holds only
attribute state, and application must completely draw each frame
retained mode: library compiles and displays from scenegraph
that it maintains, a complex DAG. It is a display-centered extract
of the Application Model
21/39
Application Distinctions: Two Basic Paradigms
22/39
Sample-based Graphics (1/3)
Sample-based graphics: Discrete samples
are used to describe visual information
pixels can be created by digitizing images,
using a sample-based “painting” program,
etc.
often some aspect of the physical world is
sampled for visualization, e.g.,
temperature across the US
example programs: Adobe Photoshop™,
GIMP™ , Adobe AfterEffects™ (which came
out of CS123/CS224!)
23/39
Sample-based Graphics (2/3)
Pixels are point locations with associated sample values, usually of light
intensities/colors, transparency, and other control information
When we sample an image, we sample the point location along the
continuous signal and we cannot treat the pixels as little circles or squares,
though they may be displayed as such
CRT* beam illumination pattern
light
intensity
1 pixel
Visualization of a Can’t visually
mathematical pixel LCD display resolve adjacent
grid pixels on CRT
* Cathode Ray Tube, like those really old TVs 24/39
Sample-based Graphics (3/3)
Samples created directly in Paint-type program, or by sampling of continuous (analog) visual
materials (light intensity/color measured at regular intervals) with many devices including:
flatbed and drum scanners
(e.g., https://luminous-landscape.com/drum-scans/ )
digital still and motion (video) cameras
Sample values can also be input numerically (e.g., with numbers from computed dataset)
Once an image is defined as pixel-array, it can be manipulated
Image editing: changes made by user, such as cutting and pasting sections, brush-type tools, and
processing selected areas
Image processing: algorithmic operations that are performed on image (or pre-selected portion of
image) without user intervention. Blurring, sharpening, edge-detection, color balancing, rotating,
warping. These are front-end processes to Computer Vision, Computational Photography
25/39
Sampling an Image
Lets do some sampling of CIT building
3D scene
A color value is measured at every grid point and used to color corresponding
grid square 0 = white, 5 = gray, 10 =
black
27/39
What’s the Disadvantage?
WYSIAYG (What You See Is All You Get): No additional
information
no depth information
can’t examine scene from different point of view
at most can play with the individual pixels or groups of
pixels to change colors, enhance contrast, find edges, etc.
But increasingly great success in image-based rendering
to fake 3D scenes and arbitrary camera positions. New
images constructed by interpolation, composition,
warping and other operations.
For a computational and cognitive science perspective,
Take James Tompkin’s Computer Vision (CSCI1430)
“Scene Reconstruction from High Spatio-Angular
Resolution Light Fields” by Kim, Zimmer et al., 2013
28/39
Geometry-Based Graphics (1/2)
Geometry-based graphics (also called scalable
vector graphics or object-oriented graphics):
geometrical model is created, along with various
appearance attributes, and is then sampled for
visualization (rendering, a.k.a image synthesis)
often some aspect of physical world is visually
simulated, or “synthesized”
examples of 2D apps: Adobe Illustrator™ and Corel
CorelDRAW™
examples of 3D apps: Autodesk’s AutoCAD™,
Autodesk’s (formerly Alias|Wavefront’s) Maya™,
Autodesk’s 3D Studio Max™
29/39
Geometry-Based Graphics (2/2)
Geometry-based graphics applications
Store mathematical descriptions, or “models,” of geometric elements (lines,
polygons, polyhedrons, polygonal meshes…) and associated attributes (e.g., color,
material properties).
Geometric elements are primitive shapes, primitives for short.
Images are created via sampling of geometry for viewing, but not stored as part of
model.
Users cannot usually work directly with individual pixels in geometry-based
programs; as user manipulates geometric elements, program resamples and
redisplays elements
Increasingly rendering combines geometry- and sample-based graphics, both as
performance hack and to increase quality of final product
CG animated characters (geometry) on painted or filmed scene images (samples)
30/39
What is Geometric Modeling?
What is a model?
Captures salient features (data, behavior) of object/phenomenon being modeled
data includes geometry, appearance, attributes…
note similarity to OOP ideas
Modeling allows us to cope with complexity
Our focus: modeling and viewing simple everyday objects
Consider this:
Through 3D computer graphics, we have abstract, easily changeable 3D forms, for the first
time in human history
Has revolutionized working process of many fields – science, engineering, industrial design,
architecture, commerce, entertainment, etc. Profound implications for visual thinking and visual
literacy
“Visual truth” is gone in the Photoshop and FX-saturated world (but consider painting and
photography…) – seeing no longer is believing…(or shouldn’t be!)
31/39
Modeling vs. Rendering
Modeling
Rendering
Create models Take “picture” with camera
Apply materials to models
Place models around scene
Both can be done with commercial software:
Place lights in scene Autodesk MayaTM ,3D Studio MaxTM, BlenderTM, etc.
Place the camera
Point Light
Spot
Light
Directional Light
Ambient
Light
32/39
Decomposition of a Geometric Model
Divide and Conquer
Hierarchy of geometrical components
Reduction to primitives (e.g., spheres, cubes, etc.)
Simple vs. not-so-simple elements (nail vs. screw)
Head
Shaft
Point
composition decomposition
33/39
Hierarchical (Tree) Diagram of Nail
Object to be modeled is (visually) analyzed, and then decomposed into collections of
primitive shapes.
Tree diagram provides visual method of expressing “composed of” relationships of model
Such diagrams are part of 3D program interfaces (e.g., 3D Studio MAX, Maya)
As a data structure to be rendered, it is called a scenegraph
34/39
Composition of a Geometric Model
Translate
Scale and Translate
Primitives Composition
in their own modeling in world (root)
coordinate system coordinate system
Primitives created in decomposition process must be assembled to
create final object. Done with affine transformations, T, R, S (as in
above example). Order matters – these are not commutative!
35/39
Upcoming Topics
We manipulated primitive shapes with geometric
transformations (translation, rotation, scale). These
transformations are essential for model organization, process
of composing complex objects from simpler components.
Hierarchical models and geometric transformations are also
essential for animation – create and edit scenegraphs
Once object’s geometry is established, must be viewed on
screen: map from 3D geometry to 2D projections for viewing,
and from 2D to 3D for 2D input devices (e.g., the mouse or
pen/stylus, or touch)
While mapping from 3D to 2D, object (surface) material
properties and lighting effects are used in rendering one’s
constructions. This
rendering process is also called image synthesis
36/39
In Summary
Computer graphics involves both real-time / interactive applications and batch / offline
applications
Both equally important, but different use cases
Photo-realism has really advanced
But it still takes dozens of hours on fastest computers to mimic physics of photons interacting with
physical environments
Hardware evolution from vector to raster graphics
Vector graphics survives as Scalable Vector Graphics, which transforms without artifacts
Geometry-based vs. image-based graphics
Mathematical definition vs. pixel manipulation
Pixels are discrete samples of continuous functions
Causes artifacts (“jaggies”/ “aliases”) to appear (we will study fixes – “anti-aliasing”)
Geometric models typically constructed hierarchically
Scene graph data structure
37/39
What is Graphics Processing Unit (GPU)?
It is a processor optimized for 2D/3D graphics, video, visual computing,
and display.
It is highly parallel, highly multithreaded multiprocessor optimized for
visual computing.
It provide real-time visual interaction with computed objects via
graphics images, and video.
It serves as both a programmable graphics processor and a scalable
parallel computing platform.
Heterogeneous Systems: combine a GPU with a CPU
FIGURE A.2.2
Contemporary PCs with
Intel and AMD CPUs. See
Chapter 6 for an
explanation of the
components and
interconnects in this
figure. Copyright © 2009
Elsevier
The origin
In this example, the origin or (0,0) is in the bottom-left of the screen. It is also possible to have the
origin in the top-left, in which case the rows (y coordinates) are numbered downwards.