05-440 / 05-640: Interaction Techniques,

Spring, 2022

http://www.cs.cmu.edu/~bam/uicourse/05440inter2019/
Brad Myers
Human Computer Interaction Institute

Lecture 1
Spring, 2022 1
© 2022 - Brad Myers
 What is This?
 How many “states” can it be in?

© 2022 - Brad Myers

 Example: check box
 How many “states” can it be in?
 Checked, not-checked, partial checked
 Disabled, not-disabled
 Hover, not-hover (can’t be hover+disabled)
 Pressed-inside, pressed-outside, not-pressed
(can’t be pressed + disabled, can’t be pressed-
inside + not-hover)
 Keyboard focus, not-focus
 2^4 * 3 = 48, but many are not possible
 Implementers often forget about the release-outside
case & interface gets confused (Flash
implementations) 3

© 2022 - Brad Myers

Course:
 05-440 / 05-640: Interaction Techniques
 Intended for Undergraduates and Masters – same course

 Fourth time offered

 2014 http://www.cs.cmu.edu/~bam/uicourse/2014inter/ (was videotaped)

 2016: http://www.cs.cmu.edu/~bam/uicourse/05440inter2016/

 2019: http://www.cs.cmu.edu/~bam/uicourse/05440inter2019/

 2022: http://www.cs.cmu.edu/~bam/uicourse/05440inter2022

 Tues-Thurs, 1:25PM - 2:45PM

 Attendance in lectures expected
 In-class quizzes on required readings

 Class participation counts as part of grades

 First 2 weeks by Zoom (link in Canvas)

 Recordings also will be on Canvas – do not share the recordings
 Then in-person (hopefully)
 Room: WEH 4709
4
 Maybe still recorded? © 2022 - Brad Myers
Waitlist
 There are 26 people registered across both course
numbers
 19 on the waitlist (6 in 440; 13 in 640) = 45
 28 allowed in the room – hard limit
 As an elective, I expect a lot of people to drop, so many
from waitlist will get in
 If you decide not the take the course, please drop it
promptly
 If you are on the waitlist, OK to attend through Zoom for 2
weeks
 Recommend starting on the homework as well!
5

© 2022 - Brad Myers

IxT
 “IxT” = interaction technique
 Like “IxD” = interaction design
 https://www.interaction-design.org/literature/topics/interaction-design

 Abbreviation from
2019 course!

© 2022 - Brad Myers

 Course Web page:
 Course Web page:
http://www.cs.cmu.edu/~bam/uicourse/05440inter2022
 www.uicourse.org (temporarily)
 Course schedule is tentative
 Note homework schedule
 Some readings are CMU-only, use CMU network or
VPN
 Note required readings – from my textbook!

© 2022 - Brad Myers

Course Textbook
 We will be using a draft of my new book!
Pick, Click, Flick! The Story of Interaction Techniques
 Free PDF on Canvas and Google Drive
 Please do not copy, distribute or publicly post this
draft!
 Please let me know about typos, corrections, errors,
parts that are not clear, or things that are missing!
There will be rewards for letting me know about them!
 See also: www.ixtbook.com

© 2022 - Brad Myers

Instructor
 Brad Myers
 Human Computer Interaction Institute
 Office: Newell-Simon Hall (NSH) 3517
 Phone: x8-5150
 E-mail: [email protected]
 http://www.cs.cmu.edu/~bam
 Office hours: TBD.

 My cat: “Cupcake”

© 2022 - Brad Myers

Teaching Assistant
 Chaoran Chen
 [email protected]
 METALS (Masters)
 https://chenchaoran1234.wixsite.com/mysite
 Office hours:
 Maybe Tuesdays, 12:30-1:30 EST starting
next week?

10
© 2022 - Brad Myers
 Why am I teaching this course?
 I was at MIT Media Lab (then “Architecture
Machine Group), 1976-1979
 At Xerox PARC, 1976-1980
 Designed one of the first commercial
window managers, 1980-1984
 First to put progress bars into icons, and collect
icons in a window, etc.
 Studies of two-handed UIs and progress bars
with Bill Buxton, 1984 - 1988
 “All the Widgets” history video, 1990
 "A Brief History of Human Computer Interaction
Technology.” ACM interactions, 1998
 With students, invented new text input techniques
 Significant consulting on patents on interaction
techniques, 1988-present
 I am (literally) writing the book on Interaction Techniques! 11

© 2022 - Brad Myers

Course Requirements
 This course does not require any
programming
 Prerequisites:
 Students should preferably have taken
UCRE/Methods or DHCS or Intro. HCI Tech
Execs or equivalent
 However, I am allowing students interested in
history of technology to enroll without any
background in HCI or experience in programming
12

© 2022 - Brad Myers

Education Goals
 After taking this course, students will be able to:
 Articulate design issues regarding interaction
techniques.
 Design a new interaction technique given a set of
requirements and constraints.
 Evaluate interaction techniques using the appropriate
tests for performance and usability.
 Describe the historical progression of the most
important interaction techniques and the factors that
impacted their evolution and eventual widespread
adoption.
13

© 2022 - Brad Myers

Lecture Topics
 General topics
 Many specific kinds of interaction techniques
 Lots of history
 Useful to have examples of different approaches
 See schedule
 What would you like to hear about?

14

© 2022 - Brad Myers

 Guest Lectures by Inventors
 https://www.cs.cmu.edu/~bam/uicourse/05440inter/index.html#ixtinventors
 2022:
 Bill Buxton – 2 handed interaction, many advanced interactions, tools to use IxT
 Gregg Vanderheiden –many assistive techniques
 Alex Mogilevsky, wavy underlines for spelling and grammar errors in Microsoft Office
 Others invited…
 2019:
 Bill Atkinson – original Apple Macintosh
 Dan Bricklin – spreadsheets
 Rob Haitani – Palm Pilot UI
 Don Hopkins – pie menus
 Shumin Zhai – swype phone text entry
 2016:
 Norm Cox – icon designs
 Bruce Horn – original Macintosh UI
 Loren Brichter – pull-to-refresh
 Tandy Trower – Microsoft Windows designs
 May-Li Khoe – Apple's Force Touch, 3D Touch
 2014:
 David C. Smith – Icons
 Larry Tesler – Copy-and-paste
15
 Ted Selker – IBM pointing stick
 Chris Harrison – many new ones
Grading
 Homeworks
 Midterm
 Final Project
 Final test (on second half)
 See how much each is worth on homeworks page
 It is fine with me if everyone gets “A”

 Taking it Pass/Fail is fine

 Auditing is not fine if return to in-person
16

© 2022 - Brad Myers

 Homeworks
 Mostly measurement activities on
your own
 Homework #1:
 Measuring the speed and accuracy of
clicking with various pointing devices
 Mouse, laptop touchpad, IBM pointing
stick, fingers on smartphone, fingers on
tablet, stylus on phone, stylus on tablet,
Nintendo Wii controller, … (what else?)
 Discussion of your measurements
with respect to the historical record
 See the late policy 17

© 2022 - Brad Myers

Final Project Ideas
 See course homepage:
http://www.cs.cmu.edu/~bam/uicourse/05440inter2022/index.html#finalprojectideas
 Create or edit a Wikipedia page about an interaction technique or
about an inventor of one, with appropriate citations
 Promised our guest speakers that we would do this for them!
 Document, evaluate and critique one or more existing interaction
techniques with appropriate evidence
 Interview an inventor of a particular interaction technique
 Pick a particular, important milestone design, and enumerate and
describe all of the novel interaction techniques introduced in that
system.
 Invent a new interaction technique and evaluate it
 Create a project of your own design, in consultation with the professor.
18

© 2022 - Brad Myers

Introduction to this Course: What
is an Interaction Technique and
Why are They Important?

Lecture 1
Spring, 2019

19
© 2022 - Brad Myers
“Interaction Techniques”
 Scroll bars, buttons, text fields
 But also:
 Copy-and-paste
 Text entry in PC or phone
 Drawing a new object in an editor
 Selecting a cell in a spreadsheet
 How high level? Text editor widget,
but not Word
 Scroll bar is composed
of buttons, etc.
20

© 2022 - Brad Myers

What are some other examples?

21

© 2022 - Brad Myers

 Some examples
 Visual Basic “controls”
 Physical controls

22

© 2022 - Brad Myers

 Other names
 “Widgets” (Wikipedia: “GUI Widget”)
 Note that there have been no cross references in Wikipedia
between “Interaction Technique” and “Widget”
 See my video “All the Widgets”
 But not the same as Apple dashboard widgets
 GUI “elements”
 “Gadgets”
 But not the same
as Scott Hudson’s
 “Controls”
 (MS Windows)
 “Components”
 Too generic
 “Behaviors” 23

 = the interaction part © 2022 - Brad Myers

Definitions
 My definition:
An “interaction technique” starts when the
user does something that causes a computer
to respond, and includes the direct feedback
from the computer to the user. Interaction
techniques are generally reusable across
various applications.

24

© 2022 - Brad Myers

Definitions
 Wikipedia’s definition:
“An interaction technique, user interface
technique or input technique is a combination
of hardware and software elements that provides
a way for computer users to accomplish a single
task.”
 (has changed periodically)

25

© 2022 - Brad Myers

Definitions
 Foley & van Dam, 1990:
“An interaction technique is a way of using a
physical input/output device to perform a
generic task in a human-computer dialogue.”

26

© 2022 - Brad Myers

What is not an interaction
technique?
 Whole applications (Microsoft Word)
 Dashboard “widgets” – small apps for the desktop
 Output only (no interactions)
 Visualizations
 But many come with specialized interactions, then they
might count? https://support.google.com/analytics/answer/1068216?hl=en#zippy=%2Cin-this-article

 Animations
 Movies
 …
27

© 2022 - Brad Myers

 Why Study Interaction
Techniques?
 Used extensively
 Everyone who uses a computer
uses copy-paste, etc.
 So can have an enormous impact
 Interesting historically
 Why do we do things the way we do?
 Is there a good reason?
 Example: which way does the arrow point in a scroll bar?
 And new interaction techniques are created all the time:
 Patent on “Bounce at end of scrolling” for iPhone submitted by Bas Ording in 2007 (right
before 1st iPhone was released in 2007) US 7,469,381
 Try it! iPhone vs. Samsung
 “Pull down to refresh” – patent submitted in 2010 by Twitter, became popular in 2013!
 US 8,448,084
 Inventor was a guest speaker in 2016!
 Many new CHI & UIST conference papers every year with new ones 28

© 2022 - Brad Myers

Why Study Interaction
Techniques, cont.
 Interaction Techniques have a high economic value
 Often the subject of patents and lawsuits
 Can’t patent overall look and feel
 “Apple Wins Over Jury in Samsung Patent Dispute, Awarded
$1.05 Billion in Damages (Live Blog)” 2012, link
 “Jury orders Samsung to pay $290M to Apple in patent case”
2013, link
 Need new ones
 “Desktop metaphor” is getting tired
 Macs & PCs look and work pretty similar to each other and to the
designs of the 1980’s (40 years ago)
 Text entry on smartphones is a big barrier
 Text entry on smartphones!!!
 Selecting individual elements, characters on smartphones 29

© 2022 - Brad Myers

Problem
 April 29, 1991

30

© 2022 - Brad Myers

 Problem

 Appliances are too complex

31

© 2022 - Brad Myers

 Problem

 Too many remotes

32

© 2022 - Brad Myers

Why are Interaction
Techniques Hard to Design?
 Surprisingly large number of design decisions & detailed
design
 Individual differences and preferences
 Lots of details that impact human performance
 How far does the cursor move when you move the mouse 1
inch?
 Trick question – depends on mouse speed
 Complex formula developed through experimentation
 How far does the content move on an iPhone when you flick
your finger?
 Needs to work for long distance, and highly accurate local movements
 Nokia phones released just after the iPhone got this all wrong
33

© 2022 - Brad Myers

Example: Drawing a new object
 What happens when move upwards past
start point?

34

© 2022 - Brad Myers

Many other Details and
Advanced Techniques
 What does shaking an iPhone do?
 How do you drag around the cursor in an
iPhone text area?
 What does a “Copy” (^C) operation do?
 What does Undo (^Z) do after you have done
a Copy or a Save operation?
 … and many more!
 You will learn these in this course!
35

© 2022 - Brad Myers

Designing Interaction
Techniques
 Must take into account device characteristics
 Must take into account human characteristics
 Look
 Styling
 3D look and feel – Smith’s ARK (1986), up through Windows
7
 Flat squares – Windows Phone and Windows 8, 10
 Feedback for behaviors
 Animation effects – from 1993
 Feel
 Specific implementation of the behavior
 Details matter 36

© 2022 - Brad Myers

Why Hard, continued
 Need to support keyboard and assistive uses
 Need to abort and undo interactions
 Localization and Internationalization
 Left-to-right vs. right-to-left (up-to-down?)
 Different sizes in different languages

37

© 2022 - Brad Myers

More Definitions
 “User” - a person who will use a product
 “Designer - the person who creates (or designs) the
product
 “Implementer” or “Developer” – who also creates
 “User interface” (UI), - everything the user
encounters when using a product
 “Usability” – measures the quality of the UI
 “Input” and “Output” - always used from the
computer’s point of view – input is “in” to the
computer; output is out from the computer
38

© 2022 - Brad Myers

Measuring Interaction
Techniques
 What are relevant quality metrics for interaction
techniques?
 For evaluating them?

39

© 2022 - Brad Myers

Measuring Interaction
Techniques
 What are relevant quality metrics for interaction
techniques?
 (same as other HCI usability metrics!)
 Efficiency (speed)
 Error rate
 Learnability
 Discoverability
 Memorability
 Aesthetics & emotional impact
 Satisfaction (Pleasurable)
 Consistency with other interactions
 Etc.
40

© 2022 - Brad Myers

Measuring Interaction
Techniques
 But also generalizability
 How often can be used?
 Different applications?
 Different kinds of input values?
 Dimensionality
 One D (menu, slider) or 2-D (mouse), or more
 How many items? (pick among 5 items vs. among
100 or 1,000)

41

© 2022 - Brad Myers

 Affordances
 “Perceived and actual properties of the
thing, primarily those fundamental
properties that determine how the thing
could possibly be used.” (Norman DOET
book, p. 9)
 “When affordances are taken advantage of,
the user knows what to do just by looking”
 Helps people understand what to do with
the control

42

© 2022 - Brad Myers