Semantic

Interaction
Design

Daniel Rosenberg

San Jose State Univ.

Human Factors Engineering Program

[Link]
What is the first thing you do when
you start to create/define/specify
an Interaction Design?
 Don’t do that!

There is a better way to start…

 What Chapter 1

Semantic IxD is a proven (cognitive science based)

scalable UX design method that ensures maximum
usability is achieved 10X more effectively & efficiently
than current UX practice today.
 Origin of Semantic Interaction Design (IxD)
theory goes back decades!
2012 2020
Human Factors Cognitive Ergonomics:
• Task action grammar/Complexity
models
• Reisner (1979)
• Spence and Apperley (1984)

+
Cognitive Science:
• Stages of Action (Norman 2015)
• Design by Levels (Foley 1995)
• Activity Theory (Nardi 1996)
• Consistency (Shneiderman 1988)

+
Computer Science
• Objected-Oriented UI (Collins 1995)
Practice Foundation Complete System
Science starts here
 Learn about an innovative IxD method
Deliver designs faster/smarter Deliver optimal UX designs
• Fewer iterations • Minimum number of screens
• Minimize feature creep rework • Shortest flows
• Fewer stakeholder meetings • Lowest cognitive load possible
• Science based tradeoff discussions • Ready to scale for next version
(not opinions, not trial & error)

10X 10X
Efficiency Effectiveness
(your labor) (design quality)
Can you support the 10X claim?

Case studies in
the book

= Over a 100X
Improvement!
 Before - Can you support this 10x claim?

800+ screens
 After - Can you support this claim?
= An 18X
Improvement!

45 screens (5 main ones)

 When Chapter 1

At the heart of the User Centered Design step.

Focus – Only Interaction Design step
(not full UCD life cycle)
• Other phases are important too
• Necessary but not sufficient
 Why Chapter 1
Because Design Darwinism does not work!

Current design methods don’t scale to complex systems,

are inefficient and often lead to product failure at
significant expense because

Real quality is not achieved by eliminating defects – occurs through design

*Design Darwinism = Believing A/B testing is a legitimate form of iteration

 How (theory)
Chapter 2

Leverage 2 cognitive science principles at 4 modular levels of

Interaction Design
 Cognitive Science Foundation of Semantic
Interaction Design
1. Language is the basis of conscious thought

2. Language grammar correlates with cognitive complexity

– Cognitive load in Interaction Design can be measured (lab)
–
NEW Cognitive load for Interaction Design can be predicted in advance
Simple Graphical User Interface (GUI) Semantic
Example (Interaction Design Grammar)
 Consistency vs. Cognitive Load

ACTIONS
Cut Remove Copy Duplicate Paste Stamp Print Publish
Character X X X X
OBJECTS

Word X X X X
Paragraph X X X X
Page X X X X
Document X X X X

Sparse Object-Action Grammar D!

BA
is
rse
a
Sp
Consistency vs. Cognitive Load

ACTIONS
Cut Copy Paste Print
Character X X X X
OBJECTS

Word X X X X
Paragraph X X X X
Page X X X X
Document X X X X

Dense Object-Action Grammar

Consistency vs. Cognitive Load Calculation
Real Conceptual Model 10x deconstruction in practice!

800 45
Screens Screens

Only 5 primary screens

Create Update Void Accept Reject Delegate Approve Refer Transfer Pin
Patient X X X X X X
Medical record X X X X X X X X X X
Treatment plan X X X X X X X X
Appointment X X X X X X
Task X X X X X X X X X
Messages X X X X X X X
Note X X X X X X
Care team X X X X X X
Only 10 objects and 10 actions can describe an entire EMR solution
 How (practice)
Chapters 4–7
Apply Semantic Interaction Design across all 4
modular framework levels

User Flow, Micro-Interaction Design

Layer 1- Grammar
Where do Objects and Actions come from?
Linguistic structure can be mapped to experience design as follows:

Language construct UX construct Example

Noun Object Box

Verb Action Copy

Adjective Attribute Color (Yellow)

Process
Conversion of User Stories into conceptual model
Do the math! – 7X difference in cognitive load

Table 4 calculation: Objects x Actions = 66

Table 5 calculation: Objects + Actions = 9
9 divided by 66 is approximately 14% or 7.3X improvement
 Figure out the Object Attributes!
Animal Money People Services Event
Species Donation Employee Medical type
Age Sponsorship Customer Adoption time
Breed Volunteer Board M. Membership place
Health Vehicle Vet Volunteer cost
Personality Volunteer duration
Exercise attendees
Kids friendly
Other pets

Attributes don’t add significant cognitive load!

They rely on recognition not recall (the two types of human memory)
 Prioritization!
Task = Object + Action(s) combination

Object-Actions pairs don’t have the same level of importance

– To the user
– To the business model
 Make Common Tasks Easily Accessible
• Two dimensions to evaluate “common”:
– Frequency (how often)
– Volume (how many)

By Many By Few
Most users will do Few users do task,
Frequent task often but those who do do
it often
Most users do task, Few users do task,
Rare but only rarely and only rarely

Micro revenue Macro revenue

$ Financial generation (e.g. generation (e.g. paid
advertising) subscription)
 Use this knowledge immediately

Actions + Objects 1. Alternative Heuristic evaluation method

Cognitive Load

• assess cognitive load

• find mismatches between user mental model and UX
Functionality
2. UX Design method of new projects
• Minimize screen count and complexity (increase
Versus quality)
• Prioritize tasks before creating mocks (save time)
Actions X Objects
3. UX Evolution of existing products
Cognitive Load

• Add features as new attributes of existing objects

• Minimizes complexity
Functionality • Slows growth of cognitive load (due to feature creep)
Layer 2 - Visualization
Visualization Deconstruction

All pattern levels are

optimized to present
and manipulate
actions, objects and
attributes in different
ways
 Building a Calendar Page UX
Components Widget (little) Widget (bigger) Calendar Archetype Screen

Grid Day/Week views

Buttons 35 buttons Change month icons
labels Show selection Add button
42 labels Search box
“Today” button
Color coding
Visualization Deconstruction
 Component Patterns
Name Attribute Action Object Logic/Behavior Use today:
Label x x x The name attribute of anything, in hypertext a navigate action

Radio button X Selection of on state within a set of many attributes Are you choosing
Toggle Button X On/off state for only one attribute
the optimal
component to
Checkbox X One of many states of given attribute
express your
Drop list X A collection of attribute values
attributes?
Combo box X X A collection of attribute values with the action to add a new attribute in place

Value Slider X Point and click way to choose the numeric value of an attribute

Button X Click to execute action it represents

Hypertext link X Special case of button with only action to be view/navigate

Don’t use trial &
Icon x X Mostly for actions, some can show visual state error or intuition
Field (value) X Type a value which represents an attribute (typically on of many) for a parent object

Text entry box X x Mostly for annotation which is an attribute

Menu x X Mostly for actions but sometimes change the state of an attribute

Tabs x X Used for granular unit of division for functionality

Legend: X (Primary) x (Secondary)

Example – Menu as Actions versus Objects

Menu of
Actions related
to profile

Menu of Jobs (Objects)

 Bad Example
Tabs = 2
Attributes
and 1
Action
Visualization Deconstruction
 Widget Patterns
Name Attribute Action Object Logic/Behavior
Use today:
List X Table with one column (can be horizontal like a Carrousel or Cover flow)

Table x X Table itself or each row can be an object

Are you
Tree control X Represents hierarchy of Object
choosing the
H-Grid Combination of tree and table, so set attributes be shown as well
optimal widget
Master-detail X X Combination of list of objects and its attributes or Attributes and sub-attributes
to express your
Form x X Logically represents an object, fields and controls inside are attributes
objects?
Card x X Expanding (Detail)/Collapsing (Summary) element used for a series of object.

Chat box x X Conversation is an object as are the people you have it with

Filter panel X X Uses the action of turning on/off specific attributes to filter data

Shuttle control X Two list boxes side by side that allow the selection of objects
Don’t use trial &
Picker X Visual palette for colors, fonts, shadow and other visual attributes error or intuition!
Context maybe one object, purposes is to set attributes and commit on final
Wizard x X action
Property sheet X Container of attributes of given object type

Leader board X X Gamification list of object (people or avatars) sorted in top to worst order

Media control X Stop, start, fast forward audio or video

Legend: X (Primary) | x (Secondary)
Card Control – Semantic error example

Many attributes inside card

Local action
(CM transactional)

Local action
(not grammar relevant)
 Non-grammatic Widgets
Time Container only
– Feed – Dialog box
– Media controls (VCR)
– Toolbar
– Timeline
– Breadcrumbs – Portlet (html
– Search box iFrame)
– Card
Space – Page
– Zoom controls – Screen
– Pan
– Map
– Search results
Visualization Deconstruction
 Screen Archetype Patterns
Name Attribute Action Object Logic/Behavior
Use today:
Menu page x X Object usage can have implicit or explicit selection, to display actions mostly a list

Catalog X X X Select item to purchase (put in cart) Are you

Funnel X X X eCommerce “Shopping Cart” to pay and ship. As seen in tax preparation apps. choosing the
Desktop X X X Select item to object to open/run optimal
Portal X X iFrames represent object, content within attributes archetype to
Workflow X X X Multiple step process spanning a single object, actions and attributes embedded express your
Social X x x Organized content over time, mostly attributes of people, people are the object Interaction
Container organizer X X Master-Master level for object, detail are content of object plus search and create
Design system?
Tool & Canvas X X X Objects in canvas, (could be simple box or full engineering CAD drawing)
Don’t use trial &
Workspace X X X Combines editors, prop sheet and toolbars error or intuition
Administration tool X X X Object on the left in hierarchy, props on right

Dashboard X X Frames represent objects, content inside are attributes, filters in panels or toolbars

Report X X Report is the parent object, content is attributes of same

Calendar X x X Representation of time in a grid or linear layout (top to bottom or left to right)

Legend: X (Primary) x (Secondary)

 Archetype: Social (Feed and suggestions)
Global
Actions
Local
Object Actions
Objects
Local
Action Object Actions

Attribute Local
filters Actions
Content = Attribute

Actions
Local
Actions

Local
Feed Widget in the center Actions
Visualization Deconstruction

Use today:

Are you consistently

expressing the
grammar across all
your design
decisions?
 Medical Interaction Design Language
examples

1. Behavior the Appointment Object inside different archetypes

2. Consistent use of the Void Action to remove errors

A day in the life of…

 EMR Conceptual Model
Create Update Void Accept Reject Delegate Approve Refer Transfer Pin

Patient X X X X X X
Medical record X X X X X X X X X X

Treatment plan X X X X X X X X

Appointment X X X X X X
Task X X X X X X X X X
Messages X X X X X X X
Note X X X X X X
Care team X X X X X X
 Interaction Design Language: Object
Example - Appointment

Appointment card Card widget in

has explicit swim lane –
patient (Object) collapsed state
context

Swimlane Archetype Pattern

Interaction Design Language: Object
Example - Appointment

Click to expand
widget for
details

Local action
menu

Swimlane Archetype Pattern

Interaction Design Language: Object
Example - Appointment

Local actions assume

patient object context

Swimlane Archetype Pattern

Interaction Design Language: Object
Example - Appointment

Same widget
design
pattern for
appointment

Calendar Archetype Pattern

Interaction Design Language: Object
Example - Appointment

Local action menu

context assume
Appointment object

Calendar Archetype Pattern

Interaction Design Language: Object
Example - Appointment

Next upcoming
appointment
summary in widget

Portal Archetype Pattern

 Conceptual model
Create Update Void Accept Reject Delegate Approve Refer Transfer Pin
Patient X X X X X X
Medical record X X X X X X X X X X
Treatment plan X X X X X X X X
Appointment X X X X X X
Task X X X X X X X X X
Messages X X X X X X X
Note X X X X X X
Care team X X X X X X
Interaction Design Language: Action
Example – Void Data

Void action – removes mistake. Data is

never “deleted” in an EMR for forensic
reasons

Canvas Archetype Pattern

Interaction Design Language: Action
Example – Void Data

View control checked

Strikeout
visualization for
void data

Canvas Archetype Pattern

Interaction Design Language: Action
Example – Void Data

View control not checked

Voided Diagnosis card

widget removed from
canvas

Canvas Archetype Pattern

 Interaction Design Language: Action
Example – Void Data

Void action can be applied to

individual attributes at a
detailed level within the
diagnosis
Two level
tab widget
embedded
inside card Side note:
widget Click to expand card “implicit
action” behavior is identical
to appointment card widget

Canvas Archetype Pattern

Interaction Design Language: Action
Example – Void Data

View control not checked

Voided Diagnosis card widget

removed from canvas

Canvas Archetype Pattern

Interaction Design Language: Action
Example – Void Data

Same action design pattern

Menu (list) Archetype Pattern

Interaction Design Language: Action
Example – Void Data

Same action design

pattern

Local action
menu

Social (Clinical Notes - Feed) Archetype Pattern

Interaction Design Language: Action
Example – Void Data

Same action design

pattern

Local action
Strikeout
menu disabled
visualization for void
rule
data

Social (Clinical Notes - Feed) Archetype Pattern

Visualization Deconstruction

Use today:

Have you chosen the

most effective UX
architecture to
navigate across all
your primary objects?
 UX Architecture

Sequential

Hierarchical
Hub & Spoke

Matrix Network
 UX Architecture – Performance characteristics

Sequential Hierarchical Hub & Spoke Matrix Network

Location Awareness High Medium Medium Medium Low-Med

Visual Search High High High Medium Med-High
Effectiveness
Task Speed Low Low Low Low-Med Low-Med
Working Memory Load Low Medium Medium Medium Med-High
Learnability Easy Fast & Easy Fast & Easy Medium Low-Med
Effectiveness High High High Medium Med-High
Efficiency Low Low Low High High
Satisfaction Medium Med-High Med-High Low-Med Medium
 Who: Digital Natives recognize Interaction
Design patterns
• 5B of the 7.7B people on earth interpret GUI design patterns as Actions
and Objects (Interaction Design Grammar)
– Thanks to the internet and smart phone global penetration

• The other 2.7B whether literate or not…

– Must linguistically associate physical world metaphors with on screen
objects and actions to participate in HCI
– They will be (self) taught based on their own natural language of nouns
and verbs to understand what any new tool can do

99% of the digital product and service economy are in the first 5 Billion
Layer 3 - Flow
Use today:

Are you minimizing the number

of steps and screens across all
your entire product system?
 Actions propel Objects through Flow
Actions
Objects Donate Adopt Schedule Share Learn

Dog X X X X X

Owner X X X X X

Organization X X X X

Money X X

Most BOXES will represent OBJECTS

Most LINES will represent ACTIONS

Simple Donation Flow

Actions propel Objects through Flow
Organization Object Owner
Object

Volunteer
User
Signup
Screen Profile
Screen

Pay Member?
Home Donate Action Y
Screen CTA Donate Thank you
B1 Button Screen D1 Screen N

Multiple Money Organization

Objects Object Object Home
Screen
H2

Simple Donation Flow Multiple

Objects
Layer 4 - Game Theory
Use today:

Can you motivate and

guide human behavior to
favor specific object-
action pairs (aka tasks)?
 Game Theory for Interaction Design is part
of UX Magic
Captology
• Using technology to
persuade user to
Gameful Interaction change behavior
Design
• Substitution of a game
as a proxy for a
transactional task
Gamification • Intrinsically fulfilling
• Addition of reward • Generate feeling of
elements to competence &
incentivize autonomy
Pure Games
productive work
• Entertainment
• Learning/Education (war)
• Voluntary engagement
• Rules and competition (not free play)
• Lack of consequences
• Compelling narrative!
 Mapping to UX Magic – Conceptual Models

• Gamification - provides Action incentives

• Gameful Interaction Design - utilizes Object substitution
• Captology focuses on Attribute manipulation
– When used in tools and online services
 Reasons to Apply Game Theory in
Interaction Design
Increasing Decreasing
Productivity Errors
Performance Boredom during repetitive tasks
Satisfaction Unsafe behavior
Sales/Revenue Energy consumption
Community size Gambling
Health and well-being Conflict
Sustainability Absenteeism

Incentivizing user tasks (object-action pair)

 Human Motivators to target
Intrinsic Extrinsic
Autonomy Benefits
Belonging Bonus/Reward
Creativity Competition
Curiosity Fear of failure
Learning Fear of loss
Love Fear of punishment
Mastery Greed
Meaning Praise
Pleasure Status
 Game Intrinsic Extrinsic
Mechanics
Autonomy Benefits
Achievement
Belonging Bonus/Reward
Connection
Creativity Competition
Constraints
Curiosity Fear of failure
Feedback
Learning Fear of loss
Narrative
Love Fear punishment
Ownership
Mastery Greed
Privilege
Meaning Praise
Reciprocity
Pleasure Status
Recognition

Role play
Self
Expression Team competition
Sports
(Social connection)
Urgency
Game Theory is everywhere
Interaction Design can outperform medication

BlueStar – Diabetes Solution from WellDoc Inc.

 End of Preview
• What
• When
• Why
• How

dan@[Link] Semantic Interaction Design

Semantic IxD – Only Interaction Design step

Don’t forget to do the other UCD Steps!

 Semantic Interaction Design Framework
We have completed our preview of the Semantic Interaction Design core – but there is much more….

Chapter 10 – Chess (future of UX design & designers)

Chapter 9 – Myth (UI style guide applicability)
Chapter 8 – Elegance (graphic design for Semantics)

Preview Chapters 4–7

Preview Chapters 1–3

 UX Magic – Book by the Interaction
Design Foundation
We only scratched the surface of the
Semantic Interaction Design topic.

To dig in deep:
• Get the book
• Study it
• Do the related exercises
• Teach the method to others…

You can purchase this new book from the Interaction Design Foundation on Amazon
 Addendum
For individuals:

• If you found this new Semantic IxD approach compelling please tell your UX colleagues at
other companies about it. Post on social media and help spread the word within the global
UX design community.

• Join the Interaction Design Foundation (the publisher) and learn more about IxD in general…

For UX leaders/manager:

• I have been giving corporate training classes on this the Semantic IxD method for years
• It includes how it integrates into Data Visualization and Information Architecture which are
not covered in the book.

dan@[Link]
The End – Q&A
dan@[Link]