USOO7469381 B2

(12) United States Patent (10) Patent No.: US 7.469,381 B2

Ording (45) Date of Patent: Dec. 23, 2008
(54) LIST SCROLLING AND DOCUMENT 6,489,951 B1 12/2002 Wong et al. ................. 345,173
TRANSLATION, SCALING, AND ROTATION 6,567,102 B2 5/2003 Kung ......................... 345,660
ON A TOUCH-SCREEN DISPLAY

(75) Inventor: Bas Ording, San Francisco, CA (US)

Continued
(73) Assignee: Apple Inc., Cupertino, CA (US) (Continued)
FOREIGN PATENT DOCUMENTS
(*) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 EP O 635 779 A1 1, 1995
U.S.C. 154(b) by 0 days.
(21) Appl. No.: 11/956,969
Continued
(22) Filed: Dec. 14, 2007 ( )
OTHER PUBLICATIONS
(65) Prior Publication Data
US 2008/O168404 A1 Jul. 10, 2008
ul. TU,
Microsoft Word 2003 Screen Shots.
Continued
Related U.S. Application Data ( )
60) Provisional application No. 60/937,993, filed on Jun. Primary Examiner Boris Pesin
(60) filed
S"S"ENEE
on Jun. 28, 2007, provisional application No.
(4) Attorney, Agent, or Firm Morgan, Lewis & Bockius
LLP
60/945,858, filed on Jun. 22, 2007, provisional appli
cation No. 60/879,469, filed on Jan. 8, 2007, provi- (57) ABSTRACT
sional application No. 60/883,801, filed on Jan. 7,
2007, provisional application No. 60/879,253, filed on
Jan. 7, 2007. In accordance with some embodiments, a computer-imple
s mented method for use in conjunction with a device with a
(51) Int. Cl. touch screen display is disclosed. In the method, a movement
G06F 3/0 (2006.01) of an object on or near the touch screen display is detected. In
(52) U.S. Cl. ....................... 715/702; 715/764; 715/863; response to detecting the movement, an electronic document
715/864; 715/769 displayed on the touch screen display is translated in a first
(58) Field of Classification Search ................. 715/764, direction. If an edge of the electronic document is reached
715/769,702, 863, 864 while translating the electronic document in the first direction
See application file for complete search history. while the object is still detected on or near the touch screen
(56) References Cited display, an area beyond the edge of the document is displayed.
After the object is no longer detected on or near the touch
U.S. PATENT DOCUMENTS screen display, the document is translated in a second direc
5,495,566 A 2/1996 Kwatinetz ................... 395,157
tion until the area beyond the edge of the document is no
5,844,547 A 12, 1998 Minakuchi et al. .......... 345,173
longer displayed.
5,867,158 A 2, 1999 Murasaki et al. ............ 345,341
6,034,688 A 3/2000 Greenwood et al. ......... 345.353 20 Claims, 38 Drawing Sheets
Portals utifurctic Device
208 1. y

390A

-1402 Currentti me 34 306

Welcome 394
http:iiwww.company,Comistart 3:08 ( falo
Block
3914-7

3912
 US 7.469,381 B2
Page 2

U.S. PATENT DOCUMENTS 2006/0181510 Al 8/2006 Faith .......................... 345,158

2006/0187215 A1 8/2006 Rosenberg etal 345,173
6,661.409 B2 12/2003 Demartines et al. ......... 345,173 2006/0197753 A1 9/2006 Hotelling .................... 345,173
6,690,387 B2 * 2/2004 Zimmerman et al. ........ 345,684 2006/0238495 A1 10, 2006 Davis ......................... 345,156
6,707.449 B2 3, 2004 Hinckley et al. ..... ... 345,173 2007,0008066 A1 1/2007 Fukuda .... 340,552
6,809,724 B1 10/2004 Shiraishi et al. ... 34.5/169 2007/0024646 A1 2/2007 Saarinen et al. ............. 345,660
6,907,575 B2 6/2005 Duarte ....... 715,784 2007/0046646 A1 3/2007 Kwon et al. ................ 345,173
6,912.462 B2 6/2005 Ogaki ..... 701,208 2007, OO67745 A1 3, 2007 Choi et al. T15,863
6,972,776 B2 12/2005 Davis et al. .... ... 345,684 2007/0109275 A1 5/2007 Chuang ...................... 345,173
6,975,306 B2 12/2005 Hinckley et al. ............ 345/173 2007/O120835 A1 5/2007 Sato ........................... 345,173
7,009,599 B2 3/2006 Pihlaja ............. ... 345,173 2007/O150826 A1 6/2007 Anzures et al. . 71.5/772
7,046.230 B2 5/2006 Zadesky et al. ... 345,156 2007/0150842 A1 6/2007 Chaudhri et al. ............ T15,863
7,075,512 B1 7/2006 Fabre et al. .... ... 345,156 2007/0152978 A1 7/2007 KOcienda et al. ............ 345,173
7,102,626 B2 9/2006 Denny, III ... ... 345,179 2007. O152979 A1 7, 2007 JobS et al. ... 345,173
7,154,534 B2 12/2006 Seki et al. ... 348,207.1 2007/015.5434 A1 7/2007 JobS et al. ................... 455,565
7,155,048 B2 12/2006 Ohara ........ ... 382,132 2007/0156364 A1 7/2007 Rothkopf .................... 702/117
7,181,373 B2 2/2007 Le Cocq et al. ... 703.1 2008/O104544 A1* 5/2008 Collins et al. ............... T15,846
7,184,796 B2 2/2007 Karidis et al. ..... ... 455,566
7,240,291 B2 7/2007 Card et al. ..... ... 715,776 FOREIGN PATENT DOCUMENTS
2003/0095135 A1 5/2003 Kaasila et al. . ... 345,613
2003/0095697 A1 5, 2003 Wood et al. ... ... 382/131 EP O 701 220 A 3, 1996
2003/0184525 A1 10, 2003 Tsai ........... 345/173 EP O 880 091 A2 11/1998
2004/0021676 A1 2/2004 Chen et al. 345/684 WO WO O2/O1338 A1 1/2002
2004/0027398 A1* 2/2004 Jaeger ........ ... 345/863 WO WOO3/060622 A2 7, 2003
2004/0080541 A1 4/2004 Saiga et al. .... 345.80s WO WO 2005/052773 A2 6/2005
2004/0155888 A1 8/2004 Padgitt et al. .. so WO WO 2006/003591 A 1/2006
2004/0263.486 All 12/2004 Seni ........... 4517 WO WO 2006/020305 A2 2/2006
2005, OO12723 A1 1/2005 Pallakoff ... 345,173
2005/0145807 A1 7/2005 Lapstun etal 250,566 OTHER PUBLICATIONS
2005/0168488 A1 8/2005 Montague ... ... 345,659 Photo Mesa 3.1.2 2006 Screen Shots.
2005/O198588 A1 9, 2005 Lin et al. .... ... 15,784 International Search Report and Written Opinion for International
2005/0237308 A1 10, 2005 Autio et al. . ... 34.5/173 Application No. PCT/US2008/050292, mailed Sep. 19, 2008.
2005/0270269 A1 12, 2005 Tokkonen ... ... 345,156 IBM, “Scroll Control Box, IBM Technical Disclosure Bulletin, vol.
2005/0275618 A1 12/2005 Juh et al. ..., , 345,156 38, No. 04, Apr. 1993, pp. 399-403.
2006, OO25218 A1 2, 2006 Hotta ............ ... 463,37 Tidwell et al., “Magnetism.” Designing Interfaces, Nov. 2005, Sec
2006/0026521 A1 2/2006 Hotelling et al. 7157702 tion 85.
2006, OO48073 A1 3, 2006 Jarrett et al. ... 15,784 Invitation to Pay Additional Fees for International Application No.
2006/0055669 A1 3/2006 Das ............ 345,156 PCT/US2008/050292, mailed Jul 18, 2008.
2006, OO61551 A1 3/2006 Fateh ......................... 345,158 Miller, D., "Personal Java Application Environment.” Sun
2006, OO77544 A1 4/2006 Stark .......................... 359,448 MicroSystems, http://java. Sun.com/products/personalava touch
2006/0082549 A1 4/2006 Hoshino et al. ... 345,157 able?, Jun. 8, 1999, 13 pages.
2006/00945.02 A1 5/2006 Katayama et al. ............. 463,31
2006/0156249 A1 7/2006 Blythe et al. ................ 715,781 * cited by examiner
U.S. Patent Dec. 23, 2008 Sheet 1 of 38 US 7.469,381 B2

Memory Portable Multifunction Device

102 100
126 136
Operating System Applications (Continued)
128 148
Communication Module Calendar Module
130 149
Contact/Motion Module Widget Modules
132 149-1
Graphics Module Weather Widget 149-2
134
Text input Module Stocks Widget 149-3
135
GPS Module 136 Calculator Widget 149-4
Applications 137 Alarm Clock Widget 149-5
Contacts Module Dictionary Widget
138
Telephone Module 139 149-6
Video Conference Module
140
User-Created Widget(s) 150
E-mail Client Module
141 Widget Creator Module 151
nstant Messaging Module 142
Search Module
Blogging Module 143
Camera Module 162
144 Power
Image Management Module 145
Video Player Module 146
Music Player Module 147 External 124
Browsing Module Port
118
103 O3
104 RF circuitry u? 108
122 Controller
Speaker
Peripherals 103 111
Interface Audio
Circuitry
(
103
120 Microphone
Proximity 113
Sensor 166

I/O Subsystem 158

106
Display Optical Other input 160
Controller Sensor(s) Controller(s)
Controller
156-1
103 103 103

Touch-Sensitive
Optical Other input 116
112 Display System Sensor(s) Control
164 Devices

Figure 1
U.S. Patent Dec. 23, 2008 Sheet 2 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Optical Proximity
Speaker 111 SensOr 164 SensOr 166

TOUCh SCreen 11

- s

Microphone Home
113 204 /
^- ---

Figure 2
U.S. Patent Dec. 23, 2008 Sheet 3 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time M 308

310 \ Day and Date M 312
Wallpaper image
314

Slide to unlock

TOUCh SCreen 11

Microphone ? HOme
113 \ 204
>
U.S. Patent Dec. 23, 2008 Sheet 4 of 38 US 7.469,381 B2

Portable Multifunction Device

(3) (3) (3)

PhotoS
144

Jan
75° 15
Weather Stocks Blog Calendar
149-1 149-2 142 148
--- X × - ---

Y / User
- - X -- / ABC Created
N / 8. A \ a N Wi d 9 et -
Calculator Alarm Dictionary Widget
149-3 149-4 149-5 149-6

BrOWSer Music
147 146

Touch SCreen 11

Microphone Home
113 204

Figure 4
U.S. Patent Dec. 23, 2008 Sheet 5 Of 38 US 7.469,381 B2

500
502

Detect a movement of an object (e.g., a finger) on or near a touch screen

display of a device.
504
Scroll a list of items displayed on the touch screen display in a first
direction (e.g., vertical or horizontal).
Scroll the list at a speed corresponding to a speed of movement of 506
the object. |

Scroll the list in accordance with a simulation of an equation of motion 508

| having friction. |

- s 510
u- - ls a /s 512
- terminus Of the s /'
u- list reached while Scrolling the is No PrOCeSS
<s list in the first direction while the object u- D-D Complete
s is still detected On Or near -
s the touch SCreen u
is display? -

514

516

518

520
After the object is no longer detected on or near the touch screen display,
scroll the list in a second direction opposite the first direction until the area
beyond the terminus of the list is no longer displayed.
522

Make the terminus of the list appear to be elastically attached to an 524

edge of the touch screen display or to an edge displayed on the touch
| screen display. |

Figure 5
U.S. Patent Dec. 23, 2008 Sheet 6 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404 406

X Y-3502 - 3504
Mailboxes Inbox
X. -/
6 -

3512
Bruce Walker"506-2January 30, 2006 351 0-2
Lunch meeting today'? 3530
351O-3
Kim Brook." 3508-3
January 30, 2006
Draft agreement
Bob Adams January 29, 2006
Project Orion 3514
3512-2
Darin Adler January 29, 2006
fantasy football
Aaron Jones January 29, 2006
Fwd: status report
Kim BrOOk January 28, 2006
Re: proposal

Figure 6A
U.S. Patent Dec. 23, 2008 Sheet 7 Of 38 US 7.469,381 B2

Portable Multifunction Device

206 100

Current Time 404 IC 406

w
. . ... c.3502 a 3504
&
Mailboxes” Inbox (6
3506-1 3510-1
Aaron Jones' 3508-1 January 30, 2006
Re: Project Oriori 3534
3512-1 3506-2 3510-2
db Bruce Walker January 30, 2006
Lunch meeting today' 3530
351O-3
Kim Brook 3508-3: January 30, 2006
Draft agreement 3514
Bob Adams January 29, 2006
Project Orion W
3512-2
Darin Adler January 29, 2006
fantasy football
Aaron Jones January 29, 2006
Fwd: status report

Microphone Home \
113 \ 204 /

Figure 6B
U.S. Patent Dec. 23, 2008 Sheet 8 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

1402 Current Time 404

( . . ... 3502
Mailboxes Inbox
3538
536
3506-1 351O-1
Aaron Jones
Re: Project Orion'
January 30, 2006
3534
3512-1 3506-2 3510-2
d Bruce Walker January 30, 2006
Lunch meeting today? 3530
351O-3
Kim Brook. January 30, 2006
Draft agreement 3514
Bob Adams January 29, 2006
Project Orion W
Darin Adler January 29, 2006
fantasy football

Microphone Home
113 204 /

Figure 6C
U.S. Patent Dec. 23, 2008 Sheet 9 Of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404 C. 406

-

. . ... -3502 - 3504

x,
Mailboxes” Inbox ^
6 -

351O-1
Aaron Jones
.
"'January 30, 2006
.3508-1
Re: Project Orion 3534
3512 506-2
Bruce Walker January 30, 2006?"
Lunch meeting today' 3530

Kim Brook”. January 30, 2006"

Draft agreement
Bob Adams January 29, 2006
Project Orion
2 Darin Adler January 29, 2006
fantasy football
Aaron Jones January 29, 2006
Fwd: status report

Figure 6D
U.S. Patent Dec. 23, 2008 Sheet 10 of 38 US 7.469,381 B2

700
702

Detect a movement of an object (e.g., a finger) on or near a touch screen

display.
704
Translate an electronic document displayed on the touch screen display in
a first direction (e.g., vertical, horizontal, or diagonal).
Translate the electronic document at a speed of translation 706
corresponding to a speed of movement of the object.
Translate the electronic doCument in acCOrdance With a simulation of 708
| an equation of motion having friction.

- s 710
- ls an s /
u- edge of the s /1 712
- electronic document reached is
u- while translating the electronic document in N Process
is the first direction while the object is still u- Complete
s detected On Or near u
is the touch screen -

71.4

716

718

720
After the object is no longer detected on or near the touch screen display,
translate the document in a second direction (e.g., opposite the first
direction) until the area beyond the edge of the document is no longer
displayed.
722
Translate the document using a damped motion.
Make the edge of the electronic document appear to be elastically 724
attached to an edge of the touch screen display or to an edge
displayed on the touch Screen display.

Figure 7
U.S. Patent Dec. 23, 2008 Sheet 11 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404

WelCOme 3904

http://www.company.com/start 3908

BOCk 3
3914-3

BOCk 4
3914-4
V
M
V
BOCk 5
3914-5

/ y
Microphone Y / Home
113 204 /
--- -

Figure 8A
U.S. Patent Dec. 23, 2008 Sheet 12 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

BOCk 1 BOCK 7
3914-1
Block 4 3914-7
3914-4

BOCK 5
3925.3914-5

Microphone (Hg)
113 204

Figure 8B
U.S. Patent Dec. 23, 2008 Sheet 13 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404

Welcome 3904

http://WWW.Company.com/start 3908
3914-1 Block 4 BOCk 7
3914-4 3914-7

BOCk 5
3914-5

w
M
W
W
Blockg
3914-6

/ N
Microphone HOme
113 204 X

Figure 8C
U.S. Patent Dec. 23, 2008 Sheet 14 of 38 US 7.469,381 B2

Portable Multifunction DeVice

206 100

Current Time 404

WelCOme 3904

http://WWW.Company.com/start 3908

BOCk 1 BOCK 7
3914-1
Block 4 3914-7
3914-4

Block 5
3914-5

Microphone ( Home
113 204
--- ---

Figure 8D
U.S. Patent Dec. 23, 2008 Sheet 15 Of 38 US 7.469,381 B2

OO
902

Display an electronic document at a first magnification on a touch screen

display. The electronic document has a document length and a document
Width.

904

Detect a gesture (e.g., a pinching gesture) on or near the touch screen

display corresponding to a command to Zoom out by a user-specified
amount.

906

In response to detecting the gesture, display the electronic document at a

magnification less than the first magnification.

u- ls the s -910
u- document length or s
u-
s
document width entirely displayed while No
>->
Process
st the gesture is still detected on or near ul- Complete
s the touch SCreen u
N display? u

912

Display the electronic document at a magnification wherein areas beyond

opposite edges of the electronic document are displayed.

914

Upon detecting termination of the gesture, display the electronic

document at a magnification wherein the areas beyond opposite edges of
the electronic document are no longer displayed.

Figure 9
U.S. Patent Dec. 23, 2008 Sheet 16 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404

WelCOme 3904

http://www.company.com/start 3908

Block 3
3914-3

es Block 4
3951
Y 6. 3914-4

Block 5
3914-5

Microphone (Hiye
113 - 204 /
Y- -

Figure 10A
U.S. Patent Dec. 23, 2008 Sheet 17 Of 38 US 7.469,381 B2

Portable Multifunction Device

100

Current Time 4.
Welcome 3904

http://WWW.COmpany.Com/start 3908

3955
3955

Block 4
3914.4
BOCk 5
up to
3914-5
395
Block 2 a.
3914-2

Microphone He
113 1. 204 /
Figure 10B
U.S. Patent Dec. 23, 2008 Sheet 18 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404

WelCOme 3904

http://WWW.Company.com/start 3908 (
Web page
3912

Microphone ? Home
113 \ 204 /

Figure 10C
U.S. Patent Dec. 23, 2008 Sheet 19 Of 38 US 7.469,381 B2

1102

Display at least a first portion of an electronic document at a first

magnification on a touch screen display.

1 104

Detect a gesture (e.g., a de-pinching gesture) on or near the touch screen

display corresponding to a command to Zoom in by a user-specified
amount.

1106

In response to detecting the gesture, display decreasing portions of the

electronic document at increasing magnifications.

u-1 - - 1110
u- Upon detecting s N
u-1 termination of the gesture, does the No PrOCeSS
is magnification exceed a predefined u- Complete
s magnification? u1

Yes
1112

Display a respective portion of the electronic document at the predefined

magnification.

Figure 11
U.S. Patent Dec. 23, 2008 Sheet 20 of 38 US 7.469,381 B2

Portable Multifunction Device

100

Current Time 404

Welcome 3904

http://www.company.com/start 908

Block 3
3914-3
Block 1893
Y P

alty
3933
Block 4
e is 12
3914-4

BOCK 5
3914-5

N
Microphone Home
113 \ 204 /
X

Figure 12A
U.S. Patent Dec. 23, 2008 Sheet 21 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404

Welcome 3904

http://www.company.com/start 3908

Web page
3912
893

M
/
a
3933P
BOCk 1
3914-1

/
Microphone
113
f
He
204

Figure 12B
U.S. Patent Dec. 23, 2008 Sheet 22 of 38 US 7.469,381 B2

Portable Multifunction Device

2O6 100

Current Time 404

Welcome 3904

http://WWW.Company.com/start 3908

Web page
3912

Microphone
113
He
204 ,
X.
-- ---

Figure 12C
U.S. Patent Dec. 23, 2008 Sheet 23 of 38 US 7.469,381 B2

/
-
i
39 70 9.
(S)-eleuoueeoow euoH euOudo.JOW

99 JOSueS 79 JOSueSY (i.

U.S. Patent Dec. 23, 2008 Sheet 24 of 38 US 7.469,381 B2

39 70% LL
(S)-eleuoueeoow euOH euOudo.JOIN

99 JOSueS 79 JosueS o
|eodo Jeseeds
U.S. Patent Dec. 23, 2008 Sheet 25 Of 38 US 7.469,381 B2

39 70% 9 L
(S)-eleuoueeoow euOH euOudo JoW

99 JosueS 79 JOSueSY /
U.S. Patent Dec. 23, 2008 Sheet 26 of 38 US 7.469,381 B2

1402

Detect a multifinger twisting gesture on or near a touch screen display.

The multifinger twisting gesture has a corresponding degree of rotation.

-
-- N
/- 1404
N
/1 1406
- DOes the N
- Corresponding degree of Yes Execute a 90
{ rotation eXCeed a D-D screen rotation
predefined degree of - COmmand.
N
rotation? -
N -
No
1408

Execute a screen rotation Command with an acute angle of rotation.

1410

Upon ceasing to detect the multifinger twisting gesture, executing a

Screen rotation Command with an angle of rotation opposite to the acute
angle.

Figure 14
U.S. Patent Dec. 23, 2008 Sheet 27 Of 38 US 7.469,381 B2

Optical Proximity
Speaker 111 SensOr 164 SensOr 166

To Figure 15B

Microphone Home Accelerometer(s)

113 204 168

Figure 15A
U.S. Patent Dec. 23, 2008 Sheet 28 of 38 US 7.469,381 B2

Optical Sensor Proximity

Home Accelerometer(s)

Figure 15B
U.S. Patent Dec. 23, 2008 Sheet 29 Of 38 US 7.469,381 B2

Optical Proximity
Speaker 111 Sensor 164 Sensor 166

Microphone Home Accelerometer(s)

113 204 168

Figure 15C
U.S. Patent Dec. 23, 2008 Sheet 30 of 38 US 7.469,381 B2

Optical Proximity
Speaker 111 Sensor 164 Sensor 166

To Figure 15E

Microphone Home Accelerometer(s)

113 204 168

Figure 15D
U.S. Patent Dec. 23, 2008 Sheet 31 of 38 US 7.469,381 B2

Optical Proximity
Speaker 111 SensOr 164 SensOr 166

Microphone Home Accelerometer(s)

113 204 168

Figure 15E
U.S. Patent Dec. 23, 2008 Sheet 32 of 38 US 7.469,381 B2

Optical Proximity
1604-L Speaker 111 Sensor 164 Sensor 166

To Figure 16B

Microphone Home Accelerometer(s) 1604-R

113 204 168

Figure 16A
U.S. Patent Dec. 23, 2008 Sheet 33 Of 38 US 7.469,381 B2

Figure 16B
U.S. Patent Dec. 23, 2008 Sheet 34 of 38 US 7.469,381 B2

Figure 16C
U.S. Patent Dec. 23, 2008 Sheet 35. Of 38 US 7.469,381 B2

To Figure 16E

Microphone Home Accelerometer(s) 1 604-R

113 204 168

Figure 16D
U.S. Patent Dec. 23, 2008 Sheet 36 of 38 US 7.469,381 B2

1604-L
Optical Proximity
Speaker 111 SenSOr 164 Sensor 166

To Figure 16F

Microphone Home Accelerometer(s) S

113 204 168 1604-R

Figure 16E
U.S. Patent Dec. 23, 2008 Sheet 37 Of 38 US 7.469,381 B2

Optical Sensor Proximity

Microphone Home Accelerometer(s)

113 204 168

Figure 16F
U.S. Patent Dec. 23, 2008 Sheet 38 of 38 US 7.469,381 B2

Memory 1770
Operating System 126
Communication Module 128
Contact/Motion Module
Graphics Module
134
TextInput Module 136
Applications 137
Contacts Module 138
Device 1700 Telephone Module 139
s VideO COnference MOdule 140
E-mail Client Module 141
1710 instant Messaging Module 142
CPU(s) Blogging Module 143
Camera Module
1720 144
Image Management Module 145
Video Player Module
1730 146
Music Player Module 147
User interface Browsing Module 148
Calendar Module 149
Tosen
ISplay 1740 Widget Modules 149-1
Weather Widge 149-2
Keyboard/Mouse Stocks Widget 149-3
Calculator Widget 149-4
1750 NetWOrk Alarm Clock Widget 149-5
Dictionary Widdet
1760 Communications ry 9
Interface - - 1496
User-Created Widget(s) 150
Widget Creator Module 151
Search Module

Figure 17
 US 7,469,381 B2
1. 2
LIST SCROLLING AND DOCUMENT to access, store and manipulate data. These conventional user
TRANSLATION, SCALING, AND ROTATION interfaces often result in complicated key sequences and
ON A TOUCH-SCREEN DISPLAY menu hierarchies that must be memorized by the user.
RELATED APPLICATIONS
Many conventional user interfaces, such as those that
include physical pushbuttons, are also inflexible. This is
This application claims priority to U.S. Provisional Patent unfortunate because it may prevent a user interface from
Application Nos. 60/937,993, “Portable Multifunction being configured and/or adapted by eitheran application run
Device.” filed Jun. 29, 2007; 60/946,971, “List Scrolling and ning on the portable device or by users. When coupled with
Document Translation, Scaling, and Rotation on a Touch 10 the time consuming requirement to memorize multiple key
Screen Display” filed Jun. 28, 2007; 60/945,858, “List sequences and menu hierarchies, and the difficulty in activat
Scrolling and Document Translation on a Touch-Screen Dis ing a desired pushbutton, such inflexibility is frustrating to
most uSerS.
play,” filed Jun. 22, 2007; 60/879,469, “Portable Multifunc
tion Device filed Jan. 8, 2007; 60/883,801, “List Scrolling As a result of the Small size of display screens on portable
and Document Translation on a Touch-Screen Display, filed 15 electronic devices and the potentially large size of electronic
Jan. 7, 2007; and 60/879,253, “Portable Multifunction files, frequently only a portion of a list or of an electronic
Device.” filed Jan. 7, 2007. All of these applications are document of interest to a user can be displayed on the screen
incorporated by reference herein in their entirety. at a given time. Users thus will frequently need to scroll
This application is related to the following applications: (1) displayed lists or to translate displayed electronic documents.
U.S. patent application Ser. No. 10/188,182, “TouchPad For Users also will need to rotate and to scale (i.e., magnify or
Handheld Device.” filed on Jul. 1, 2002; (2) U.S. patent appli de-magnify) displayed electronic documents. However, the
cation Ser. No. 10/722,948, “Touch Pad For Handheld limitations of conventional user interfaces can cause these
Device.” filed on Nov. 25, 2003; (3) U.S. patent application actions to be awkward to perform.
Ser. No. 10/643,256, “Movable TouchPad With Added Func Furthermore, Scrolling displayed lists and translating elec
tionality.” filed on Aug. 18, 2003; (4) U.S. patent application 25
tronic documents can be awkward on both portable and non
Ser. No. 10/654,108, “Ambidextrous Mouse.” filed on Sep. 2, portable electronic devices with touch-screen displays. A user
2003; (5) U.S. patent application Ser. No. 10/840,862, “Mul may become frustrated if the scrolling or translation does not
tipoint Touchscreen filed on May 6, 2004; (6) U.S. patent reflect the user's intent. Similarly, a user may become frus
application Ser. No. 10/903,964, “Gestures For Touch Sensi trated if rotation and Scaling of electronic documents does not
tive Input Devices.” filed on Jul. 30, 2004; (7) U.S. patent 30 reflect the users intent.
application Ser. No. 11/038,590, “Mode-Based Graphical Accordingly, there is a need for devices with touch-screen
User Interfaces For Touch Sensitive Input Devices' filed on displays with more transparent and intuitive user interfaces
Jan. 18, 2005; (8) U.S. patent application Ser. No. 11/057, for scrolling lists of items and for translating, rotating, and
050, “Display Actuator.” filed on Feb. 11, 2005; (9) U.S. Scaling electronic documents that are easy to use, configure,
Provisional Patent Application No. 60/658,777, “Multi 35
and/or adapt.
Functional Hand-Held Device.” filed Mar. 4, 2005; (10) U.S.
patent application Ser. No. 1 1/367,749, “Multi-Functional SUMMARY
Hand-Held Device.” filed Mar. 3, 2006; and (11) U.S. Provi
sional Patent Application No. 60/824,769, “Portable Multi
function Device.” filed Sep. 6, 2006. All of these applications 40 The above deficiencies and other problems associated with
are incorporated by reference herein in their entirety. user interfaces for portable devices and devices with touch
sensitive displays are reduced or eliminated by the disclosed
TECHNICAL FIELD device. In some embodiments, the device has a touch-sensi
tive display (also known as a “touchscreen') with a graphical
The disclosed embodiments relate generally to devices 45 user interface (GUI), one or more processors, memory and
with touch-screen displays, and more particularly to scrolling one or more modules, programs or sets of instructions stored
lists and to translating, rotating, and Scaling electronic docu in the memory for performing multiple functions. In some
ments on devices with touch-screen displays. embodiments, the user interacts with the GUI primarily
through finger contacts and gestures on the touch-sensitive
BACKGROUND 50 display. In some embodiments, the functions may include
telephoning, video conferencing, e-mailing, instant messag
As portable electronic devices become more compact, and ing, blogging, digital photographing, digital videoing, web
the number of functions performed by a given device browsing, digital music playing, and/or digital video playing.
increases, it has become a significant challenge to design a Instructions for performing these functions may be included
user interface that allows users to easily interact with a mul 55 in a computer program product configured for execution by
tifunction device. This challenge is particularly significant for one or more processors.
handheld portable devices, which have much smaller screens In accordance with some embodiments, a computer-imple
than desktop or laptop computers. This situation is unfortu mented method for use in conjunction with a device with a
nate because the user interface is the gateway through which touch screen display is disclosed. In the method, a movement
users receive not only content but also responses to user 60 of an object on or near the touch screen display is detected. In
actions or behaviors, including user attempts to access a response to detecting the movement, an electronic document
device's features, tools, and functions. Some portable com displayed on the touch screen display is translated in a first
munication devices (e.g., mobile telephones, sometimes direction. If an edge of the electronic document is reached
called mobile phones, cell phones, cellular telephones, and while translating the electronic document in the first direction
the like) have resorted to adding more pushbuttons, increas 65 while the object is still detected on or near the touch screen
ing the density of push buttons, overloading the functions of display, an area beyond the edge of the document is displayed.
pushbuttons, or using complex menu systems to allow a user After the object is no longer detected on or near the touch
 US 7,469,381 B2
3 4
screen display, the document is translated in a second direc area beyond the edge of the document is no longer displayed,
tion until the area beyond the edge of the document is no after the object is no longer detected on or near the touch
longer displayed. screen display.
In accordance with some embodiments, a graphical user In accordance with some embodiments, a computer-imple
interface on a device with a touch screen display is disclosed, mented method for use in conjunction with a device with a
comprising a portion of an electronic document displayed on touch screen display is disclosed. In the method, a movement
the touch screen display and an area beyond an edge of the of an object on or near the touch screen display is detected. In
document. In the graphical user interface, in response to response to detecting the movement, a list of items displayed
detecting a movement of an object on or near the touch screen on the touchscreen display is scrolled in a first direction. If a
display, the electronic document is translated in a first direc 10 terminus of the list is reached while scrolling the list in the
tion. If the edge of the electronic document is reached while first direction while the object is still detected on or near the
translating the electronic document in the first direction while touch screen display, an area beyond the terminus of the list is
the object is still detected on or near the touchscreen display, displayed. After the object is no longer detected on or near the
the area beyond the edge of the document is displayed. After touch screen display, the list is scrolled in a second direction
the object is no longer detected on or near the touch screen 15 opposite the first direction until the area beyond the terminus
display, the document is translated in a second direction until of the list is no longer displayed.
the area beyond the edge of the document is no longer dis In accordance with some embodiments, a graphical user
played. interface on a device with a touch screen display is disclosed,
In accordance with Some embodiments, a device is dis comprising a portion of a list of items displayed on the touch
closed, comprising a touch screen display, one or more pro screen display and an area beyond a terminus of the list. In
cessors, memory, and one or more programs. The one or more response to detecting a movement of an object on or near the
programs are stored in the memory and configured to be touch screen display, the list is scrolled in a first direction. If
executed by the one or more processors. The one or more the terminus of the list is reached while scrolling the list in the
programs include instructions for detecting a movement of an first direction while the object is still detected on or near the
object on or near the touch screen display and instructions for 25 touch screen display, the area beyond the terminus of the list
translating an electronic document displayed on the touch is displayed. After the object is no longer detected on or near
screen display in a first direction, in response to detecting the the touch screen display, the list is scrolled in a second direc
movement. The one or more programs also include instruc tion opposite the first direction until the area beyond the
tions for displaying an area beyond an edge of the electronic terminus of the list is no longer displayed.
document if the edge of the electronic document is reached 30 In accordance with Some embodiments, a device is dis
while translating the electronic document in the first direction closed, comprising a touch screen display, one or more pro
while the object is still detected on or near the touch screen cessors, memory, and one or more programs. The one or more
display. The one or more programs further include instruc programs are stored in the memory and configured to be
tions for translating the document in a second direction until executed by the one or more processors. The one or more
the area beyond the edge of the document is no longer dis 35 programs include instructions for detecting a movement of an
played, after the object is no longer detected on or near the object on or near the touch screen display and instructions for
touch screen display. scrolling a list of items displayed on the touch screen display
In accordance with Some embodiments, a computer-pro in a first direction in response to detecting the movement. The
gram product is disclosed, comprising a computer readable one or more programs also include instructions for displaying
storage medium and a computer program mechanism embed 40 an area beyond a terminus of the list if the terminus of the list
ded therein. The computer program mechanism comprises is reached while scrolling the list in the first direction while
instructions, which when executed by a device with a touch the object is still detected on or near the touchscreen display.
screen display, cause the device to detect a movement of an The one or more programs further include instructions for
object on or near the touch screen display and to translate an scrolling the list in a second direction opposite the first direc
electronic document displayed on the touch screen display in 45 tion until the area beyond the terminus of the list is no longer
a first direction, in response to detecting the movement. The displayed, after the object is no longer detected on or near the
instructions also cause the device to display an area beyond an touch screen display.
edge of the electronic document if the edge of the electronic In accordance with some embodiments, a computer-pro
document is reached while translating the electronic docu gram product is disclosed, comprising a computer readable
ment in the first direction while the object is still detected on 50 storage medium and a computer program mechanism embed
or near the touch screen display. The instructions further ded therein. The computer program mechanism comprises
cause the device to translate the document in a second direc instructions, which when executed by a device with a touch
tion until the area beyond the edge of the document is no screen display, cause the device to detect a movement of an
longer displayed, after the object is no longer detected on or object on or near the touchscreen display and to Scroll a list of
near the touch screen display. 55 items displayed on the touchscreen display in a first direction
In accordance with some embodiments, a device with a in response to detecting the movement. The instructions also
touch screen display is disclosed. The device comprises cause the device to display an area beyond a terminus of the
means for detecting a movement of an object on or near the list if the terminus of the list is reached while scrolling the list
touch screen display and means for translating an electronic in the first direction while the object is still detected on or near
document displayed on the touch screen display in a first 60 the touch screen display. The instructions further cause the
direction, in response to detecting the movement. The device device to scroll the list in a second direction opposite the first
also comprises means for displaying an area beyond an edge direction until the area beyond the terminus of the list is no
of the electronic document if the edge of the electronic docu longer displayed, after the object is no longer detected on or
ment is reached while translating the electronic document in near the touch screen display.
the first direction while the object is still detected on or near 65 In accordance with some embodiments, a device with a
the touch screen display. The device further comprises means touch screen display is disclosed. The device comprises
for translating the document in a second direction until the means for detecting a movement of an object on or near the
 US 7,469,381 B2
5 6
touch screen display and means for Scrolling a list of items In accordance with some embodiments, a computer-imple
displayed on the touch screen display in a first direction in mented method of displaying an electronic document having
response to detecting the movement. The device also com a document length and a document width, for use at a device
prises means for displaying an area beyond a terminus of the with a touchscreen display, includes displaying the electronic
list if the terminus of the list is reached while scrolling the list documentata first magnification and detecting a gesture on or
in the first direction while the object is still detected on or near near the touch screen display corresponding to a command to
the touch screen display. The device further comprises means Zoom out by a user-specified amount. In response to detecting
for Scrolling the list in a second direction opposite the first the gesture, the electronic document is displayed at a magni
direction until the area beyond the terminus of the list is no fication less than the first magnification. If the document
longer displayed, after the object is no longer detected on or 10 length or document width is entirely displayed while the
near the touch screen display. gesture is still detected on or near the touchscreen display, the
In accordance with some embodiments, a computer-imple electronic document is displayed at a magnification wherein
mented method for use at a device with a touchscreen display areas beyond opposite edges of the electronic document are
includes detecting a multifinger twisting gesture on or near displayed, and upon detecting termination of the gesture, the
the touch screen display. The multifinger twisting gesture has 15 electronic document is displayed at a magnification wherein
a corresponding degree of rotation. If the corresponding the areas beyond opposite edges of the electronic document
degree of rotation exceeds a predefined degree of rotation, a are no longer displayed.
90° screen rotation command is executed. If the correspond In accordance with some embodiments, a graphical user
ing degree of rotation is less than the predefined degree of interface on a device with a touch screen display includes an
rotation, a screen rotation command with an acute angle of electronic document having a document length and a docu
rotation is executed and, upon ceasing to detect the multifin ment width, to be displayed on the touch screen display at
ger twisting gesture, a screen rotation command with an angle multiple magnifications including a first magnification, and
of rotation opposite to the acute angle is executed. areas beyond opposite edges of the electronic document. In
response to detecting a gesture on or near the touch screen
In accordance with some embodiments, a device includes a 25 display corresponding to a command to Zoom out by a user
touch screen display, one or more processors, memory, and specified amount, wherein the gesture is detected while dis
one or more programs. The one or more programs are stored playing the electronic document at the first magnification, the
in the memory and configured to be executed by the one or electronic document is displayed at a magnification less than
more processors. The one or more programs include: instruc the first magnification. If the document length or document
tions for detecting a multifinger twisting gesture on or near 30 width is entirely displayed while the gesture is still detected
the touch screen display, wherein the multifinger twisting on or near the touchscreen display, the electronic document is
gesture has a corresponding degree of rotation; instructions displayed at a magnification wherein the areas beyond oppo
for executing a 90° screen rotation command, if the corre site edges of the electronic document are displayed, and upon
sponding degree of rotation exceeds a predefined degree of detecting termination of the gesture, the electronic document
rotation; and instructions for executing a screen rotation com 35 is displayed at a magnification wherein the areas beyond
mand with an acute angle of rotation and for executing, upon opposite edges of the electronic document are no longer dis
ceasing to detect the multifinger twisting gesture, a screen played.
rotation command with an angle of rotation opposite to the In accordance with some embodiments, a device includes a
acute angle, if the corresponding degree of rotation is less touch screen display, one or more processors, memory, and
than the predefined degree of rotation. 40 one or more programs. The one or more programs are stored
In accordance with Some embodiments, a computer-pro in the memory and configured to be executed by the one or
gram product includes a computer readable storage medium more processors. The one or more programs include: instruc
and a computer program mechanism embedded therein. The tions for displaying an electronic document at a first magni
computer program mechanism includes instructions, which fication; instructions for detecting a gesture on or near the
when executed by a device with a touch screen display, cause 45 touch screen display corresponding to a command to Zoom
the device to: detect a multifinger twisting gesture on or near out by a user-specified amount; instructions for displaying the
the touch screen display, wherein the multifinger twisting electronic document at a magnification less than the first
gesture has a corresponding degree of rotation; execute a 90° magnification, in response to detecting the gesture; instruc
screen rotation command, if the corresponding degree of tions for displaying the electronic document at a magnifica
rotation exceeds a predefined degree of rotation; and execute 50 tion wherein areas beyond opposite edges of the electronic
a screen rotation command with an acute angle of rotation document are displayed, if a document length or a document
and, upon ceasing to detect the multifinger twisting gesture, width is entirely displayed while the gesture is still detected
execute a screen rotation command with an angle of rotation on or near the touch screen display; and instructions for
opposite to the acute angle, if the corresponding degree of displaying the electronic document at a magnification
rotation is less than the predefined degree of rotation. 55 wherein the areas beyond opposite edges of the electronic
In accordance with some embodiments, a device with a document are no longer displayed, upon detecting termina
touch screen display includes: means for detecting a multi tion of the gesture.
finger twisting gesture on or near the touch screen display, In accordance with some embodiments, a computer-pro
wherein the multifinger twisting gesture has a corresponding gram product includes a computer readable storage medium
degree of rotation; means for executing a 90° screen rotation 60 and a computer program mechanism embedded therein. The
command, if the corresponding degree of rotation exceeds a computer program mechanism includes instructions, which
predefined degree of rotation; and means for executing a when executed by a device with a touch screen display, cause
screen rotation command with an acute angle of rotation and, the device to: display an electronic document at a first mag
upon ceasing to detect the multifinger twisting gesture, for nification; detect a gesture on or near the touch screen display
executing a screen rotation command with an angle of rota 65 corresponding to a command to Zoom out by a user-specified
tion opposite to the acute angle, if the corresponding degree amount; display the electronic document at a magnification
of rotation is less than the predefined degree of rotation. less than the first magnification, in response to detecting the
 US 7,469,381 B2
7 8
gesture; display the electronic document at a magnification In accordance with some embodiments, a computer-pro
wherein areas beyond opposite edges of the electronic docu gram product includes a computer readable storage medium
ment are displayed, if a document length or a document width and a computer program mechanism embedded therein. The
is entirely displayed while the gesture is still detected on or computer program mechanism includes instructions, which
near the touch screen display; and display the electronic when executed by a device with a touch screen display, cause
document at a magnification wherein the areas beyond oppo the device to: display at least a first portion of an electronic
site edges of the electronic document are no longer displayed, document at a first magnification; detect a gesture on or near
upon detecting termination of the gesture. the touch screen display corresponding to a command to
In accordance with some embodiments, a device with a Zoom in by a user-specified amount; display decreasing por
touch screen display includes: means for displaying an elec 10 tions of the electronic documentat increasing magnifications,
tronic document at a first magnification; means for detecting in response to detecting the gesture; and display a respective
a gesture on or near the touchscreen display corresponding to portion of the electronic document at a predefined magnifi
a command to Zoom out by a user-specified amount; means cation if upon detecting termination of the gesture, the mag
for displaying the electronic documentata magnification less nification exceeds a predefined magnification.
15 In accordance with some embodiments, a device with a
than the first magnification, in response to detecting the ges
ture; means for displaying the electronic document at a mag touch screen display includes means for displaying at least a
nification wherein areas beyond opposite edges of the elec first portion of an electronic documentata first magnification;
tronic document are displayed, if a document length or a means for detecting a gesture on or near the touch screen
document width is entirely displayed while the gesture is still display corresponding to a command to Zoom in by a user
detected on or near the touch screen display; and means for specified amount; means for displaying decreasing portions
displaying the electronic document at a magnification of the electronic document at increasing magnifications, in
wherein the areas beyond opposite edges of the electronic response to detecting the gesture; and means for displaying a
document are no longer displayed, upon detecting termina respective portion of the electronic document at a predefined
tion of the gesture. magnification if, upon detecting termination of the gesture,
In accordance with some embodiments, a computer-imple
25 the magnification exceeds the predefined magnification.
mented method of displaying an electronic document, for use The disclosed embodiments provide for easy and intuitive
at a device with a touch screen display, includes displaying at scrolling of lists and translating of electronic documents on a
least a first portion of the electronic document at a first mag device with a touch screen display, and for easy and intuitive
nification and detecting a gesture on or near the touch screen 30
rotation and Scaling of electronic documents on a device with
display corresponding to a command to Zoom in by a user a touch screen display.
specified amount. In response to detecting the gesture,
decreasing portions of the electronic document are displayed BRIEF DESCRIPTION OF THE DRAWINGS
at increasing magnifications. Upon detecting termination of
the gesture, if the magnification exceeds a predefined magni 35 For a better understanding of the aforementioned embodi
fication, a respective portion of the electronic document is ments of the invention as well as additional embodiments
displayed at the predefined magnification. thereof, reference should be made to the Description of
In accordance with some embodiments, a graphical user Embodiments below, in conjunction with the following draw
interface on a device with a touch screen display includes ings in which like reference numerals refer to corresponding
decreasing portions of an electronic document, to be dis 40
parts throughout the figures.
played on the touch screen display at increasing magnifica FIG. 1 is a block diagram illustrating a portable multifunc
tions. The decreasing portions of the electronic document tion device with a touch-sensitive display in accordance with
include a first portion. In response to detecting a gesture on or Some embodiments.
near the touch screen display corresponding to a command to FIG. 2 illustrates a portable multifunction device having a
Zoom in by a user-specified amount, wherein the gesture is 45 touch screen in accordance with some embodiments.
detected while displaying at least the first portion of an elec FIG.3 illustrates an exemplary user interface for unlocking
tronic document at a first magnification, the decreasing por a portable electronic device in accordance with some embodi
tions of the electronic document are displayed at the increas mentS.
ing magnifications. Upon detecting termination of the FIG. 4 illustrates an exemplary user interface for a menu of
gesture, if the magnification exceeds a predefined magnifica 50 applications on a portable multifunction device in accordance
tion, a respective portion of the electronic document is dis with some embodiments.
played at the predefined magnification. FIG. 5 is a flow diagram illustrating a method of scrolling
In accordance with some embodiments, a device includes a through a list in accordance with some embodiments.
touch screen display, one or more processors, memory, and FIGS. 6A-6D illustrate an exemplary user interface for
one or more programs. The one or more programs are stored 55
managing an inbox in accordance with some embodiments.
in the memory and configured to be executed by the one or FIG. 7 is a flow diagram illustrating a method of translating
more processors. The one or more programs include: instruc an electronic document in accordance with some embodi
tions for displaying at least a first portion of an electronic mentS.
document at a first magnification; instructions for detecting a
gesture on or near the touch screen display corresponding to 60 FIGS. 8A-8D illustrate an exemplary user interface for a
a command to Zoom in by a user-specified amount; instruc browser in accordance with Some embodiments.
tions for displaying decreasing portions of the electronic FIG.9 is a flow diagram illustrating a process of displaying
document at increasing magnifications, in response to detect an electronic document at multiple magnifications in accor
ing the gesture; and instructions for displaying a respective dance with some embodiments.
portion of the electronic document at a predefined magnifi 65 FIGS. 10A-10C illustrate the display of an electronic docu
cation if, upon detecting termination of the gesture, the mag ment at multiple magnifications in accordance with some
nification exceeds the predefined magnification. embodiments.
 US 7,469,381 B2
10
FIG. 11 is a flow diagram illustrating a process of display device, such as the touchscreen. One or more functions of the
ing an electronic document at multiple magnifications in touch screen as well as corresponding information displayed
accordance with Some embodiments. on the device may be adjusted and/or varied from one appli
FIGS. 12A-12C illustrate the display of an electronic docu cation to the next and/or within a respective application. In
ment at multiple magnifications in accordance with some this way, a common physical architecture (such as the touch
embodiments. screen) of the device may support the variety of applications
FIGS. 13A-13C illustrate the display of an electronic docu with user interfaces that are intuitive and transparent.
ment at multiple magnifications in accordance with some The user interfaces may include one or more soft keyboard
embodiments.
FIG. 14 is a flow diagram illustrating a process of executing 10
embodiments. The soft keyboard embodiments may include
a screen rotation command in accordance with some embodi standard (QWERTY) and/or non-standard configurations of
mentS.
symbols on the displayed icons of the keyboard. Such as those
FIGS. 15A-15E illustrate rotating the display of an elec described in U.S. patent applications Ser. No. 1 1/459,606,
tronic document or other digital object in accordance with “Keyboards For Portable Electronic Devices.” filed Jul. 24,
Some embodiments. 15
2006, and Ser. No. 1 1/459,615, “Touch Screen Keyboards For
Portable Electronic Devices, filed Jul. 24, 2006, the contents
FIGS. 16A-16F illustrate an exemplary screen rotation of which are hereby incorporated by reference herein in their
gesture in accordance with some embodiments. entirety. The keyboard embodiments may include a reduced
FIG. 17 is a block diagram illustrating a device with a number of icons (or soft keys) relative to the number of keys
touch-screen display in accordance with some embodiments. in existing physical keyboards, such as that for a typewriter.
DESCRIPTION OF EMBODIMENTS This may make it easier for users to select one or more icons
in the keyboard, and thus, one or more corresponding sym
Reference will now be made in detail to embodiments, bols. The keyboard embodiments may be adaptive. For
examples of which are illustrated in the accompanying draw example, displayed icons may be modified in accordance
ings. In the following detailed description, numerous specific 25
with user actions, such as selecting one or more icons and/or
details are set forth in order to provide a thorough understand one or more corresponding symbols. One or more applica
ing of the present invention. However, it will be apparent to tions on the portable device may utilize common and/or dif
one of ordinary skill in the art that the present invention may ferent keyboard embodiments. Thus, the keyboard embodi
be practiced without these specific details. In other instances, ment used may be tailored to at least some of the applications.
well-known methods, procedures, components, circuits, and 30
In some embodiments, one or more keyboard embodiments
networks have not been described in detailso as not to unnec may be tailored to a respective user. For example, based on a
essarily obscure aspects of the embodiments. word usage history (lexicography, slang, individual usage) of
Embodiments of a portable multifunction device, user the respective user. Some of the keyboard embodiments may
interfaces for Such devices, and associated processes for be adjusted to reduce a probability of a user error when
using such devices are described. In some embodiments, the 35
selecting one or more icons, and thus one or more symbols,
device is a portable communications device Such as a mobile when using the Soft keyboard embodiments.
telephone that also contains other functions, such as PDA Attention is now directed towards embodiments of the
and/or music player functions. device. FIG. 1 is a block diagram illustrating a portable mul
The user interface may include a physical click wheel in tifunction device 100 with a touch-sensitive display 112 in
addition to a touchscreen or a virtual click wheel displayed on 40
accordance with some embodiments. The touch-sensitive dis
the touch screen. A click wheel is a user-interface device that play 112 is sometimes called a “touch screen” for conve
may provide navigation commands based on an angular dis nience. The device 100 may include a memory 102 (which
placement of the wheel or a point of contact with the wheel by may include one or more computer readable storage medi
a user of the device. A click wheel may also be used to provide ums), a memory controller 122, one or more processing units
a user command corresponding to selection of one or more 45 (CPUs) 120, a peripherals interface 118, RF circuitry 108,
items, for example, when the user of the device presses down audio circuitry 110, a speaker 111, a microphone 113, an
on at least a portion of the wheel or the center of the wheel. input/output (I/O) subsystem 106, other input or control
Alternatively, breaking contact with a click wheel image on a devices 116, and an external port 124. The device 100 may
touch screen Surface may indicate a user command corre include one or more optical sensors 164. These components
sponding to selection. For simplicity, in the discussion that 50 may communicate over one or more communication buses or
follows, a portable multifunction device that includes a touch signal lines 103.
screen is used as an exemplary embodiment. It should be It should be appreciated that the device 100 is only one
understood, however, that some of the user interfaces and example of a portable multifunction device 100, and that the
associated processes may be applied to other devices, such as device 100 may have more or fewer components than shown,
personal computers and laptop computers, that may include 55 may combine two or more components, or a may have a
one or more other physical user-interface devices, such as a different configuration or arrangement of the components.
physical click wheel, a physical keyboard, a mouse and/or a The various components shown in FIG. 1 may be imple
joystick. mented in hardware, software or a combination of both hard
The device Supports a variety of applications, such as a ware and Software, including one or more signal processing
telephone application, a video conferencing application, an 60 and/or application specific integrated circuits.
e-mail application, an instant messaging application, a blog Memory 102 may include high-speed random access
ging application, a digital camera application, a digital video memory and may also include non-volatile memory. Such as
camera application, a web browsing application, a digital one or more magnetic disk storage devices, flash memory
music player application, and/or a digital video player appli devices, or other non-volatile Solid-state memory devices.
cation. 65 Access to memory 102 by other components of the device
The various applications that may be executed on the 100, such as the CPU 120 and the peripherals interface 118,
device may use at least one common physical user-interface may be controlled by the memory controller 122.
 US 7,469,381 B2
11 12
The peripherals interface 118 couples the input and output dials, slider Switches, joysticks, click wheels, and so forth. In
peripherals of the device to the CPU 120 and memory 102. some alternate embodiments, input controller(s) 160 may be
The one or more processors 120 run or execute various soft coupled to any (or none) of the following: a keyboard, infra
ware programs and/or sets of instructions stored in memory red port, USB port, and a pointer device such as a mouse. The
102 to perform various functions for the device 100 and to one or more buttons (e.g., 208, FIG. 2) may include an
process data. up/down button for volume control of the speaker 111 and/or
In some embodiments, the peripherals interface 118, the the microphone 113. The one or more buttons may include a
CPU 120, and the memory controller 122 may be imple push button (e.g., 206, FIG. 2). A quick press of the push
mented on a single chip. Such as a chip 104. In some other button may disengage a lock of the touch screen 112 or begin
embodiments, they may be implemented on separate chips. 10 a process that uses gestures on the touch screen to unlock the
The RF (radio frequency) circuitry 108 receives and sends device, as described in U.S. patent application Ser. No.
RF signals, also called electromagnetic signals. The RF cir 1 1/322.549, "Unlocking a Device by Performing Gestures on
cuitry 108 converts electrical signals to/from electromagnetic an Unlock Image.” filed Dec. 23, 2005, which is hereby
signals and communicates with communications networks incorporated by reference herein in its entirety. A longerpress
and other communications devices via the electromagnetic 15 of the push button (e.g., 206) may turn power to the device
signals. The RF circuitry 108 may include well-known cir 100 on or off. The user may be able to customize a function
cuitry for performing these functions, including but not lim ality of one or more of the buttons. The touch screen 112 is
ited to an antenna system, an RF transceiver, one or more used to implement virtual or soft buttons and one or more soft
amplifiers, a tuner, one or more oscillators, a digital signal keyboards.
processor, a CODEC chipset, a subscriber identity module The touch-sensitive display system 112 provides an input
(SIM) card, memory, and so forth. The RF circuitry 108 may interface and an output interface between the device and a
communicate with networks. Such as the Internet, also user. The display controller 156 receives and/or sends elec
referred to as the World Wide Web (WWW), an intranet trical signals from/to the display system 112. The display
and/or a wireless network, Such as a cellular telephone net system 112 displays visual output to the user. The visual
work, a wireless local area network (LAN) and/or a metro 25 output may include graphics, text, icons, video, and any com
politan area network (MAN), and other devices by wireless bination thereof (collectively termed “graphics”). In some
communication. The wireless communication may use any of embodiments. Some or all of the visual output may corre
a plurality of communications standards, protocols and tech spond to user-interface objects, further details of which are
nologies, including but not limited to Global System for described below.
Mobile Communications (GSM), Enhanced Data GSM Envi 30 A touch screen in display system 112 is a touch-sensitive
ronment (EDGE), wideband code division multiple access Surface that accepts input from the user based on haptic and/or
(W-CDMA), code division multiple access (CDMA), time tactile contact. The display system 112 and the display con
division multiple access (TDMA), Bluetooth, Wireless Fidel troller 156 (along with any associated modules and/or sets of
ity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g instructions in memory 102) detect contact (and any move
and/or IEEE 802.11n), voice over Internet Protocol (VoIP), 35 ment or breaking of the contact) on the display system 112
Wi-MAX, a protocol for email, instant messaging, and/or and converts the detected contact into interaction with user
Short Message Service (SMS)), or any other suitable com interface objects (e.g., one or more soft keys, icons, web
munication protocol, including communication protocols not pages or images) that are displayed on the touch screen. In an
yet developed as of the filing date of this document. exemplary embodiment, a point of contact between a touch
The audio circuitry 110, the speaker 111, and the micro 40 screen in the display system 112 and the user corresponds to
phone 113 provide an audio interface between a user and the a finger of the user.
device 100. The audio circuitry 110 receives audio data from The touch screen in the display system 112 may use LCD
the peripherals interface 118, converts the audio data to an (liquid crystal display) technology, or LPD (light emitting
electrical signal, and transmits the electrical signal to the polymer display) technology, although other display tech
speaker 111. The speaker 111 converts the electrical signal to 45 nologies may be used in other embodiments. The touch
human-audible sound waves. The audio circuitry 110 also screen in the display system 112 and the display controller
receives electrical signals converted by the microphone 113 156 may detect contact and any movement or breaking
from sound waves. The audio circuitry 110 converts the elec thereofusing any of a plurality of touch sensing technologies
trical signal to audio data and transmits the audio data to the now known or later developed, including but not limited to
peripherals interface 118 for processing. Audio data may be 50 capacitive, resistive, infrared, and Surface acoustic wave tech
retrieved from and/or transmitted to memory 102 and/or the nologies, as well as other proximity sensor arrays or other
RF circuitry 108 by the peripherals interface 118. In some elements for determining one or more points of contact with
embodiments, the audio circuitry 110 also includes aheadset a touch screen in the display system 112. A touch-sensitive
jack (not shown). The headset jack provides an interface display in Some embodiments of the display system 112 may
between the audio circuitry 110 and removable audio input/ 55 be analogous to the multi-touch sensitive tablets described in
output peripherals, such as output-only headphones orahead the following U.S. Pat. Nos. 6.323,846 (Westerman et al.),
set with both output (e.g., a headphone for one or both ears) 6,570,557 (Westermanet al.), and/or 6,677,932 (Westerman),
and input (e.g., a microphone). and/or U.S. Patent Publication 2002/0015024A1, each of
The I/O subsystem 106 couples input/output peripherals on which is hereby incorporated by reference herein in their
the device 100, such as the display system 112 and other 60 entirety. However, a touch screen in the display system 112
input/control devices 116, to the peripherals interface 118. displays visual output from the portable device 100, whereas
The I/O subsystem 106 may include a display controller 156 touch sensitive tablets do not provide visual output. The touch
and one or more input controllers 160 for other input or screen in the display system 112 may have a resolution in
control devices. The one or more input controllers 160 excess of 100 dpi. In an exemplary embodiment, the touch
receive/send electrical signals from/to other input or control 65 screen in the display system has a resolution of approximately
devices 116. The other input/control devices 116 may include 168 dpi. The user may make contact with the touch screen in
physical buttons (e.g., push buttons, rocker buttons, etc.), the display system 112 using any suitable object or append
 US 7,469,381 B2
13 14
age. Such as a stylus, a finger, and so forth. In some embodi Sources (e.g., battery, alternating current (AC)), a recharging
ments, the user interface is designed to work primarily with system, a power failure detection circuit, a power converter or
finger-based contacts and gestures, which are much less pre inverter, a power status indicator (e.g., a light-emitting diode
cise than stylus-based input due to the larger area of contact of (LED)) and any other components associated with the gen
a finger on the touchscreen. In some embodiments, the device 5 eration, management and distribution of power in portable
translates the rough finger-based input into a precise pointer/ devices.
cursor position or command for performing the actions The device 100 may also include one or more optical
desired by the user. sensors 164. FIG. 1 shows an optical sensor coupled to an
A touch-sensitive display in Some embodiments of the optical sensor controller 158 in I/O subsystem 106. The opti
display system 112 may be as described in the following 10 cal sensor 164 may include charge-coupled device (CCD) or
applications: (1) U.S. patent application Ser. No. 1 1/381.313, complementary metal-oxide semiconductor (CMOS) pho
“Multipoint TouchSurface Controller” filed on May 2, 2006: totransistors. The optical sensor 164 receives light from the
(2) U.S. patent application Ser. No. 10/840,862, “Multipoint environment, projected through one or more lens, and con
Touchscreen.” filed on May 6, 2004; (3) U.S. patent applica verts the light to data representing an image. In conjunction
tion Ser. No. 10/903,964, “Gestures For Touch Sensitive 15 with an imaging module 143, the optical sensor 164 may
Input Devices, filed on Jul. 30, 2004; (4) U.S. patent appli capture still images or video. In some embodiments, an opti
cation Ser. No. 11/048.264, “Gestures For Touch Sensitive cal sensor is located on the back of the device 100, opposite
Input Devices.” filed on Jan. 31, 2005; (5) U.S. patent appli the touchscreen display 112 on the front of the device, so that
cation Ser. No. 11/038,590, “Mode-Based Graphical User the touch screen display may be used as a viewfinder for
Interfaces For Touch Sensitive Input Devices.” filed on Jan. either still and/or video image acquisition. In some embodi
18, 2005; (6) U.S. patent application Ser. No. 1 1/228,758, ments, an optical sensor is located on the front of the device so
“Virtual Input Device Placement On A Touch Screen User that the user's image may be obtained for videoconferencing
Interface filed on Sep. 16, 2005; (7) U.S. patent application while the user views the other video conference participants
Ser. No. 1 1/228.700, “Operation Of A Computer With A on the touch screen display. In some embodiments, the posi
Touch Screen Interface.” filed on Sep. 16, 2005; (8) U.S. 25 tion of the optical sensor 164 can be changed by the user (e.g.,
patent application Ser. No. 1 1/228.737, Activating Virtual by rotating the lens and the sensor in the device housing) so
Keys Of A Touch-Screen Virtual Keyboard, filed on Sep. 16, that a single optical sensor 164 may be used along with the
2005; and (9) U.S. patent application Ser. No. 1 1/367,749, touch screen display for both video conferencing and still
“Multi-Functional Hand-Held Device filed on Mar. 3, 2006. and/or video image acquisition.
All of these applications are incorporated by reference herein 30 The device 100 may also include one or more proximity
in their entirety. sensors 166. FIG. 1 shows a proximity sensor 166 coupled to
In some embodiments, in addition to the touch screen, the the peripherals interface 118. Alternately, the proximity sen
device 100 may include a touchpad (not shown) for activating sor 166 may be coupled to an input controller 160 in the I/O
or deactivating particular functions. In some embodiments, subsystem 106. The proximity sensor 166 may perform as
the touchpad is a touch-sensitive area of the device that, 35 described in U.S. patent application Ser. Nos. 1 1/241,839,
unlike the touch screen, does not display visual output. The “Proximity Detector In Handheld Device.” filed Sep. 30,
touchpad may be a touch-sensitive Surface that is separate 2005, and 11/240,788, “Proximity Detector In Handheld
from the touch screen in the display system 112 or an exten Device.” filed Sep. 30, 2005, which are hereby incorporated
sion of the touch-sensitive surface formed by the touch by reference herein in their entirety. In some embodiments,
SCC. 40 the proximity sensor turns off and disables the touch screen
In some embodiments, the device 100 may include a physi 112 when the multifunction device is placed near the user's
cal or virtual click wheel as an input control device 116. A ear (e.g., when the user is making a phone call). In some
user may navigate among and interact with one or more embodiments, the proximity sensor keeps the screen off when
graphical objects (henceforth referred to as icons) displayed the device is in the user's pocket, purse, or other dark area to
in the display system 112 by rotating the click wheel or by 45 prevent unnecessary battery drainage when the device is a
moving a point of contact with the click wheel (e.g., where the locked State.
amount of movement of the point of contact is measured by its In some embodiments, the Software components stored in
angular displacement with respect to a center point of the memory 102 may include an operating system 126, a com
click wheel). The click wheel may also be used to select one munication module (or set of instructions) 128, a contact/
or more of the displayed icons. For example, the user may 50 motion module (or set of instructions) 130, a graphics module
press down on at least a portion of the click wheel or an (or set of instructions) 132, a text input module (or set of
associated button. User commands and navigation commands instructions) 134, a Global Positioning System (GPS) module
provided by the user via the click wheel may be processed by (or set of instructions) 135, and applications (or set of instruc
an input controller 160 as well as one or more of the modules tions) 136.
and/or sets of instructions in memory 102. For a virtual click 55 The operating system 126 (e.g., Darwin, RTXC, LINUX,
wheel, the click wheel and click wheel controller may be part UNIX, OSX,WINDOWS, or an embedded operating system
of the display system 112 and the display controller 156, such as VxWorks) includes various software components
respectively. For a virtual click wheel, the click wheel may be and/or drivers for controlling and managing general system
either an opaque or semitransparent object that appears and tasks (e.g., memory management, storage device control,
disappears on the touch screen display in response to user 60 power management, etc.) and facilitates communication
interaction with the device. In some embodiments, a virtual between various hardware and software components.
click wheel is displayed on the touch screen of a portable The communication module 128 facilitates communica
multifunction device and operated by user contact with the tion with other devices over one or more external ports 124
touch screen. and also includes various Software components for handling
The device 100 also includes a power system 162 for pow 65 data received by the RF circuitry 108 and/or the external port
ering the various components. The power system 162 may 124. The external port 124 (e.g., Universal Serial Bus (USB),
include a power management system, one or more power FIREWIRE, etc.) is adapted for coupling directly to other
 US 7,469,381 B2
15 16
devices or indirectly over a network (e.g., the Internet, wire alarm clock widget 149-4, dictionary widget 149-5, and
less LAN, etc.). In some embodiments, the external port is a other widgets obtained by the user, as well as user
multi-pin (e.g., 30-pin) connector that is the same as, or created widgets 149-6;
similar to and/or compatible with the 30-pin connector used widget creator module 150 for making user-created wid
on iPod (trademark of Apple Computer, Inc.) devices. 5 gets 149-6; and/or search module 151.
The contact/motion module 130 may detect contact with Examples of other applications 136 that may be stored in
the touch screen in the display system 112 (in conjunction memory 102 include memo pad and other word processing
with the display controller 156) and other touch sensitive applications, JAVA-enabled applications, encryption, digital
devices (e.g., a touchpad or physical click wheel). The con rights management, Voice recognition, and Voice replication.
tact/motion module 130 includes various software compo 10
In conjunction with display system 112, display controller
nents for performing various operations related to detection 156, contact module 130, graphics module 132, and text input
of contact, such as determining if contact has occurred, deter module 134, the contacts module 137 may be used to manage
mining if there is movement of the contact and tracking the an address book or contact list, including: adding name(s) to
movement across the touch screen in the display system 112, the address book, deleting name(s) from the address book;
and determining if the contact has been broken (i.e., if the 15
associating telephone number(s), e-mail address(es), physi
contact has ceased). Determining movement of the point of cal address(es) or other information with a name; associating
contact may include determining speed (magnitude), Velocity an image with a name; categorizing and Sorting names; pro
(magnitude and direction), and/or an acceleration (a change viding telephone numbers or e-mail addresses to initiate and/
in magnitude and/or direction) of the point of contact. These or facilitate communications by telephone 138, video confer
operations may be applied to single contacts (e.g., one finger ence 139, e-mail 140, or IM 141; and so forth.
contacts) or to multiple simultaneous contacts (e.g., “multi In conjunction with RF circuitry 108, audio circuitry 110,
touch'/multiple finger contacts). In some embodiments, the speaker 111, microphone 113, display system 112, display
contact/motion module 130 and the display controller 156 controller 156, contact module 130, graphics module 132,
also detects contact on a touchpad. In some embodiments, the and text input module 134, the telephone module 138 may be
contact/motion module 130 detects movement of one or more 25
used to enter a sequence of characters corresponding to a
objects on or near the touch screen and/or the touchpad. In telephone number, access one or more telephone numbers in
some embodiments, the contact/motion module 130 and the the address book 137, modify a telephone number that has
controller 160 detects contact on a click wheel 116. been entered, dial a respective telephone number, conduct a
The graphics module 132 includes various known software conversation and disconnect or hang up when the conversa
components for rendering and displaying graphics on the 30
tion is completed. As noted above, the wireless communica
display system 112, including components for changing the tion may use any of a plurality of communications standards,
intensity of graphics that are displayed. As used herein, the protocols and technologies.
term 'graphics includes any object that can be displayed to In conjunction with RF circuitry 108, audio circuitry 110,
a user, including without limitation text, web pages, icons speaker 111, microphone 113, display system 112, display
(such as user-interface objects including Soft keys), digital 35
controller 156, optical sensor 164, optical sensor controller
images, videos, animations and the like. 158, contact module 130, graphics module 132, text input
The text input module 134, which may be a component of module 134, contact list 137, and telephone module 138, the
graphics module 132, provides soft keyboards for entering videoconferencing module 139 may be used to initiate, con
text in various applications (e.g., contacts 137, e-mail 140, IM 40 duct, and terminate a video conference between a user and
141, blogging 142, browser 147, and any other application one or more other participants.
that needs text input). In conjunction with RF circuitry 108, display system 112,
The GPS module 135 determines the location of the device display controller 156, contact module 130, graphics module
and provides this information for use in various applications 132, and text input module 134, the e-mail client module 140
(e.g., to telephone 138 for use in location-based dialing, to 45 may be used to create, send, receive, and manage e-mail. In
camera 143 and/or blogger 142 as picture/video metadata, conjunction with image management module 144, the e-mail
and to applications that provide location-based services Such module 140 makes it very easy to create and send e-mails with
as weather widgets, local yellow page widgets, and map/ still or video images taken with camera module 143.
navigation widgets). In conjunction with RF circuitry 108, display system 112,
The applications 136 may include the following modules 50 display controller 156, contact module 130, graphics module
(or sets of instructions), or a Subset or Superset thereof: 132, and text input module 134, the instant messaging module
a contacts module 137 (sometimes called an address book 141 may be used to enter a sequence of characters corre
or contact list); sponding to an instant message, to modify previously entered
a telephone module 138; characters, to transmit a respective instant message (for
a video conferencing module 139: 55 example, using a Short Message Service (SMS) or Multime
an e-mail client module 140; dia Message Service (MMS) protocol), to receive instant
an instant messaging (IM) module 141; messages and to view received instant messages. In some
a blogging module 142: embodiments, transmitted and/or received instant messages
a camera module 143 for still and/or video images: may include graphics, photos, audio files, video files and/or
an image management module 144; 60 other attachments as are supported in a MMS and/or an
a video player module 145: Enhanced Messaging Service (EMS).
In conjunction with RF circuitry 108, display system 112,
a music player module 146; display controller 156, contact module 130, graphics module
a browser module 147: 132, text input module 134, image management module 144,
a calendar module 148; 65 and browsing module 147, the blogging module 142 may be
widget modules 149, which may include weather widget used to send text, still images, video, and/or other graphics to
149-1, stocks widget 149-2, calculator widget 149-3. a blog (e.g., the user's blog).
 US 7,469,381 B2
17 18
In conjunction with display system 112, display controller Each of the above identified modules and applications cor
156, optical sensor(s) 164, optical sensor controller 158, con respond to a set of instructions for performing one or more
tact module 130, graphics module 132, and image manage functions described above. These modules (i.e., sets of
ment module 144, the camera module 143 may be used to instructions) need not be implemented as separate Software
capture still images or video (including a video stream) and programs, procedures or modules, and thus various Subsets of
store them into memory 102, modify characteristics of a still these modules may be combined or otherwise re-arranged in
image or video, or delete a still image or video from memory various embodiments. In some embodiments, memory 102
102. may store a Subset of the modules and data structures identi
In conjunction with display system 112, display controller fied above. Furthermore, memory 102 may store additional
156, contact module 130, graphics module 132, text input 10 modules and data structures not described above.
module 134, and camera module 143, the image management In some embodiments, the device 100 is a device where
module 144 may be used to arrange, modify or otherwise operation of a predefined set of functions on the device is
manipulate, label, delete, present (e.g., in a digital slide show performed exclusively through a touch screen in the display
or album), and store still and/or video images. system 112 and/or a touchpad. By using a touch screen and/or
In conjunction with display system 112, display controller 15 a touchpad as the primary input/control device for operation
156, contact module 130, graphics module 132, audio cir of the device 100, the number of physical input/control
cuitry 110, and speaker 111, the video player module 145 may devices (such as push buttons, dials, and the like) on the
be used to display, present or otherwise playback Videos (e.g., device 100 may be reduced.
on the touch screen or on an external, connected display via The predefined set of functions that may be performed
external port 124). exclusively through a touch screen and/or a touchpad
In conjunction with display system 112, display system includes navigation between user interfaces. In some embodi
controller 156, contact module 130, graphics module 132, ments, the touchpad, when touched by the user, navigates the
audio circuitry 110, speaker 111, RF circuitry 108, and device 100 to a main, home, or root menu from any user
browser module 147, the music player module 146 allows the interface that may be displayed on the device 100. In such
user to download and play back recorded music and other 25 embodiments, the touchpad may be referred to as a “menu
sound files stored in one or more file formats, such as MP3 or button.” In some other embodiments, the menu button may be
AAC files. In some embodiments, the device 100 may include a physical push button or other physical input/control device
the functionality of an MP3 player, such as an iPod (trade instead of a touchpad.
mark of Apple Computer, Inc.). FIG. 2 illustrates a portable multifunction device 100 hav
In conjunction with RF circuitry 108, display system 112, 30 ing a touch screen 112 in accordance with Some embodi
display system controller 156, contact module 130, graphics ments. The touchscreen may display one or more graphics. In
module 132, and text input module 134, the browser module this embodiment, as well as others described below, a user
147 may be used to browse the Internet, including searching, may select one or more of the graphics by making contact or
linking to, receiving, and displaying web pages or portions touching the graphics, for example, with one or more fingers
thereof, as well as attachments and other files linked to web 35 202 (not drawn to scale in the figure). In some embodiments,
pageS. selection of one or more graphics occurs when the userbreaks
In conjunction with RF circuitry 108, display system 112, contact with the one or more graphics. In some embodiments,
display system controller 156, contact module 130, graphics the contact may include a gesture. Such as one or more taps,
module 132, text input module 134, e-mail module 140, and one or more Swipes (from left to right, right to left, upward
browser module 147, the calendar module 148 may be used to 40 and/or downward) and/or a rolling of a finger (from right to
create, display, modify, and store calendars and data associ left, left to right, upward and/or downward) that has made
ated with calendars (e.g., calendar entries, to do lists, etc.). contact with the device 100. In some embodiments, inadvert
In conjunction with RF circuitry 108, display system 112, ent contact with a graphic may not select the graphic. For
display system controller 156, contact module 130, graphics example, a Swipe gesture that Sweeps over an application icon
module 132, text input module 134, and browser module 147. 45 may not select the corresponding application when the ges
the widget modules 149 are mini-applications that may be ture corresponding to selection is a tap.
downloaded and used by a user (e.g., weather widget 149-1, The device 100 may also include one or more physical
stocks widget 149-2, calculator widget 149-3, alarm clock buttons, such as "home” or menu button 204. As described
widget 149-4, and dictionary widget 149-5) or created by the previously, the menu button 204 may be used to navigate to
user (e.g., user-created widget 149-6). In some embodiments, 50 any application 136 in a set of applications that may be
a widget includes an HTML (Hypertext Markup Language) executed on the device 100. Alternatively, in some embodi
file, a CSS (Cascading Style Sheets) file, and a JavaScriptfile. ments, the menu button is implemented as a soft key in a GUI
In some embodiments, a widget includes an XML (Extensible in touch screen 112.
Markup Language) file and a JavaScript file (e.g., Yahoo! In one embodiment, the device 100 includes a touchscreen
Widgets). 55 112, a menu button 204, a push button 206 for powering the
In conjunction with RF circuitry 108, display system 112, device on/off and locking the device, and Volume adjustment
display system controller 156, contact module 130, graphics button(s) 208. The push button 206 may be used to turn the
module 132, text input module 134, and browser module 147. power on/off on the device by depressing the button and
the widget creator module 150 may be used by a user to create holding the button in the depressed state for a predefined time
widgets (e.g., turning a user-specified portion of a web page 60 interval; to lock the device by depressing the button and
into a widget). releasing the button before the predefined time interval has
In conjunction with display system 112, display system elapsed; and/or to unlock the device or initiate an unlock
controller 156, contact module 130, graphics module 132, process. In an alternative embodiment, the device 100 also
and text input module 134, the search module 151 may be may accept verbal input for activation ordeactivation of some
used to search for text, music, Sound, image, video, and/or 65 functions through the microphone 113.
other files in memory 102 that match one or more search Attention is now directed towards embodiments of user
criteria (e.g., one or more user-specified search terms). interfaces (“UI”) and associated processes that may be imple
 US 7,469,381 B2
19 20
mented on a portable multifunction device 100 and/or on a menu button enables a user to access any desired application
device 1700 with a touch-screen display (FIG. 17). with at most two inputs, such as activating the menu button
FIG.3 illustrates an exemplary user interface for unlocking 204 and then activating the desired application (e.g., by a tap
a portable electronic device inaccordance with some embodi or other finger gesture on the icon corresponding to the appli
ments. In some embodiments, user interface 300 includes the cation).
following elements, or a Subset or Superset thereof: In some embodiments, UI 400 provides integrated access
Unlock image 302 that is moved with a finger gesture to to both widget-based applications and non-widget-based
unlock the device; applications. In some embodiments, all of the widgets,
Arrow 304 that provides a visual cue to the unlock gesture: whether user-created or not, are displayed in UI 400. In other
Channel 306 that provides additional cues to the unlock 10 embodiments, activating the icon for user-created widget
gesture; 149-6 may lead to another UI (not shown) that contains the
Time 308: user-created widgets or icons corresponding to the user-cre
Day 310; ated widgets.
Date 312; and In Some embodiments, a user may rearrange the icons in UI
Wallpaper image 314. 15
400, e.g., using processes described in U.S. patent application
In some embodiments, the device detects contact with the Ser. No. 1 1/459,602, “Portable Electronic Device With Inter
touch-sensitive display (e.g., a user's finger making contact face Reconfiguration Mode, filed Jul. 24, 2006, which is
on or near the unlock image 302) while the device is in a hereby incorporated by reference herein in its entirety. For
user-interface lock state. The device moves the unlock image example, a user may move application icons in and out of tray
302 in accordance with the contact. The device transitions to 408 using finger gestures.
a user-interface unlock state if the detected contact corre
sponds to a predefined gesture, such as moving the unlock In some embodiments, UI 400 includes a gauge (not
image across channel 306. Conversely, the device maintains shown) that displays an updated account usage metric for an
the user-interface lock state if the detected contact does not account associated with usage of the device (e.g., a cellular
correspond to the predefined gesture. As noted above, pro 25 phone account), as described in U.S. patent application Ser.
cesses that use gestures on the touch screen to unlock the No. 1 1/322.552, “Account Information Display For Portable
device are described in U.S. patent application Ser. No. Communication Device filed Dec. 23, 2005, which is
1 1/322.549, “Unlocking a Device by Performing Gestures on hereby incorporated by reference herein in its entirety.
an Unlock Image filed Dec. 23, 2005, which is hereby As discussed above, UI 400 may display all of the available
incorporated by reference herein in its entirety. applications 136 on one screen so that there is no need to
FIG. 4 illustrates an exemplary user interface for a menu of scroll through a list of applications. However, in some
applications on a portable multifunction device inaccordance embodiments a touch-sensitive display may include a GUI
with Some embodiments. In some embodiments, user inter with one or more windows that display only a portion of a list
face 400 includes the following elements, or a subset or of items (e.g., information items) or of an electronic docu
superset thereof: 35 ment. In response to detecting a movement of an object on or
Signal strength indicator 402 for wireless communication; near the touch-sensitive display, the list may be scrolled or the
Time 404; electronic document may be translated. Detecting the move
Battery status indicator 406; ment of the object may include determining speed (magni
Tray 408 with icons for frequently used applications, such tude), Velocity (magnitude and direction), and/oran accelera
as one or more of the following: 40 tion (including magnitude and/or direction) of the object.
Phone 138: Scrolling through the list or translating the document may be
E-mail client 140, which may include an indicator 410 of accelerated in response to an accelerated movement of the
the number of unread e-mails; object. In some embodiments, the scrolling and acceleration
Browser 147; and of the scrolling, or translation and acceleration of the trans
Music player 146; and 45 lation, may be in accordance with a simulation of a physical
Icons for other applications, such as one or more of the device having friction, i.e., damped motion. For example, the
following: scrolling or translation may correspond to a simulation of a
IM 141: force law or equation of motion having a mass or inertial term,
Image management 144; as well as a dissipative term. In some embodiments, the
Camera 143: 50 simulation may correspond to a cylinder rotating about its
Video player 145: aX1S.

Weather 149-1; In some embodiments, accelerated movement of the

Stocks 149-2: detected object may include an accelerated movement of a
Blog 142: point of contact followed by a breaking of the point of contact.
Calendar 148; 55 For example, the user may make contact with the touch
Calculator 149-3: sensitive display, Swipe or Sweep one or more of his or her
Alarm clock 149-4, fingers along the display (i.e., move and/or accelerate the
Dictionary 149-5; and point of contact), and optionally, break the point of contact
User-created widget 149-6. with the display, i.e., move the one or more fingers away from
In some embodiments, UI 400 displays all of the available 60 the display. The Swipe or Sweep may be along a predefined
applications 136 on one screen so that there is no need to axis of the touch-sensitive display or may be within a prede
scroll through a list of applications (e.g., via a scroll bar or via termined angle of a predefined direction on the touch-sensi
a Swipe gesture). In some embodiments, as the number of tive display. In other embodiments, the accelerated move
applications increases, the icons corresponding to the appli ment of the point of contact may include a first user gesture
cations may decrease in size so that all applications may be 65 oriented along a predefined axis of the touch-sensitive display
displayed on a single screen without Scrolling. In some or oriented within a predetermined angle of a predefined
embodiments, having all applications on one screen and a direction on the touch-sensitive display.
 US 7,469,381 B2
21 22
Scrolling through the list of items or translating the elec letter in the alphabet that has one or more entries. In some
tronic document may be further accelerated in response to embodiments, the scrolling may be in accordance with a
detection of a second movement of an object on or near the simulation of an equation of motion having friction.
touch-sensitive display, such as a second Sweeping motion of The scrolling may include scrolling through a respective
the point of contact along the predefined axis or within the 5 information item subset if the point of contact moves over a
predetermined angle of a predefined direction on the touch corresponding respective index item in the index items. The
sensitive display and/or a second user gesture oriented along scrolling may have an associated Scroll speed based on a
the predefined axis or within the predetermined angle of a speed of movement of the point of contact over the respective
predefined direction on the touch-sensitive display. For index item and the number of items in the information item
example, the user may swipe one or more of his or her fingers 10 Subset corresponding to the respective index item. For
along the touch-sensitive display two or more times. example, the scroll speed may be faster for subsets that have
The scrolling through the list of items or the translation of more entries than subsets with fewer entries. The scrolling
the electronic document may be stopped in accordance with may include scrolling through all items in a plurality of the
the user breaking the point of contact and then establishing a information item Subsets in response to the point of contact
Substantially stationary point of contact with the touch-sen 15 moving over the corresponding index items in the displayed
sitive display for at least a pre-determined period of time. For index.
example, after Swiping one or more of his or her fingers along If it is determined that the point of contact with the index
the touch-sensitive display and breaking the point of contact, corresponds to a respective index item in the index, the list of
the user may touch the display and hold the one or more information items may be scrolled to a corresponding Subset
fingers that are touching the display stationary (or approxi of the list of information items. For example, if the user
mately stationary) for one or more seconds, or fractions of a selects an index item, such as the letter R, in the set of index
second. symbols, the list of items may be smoothly scrolled to the
The direction of scrolling or translation may be reversed in corresponding subset for the letter R in the list of items.
response to intersecting a virtual boundary corresponding to Alternatively, the displayed list of information items jump
a terminus of the list or an edge of the electronic document. 25 directly from a current scroll position to a scroll position in
The scrolling reversal or translation reversal may correspond which information items corresponding to the index item R
to a damped motion. For example, during scrolling, a dis are displayed.
played portion of the list of items may appear to bounce off of In the present document, the term “if may be construed to
a boundary of the window in the touch-sensitive display when mean “when, or “upon, or “in response to determining,” or
a beginning or an end of the list of items is reached. Similarly, 30 “in response to detecting depending on the context Simi
during translation, a displayed portion of the electronic docu larly, the phrase “if it is determined or “if a stated condition
ment may appear to bounce off of a boundary of the window or event is detected” may be construed to mean “upon deter
in the touch-sensitive display when an edge of the document mining or “in response to determining or “upon detecting
is reached. The apparent bounce may correspond to a simu the stated condition or event, or “in response to detecting the
lation of a viscous or elastic ball having momentum in a first 35 stated condition or event, depending on the context.
direction Striking an immovable and/or inelastic object, Such If the point of contact with the touch-sensitive display
as a wall. The Subsequent motion of the document (the motion corresponds to a user selection of a respective information
of which corresponds to the ball in the aforementioned anal item in the list of information items, information correspond
ogy) may be damped, for example, by including a friction or ing to the respective information item may be displayed on
dissipative term in the simulation. A parameter corresponding 40 the touch-sensitive display. For example, if the user selects a
to the friction term in the simulation may be adjustable, respective name, the corresponding contact information may
allowing the document to reach equilibrium in contact with be displayed.
the virtual boundary, or displaced from the virtual boundary. While scrolling through respective information subsets, an
In Some embodiments movement of the point of contact by index symbol may displayed in conjunction with each respec
the user over an index on the touch-sensitive display may be 45 tive information item Subset. In some embodiments, respec
determined. In some embodiments, the index may be dis tive index symbols may be displayed adjacent to correspond
played in a first region or a first window of the touch-sensitive ing Subsets (such as displayed text) of the list of information
display while the list of items or information items during the items. In some embodiments, a respective index symbol may
scrolling may be displayed in a second region or a second be displayed at an upper edge of a window containing the
window of the touch-sensitive display. The displayed index 50 displayed text of the respective information item subset.
may have a sequence of index items. In an exemplary embodi The index symbol corresponding to a respective informa
ment, the sequence of index items may include letters in the tion subset may be translucently displayed over the respective
alphabet, i.e., the index may include an alphabetical index. information item subset. The translucently displayed index
The list of information items may include an alphabetically symbol may have a different font color than that used to
ordered list of information items. The alphabetically ordered 55 display text in the information item Subset, and/or it may be
list of information items may include contact information, for displayed using a larger font than the font used to display text
example, in a user's contact list or address book. in the information item subset.
In response to movement of the user's point of contact over If the list of information items contains no items for a
a displayed index, the list of information items on the touch particularindex symbol, i.e., no entries for a particular Subset,
sensitive display may be scrolled. The list of information 60 a first index symbol preceding a particular index symbol and
items may include a sequence of information item Subsets a second index symbol following the index symbol may be
corresponding to the sequence of index items. The Subsets displayed in conjunction with scrolling through the list of
may include one or more categories. For example, a respec information items from the information Subset corresponding
tive category may include contact information for one or to the first index symbol to the information subset corre
more individuals whose first and/or last names begin with one 65 sponding to the second index symbol. The particular index
or more respective letters, such as the letters. In an exem symbol may not be displayed in conjunction with the dis
plary embodiment, there is a Subset corresponding to each played text of the list of information items during the scroll
 US 7,469,381 B2
23 24
through. For example, display of a respective index symbol In response to detecting the movement, a list of items
may be skipped when the list of information items contains no displayed on the touch screen display is scrolled in a first
items for the particular index symbol. direction (504). In some embodiments, the list is a list of
In some embodiments, the list scrolling described here email messages, as illustrated in FIGS. 6A-6D. In some
operates without displaying a scroll bar. Similarly, in some embodiments, the list of items is a list of instant message
embodiments, the translation of electronic documents conversations, a list of favorite phone numbers, a list of con
described here operates without displaying scroll bars. The tact information (sometimes called a contact list or address
user's Sweeping motion on the touch-sensitive display opera book list), a list of labels, a list of email folders, a list of email
tion may be performed directly on top of the displayed list or addresses, a list of physical addresses, a list of ringtones, a list
displayed electronic document, and may include a Sweeping 10 of album names, or a list of bookmarks. In some embodi
or gliding motion, near or in contact with the display's Sur ments, the first direction is a vertical direction; in some other
face, alongapath anywhere within a display window in which embodiments, the first direction is a horizontal direction. In
the list or electronic document is displayed. While a scroll bar some embodiments, scrolling the list in the first direction
could potentially be displayed in conjunction with the dis prior to reaching a terminus of the list has an associated
played list, the scrolling or translation described here can be 15 scrolling speed corresponding to a speed of movement of the
independent of any such scroll bar. In some embodiments, if object (506). In some embodiments, the list is scrolled in
a scroll bar is used, then an upward movement of a point of accordance with a simulation of an equation of motion having
contact on the scroll bar may cause earlier entries in the list to friction (508).
be displayed, whereas a downward movement of the point of If a terminus of the list is reached (e.g., upon reaching the
contact on the scrollbar may cause later entries in the list to be terminus of the list) while scrolling the list in the first direc
displayed. tion while the object is still detected on or near the touch
In some embodiments, Scrolling or translation may be in screen display, an area beyond the terminus of the list is
accordance with a speed of movement of a detected object, displayed (510-Yes, 514). In some embodiments, the list has
Such as a speed of movement of a point of contact. The speed a first item and a last item and the terminus is either the first
may be a time average of values determined during several 25 item or the last item. For example, in FIG. 6B the email 3534
time intervals. In an exemplary embodiment, the speed, from Aaron Jones is the first item and thus the terminus of the
velocity and/or acceleration may be determined over five time corresponding list of emails. In some embodiments, the area
intervals, where a respective time interval corresponds to an beyond the terminus of the list is white (516). In some
inverse of a frame rate, such as 0.0167s, of a display. In some embodiments, the list of items has a background and the area
embodiments, the speed, Velocity and/or acceleration may be 30 beyond the terminus of the list is visually indistinct from the
determined even when a variable frame rate is used. Such as background (518). For example, in FIG. 6C both the area
when one or more frames are skipped or not displayed. In 3536 and the background of the listed emails are white.
these embodiments, the speed, Velocity, and/or acceleration After the object is no longer detected on or near the touch
may be determined two or more times for the respective time screen display, the list of items is scrolled in a second direc
interval and/or may be projected based on values determined 35 tion opposite the first direction until the area beyond the
in a preceding and/or a Subsequent time interval. terminus of the list is no longer displayed (520). In some
In some embodiments, the Scrolling or translation after a embodiments, the list is scrolled in the second direction using
user optionally breaks the contact may be in accordance with a damped motion (522). In some embodiments, the change
the change in the acceleration and the speed or the Velocity in from scrolling the list in the first direction to scrolling the list
one or more time intervals prior to the breaking of the contact. 40 in the second direction until the area beyond the terminus of
For example, the velocity V, of scrolling or translation one or the list is no longer displayed makes the terminus of the list
more time intervals after breaking contact may be determined appear to be elastically attached to an edge of the touchscreen
uS1ng display or to an edge displayed on the touch screen display
(524).
45 In some embodiments, Scrolling in the first direction prior
to reaching the terminus of the list has a first associated
where V is a current value of the velocity when the contact is scrolling distance that corresponds to a distance of movement
broken, a is a current value of the acceleration when the of the object prior to reaching the terminus of the list. For
contact is broken and At is an elapsed time. Such as one time example, a scrolling distance prior to reaching the terminus of
interval. The Velocities and/or acceleration in Such a calcula 50 the list shown in FIGS. 6A-6D may correspond to a distance
tion may be projected along an axis or direction of the Scroll traversed on the touch screen display by the Swipe gesture
ing or translation. In some embodiments, in Subsequent time 3514 before the terminus is reached. Displaying an area
intervals following the determination of the velocity based on beyond the terminus of the list includes scrolling the list in the
the acceleration and/or the velocity in one or more time inter first direction for a second associated Scrolling distance that is
vals prior to the breaking of the contact, the velocity of the 55 less than a distance of movement of the object after the
scrolling or translation may be tapered. For example, in each terminus is reached. For example, in FIG. 6C, after the ter
successive time interval the velocity may be decreased by 5%. minus is reached the list is scrolled for a distance 3538, which
When the velocity crosses a lower threshold, it may be set to may be less than a distance traversed on the touch screen
ZO.
display by the swipe gesture 3514 after the terminus is
FIG. 5 is a flow diagram illustrating a method 500 of 60 reached.
scrolling through a list in accordance with some embodi In some embodiments, Scrolling in the first direction prior
ments. The method 500 provides a simple visual indicator to to reaching a terminus of the list has a first associated Scroll
a user that a terminus of a list has been reached. ing speed that corresponds to a speed of movement of the
Movement of an object is detected on or near a touchscreen object. For example, a scrolling speed prior to reaching the
display of a device (502). In some embodiments, the object is 65 terminus of the list shown in FIGS. 6A-6D may correspond to
a finger. In some embodiments, the device is a portable mul a speed on the touchscreen display of the swipe gesture 3514
tifunction device. before the terminus is reached. Displaying an area beyond the
 US 7,469,381 B2
25 26
terminus of the list includes scrolling the list in the first upward and/or vertically downward Swipe gestures on the
direction at a second associated Scrolling speed. The second touch screen. In the example of FIG. 6A, a portion of a list of
associated Scrolling speed is slower than the first associated emails is displayed in the screen area, including a top dis
scrolling speed. For example, in FIG. 6C, displaying the area played email 3530 from Bruce Walker and a bottom displayed
3536 beyond the terminus of the list may include scrolling the email 3532 from Kim Brook. A user performs a vertically
list at a speed that is slower than the scrolling speed before the downward swipe gesture 3514 to scroll toward the top of the
terminus is reached. In some embodiments, the second asso list. The vertically downward gesture 3514, which may be a
ciated speed is a fraction (e.g., one-half or one-third) of the finger gesture, corresponds to the movement of an object on
first associated speed. In some embodiments, the second or near the touch screen that is detected in operation 502 of
associated speed is the square root of the first associated 10 process 500 (FIG. 5). The vertically downward gesture 3514
speed. need not be exactly vertical; a Substantially vertical gesture is
Ifa terminus of the list is not reached while scrolling the list Sufficient. In some embodiments, a gesture within a predeter
in the first direction while the object is still detected on or near mined angle of being perfectly vertical results in Vertical
the touch screen display, the process 500 is complete (510 scrolling. In one embodiment, a gesture within 27 degrees of
No. 512). The process 500 may be re-initiated upon subse 15 being perfectly vertical results in Vertical scrolling.
quent detection of another movement of an object on or near As a result of detecting the vertically downward gesture
the touch screen display (502). 3514, in FIG. 6B the displayed emails have shifted down,
FIGS. 6A-6D illustrate the scrolling of a list of items to a such that the previous bottom displayed email 3532 from Kim
terminus of the list, at which point an area beyond the termi Brook is no longer displayed, the previous top displayed
nus is displayed and the list is then scrolled in an opposite email 3530 from Bruce Walker is now second from the top,
direction until the area beyond the terminus is no longer and the email 3534 from Aaron Jones, which was not dis
displayed, in accordance with some embodiments. While played in FIG. 6A, is now displayed at the top of the list. This
FIGS. 6A-6D illustrate this scrolling in the context of a por shifting of emails is an example of the scrolling described in
table multifunction device 100, this scrolling is not limited to operation 504 of process 500 (FIG. 5).
portable multifunction devices. In the example of FIGS. 25 In this example, the email 3534 from Aaron Jones is the
6A-6D, the listed items are email messages: FIGS. 6A-6D first email in the list and thus is the terminus of the list. Upon
illustrate an exemplary user interface 3500A for managing an reaching this email 3534, in response to continued detection
inbox in accordance with some embodiments. An analogous of the vertically downward gesture 3514, an area 3536 (FIG.
user interface may be used to display and manage other mail 6C) above the first email 3534 (i.e., beyond the terminus of
boxes (e.g., drafts, sent, trash, personal, etc.). In addition, 30 the list) is displayed, as described in operation 514 of process
other types of lists are possible, including but not limited to 500 (FIG. 5). In some embodiments, the area displayed
lists of instant message conversations, favorite phone num beyond the terminus of the list is visually indistinct from the
bers, contact information, labels, email folders, email background of the list, as described in operation 518 of pro
addresses, physical addresses, ringtones, album names or cess 500 (FIG. 5). In FIG. 6C, both the area 3536 and the
bookmarks. 35 background of the emails (e.g., emails 3534 and 3530) are
In some embodiments, user interface 3500A include the white and thus are visually indistinct.
following elements, or a Subset or Superset thereof: Once vertically downward gesture 3514 is complete, such
402,404, and 406, as described above; that a corresponding object is no longer detected on or near
a create email icon 3310 that when activated (e.g., by a the touch screen display, the list is scrolled in an opposite
finger tap on the icon) initiates display of a UI to create 40 direction until the area 3536 is no longer displayed. FIG. 6D
a new email message; illustrates the result of this scrolling in the opposite direction,
mailboxes icon 3502 that when activated (e.g., by a finger which corresponds to operation 520 of process 500 (FIG. 5):
tap on the icon) initiates the display of a UI listing email the email 3534 from Aaron Jones is now displayed at the top
mailboxes (i.e., folders); of the screen area allotted to the list and the area 3536 is not
unread messages icon 3504 that displays the number of 45 displayed.
unread messages in the inbox; In the example of FIGS. 6A-6D, a vertically downward
names 3506 of the senders of the email messages; gesture resulted in display of an area beyond the first item in
subject lines 3508 for the email messages: the list. Similarly, a vertically upward gesture may result in
dates 3510 of the email messages; display of an area beyond the last item of the list, if the
unread message icons 3512 that indicate messages that 50 Vertically upward gesture continues once the list has been
have not been opened; scrolled to the last item. The last item may be considered a
preview pane separator 3518 that separates the list of mes terminus of the list, similar to the first item. As discussed
Sages from a preview of a selected message in the list; above, the gesture need not be exactly vertical to result in
settings icon 3520 that when activated (e.g., by a finger tap Vertical Scrolling; a gesture within a predefined range of
on the icon) initiates the display of a UI to modify 55 angles from perfectly vertical is sufficient.
Settings; In some embodiments, instead of Scrolling a list of items in
move message icon 3522 that when activated (e.g., by a one dimension, a user may desire to translate an electronic
finger tap on the icon) initiates the display of a UI to document in two dimensions. If the electronic document fills
move messages; more than the screen area allotted to display the document,
Delete symbol icon 3524 that when activated (e.g., by a 60 the screen will only display a portion of the document. The
finger tap on the icon) initiates display of a UI to confirm user may translate the electronic document to view portions
that the user wants to delete the selected email; of the document that are not initially displayed.
Reply/Forward icon 3526 that when activated (e.g., by a FIG. 7 is a flow diagram illustrating a method 700 of
finger tap on the icon) initiates display of a UI to select translating an electronic document in accordance with some
how to reply or forward the selected email; 65 embodiments. The method 700 provides a simple visual indi
If the list of emails fills more than the allotted screen area, cator to a user that one or more edges of an electronic docu
the user may scroll through the emails using vertically ment are being displayed.
 US 7,469,381 B2
27 28
Movement of an object is detected on or near a touchscreen associated translating distance is less thana distance of move
display of a device (702). In some embodiments, the object is ment of the object after reaching the edge of the electronic
a finger. In some embodiments, the device is a portable mul document. For example, in FIG. 8C, after the edge is reached
tifunction device. the web page 3912 is translated by a distance indicated by
In response to detecting the movement, an electronic docu opposing arrows 3928-1 and 3928-2, which may be less than
ment displayed on the touch screen display is translated in a a distance traversed on the touch screen display by the Swipe
first direction (704). In some embodiments, the electronic gesture 3925 after the terminus is reached.
document is a web page, as illustrated in FIGS. 8A-8D. In In some embodiments, translating in the first direction
Some embodiments, the electronic document is a digital prior to reaching an edge of the electronic document has a first
image. In some embodiments, the electronic document is a 10 associated translating speed that corresponds to a speed of
word processing, spreadsheet, email, or presentation docu movement of the object. For example, a speed of translation
ment. In some embodiments, the first direction is a vertical prior to reaching the edge of the web page 3912 shown in
direction, a horizontal direction, or a diagonal direction. In FIGS. 8A-8D may correspond to a speed of movement of the
Some embodiments, the first direction corresponds to the swipe gesture 3925. Displaying an area beyond the edge of
direction of movement of the object detected on or near the 15 the electronic document includes translating the electronic
display but is not necessarily identical to the direction of document in the first direction at a second associated trans
movement of the object. lating speed. The second associated translating speed is
In some embodiments, translating the electronic document slower than the first associated translating speed. For
in the first direction prior to reaching an edge of the electronic example, in FIG. 8C, displaying the area 3930 beyond the
document has an associated speed of translation correspond edge of the web page 3912 may include translating the web
ing to a speed of movement of the object (706). In some page 3912 at a speed that is slower than the speed of transla
embodiments, the electronic document is translated in accor tion before the edge is reached. In some embodiments, the
dance with a simulation of an equation of motion having second associated speed is a fraction (e.g., one-half or one
friction (708). third) of the first associated speed. In some embodiments, the
If an edge of the electronic document is reached (e.g., upon 25 second associated speed is the square root of the first associ
reaching the edge of the document) while translating the ated speed.
electronic document in the first direction while the object is If an edge of the electronic document is not reached while
still detected on or near the touch screen display, an area translating the electronic document in the first direction while
beyond the edge of the electronic document is displayed the object is still detected on or near the touchscreen display,
(710-Yes, 714). In some embodiments, the area beyond the 30 the process 700 is complete (710-No. 712). The process 700
edge of the electronic document is black, gray, a solid color, may be re-initiated upon Subsequent detection of another
or white (716). In some embodiments, the area beyond the movement of an object on or near the touch screen display
edge of the electronic document is visually distinct from the (702).
document (718). For example, the area 3930 beyond the edge FIGS. 8A-8D illustrate the translation of an electronic
of the web page 3912 in FIG. 8C is black, in contrast to the 35 document to an edge of the document, at which point an area
white background of the web page 3912. In some other beyond the edge is displayed and the document is then trans
embodiments, a wallpaper image such as a picture or pattern lated in a second direction until the area beyond the edge of
may be displayed in the area beyond the edge of the electronic the document is no longer displayed, inaccordance with some
document. embodiments. While FIGS. 8A-8D illustrate this translation
After the object is no longer detected on or near the touch 40 in the context of a portable multifunction device 100, this
screen display, the electronic document is translated in a translation is not limited to portable multifunction devices. In
second direction until the area beyond the edge of the elec the example of FIGS. 8A-8D, the document is a web page
tronic document is no longer displayed (720). For example, in 3912; FIGS. 8A-8D illustrate an exemplary user interface for
FIG.8D the web page 3912 has been translated such that the a browser in accordance with some embodiments. An analo
area 3930 beyond its edge is no longer displayed. In some 45 gous user interface may be used to display other types of
embodiments, the second direction is opposite the first direc electronic documents, such as word processing, spreadsheet,
tion. In some embodiments, the electronic document is trans email, presentation documents, or digital images.
lated in the second direction using a damped motion (722). In In some embodiments, user interface 3900A of FIGS.
Some embodiments, the change from translating the elec 8A-8D includes the following elements, or a subset or super
tronic document in the first direction to translating the elec 50 set thereof:
tronic document in the second direction until the area beyond 402, 404, and 406, as described above;
the edge of the electronic document is no longer displayed Previous page icon 3902 that when activated (e.g., by a
makes the edge of the electronic document appear to be finger tap on the icon) initiates display of the previous
elastically attached to an edge of the touch screen display or web page;
to an edge displayed on the touch screen display (724). 55 Web page name 3904;
In some embodiments, translating in the first direction Next page icon 3906 that when activated (e.g., by a finger
prior to reaching an edge of the electronic document has a first tap on the icon) initiates display of the next web page;
associated translating distance that corresponds to a distance URL (Uniform Resource Locator) entry box 3908 for
of movement of the object prior to reaching the edge of the inputting URLs of web pages;
electronic document. For example, a distance of translation of 60 Refresh icon 3910 that when activated (e.g., by a finger tap
the web page 3912 shown in FIGS. 8A-8D prior to reaching on the icon) initiates a refresh of the web page;
the edge of the document may correspond to a distance tra Web page 3912 or other structured document, which is
versed on the touch screen display by the swipe gesture 3925 made of blocks 3914 of text content and other graphics
before the edge is reached. In some embodiments, displaying (e.g., images):
an area beyond the edge of the electronic document includes 65 Settings icon 3916 that when activated (e.g., by a finger tap
translating the electronic document in the first direction for a on the icon) initiates display of a settings menu for the
second associated translating distance, wherein the second browser;
 US 7,469,381 B2
29 30
Bookmarks icon 3918 that when activated (e.g., by a finger until the area 3930 is no longer displayed. FIG.8D illustrates
tap on the icon) initiates display of a bookmarks list or the result of this translation, which corresponds to operation
menu for the browser; 720 of process 700 (FIG. 7): block 9 (3914-9) is now dis
Add bookmark icon 3920 that when activated (e.g., by a played in the lower right-hand corner of the portion of the
finger tap on the icon) initiates display of a UI for adding screen allotted to display the web page 3912 and the area 3930
bookmarks; and not displayed. In some embodiments, the direction of trans
New window icon 3922 that when activated (e.g., by a lation is not necessarily opposite to the original direction but
finger tap on the icon) initiates display of a UI for adding may be in any direction Such that, upon completion of the
new windows to the browser. translation, the area beyond the edge(s) of the electronic
In some embodiments, the device analyzes the render tree 10 document is no longer displayed.
of the web page3912 to determine the blocks 3914 in the web FIG. 9 is a flow diagram illustrating a process 900 of
page. In some embodiments, a block 3914 corresponds to a displaying an electronic document having a document length
render node that is: replaced; a block; an inline block; or an and a document width, in accordance with Some embodi
inline table.
ments. The process 900 provides a simple visual indicator to
In FIG. 8A, the web page fills more than the allotted screen 15
a user that an electronic document is being displayed at a
area: only the left sides of block 7 (3914-7) and block 8 minimum magnification (e.g., the electronic document can
(3914-8) are displayed and only the top left corner of block 9 not be Zoomed out and/or demagnified further).
(3914-9) is displayed. To view the partially displayed blocks,
a user may translate the displayed document by gesturing on The process 900 is performed at a device with a touch
the touch screen in accordance with some embodiments. screen display. In some embodiments, the device is a portable
In some embodiments, in response to a Substantially verti multifunction device. In some embodiments, the electronic
cal upward (or downward) swipe gesture by the user, the web document is a web page (e.g., web page 3912, FIGS. 10A
page (or, more generally, other electronic documents) may 10C). In some embodiments, the electronic document is a
translate one-dimensionally upward (or downward) in the digital image. In some embodiments, the electronic document
Vertical direction. In some embodiments, a gesture is consid 25 is a word processing, spreadsheet, email or presentation
ered substantially vertical if it is within a predetermined angle document.
of being perfectly vertical. For example, in response to an The electronic document is displayed (902) at a first mag
upward Swipe gesture by the user that is within a predeter nification on the touch screen display. A gesture is detected
mined angle (e.g., 27°) of being perfectly vertical, the web (904) on or near the touch screen display corresponding to a
page may scroll one-dimensionally upward in the vertical 30 command to Zoom out by a user-specified amount. In some
direction. embodiments, the gesture is a pinching gesture (e.g., gesture
Conversely, in some embodiments, in response to a gesture 3951/3953, FIG. 10A).
that is not within a predetermined angle (e.g., 27°) of being In response to detecting the gesture, the electronic docu
perfectly vertical, the web page may translate two-dimen ment is displayed (906) at a magnification less than the first
sionally (i.e., with simultaneous movement in both the verti 35 magnification. For example, the web page 3912 is shown at a
cal and horizontal directions). For example, in response to an lesser magnification in FIG. 10B than in FIG. 10A.
upward Swipe gesture by the user that is not within a prede If the document length or document width is not entirely
termined angle (e.g., 27°) of being perfectly vertical, the web displayed (908-No) while the gesture is still detected on or
page may translate two-dimensionally along the direction of near the touch screen display, the process 900 is complete
the Swipe. 40
(910).
In the example of FIG. 8A, an upward swipe gesture 3925 If, however, the document length (e.g., 3957, FIG. 10B) or
is not within a predetermined angle of being perfectly verti document width (e.g., 3959, FIG. 10B) is entirely displayed
cal. Therefore, as a result of detecting the upward Swipe (908-Yes) while the gesture (e.g.,3951/3953) is still detected
gesture 3925, the web page is translated in two dimensions. In on or near the touchscreen display, the electronic document is
this example, the translation is approximately diagonal. FIG. 45
displayed (912) at a magnification wherein areas beyond
8B illustrates the result of this translation: blocks 8 (3914-8) opposite edges of the electronic document (e.g., areas 3955,
and 9 (3914-9) are now fully displayed; blocks 1 (3914-1)and FIG. 10B) are displayed.
2 (3914-2) are now only partially displayed, and block 3 In some embodiments, the areas beyond opposite edges of
(3914-3) is no longer displayed at all. This translation is an the electronic document include an area beyond a top edge of
example of the translation described in operation 704 of pro 50
cess 700 (FIG. 7). the document and an area beyond a bottom edge of the docu
In FIG. 8B, block 9 (3914-9) is in the lower right-hand ment. In some embodiments, the areas beyond opposite edges
corner of the web page 3912; both the bottom and right edges of the electronic document include an area beyond a right
of the web page have been reached while translating the web edge of the document and an area beyond a left edge of the
page. Upon reaching these edges of the document, in 55 document. In some embodiments, the areas beyond opposite
response to continued detection of the upward gesture 3925, edges of the electronic document include an area beyond atop
an area 3930 (FIG. 8C) beyond the bottom and right edges of edge of the document, an area beyond a bottom edge of the
the web page is displayed. In some embodiments, the area document, an area beyond a right edge of the document, and
displayed beyond the edge(s) of an electronic document is an area beyond a left edge of the document (e.g., FIG. 10B).
visually distinct from the document, as described in operation 60 In some embodiments, the areas beyond opposite edges of
718 of process 700 (FIG.7). In FIG.8C, the area 3930 is black the electronic document are black, gray, a Solid color, or
and thus is visually distinct from the white background of the white. In some embodiments, the areas beyond opposite
web page 3912. edges of the electronic document are visually distinct from
Once the upward gesture 3925 is complete, such that a the document. For example, the areas 3955 (FIG. 10B) are
corresponding object is no longer detected on or near the 65 black and thus are visually distinct from the web page 3912.
touch screen display, the web page 3912 is translated (e.g., in Upon detecting termination of the gesture, the electronic
a direction opposite to the original direction of translation) document is displayed (914) at a magnification wherein the
 US 7,469,381 B2
31 32
areas beyond opposite edges of the electronic document are gesture 3931/3933, a portion of the web page 3912 is dis
no longer displayed. For example, the areas 3955 are not played at the predefined magnification, as illustrated in FIG.
displayed in FIG. 10C. 12C, and a portion of the digital image 1302 is displayed at the
FIGS. 10A-10C illustrate the display of an electronic docu predefined magnification, as illustrated in FIG. 13C.
ment at multiple magnifications in accordance with some In some embodiments, immediately prior to detecting ter
embodiments. While FIGS. 10A-10C illustrate displaying mination of the gesture, a last decreased portion of the elec
these multiple magnifications in the context of a portable tronic document is displayed at a first resolution. Upon
multifunction device 100, displaying these multiple magnifi detecting termination of the gesture, the respective portion of
cations is not limited to portable multifunction devices. In the the electronic document is displayed at a second resolution
example of FIGS. 10A-10C, the document is a web page 10 that is greater than the first resolution.
3912; FIGS. 10A-10C (like FIGS. 8A-8D) illustrate an exem FIGS. 12A-12C illustrate the display of an electronic docu
plary user interface for a browser in accordance with some ment at multiple magnifications in accordance with some
embodiments. An analogous user interface may be used to embodiments. While FIGS. 12A-12C illustrate displaying
display other types of electronic documents, such as digital these multiple magnifications in the context of a portable
images or word processing, spreadsheet, email, or presenta 15 multifunction device 100, displaying these multiple magnifi
tion documents. cations is not limited to portable multifunction devices. In the
In FIG. 10A, the web page 3912 is displayed at a first example of FIGS. 12A-12C, the document is a web page
magnification. The web page 3912 fills more than the allotted 3912; FIGS. 12A-12C (like FIGS. 8A-8D) illustrate an exem
screen area: only the left sides of block 7 (3914-7) and block plary user interface for a browser in accordance with some
8 (3914-8) are displayed and only the top left corner of block embodiments. An analogous user interface may be used to
9 (3914-9) is displayed. display other types of electronic documents, such as digital
In response to detecting a pinching gesture 3951/3953 images or word processing, spreadsheet, email, or presenta
(FIG. 10A), the web-page is displayed at a magnification less tion documents.
than the first magnification, as shown in FIG. 10B. If a docu In FIG. 12A, a first portion of the web page 3912 is dis
ment length 3957 or a document width 3959 is entirely dis 25
played at a first magnification. The web page 3912 fills more
played while the gesture 3951/3953 is still detected, areas than the allotted screen area: only the left sides of block 7
3955 beyond opposite edges of the web page 3912 are dis (3914-7) and block 8 (3914-8) are displayed and only the top
played. Upon detecting termination of the gesture 3951/3953, left corner of block 9 (3914-9) is displayed.
the web page 3912 is displayed at a magnification wherein the In response to detecting a de-pinching gesture 3931/3933
areas 3955 are no longer displayed, as shown in FIG. 10C. 30
(FIG. 12A), decreasing portions of the web-page 3912 are
FIG. 11 is a flow diagram illustrating a process 1100 of displayed at increasing magnifications compared to the mag
displaying an electronic document at multiple magnifications nification shown in FIG. 12A. For example, the portion of the
in accordance with some embodiments. The process 1100 web page 3912 shown in FIG. 12B is smaller than and has a
provides a simple visual indicator to a user that an electronic higher magnification than the portion of the web page 3912
document is being displayed at a maximum magnification 35
shown in FIG. 12A.
(e.g., the electronic document cannot be Zoomed in and/or In the example of FIG. 12B, the magnification exceeds a
magnified further). predefined magnification. Upon detecting termination of the
The process 1100 is performed at a device with a touch gesture 3931/3933, a portion of the web page 3912 is dis
screen display. In some embodiments, the device is a portable played at the predefined magnification, as illustrated in FIG.
multifunction device. In some embodiments, the electronic 40
12C
document is a web page (e.g., web page 3912, FIGS. 12A
12C). In some embodiments, the electronic document is a FIGS. 13 A-13C illustrate the display of an electronic docu
digital image (e.g., digital image 1302, FIGS. 13 A-13C). In ment at multiple magnifications in accordance with some
Some embodiments, the electronic document is a word pro embodiments. While FIGS. 13 A-13C illustrate displaying
cessing, spreadsheet, email or presentation document. 45 these multiple magnifications in the context of a portable
At least a first portion of the electronic document is dis multifunction device 100, displaying these multiple magnifi
played (1102) at a first magnification. A gesture is detected cations is not limited to portable multifunction devices. In the
(1104) on or near the touch screen display corresponding to a example of FIGS. 13 A-13C, the document is a digital image
command to Zoom in by a user-specified amount. In some 1302 that includes an image of a person 1304.
embodiments, the gesture is a de-pinching gesture (e.g., 50 In FIG. 13A, a digital image 1302 is displayed at a first
3931/3933, FIGS. 12A and 13A). magnification. In response to detecting a de-pinching gesture
In response to detecting the gesture, decreasing portions of 3931/3933, decreasing portions of the digital image 1302 are
the electronic document are displayed (1106) at increasing displayed at increasing magnifications compared to the mag
magnifications. For example, in FIG.12B a decreased portion nification shown in FIG. 13A. For example, the portion of the
of the web page 3912 is displayed at a higher magnification 55 digital image 1302 shown in FIG. 13B is smaller than and has
than the portion in FIG. 12A, and in FIG. 13B a decreased a higher magnification than the portion of the digital image
portion of the digital image 1302 is displayed at a higher 1302 shown in FIG. 13A.
magnification than the portion in FIG. 13A. In the example of FIG. 13B, the magnification exceeds a
If, upon detecting termination of the gesture, the magnifi predefined magnification. Upon detecting termination of the
cation does not exceed a predefined magnification (1108-No), 60 gesture 3931/3933, a portion of the digital image 1302 is
the process 1100 is complete (1110). displayed at the predefined magnification, as illustrated in
If, however, upon detecting termination of the gesture, the FIG. 13C.
magnification exceeds a predefined magnification FIG. 14 is a flow diagram illustrating a process 1400 of
(1108-Yes), a respective portion of the electronic document is executing a screen rotation command in accordance with
displayed (1112) at the predefined magnification. In the 65 some embodiments. The process 1400 provides a simple
examples of FIGS. 12B and 13B, the magnification exceeds a visual indicator to a user that the user has not provided a
predefined magnification. Upon detecting termination of the sufficient gesture to initiate a 90° screen rotation command.
 US 7,469,381 B2
33 34
The process 1400 is performed at a device with a touch ments, the first sense of rotation is a clockwise rotation (e.g.,
screen display. In some embodiments, the device is a portable FIG.16C). The simultaneous rotation of the two thumbs has
multifunction device. a corresponding degree of rotation.
A multifinger twisting gesture (e.g., 1506, FIG. 15A, or In some embodiments, the sense of rotation for each thumb
1508, FIG.15C) is detected (1402) on or near the touchscreen 5 is detected by monitoring the change in orientation of the
display. The multifinger twisting gesture has a corresponding contact area of the thumb with the touch screen display. For
degree of rotation. In some embodiments, the multifinger example, if the contact area of the thumb is elliptical, the
twisting gesture includes gestures by two thumbs 1604-L and change in the orientation of an axis of the ellipse may be
1604-R (FIGS. 16A and 16D) detected (e.g., from contact ellipse 1606-L in FIG. 16A to
If the corresponding degree of rotation exceeds a pre 10 contact ellipse 1608-L in FIG. 16B, as shown on an enlarged
defined degree of rotation (1404-Yes), a 90° screen rotation portion of touch screen 112 in FIG. 16C). In some embodi
command is executed (1406). For example, the digital image ments, the change in the orientation of the axis of the ellipse
1502 of FIGS. 15A and 16A is rotated from a portrait orien determines the corresponding degree of rotation. In some
tation to a landscape orientation, as shown respectively in embodiments, at least Some of a user's other fingers (i.e.,
FIGS. 15B and 16B. 15 fingers other than thumbs 1604-L and 1604-R) support the
If the corresponding degree of rotation does not exceed a device 100 by contacting the backside of the device.
predefined degree of rotation (1404-No), a screen rotation In some embodiments, the first sense of rotation is a coun
command with an acute angle of rotation (i.e., less than 90°) terclockwise rotation. For example, if thumb 1604-L is ini
is executed (1408). For example, the digital image 1502 of tially on the lower left side of touchscreen 112 (rather than the
FIGS. 15C and 16D is rotated by an acute angle, as shown upper left side in FIG.16A), thumb 1604-R is initially on the
respectively in FIGS. 15D and 16E. Upon ceasing to detect upper right side of touch screen 112 (rather than the lower
the multifingertwisting gesture, a screen rotation commandis right side in FIG.16A), and the thumbs are moved apart from
executed (1410) with an angle of rotation opposite to the each other, then the sense of rotation detected by the touch
acute angle (e.g., with the result shown in FIGS. 15E and screen 112 will be counterclockwise for both thumbs.
16F). 25 If the corresponding degree of rotation exceeds a pre
FIGS. 15A-15E illustrate rotating the display of an elec defined degree of rotation, a 90° screen rotation command is
tronic document or other digital object in accordance with executed. For example, display of the digital image 1502 is
some embodiments. While FIGS. 15A-15E illustrate display rotated from the portrait orientation of FIG. 16A to a land
rotation in the context of a portable multifunction device 100, scape orientation in FIG. 16B.
display rotation is not limited to portable multifunction 30 If, however, the corresponding degree of rotation does not
devices. In the example of FIGS. 15A-15E, the electronic exceed a predefined degree of rotation, a screen rotation com
document is a digital image 1502. mand with an acute angle of rotation is executed. For
In FIGS. 15A and 15C, the digital image 1502 is displayed example, the digital image 1502 in FIG. 16D is rotated by an
in a portrait orientation. A multifinger twisting gesture 1506 acute angle, with the result shown in FIG. 16E. Once detec
(FIG.15A) or 1508 (FIG.15C) is detected on the touchscreen 35 tion of the two thumbs 1604-L and 1604-R ceases, a screen
display. The multifinger twisting gesture 1506 or 1508 has a rotation command with an angle of rotation opposite to the
corresponding degree of rotation. In some embodiments, the acute angle is executed, thereby restoring the digital image
degree of rotation corresponds to a degree of rotation of an 1502 to a portrait orientation, as shown in FIG. 16F.
axis between the contact points on the touchscreen display of While FIGS. 6A-6D, 8A-8D, 10A-10C, 12A-12C, 13 A
the two fingers in the multifinger gesture (e.g., an axis 40 13C, 15A-15E, and 16A-16F illustrate scrolling, translation,
between the center points or centroids of the contact regions Scaling, and rotation operations in the context of a portable
of the two fingers). multifunction device 100, similar operations may be per
In the example of FIG. 15A, the multifinger twisting ges formed on any device with a touch-screen display, in accor
ture 1506 has a corresponding degree of rotation that exceeds dance with some embodiments. The device, such as device
a predefined degree of rotation. Thus, a 90° screen rotation 45 1700 below, may or may not be portable and the function or
command is executed, with the result that the digital image is functions performed by the device may vary.
displayed in a landscape orientation, as shown in FIG.15B. In FIG. 17 is a block diagram illustrating a device 1700 with
the example of FIG. 15C, however, the multifinger twisting a touch-screen display in accordance with some embodi
gesture 1508 has a corresponding degree of rotation that does ments. Device 1700 need not be portable. The device 1700
not exceed a predefined degree of rotation. A screen rotation 50 typically includes one or more processing units (CPUs)
command with an acute angle of rotation is executed, with the 1710, one or more network or other communications inter
result shown in FIG. 15D. Upon ceasing to detect the multi faces 1760, memory 1770, and one or more communication
finger twisting gesture 1508, a screen rotation command with buses 1720 for interconnecting these components. The com
an angle opposite to the acute angle is executed, with the munication buses 1720 may include circuitry (sometimes
result that the portrait orientation of the digital image 1502 is 55 called a chipset) that interconnects and controls communica
restored, as shown in FIG. 15E. tions between system components. The device 1700 includes
FIGS. 16A-16F illustrate an exemplary screen rotation a user interface 1730 comprising a touch-screen display 1740.
gesture in accordance with some embodiments. While FIGS. The user interface 1730 also may include a keyboard and/or
16A-16F illustrate this screen rotation gesture in the context mouse (or other pointing device) 1750. Memory 1770
of a portable multifunction device 100, this screen rotation 60 includes high-speed random access memory, such as DRAM,
gesture is not limited to portable multifunction devices. In the SRAM, DDR RAM or other random access solid state
example of FIGS. 16A-16F, this screen rotation gesture is memory devices; and may include non-volatile memory, Such
used to rotate the digital image 1502. as one or more magnetic disk storage devices, optical disk
In FIG.16A, the device 100 displays the digital image 1502 storage devices, flash memory devices, or other non-volatile
in a portraitorientation. Simultaneous rotation of two thumbs 65 solid state storage devices. Memory 1770 may optionally
(e.g., 1604-L and 1604-R) in a first sense of rotation is include one or more storage devices remotely located from
detected on the touch screen display 112. In some embodi the CPU(s) 1710. In some embodiments, memory 1770 stores
 US 7,469,381 B2
35 36
programs, modules, and data structures analogous to the pro 5. The computer-implemented method of claim 1, wherein
grams, modules, and data structures stored in the memory 102 the first direction is a vertical direction, a horizontal direction,
of portable multifunction device 100 (FIG. 1), or a subset or a diagonal direction.
thereof. Furthermore, memory 1770 may store additional 6. The computer-implemented method of claim 1, wherein
programs, modules, and data structures (not shown) not the electronic document is a web page.
present in the memory 102 of portable multifunction device 7. The computer-implemented method of claim 1, wherein
1OO. the electronic document is a digital image.
Each of the above identified elements in FIG. 17 may be 8. The computer-implemented method of claim 1, wherein
stored in one or more of the previously mentioned memory the electronic document is a word processing, spreadsheet,
devices. Each of the above identified modules corresponds to 10 email or presentation document.
a set of instructions for performing a function described 9. The computer-implemented method of claim 1, wherein
above. The above identified modules or programs (i.e., sets of the electronic document includes a list of items.
instructions) need not be implemented as separate Software 10. The computer-implemented method of claim 1,
programs, procedures or modules, and thus various Subsets of wherein the second direction is opposite the first direction.
these modules may be combined or otherwise re-arranged in 15
11. The computer-implemented method of claim 1,
various embodiments. In some embodiments, memory 1770 wherein translating in the first direction prior to reaching an
may store a Subset of the modules and data structures identi edge of the document has an associated speed of translation
fied above. Furthermore, memory 1770 may store additional that corresponds to a speed of movement of the object.
modules and data structures not described above. 12. The computer-implemented method of claim 1,
The foregoing description, for purpose of explanation, has whereintranslating in the first direction is in accordance with
been described with reference to specific embodiments. How a simulation of an equation of motion having friction.
ever, the illustrative discussions above are not intended to be 13. The computer-implemented method of claim 1,
exhaustive or to limit the invention to the precise forms dis wherein the area beyond the edge of the document is black,
closed. Many modifications and variations are possible in gray, a Solid color, or white.
view of the above teachings. The embodiments were chosen 25
14. The computer-implemented method of claim 1,
and described in order to best explain the principles of the wherein the area beyond the edge of the document is visually
invention and its practical applications, to thereby enable distinct from the document.
others skilled in the art to best utilize the invention and vari
ous embodiments with various modifications as are Suited to 15. The computer-implemented method of claim 1,
the particular use contemplated. 30 wherein translating the document in the second direction is a
damped motion.
What is claimed is: 16. The computer-implemented method of claim 1,
1. A computer-implemented method, comprising: wherein changing from translating in the first direction to
at a device with a touch screen display: translating in the second direction until the area beyond the
displaying a first portion of an electronic document; 35 edge of the document is no longer displayed makes the edge
detecting a movement of an object on or near the touch of the electronic document appear to be elastically attached to
screen display; an edge of the touch screen display or to an edge displayed on
in response to detecting the movement, translating the the touch screen display.
electronic document displayed on the touch screen 17. The computer-implemented method of claim 1,
display in a first direction to display a second portion 40 wherein translating in the first direction prior to reaching the
of the electronic document, wherein the second por edge of the electronic document has a first associated trans
tion is different from the first portion; lating distance that corresponds to a distance of movement of
in response to an edge of the electronic document being the object prior to reaching the edge of the electronic docu
reached while translating the electronic document in ment; and wherein displaying an area beyond the edge of the
the first direction while the object is still detected on 45 electronic document comprises translating the electronic
or near the touch screen display: document in the first direction for a second associated trans
displaying an area beyond the edge of the document, lating distance, wherein the second associated translating
and distance is less than a distance of movement of the object after
displaying a third portion of the electronic document, reaching the edge of the electronic document.
wherein the third portion is smaller than the first 50 18. The computer-implemented method of claim 1,
portion; and wherein translating in the first direction prior to reaching the
in response to detecting that the object is no longer on edge of the electronic document has a first associated trans
or near the touch screen display, translating the lating speed that corresponds to a speed of movement of the
electronic document in a second direction until the object, and wherein displaying an area beyond the edge of the
area beyond the edge of the electronic document is 55 electronic document comprises translating the electronic
no longer displayed to display a fourth portion of document in the first direction at a second associated trans
the electronic document, wherein the fourth por lating speed, wherein the second associated translating speed
tion is different from the first portion. is slower than the first associated translating speed.
2. The computer-implemented method of claim 1, wherein 19. A device, comprising:
the first portion of the electronic document, the second por 60 a touch screen display;
tion of the electronic document, the third portion of the elec one or more processors;
tronic document, and the fourth portion of the electronic memory; and
document are displayed at the same magnification. one or more programs, wherein the one or more programs
3. The computer-implemented method of claim 1, wherein are stored in the memory and configured to be executed
the movement of the object is on the touch screen display. 65 by the one or more processors, the programs including:
4. The computer-implemented method of claim 1, wherein instructions for displaying a first portion of an electronic
the object is a finger. document;
 US 7,469,381 B2
37 38
instructions for detecting a movement of an object on or display a first portion of an electronic document;
near the touch screen display; detect a movement of an object on or near the touch screen
instructions for translating the electronic document dis- display;
played on the touch screen display in a first direction translate the electronic document displayed on the touch
to display a second portion of the electronic docu- 5 Screen display in a first direction to display a second
ment, wherein the second portion is different from the portion of the electronic document, wherein the second
first portion, in response to detecting the movement; portion is different from the first portion, in response to
instructions for displaying an area beyond an edge of the detecting the movement
electronic document and displaying a third portion of display an area beyond an edge of the electronic document
the electronic document, wherein the third portion is 10 and display a third portion of the electronic document,
Smaller than the first portion, in response to the edge
of the electronic document being reached while trans wherein the third portion is smaller than the first portion,
lating the electronic document in the first direction if the edge of the electronic document is reached while
while the object is still detected on or near the touch translating the electronic document in the first direction
screen display; and 15 while the object is still detected on or near the touch
instructions for translating the electronic document in a Screen display; and
second direction until the area beyond the edge of the translate the electronic document in a second direction
electronic document is no longer displayed to display until the area beyond the edge of the electronic docu
a fourth portion of the electronic document, wherein ment is no longer displayed to display a fourthportion of
the fourth portion is different from the first portion, in 20 the electronic document, wherein the fourth portion is
response to detecting that the object is no longer on or different from the first portion, in response to detecting
near the touch screen display. that the object is no longer on or near the touch screen
20. A computer readable storage medium having stored display.
therein instructions, which when executed by a device with a
touch screen display, cause the device to: k . . . .
 UNITED STATES PATENT AND TRADEMARK OFFICE
CERTIFICATE OF CORRECTION

PATENT NO. : 7.469,381 B2 Page 1 of 1

APPLICATIONNO. : 11/956969
DATED : December 23, 2008
INVENTOR(S) : Ording
It is certified that error appears in the above-identified patent and that said Letters Patent is
hereby corrected as shown below:

Column 38, line 8, please insert -- ; -- after movement.

Signed and Sealed this

Seventeenth Day of February, 2009

4 (O-e-
JOHN DOLL
Acting Director of the United States Patent and Trademark Office
 US007469381C1

(12) EX PARTE REEXAMINATION CERTIFICATE (8187th)

United States Patent (10) Number: US 7.469,381 C1
Ording (45) Certificate Issued: Apr. 26, 2011
(54) LIST SCROLLING AND DOCUMENT (56) References Cited
TRANSLATION, SCALING, AND ROTATION
ON A TOUCH-SCREEN DISPLAY U.S. PATENT DOCUMENTS
5,463,725 A 10/1995 Henckel et al. ............. 395/155
(75) Inventor: Bas Ording, San Francisco, CA (US) 6,690,387 B2 2/2004 Zimmerman et al.
2005/O195154 A1 9, 2005 Robbins et al.
(73) Assignee: Apple Inc., Cupertino, CA (US) FOREIGN PATENT DOCUMENTS
Reexamination Request: AU 2007283771 A1 4, 2008
No. 90/010,963, Apr. 28, 2010 CN 1695.105. A 1 1/2005
JP 02140822 5, 1990
Reexamination Certificate for: JP O327 1976 12/1991
Patent No.: 7.469,381 OTHER PUBLICATIONS
Issued: Dec. 23, 2008
Appl. No.: 11A956,969 Forlines et al., Glimpse: A Novel Input Model for Multi
Filed: Dec.e -1 rs 2007 Level Devices, Apr. 2005, 6 pages total, Mitsubishi Electric
Research Laboratories, Cambridge, MA (Exhibit B).
Certificate of Correction issued Feb. 17, 2009. Millhollon et al., Microsoft Office Word 2003 Inside Out,
2003, Microsoft Press, Redmond, Washington, pp. 93,
762–765. (Exhibit C).
Related U.S. Application Data
(60) Provisional application No. 60/879,253, filed on Jan. 7, Primary Examiner—Rachna S. Desai
2007, provisional application No. 60/883,801, filed on Jan. (57) ABSTRACT
7, 2007, provisional application No. 60/879,469, filed on
Jan. 8, 2007, provisional application No. 60/945,858, filed In accordance with some embodiments, a computer
on Jun. 22, 2007, provisional application No. 60/946,971,
filed on Jun. 28, 2007, and provisional application No. implemented method for use in conjunction with a device
60/937,993, filed on Jun. 29, 2007. with a touch screen display is disclosed. In the method, a
(51) Int. Cl. movement of an object on or near the touch screen display is
G06F 3/0 (2006.01) detected. In response to detecting the movement, an elec
G06F 3/048 (2006.01) tronic document displayed on the touch screen display is
G06F 3/033 (2006.01) translated in a first direction. If an edge of the electronic
G06F 3/4 (2006.01) document is reached while translating the electronic docu
ment in the first direction while the object is still detected on
or near the touch screen display, an area beyond the edge of
(52) U.S. Cl. ........................ 715/702; 715/764; 715/769; the document is displayed. After the object is no longer
715/863; 715/864 detected on or near the touch screen display, the document is
(58) Field of Classification Search ................... 71.57702 translated in a second direction until the area beyond the
See application file for complete search history. edge of the document is no longer displayed.
Prassifiction newlce
2i o
Optical
Proximity
390A (speatera) G.G.) G.
Current Time 404
-
Welcome 394 328 s
http:lwww.company.com/start 3908 ( file
EBlock 4 Block
3914
 US 7,469,381 C1
1. 2
EX PARTE AS A RESULT OF REEXAMINATION, IT HAS BEEN
REEXAMINATION CERTIFICATE DETERMINED THAT:
ISSUED UNDER 35 U.S.C. 307 5 The patentability of claims 1-20 is confirmed.
NO AMENDMENTS HAVE BEEN MADE TO
THE PATENT k . . . .
 US007469381 C2

(12) EX PARTE REEXAMINATION CERTIFICATE (9758th)

United States Patent (10) Number: US 7.469,381 C2
Ording (45) Certificate Issued: Jul. 17, 2013
(54) LIST SCROLLING AND DOCUMENT (52) U.S. Cl.
TRANSLATION, SCALING, AND ROTATION USPC ........... 715/702; 715/764; 715/769; 715/863;
ON A TOUCH-SCREEN DISPLAY 71.5/864

(75) Inventor: Bas Ording, San Francisco, CA (US) (58) Field of Classification Search
None
(73) Assignee: Apple Inc., Cupertino, CA (US) See application file for complete search history.
Reexamination Request: (56) References Cited
No. 90/012,304, May 23, 2012 To view the complete listing of prior art documents cited
Reexamination Certificate for: during the proceeding for Reexamination Control Number
Patent No.: 7.469,381 90/012,304, please refer to the USPTO's public Patent
Issued: Dec. 23, 2008 Application Information Retrieval (PAIR) system under the
Appl. No.: 11/956,969 Display References tab.
Filed: Dec. 14, 2007
Primary Examiner — Dennis Bonshock
Reexamination Certificate C1 7.469,381 issued Apr. 26, 2011 (57) ABSTRACT
Certificate of Correction issued Feb. 17, 2009 In accordance with some embodiments, a computer-imple
Related U.S. Application Data mented method for use in conjunction with a device with a
touch screen display is disclosed. In the method, a movement
(60) Provisional application No. 60/879,253, filed on Jan. of an object on or near the touch screen display is detected. In
7, 2007, provisional application No. 60/883,801, filed response to detecting the movement, an electronic document
on Jan. 7, 2007, provisional application No. displayed on the touch screen display is translated in a first
60/879,469, filed on Jan. 8, 2007, provisional direction. If an edge of the electronic document is reached
application No. 60/945,858, filed on Jun. 22, 2007, while translating the electronic document in the first direction
provisional application No. 60/946,971, filed on Jun. while the object is still detected on or near the touch screen
28, 2007, provisional application No. 60/937,993, display, an area beyond the edge of the document is displayed.
filed on Jun. 29, 2007. After the object is no longer detected on or near the touch
screen display, the document is translated in a second direc
(51) Int. C. tion until the area beyond the edge of the document is no
G06F 3/0 (2006.01) longer displayed.

Portable Multifunction bewice

100
Optical Proximity

1 402 Current Time 404 48

-
Welcome 3904 3906

http://www.company.com/start 3908 eft

3914-1 Block 4 Block 7

W
W

Block
3914-6
 US 7,469,381 C2
1.
EX PARTE
REEXAMINATION CERTIFICATE
ISSUED UNDER 35 U.S.C. 307
THE PATENT IS HEREBY AMENDED AS
INDICATED BELOW.

AS A RESULT OF REEXAMINATION, IT HAS BEEN

DETERMINED THAT: 10

The patentability of claims 14, 17, 18 and 19 is confirmed.

Claims 1-13, 15, 16 and 20 are cancelled.
k k k k k