Foundational

Python for Data

Science

9780136624356_print.indb 1 30/07/21 4:16 PM

This page intentionally left blank
 Foundational
Python for Data
.
Science
Kennedy R. Behrman

Boston • Columbus • New York • San Francisco • Amsterdam • Cape Town

Dubai • London • Madrid • Milan • Munich • Paris • Montreal • Toronto • Delhi • Mexico City
São Paulo • Sydney • Hong Kong • Seoul • Singapore • Taipei • Tokyo

9780136624356_print.indb 3 30/07/21 4:16 PM

 Many of the designations used by manufacturers and sellers to distinguish their products Editor-in-Chief
are claimed as trademarks. Where those designations appear in this book, and the Mark Taub
publisher was aware of a trademark claim, the designations have been printed with initial
Acquisitions Editor
capital letters or in all capitals.
Malobika
The author and publisher have taken care in the preparation of this book, but make no Chakraborty
expressed or implied warranty of any kind and assume no responsibility for errors or Development Editor
omissions. No liability is assumed for incidental or consequential damages in connection Mark Renfrow
with or arising out of the use of the information or programs contained herein.
Managing Editor
For information about buying this title in bulk quantities, or for special sales opportunities Sandra Schroeder
(which may include electronic versions; custom cover designs; and content particular to
your business, training goals, marketing focus, or branding interests), please contact our Senior Project
Editor
corporate sales department at [email protected] or (800) 382-3419.
Lori Lyons
For government sales inquiries, please contact [email protected].  Copy Editor
For questions about sales outside the U.S., please contact [email protected].  Kitty Wilson
Production
Visit us on the Web: informit.com/aw.
Manager
Library of Congress Control Number: 2021940284 Aswini Kumar/
codeMantra
Copyright © 2022 Pearson Education, Inc.
Indexer
All rights reserved. This publication is protected by copyright, and permission must be Timothy Wright
obtained from the publisher prior to any prohibited reproduction, storage in a retrieval
Proofreader
system, or transmission in any form or by any means, electronic, mechanical, photocopying,
Abigail Manheim
recording, or likewise. For information regarding permissions, request forms and the
appropriate contacts within the Pearson Education Global Rights & Permissions Department, Compositor
please visit www.pearson.com/permissions/. codeMantra
ISBN-13: 978-0-13-662435-6
ISBN-10: 0-13-662435-9
ScoutAutomatedPrintCode

9780136624356_print.indb 4 30/07/21 4:16 PM

 ❖

This book is dedicated to Tatiana, Itta, and Maple,

who is probably still under the bed.
❖

9780136624356_print.indb 5 30/07/21 4:16 PM

 Contents at a Glance
Preface  xiii

I: Learning Python in a Notebook Environment  1

1 Introduction to Notebooks  3

2 Fundamentals of Python  13

3 Sequences  25

4 Other Data Structures  37

5 Execution Control  55

6 Functions  67

II: Data Science Libraries  83

7 NumPy  85

8 SciPy  103

9 Pandas  113

10 Visualization Libraries  135

11 Machine Learning Libraries  153

12 Natural Language Toolkit  159

III: Intermediate Python  171

13 Functional Programming  173

14 Object-Oriented Programming  187

15 Other Topics  201

A Answers to End-of-Chapter Questions  215

Index  221

9780136624356_print.indb 6 30/07/21 4:16 PM

 Table of Contents
Preface  xiii

I: Learning Python in a Notebook Environment 1

1 Introduction to Notebooks 3
Running Python Statements 4
Jupyter Notebooks 4
Google Colab 5
Colab Text Cells 6
Colab Code Cells 9
Colab Files 9
Managing Colab Documents 10
Colab Code Snippets 11
Existing Collections 11
System Aliases 11
Magic Functions 12
Summary 12
Questions 12

2 Fundamentals of Python 13
Basic Types in Python  14
High-Level Versus Low-Level Languages 15
Statements 15
Performing Basic Math Operations 21
Using Classes and Objects with Dot Notation 22
Summary 22
Questions 23

3 Sequences 25
Shared Operations 25
Testing Membership 26
Indexing 26
Slicing 27
Interrogation 27
Math Operations 28
Lists and Tuples 29

9780136624356_print.indb 7 30/07/21 4:16 PM

 viii Contents

Creating Lists and Tuples 29

Adding and Removing List Items 30
Unpacking 31
Sorting Lists 32
Strings 32
Ranges 34
Summary 35
Questions 35

4 Other Data Structures 37

Dictionaries 37
Creating Dictionaries 38
Accessing, Adding, and Updating by Using Keys 38
Removing Items from Dictionaries 39
Dictionary Views 40
Checking to See If a Dictionary Has a Key 43
The get Method 43
Valid Key Types 44
The hash Method 45
Sets 46
Set Operations 48
Frozensets 53
Summary 53
Questions 53

5 Execution Control 55
Compound Statements 55
Compound Statement Structure 56
Evaluating to True or False  56
if Statements 59
while Loops 62
for Loops 63
break and continue Statements 64
Summary 65
Questions 65

6 Functions 67
Defining Functions 67
Control Statement 68

A01_Behrman_FM_pi-xvi.indd 8 31/07/21 6:22 PM

 Contents ix

Docstrings 68
Parameters 69
Return Statements 75
Scope in Functions 75
Decorators 76
Anonymous Functions 80
Summary 81
Questions 81

II: Data Science Libraries 83

7 NumPy 85
Installing and Importing NumPy 86
Creating Arrays 86
Indexing and Slicing 89
Element-by-Element Operations 91
Filtering Values 92
Views Versus Copies 94
Some Array Methods 95
Broadcasting 98
NumPy Math 100
Summary 102
Questions 102

8 SciPy 103
SciPy Overview 103
The scipy.misc Submodule 104
The scipy.special Submodule 105
The scipy.stats Submodule 105
Discrete Distributions 105
Continuous Distributions 108
Summary 111
Questions 111

9 Pandas 113
About DataFrames 113
Creating DataFrames 114
Creating a DataFrame from a Dictionary 114

9780136624356_print.indb 9 30/07/21 4:16 PM

 x Contents

Creating a DataFrame from a List of Lists 115

Creating a DataFrame from a File 116
Interacting with DataFrame Data 117
Heads and Tails 117
Descriptive Statistics 118
Accessing Data 120
Bracket Syntax 121
Optimized Access by Label 123
Optimized Access by Index 124
Masking and Filtering 125
Pandas Boolean Operators 126
Manipulating DataFrames 127
Manipulating Data 129
The replace Method 131
Interactive Display 133
Summary 133
Questions 133

10 Visualization Libraries 135
matplotlib 135
Styling Plots 137
Labeled Data 140
Plotting Multiple Sets of Data 141
Object-Oriented Style 143
Seaborn 144
Seaborn Themes 145
Plotly 148
Bokeh 149
Other Visualization Libraries 151
Summary 151
Questions 151

11 Machine Learning Libraries 153

Popular Machine Learning Libraries 153
How Machine Learning Works 154
Transformations 154
Splitting Test and Training Data 155
Training and Testing 156

9780136624356_print.indb 10 30/07/21 4:16 PM

 Contents xi

Learning More About Scikit-learn 157

Summary 157
Questions 157

12 Natural Language Toolkit 159

NLTK Sample Texts 159
Frequency Distributions 161
Text Objects 165
Classifying Text 166
Summary 169
Exercises 169

III: Intermediate Python  171

13 Functional Programming 173
Introduction to Functional Programming 173
Scope and State 174
Depending on Global State 174
Changing State 175
Changing Mutable Data 176
Functional Programming Functions 177
List Comprehensions 179
List Comprehension Basic Syntax 179
Replacing map and filter 180
Multiple Variables 181
Dictionary Comprehensions 181
Generators 182
Generator Expressions 182
Generator Functions 183
Summary 184
Questions 185

14 Object-Oriented Programming 187
Grouping State and Function 187
Classes and Instances 188
Private Methods and Variables 190
Class Variables 190
Special Methods 191
Representation Methods 192

9780136624356_print.indb 11 30/07/21 4:16 PM

 xii Contents

Rich Comparison Methods 192

Math Operator Methods 195
Inheritance 196
Summary 199
Questions 199

15 Other Topics 201
Sorting 201
Lists 201
Reading and Writing Files 204
Context Managers 205
datetime Objects 206
Regular Expressions 207
Character Sets 208
Character Classes 209
Groups 209
Named Groups 210
Find All 210
Find Iterator 211
Substitution 211
Substitution Using Named Groups 211
Compiling Regular Expressions 211
Summary 212
Questions 212

A Answers to End-of-Chapter Questions 215

Index 221

9780136624356_print.indb 12 30/07/21 4:16 PM

 Preface
The Python language has been around for a long time and worn many hats. Its original
implementation was started by Guido van Rossum in 1989 as a tool for system administration
as an alternative to Bash scripts and C programs.1 Since its public release in 1991, it has evolved
for use in a myriad of industries. These include everything from web-development, film,
government, science, and business.2

I was first introduced to Python working in the film industry, where we used it to automate
data management across departments and locations. In the last decade, Python has become a
dominant tool in Data Science.

This dominance evolved due to two developments: the Jupyter notebook, and powerful third-
party libraries. In 2001 Fernando Perez began the IPython project, an interactive Python
environment inspired by Maple and Mathematica notebooks.3 By 2014, the notebook-specific
part of the project was split off as the Jupyter project. These notebooks have excelled for
scientific and statistical work environments. In parallel with this development, third-party
libraries for scientific and statistical computing were developed for Python. With so many
applications, the functionality available to a Python programmer has grown immensely. With
specialized packages for everything from opening web sockets to processing natural language
text, there is more available than a beginning developer needs.

This project was the brainchild of Noah Gift.4 In his work as an educator, he found that
students of Data Science did not have a resource to learn just the parts of Python they needed.
There were many general Python books and books about Data Science, but not resources for
learning just the Python needed to get started in Data Science. That is what we have attempted
to provide here. This book will not teach the Python needed to set up a web page or perform
system administration. It is also not intended to teach you Data Science, but rather the Python
needed to learn Data Science.

I hope you will find this guide a good companion in your quest to grow your Data Science
knowledge.

Example Code
Most of the code shown in examples in this book can be found on GitHub at:
https://github.com/kbehrman/foundational-python-for-data-science.

1 https://docs.python.org/3/faq/general.html#why-was-python-created-in-the-first-place
2 https://www.python.org/success-stories/
3 http://blog.fperez.org/2012/01/ipython-notebook-historical.html
4 https://noahgift.com

A01_Behrman_FM_pi-xvi.indd 13 04/08/21 7:35 PM

 Figure Credits
Figure Credit Attribution
Cover Boris Znaev/Shutterstock
Cover Mark.G/Shutterstock
Figure 1-01 Screenshot of Colab Dialogue © 2021 Google
Figure 1-02 Screenshot of Renaming Notebook © 2021 Google
Figure 1-03 Screenshot of Google Drive © 2021 Google
Figure 1-04 Screenshot of Editing Text Cells © 2021 Google
Figure 1-05 Screenshot of Formatting Text © 2021 Google
Figure 1-06 Screenshot of Lists © 2021 Google
Figure 1-07 Screenshot of Headings © 2021 Google
Figure 1-08 Screenshot of Table of Contents © 2021 Google
Figure 1-09 Screenshot of Hiding Cells © 2021 Google
Figure 1-10 Screenshot of LaTeX Example © 2021 Google
Figure 1-11 Screenshot of A Files © 2021 Google
Figure 1-12 Screenshot of Upload Files © 2021 Google
Figure 1-13 Screenshot of Mount Google Drive © 2021 Google
Figure 1-14 Screenshot of Code Snippets © 2021 Google

A01_Behrman_FM_pi-xvi.indd 14 30/07/21 5:34 PM

 Register Your Book
Register your copy of Foundational Python for Data Science on the InformIT site for convenient
access to updates and/or corrections as they become available. To start the registration
process, go to informit.com/register and log in or create an account. Enter the product ISBN
9780136624356 and click Submit. Look on the Registered Products tab for an Access Bonus
Content link next to this product, and follow that link to access any available bonus materials.
If you would like to be notified of exclusive offers on new editions and updates, please check
the box to receive email from us.

9780136624356_print.indb 15 30/07/21 4:16 PM

 Acknowledgments
The idea for this book first came from Noah Gift. It is he who really identified the need for a
specialized introduction to Python targeted at students of Data Science. Thank you for that,
Noah. And thank you to Colin Erdman who, acting as technical editor, brought an attention to
detail that was much appreciated and needed. I also want to thank the Pearson team, including
Malobika Chakraborty, who guided me through the whole process, Mark Renfrow, who came in
and helped get the project done, and Laura Lewin, who helped get it going.

9780136624356_print.indb 16 30/07/21 4:16 PM

 About the Author
Kennedy Behrman is a veteran software engineer. He began using Python to manage
digital assets in the visual effects industry and has used it extensively since. He has
authored various books and training programs around Python education. He currently
works as a senior data engineer at Envestnet.

9780136624356_print.indb 17 30/07/21 4:16 PM

This page intentionally left blank
 3
Sequences

Errors using inadequate data are much less than those using no data at all.
Charles Babbage

In This Chapter
QQ Shared sequence operations
QQ Lists and tuples
QQ Strings and string methods
QQ Ranges

In Chapter 2, “Fundamentals of Python,” you learned about collections of types. This chapter
introduces the group of built-in types called sequences. A sequence is an ordered, finite collection.
You might think of a sequence as a shelf in a library, where each book on the shelf has a location
and can be accessed easily if you know its place. The books are ordered, with each book (except
those at the ends) having books before and after it. You can add books to the shelf, and you
can remove them, and it is possible for the shelf to be empty. The built-in types that comprise a
sequence are lists, tuples, strings, binary strings, and ranges. This chapter covers the shared
characteristics and specifics of these types.

Shared Operations
The sequences family shares quite a bit of functionality. Specifically, there are ways of using
sequences that are applicable to most of the group members. There are operations that relate to
sequences having a finite length, for accessing the items in a sequence, and for creating a new
sequence based a sequence’s content.

M03_Behrman_C03_p025-036.indd 25 31/07/21 6:21 PM

 26 Chapter 3  Sequences

Testing Membership
You can test whether an item is a member of a sequence by using the in operation. This
operation returns True if the sequence contains an item that evaluates as equal to the item in
question, and it returns False otherwise. The following are examples of using in with differ-
ent sequence types:
'first' in ['first', 'second', 'third']
True

23 in (23,)
True

'b' in 'cat'
False

b'a' in b'ieojjza'
True

You can use the keyword not in conjunction with in to check whether something is absent from
a sequence:
'b' not in 'cat'
True

The two places you are most likely to use in and not in are in an interactive session to explore
data and as part of an if statement (see Chapter 5, “Execution Control”).

Indexing
Because a sequence is an ordered series of items, you can access an item in a sequence by using
its position, or index. Indexes start at zero and go up to one less than the number of items. In an
eight-item sequence, for example, the first item has an index of zero, and the last item an index of
seven.

To access an item by using its index, you use square brackets around the index number. The
following example defines a string and accesses its first and last substrings using their index
numbers:
name = "Ignatius"
name[0]
'I'

name[4]
't'

You can also index counting back from the end of a sequence by using negative index numbers:
name[-1]
's'

name[-2]
'u'

M03_Behrman_C03_p025-036.indd 26 31/07/21 6:21 PM

 Shared Operations 27

Slicing
You can use indexes to create new sequences that represent subsequences of the original. In
square brackets, supply the beginning and ending index numbers of the subsequence separated
by a colon, and a new sequence is returned:
name = "Ignatius"
name[2:5]
'nat'

The subsequence that is returned contains items starting from the first index and up to, but not
including, the ending index. If you leave out the beginning index, the subsequence starts at the
beginning of the parent sequence; if you leave out the end index, the subsequence goes to the end
of the sequence:
name[:5]
'Ignat'

name[4:]
'tius'

You can use negative index numbers to create slices counting from the end of a sequence. This
example shows how to grab the last three letters of a string:
name[-3:]
'ius'

If you want a slice to skip items, you can provide a third argument that indicates what to count
by. So, if you have a list sequence of integers, as shown earlier, you can create a slice just by using
the starting and ending index numbers:
scores = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]
scores[3:15]
[3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]

But you can also indicate the step to take, such as counting by threes:
scores[3:15:3]
[3, 6, 9, 12]

To count backward, you use a negative step:

scores[18:0:-4]
[18, 14, 10, 6, 2]

Interrogation
You can perform shared operations on sequences to glean information about them. Because a
sequence is finite, it has a length, which you can find by using the len function:
name = "Ignatius"
len(name)
8

M03_Behrman_C03_p025-036.indd 27 31/07/21 6:21 PM

 28 Chapter 3  Sequences

You can use the min and max functions to find the minimum and maximum items, respectively:
scores = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]
min(scores)
0

max(name)
'u'

These methods assume that the contents of a sequence can be compared in a way that implies
an ordering. For sequence types that allow for mixed item types, an error occurs if the contents
cannot be compared:
max(['Free', 2, 'b'])
-----------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-15-d8babe38f9d9> in <module>()
----> 1 max(['Free', 2, 'b'])
TypeError: '>' not supported between instances of 'int' and 'str'

You can find out how many times an item appears in a sequence by using the count method:
name.count('a')
1

You can get the index of an item in a sequence by using the index method:
name.index('s')
7

You can use the result of the index method to create a slice up to an item, such as a letter in a
string:
name[:name.index('u')]
'Ignati'

Math Operations
You can perform addition and multiplication with sequences of the same type. When you do, you
conduct these operations on the sequence, not on its contents. So, for example, adding the list [1]
to the list [2] will produce the list [1,2], not [3]. Here is an example of using the plus (+) operator
to create a new string from three separate strings:
"prefix" + "-" + "postfix"
'prefix-postfix'

The multiplication (*) operator works by performing multiple additions on the whole sequence,
not on its contents:
[0,2] * 4
[0, 2, 0, 2, 0, 2, 0, 2]

M03_Behrman_C03_p025-036.indd 28 31/07/21 6:21 PM

 Lists and Tuples 29

This is a useful way of setting up a sequence with default values. For example, say that you want
to track scores for a set number of participants in a list. You can initialize that list so that it has an
initial score for each participant by using multiplication:
num_participants = 10
scores = [0] * num_participants
scores
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Lists and Tuples

Lists and tuples are sequences that can hold objects of any type. Their contents can be of mixed
types, so you can have strings, integers, instances, floats, and anything else in the same list.
The items in lists and tuples are separated by commas. The items in a list are enclosed in square
brackets, and the items in a tuple are enclosed in parentheses. The main difference between lists
and tuples is that lists are mutable, and tuples are immutable. This means that you can change
the contents of a list, but once a tuple is created, it cannot be changed. If you want to change the
contents of a tuple, you need to make a new one based on the content of the current one. Because
of the mutability difference, lists have more functionality than tuples—and they also use more
memory.

Creating Lists and Tuples

You create a list by using the list constructor, list(), or by just using the square bracket syntax.
To create a list with initial values, for example, simply supply the values in brackets:
some_list = [1,2,3]
some_list
[1, 2, 3]

You can create tuples by using the tuple constructor, tuple(), or using parentheses. If you want
to create a tuple with a single item, you must follow that item with a comma, or Python will
interpret the parentheses not as indicating a tuple but as indicating a logical grouping. You can
also create a tuple without parentheses by just putting a comma after an item. Listing 3.1 provides
examples of tuple creation.

Listing 3.1  Creating Tuples

tup = (1,2)
tup
(1,2)

tup = (1,)
tup
(1,)

tup = 1,2,
tup
(1,2)

M03_Behrman_C03_p025-036.indd 29 31/07/21 6:21 PM

 30 Chapter 3  Sequences

Warning
A common but subtle bug occurs when you leave a trailing comma behind an argument to
a function. It turns the argument into a tuple containing the original argument. So the second
argument to the function my_function(1, 2,) will be (2,) and not 2.

You can also use the list or tuple constructors with a sequence as an argument. The following
example uses a string and creates a list of the items the string contains:
name = "Ignatius"
letters = list(name)
letters
['I', 'g', 'n', 'a', 't', 'i', 'u', 's']

Adding and Removing List Items

You can add items to a list and remove items from a list. To conceptualize how it works, think of a
list as a stack of books. The most efficient way to add items to a list is to use the append method,
which adds an item to the end of the list, much as you could easily add a book to the top of a
stack. To add an item to a different position in the list, you can use the insert method, with the
index number where you wish to position the new item as an argument. This is less efficient than
using the append method as the other items in the list may need to move to make room for the
new item; however, this is typically an issue only in very large lists. Listing 3.2 shows examples of
appending and inserting.

Listing 3.2  Appending and Inserting List Items

flavours = ['Chocolate', 'Vanilla']
flavours
['Chocolate', 'Vanilla']

flavours.append('SuperFudgeNutPretzelTwist')
flavours
['Chocolate', 'Vanilla', 'SuperFudgeNutPretzelTwist']

flavours.insert(0,"sourMash")
flavours
['sourMash', 'Chocolate', 'Vanilla', 'SuperFudgeNutPretzelTwist']

To remove an item from a list, you use the pop method. With no argument, this method removes
the last item. By using an optional index argument, you can specify a specific item. In either case,
the item is removed from the list and returned.

The following example pops the last item off the list and then pops off the item at index 0. You can see
that both items are returned when they are popped and that they are then gone from the list:
flavours.pop()
'SuperFudgeNutPretzelTwist'

M03_Behrman_C03_p025-036.indd 30 31/07/21 6:21 PM

 Lists and Tuples 31

flavours.pop(0)
'sourMash'

flavours
['Chocolate', 'Vanilla']

To add the contents of one list to another, you use the extend method:
deserts = ['Cookies', 'Water Melon']
desserts
['Cookies', 'Water Melon']

desserts.extend(flavours)
desserts
['Cookies', 'Water Melon', 'Chocolate', 'Vanilla']

This method modifies the first list so that it now has the contents of the second list appended to
its contents.

Nested List Initialization

There is a tricky bug that bites beginning Python developers. It involves combining list mutabil-
ity with the nature of multiplying sequences. If you want to initialize a list containing four sub-
lists, you might try multiplying a single list in a list like this:
lists = [[]] * 4
lists
[[], [], [], []]

This appears to have worked, until you modify one of the sublists:
lists[-1].append(4)
lists
[[4], [4], [4], [4]]

All of the sublists are modified! This is because the multiplication only initializes one list and
references it four times. The references look independent until you try modifying one. The
solution to this is to use a list comprehension (discussed further in Chapter 13, “Functional
Programming”):
lists = [[] for _ in range(4)]
lists[-1].append(4)
lists
[[], [], [], [4]]

Unpacking
You can assign values to multiple variables from a list or tuple in one line:
a, b, c = (1,3,4)
a
1

M03_Behrman_C03_p025-036.indd 31 31/07/21 6:21 PM

 32 Chapter 3  Sequences

b
3

c
4

Or, if you want to assign multiple values to one variable while assigning single ones to the others,
you can use a * next to the variable that will take multiple values. Then that variable will absorb
all the items not assigned to other variables:
*first, middle, last = ['horse', 'carrot', 'swan', 'burrito', 'fly']
first
['horse', 'carrot', 'swan']

last
'fly'

middle
'burrito'

Sorting Lists
For lists you can use built-in sort and reverse methods that can change the order of the
contents. Much like the sequence min and max functions, these methods work only if the
contents are comparable, as shown in these examples:
name = "Ignatius"
letters = list(name)
letters
['I', 'g', 'n', 'a', 't', 'i', 'u', 's']

letters.sort()
letters
['I', 'a', 'g', 'i', 'n', 's', 't', 'u']

letters.reverse()
letters
['u', 't', 's', 'n', 'i', 'g', 'a', 'I']

Strings
A string is a sequence of characters. In Python, strings are Unicode by default, and any Unicode
character can be part of a string. Strings are represented as characters surrounded by quotation
marks. Single or double quotations both work, and strings made with them are equal:
'Here is a string'
'Here is a string'

"Here is a string" == 'Here is a string'

True

M03_Behrman_C03_p025-036.indd 32 31/07/21 6:21 PM

 Strings 33

If you want to include quotation marks around a word or words within a string, you need to use
one type of quotation marks—single or double—to enclose that word or words and use the other
type of quotation marks to enclose the whole string. The following example shows the word is
enclosed in double quotation marks and the whole string enclosed in single quotation marks:
'Here "is" a string'
'Here "is" a string'

You enclose multiple-line strings in three sets of double quotation marks as shown in the follow-
ing example:
a_very_large_phrase = """
Wikipedia is hosted by the Wikimedia Foundation,
a non-profit organization that also hosts a range of other projects.
"""

With Python strings you can use special characters, each preceded by a backslash. The special
characters include \t for tab, \r for carriage return, and \n for newline. These characters are
interpreted with special meaning during printing. While these characters are generally useful,
they can be inconvenient if you are representing a Windows path:
windows_path = "c:\row\the\boat\now"
print(windows_path)

ow heoat
ow

For such situations, you can use Python’s raw string type, which interprets all characters literally.
You signify the raw string type by prefixing the string with an r:
windows_path = r"c:\row\the\boat\now"
print(windows_path)
c:\row\the\boat\now

As demonstrated in Listing 3.3, there are a number of string helper functions that enable you to
deal with different capitalizations.

Listing 3.3  String Helper Functions

captain = "Patrick Tayluer"
captain
'Patrick Tayluer'

captain.capitalize()
'Patrick tayluer'

captain.lower()
'patrick tayluer'

captain.upper()
'PATRICK TAYLUER'

M03_Behrman_C03_p025-036.indd 33 31/07/21 6:21 PM

 34 Chapter 3  Sequences

captain.swapcase()
'pATRICK tAYLUER'

captain = 'patrick tayluer'

captain.title()
'Patrick Tayluer'

Python 3.6 introduced format strings, or f-strings. You can insert values into f-strings at runtime
by using replacement fields, which are delimited by curly braces. You can insert any expression,
including variables, into the replacement field. An f-string is prefixed with either an F or an f, as
shown in this example:
strings_count = 5
frets_count = 24
f"Noam Pikelny's banjo has {strings_count} strings and {frets_count} frets"
'Noam Pikelny's banjo has 5 strings and 24 frets'

This example shows how to insert a mathematic expression into the replacement field:
a = 12
b = 32
f"{a} times {b} equals {a*b}"
'12 times 32 equals 384'

This example shows how to insert items from a list into the replacement field:
players = ["Tony Trischka", "Bill Evans", "Alan Munde"]
f"Performances will be held by {players[1]}, {players[0]}, and {players[2]}"
'Performances will be held by Bill Evans, Tony Trischka, and Alan Munde'

Ranges
Using range objects is an efficient way to represent a series of numbers, ordered by value. They are
largely used for specifying the number of times a loop should run. Chapter 5 introduces loops.
Range objects can take start (optional), end, and step (optional) arguments. Much as with slicing,
the start is included in the range, and the end is not. Also as with slicing, you can use negative
steps to count down. Ranges calculate numbers as you request them, and so they don’t need
to store more memory for large ranges. Listing 3.4 demonstrates how to create ranges with and
without the optional arguments. This listing makes lists from the ranges so that you can see the
full contents that the range would supply.

Listing 3.4  Creating Ranges

range(10)
range(0, 10)

list(range(1, 10))
[1, 2, 3, 4, 5, 6, 7, 8, 9]

M03_Behrman_C03_p025-036.indd 34 31/07/21 6:21 PM

 Questions 35

list(range(0,10,2))
[0, 2, 4, 6, 8]

list(range(10, 0, -2))
[10, 8, 6, 4, 2]

Summary
This chapter covers the import group of types known as sequences. A sequence is an ordered,
finite collection of items. Lists and tuples can contain mixed types. Lists can be modified after
creation, but tuples cannot. Strings are sequences of text. Range objects are used to describe ranges
of numbers. Lists, strings, and ranges are among the most commonly used types in Python.

Questions
1. How would you test whether a is in the list my_list?

2. How would you find out how many times b appears in a string named my_string?

3. How would you add a to the end of the list my_list?

4. Are the strings 'superior' and "superior" equal?

5. How would you make a range going from 3 to 13?

M03_Behrman_C03_p025-036.indd 35 31/07/21 6:21 PM

 Index

A
anonymous functions, 80
Anscombe, F., 135
apply() method, 132–133
arguments, 30
arrays
broadcasting, 98–99
expanding dimensions, 99–100
changing values in, 91
copies, changing values in, 95
creating, 86–88
one-dimensional, 87
two-dimensional, 88
using reshape method, 88–89
element-by-element operations, 91–92
filtering values, 92–94
indexing, 89–90
matrix operations, 96–97
methods, 95–96
one-dimensional, 87
sequences and, 91
setting type automatically, 97
setting type explicitly, 97–98
slicing, 89–90
two-dimensional, 88
indexing and slicing, 90
views, 94
changing values in, 94
assert statements, 16–17
assignment statements, 17

Z02_Behrman_Index_p221-000.indd 221 31/07/21 2:34 PM

 222 attributes

attributes, 22 list(), 29
axes, 136, 143–144 tuple(), 29
context managers, 205
B continue statements, 19
binomial distribution, 105–107 continuous distributions, 108
Bokeh, 149–150 exponential distribution, 110

Boolean operators, 14, 58–59, 125 normal distribution, 108–110

DataFrames and, 126–127 uniform distribution, 110–111

bracket syntax, 121–122 control statements, 56, 68

break statement, 64 copies, changing values in, 95
break statements, 19 corpus readers, 160
broadcasting, 98–99 loading text, 160–161

expanding dimensions, 99–100 tokenizers, 161

built-in types, 14 corpuses, downloading, 166–167

creating
C arrays, 86–88
one-dimensional, 87
cells, 4–5
two-dimensional, 88
character classes, 209
using reshape method, 88–89
character sets, 208–209
columns, 128
classes, 22, 188–189
DataFrames, 114
datetime.date, 207
from a dictionary, 114–115
inheritance, 196–199
from a file, 116
variables, 190–191
from a list of lists, 115–116
classifier classes, 166
datetime object, 206
code blocks, 56, 63–64
dictionaries, 38
collocations() method, 165
lists, 29–30
columns
tuples, 29–30
creating, 128
updating, 129
D
comparison operators, 57–58, 93–94
DataFrames, 113
compiling regular expressions, 211–212
accessing data, 120–121
compound statements, 55
apply() method, 132–133
if, 59–62
Boolean operators, 126–127
structure, 56
bracket syntax, 121–122
concordance() method, 165
columns
constructors
creating, 128
dict(), 38
updating, 129

Z02_Behrman_Index_p221-000.indd 222 31/07/21 2:34 PM

 fileids() method 223

creating, 114 creating DataFrames from, 114–115

from a dictionary, 114–115 get method, 43–44
from a file, 116 hash() method, 45
from a list of lists, 115–116 key/value pairs
data manipulation, 129–131 adding, 39
describe method, 118 updating, 39
exclude argument, 120 removing items, 39–40
include keyword, 119–120 valid key types, 44–45
percentiles argument, 118–119 dictionary comprehensions, 181
head method, 117 dictionary views, 40–42
interacting with, 117 dict_key, 41–42
interactive display, 133 key_item, 42
manipulating, 127–128, 129 difference() method, 51
masking and filtering, 125–126 discrete distributions, 105
methods, 128 binomial distribution, 105–107
optimized access Poisson distribution, 107–108
by index, 124 disjoint sets, 48
by label, 123–124 dispersion_plot() method, 165–166
replace method, 131–132 docstrings, 68–69
sorting, 204 dot notation, 22
tail method, 118 downloading, corpuses, 166–167
datetime object, 207
creating, 206 E
setting the time zone, 207
elif statement, 62
translating strings to, 207
else statement, 61
decorators, 76–77, 79
equality operators, 56–57, 125
syntax, 79–80
estimators, 156
del() function, 40
exceptions, 18–19
delete statements, 18
exponential distribution, 110
describe method, 118
expressions, 16
exclude argument, 120
generator, 182–183
include keyword, 119–120
extend method, 31
percentiles argument, 118–119
dict() constructor, 38
F
dict_key view, 41–42
figures, 136
dictionaries, 37–39
fileids() method, 160
checking for keys, 43
creating, 38

Z02_Behrman_Index_p221-000.indd 223 31/07/21 2:34 PM

 224 files

files inheriting, 174

creating DataFrames from, 116 outer, 175–176
Google Colab, 9–10 state, 174
opening, 205 functions, 15, 67
reading and writing, 204–205 anonymous, 80
filter() function, 179 control statement, 68
replacing with a list comprehension, 180 datetime.now(), 206
filtering, DataFrames, 125–126 decorators, 76–77, 79
find iterator, 211 syntax, 79–80
findall() method, 165 del(), 40
flattening nested lists, 167 docstring, 68–69
for loops, 63–64 generator, 183–184
FreqDist class helper, 33–34
built-in plot method, 164 lambda, 179
methods, 164 len, 27
frequency distributions, 161–162 max, 28
filtering stopwords, 163–164 min, 28
removing punctuation, 162–163 nested, 77
frozensets, 53 nested wrapping, 78–79
f-strings, 34 open(), 204–205
functional programming, 173, 174–175 as a parameter, 78
changing mutable data, 176–177 parameters, 69–70
dictionary comprehensions, 181 default value, 71–72
filter() function, 179 keyword assignments, 70–71
generator(s), 182 keyword wildcard, 74–75
expressions, 182–183 mutable defaults, 72–73
functions, 183–184 positional wildcard, 74
lambda functions, 179 positional-only, 73
list comprehensions, 179 re.compile(), 211
conditionals and, 181 re.findall(), 210
multiple variables, 181 re.finditer(), 211
replacing map() and filter() with, re.match(), 207–208
180 re.search(), 208
syntax, 179–180 return statements, 75
map() function, 177–178 reversed, 41
operator module, 179 scope, 75–76
reduce() function, 178, 179 sorted(), 202–204
scope, 173–174 wrapping, 77–78
future statements, 20

Z02_Behrman_Index_p221-000.indd 224 31/07/21 2:34 PM

 list() constructor 225

G inheriting scope, 174

installing, NumPy, 86
generator(s), 182
instances, 188
expressions, 182–183
interacting with DataFrame data, 117
functions, 183–184
interrogation, 27–28
get method, 43–44
intersections, 51
global statements, 20
ints, 14
Google Colab, 5–6
numerator attribute, 22
code cells, 9
issuperset() method, 50
Code Snippets, 11
items() method, 40
existing collections and, 11
files, 9–10
headings, 7–8
J-K
LaTeX, 8–9 JSON files, opening and reading, 205
notebooks, managing, 10 Jupyter notebooks, 4–5
shell commands, 11–12 Keras, 153
system aliases, 11–12 key_item view, 42
text cells, 6–8 keys() method, 40
groups, 209–210 key/value pairs, 37
named, 210 adding, 39
updating, 39
H
hash() method, 45 L
head method, 117 labels, DataFrames access and, 123–124
helper functions, 33–34 lambda functions, 80, 179
high-level programming languages, 15 LaTeX, 8–9
len function, 27
I libraries. See also NumPy; SciPy

if statements, 59–62 machine learning, 153–154

immutable objects, 44–45 SciPy, 103

third-party, 85
import statements, 19–20
visualization, matplotlib, 135–136
index method, 28
list comprehensions, 179
indexing, 26
conditionals and, 181
arrays, 89–90
multiple variables, 181
DataFrames and, 124
replacing map() and filter() with, 180
inequality operators, 56–57
syntax, 179–180
inheritance, 196–199
list() constructor, 29

Z02_Behrman_Index_p221-000.indd 225 31/07/21 2:34 PM

 226 lists

lists, 29 matrix operations, 96–97

adding and removing items, 30–31 max function, 28
creating, 29–30 methods, 188–190
creating DataFrames from, 115–116 to_bytes(), 187–188
flattening, 167 apply(), 132–133
nested, 31 arrays and, 95–96
sorting, 32, 201–204 collocations(), 165
unpacking, 31–32 concordance(), 165
loops count, 28
break statement, 64 DataFrames, 128
for, 63–64 describe, 118
while, 62–63 exclude argument, 120
low-level programming languages, 15 include keyword, 119–120
percentiles argument, 118–119
M difference(), 51

machine learning, 153. See also Scikit- disjoint(), 48

learn dispersion_plot(), 165–166
overfitting, 155 extend, 31
splitting test and training data, 155–156 fileids(), 160
supervised versus unsupervised learning, findall(), 165
154 get, 43–44
transformations, 154–155 hash(), 45
magic functions, 12 head, 117
manipulating DataFrames, 127–128, 129 index, 28
map() function, 177–178 inheritance, 196–199
replacing with a list comprehension, 180 intersection(), 51
Markdown, 6 issuperset(), 50
math operations, 21–22 items(), 40
math operator methods, 195–196 keys(), 40
matplotlib, 135–136 math operator, 195–196
colors, 139 min(), 125
creating multiple axes, 143–144 pop, 30
labeled data, 140–141 private, 190
line styles, 138 public, 190
marker types, 137–138 replace, 131–132
object-oriented style, 143 representation, 192
plotting multiple sets of data, 141–143 reverse, 32
styling plots, 137, 139–140 rich comparison, 192–195

Z02_Behrman_Index_p221-000.indd 226 31/07/21 2:34 PM

 object-oriented programming 227

similar(), 165 methods, 164

sort, 32 frequency distributions, 161–162
sort(), 201–202 filtering stopwords, 163–164
special, 191 removing punctuation, 162–163
subset(), 49 sample texts, 159–160
symmetric difference(), 51 Text class, 165
tail, 118 collocations() method, 165
union(), 50 concordance() method, 165
values(), 40 dispersion_plot() method, 165–166
min function, 28 findall() method, 165
min() method, 125 similar() method, 165
MinMaxScaler transformer, 154–155 NoneType, 15
multiple statements, 16 nonlocal statements, 20
mutable objects, 44, 176–177 normal distribution, 108–110
notebooks, 4–5
N Google Colab, 5–6

named groups, 210 Jupyter, 4–5

substitution and, 211 managing, 10

natural language processing, 159 numerics, 14

Natural Language Processing with Python, NumPy. See also arrays; SciPy
169 creating arrays, 86–87
nested functions, 77 installing and importing, 86
nested lists, 31 polynomials, 100–101
nested wrapping functions, 78–79
NLTK (Natural Language Toolkit), 159 O
classifier classes, 166 object-oriented programming, 187
defining features, 168 classes, 188–189
downloading corpuses, 166–167 variables, 190–191
flattening nested lists, 167 inheritance, 196–199
labeling data, 167 instances, 188
training and testing, 168–169 methods, 188–190
corpus readers, 160 math operator, 195–196
loading text, 160–161 representation, 192
tokenizers, 161 rich comparison, 192–195
fileids() method, 160 special, 191
FreqDist class objects, 187–188
built-in plot method, 164 private methods, 190

Z02_Behrman_Index_p221-000.indd 227 31/07/21 2:34 PM

 228 objects

objects, 22, 187–188 pop method, 30

datetime, creating, 206 print statements, 20–21
evaluation, 59 private methods, 190
immutable, 44–45 procedural programming, 174
mutable, 44, 176–177 programming languages, high-level versus
range, 34–35 low-level, 15
one-dimensional arrays, 87 proper subsets, 49
open() function, 204–205 public methods, 190
in operator, 26, 40 Punkt tokenizer, 161
or operator, 59 Python, types, 14–15
operators, 21–22 PyTorch, 154
Boolean, 58–59
Boolean operators, 126–127 Q-R
comparison, 57–58, 93–94 quotation marks, strings and, 33
equality/inequality, 56–57, 125 raise statements, 18–19
in, 40 ranges, 34–35
math, 28–29 raw strings, 33
or, 59 reading files, 204–205
walrus, 60 re.compile() function, 211
overfitting, 155 reduce() function, 178, 179
re.findall() function, 210
P re.finditer() function, 211
packages, zoneinfo, 207 regular expressions, 207–208
Pandas DataFrames. See DataFrames compiling, 211–212
parameters groups, 209–210
default value, 71–72 named groups, 210
functions as, 78 substitution, 211
keyword assignments, 70–71 using named groups, 211
keyword wildcard, 74–75 re.match() function, 207–208
mutable defaults, 72–73 removing, items from dictionaries, 39–40
positional wildcard, 74 replace method, 131–132
positional-only, 73 representation methods, 192
parser, 14 re.search() function, 208
pass statements, 18 return statements, 18, 75
Plotly, 148–149 reverse method, 32
Poisson distribution, 107–108 rich comparison methods, 192–195
polynomials, 100–101 running statements, 4

Z02_Behrman_Index_p221-000.indd 228 31/07/21 2:34 PM

 statements 229

S testing membership, 26
tuples, 29
Scikit-learn, 154
unpacking, 31–32
estimators, 156
sets, 46–48
MinMaxScaler transformer, 154–155
difference between, 51
splitting test and training data, 155–156
disjoint, 48
training a model, 156
proper subsets, 49
training and testing, 156
subsets and, 49
tutorials, 157
supersets and, 50
SciPy, 103
symmetric difference, 51
continuous distributions, 108
union, 50
exponential distribution, 110
updating, 51–52
normal distribution, 108–110
shared operations, 25
uniform distribution, 110–111
similar() method, 165
discrete distributions, 105
slicing, 27
binomial distribution, 105–107
arrays, 89–90
Poisson distribution, 107–108
DataFrames, 122
scipy.misc submodule, 104–105
sort() method, 201–202
scipy.special submodule, 105
sort method, 32
scipy.stats submodule, 105
sorted() function, 202–204
scope, 20, 75–76, 173–174
sorting, lists, 32, 201–204
inheriting, 174
special characters, 33
Seaborn, 144–145
statements, 15–16
plot types, 148
assert, 16–17
themes, 145–147
assignment, 17
sequences, 14, 25
break, 19, 64
arrays and, 91
code blocks, 56, 63–64
frozensets and, 53
continue, 19, 64–65
indexing, 26
delete, 18
interrogation, 27–28
elif, 62
intersections, 51
else, 61
lists, 29
expression, 16
adding and removing items, 30–31
future, 20
nested, 31
global, 20
sorting, 32
if, 59–62
unpacking, 31–32
import, 19–20
math operations, 28–29
multiple, 16
slicing, 27
nonlocal, 20

Z02_Behrman_Index_p221-000.indd 229 31/07/21 2:34 PM

 230 statements

pass, 18 time series data, 206

print, 20–21 time zone, setting for datetime object, 207
raise, 18–19 to_bytes() method, 187–188
return, 18, 75 tokenizers, 161
running, 4 transformations, 154–155
yield, 18 tuple() constructor, 29
stopwords, 163–164 tuples, 29
strings, 14, 32–33 creating, 29–30
f-, 34 unpacking, 31–32
helper functions, 33–34 two-dimensional arrays, 88
quotation marks and, 33 indexing and slicing, 90
raw, 33 types, 14–15. See also sequences
special characters, 33
translating to datetime object, 207 U
submodules
uniform distribution, 110–111
scipy.misc, 104–105
union() method, 50
scipy.special, 105
updating
scipy.stats, 105
columns, -129
subset() method, 49
sets, 51–52
substitution, 211
supersets, 50 V
symmetric difference() method, 51
values() method, 40
syntax
variables, 190–191
bracket, 121–122
views, 94
decorators, 79–80
changing values in, 94
list comprehensions, 179–180
visualization libraries, 151
Bokeh, 149–150
T
matplotlib, 135–136
tail method, 118
colors, 139
TensorFlow, 153 creating multiple axes, 143–144
text cells, 6–8 labeled data, 140–141
Text class, 165 line styles, 138
collocations() method, 165 marker types, 137–138
concordance() method, 165 object-oriented style, 143
dispersion_plot() method, 165–166 plotting multiple sets of data,
findall() method, 165 141–143
similar() method, 165 styling plots, 137, 139–140
third-party libraries, 85 Plotly, 148–149

Z02_Behrman_Index_p221-000.indd 230 31/07/21 2:34 PM

 zoneinfo package 231

Seaborn, 144–145 wrapping functions, 77–78

plot types, 148 writing file data, 204–205
themes, 145–147
X-Y-Z
W yield statements, 18
walrus operator, 60 zoneinfo package, 207
while loops, 62–63

Z02_Behrman_Index_p221-238.indd 231 31/07/21 8:07 PM