Instant Download Expert Twisted: Event-Driven and Asynchronous Programming With Python 1st Edition Mark Williams PDF All Chapter
Instant Download Expert Twisted: Event-Driven and Asynchronous Programming With Python 1st Edition Mark Williams PDF All Chapter
com
https://textbookfull.com/product/expert-twisted-
event-driven-and-asynchronous-programming-with-
python-1st-edition-mark-williams/
https://textbookfull.com/product/expert-twisted-event-driven-and-
asynchronous-programming-with-python-1st-edition-mark-williams-2/
textbookfull.com
https://textbookfull.com/product/using-asyncio-in-python-
understanding-python-s-asynchronous-programming-features-caleb-
hattingh/
textbookfull.com
https://textbookfull.com/product/rare-metal-technology-2019-gisele-
azimi/
textbookfull.com
https://textbookfull.com/product/the-complete-guidebook-to-yosemite-
national-park-steven-p-medley/
textbookfull.com
https://textbookfull.com/product/essentials-of-negotiation-6e-test-
bank-6th-edition-roy-j-lewicki/
textbookfull.com
https://textbookfull.com/product/sex-hormones-exercise-and-women-
scientific-and-clinical-aspects-1st-edition-anthony-c-hackney-eds/
textbookfull.com
https://textbookfull.com/product/drug-abuse-and-antisocial-behavior-a-
biosocial-life-course-approach-1st-edition-christopher-p-salas-wright/
textbookfull.com
Advances in Computers Volume 108 1st Edition Atif Memon
https://textbookfull.com/product/advances-in-computers-volume-108-1st-
edition-atif-memon/
textbookfull.com
Expert Twisted
Event-Driven and Asynchronous
Programming with Python
—
Mark Williams
Cory Benfield
Brian Warner
Moshe Zadka
Dustin Mitchell
Kevin Samuel
Pierre Tardy
Expert Twisted
Event-Driven and Asynchronous
Programming with Python
Mark Williams
Cory Benfield
Brian Warner
Moshe Zadka
Dustin Mitchell
Kevin Samuel
Pierre Tardy
Expert Twisted
Mark Williams Cory Benfield
Pasadena, CA, USA London, UK
Pierre Tardy
Toulouse, France
Introduction������������������������������������������������������������������������������������������������������������xix
Part 1: Foundations���������������������������������������������������������������������������������������� 1
Chapter 1: An Introduction to Event-Driven Programming with Twisted����������������� 3
A Note About Python Versions������������������������������������������������������������������������������������������������������� 4
What Is Event-Driven Programming?�������������������������������������������������������������������������������������������� 4
Multiple Events����������������������������������������������������������������������������������������������������������������������������� 5
Application(tkinter.Tk()).mainloop()����������������������������������������������������������������������������������������������� 6
Multiplexing and Demultiplexing��������������������������������������������������������������������������������������������������� 7
The select Multiplexer������������������������������������������������������������������������������������������������������������������� 9
Its History, Its Siblings, and Its Purpose���������������������������������������������������������������������������������� 9
select and Sockets������������������������������������������������������������������������������������������������������������������ 9
The How and Why of Socket Events�������������������������������������������������������������������������������������� 11
Handling Events��������������������������������������������������������������������������������������������������������������������� 12
An Event Loop with select����������������������������������������������������������������������������������������������������� 13
Event-Driven Clients and Servers������������������������������������������������������������������������������������������ 15
Non-blocking I/O������������������������������������������������������������������������������������������������������������������������� 18
Knowing When to Stop���������������������������������������������������������������������������������������������������������� 18
Tracking State������������������������������������������������������������������������������������������������������������������������ 19
State Makes Programs Complex������������������������������������������������������������������������������������������� 23
Managing Complexity with Transports and Protocols����������������������������������������������������������������� 23
Reactors: Working with Transports���������������������������������������������������������������������������������������� 24
v
Table of Contents
vi
Table of Contents
Coroutines in Python������������������������������������������������������������������������������������������������������������������� 88
Coroutines with yield from���������������������������������������������������������������������������������������������������� 88
Coroutines async and await�������������������������������������������������������������������������������������������������� 90
Awaiting Deferreds��������������������������������������������������������������������������������������������������������������������� 95
Coroutines to Deferreds with ensureDeferred���������������������������������������������������������������������������� 97
Multiplexing Deferreds���������������������������������������������������������������������������������������������������������������� 99
Testing Deferreds���������������������������������������������������������������������������������������������������������������������� 102
Summary���������������������������������������������������������������������������������������������������������������������������������� 106
vii
Table of Contents
Client����������������������������������������������������������������������������������������������������������������������������������� 159
Registry������������������������������������������������������������������������������������������������������������������������������� 160
Build������������������������������������������������������������������������������������������������������������������������������������ 160
Multi-stage Build����������������������������������������������������������������������������������������������������������������� 161
Python on Docker���������������������������������������������������������������������������������������������������������������������� 163
Deployment Options������������������������������������������������������������������������������������������������������������ 163
Full env�������������������������������������������������������������������������������������������������������������������������������� 163
Virtualenv���������������������������������������������������������������������������������������������������������������������������� 169
Pex��������������������������������������������������������������������������������������������������������������������������������������� 170
Build Options����������������������������������������������������������������������������������������������������������������������� 172
One Big Bag������������������������������������������������������������������������������������������������������������������������� 172
Copying Wheels Between Stages���������������������������������������������������������������������������������������� 172
Copying Environment Between Stages������������������������������������������������������������������������������� 173
Copying the Pex Executable Between Stages��������������������������������������������������������������������� 173
Automation with Dockerpy�������������������������������������������������������������������������������������������������� 173
Twisted on Docker�������������������������������������������������������������������������������������������������������������������� 174
ENTRYPOINT and PID 1�������������������������������������������������������������������������������������������������������� 174
Custom Plugins�������������������������������������������������������������������������������������������������������������������� 174
NColony������������������������������������������������������������������������������������������������������������������������������� 175
Summary���������������������������������������������������������������������������������������������������������������������������������� 178
viii
Table of Contents
ix
Table of Contents
x
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Table of Contents
Index��������������������������������������������������������������������������������������������������������������������� 373
xii
About the Authors
Mark Williams works on Twisted. At eBay and PayPal, he worked on high-performance
Python web services (over a billion requests a day!), application and information
security, and porting enterprise, Java-only libraries to Python.
Cory Benfield is an open source Python developer heavily involved in the Python HTTP
community. He's a Requests core contributor, a urllib3 core contributor, and the lead
maintainer of the Hyper Project, a collection of HTTP and HTTP/2 tools for Python. For
his sins, he also helps out with the Python Cryptographic Authority on PyOpenSSL.
Brian Warner is a security engineer and software developer, having worked at Mozilla
on Firefox Sync, the Add-On SDK, and Persona. He is co-founder of the Tahoe-LAFS
distributed secure filesystem, and develops secure storage and communication tools.
Moshe Zadka has been part of the open source community since 1995, made his first
core Python contributions in 1998, and is a founding member of the Twisted open
source project. He also loves to teach Twisted and Python, having given tutorials at
several conferences as well as regularly blogging.
Dustin Mitchell has contributed to Buildbot and is a member of the TaskCluster team
at Mozilla, having also worked on the Release Engineering, Release Operations, and
Infrastructure teams.
Kevin Samuel has been a Dev and trainer since Python 2.4 and has been putting
his skills to work in East Europe, North America, Asia, and West Africa. He has been
working closely with the Crossbar.io team and is an active member of the French Python
community.
Pierre Tardy is a continuous integration specialist with Renault Software Labs, and he is
currently the lead committer for Buildbot.
xiii
About the Technical Reviewers
Julian Berman is a New York-based software developer and
open source contributor. He is the author of the jsonschema
Python library, an occasional contributor to the Twisted
ecosystem, and an active member of the Python community.
Shawn Shojaie lives in the clement chaparral of California's Bay Area, where he works
as a back-end software engineer. He has worked at Intel, NetApp, and now SimpleLegal,
where he happily builds web-based applications for legal services. He spends weekdays
writing Django and tuning PostgreSQL, and his weekends contributing to open source
projects like django-pylint, occasionally editing technical essays. Find out more at him at
shawnshojaie.com.
Tom Most is a software engineer in the telecommunications industry. He is a Twisted
committer with 10 years of experience of applying Twisted to web services, client
libraries, and command-line applications. He is the maintainer of Afkak, the Twisted
Kafka client. He can be found online at freecog.net and reached at [email protected].
xv
Acknowledgments
Thanks to my wife, Jennifer Zadka, without whose support I could not have done it.
Thanks to my parents, Yaacov and Pnina Zadka, who taught me how to learn.
Thanks to Glyph Lefkowitz, for teaching me things about Python, about programming,
and about being a good person.
—Moshe Zadka
Thanks to Mahmoud Hashemi and David Karapetyan for their feedback. Thanks to
Annie for putting up with me while I wrote
—Mark Williams
xvii
Introduction
Twisted has recently celebrated its sweet sixteen birthday. It has been around for a
while; and in that time, it grew to be a powerful library. In that time, some interesting
applications have been built on top of it. In that time, many of us learned a lot about how
to use Twisted well, how to think about networking code, and how to architect event-
based programs.
After going through the introductory materials that we have on the Twisted site,
a common thing to hear is “What now? How can I learn more about Twisted?” The
usual way we answered that question is with a question: “What do you want to do with
Twisted?” This book shows how to do interesting things with Twisted.
Each of the contributors to this book has done slightly different things with Twisted
and learned different lessons. We are excited to present all of these lessons, with the
goals of making them common knowledge in the community.
Enjoy!
xix
PART 1
Foundations
CHAPTER 1
An Introduction to
Event-Driven
Programming with Twisted
Twisted is a powerful, well-tested, and mature concurrent networking library and
framework. As we’ll see in this book, many projects and individuals have used it to great
effect for more than a decade.
At the same time, Twisted is large, complicated, and old. Its lexicon teems with
strange names, like “reactor,” “protocol,” “endpoint,” and “Deferred.” These describe a
philosophy and architecture that have baffled both newcomers and old hands with years
of Python experience.
Two fundamental programming paradigms inform Twisted’s pantheon of APIs:
event-driven programming and asynchronous programming. The rise of JavaScript
and the introduction of asyncio into the Python standard library have brought both
further into the mainstream, but neither paradigm dominates Python programming
so completely that merely knowing the language makes them familiar. They remain
specialized topics reserved for intermediate or advanced programmers.
This chapter and the next introduce the motivations behind event-driven and
asynchronous programming, and then show how Twisted employs these paradigms.
They lay the foundation for later chapters that explore real-world Twisted programs.
We’ll begin by exploring the nature of event-driven programming outside of the
context of Twisted. Once we have a sense of what defines event-driven programming,
we’ll see how Twisted provides software abstractions that help developers write clear
and effective event-driven programs. We’ll also stop along the way to learn about
some of the unique parts of those abstractions, like interfaces, and explore how they’re
documented on Twisted’s website.
3
© Mark Williams, Cory Benfield, Brian Warner, Moshe Zadka, Dustin Mitchell, Kevin Samuel, Pierre Tardy 2019
M. Williams et al., Expert Twisted, https://doi.org/10.1007/978-1-4842-3742-7_1
Chapter 1 An Introduction to Event-Driven Programming with Twisted
By the end of this chapter you’ll know Twisted terminology: protocols, transports,
reactors, consumers, and producers. These concepts form the foundation of Twisted’s
approach to event-driven programming, and knowing them is essential to writing useful
software with Twisted.
import sys
line = sys.stdin.readline().strip()
if line == "h":
print("Hello")
else:
print("World")
The availability of a line of input over standard input is an event. Our program
pauses on sys.stdin.readline(), which asks the operating system to allow the user to
input a complete line. Until one is received, our program can make no progress. When
the operating system receives input, and Python’s internals determine it’s a line, sys.
stdin.readline() resumes our program by returning that data to it. This resumption
is the event that drives our program forward. Even this simple program, then, can be
understood as an event-driven one.
4
Chapter 1 An Introduction to Event-Driven Programming with Twisted
Multiple Events
A program that receives a single event and then exits doesn’t benefit from an event-
driven approach. Programs in which more than one thing can happen at a time,
however, are more naturally organized around events. A graphical user interface implies
just such a program: at any moment, a user might click a button, select an item from a
menu, scroll through a text widget, and so on.
Here’s a version of our previous program with a Tkinter GUI:
class Application(tkinter.Frame):
def __init__ (self, root):
super(Application,self). __init__ (root)
self.pack()
self.helloButton = tkinter.Button(self,
text="Say Hello",
command=self.sayHello)
self.worldButton = tkinter.Button(self,
text="Say World",
command=self.sayWorld)
self.output = scrolledtext.ScrolledText(master=self)
self.helloButton.pack(side="top")
self.worldButton.pack(side="top")
self.output.pack(side="top")
def outputLine(self, text):
self.output.insert(tkinter.INSERT, text+ '\n')
def sayHello(self):
self.outputLine("Hello")
def sayWorld(self):
self.outputLine("World")
5
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Chapter 1 An Introduction to Event-Driven Programming with Twisted
Application(tkinter.Tk()).mainloop()
This version of our program presents the user with two buttons, either of which can
generate an independent click event. This differs from our previous program, where only
sys.stdin.readline could generate the single “line ready” event.
We cope with the possible occurrence of either button’s event by associating event
handlers with each one. Tkinter buttons accept a callable command to invoke when they
are clicked. When the button labeled “Say Hello” generates a click event, that event
drives our program to call Application.sayHello as shown in Figure 1-1. This, in turn,
outputs a line consisting of Hello to a scrollable text widget. The same process applies to
the button labeled “Say Hello” and Application.sayWorld.
Figure 1-1. Our Tkinter GUI application after a series of clicks of “Say Hello” and
“Say World”
6
Chapter 1 An Introduction to Event-Driven Programming with Twisted
3. An event loop waits for events and invokes the event handler
associated with each. Not all event-driven programs have an event
loop; our sys.stdin.readline example did not because it only
responds to a single event. However, most resemble our tkinter
example in that they process many events before finally exiting.
These kinds of programs use an event loop.
def mainloop(self):
while self.running:
ready = [button for button in self.buttons if button.hasEvent()]
if ready:
self.dispatchButtonEventHandlers(ready)
mainloop continually polls each button for a new event, dispatching event handlers
only for those that have an event ready. When no events are ready, the program makes
no progress because no action has been taken that requires a response. An event-driven
program must suspend its execution during these periods of inactivity.
7
Chapter 1 An Introduction to Event-Driven Programming with Twisted
The while loop in our mainloop example suspends its program until one of the
buttons has been clicked and sayHello or sayWorld should run. Unless the user is
supernaturally fast with a mouse, this loop spends most of its time checking buttons that
haven’t been clicked. This is known as a busy wait because the program is actively busy
waiting.
A busy wait like this pauses a program’s overall execution until one of its event
sources reports an event, and so it suffices as a mechanism to pause an event loop.
The inner list comprehension that powers our implementation’s busy wait asks
a critical question: Has anything happened? The answer comes from the ready
variable, which contains all buttons that have been clicked in a single place. The
truthiness of ready decides the answer to the event loop’s question: when ready is
empty and thus falsey, no buttons have been clicked and so nothing has happened.
When it’s truthy, however, at least one has been clicked, and so something has
happened.
The list comprehension that constructs ready coalesces many separate inputs
into one. This is known as multiplexing, while the inverse process of separating
different inputs out from a single coalesced input is known as demultiplexing.
The list comprehension multiplexes our buttons into ready while the
dispatchButtonEventHandlers method demultiplexes them out by invoking each
event’s handler.
We can now refine our understanding of event loops by precisely describing how
they wait for events:
8
Chapter 1 An Introduction to Event-Driven Programming with Twisted
While Twisted can integrate with graphical user interfaces, and in fact has special
support for tkinter, it is at its heart a networking engine. Sockets, not buttons, are the
fundamental object in networking, and operating systems expose efficient primitives for
multiplexing socket events. Twisted’s event loop uses these primitives to wait for events.
To understand Twisted’s approach to event-driven programming, we must understand
the interaction between these sockets and these multiplexing networking primitives.
9
Chapter 1 An Introduction to Event-Driven Programming with Twisted
A full explanation of the socket API is beyond the scope of this book. Indeed, we
expect that the parts we discuss will lead you to prefer Twisted! The preceding code,
however, contains more fundamental concepts than irrelevant details:
client and server are two ends of the same TCP connection. An established TCP
connection has no concept of “client” and “server”; our client socket has the same
privileges to read, write, or close the connection as our server:
With that the rest, the which the Castle kept, xxiv
About him flockt, and hard at him did lay;
But he them all from him full lightly swept,
As doth a Steare, in heat of sommers day,[407]
With his long taile the bryzes brush away.
Thence passing forth, into the hall he came,
Where of the Lady selfe in sad dismay
He was ymett, who with vncomely shame
Gan him salute, and fowle vpbrayd with faulty blame.
The Dwarfe his way did hast, and went all night; xxx
But Calidore did with her there abyde
The comming of that so much threatned Knight,
Where that discourteous Dame with scornfull pryde,
And fowle entreaty him indignifyde,
That yron heart it hardly could sustaine:
Yet he, that could his wrath full wisely guyde,
Did well endure her womanish disdaine,
And did him selfe from fraile impatience refraine.
The wretched man, that all this while did dwell xliii
In dread of death, his heasts did gladly heare,
And promist to performe his precept well,
And whatsoeuer else he would requere.
So suffring him to rise, he made him sweare
By his owne sword, and by the crosse thereon,
To take Briana for his louing fere,
Withouten dowre or composition;
But to release his former foule condition.
FOOTNOTES:
[401] vii 6 replide) 1596
[402] ix i leaue 1596
[403] x 8 withall, 1596 withall; 1609
[404] xiii 9 pay 1596
[405] xvi 6 vnable 1609
[406] xxiii 6 carkarsse 1596
[407] xxiv 4 day. 1596
[408] xxv 9 requight 1596
[409] xxviii 6 Ere he] Ere thou 1596
[410] xxxi 1 light, 1596
[411] xxxvi 9 breathe 1609
[412] xxxviii 1 hie, 1596
[413] xl 9 earne 1609
[414] xlii 4 sith 1609
Cant. II.
FOOTNOTES:
[415] iii 2 deed and word] act and deed 1596
[416] 3 eyes] eares edd.
[417] 4 eares] eyes edd.
[418] v 7 lincolne 1596
[419] vi 7 launce 1609
[420] vii 2 what 1596
[421] viii 1 him 1596
[422] ix 7 enranging 1609
[423] xiv 3 Sayd] Staid 1609
[424] neither 1596, 1609
[425] xv 8 since] sith 1609
[426] xvi 5 iolliment, 1596
[427] xix 6 hot 1609
[428] xxii 6 Punching 1609
[429] xxvii 1 may 1596
[430] xxx 6 wrong 1596
[431] xxxiii 2 sildome 1609
[432] 7 since] sith 1609
[433] xxxiv 4 faire 1596
[434] xxxvii 6 drad 1609
[435] xxxix 2 implements] ornaments 1609
Cant. III.