Learning Advanced Python by Studying Open Source Projects 1st Edition Li All Chapters Instant Download
Learning Advanced Python by Studying Open Source Projects 1st Edition Li All Chapters Instant Download
com
https://textbookfull.com/product/learning-
advanced-python-by-studying-open-source-
projects-1st-edition-li/
https://textbookfull.com/product/firm-sponsored-developers-in-open-
source-software-projects-a-social-capital-perspective-dirk-homscheid/
textbookfull.com
https://textbookfull.com/product/deep-learning-for-remote-sensing-
images-with-open-source-software-1st-edition-remi-cresson/
textbookfull.com
https://textbookfull.com/product/how-open-source-ate-software-
understand-the-open-source-movement-and-so-much-more-gordon-haff/
textbookfull.com
https://textbookfull.com/product/learning-d3-js-data-visualization-
aendrew-rininsland/
textbookfull.com
Strategy-Making in the EU: From Foreign and Security
Policy to External Action Pol Morillas
https://textbookfull.com/product/strategy-making-in-the-eu-from-
foreign-and-security-policy-to-external-action-pol-morillas/
textbookfull.com
https://textbookfull.com/product/design-ethnography-epistemology-and-
methodology-francis-muller/
textbookfull.com
https://textbookfull.com/product/interactive-modeling-and-simulation-
in-business-system-design-1st-edition-ella-roubtsova-auth/
textbookfull.com
https://textbookfull.com/product/web-security-for-developers-real-
threats-practical-defense-1st-edition-malcolm-mcdonald/
textbookfull.com
https://textbookfull.com/product/mind-over-matter-a-comprehensive-
guide-to-discovering-your-psychic-powers-auerbach/
textbookfull.com
Curtiss P 40 Long nosed Tomahawks 1st Edition Carl
Molesworth
https://textbookfull.com/product/curtiss-p-40-long-nosed-
tomahawks-1st-edition-carl-molesworth/
textbookfull.com
Learning Advanced Python by
Studying Open Source Projects
This book is one of its own kind. It is not an encyclopedia or a hands-on tutorial that traps
readers in the tutorial hell. It is a distillation of just one common Python user’s learn-
ing experience. The experience is packaged with exceptional teaching techniques, careful
dependence unraveling and, most importantly, passion.
Learning Advanced Python by Studying Open Source Projects helps readers overcome the
difficulty in their day-to-day tasks and seek insights from solutions in famous open source
projects. Different from a technical manual, this book mixes the technical knowledge, real-
world applications and more theoretical content, providing readers with a practical and
engaging approach to learning Python.
Throughout this book, readers will learn how to write Python code that is efficient,
readable and maintainable, covering key topics such as data structures, algorithms, object-
oriented programming and more. The author’s passion for Python shines through in
this book, making it an enjoyable and inspiring read for both beginners and experienced
programmers.
Rongpeng Li (Ron) is a YouTube educator and animator. He has a consistent passion for
education. He has published two books on statistics and scientific simulation.
Chapman & Hall/CRC
The Python Series
Rongpeng Li
Cover image: Rongpeng Li
© 2024 Rongpeng Li
Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been
acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or
utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including
photocopying, microfilming, and recording, or in any information storage or retrieval system, without written
permission from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com or contact the
Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. For works that are
not available on CCC please contact [email protected]
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for
identification and explanation without intent to infringe.
DOI: 10.1201/9781003316909
Typeset in Minion
by codeMantra
To Yan, the brightest light in my life.
To my family, whose decisions made my life today possible.
To Holly and Prosper, the endless source of joy in the room.
Contents
Preface, x
Acknowledgments, xi
IntroductIon 1
PURPOSE AND SCOPE OF THIS BOOK 1
OVERVIEW OF THE APPROACH TAKEN 3
NOTE 4
C
A GENTLE INTRODUCTION TO PYTHON’S DATA MODEL 5
CUSTOMIZED COMPARISON 7
A MANAGED ITERATION BEHAVIOR 12
ATTRIBUTES, FUNCTION OR DICTIONARY? 16
SUMMARY 20
NOTES 20
C
INTRODUCTION 21
DESCRIPTORS AND ATTRIBUTE LOOKUP ORDER 21
Descriptor Demystified 21
Lazy Evaluation in Matplotlib 25
METACLASS AND ITS USAGE IN ELASTICSEARCH DSL 30
Understanding Metaclass Using Meta-Recipe 30
Use Metaclass to Model Documents in Elasticsearch DSL 35
SUMMARY 40
APPENDIX 40
NOTES 41
viii ◾ Contents
C
CONCURRENCY FROM A TOP-DOWN PERSPECTIVE 42
Operating System and Concurrency 45
Introducing Global Interpreter Lock (GIL) 47
MULTIPROCESSING FOR CPU BOUND TASKS 48
Parallel Pandas Apply in pandarallel 52
MULTITHREADING FOR I/O BOUND TASKS 58
SUMMARY 63
NOTES 63
C
A SHIFT OF PARADIGM 64
EVENT-DRIVEN SIMULATION 65
ASYNC AS A PATTERN 71
SUMMARY 75
APPENDIX 75
NOTES 76
C
INTRODUCTION 77
THE DECORATOR FOR RETRYING A FUNCTION 77
CONTEXT MANAGER IN A NUTSHELL 81
DIVE INTO THE AIOSQLITE EXAMPLE 84
Connection as an Executor and a Scheduler 85
Connection and Cursor as Async Context Managers 88
SUMMARY 93
NOTES 93
C
UNDERSTAND YOUR BUSINESS 94
A Quick Overview of the Business 95
MODEL THE BUSINESS ENTITIES WITH OOP 97
Design the Core Entities 97
Establish the Relationship between Classes 100
The Benefits of Universal interface 104
Sometimes No OOP is the Best Design 106
Contents ◾ ix
SUMMARY 108
NOTES 108
C
INTRODUCTION 109
FIXTURE AND PARAMETERIZATION 109
Parameterization 111
Resources and Fixture 113
MONKEY PATCH 115
Modify the Built-in Print 115
More Powerful Monkey Patching 117
PROPERTY-BASED TEST 118
SUMMARY 122
NOTES 122
INDEX, 123
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
Preface
x
Acknowledgments
I would like to thank the creators, maintainers and all contributors to the open source
projects I used as examples in this book.
It is unimaginable if those amazing resources are not freely accessible to me and to
everyone around the globe. The open source movement has astronomically impacted not
only the software engineering but also the advance of technology universally.
Thank you, from the bottom of my heart.
xi
Introduction
1. The Python data model and Python classes are fundamentally important as they are
the basics to learn everything else. They are basic but they are not obviously easy to
master. Many Python users, or most Python users didn’t graduate with a computer
science master’s degree. They started using Python by copying and pasting so long as
the code worked. At some point in their learning or career path, the lack of founda-
tion will bite. I want to solidify your foundations.
2. The other four topics are deemed as a gap between so-so engineers and more pro-
fessional engineers. I picked these four because they are the ones that I find easy to
break the cyclic dependencies of the advanced topics. For example, it is impossible
to talk about concepts like fixture without talking about decorators first. It is hard
DOI: 10.1201/9781003316909-1 1
2 ◾ Learning Advanced Python by Studying Open Source Projects
to imagine context manager without talking about the idea of coroutine first: well
technically you can. I try to chop the open source projects that utilize multiple such
concepts into digestible pieces. The example of aiosqlite is probably the most typical
example. Please do read Chapter 5 and let me know whether I did an acceptable job.
The following topics are not considered important enough for most readers.
Well, those topics are very important if you are working on fine-tuning Python perfor-
mances: not because of bad code but because of Python itself. For example, some high-
performance computing work requires such skills, but most Python users don’t ever write
a line of C in their whole career. Similarly, most people don’t publish Python packages as
well.
There are great resources on these two topics. I would recommend the official doc for the
former and the book Publishing Python Packages1 by Dane Hillard for the latter. Honestly
speaking, the fact that the second topic is enough for an independent book also makes it
impractical to be included in this book. The first topic is also big enough for an indepen-
dent book.
The following topics are indeed important, but they are not really Python specific.
The first draft of this book included documentation as an independent chapter. Popular
Python open source libraries often have fabulous documentation. However, the more I
wrote about it, the more I feel that they are not advanced Python, they are skills related to
advanced Programming. Everyone who wrote code needs to know them. It is not suitable to
include them in a book about Python.
Similar ideas apply to project management. Since most open source projects are devel-
oped for the public, you can read the comments, the interactions and sometimes spicy
exchanges between developers and users. I personally find this topic super interesting while
preparing the materials. However, they are, again, not Python specific. I have to drop it.
Let’s discuss the purpose of this book. This book’s purpose is to help moderately good
Python users to become professional Python users.
What is a Python user? What is moderately good? What makes a Python user
professional?
A Python user is a superset of a Python developer. When I was a PhD student, I used
Python a lot. I was not a Python developer. Namely, I wrote Python as a tool to do some-
thing else, rather than for the sake of writing Python. On the contrary, a Python devel-
oper’s job is to write high-quality Python applications.
Introduction ◾ 3
However, I find Python users’ work often requires very high technical skills. For exam-
ple, a Jupyter Notebook is far from enough for a researcher. The researchers sometimes
hack their way through the development and write terrible code. This book aims at the
aspirational and ambitious Python users so they can use Python in the most efficient way.
Of course, developers are welcome.
So, what is exactly being moderately good that qualifies you to read this book?
If you
2. are familiar with basic object-oriented programming concepts like class, instance
and inheritance.
3. understand basic operating system concepts like process, thread and memory.
4. are familiar with tools/platforms like Git, GitHub and VS Code.
I hope this gives a good idea of what topics this book covers, and what kind of prerequisites
I am expecting the readers to have.
NOTE
Chapter 1
cars = list()
cars.append(‘bmw’)
cars.extend([‘audi’, ‘toyota’])
assert cars[0] == ‘bmw’
last_car = cars.pop()
assert last_car == ‘toyota’
assert len(cars) == 2
For a dictionary, you can perform a different set of operations as shown in code snippet
1.2. You can get a member of the dictionary. Note that both list and dict support the
len() method.
fruit_prices = dict()
fruit_prices[‘apple’] = 0.5
fruit_prices[‘orange’] = 0.25
assert len(fruit_prices) == 2
assert fruit_prices.get(‘pear’) == None
What makes lists and dictionaries behave in different ways? What if we want to modify
their behaviors and create a hybrid data structure to suit our needs? This will be the main
topic of this chapter: the data model2 for built-in Python data types.
DOI: 10.1201/9781003316909-2 5
6 ◾ Learning Advanced Python by Studying Open Source Projects
In Python, everything is an instance of the object class. You can think of object as the
ancestor of all things in Python, including the built-in types like int, str, list,
dict and all user-defined classes. Let’s try some examples with code snippet 1.3.
isinstance(“California”,object)
# True
isinstance(int,object)
# True
isinstance(list(),object)
# True
class Car:
pass
isinstance(Car(),object)
# True
isinstance(Car, object)
# True
A Car instance is an object instance. Also, the Car class itself is an object instance. The
question is that if all stuffs are instances of the object class, then what makes a list dif-
ferent from a dict?
Let’s check the supported methods of a list and a dict.
The dir() function returns a list of attributes and methods supported by an object. We
sort the list in alphabetical order to make it easier to compare the results. Code snippet 1.4
shows the result for a list instance.
sorted(dir(list()))
CODE 1.4 The attributes and methods supported by a Python list object.
Let’s check what methods are supported by a list instance but not supported by a dict
instance and vice versa. To do that, we need to build two sets and take a difference as
shown in code snippet 1.5.
set(dir(list())) - set(dir(dict()))
The Data Model of Python ◾ 7
set(dir(dict())) - set(dir(list()))
# {‘__ior__’, ‘__or__’, ‘__ror__’, ‘fromkeys’, ‘get’,
‘items’, ‘keys’, ‘popitem’, ‘setdefault’, ‘update’, ‘values’}
CODE 1.5 Compare the differences of supported methods for a list and a set.
You may notice that both list and set support the __gt__ methods. It means that you
can compare two lists or two dictionaries. It seems that they follow the same protocol here
that defines what they can do: they can be compared against their own kinds.
However, a list supports the __add__ method while a dict instance does not. This
hints that they follow different protocols: lists can be added directly, while dictionaries
cannot.
In the following sections, you will learn how to create your own data structures that fol-
low different protocols and control exactly how they behave.
CUSTOMIZED COMPARISON
Let’s begin with a scenario. You are hired by a major car dealer to create an application that
will help them to keep track of their customers’ information. One key feature of the applica-
tion is to compare the in-house creditability of customers. For example, if two customers
bid for the same car, the application needs to tell your boss which customer is more credible.
We need a Customer class. Code snippet 1.6 creates it.
class Customer:
def __init__(self, first_name: str, last_name: str, credit_
score: int, credit_limit: int, in_debt: bool, monthly_income: int):
self.first_name = first_name
self.last_name = last_name
self.credit_score = credit_score
self.credit_limit = credit_limit
self.in_debt = in_debt
self.monthly_income = monthly_income
def __repr__(self):
debt_status = “not in debt” if self.in_debt == False else
“in debt”
return f”Customer {self.first_name} {self.last_name},
{debt_status},” \
f” with a credit score of {self.credit_score},”\
f” credit limit of {self.credit_limit}” \
f” and a monthly income of {self.monthly_income}.”
Visit https://textbookfull.com
now to explore a rich
collection of eBooks, textbook
and enjoy exciting offers!
8 ◾ Learning Advanced Python by Studying Open Source Projects
The __repr__ method is used to print the object. If you don’t define it, Python will print
the object’s memory address like < __main__.Customer object at0x1083c4ee0>.
We can create a few customers and play around with them as shown in code snippet 1.7.
print(melissa_miller)
# Customer Melissa Miller, not in debt, with a credit score of
700, credit limit of 250000 and a monthly income of 9000.
To compare the financial credibility of the customers, the car dealer has created a set of
rules based on their historical experiences. The rules are organized in order such that the
rules are checked in order. These rules are:
However, if the one who gets paid higher is in debt while another who gets paid lower is
not, the difference of monthly income must be larger than 4000. Otherwise, the one who
is not in debt is more credible.
Ideally, we want to compare the john _ smith and richard _ dawkins objects in
code snippet 1.8. However, we can’t do it because Customers don’t support such operations.
We have two solutions. One is to write a helper function that compares two customers.
Every time we need to compare customers, we can pass the function to something like the
key parameter of the sort() method. Another solution is to enable native support for
Customer comparison.
Which solution is better? The logic of comparison needs to go somewhere in the code. The
question is where. The canonical way is in the Customer class. There are two reasons. First,
by enabling syntax like john _ smith < richard _ dawkins, we aligned our syntax with
the Pythonic way of doing comparison. It is much easier for your coworkers to integrate the
code into their projects. Second, the built-in comparison logic is more robust and error-free
than a standalone function. When you ship your code to someone else, you have a better
control over the code unless the code users deliberately overwrite the comparison logic in
the Customer class, which is much less likely than them writing another standalone com-
parison function.
Let’s pause our concerns for a while to learn from the pros and see how this is done in
the SymPy library. SymPy is an open source Python library for symbolic computation.
It can be used to perform algebraic computations and symbolic differentiation, etc. Code
snippet 1.9 shows a differentiation example.
Back to SymPy, instead of studying its implementation, let’s investigate a bug to pierce
into its core. The developers found a bug that alternating two objects’ order in comparison
10 ◾ Learning Advanced Python by Studying Open Source Projects
gives different results in early 2021. As stated in issue 20796,3 the two comparisons in code
snippet 1.10 should both return False. Based on the types of them, one represents numeri-
cal value and another logical. Objects with different data types should always be different.
S(0.0) == S.false
# True
S.false == S(0.0)
# False
S here stands for Singleton. S.false means that there is only one such thing as a math-
ematical false in the whole mathematical universe. There can only be one false and there
can only be one 0.
Now, let’s get to the implementation of the __eq__ method of the S class. Pull request
208014 fixed the issue but I already copied the pre-PR implementation to snippet 1.11.
In snippet 1.11, the self represents a float number.
CODE 1.11 Equality comparison of a float number with a bool, before pull request
20801.
The __eq__ method takes two arguments, the first one is the object itself and the second
one is the object to be compared with. When you write comparison like A == B, The __eq__
method of the object on the left side of the comparison is called: in the example, A.
In the line denoted with comment 1, we first check the Boolean representation of the
object itself. If it is False, it means the object is a 0, probably with arbitrary precision. In that
case, we check the other object. If it is also equivalent to 0, then they are equal.
In the line denoted with comment 2, we check whether the other object is a Boolean
object or not. If it is, then we always return False. A number is never the same with a
Boolean, mathematically.
A quick note if you are not catching up. How did we know that the float number’s __eq__
method is wrong? Since S.false == S(0.0) gives the correct answer, which is False, we
then know the S.false. __eq __(S(0.0)) returns the right answer. This is how we know
that the issue is in the implementation of __eq__ for the float number Singleton object.
The devil lies in the order of the two lines I commented. When a float number is equiva-
lent to 0, not self in the first if statement is evaluated to be True, therefore if the other object
is a False object, a True value is returned. These two if statements need to be switched. If
the types are different, then they are never equivalent. This is exactly what the pull request
20801 fixes.
Notice that the practice is very similar to our proposed implementation of __lt__
or __gt__ methods for the Customer class. The order of criteria matters.
Go back on our original question, code snippet 1.12 is implements the __lt__ method
for the Customer class. It is probably the most elegantly written code. In production,
make sure you write comprehensive tests, too.
if self.credit_limit != other.credit_limit:
return self.credit_limit < other.credit_limit
income_diff = abs(self.income - other.income)
CODE 1.12 Implementation of the less than logic for Customer class.
This dunder method allows users to sort customers natively as shown in code snippet 1.13.
So far everything looks great. Let’s do something even fancier in the next section.
»Blandois?»
»Minkä sattuman?»
»Ette ole vielä sanonut, sir, kuinka tämä kaikki liikuttaa minua ja
minkä palveluksen aiotte pyytää minulta. Tahtoisitteko puhua siitä
nyt?»
»Mr Gowan ei ole nähnyt Blandoisia sen koommin kuin tämä lähti
Englantiin eikä muutenkaan tiedä hänestä mitään. Hän oli vain
satunnainen ulkomailla saatu tuttava.»
Arthur pyysi häntä antamaan Sen. Miss Wade meni lipastolle, avasi
sen ja otti eräästä sisälaatikosta muutaman kokoonkäärityn
paperiarkin. Osoittamatta minkäänlaista sovinnollisuutta, tuskin
kääntyen hänen puoleensa, pikemmin puhuen vain kuvastimellensa
puolustaakseen omaa itsepintaisuuttansa, sanoi hän antaessaan
paperit:
Hänellä ei ollut mitään syytä epäillä heitä; tytön käytös oli niin
luonnollista, että se jo sinänsä olisi ollut vakuuttava, jos hänellä olisi
ollut joitakin ennakkoepäilyksiä. Hän vastasi: »Minun täytyy hakea
tietoja muualta.»
»Kutka?»
»Koska tahdoin nähdä sen. Koska tunsin, että olisi hauskaa taas
nähdä se.»
Itsekiduttajan tarina
Vielä toinenkin tapa hänellä oli. Toisinaan kun hän näki näillä
keinoin varmasti syösseensä minut synkän epätoivon mietiskelyihin,
kiinnitti hän lasten huomion siihen, osoittaakseen millainen ero
hänen ja minun välillä oli. »Hiljaa! Miss Wade-parka ei voi hyvin.
Älkää melutko kultaseni, hänen päätänsä kivistää. Tulkaa
lohduttamaan häntä. Tulkaa kysymään joko hän voi paremmin,
pyytäkää häntä rupeamaan levolle. Toivon ettei teillä ole mitään
huolia, ma'am. Älkää olko pahoillanne, ma'am, älkää olko
surullinen!»
Tämä kävi sietämättömäksi. Hänen armonsa, emäntäni, tuli
eräänä päivänä sisään kun olin yksin ja äärimmäisen tietoinen siitä,
etten minä kestäisi tätä. Sanoin hänelle että minun täytyi lähteä
pois, koska en sietänyt tuota naista, Dawesia.
Tiesin edeltäpäin jo, että hän sanoisi niin, olin valmistunut siihen;
vastasin vain, ettei minun sopinut sanoa emäntääni vastaan, minun
täytyi lähteä.
»Sitä en sanonut.»
En voi välttää sanomasta, että hän ihaili minua, mutta jos voisin,
niin välttäisin. Turhamaisuudella ei ole mitään osaa tässä, sillä
hänen ihailunsa kiusasi minua. Hän ei vaivautunut salaamaan sitä,
niin että minusta tuntui, eläessäni noiden rikkaiden ihmisten
loukossa, kuin hän olisi ostanut minut ulkomuotoni tähden ja
asettaisi ostamansa tavaran näytteille puolustaakseen kauppaansa.
Huomasin, että he mielessään arvioivat minua ja olivat uteliaita
tietämään minkä arvoinen oikeastaan olin. Päätin, etteivät he saisi
sitä tietää. Olin liikkumaton ja vaiti heidän seurassaan ja olisin
sallinut kenen hyvänsä tappaa minut mieluummin kuin olisin
millään tavoin vaivautunut saavuttaakseni heidän hyväksymisensä.
Tämän varjolla alkoi hän nyt mitata takaisin minulle. Hän saattoi
tuntimäärin pysyä poissa kenen muun kanssa hyvänsä. Istuin
kerran puolen iltaa yksin ja huomaamattomana, jolla välin hän
keskusteli nuoren serkkunsa, oppilaani, kanssa. Näin koko ajan
ihmisten silmistä, että heidän mielestänsä nämä kaksi paljoa
paremmin sopivat yhteen kuin hän ja minä. Arvaten ihmisten
ajatukset istuin ja katselin nuorta paria, kunnes huomasin, että se
saattoi minut naurettavaksi, ja raivosin itseäni vastaan, koska olin
rakastanut häntä.
Daniel Doyce oli yksi näitä valittuja. Sillä kertaa ei voitu sanoa,
viipyisikö hän poissa kuukausia vai vuosia Hänen
matkavalmistuksensa ja heidän yhteisen liikkeensä kaikkien
yksityisseikkojen ja tulosten tunnontarkka järjestely vaati paljon
työtä lyhyessä ajassa, ja Clennam ahersi yöt päivät. Ensimmäisenä
lomapäivänään livahti hän Kanaalin poikki ja yhtä nopeasti taas
samalla laivalla takaisin sanomaan hyvästi Doycelle.