PDF Experimentation for Engineers From A B testing to Bayesian optimization 1st Edition David Sweet download
PDF Experimentation for Engineers From A B testing to Bayesian optimization 1st Edition David Sweet download
com
https://ebookmeta.com/product/experimentation-for-engineers-
from-a-b-testing-to-bayesian-optimization-1st-edition-david-
sweet-2/
OR CLICK HERE
DOWLOAD NOW
https://ebookmeta.com/product/experimentation-for-engineers-from-a-b-
testing-to-bayesian-optimization-1st-edition-david-sweet/
ebookmeta.com
https://ebookmeta.com/product/experimentation-for-engineers-from-a-b-
testing-to-bayesian-optimization-meap-v06-david-sweet/
ebookmeta.com
https://ebookmeta.com/product/experimentation-for-engineers-from-a-b-
testing-to-bayesian-optimization-meap-version-6-david-sweet/
ebookmeta.com
https://ebookmeta.com/product/advanced-machining-and-finishing-1st-
edition-kapil-gupta-alokesh-pramanik/
ebookmeta.com
Economics, Ecology, and Policy for the Bioeconomy: A
Holistic Approach First Edition Sanja Tišma
https://ebookmeta.com/product/economics-ecology-and-policy-for-the-
bioeconomy-a-holistic-approach-first-edition-sanja-tisma/
ebookmeta.com
https://ebookmeta.com/product/dk-eyewitness-top-10-florence-and-
tuscany-12th-edition-dk-eyewitness/
ebookmeta.com
https://ebookmeta.com/product/myths-and-traditions-of-central-
european-university-culture-1st-edition-fasora/
ebookmeta.com
https://ebookmeta.com/product/phage-display-2nd-edition-michael-hust/
ebookmeta.com
Berlitz Pocket Guide Bologna Travel Guide eBook 2nd
Edition Berlitz Publishing
https://ebookmeta.com/product/berlitz-pocket-guide-bologna-travel-
guide-ebook-2nd-edition-berlitz-publishing/
ebookmeta.com
inside front cover
David Sweet
To comment go to liveBook
Manning
Shelter Island
For online information and ordering of these and other Manning books,
please visit www.manning.com. The publisher offers discounts on these
books when ordered in quantity.
ISBN: 9781617298158
dedication
To B and Iz.
contents
front matter
preface
acknowledgments
about this book
about the author
about the cover illustration
1 Optimizing systems by experiment
1.1 Examples of engineering workflows
Machine learning engineer’s workflow
Quantitative trader’s workflow
Software engineer’s workflow
1.2 Measuring by experiment
Experimental methods
Practical problems and pitfalls
1.3 Why are experiments necessary?
Domain knowledge
Offline model quality
Simulation
2 A/B testing: Evaluating a modification to your system
2.1 Take an ad hoc measurement
Simulate the trading system
Compare execution costs
2.2 Take a precise measurement
Mitigate measurement variation with replication
2.3 Run an A/B test
Analyze your measurements
Design the A/B test
Measure and analyze
Recap of A/B test stages
3 Multi-armed bandits: Maximizing business metrics while
experimenting
3.1 Epsilon-greedy: Account for the impact of evaluation on
business metrics
A/B testing as a baseline
The epsilon-greedy algorithm
Deciding when to stop
3.2 Evaluating multiple system changes simultaneously
3.3 Thompson sampling: A more efficient MAB algorithm
Estimate the probability that an arm is the best
Randomized probability matching
The complete algorithm
4 Response surface methodology: Optimizing continuous
parameters
4.1 Optimize a single continuous parameter
Design: Choose parameter values to measure
Take the measurements
Analyze I: Interpolate between measurements
Analyze II: Optimize the business metric
Validate the optimal parameter value
4.2 Optimizing two or more continuous parameters
Design the two-parameter experiment
Measure, analyze, and validate the 2D experiment
5 Contextual bandits: Making targeted decisions
5.1 Model a business metric offline to make decisions online
Model the business-metric outcome of a decision
Add the decision-making component 128
Run and evaluate the greedy recommender
5.2 Explore actions with epsilon-greedy
Missing counterfactuals degrade predictions
Explore with epsilon-greedy to collect counterfactuals
5.3 Explore parameters with Thompson sampling
Create an ensemble of prediction models
Randomized probability matching
5.4 Validate the contextual bandit
6 Bayesian optimization: Automating experimental
optimization
6.1 Optimizing a single compiler parameter, a visual
explanation
Simulate the compiler
Run the initial experiment
Analyze: Model the response surface
Design: Select the parameter value to measure next
Design: Balance exploration with exploitation
6.2 Model the response surface with Gaussian process
regression
Estimate the expected CPU time
Estimate uncertainty with GPR
6.3 Optimize over an acquisition function
Minimize the acquisition function
6.4 Optimize all seven compiler parameters
Random search
A complete Bayesian optimization
7 Managing business metrics
7.1 Focus on the business
Don’t evaluate a model
Evaluate the product
7.2 Define business metrics
Be specific to your business
Update business metrics periodically
Business metric timescales
7.3 Trade off multiple business metrics
Reduce negative side effects
Evaluate with multiple metrics
8 Practical considerations
8.1 Violations of statistical assumptions
Violation of the iid assumption
Nonstationarity
8.2 Don’t stop early
8.3 Control family-wise error
Cherry-picking increases the false-positive rate
Control false positives with the Bonferroni correction
8.4 Be aware of common biases
Confounder bias
Small-sample bias
Optimism bias
Experimenter bias
8.5 Replicate to validate results
Validate complex experiments
Monitor changes with a reverse A/B test
Measure quarterly changes with holdouts
8.6 Wrapping up
appendix A Linear regression and the normal equations
appendix B One factor at a time
appendix C Gaussian process regression
index
front matter
preface
When I first entered the industry, I had the training of a theoretician but
was presented with the tasks of an engineer. As a theoretician, I had
worked with models using pen-and-paper or simulation. Where the model
had a parameter, I—the theoretician—would try to understand how the
model would behave with different values of it. But now I—the engineer—
had to commit to a single value: the one to use in a production system.
How could I know what value to choose?
Over the years, the methods applied by the teams I have been on, and by
engineers in trading and technology generally, have become ever more
precise and efficient. They have been used to optimize the execution of
stock trades, market making, web search, online advertising, social media,
online news, low-latency infrastructure, and more. As a result, trade
execution has become cheaper and more fairly priced. Users regularly
claim that web search and social media recommendations are so good
that they worry their phones might be eavesdropping on them (they’re
not).
While this research was being done, the methods were being applied in
industry: first in agriculture (Fisher’s methods), then in chemical and
process industries (response surface methods). Later (from the 1950s to
the 1980s) experimentation merged with statistical process control to give
us the quality movements in manufacturing, exemplified by Toyota’s Total
Quality Management, and later, popularized by Six Sigma.
In 1975, J. Mockus wrote “On the Bayes methods for seeking the
extremal point,” the foundation for Bayesian optimization (chapter 6),
which takes an alternative approach to modeling a response surface and
combines it with ideas from sequential analysis. This method was
developed over the decades since by many researchers, including D.
Jones et al., who wrote “Efficient global optimization of expensive black-
box functions,” which, in 1998, applied some modern ideas to the method,
making it look much more like the approach presented in this book.
I hope this book saves you some time by putting all the bits and pieces
I’ve collected in one place and stitching them together into a single,
coherent unit.
acknowledgments
I am grateful to so many people for their hard work, for their support, and
for their faith that this book could be brought into existence.
Several people sat with me for interviews. I appreciate the time and
support of P.B., B.S., M.M., and Yan Wu (of Bond).