Optimization 101 –

Bruce Robinson
[email protected]

October 8, 2005
Presentation Overview
 Mechanical systems and modeling
 Fitness – what is it and what are the possibilities
 Back-fitting as a loaded gun – what are the issues and
pitfalls
 Near-optimal solutions and a high level overview of how
its done
 Walk-forward methodology
 --- BREAK ---
 Mechanics of the IO optimizer
 Three functions
 Detailed example of back-fitted optimization
 Sensitivity analysis and how it predicts robustness - with
example
 Detailed example of walk-forward optimization, and use of
sensitivity analysis
 How to use it in your trading – some guidelines
Presentation Goal -
System vs. discretionary
trading
“Dad – are all stops signs like mandatory, or are
some just like recommended?”
 There are successful mechanical traders and
successful discretionary traders
 Both back-test, although differently, and over
different time periods
 Discretionary traders use the “computer
between their ears”
 Discretionary traders tend to factor in more
complex patterns, but don’t take all signals
 Mechanical traders must take all signals to
achieve expected results – but most don’t take
Discretionary traders
 Discretionary traders process –
 Patterns they articulate
 Patterns that they do not, OR can not
articulate
 Patterns may involve price action
alone, or other information such as
markets, sectors, age of move, etc.
 Fall into two categories
 Setups
 Projections of future effects
Mechanical system
traders
 Rationale
 Mechanical system can be back-tested
 In theory, it removes emotions from trading
 Modern T/A programs have made
development much easier
 There are a number of traps waiting for a
developer of trading systems
 Back-fitting is one of the worst !
 Most system traders will exercise
discretion thinking that they can improve
on results
Why do we model, backtest, and
optimize -
 Model
 To define an “edge” in timing, selection, money
management
 A model approximates history
 Are we modeling an underlying crowd behavior, or
a statistical relationship with no cause and effect ?
 Storks nesting in post-WWII Copenhagen -> birth
rate
 Accuracy is required, but not exactness
 Back-test
 To verify that edge, know the odd’s
 To test across different markets
 To provide confidence going forward
 Optimize
 To maximize or minimize those factors that are
most important
 CAR, MDD, Sharpe, UPI, etc.
What is fitness?
 We have to decide on what we are going
to optimize on
 It can be a single measure, or a weighted
combination of measures
 Some typical candidates
 CAR – compounded annual return
 CAR / MDD
 UPI - Ulcer Performance Index – penalize net.
volatility only
 Sharpe ratio – neg. volatility will follow pos.
volatility
 Combination measures
 Fitness = CAR / Trades ^ 0.5 / MDD ^ 2 / UI ^ 0.5
 Risk adjusted measures will often need
Trading System Mining
Pitfalls-
 “A fossicker is one who
searches
(unsystematically) for any
object by which to make
gain - to dig - to hunt - to
bustle about - or to
rummage... “
 Back-fitting to history
usually yields negative
surprises
Back-testing
???

2001 2003 TODAY

BACKTEST AND
OPTIMIZE

 Big Question – What percentage of the

back-test return, MDD, etc. should you
expect going forward ?
The back-fitting sensitivity
problem -
Other issues -
 Back-testing
 Different market action vs. recent
market action
 Test over bull, bear, sideways markets
 Number of signal samples - statistical
significance
 Issue for intermediate term systems
 Survivorship bias
 Bad performing stocks fold, or merge
 Good funds’ record is bought
 Bad funds record is buried

 Selection bias
AI optimization -
 AI optimization offers many new
capabilities
 To date we have fitted as much data as
possible and “hoped”
 Mechanization is one of the most
important capabilities
 Eliminates biases - selection, event knowledge,
etc.
 Less sensitive solutions can be found for a
point in time
 We can “time travel” to points in the past
and look at walk forward performance
 A “spliced” equity curve can be assembled
to look at what would have happened if
Keywords –
 Near optimal
 Robust
 Out of sample, walk-forward
 Many of these concepts were
pioneered by Rober Pardo
Near optimality -
Robustness -
Rolling walk forward
testing

1998 2000 2002 TODAY

 Green segments represent equity that

COULD have been achieved !
Anchored walk forward
testing

1998 2000 2002 TODAY

 Green segments represent equity

that COULD have been achieved !
Walk forward issues
 Rolling walk forward –
 How far back to back-test and optimize
over
 Attractive to look over recent history

 Markets change when you least expect

 Anchored walk forward –

 Encompasses different markets
 Compromises over those different
environments
Walk forward efficiency
 Several approaches to walk forward efficiency
including Pardo’s
 One approach is compound annual return (CAR)
of walk forward periods / CAR of solution
optimized over entire period
 The following alternative, though, performs
measurements in the same period as a kind of
“apples to apples” approach
 Efficiency (WFE) -
 Walk forward period / value for walk forward period
when optimized through end of walk forward period
 Example –
 LB = Optimize 1/2/1998 – 1/2/2002
 WF = Walk forward 1/2/2002 – 1/2/2003
 TOT = Optimize 1/2/1998 – 1/2/2003
 IDEAL = Segment of TOT for 1/2/2002 – 1/2/2003
Why Pardo -
 Design, Testing, and
Optimization of Trading Systems
- Robert Pardo
 Originally written in 1992

 His book is a good set of

guidelines
 Other authors such as Perry
Kaufman have also written about
this topic
Pardo Cliff Notes -
 First – Pardo is a guide
 Difference between necessary and sufficient
 Not A -> Not B
 A -> B
 Most of Pardo’s guidelines are sufficient
 In other words, if you have them, you should
have a robust system, BUT, is you don’t, you
may still have a robust system.
 “Overfitting occurs when excessive attention is
paid to creating a curve or testing a trading
model with past data and not enough attention is
paid to judging the predictive value of the curve
or trading model.”
 Much of Pardo’s work deals with over-fitting, how
to detect it, and how to avoid it
 Pardo’s operational
recommendations -
 Know the point at which to abandon a
system
 Models have a “shelf life”
 Re-optimization can be too frequent OR too
infrequently
 Systems must have a statistically
significant sample size
 Intermediate systems – as much data as is
available
 Swing systems – multiple markets vs. recent
market debate
 Monitor the equity curve
 Trade the equity curve – trend line, MA,
MDD, etc.
What is the problem to
solve?
 Today we optimize to today, fit, and
hope !
 Many/most system models are too
complex for standard optimization
techniques
 We do not know how a model would
have performed in the past as it was
optimized and traded forward.
 “In theory there is no difference
between theory and practice. In
System off-roading -
A Solution -
 What if we had ways to find a near
optimal, reasonably robust solution (if one
exists) to any system, at any point in
time?
 And, we could GO BACK IN TIME see how
such a system would do OOS in a walk
forward test.
 We could piece together those OOS tests
over time to get a result that could have
been achieved.
 We would no longer be optimizing and
hoping.

BREAK
Artifical Intelligence
Optimizations -
 The idea is to use AI based search techniques
that model techniques in nature to find a near-
optimal, robust solution.
 Solutions are expected to be the best that can be
detected by intelligent searches in a reasonable
timeframe.
 Timeframe for full search is impractical ! (NP
Complete)
 RUTTR – 14 variables (9 without 4 levels and 1 lookback)
 51 x 51 x 41 x 21 x 21 x 51 x 101 x 15 x 21 x 21 x 18 x
100 x 100 x 5 =
1,442,204,034,769,858,500,000 combinations
1,483,131,035,700,000 combinations without
levels and lookback
2,351,489 years at 1,728,000 per day
AB and IO -
 AB lends itself to this approach because –
 Has very fast backtester
 Allows user functions
 Support COM automation by controlling script
 IO – product written by Fred Tonetti as AB add-on
 IO uses particle swarm and differential evolution
 Manipulates the Optimize() statements in the
source and “feeds in” potential solution points
 Exports performance results and iterates
 Animates the ongoing equity charts and produces
statistical data and charts in browser
 Also produces numerous CSV files of results
 Will “dim the lights”
Setup for background IO
run -
AB tool used to run IO -
Goals and penalties -
 Goal assigns a minimum desired value to a
fitness field name (AB optimization result
column)
 Values below the goal result in an
adjustment factor that is applied to fitness.
 Two types of scaling – multiplier and
exponential
 Example of multiplier
 Fitness is UPI
 UPI of solution point = 2
 CAR = 15, CAR goal = 20
 Fitness adjustment is .75 * 2 = 1.5
Sensitivity
 Definition - change in fitness with
respect to “distance” from solution
 Rationale – since change in inputs
can’t be accurately predicted,
approach is to measure change in
output based on change in
parameters
 An average percentage is
determined based on samples
 The solution point fitness is adjusted
by a sensitivity goal
The sensitivity adjusted
fitness -
IO directives for traditional
optimization
//IO: StatusWindow: N
//IO: SaveCancelled: Y Disable the
directive
// Remove in final "production" run
//xIO: FitnessTime: 300 5 minute
//xIO: SenFinalTests: 1000 limit
1000 samples for
sensitivity
//IO: SenOptGoalPct: 85 Penalize result < 85%
//IO: SenOptTests: 10 sensitivity
10 samples to determine
//IO: Fitness: UPI sensitiviy
//IO: Goal: CAR: 15 CAR goal to prevent high
UPI, low CAR
//xIO: BegISDate: 01/01/1995
//xIO: EndISDate: 01/01/2001
//xIO: EndOSDate: 9/15/2005 Walk-forward directives
//xIO: WFAuto: Anchored: Every: Yeardisabled
Timing signal logic -
BuyPrice = SellPrice = ShortPrice = CoverPrice = Close;
SetTradeDelays( 0, 0, 0, 0 );
emalen = Optimize( "emalen", 4, 2, 100, 1 );
longfilt = Optimize( "longfilt", 0.014, 0.001,
0.050, 0.001 );
shortfilt = Optimize( "shortfilt", 0.016, 0.001,
0.050, 0.001 );
fund = Close;
fundema = EMA( fund, emalen );
Buy = Cross( fund, fundema * ( 1 + longfilt ) );
Sell = Cross( fundema * ( 1 - shortfilt ),
fund );
No needed in
BuyState = Flip( Buy, Sell ); this case – but
SellState = NOT BuyState; always safe
Buy = ExRem( BuyState, SellState );
Sell Turns off
= ExRem( SellState, BuyState ); short
trading
Short = 0; //Sell;
Cover = 0; //Buy;
SetOption( "ReverseSignalForcesExit", True );
Graphical signal logic -

Note that area

between green and
red requires state
logic
State and boolean signal
logic -
 Area between red and green can’t be
described by Boolean logic
 Fund is in that area AFTER BUY and AFTER SELL
 Impulse signal occurs when fund crossed
up thru green or down thru red EMA’s
 Design pattern
 These impulses are switched into state by the
Flip() function
 Flip() also resolves combinations of signals
 AB’s trading engine want impulse form which is
obtained with the ExRem() function
IO Startup Screen -
Next – IO Optimizer in detail
Traditional optimization -
Portfolio indicator -

Days in
drawdown

Equity &
signals
Regressi
on line
Reference
Current ticker
drawdown
Parameter sensitivity
distribution -
Report of traditional
backtest -
1 year walk forward
optimization -
Walk forward vs. traditional
from 2000 -
Walk forward vs. traditional
from 1996 -
Comparison of walk-forward to
traditional -
 Primary difference is that the 2003
run-up was not captured by the walk-
forward
 Walk-forward actually did better in
2000, and 1997-1998
 The traditional back-fit must make
compromises to handle all periods
 But the walk-forward is more
sensitive to changes in the early
years
 Note differences in number of trades
Additional IO
functionality -
 Perform sensitivity testing on a given
system
 Invoke IO from the command line and run
batch directive files
 Process user defined metrics
 Turn off sensitivity testing of individual
variables
 Optimization of groups of variables
 Walk forward on every signal
 Many more options that are documented in
the IO.DOC file
How to use in your
trading -
 Model your trading ideas
 Don’t incorporate hidden fixed assumptions/filters
 FilteredEMA example is a stop and reverse system
 A system that went flat might allow for better filters and signals
with less retracement
 Utilize Optimize() statements for all system parameters –
not just indicator parameters
 For example - PositionSize, WorstRankHeld, ApplyStops, etc.
 Perform IS (in sample) over several types of markets
 Perform a walk-forward analysis
 Now, there is no reason not to – we no longer have to fit and
hope
 Use a problematic period – for example, 2001-2002
 Use common sense –
 Don’t re-optimize too frequently
 Don’t have short look-back periods
 Expect to iterate your model
 Look at it as improving your odds for success
Conclusions -
 The example used was a teaching example, not a
practical system
 IO allows optimization of previously intractable
systems
 But, it’s value is in allowing walk forward
optimization of any system
 This allows you to see how a system would have
performed in the past
 It allows you to see how it may perform in the
future and to assess its stability
 You may find results you didn’t want to find
 Some high performance systems may not walk forward
well
 But, others will
 “ The future ain't what it used to be “ – Yogi Berra
 Remember, it is like going …
IO -
Questions and Answers
Preview of Optimization
201 -
 Knowledge of AFL programming will be needed
 Construction and optimization of timing signals
 Level vs. state
 Using IO to turn signal components on and off
 Explore which components add value
 Optimization of ticker selection and weights
 Assess portfolio mix
 IO optimization of rotational systems
 Special case of rotational system = walk-forward system
 Selecting look-back and walk-forward periods
 Fine-tuning the directives and options
 Minimizing run time
 Design patterns
 Why system design is partly a science and partly
an art