The value of optimization
by Louis P. Lukac and B. Wade Brorsen
Optimization based on historical data is used widely in the futures industry to determine the most
profitable parameter sets for a trading system. The underlying premise is that if some parameter set
worked well in the past, it should work well in the future.
Simply picking the best parameter set from historical data and stating the accompanying profit level
(called in-sample profits) is of no real value because you cannot see into the future to know what the best
parameter set will be. Actual or blind simulation (out-of-sample) results are always much worse than
in-sample results, which has led many to question the value of optimization.
There seems to be little agreement on how much data should be used in optimizing parameters, how
frequently to re-optimize or whether optimization is even of any value at all. Even the large public futures
fund managers who rely heavily on technical analysis cannot agree. A survey of public futures fund
managers by B. Wade Brorsen and Scott H. Irwin in The Review of Futures Markets in 1987 showed that
the data used by advisors to select parameters ranged from all the available historical data to as little as
two years worth. Their frequency of re-optimization ranged from twice a year to once every five years.
Several advisors did not optimize at all, they simply used the same parameter across all commodities.
Large amounts of time and money are devoted to finding the optimal parameters for a system. With more
and more money being managed technically, and more software available for optimization, the need to
test the validity or limits of optimization grows greater. An invalid framework could lead to invalid
results and a false sense of security, not to mention the potential loss of money.
We tested various re-optimization strategies to determine if optimization is of any value for predicting
correct sets of parameters and to determine if one re-optimization strategy is better than another.
Specifically, we statistically compared 10 different re-optimizations to each other and to a random
selection strategy using two different trading systems.