Here, in part 1 of two, these Contributing Writers explain the steps necessary to evaluate trading system behavior with the use of statistics.
by Jeffrey Owen Katz, Ph.D., and Donna L. McCormick
Most systems I have seen are simply optimized by being tested
with different parameters, examined for profitability and
sometimes assessed for a few other characteristics. On occa-sion, I have encountered systems tested on perhaps one or
two out-of-sampleŻ periods to determine whether performance would hold up. However, few who write about system development or developers themselves have attempted to estimate probabilities or compute statistical analyses on trading systems to assess likely future system performance. For the sake of simplicity, I have often been guilty of omitting such analyses from my own work.
However, I often get comments regarding issues that statistics directly bear upon, such as sample size, replicability, curve-fitting, generalization and determining whether a system will trade profitably or fall apart when finally put to the test. I have also had discussions with system developers who are reluctant to apply any optimization strategy because of a fear of curve-fitting rooted in a lack of knowledge about the statistical issues involved.