V.15:7 (311-317): Evaluating Trading Systems with Statistics by J. Katz, Ph.D.

Here, in part 1 of two, these Contributing Writers explain the steps necessary to evaluate trading system behavior with the use of statistics.

by Jeffrey Owen Katz, Ph.D., and Donna L. McCormick

Most systems I have seen are simply optimized by being tested with different parameters, examined for profitability and sometimes assessed for a few other characteristics. On occa-sion, I have encountered systems tested on perhaps one or two out-of-sampleŻ periods to determine whether performance would hold up. However, few who write about system development or developers themselves have attempted to estimate probabilities or compute statistical analyses on trading systems to assess likely future system performance. For the sake of simplicity, I have often been guilty of omitting such analyses from my own work.

However, I often get comments regarding issues that statistics directly bear upon, such as sample size, replicability, curve-fitting, generalization and determining whether a system will trade profitably or fall apart when finally put to the test. I have also had discussions with system developers who are reluctant to apply any optimization strategy because of a fear of curve-fitting rooted in a lack of knowledge about the statistical issues involved.