Letters To S&C by Technical Analysis, Inc.
BACKTESTING WITH OUT-OF-SAMPLE DATA
I have been developing an intermediate-term stock trading model, which captures weekly closes of stock prices (approximately 550) and major indexes from 12/31/1999 to 1/27/2012. My model has been optimized and backtested for the in-sample data from 12/31/1999 to 06/25/2010 with excellent results. The data from 6/25/2010 to 1/27/2012 has been forward-tested (out of sample) and has a return of 32.05% compared to a 24.81% return of the Dow Jones Industrial Average (dividends not included).
In order to determine the feasibility of my model and also to enhance my confidence in it, I would like to know: If I downloaded several additional stocks to my database from 2000 to present and run them against my rules, would the additional data be considered out-of-sample data?
Looking at the definitions of in-sample and out-of-sample testing may help you determine whether testing on additional data would qualify as out-of-sample. When you run a system on the same historical data set that was used to develop the system, it is considered in-sample data. Out-of-sample testing is when you run the same system on data that it has not encountered before in the same market and time frame.