Backtesting is a term used in modeling, referring to testing a predictive model on historical data. It is a prediction and a particular type of cross-validation applied to a previous timeframe. For example, in a business strategy, investment strategy, or (financial) risk model, backtesting seeks to estimate a strategy's or model's performance during a previous period. This requires a simulation of the conditions above with sufficient detail. This is the first limitation of backtesting: it requires detailed, reliable historical data. Secondly, there is a limit to modeling strategies; they are not to influence historical prices. Finally, backtesting, like other models, is limited by possible over-adjustment. Despite these limitations, backtesting provides information unavailable when models and strategies are tested with synthetic data.
The first step in backtesting is to select the threshold values within a period covered by the historical data. Then, historical data are truncated at the threshold for each threshold value. Next, the forecast model is trained and applied to truncated data. The forecasts thus obtained are compared with the complete original data. Finally, an average forecast error is established for all thresholds. This error can be read as an estimate of the misconception associated with the model when making forecasts (for future data). Choosing the most appropriate threshold values requires a minimum of knowledge. As a rule, increasing the number of threshold values improves resistance to overfitting problems. In stock optimization, since there are hundreds of SKUs to analyze, a few threshold values are enough to determine whether one forecast method is better than the others.