A Backtesting Protocol in the Era of Machine Learning

On top of its tremendous success in physical and biological sciences, machine learning has great potential in investment management. For instance, as was previously discussed in one of the recent blogposts (Empirical Asset Pricing via Machine Learning), machine learning can aid investors to design investment strategies. Due to its success in other fields, it is often perceived that machine learning would be a “magic box” that has the power to find repeatable patterns in data, which can be translated into profitable strategies.

Discovering profitable strategies is a notoriously challenging task. Although machine learning wields great power, a great danger comes with it as well. Namely, the ease of deceiving oneself that they have found the “holy grail”. One can face the risk of misapplying the machine learning toolbox. Just as for any other tool, it is helpful to have a manual that would guide one through machine learning application in investment management. Fortunately, Arnott, Harvey, and Markowitz (2018) developed a protocol for quantitative finance with a primary focus on investment strategy backtesting. So here is the summary of key suggestions to guide the process.

1. Research Motivation. It is best to have an investment idea with an ex-ante economic foundation. Humans are natural data miners as we continuously search for patterns, and machine learning brings this trait to a whole other level. In contrast to the past, data and computing power are abundant. Today one can start research without even specifying a hypothesis by delegating this task to the algorithm. When that is the case, the strategy is likely to fail when implemented in live trading.
2. Multiple Testing and Statistical Methods. Testing multiple strategies, altering combinations of variables, choosing data, and type of test can lead to a false discovery through p-hacking. Hence, it is crucial to keep track of different model specifications and penalize results in certain cases. Some good practices include multiple hypothesis testing, measuring correlations between strategies, and cross-correlations between signals. These methods allow controlling the portion of false-positive results.
3. Sample Choice and Data. Justify the test sample prior to running the test and ensure that the processed data is of a high quality. One should not fall into anecdotal saying that states torture the data until it confesses.
4. Cross-Validation. It is important to remember that live trading is the only true out of sample, while the holdout sample is not. Although cross-validation is extremely useful, it has weaknesses and requires a large amount of data.
5. Research Culture. Reward good science instead of good results. To avoid investing in false strategies, one should be honest and keep in mind that on its way to discovering a profitable strategy, it is expected that most experiments are likely to fail.

In its essence, the protocol is simple yet powerful. The main goal of the protocol is to improve the outcomes of the investment process. Shortly speaking, to improve outcomes, one should avoid investing in false-positive strategies, which are often a result of backtest overfitting, and maximize the chance of discovering new “true” strategies.

For a deeper dive into the topic, please look at the paper on SSRN.