Prasant.net

We are surrounded by predictive models. Every industry uses it, from the most basic of demand forecasting algorithms built into an ERP system (SAP, Oracle) to the complex algorithms used in the Hedge Fund/HFT industry. While there is no accurate way to predict a future data point no matter how good your algorithm, most users of such systems hope to be within a certain error limit, the proverbial ball park figure.

How can we be reasonably certain of our research and tests based on historical data though? There are quite a few challenges involved.

1. How much data is enough data?

Is backtesting on 3 or 5 years of tick data sufficient for you to have to confidence to put in your money or your client’s money on the system? It depends. If your time frame isn’t long enough, your model may not be able to capture different scenarios in the market.
Is having 20 years of data enough? Again, it depends. Data too old may not be relevant to you at all since markets change over time. And what type of data would be appropriate? Tick data, or EOD data?

As a general rule I have followed, systems based on volatility indicators like Donchian Channels or Bollinger Bands seem appropriate for smaller timeframes such as 1-minute or 5-minute data.

Trend following systems on the other hand, require larger time frames, such as EOD data to cut out the noise levels. An MACD based system would cause you to go mad in the scenario such as the one below I faced in 2008. Trading the Euro Bund futures, something triggered a violent drop. The price fell nearly 170 odd points in a matter of a few minutes. No one had been assassinated, no nuclear war had broken out. Strangely, it was not even a fat finger trade. Someone actually meant it. While we were waiting for an announcement from Eurex for a reversal, none came through.

In other words, it is not the quantum of data that is important, but the nature of data. Do you have enough to test your system in a variety of scenarios? Bull markets have a strange way to make people believe they are great traders, the actual test is when a bear run occurs.

2. Are you overfitting?

We have more data than we have ever had in history, yet, are we necessarily doing things better with it?  Sometimes too many data parameters render testing results useless for the real world. This is not to say that optimising your system is bad or undesirable.

Our predictive model is only as good as the predictions it makes, and not just good looking equity curves for historical data. And this is where optimisation plays a part. As with the previous point, it is vital to ensure that your parameters reflect real world scenarios, and that you test them across a variety of scenarios from the past. Going overboard with optimisation results in a scenario where your historical testing results are the “Best Case Scenario” under a given set of conditions. One way to check for this, is to see the magnitude of change occurring (positive or negative) for a very small change in a given parameter. If the test result is significantly different due to a minor change introduced, that is a good sign that you might have over optimised. The more parameters you add to this soup, the greater the complexity, the greater the divergence from a real world trading scenario.

So to have real predictive power, it is vital to not have too many parameters, and also that the parameters introduced come into play frequently.