Table of Contents
The backtest is a way to check the effectiveness of a strategy in the past. Does this tool really work?
When you start in the world of trading first things you learn is the concept of backtest. That is, before using a strategy it is advisable, if not essential, to check the results of some rules in previous periods. We call these rules trading system or simply system. The concept itself, or at least the idea, is very good. Since although now it seems obvious, it has not always been. Moreover, there are still traders or investors who prefer, by mistake or omission, to entrust their capital to the future of destiny.
Obviously, everyone speculates on their capital as they see fit. Of course, having at a click the means to at least try to check, and with relative ease, the returns that a strategy has had in the past, it seems at least absurd not to do so.
Note: We omit those parts of a non-quantifiable analysis. Something that happens in all types of analysis. There is always something we omitted.
Past returns do not guarantee future returns
Some of those who are reluctant to quantify their strategies may claim – and very well alleged – that past returns do not guarantee future returns. But, taking into account that they are not right, I always reach the following conclusion: if you cannot assure that what has worked, will continue to work, what makes you think that what did not work, will work now. It could work? Yes, but it seems more an act of faith than anything else.
Hope is the last thing you lose because, of course, before losing it, what you will surely lose is your capital.
The backtest does not work either
With the mind set on the idea that a backtest is better than relying on astrology, we must continue to refine so as not to make the same mistakes that many traders have made, commit and, unfortunately, will continue to make.
At this point, we must put oil on canvas to affirm that a backtest is better than relying on the randomness of fate, but it is far from enough.
Why is it not enough?
A backtest is enough to check if, if we had used a certain trading system in the past, we would have generated certain results. But the tool ends there. The word itself says it, ‘Back’ (past) and ‘test’. Extrapolate, without analyzing more, results do not cease to be – although to a lesser extent – another act of faith. Since by randomness it could continue to work, and have found a system that works without knowing why or that works and you don’t know until when. This way of proceeding of some quantitative analysts contrasts with their unceasing criticism of the technical analysis. That is, they criticize something that they unconsciously apply daily.
What to analyze?
Assuming that a system has fixed parameters, its validity must be checked in different market environments. Even in environments that do not exist. Check how a system would have worked in high environments volatility, of low volatility, before and after structural changes, in bull, bear and side markets. And so, we could go on almost indefinitely.
If the system has variable parameters, which usually happens in most cases, we will do the same process, but taking into account that the system is modifiable and, therefore, optimizable. And, the very fact of being optimizable, makes it susceptible to being over optimized. This point is of vital importance in order to try to obtain stable returns in the future.
The usual step after finding a strategy that worked well in the past is to try to optimize the model. Big mistake. First we should put it in tension, or what I call stressing the system. Put it to work in the worst possible environment known for such systems. Thus, for example, if we have a trend system, it will have to be put into operation in prolonged lateral periods to see how it behaves when there is no favorable scenario for the generation of system performance. The rationale for the above is that we do not know what will happen in the future, so putting ourselves in the worst possible scenario takes us as far as possible from the inevitable (and desirable) randomness.
What to do besides stress it?
The concepts that change everything are the forward test and the out of sample test. But, if we don’t know the future, how are we going to test something about something we don’t know? We have two options, which we will see soon. On the other hand, we have the concept of out of sample. The choice of this sample – which I recommend is enough (not just one) and with probability distributions that present disparate characteristics – is essential to get a system that works. The idea is that the backtest and optimization be carried out in different periods. Thus, free samples will remain. Although this is to the taste of the analyst. It can be done differently but we can fall into statistical errors that are not the objective of this article.
- The first way to carry out the process is what we will call traditional: We make a system, we optimize it and after looking at some metrics we put it to work with fictitious money or with little real money. If everything goes well, we put it to work in real.
- The second way to carry out the process is what we will call ‘new’, although in reality again it has little: We perform a system, we optimize it, we check the stability of the parameters over time, we perform out of sample tests, artificial forward tests and We put it to work with a real forward test. If everything goes well, we put it to work in real.
The second way of proceeding, with respect to the first, is based on two concepts: stability of the parameters in time and artificial forward tests. Artificial forward tests are not a type of out-of-sample tests that attempt to simulate a real forward test. Let’s think about the following:
We have carried out a process for a system for the last year. Putting it into operation from this month (July) until the end of the year (December) is practically the same as passing all 6 months and simulating the forward test from January to July. It is not the same, because real conditions always offer us difficult situations to invent, but we move forward and get better results. And after those ‘inventions’ because they are really inventions, we perform the forward test in real time. That is what I mean by artificial forward tests. Some will not like it this way, but to think otherwise is a mental bias. If you had discovered this strategy 6 months before you would have done the same.
On the other hand, we have the stability of the system parameters over time. For me, this is the most important metric and tells us if the system is over optimized. If the parameters remain stable over time after optimizations every X periods, it means that the parameters are less likely to have been over optimized than others that vary more. If we add to this that for each of the optimizations we perform an artificial forward test and the results are also stable, we are facing a system that is likely to be really profitable.
All this can be entangled much more. Although it seems complex but it is not. It is heavy, but it is simpler than the mechanism of a botijo. As always, everyone has their way of doing things, this is not the only one, but what I wanted to make clear is that a backtest without travel companions is useless and unusable. At least, of course, in the world of trading.