We’ve been through the reasons that a black box system is no way to trade, and why high yield accounts managed by people may not work out either. Then a cautionary tale about the risks of setting stops on the market, and a look at one of the latest technical tools that is starting to be recognized in the West. Now we look at one of the most useful techniques for would-be traders to master in order to prove their concepts before risking real money.
Backtesting is key to knowing that your system, however innovative and original, is actually going to work to make you a profit. If you think you have it right, you can always try it out in the market straight away, but that’s not much fun, and expensive, if it doesn’t work. You can paper trade for a while, and probably should so that you get used to the system, but that’s not very effective to prove and improve the system.
A few years ago, you might have done a backtest by hand, going through charts and marking on the trading points, but now there’s no question that you can do it easily with a personal computer. Testing your proposed system on historical data will let you see how good it really is, and perhaps find opportunities for improvement, all before you risk a penny.
Of course, a successful backtest, while important, is no guarantee that the system will make money in the future. But it should arm you with valuable information about what to expect – for instance, could your system have ten losses in a row, but still come out on top overall? If you know in advance, then you will not abandon using it after eight losses, thinking it’s not working.
Setting It Up
The way you program the system will depend on your software, whether TradeStation, MetaStock, AmiBroker or another. Once the basic system is encoded, you will then test on historic data, and try and improve the performance. There are two pitfalls to this.
The first is referred to as curve fitting, and that’s when you take pains to fi t the system to the data to come up with the best possible performance. The system finished up looking great, based on the set of data you wrote it for. What you have really done is match the system to the data. Sometimes traders will even just extract a part of the chart, and make the system work well on that. If you don’t have a system that can cope with different market conditions, then you probably shouldn’t use it.
The other pitfall is similar to this, and called over-optimization. You can take this too far, too, particularly as you may find that your software accommodates automatic optimizing. For instance, TradeStation can find the best parameters for any indicator you are using. You have to realize that, with a basically sound and effective system, there are a range of effective values for the parameters, and you cannot find one set that will perform perfectly in the future. The answer is to determine the range that works acceptably, and take a middle value. If your system results are particularly sensitive to small changes in one value, then you should go back and see why, as that suggests you have it too highly tuned in to the data set.
When you have tweaked the system sufficiently, but not too much, you can then try it out on a new set of data. You should never optimize the system using all the data that you have, as you need a new sample to give more results. This will show whether you have over optimized or curve fitted the system too much for general use. If your system is good, it should work well on the new sample; if it doesn’t, try to figure what went wrong on the new sample, but don’t make the mistake of optimizing on that set of data too – keep the data apart, so that it can be used in testing again.
When you are backtesting, look at the number of trades that the system makes. It should be at least thirty in order to be statistically valid. If it is less, you need more data points, otherwise a couple of extreme results could be distorting the outcome.
Finally when running the tests, you can put the system to work on different securities, and see the results. Again, a sound system should be consistent, and not tuned in to just one stock, otherwise you may well be disappointed with the performance in the future.
Interpreting It
All the results that you get from backtesting aren’t a lot of use if you don’t know how to use them. It’s not just a matter of seeing which set of results gives you the greatest bottom line, as there are other considerations that your test results will reveal to you. In fact, at this stage your results won’t even account for any commissions and slippage, so the positive return may be much less than you think.
Typically the software will give you a total net profi t. If this isn’t positive, your system has failed at the fi rst hurdle, and you need to develop another. But the number you see here is not that good an indication of how it will work out for you in practice. You need to look at the number of trades, the maximum drawdown, the consistency month after month, the size of gains and losses, and the other data.
First, if you are comparing the results of two systems you want to choose the system that gives you fewer trades. This keeps your commissions and slippage costs to a minimum. Bear in mind that you do want to see the results of at least thirty trades, though, as stated above, to get a good idea of the performance.
Second, don’t pay much attention to the percentage of trades that were profitable. Some traders worry about this, but it might be 50% or less and you can still have a good system. What matters is how large the wins are compared with the losses.
Look next at the size of the largest wins and losses. For instance, is the profitdue to just one trade, which may not happen again? If the largest gain accounts for a major part of the profit, then it’s not a good system. The largest loss should not be as much as the largest gain, as you are looking for a system that cuts your losses short, so if the loss is nearly as big as the gain you may need to reconsider how you define your stop loss level. Going by the general advice of reward to risk ratio being 2 to 1 or 3 to 1 you need to see a clear difference in the amounts.
I spoke about the number of consecutive losses previously. If this is a big number, you may find it hard to stick with the plan and keep trading. On the other hand, if you know it can happen because you have seen it in testing then you may be able to stomach it. The amount of the drawdown is very important, as it tells you what you have to be prepared for to use the system. It will tell you how much money you need to weather the worst period, at least of the test data. You should be able to go through a losing streak twice this size, in case one follows another, if you want to sure that you can continue trading. The size of the drawdown is an indication of the risk, so given an otherwise equal pair of systems you should choose the lower drawdown.
The profit factor is the gross profit divided by the gross loss. You want for this to be at least 1.5, and preferably more than 2. Anything close to one is not far from failing to make a profi t, and it wouldn’t take much to tip it into losing territory. Lastly, you want to consider the distribution of the returns. It’s preferable to have a system that gives steady small equity increases every month rather than wild swings over time. You can generate month-by-month charts of equity to be able to discern this.
Commissions and Slippage
Even after all this analysis of the results, you still need to figure in the costs of trading to see if the system you are proposing to use will be worthwhile. If you use a discount broker such as Scottrade you will know the costs of the trades accurately, at $7 each or $14 for a round trip – some other brokers charge according to the value traded. This is where a system that relies on trading frequently and making small profits will fail to prove profitable.
Finally, and depending on your chosen market, you need to make an allowance for slippage. This is the figure that represents the difference between what you thought you were trading for, and what the real numbers may be. At best this is a guess, but without accounting for this, you may find an apparently profitable system that does not work out in practice.
As you can see, there is a fair amount of work involved in testing and implementing a trading system. Those who tell you that you can just plug-in and go are trying to delude you, and anything worthwhile takes some time and effort. If you truly want the independence and security of knowing that you can trade successfully for a living, then you have to be prepared to do the homework. When you do, you will develop skills that will last for the rest of your life.
