Studying the Anatomy and the Dangers of TradeStation®
By Bob Pelletier (Adapted from the CSI Technical Journal, January 2000)
I have been attempting to unravel the mysteries of TradeStation (TS) for several months now, and have recently focused my attention on its "Strategy Performance" report that offers an "Account Size Required" for typical derived systems. The capital stake for most efforts was minimal, and I was eager to see if I might score handsomely with the small capital outlay TS suggests. Could it really be as good as it looked? I put on my detective hat to find out.
I selected the day-session T-Bond contract for the TradeStation study, because it is a very liquid and popular market. Due to the flawed nature of the Historybank.com database that comes with TradeStation, the first step was to substitute clean CSI data for TradeStation analysis. Many things can go wrong when attempting to develop a system, and the last thing I wanted to do was start out with a database that could not be trusted. Omega's Historybank.com, according to the Futures Magazine study published in September 1999, has an error rate that is 485% as high as the error rate of "the best" firm studied, which was CSI.
It took about an hour of my time to wrestle through the many options and introduce the range of parameters that TS would use to optimize the results. The method for achieving a profitable system with TS was not simple. I was concerned that I wasn't even given the opportunity to specify the trading commission and the expensive slippage tax, although I do remember having seen such a prompt on one occasion in an earlier exercise. I'm sure happy I don't have to make a regular habit of playing with that package. There are too few hours left in my life to spend another one struggling through that again.
TradeStation studied T. Bonds for me over a two-year period from December 12, 1995 through December 12, 1997. It suggested a stop parameter and a slow and a fast moving average that seemed reasonable, based on a narrow range of settings for the three parameters. The Strategy Performance report showed that the system produced a very respectable $4,156 profit (before commissions) with a required account size of only $1,438. Such a situation would reward the user with a "Return on Account" of an astounding 289%. With results like this, I thought, Bill Gates would have to move over, because I had already begun making plans to buy him out well before the Millennium became a blur in the past.
The Strategy Performance report also showed that I could achieve those results in just 15 simple trades. With such a small investment, I was aching to call my broker to make that first trade. Back to reality - this is only a test! I needed to get a feel for whether TradeStation was on the up and up, and know whether or not the performance derived in hindsight would be possible in the future. Of course, it seemed remote that anything could go wrong in actual practice. After all, the performance summary showed that Gross Profits exceeded Gross Losses by a 2.4 ratio. How could I go wrong?
The next part of my investigation involved evaluating the TradeStation results with Unfair Advantage's® (UA's) new Trading System Performance EvaluatorTM (TSPE). In the November 1999 and December 1999 issues of the CSI Technical Journal, we wrote about how one might certify or at least assess the risk of trading or repeating a given performance record. We talked about how this would be accomplished, and commented on how one might interpret the results. Please see those issues for more information.
I entered each trade for the two-year test period into TSPE, and factored in a $100 commission and slippage expense per trade to compensate for these figures' exclusion from the TS analysis. In spite of Omega leaving out these real-world requirements, they were forced in place. The end results weren't good. TSPE said my account size should be $30,000 and that my chance of achieving a $2,656 goal (TS's after-commission result) on a $30,000 investment would be about 89% (see right dot on chart). The probability of preserving my $30,000 capital was 98%. The probability of achieving the same goal on TS's $1,438 investment was only about 30% (left dot) and the probability of preserving my $1,438 capital was about the same. How could such a good system operate so poorly in projected operation? And how does TradeStation justify telling me that I will get rich when the likelihood of going bankrupt and losing my $1,438 capital stake is at least 70%?
The three curves from top to bottom display as a function of invested capital (the horizontal scale), the probability (vertical scale) of remaining solvent, of retaining invested capital, and of reaching a specified goal. The dollar goal is an input not shown on the chart. The larger the dollar goal, the larger will be the separation between the bottom two curves. The more marginal the profitability of a given system, the larger will be the separation between the top and middle curves. To force all curves to approach certainty of goal achievement requires unlimited trading, but residual invested capital will not be eroded should the trader suspend trading early.
Before going further with my search for a profitable system, I frantically searched the TS documentation to understand the declared assumptions concerning the TS Strategy Performance Summary. Nothing! There was absolutely no mention of any declared assumptions!
I inductively determined their assumptions from the factual output. Since the $1,438 account size matched up with the account's largest drawdown, I deduced that this was how TradeStation decided the maximum required account size. The apparent underlying assumption therefore, was that trade-by-trade results are not independent. In other words, there is a dependent sequence of trading outcomes that forces the very low drawdown. The resulting account size requirement, return on investment, and all of the superlatives and statistics reported in favor of the derived system were based on the chronological (serial) arrangement of the artificially derived profit-and-loss statistics. Omega could argue that the "Account Size Required" was a pre-simulation proving figure and not a post-simulation limit, but that would be contrary to the purpose of back testing a proposed technical method.
The dependence assumption adopted by TS is a puzzling premise, given that markets are reactions to the unpredictable daily effects of weather, government policy, regional politics, employment rates, world trade, natural resource production, product substitutability, supply and demand, and many more factors too numerous to mention. Such influences strongly and absolutely contribute to market randomness, which in turn forces the necessary assumption of independence, not dependence, on the trade-by-trade results. Given the TS performance statistics, it would be impossible for the makers of TS to argue that their performance summary is based upon an assumption of trade independence. A simple change in sequence resulting in a clustering of losses would have measurably inflated the account size requirement, proving them wrong.
What is there about TSPE that makes it assess performance in such a conservative light? As we explain in CSI's TSPE documentation, TSPE makes a critical conservative assumption known as "trade-by-trade independence." Independence, in a mathematical sense, is one of those buzzwords that most people dismiss as unimportant. You can see from TSPE's high capital requirement that independence bears heavily upon the result.
When one gets into the trading pit, profits and losses tend to cluster together, as alternating profits and losses are statistically infrequent. Commission and slippage are taxes that must be paid. In the TSPE simulation, trading results are randomly sampled from the P&L list. The randomly compiled string of consecutive losses is allowed to amplify account size requirements, which will conservatively turn the odds against you.
Another characteristic of TSPE has to do with parameter control. The moving average crossover study used and analyzed by TradeStation requires two moving averages and a stop limit. Each of these controls (3 in total) contributes to a reduction in simulated profits and a worsening of losses to account for the hindsight loss of statistical freedom.
In spite of the above, I still hadn't totally given up hope on buying out Bill Gates. Giving Omega the benefit of the doubt, there was still something I needed to do to prove their result was repeatable. I wondered how that derived system would have done in the two-year period that lies after the two-year development period. If it worked there, perhaps further testing would give me some promise with my newfound TS system. A quick review of the unseen virgin period from 12/12/1997 to 12/12/1999 revealed that my system would have lost $3,843 on 15 more trades, inclusive of commission and slippage. The facts bear it out: TSPE did a good job of keeping me out of the market.
Given the TSPE results and confirmation from TradeStation, it appears that not only will I not be able to buy Bill Gates out, I won't be able to buy out Bill Cruz either. The mysteries of TradeStation are, indeed, unraveling. We can only advise caution and wariness. In spite of the ads, always read the escape disclaimer designed to protect the vendor and buyer beware. There is no free lunch.
TSPE is said to "certify" a system based upon a review of the hindsight past or based upon the prospects of repeating actual experience. In the former event (hindsight), parameter control comes into play. In the latter case (repeatability of actual results), only sample size bears on the outcome. Parameter control causes degradation and sample size serves to overcome control parameter effects. In either event, the so-called certification is measured against a probability of achievement. In other words, certainty has only to do with the declared probability, always leaving a measured chance of ruin. One should assess that chance carefully.
Trade-by-trade example of TradeStation study results:
588
-318
-475
556
-506
-725
4087
-162
-287
806
-693
493
-225
-6
-475
TSPE and TradeStation
Using Omega Research's software, traders subject a given market to one of many possible studies such as moving average crossover, stochastics, RSI, and others. Once the choice has been made, stop-loss limits can be imposed, objectives can be targeted, and the paper trading commences. TradeStation supplies a "Performance Summary" that reports the maximum number of consecutive losses, the percentage of losses, the average profit, the average loss, the largest winning trade, etc., etc. A quick look at a Performance Summary reveals several assumptions that may be mitigated through TSPE.
One important Performance Summary statistic is the number of consecutive losing trades that occurred in the actual or simulated past. If it is small and the overall profits and losses appear to alternate, then you may be in for an unpleasant result when you attempt to repeat that situation in real life. To find out how the system might play out, TSPE will simulate the trading system that derived the P&L string. Over and over again, it randomly selects (with replacement) from that P&L set. Don't be surprised if it reveals that the same set of outcomes may not be so opportune in practice, causing excessive drawdown and possible account bankruptcy.
Understanding the Discrepancies
The quantity of trades in a P&L input is very important in a TSPE evaluation, where a large quantity of trades restricts the degradation to actual experience. Any system with many trades that shows a favorable outcome in the Performance Summary will also show favorable results with TSPE, even with moderate freedom loss. This is because many, many consistent profits significantly add to the credibility of a systematic approach and repeatability becomes statistically possible. More typical situations involving smaller sample sizes are not as reliable because a small sample suggests a less likely chance of repetition. A small sample size was the downfall in our example study. Although this may, indeed, be a profitable system, the limited amount of statistical data makes it impossible to certify that profitability will be sustainable.
In a TradeStation Performance Summary, the quantity and percentage of winning trades must be viewed with a wary eye. Any trade ending with a breakeven result is counted as a profit! While every trader desires to achieve consistent winning performance, introducing faulty input through false assumptions is not helpful. The foregoing is not a criticism of Omega's software. We are simply suggesting that TSPE does much more. It will help keep you from misinterpreting the details by requiring an unbiased account of each trade.