Notice:
  The views and information expressed in this document reflect the opinions and experience of the author Robert C. Pelletier.  Neither CSI nor the author undertake or intend to provide tax advice or trading advice in any market or endorse any outside individual or firm.  All recommendations are provided for their informational value only.  Readers should consult competent financial advisors or outside counsel before making any software purchase or investment decision.  CSI does not stand behind or endorse the products of any outside firms.

Copyright (c) 1999 Commodity Systems Inc. (CSI).  All rights are reserved.

Holiday Schedule Revisiting CSI's Trading System Performance Evaluator Using CSI Data with Omega's RadarScreen We're Ready for 6% T. Bonds and Notes

    CSI will be closed for voice communication on Thursday, November 25 for the Thanksgiving holiday. U.S. exchanges will be closed, but data from those exchanges that remain open will be available according to the current Data Release Schedule. The CSI host computer will be accessible as usual throughout the holiday weekend.

 Revisiting CSI's Trading System Performance Evaluator

    The modern investor is bombarded with offers of trading systems, managed accounts and newsletters. In addition, through the application of various user-defined analysis programs, traders are conceiving more personalized systems every day. They typically explore and predict trading opportunities through hindsight historical testing.

    What all of these ventures have in common is a performance record. These records are usually characterized by claims of substantial profit. All claims, whether supported by simulated hindsight profits or actual trading performance, can be statistically analyzed and rated with the Trading System Performance Evaluator (TSPE). This program was originally released almost a decade ago, but a new version of TSPE will become available soon in an upcoming Unfair Advantage® release.

    With TSPE, every system or approach can be evaluated against a standard set of rules. In some cases, when the promoter is willing to share his performance record, that record can be evaluated before the product or service is purchased. 

TSPE's Objective

    TSPE's objective is to find the probability that a proposed dollar goal can reasonably be achieved with a given capital stake, and to compute the expected level of goal satisfaction. Your required capital stake, measured in dollars, is a key sobering output that every trader needs to know to finance the anticipated trading exercise. No one should attempt to trade a particular systematic approach without knowing in advance what capital outlay must be advanced to achieve a known probability of success. 

    In offering TSPE within UA, we will be providing a tool for uncovering the more promising trading systems or track records, and for detecting improperly conceived methods. This is a service that will guide your trading efforts and inform you whether or not there is an ample chance of succeeding in the market arena with your newly found or purchased product. 
 
Methodology
 
    The approach used by TSPE draws upon forecasting theory and random simulation to assess performance. TSPE will take any profit and loss (P&L) input record and determine whether a similar result can be repeated by chance with randomly drawn samples from the original P&L set. The derived input P&L record of Trade Station, for example, comes from the export of your system's P&L record to an Excel Spreadsheet.
 
    TSPE randomizes a representative sample of profits and losses produced from the user-supplied trading system(s) and from this, a typical record of the overall trading account is created. The account's randomized profits and losses are logged as though they were actual trades. TSPE simulates thousands of trading sessions for the system(s) under evaluation with randomized trading sequences of your P&L record. 

Each session continues until:

1. Residual capital is insufficient to meet the exchange-imposed dollar margin requirement; or
2. The dollar goal is met, or
3. Excessive trading frequency (an arbitrary time limit) returns insufficient marginal profits.

    TSPE repeats each session for a range of capital levels, keeping records on the percentage of winning sessions. The winning percentage for each capital level is presented as the probability of reaching the goal.

    The statistical terms for the techniques used are "sampling with replacement" and "Monte Carlo Simulation."  These are two of the few methods capable of solving this type of problem.
 
The Simulation Technique
 
    TSPE is a Monte Carlo simulation similar to that which is commonly used in the development of military systems hardware. In a large-scale system such as a fighter aircraft, the model would likely include design specifications such as performance and reliability requirements. These simulations require the selection of random events that would trigger the degradation of a system's operational performance in certain would-be situations. The operational performance is tested as if it were occurring in real life by experiencing system failures and measuring system effectiveness. 

    In military applications, Monte Carlo simulation is used to find problems in the design/development stage. It helps contractors avoid costly revisions and corrections in post-design reality. When applied to trading systems or promoter track records, it can uncover otherwise unforeseen problems, assumptions or weaknesses before capital is risked in actual trading.
 
Correcting For Developer Biases
 
    Before the Monte Carlo simulation takes place in a testing application, profits are degraded and losses are inflated by a factor to correct for sample size, and if appropriate, user control. Since actual trading performance is less prone to error than a P&L string computed from hindsight evaluation, only sample-size corrections are necessary for actual trade data.
For an input derived from hindsight simulated trading, the number of user-control parameters introduced must be taken into account to force additional degradation on each member of the P&L string. Two methods are used to degrade results. Both techniques decrease profits and amplify losses. The first produces a mild adjustment with a procedure called the Akaike correction. The second approach, the CSI correction, uses a more severe correction that increases losses and decreases profits more substantially. This more serious adjustment should be considered when the user wishes to produce the most conservative result possible.

    Correction is always more severe for hindsight analysis because through user-control parameters, profits are inflated based on known market behavior in the past. The magnitude of the correction is directly proportional to the parameter count and inversely proportional to sample size. Results improve with larger samples and the absence of excessive user control.

    A by-product of the simulation exercise is a statistic that discloses the drawdown potential implicit from the user's input P&L string. This statistic produces the 95% confidence level of drawdown derived from repetitive trading sessions in the simulation exercise.
 
The Model
 
    A simulation requires a model that describes the process to be analyzed. In market analysis, a simple model is the P&L string from either simulated or actual market trading. The P&L distribution derived over an extended and representative period of time is used because it explicitly shows how cumulative profits and losses combine to produce likely trading events. Other components to the market model include assumptions about slippage, commissions, margin costs and the number of controlling parameters used to produce the input P&L string.

    TSPE can accommodate an unlimited number of P&L input sets, but each set is analyzed independently of the others. A set can represent a given commodity, security, or a group of like markets that have been handled the same way. These sets can have the same or varying levels of market exposure. 
 
The Importance of Trade Independence
 
    The elements within a given P&L strings are assumed to be independent from one another, and each string is assumed to hold uncorrelated, independent outcomes of P&L that could have occurred in any sequence. Trade independence is so crucial to the evaluation process that we do not recommend using TSPE to evaluate any system where the exact trade sequence is dependent. Any mechanical trading approach that uses the result of one trade to determine the size of subsequent positions suffers from a lack of independence. A simple Martingale approach, where bets are doubled (or increased) following losses, is an example of a dependent process that TSPE shouldn't analyze.
 
The Importance of an Adequate Sample Size
 
    Sample size is implicitly supplied as the quantity of P&L inputs in a data set. Although statistical sampling theory accepts that 30 samples drawn from a normally distributed population will be representative of the population undergoing measurement, we recommend expanding this quantity whenever possible.

    This is because the typical prudent investor cuts losses and lets profits run. As investors, we literally bank on having occasional large profits and many small losses. In order to have a normally distributed P&L curve, the investor would have to log as many big losses as big profits. Fortunately, this is not normally the case, so the typical trader's distributional curve is skewed heavily to the left, where losses are more frequent. TSPE will naturally focus upon the random likelihood of consecutive losses in assessing the capital necessary to achieve a profitable outcome that will reach a user's predetermined goal. This skew factor typically inflates the sample size requirement.
 

    In sampling theory, many more trials are required to establish the relative frequency of a rare event than are required to verify a likely event. In a skewed distribution where the mean, median and mode are all pushed left (see chart 2 above), it is mandatory to introduce enough trials so rare profits have a good chance of appearing.  This situation produces the need for a larger sample size that includes the rare large-profit events. It also heavily impacts upon the capital necessary to sustain trading when many losing trades must be financed before rare large profits develop.

    For these reasons, the sample size requirement is inflated. Patience becomes not only a virtue, but a necessity to bank profits using the typical market trading system. 
 
Applications
 
    TSPE can help you avoid failure due to underfunding through a knowledge of capital requirements. It can also help you avoid any system with a low probability of success or a high likely drawdown. You can estimate the amount of time required to meet your goal based on your own knowledge of average trade duration and TSPE's calculation of the expected number of required trades to meet your goal. TSPE's expected profit achievement can help you determine if the expected return on your investment is worth the risk long before you attempt that first trade experience.
 
The Output
 
    The typical output of TSPE is summarized in the UA display chart revealing the probability of returning your capital outlay and achieving your required dollar goal. TSPE works toward an arbitrary probability level of, say, 99.99% that goal achievement is met. The UA manual offers options for specifying the guiding equations that factor in the user control and the sample size constraints appropriate to describe user risk aversion.
 
Measuring the Likelihood of Achieving Predetermined Goal
 

    Using the above chart, simply zoom in on the success probability level at which you plan to operate, and interpolate to the capital plus goal requirement you must assume. If you can afford to risk that capital level to achieve your specified goal with the odds you have selected, you should consider proceeding with your systematic trading exercise. On the 'X' scale of the above chart, point A might represent your capital outlay and point B might represent your capital outlay plus your goal. (Your goal would be the difference between B and A.)  To trade your system, you will be risking A to achieve a profit of B-A, for example.

    We wish you the best in your trading, and we sincerely hope that, should the odds you must assume and the risk you must afford be too great for your situation, you will re-think your system's approach before commencing trading.

Using CSI Data with Omega's RadarScreen

    Since Sheldon Knight's "How Clean Is Your Data?" article appeared in the September 1999 edition of Futures magazine, we have received many calls from people wanting to know how to use CSI data with their trading software. RadarScreen, which is Omega Research's 2000I product, was a program of special concern.

    Data format compatibility was not really the problem; it was more a matter of convenience (or lack thereof). While Omega requires only 10 or 12 mouse clicks to use RadarScreen with all of the stocks in their own historybank.com database, any outside database requires about 40 keystrokes per stock, plus a requirement to enter the stock symbol twice and the full stock name once. The 2.7 million required mouse clicks take the bulk of the time to identify all of CSI's stocks with Omega's 2000I product. Whether or not it was the intent, the result of these extra mouse clicks is to effectively discourage use of any third-party database.

    Since so many Omega customers wanted to use CSI data with their RadarScreen, we added a file to UA containing all the stock information, and included programming (CSI's StockScannerTM) to automatically supply the many mouse clicks required of RadarScreen to use a vendor other than Omega's private label of historybank.com. These enhancements reduce the cumbersome procedure to a totally automated 14-step process. We've included a document file called RadarScreenInstructions.doc in UA version 2.0.1 that gives instructions for using CSI's Unfair Advantage database. The steps are fairly easy to apply, but the process takes considerable disk space, and one very long step must be endured to get the job done in perhaps one full weekend.

    If you missed Mr. Knight's article, you may be wondering why anyone would go to such lengths simply to use a different data supplier. The article reported that Omega committed some 21.75 errors and omissions per year per data symbol; whereas CSI, the best overall vendor, committed 4.48 per year per data symbol (based on a controlled study using S&P 500 futures and CBOT commodity). If equivalent performance can be extrapolated to stocks, then the result of any trading experience might be dramatically improved by using CSI for your analytical work. Omega's projected error rate of 21.75 errors per year (250 trading days) translates into .087 errors per day. If RadarScreen scans a 30-day period for any given stock, the chance that all 30 days of data is error free is less than 6.6%. Investing real dollars on such slim prospects of accuracy would, it seems, certainly violate anyone's level of tolerance. A summary conclusion of the Sheldon Knight study compared the performance of Omega's data bank with CSI using a simple breakout system and found that Omega's profit performance, over a two-year period, varied by as much as 50% below that of CSI. Data source does make a HUGE difference in results, not to mention the enormous waste of time in analyzing thousands of stocks without gaining a sound conclusion and dependable course of action. To their credit, Omega allowed us to run an advertisement in their Omega magazine demonstrating the superiority of CSI data over their own.

    Omega makes a great product, but restricting vendor choice through millions of required mouse clicks and unnecessary procedure is not a way to foster great customer satisfaction. In a full page letter in S&C Magazine's Nov. '99 edition (p. 27), Bill Cruz of Omega Research said in the introductory topic sentence of his letter, "Your financial success is important to us." If he really means that, perhaps he will listen when you ask that he make his products work as well with outside databases as with his private label database.

We're Ready for 6% T. Bonds and Notes

    The CBT and MidAmerica exchanges are in the process of changing contract specifications for some of their most popular contracts, the Benchmark 30-Year U.S. Treasury Bond and the two-, five- and ten-year Treasury Notes. The December 1999 contracts are for 8% bonds and notes and the subsequent contracts (March 2000) are for 6% bonds and notes. The change, of course, reflects changes in the U.S. Treasury markets themselves.

    Discontinuities in contract specifications such as this can be problematic for investors doing long-term technical analysis, and a popular solution is to adjust the data from the older contracts to reflect the new specifications. We plan just such an adjustment through Unfair Advantage. Upon expiration of the 12/99 contract (around Dec. 21), all UA users will automatically receive adjustments that will apply a ratio to all previous T. Bond and T. Note data, bringing them to a level comparable to the new 6% prices. The adjustments can be turned on and off using the "View", "User Settings", "Data Handling," "Adjust data series for unit of measure changes" option. 

    An October 1999 Futures magazine article argued against using ratios to adjust T. Bond data, but failed to mention that this prohibition more appropriately referred to the cash market only. We consulted the CBT on using a ratio, and they concurred that it is appropriate.

    Users of Perpetual Contract Data have probably noticed the impact of the disparate yields in the Bond and Note markets. This problem will intensify until the December contracts are dropped from your Perpetual Contract series. It is possible to apply the adjustment before December 21 to get better results from Perpetual Contract Data. Applying these adjustments early will distort data on the current December contracts, however. Keep in mind that you can turn them off according to your immediate needs. To manually adjust, add the following lines to UA's cdbadjust.adm file:

44,P,,,,199912,Y,0.80039624
144,P,,,,199912,Y,0.80039624
140,P,,,,199912,Y,0.80039624
150,P,,,,199912,Y,0.88568779
250,P,,,,199912,Y,0.88568779
147,P,,,,199912,Y,0.88568779
251,P,,,,199912,Y,0.926048328
293,P,,,,199912,Y,0.926048328
208,P,,,,199912,Y,0.926048328
207,P,,,,199912,Y,0.966585572
382,P,,,,199912,Y,0.966585572
146,P,,,,199912,Y,0.966585572

    QuickTrieve® users will not receive the adjustment file, but will have the opportunity to purchase adjusted historical data after the December Note and Bond contracts expire. Please feel free to contact our Technical Support Staff if you need assistance. They can be reached at (561) 392-8663 or via e-mail at techsupport@csidata.com.