The modern investor is bombarded with offers of trading systems, managed accounts and newsletters. Everyone seems to have a proven way to make money. What all of these ventures have in common is a performance record. These records are usually characterized by claims of substantial profit. All claims, whether supported by simulated hindsight profits or actual trading performance, can be statistically analyzed and rated with the Trading System Performance Evaluator (TM).
At last, every system or approach can be evaluated against a standard set of rules. In some cases, when the promoter is willing to share his performance record, that record can be evaluated before the product or service is purchased.
Our Trading System Performance Evaluator (TSPE) does not generate trading signals or suggest market posture. Instead, it certifies the results derived from your trading system and helps you understand your prospects for sustained success.
By analyzing the profit and loss record (P&L) of any simulation exercise, TSPE helps investors avoid ill-conceived trading systems that have little hope of producing regular profits. TSPE is a Monte Carlo simulation, which, through the assumption of trade independence, computes the capital requirement necessary to achieve a predetermined goal at a specified level of confidence.
TSPE Objective
TSPE's objective is to find the probability that a proposed dollar goal can reasonably be achieved with a given capital stake, and to compute the expected level of goal satisfaction.
In offering TSPE, CSI is interested in both uncovering the more promising trading systems or track records and in warning investors against committing funds to improperly conceived methods.
TSPE's Methodology
The approach used by TSPE draws upon forecasting theory and random simulation to assess performance. TSPE will take any profit and loss (P&L) input record and determine whether a similar result can be repeated by chance with randomly-drawn samples from the original P&L set.
TSPE randomizes a representative sample of profits and losses produced from the user-supplied trading system(s) and from this, a typical record of the overall trading account is created. The account's randomized profits and losses are logged as though they were actual trades. TSPE simulates thousands of trading sessions for the system(s) under evaluation.
Each session continues until:
1. Residual capital is insufficient to meet the exchange-imposed dollar margin requirement or;
2. The dollar goal is met.
TSPE repeats this process for a range of capital levels, keeping track of the percentage of winning sessions for each. The winning percentage for each capital level is presented as the probability of reaching the goal.
The statistical terms for the techniques used are "sampling with replacement" and "Monte Carlo Simulation." These are two of the few methods capable of solving this type of problem.
TSPE's Simulation Technique
TSPE is a Monte Carlo simulation similar to that which is commonly used in the development of military systems hardware. In a large-scale system such as a fighter aircraft, the model would likely include design specifications such as performance and reliability requirements. These simulations require the selection of random events that would trigger the degradation of a system's operational performance in certain would-be situations. The operational performance is tested as if it were occurring in real life by experiencing system failures and measuring system effectiveness.
In military applications, Monte Carlo simulation is used to find problems in the design/development stage. It helps contractors avoid costly revisions and corrections in post-design reality. When applied to trading systems or promoter track records, it can uncover otherwise unforeseen problems, assumptions or weaknesses before capital is risked in actual trading.
Correcting For Developer Biases
Before the Monte Carlo simulation takes place, profits are degraded and losses are inflated by a factor to correct for sample size, and if appropriate, user control. Since actual trading performance is less prone to error than a P&L string computed from hindsight evaluation, only sample-size corrections are necessary for actual trade data.
For an input derived from hindsight simulated trading, the number of user-control parameters introduced must be taken into account to force the additional degradation on each member of the P&L string. Two methods are used to degrade results. Both techniques decrease profits and amplify losses. The first produces a mild adjustment with a procedure called the Akaike correction. The second approach, the CSI correction, uses a more severe correction which increases losses and decreases profits more substantially. This more-serious adjustment should be considered when the user wishes to produce the most conservative result possible.
Correction is always more severe for hindsight analysis because through user-control parameters, profits are inflated through a knowledge of the past. The magnitude of the correction is directly proportional to the parameter count and inversely proportional to sample size.
The TSPE Model
A TSPE simulation requires a model that describes the process to be analyzed. In market analysis, a simple model is the P&L string from simulated or actual market trading. The P&L distribution derived over an extended and representative period of time is used because it explicitly shows how cumulative profits and losses combine to produce likely trading events. Other components to the market model include assumptions about slippage, commissions, margin costs and the number of controlling parameters which were used to produce the input P&L string.
The Importance of Trade Independence
All P&L strings are assumed to be independent from one another, and each string is assumed to hold uncorrelated, independent outcomes.
Trade independence is so crucial to the evaluation process that we do not recommend using TSPE to evaluate any system where the exact trade sequence is dependent. Any mechanical trading approach that uses the result of one trade to determine the size of subsequent positions suffers from a lack of independence. A simple martingale approach where bets are doubled (or increased) following losses is an example of a dependent process that TSPE can't analyze.
The Importance of an Adequate Sample Size
Sample size is implicitly supplied as the quantity of P&L inputs in a data set. Although statistical sampling theory accepts that 30 samples drawn from a normally-distributed population will be representative of the population undergoing measurement, we recommend at least 50 P&L inputs.
This is because the typical prudent investor cuts his losses and lets his profits run. As investors, we literally bank on having occasional large profits and many small losses. In order to have a normally-distributed P&L curve, the investor would have to log as many big losses as big profits. Fortunately, this is not normally the case, so the typical trader's distributional curve is skewed heavily to the left.
The quantity of trades in a P&L input is very important in a TSPE evaluation, where a large quantity of trades restricts the degradation to actual experience. Any system with many trades that shows a favorable outcome in the performance summary will also show favorable results with TSPE, even with moderate freedom loss. This is because many, many consistent profits significantly add to the credibility of a systematic approach and repeatability becomes statistically possible. More typical situations involving smaller sample sizes are not as reliable because a small sample suggests a less likely chance of repetition.
TSPE Applications
TSPE can help you avoid failure due to underfunding through a knowledge of capital requirements. It can also help you avoid any system with a low probability of success. You can estimate the amount of time required to meet your goal based on your own knowledge of average trade duration and TSPE's calculation of the expected number of required trades to meet your goal. TSPE's expected profit achievement can help you determine if the expected return on your investment is worth the risk.
For more on why TSPE is important to investors, read "In Search of a Sure Thing" by TSPE developer, Bob Pelletier.