UA 1.66 BACK ADJUSTING CHANGES

Summary

Starting with version 1.66 of UA, CSI is offering an alternate back adjuster as well as many fixes and enhancements to the standard back adjuster. For people receiving a new CD, the alternate back adjuster will be the default. There are many subtle difference between the back adjusters in 1.66 and in previous versions. The remainder of this web document will describe these in some detail with an emphasis on getting the results you need.

Choosing the Standard or Alternate code

The choice is accessible from the Unfair Advantage main menu. Choose View -> User Settings -> General tab -> Use Alternate Back Adjust Code to make your selection.

Overview: What is the new Alternate version?

Standard Back Adjuster: A continuing effort by Bob Pelletier.; FORTRAN engine.; Backwards looking.; Willing to roll back if necessary to maintain a position.

Alternate Back Adjuster: A three week only back adjuster written by Steven Davis.; C++ engine.; Forwards looking.; Always rolls forward or exits position.

Feature differences:

Both codes implement all features exposed to the user by the UA interface including a detrend option; custom input contract selection; rolling date determination method selection; and, contract differential determination method selection. There are two features which both codes support and which are not currently selectable.

Raising negative contracts. Both codes have the ability to add as many thousand points/ticks as needed to achieve a positive series. We believe this is better than adding a minimal offset since the applied delta can be easily determined by comparison to the nearest future. Neither code currently does this by default.
Days of confirmation. Both codes allow for a roll on volume and/or open interest to roll on a number of consecutive inequality reversals. However they differ in the default number of days. The standard code has changed from 0 day of confirmation (roll as soon as the inequality reverse) to 1 day of confirmation for close to open delta calculations. The alternate code defaults to 0 days of confirmation, but expects a parameter for the number of days which is not currently accessible to the user. The risk of confusing the user with inconsistently available options may outweigh the value of selectability. The reason for the difference in defaults will be explained below in the new gap/old gap calculation discussion.
Tick Integrity In the event that UA lists an opening price which is not an even number of ticks, the delta is computed to be an even number of ticks. The standard and alternate codes handle this differently. The alternate code rounds all price to make them an even number of ticks. The standard code ignores tick considerations until it comes to writing the actual data series which is then rounded to an even number of ticks.

What exactly is meant by the delta splicing choices?

The following analysis is of a highly detailed nature which is not suited to the majority of the UA users. Proceed if you are really interested in the exact method UA uses to compute back adjusted deltas.

What is a Back Adjusted data series?

Never Ending, Continuous Price Series: A never ending, continuous series of prices in which changes in the prices are representative of the value of the underlying commodity with minimized contract-to-contract effects. For this purpose, Perpetual Contract (R) data has even less contract-to-contract effects.
A Price Series implicitly representing rolling: For historical testing, it is very handy to separate the rolling logic and the entry/exit logic. If a trade begins as a long position in December 1997 Silver(price A), rolls to March 1998 Silver (selling December Silver at price B and buying March at price C) and then exit (price D), the profit/loss on the trade ((B-A)+(D-C)) is the same as the simple exit - enter profit loss on the back adjusted series (D-(A-(B-C))) provided that the back adjusted series rolls as you do. This is the reason why there are so many options.

As more and more options have been implemented, the basic delta options have become a bit more subtle than one would think.

Delta Calculation Methods

Each of the three choices: close to close, open to open, and close to open involve a calculation of the between contract price differential which is cummulatively applied to bring historical prices to current price levels.

Close to Close

A Close to close back adjuster simulates a buy and sell at market on close order, or a spread order executed at close. If the "Roll when known" check box is selected, then the order will be executed the day after the volume and/or open interest moves. Without the check box, one must execute order on close before the the open interest and/or volume data is published. The delta in this case is the difference between the settlement price in new contract minus the settlement price in the old contract.

Open to Open

A Open to open back adjuster simulates a buy and sell at market on open order, or a spread order executed on the open. The order is executed one day later than the close to close order since one has to wait for the next close. The delta in this case is the difference between the opening price in new contract minus the opening price in the old contract.

Close to Open choices

The Close to Open back adjuster simulates exiting the current position at the close of one day and entering the new position at the open of the next. The delta is the difference between one day's close in the old contract and the next day's open in the new contract.

From the point of view of a continuous price series, the close and open between two consecutive days for a contract are generally close together so the difference in the close to open gap should be highly representative of the contract to contract differential.

From the historical testing perspective, the close to open delta simulates a trader exiting the current contract with a market on close order and then enters the new contract with a market on open order the following day. I am not aware of any position traders who trades this way. This might be because the profit opportunity may be lost in return for reducing risk only on the the roll date.

Reintroducing overnight gaps

The close to open delta price series is distinctive in that there is no gap on the roll date. This is correct: if you withdraw your position overnight than the overnight gap won't directly effect your profit/loss on the trade. Some view this as a back adjusting artifact which can be removed by reintroducing an overnight gap. UA provides the option of reintroducing the old contract gap or the new contract gap. With the reintroduction, however you have achieved either a close to close or an open to open delta with a fancier name.

			Close to Open with Old Gap
	Mar open on the 20th - Dec close on 19th 
	- (Dec open on the 20th - Dec close on the 19th)
	=	Mar open on the 20th - Dec open on the 20th
	
			Close to Open with New Gap
	Mar open on the 20th - Dec close on 19th 
	- (Mar open on the 20th - Mar close on the 19th)
	=	Mar close on the 19th - Dec close on the 19th

This is a good example of the kind of generational misunderstanding that is common since the introduction of the computer. For a numerical example, suppose we want to roll by exiting our December 1997 Silver position at the close on the 19th and entering a March 1998 Silver position at the open on the 20th:

            Dec 1997                    Mar 1998
Date        Open     Close              Open     Close
1997/11/19  510.2    521.8              516.7    528.6
1997/11/20  521.5    522.8              528.5    529.8

Then the gaps are

	Close to Open = 528.5 - 521.8 = +6.7
	
	Old Gap = 521.5 - 521.8 = -.3
	Close To Open with Old Gap = +6.7-(-.3) = +7.0 = 528.5 - 521.5

	New Gap = 528.5 - 528.6 = -0.1
	Close To Open with New Gap = +6.7-(-.1) = +6.8 = 528.6 - 521.8

Roll Day selection

The Standard and Alternate Back Adjusters provided with Unfair Advantage differ a bit in the exact interpretation of roll date. The following example illustrates. Suppose that one is rolling on open interest.

            Dec 1997                  Mar 1998
Date        Open   Close    OI        Open   Close    OI
1997/11/18  509.7  510.0  42669*      516.0  516.5  39144
1997/11/19  510.2  521.8  39006       516.7  528.6  44963*
1997/11/20  521.5  522.8  34975       528.5  529.8  46351*
1997/11/21  523.5  541.8  29868       530.2  548.6  53726*

The following table shows the dates used for the delta calculation:

                            Date/s for Delta Calculation
                            Standard          Alternate
Close To Close              20st              19th
Open To Open                20th              20st
Close To Open               20th,21st         19th,20th
Close To Open with Old Gap  20th,21st         19th,20th
Close To Open with New Gap  20th,21st         19th,20th

The difference in this case is that the alternate back adjuster is rolling based on one day's data, but one day delayed when necessary to execute the order. The standard code is using a day of confirmation in deciding to roll, but retroactively rolls. The following table shows the first date onwhich the new contract is listed as the active contract

                            Date/s for Delta Calculation
                            Standard          Alternate
Close To Close              20th              19th
Open To Open                20th              20th
Close To Open               20th              20th
Close To Open with Old Gap  20th              20th
Close To Open with New Gap  20th              20th

The difference here is that without the "Roll when known" box checked the close to close order with zero days confirmation can be executed a day earlier than the enter orders used in the order rolls types.

Conclusion

To summarize, starting with version 1.66 of UA, there are two back adjusters available. One is in FORTRAN and has seen a fair amount of field use. The alternate is in C++ and is quite new. Both have been thoroughly tested on standard and exotic data as part of our continuing quality assurance commitment. The results may differ in that the roll date may differ by a day and the handling of partial tick prices is different. Make your selection through the View->User Settings->General form.

Addition 1. Warning about abberations introduced by back-adjusting

Since I wrote the above document, many bugs have been fixed, and the adjusters are in heavy, productive use. I do often receive emails point out that abberation in the back-adjusted series which doesn't fit what contract data is supposed to do. Often these difference are quite inherent, and I fear that too many traders are under the mistaken impression that back-testing can be made so meticulous that forward-trading can really be simulated. The following section describes a number of inherent abberations and an introduction to Perpetual Contract(TM) data which has a different set of advantages and detractions.

The back adjust series may have a daily range as large as twice a limit move, because this can really happen. Suppose that there is a run-up caused by a perceived risk. The current contract could go limit down without greatly affecting the distant contract. Your exit at the open would take a limit loss, and during the day, some legislative risk could be introduced in the distant contract which charges you an additional limit loss. I don't know of it ever happening, and I think it is unlikely to happen, but it is a real possibility which should not be artificially "cleaned" from the data.

Many people make the mistake of thinking that historically back-tested results can be reproduced by forward trading in the real contract using signals generated by a back-adjusted series. This assumption that profits and loss are computed from the same series from which the system makes its decisions is only true in back-testing. The fact that it isn't true in forward-trading is inherent and very important to keep in mind when evaluating systems.

Forward-trading suffers from bad data. If the exchange reports a high for the day which includes an unreasonable trade, your system might well generate a signal. When historically back-testing, either this bad price has been caught and fixed or the back-testing software will give you full credit/loss assuming that you really could have traded in that price range. Optimized systems have an unpleasant, special aptitude for always being on the profitable-side of bad data points.

A back-testing system has no problem with rolling a large position at the open, but the reported opening price may be either an average or the first trade. The price may not be fillable even for one contract much less a large position.

If a hurricane is bearing down on the Gulf coast and your system has you long in unleaded gas, would you really roll that into a distant contract which reflects the risk premium for after the hurricane season is over? If not, then your forward trading is not going to be reflective of your back-testing.

If you exit your current position at the close, and then the market goes limit up before you can enter into the distant contract. Your back-testing system will happily fill you at the open and show your profit/loss from that point. In fact, you could be limit for several days miss out on the whole move.

If you choose to roll at the close based on open interest, you are still assuming that you can correctly judge what the interest will turn-out to be based on the ticker-stream. Since most of the volume of the day is at the close, it is easy to roll off by a day or two. The back-testing has no such problems.

Of course, when it comes to roll timing, bad data can easily throw-off rolling, especially when rolling by volume. It is very easy for a mistake in the reported volume to shift the roll date by weeks. In forward trading, of course, once you are in the distant contract, that is where you are staying, right or wrong.

When additively adjusting data series (as opposed to proportionally), many markets have a systematic rolling profit depending upon how you roll. In order to adjust for these profits, the early history must be artificially small, or even negative. It is important for system designers to level-out their indicators so that the system produces the same result regardless of adding a factor to the input prices, but how much of that 10-year return came from this artificial trend in the data? And how much of this profit would really be left after so many commissions and so much slippage?

The important thing is to understand that the input data that the system receives is not the same as the trading data upon which it acts. Not even if you try your very best to try to keep them the same. The Perpetual Contract creates a much more representative input data series. It does not pretend to match the current contract, but rather reflects the underlying future market by gradually rolling positions into ever distant contracts.

Sure limit days can still cause a position to not roll correctly, but if you are rolling rolling 1/30 th of the position that day, then the size of the abberation is reduced by 1/30th since the rest of the position acted as tested.

By smoothing out the inherent differences between the contracts, the Perpetual Contract is well suited for long-term trading or as a base-line to compare the contract data if one wishes to do shorter-term spread trading. No system can successfully predict the idiosyncratic premiums and discounts that the current contract suffers from. So does passing such information to your system as input help your system make complex trade decisions, or it is just noise that your system has to work around?

Whatever series you believe will provide the best input for your system, I think that system designers are well advised to consider the inherent abberations introduced into the data and the alternative of using the Perpetual Contract as an input to their system.

Addition 2. Trends in back-adjusted data which aren't present in the underlying (11/14/06)

To see that this can happen naturally, consider the case of a commodity with a carrying costs, but no execution costs and no volatility in the cash market. The trade of shorting the future and buying the physical would be an arbitrage opportunity unless the price of the future were no more than S*(1+r)^T where T is the number of days until settlement. If one where a manufacturer which needed the commodity, then by postponing purchase from the cash to the future, one could save the carrying costs. Thus for some market participants, there would be an arbitrage opportunity unless the price of the future were less than S*(1+r)^T. Thus we would expect the futures price to exactly match this formula when there is no costs of execution and no volatility in the cash price.

To examine the examine the long-term affects of proportionally back-adjusting such a series, let us make the additional assumption that every delivery month is current for exactly 30 days. The adjusted series on day T will be price of the then-current delivery month (S*(1+r)^(T mod 30)) multiplied by an adjustment factor for each roll over which has occurred since. Each of those factors is the distant price divided by the receding current contract which is (S*(1+r)^(U+30))/(S*(1+r)^(U)) = (1+r)^30. Since there are T divide 30 such factors, the proportional series would be S*(1+r)^T which matches what an appropriately dated super-distant contract would have been.

This series has a downward slope even that the cash series isn’t. In this simple case, the difference between the initial high price and the actual cash price is the daily carrying costs for the whole length of the series.

Real life data is not as straight-forward. In our ideal case, the proportional spread between distant contracts is constant. Real life spreads have trends. Sometimes even in the opposite direction to carrying costs! Certainly there are cases of traders "playing chicken" at the end of a contract, but there are other causes such as new-crop/old-crop distinctions.

The Lean Hogs market, for example, actually goes highly negative when additively adjusted. This would seem to say that putting bacon in the freezer for 20 years will make you rich! Don’t believe it. I suspect that this may be caused by artificially inverted spreads possibly caused by floor traders knowing when an uninformed public is going to roll their positions.

There isn't any perfect way to remove these artificial trends from the data, but short of trading each delivery month separately, make sure to assume plenty of slippage when rolling your positions.