daniel, gilles and sornette, didier and wohrmann, peter - look-ahead benchmark bias in portfolio...

Look-Ahead Benchmark Biasin Portfolio Performance Evaluation

Gilles Daniel1, Didier Sornette1 and Peter Wohrmann2

1 ETH Zurich, Department of Management, Technology and Economics2 University of Zurich

Switzerland

[August 30, 2007][version 3]

Abstract: Performance of investment managers are evaluated in comparisonwith benchmarks, such as financial indices. Due to the operational constraint thatmost professional databases do not track the change of constitution of benchmarkportfolios, standard tests of performance suffer from the “look-ahead benchmarkbias,” when they use the assets constituting the benchmarks of reference at the endof the testing period, rather than at the beginning of the period. Here, we reportthat the “look-ahead benchmark bias” can exhibit a surprisingly large amplitudefor portfolios of common stocks – while most studies have emphasized relatedsurvival biases in performance of mutual and hedge funds for which the biasescan be expected to be larger. We use the CRSP database from 1926 to 2006 andanalyze the running top 500 US capitalizations to demonstrate that this bias canaccount for a gross overestimation of performance metrics such as the Sharpe ratioas well as an underestimation of risk, as measured for instance by peak-to-valleydrawdowns. [We suggest a simple practical approach for detecting the presenceof such a bias in one’s data, based on the construction of constrained randomportfolios.]

1

[August 30, 2007] [version 3]

1 IntroductionHow does one estimate the performance of a mutual fund, of a hedge-fund ormore generally of a financial investment? How does one quantify the quality of aninvestment strategy? A time-honored approach consists first in selecting the fundsand/or strategies of interest and then in quantifying their past performance oversome time horizon. A large literature has followed this route, motivated by theeternal question of whether some investors/strategies systematically outperformothers, with its implications for market efficiency and investment opportunities.

These backtests of investment performance may appear straightforward andnatural at first sight. However, a significant literature has unearthed, studied andtried to correct for two ex-post conditioning biases, which continue to pollute eventhe most careful assessments.

• The survivorship bias: many estimates of persistence in investment per-formance are based on data sets that only contain funds or stocks that are inexistence at the end of the sample period; see, e.g. (Brown et al. 1992, Grin-blatt and Titman 1992). The corresponding survivorship bias is caused bythe fact that poor performing funds/stocks are less likely to be observed indata sets that only contain the surviving funds/stocks, because the survivalprobabilities depend on past performance. Perhaps less appreciated is thefact that stocks themselves have also a large exit rate. For instance, Knaup(2005) examines the business survival characteristics of all establishmentsthat started in the United States in the late 1990s when the boom of much ofthat decade was not yet showing signs of weakness, and finds that, if 85%of firms survive more than one year, only 45% survive more than four years.Bartelsman et al. (2003) confirm that a large number of firms enter and exitmost markets every year in a group of ten OECD countries: data coveringthe first part of the 1990s show the firm turnover rate (entry plus exit rates)to be between 15 and 20 per cent in the business sector of most countries,i.e. a fifth of firms are either recent entrants or will close down within theyear. This phenomenon of firm exits is not confined to small firms. Indeed,in the exhaustive CRSP database of about 26’900 listed US firms, coveringthe period from Jan. 1926 to Dec. 2006 (Center for Research in Secu-rity Prices, http://www.crsp.com/), we find that on average 25% ofnames disappeared after 3.3 years, 75% of names disappeared after 14 yearsand 95% of names disappeared after 34 years.

• The look-ahead bias: the use of information in a simulation that would notbe available during the time period being simulated, usually resulting in anupward shift of the results. An example is the false assumption that earningsdata become available immediately at the end of a financial period. Another

Look-Ahead Benchmark Bias2

G. Daniel, D. Sornette and P. Wohrmann

http://www.crsp.com/


example is observed in performance persistence studies, in which it is com-mon to form portfolios of funds/stocks based upon a ranking performed atthe end of a first period, together with the implicit or explicit condition thatthe funds/stocks are still in the selected ranks at the end of the second test-ing period. In other words, funds/stocks that are considered for evaluationare those which survive a minimum period of time after a ranking period(Brown et al. 1992). This bias is not remedied even if a survivorship freedata-base is used, because it reflects additional constraints on ranking.

More generally, the fact that a data set is survivorship free does not imply that stan-dard methods of analysis do not suffer from ex-post conditioning biases, which inone way or another may use (often implicit or hidden) present information whichwould not have been available in a real-time situation. In certain scientific fieldsconcerned with forecasting, such as in earthquake prediction for instance, thecommunity has recently evolved to the recognition that only real-time procedurescan avoid such biases and test the validity of models (Jordan 2006, Schorlemmeret al. 2007).

Previous works have addressed both survivorship and look-ahead biases. Brownet al. (1992) have shown that survivorship in mutual funds can introduce a biasstrong enough to account for the strength of the evidence favoring return pre-dictability previously reported. Carpenter and Lynch (1999) find, among other re-sults, that look-ahead biased methodologies (which require funds to survive a min-imum period of time after a ranking period) materially bias statistics. ter Horst etal. (2001) introduce a weighting procedure based upon probit regressions model-ing how survival probabilities depend upon historical returns, fund age and aggre-gate economy-wide shocks, which provides look-ahead bias-corrected estimatesof mutual fund performances. Baquero et al. (2005) apply the methodology ofter Horst et al. (2001) to hedge-fund performance, which requires a well-specifiedmodel that explains survival of hedge funds and how it depends upon historicalperformance. ter Horst and Verbeek (2007) extend the look-ahead bias correctionmethod of Baquero et al. (2005) to hedge-funds by correcting separately for addi-tional self-selection biases that plague hedge-fund databases (underperformers donot wish to make their performance known while funds that performed well haveless incentive to report to data vendors to attract potential investors).

Here, we present a dramatic illustration of a variant of the look-ahead bias,which surprised us by its large amplitude. We are not dealing with mutual fundsor hedge-funds, for which survival biases and look-ahead biases have been sug-gested to lead to overestimating expected returns by as much as 8% per year. Wedocument a look-ahead bias in performance of portfolios investing solely in regu-lar stocks.




2 A classical exampleIn this context, the nature of the problem can be stated concretely as follows.Assume you want to back-test a given trading strategy on past data, namely on apool of stocks such as the constituents of the S&P500 index on a given period,say January 2001 to December 2006. To do so, the natural approach would be thefollowing:

1. Obtain the list of constituents of the S&P500 index at the present time (endof December 2006);

2. For each name (stock), retrieve the closing price time series for the givenperiod;

3. Backtest the strategy on that data set, for instance by comparing it with theS&P500 benchmark.

However, doing so introduces a formidable bias, and can easily lead to an er-roneous analysis. Figure 1 dramatically illustrates the effect by comparing theperformance of two investments.

In the first investment, which is subject to the look-ahead bias, we build $1 ofan equally weighted portfolio invested in the 500 stocks constituting the S&P500index at the end of the period (29 December 2006), and hold it from 1st January2001 until 29th December 2006. Meanwhile, the second investment simply con-sists of buying $1 of the S&P500 index on 1st January 2001 and holding it until29th December 2006. Both investments are buy-and-hold strategies and shouldhave yielded similar performances if the constitution of the S&P500 index hadnot changed over this period.1 However, the list of constituents of the S&P500index is updated, usually on a monthly basis, in order to include only the largestcapitalizations of the US stock market at the current time. Consequently, this listof constituents cannot, by definition, contain a stock that for instance crashed inthe recent past. Indeed, in such a case, the stock would have left the index andbeen replaced by another stock. The only difference in the two portfolios is thatthe first investment uses a look-ahead information, namely it knows on 1st Jan-uary 2001 what will be the list of constituents of the S&P500 index at the endof the period (29th December 2006). This apparently innocuous look-ahead biasleads to a huge difference in performance, as can be seen in Figure 1 and fromsimple statistics: the first (respectively second) investment has an annual averagecompounded return of 6.4% (resp. 2.3%) and a Sharpe ratio (non-adjusted for

1Since the S&P500 index is *not* equally weighted, we should expect a slight discrepancybetween the evolution of portfolio 2 and the actual index.




risk-free rate) of 0.4 (resp. 0.1). The first investment has significant better returnbut even more important smaller risks.

Many managers would be happier to report Sharpe ratios in the range obtainedfor the first investment. Investment strategies exhibiting this kind of performancewould fuel interpretations in terms of evidence of departure from the efficient mar-ket hypothesis and/or of arbitrage opportunities. On the other hand, other punditswould observe that this look-ahead bias is really obvious, so that no one wouldfall in such a trivial trap. This quite reasonable assessment actually collides withone simple but often overlooked operational limitation: the change of constitu-tion of financial indices are most often not recorded in most standard professionaldatabases, such as Bloomberg, Reuters, Datastream or Yahoo! Finance. As thestandard goal for investment managers is at least to emulate or better to beat someindex of reference, tests on comparative investments should use a defined set ofassets on which to invest defined at the beginning of the period. However, becausethe list of constituents of the indices is unexpectedly challenging to retrieve2, it iscommon practice to use the set of assets constituting the benchmarks of referenceat the present time, rather than at the beginning of the period [Ref]. Then, neces-sarily, the kind of look-ahead bias that we report here will automatically pollutethe conclusions, with sometimes dramatic consequences, as illustrated in Figure1. We refer to this as the “look-ahead benchmark bias.”

The rest of this note is devoted to a more systematic illustration of this look-ahead bias over different periods, from 1926 to 2007.

3 The extent of the Look-Ahead Benchmark BiasWe use the data provided by the CRSP database, from which we extract the closeprice (daily), split factor (daily) and number of outstanding shares (monthly) forall US stocks from January 1926 to December 2006. This represents a total of26’892 different stocks.

We decompose the time interval from January 1926 to December 2006 into 8periods of 10 years each. For each period, we monitor the value of two portfolios.At the beginning of each ten-year period, the first (resp. second) portfolio invests$1 equally weighted on the 500 largest stock capitalizations, as determined atthe end (resp. start) of the ten-year period. The first portfolio has by definitionthe look-ahead benchmark bias, while the second portfolio is exempt from it and

2Standard & Poor’s themselves provide the list of constituents of the S&P500 index onlysince January 2000, while scripting Reuters, Bloomberg and Datastream returned only incompleteresults. In fact, it appears that both the CRSP and Compustat databases are necessary to retrievethe list of constituents of the S&P500 index at any given point in time, and these databases areusually not accessible to practitioners.




could have been implemented in real time. Figure 2 plots the time evolution ofthe value of the two portfolios. The inserted panels show that the Sharpe ratio andcontinuously-compounded average annual returns are much larger for portfolio 1compared with portfolio 2.

Figure 3 shows the time evolution of the value of an investment consisting ofbeing long $1 in portfolio 1 and short $1 in portfolio 2. In other words, it showsthe ratio of the value of portfolio 1 divided by the value of portfolio 2, for eachof the 8 periods. By construction, this hedged long-short portfolio can be imple-mented only ex-post when backtesting, and not in real-time. Its performance isconsistently good 3 over the 8 periods from 1926 to 2006, with a huge reduction ofrisks and better returns than the unbiased, ex-ante portfolio 2, thus demonstratingthe significance of the look-ahead bias. This result is robust with respect to thenumber of stocks selected in the two portfolios.

The look-ahead benchmark bias documented here is related to the work of Caiand Houge (2007) who study how additions and deletions affect benchmark per-formance. Studying changes to the small-cap Russell 2000 index from 1979-2004,Cai and Houge (2007) find that a buy-and-hold portfolio significantly outperformsthe annually rebalanced index by an average of 2.2% over one year and by 17.3%over five years. These excess returns result from strong positive momentum ofindex deletions and poor long-run returns of new issue additions.

3Only the 1937-1946 period exhibits a rather smaller gain, albeit still positive with a significantreduction of risks.




4 Constrained Random Portfolios[Let us come back to the period from 1st January 2001 to 29th December 2006,shown in Figure 1 in order to test further the amplitude and robustness of the im-pact of the look-ahead benchmark bias. Investing on the 500 constituents of theS&P500 index at the end of the testing period (29th December 2006) amounts tobias the stock selection towards good performers. We illustrate that, as a conse-quence, non-informative random strategies exhibit very good to extremely goodperformance. We generate 10 portfolios, each of them implementing a randomstrategy. A random strategy opens only long positions (we buy first, and sell later)on a subset of the 500 stocks, with an average leverage of 0.8 and an averageduration per deal of 9 days – which are common values. Given these constraints,the choice of the selected stock and the timing are random at each time step. Wedo not specify further the algorithm as all possible specific implementations givesimilar results.

Figure 4 plots the time evolution of the value of $1 invested in each of theseten random portfolios on 1st January 2001. These random portfolios providean average compounded annual return of (9.1 ± 4)% with an excellent Sharperatio (with zero risk-free interest) of 2 ± 0.8. The random portfolios (solid greenlines) strongly outperform the S&P500 index (red dashed line), and simply diffusearound the look-ahead index (black dots) with an upward asymmetry. Similarresults are obtained with other parameters of the random strategies.]




5 Concluding remarksWe have reported a surprisingly large “look-ahead benchmark bias”, which resultsfrom an information on the future ranking of stocks due to their belonging to abenchmark index at the end of the testing period. We have argued that this look-ahead benchmark bias is probably often present due to (i) the need for strategiesor investments to prove themselves against benchmark indices and (ii) the non-availability of the changes of composition of benchmark indices over the testingperiod. More generally, it is difficult if not impossible to completely exclude anylook-ahead bias in simulations of the performance of investment strategies. Aswe reviewed in the introduction, one way to address these biases requires to firstrecognize their existence, and then to model how survival probabilities depend onhistorical returns, fund age and aggregate economy-wide shocks.

But, one can never be 100% certain that all biases have been removed. This iswhy other disciplines, such as in earthquake prediction for instance, have turned toreal-time testing. Actually, real-time testing is a standard of the financial industry,since cautious investors only invest in funds that have developed a proven trackrecord established over several years. But as shown in the academic literature,success does not equate to skill and may not be predictive of future performance,as luck and survival bias can also loom large (Barras et al. 2007). There is a largeand growing literature on how to test for data snooping and fund performance (seefor instance (Lo and McKinlay 1990, Romano and Wolf 2005, Wolf 2006)). Theproblem is more generally related to the large issue of validating models ((Sornetteet al. 2007) and references therein).

Practitioners should be careful to test for the presence of such a look-aheadbias in their dataset prior to backtesting their trading strategy.

[Here, we have proposed a simple and practical way to do so. Constrainedrandom portfolios, which imitate as closely as possible the properties of the trad-ing strategy about to be tested, such as its leverage, mean invested time and theturn-over, provide a straightforward approach to assess the probability that theperformance of a given strategy can be attributed to chance.]

ReferencesBaquero, G., J. ter Horst and M. Verbeek (2005) Survival, look-ahead bias, and

the persistence in hedge fund performance, Journal of Financial and Quan-titative Analysis 40 (3), 493-517.

Barras, L., O. Scaillet and R. Wermers, False Discoveries in Mutual Fund Per-formance: Measuring Luck in Estimated Alphas, working paper (March




2007)

Bartelsman, E., Scarpetta, S., Schivardi, F. (2003) Comparative Analysis of FirmDemographics and Survival: Micro-level Evidence for the OECD Coun-tries, OECD Economics Department Working Papers, No. 348, OECD Pub-lishing, doi:10.1787/010021066480.

Brown, S.J., W. Goetzmann, R. G. Ibbotson and S. A. Ross (1992) Survivorshipbias in performance studies, Review of Financial Studies 5 (4), 553-580.

Cai, J. and T. Houge (2007) Index Rebalancing and Long-Term Portfolio Perfor-mance, Available at SSRN: http://ssrn.com/abstract=970839

Carpenter, J.N. and A. W. Lynch (1999) Survivorship bias and attrition effects inmeasures of performance persistence, Journal of Financial Economics 54(3), 337-374.

Grinblatt, M. and S. Titman (1992) The persistence of mutual fund performance,Journal of Finance 47, 1977-1984.

Jordan, T. H. (2006) Earthquake Predictabillity: Brick by Brick, SeismologicalResearch Letters 77 (1), 3-6.

Knaup, A.E. (2005) Survival and longevity in the Business Employment Dynam-ics data, Monthly Labor Review, May 2005, 50-56.

Lo, A.W. and A.C. McKinlay (1990) Data-snooping biases in tests of financialasset pricing models, Review of Financial Studies 3 (3), 431-467.

Romano, J.P. and M. Wolf (2005) Stepwise multiple testing as formalized datasnooping, Econometrica 73 (4), 1237-1282.

Schorlemmer, D., M. Gerstenberger, S. Wiemer, D.D. Jackson and D.A. Rhoades(2007) Earthquake Likelihood Model Testing, in Special Issue on WorkingGroup on Regional Earthquake Likelihood Models (RELM), SeismologicalResearch Letters 78 (1), 17.

Sornette, D., A. B. Davis, K. Ide, K. R. Vixie, V. Pisarenko, and J. R. Kamm(2007) Algorithm for Model Validation: Theory and Applications, Proc.Nat. Acad. Sci. USA 104 (16), 6562-6567.

ter Horst, J.R., T.E. Nijman and M. Verbeek (2001) Eliminating look-ahead biasin evaluating persistence in mutual fund performance, Journal of EmpiricalFinance 8, 345-373.



http://ssrn.com/abstract=970839


ter Horst, J. R. and M. Verbeek (2007) Fund Liquidation, Self-Selection andLook-Ahead Bias in the Hedge Fund Industry, Review of Finance, 1-28,doi:10.1093/rof/rfm012.

Wolf, M. (2006) The Skill of Separating Skill from Luck, working paper Univ.Zurich (August).




Jan01 Jan02 Jan03 Jan04 Jan05 Jan06 Dec060.5

0.6

0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

Date

Por

tfolio

val

ue

ex−ante (0.1, 2.3%)ex−post (0.4, 6.4%)S&P500 (0.1, 1.2%)

Figure 1: Evolution, from January 2001 to December 2006, of $1 invested intwo equally weighted buy-and-hold portfolios made of the 500 constituents ofthe S&P500 index, respectively as in 1st January 2001 (ex-ante portfolio) and29th December 2006 (ex-post portfolio). For reference, we also plot the historicalvalue of the actual S&P500 index, normalised to 1 on 1st January 2001. Theperformance of the ex-ante and ex-post portfolios are reported in the upper leftpanel and measure respectively their Sharpe ratio (without zero risk-free interestrate) and continuously compounded average annual return.




0 500 1000 1500 2000 2500 30000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Trading days

Buy

−an

d−H

old

port

folio

val

ue

1927−1936: (0.0, 0.6%)1937−1946: (0.1, 1.4%)1947−1956: (0.8, 9.6%)1957−1966: (0.8, 7.6%)1967−1976: (0.4, 4.6%)1977−1986: (1.0, 12.7%)1987−1996: (0.9, 14.3%)1997−2006: (0.7, 12.4%)

(a) biased, ex-post pickup

0 500 1000 1500 2000 2500 30000

0.5

1

1.5

2

2.5

Trading days

Buy

−an

d−H

old

port

folio

val

ue

1927−1936: (−0.2, −5.0%)1937−1946: (−0.0, −0.1%)1947−1956: (0.6, 6.4%)1957−1966: (0.4, 4.0%)1967−1976: (0.1, 1.2%)1977−1986: (0.6, 7.6%)1987−1996: (0.5, 6.7%)1997−2006: (0.3, 5.5%)

(b) ex-ante pickup

Figure 2: For each epoch of 10 years, we plot the evolution of portfolio 1 (upperpanel) which invests $1 equally weighted on the 10 largest stock capitalizations, asdetermined at the end of the ten-year period and the evolution of portfolio 2 (lowerpanel) which invests $1 equally weighted on the 10 largest stock capitalizations,as determined at the start of the ten-year period. The inserts give the Sharpe ratio(with zero risk-free rate) and continuously-compounded average annual return.The discrepancy between these two figures helps us visualise the extent of thesurvival bias for the n=10 largest capitalizations throughout time.Look-Ahead Benchmark Bias

12G. Daniel, D. Sornette and P. Wohrmann


0 500 1000 1500 2000 2500 30000.8

1

1.2

1.4

1.6

1.8

2

2.2

Trading days

Buy

−an

d−H

old

port

folio

val

ue

1927−1936: (1.1, 5.9%)1937−1946: (0.7, 1.5%)1947−1956: (2.9, 3.0%)1957−1966: (2.3, 3.5%)1967−1976: (2.1, 3.4%)1977−1986: (2.6, 4.7%)1987−1996: (2.6, 7.2%)1997−2006: (2.0, 6.5%)

Figure 3: Time evolution of the value of an investment consisting in being long $1in a portfolio 1 equally weighted on the 500 largest stock capitalizations at the endof each period and short $1 in a portfolio 2 equally weighted on the 500 largeststock capitalization at the end of each period, with compounding the returns. Theinset shows the Sharpe ratios (with zero risk-free interest) and compounded annualreturn for the 8 periods.




Jan01 Jan02 Jan03 Jan04 Jan05 Jan06 Dec060.5

1

1.5

2

2.5

3

Date

Por

tfolio

val

ue

ex−ante (0.1, 2.3%)ex−post (0.4, 6.4%)S&P500 (0.1, 1.2%)Random

Figure 4: [As in Figure 1, with in addition 5 random portfolios.]



daniel, gilles and sornette, didier and wohrmann, peter - look-ahead benchmark bias in portfolio...

Documents

performance of mutual

past performance

time period

nancial period

lookahead bias

testing period

sample period

ranking period brown