Regression to the Mean and the Markets

In this article I will talk about regression to the mean, a very useful and practical but not much understood concept for stock market players.

Regression to the Mean

Vishnu has some money that he would like to invest in a mutual fund. So he does some research on the last few years’ performance of some of the top performing funds. After thorough analysis, he selects the best performer..

Over the next few years the fund is barely able to match the benchmark and in many instances actually under-performs it.

Vishnu pulls out all his money and finds another fund manager based on his earlier selection criteria of recent high performance. He once again puts his money in this fund and immediately the fund performance deteriorates.

He scratches his head and introspects on his stock-picking ability. He is clueless as to what is going on.

That is because Vishnu has been caught squarely in the middle of a regression to the mean.

Games Statistics Play

Regression to the mean, also known as mean reversion is a statistical concept where an event that is not an average will more than likely be followed by an average event. It makes natural random variation in data appear to be real and more often than not people attribute a causal link to this random variation.

Daniel Kahneman in his seminal book “Thinking, Fast and Slow” gives an example of this phenomenon. He gives the example of a sportscaster who is seemingly able to provide causal explanation of the performance of participants during the second jump in a men’s ski jump event.

The commentator had noticed, that a good jump was more often than not followed by a poorer one, and a poor jump was followed by a better one. He attributed a better jump to a relaxed state after a bad first jump, and a poor jump to the tension associated with protecting the lead.

When seen in terms of stock markets, prolonged periods of out-performance will be followed by periods of under-performance and periods of below average performance will be followed by a bull run. For short periods, markets can and will do almost anything and swing between extremes, but over an extended period of time markets tend to even out.

Why Regress

Regression to the mean is a statistical phenomenon. It happens because of two reasons.

The first reason is the nature of the sample Vishnu has chosen to arrive at a decision. From a pool of over perhaps 5,000 mutual funds, he selected the top 10 best performers. And it was based on this elite list of over-achievers that Vishnu arrived at a buy decision

The top 10 funds are there at the top because as a group they have outperformed the market in the last few years. The same fund may not be a top performer every single year and it is more than likely that the top 10 list will vary from year to year. The average returns of the top performing funds as a group are significantly greater than the market average of all the 5,000 or so mutual funds. The out-performance could be on account of any number and combination of factors and events which one can at best surmise only in retrospect.

Having based his investment decision on the top performers, Vishnu has allowed himself to become more susceptible to the unpredictability of statistical fluctuations. Because an important rule of regression to the mean is that the more extreme the sample group, the greater is the mean reversion. This is because an extreme group (the lowest or highest %) will have a sample mean that is much further than the population mean, and therefore has a greater likelihood of regression towards the population mean.

If however any specific fund manages to stay in the top for a prolonged period of time, then the equation could change slightly. But the inevitability of regression cannot be altered. We will talk more about this in a while.

The second reason for regression to the mean is the degree of correlation between any two variables. Vishnu has tried to establish a causal link between a fund’s performance in a specific year with the expected future performance of the fund..

Generally when two variables are perfectly correlated, there will not be any mean reversion. In real world, there is nothing called perfect correlation, there will always be some random error. The less correlated the two variables, the greater would be the regression to the mean.

A schematic formula, based on the framework laid down by Daniel Kahneman for the factors which determine past performance and expected future performance:

past performance = shared factors + previous years specific factors
future performance = shared factors + luck

The shared factors could include things like fund managers stock-picking ability, asset allocation, regulatory tailwind, bull market-like condition or any number of factors. While in retrospect we can determine the previous years specific factors, there is simply no way to ascertain what the future years specific factors could be. Hence it comes under a broad heading called luck.

Given the imperfect correlation between these variables, regression to the mean is inevitable.

What about Berkshire Hathaway?

As mentioned earlier, if a fund has been a top performer for a prolonged period, we can use a modified version of the above schematic formula.

performance = high stock-picking ability + shared factors + lot of luck

This kind of equation is particularly true of Warren Buffett and the performance of Berkshire Hathaway. Berkshire Hathaway has systematically beaten the markets decade after decade by 9.6 percentage points on a cumulative annualized basis.

Yet if one inspects the directionality of the variance between Berkshire Hathaway and the markets the trend is unmistakably downwards. Berkshire is slowly but surely regressing towards the mean (the markets). Complete regression to the mean may not happen overnight, but will definitely happen some day. Nothing, including Berkshire is immune to mean reversion.

The delta between Berkshire’s return via-a-vis S&P returns (including dividends) peaked in 1982 at 15.2% on a cumulative basis. Since then it has slowly but steadily come down to 9.6% in 2014 (which is still a phenomenal number).

So yes. Regression to the mean does not spare even the superheroes.

Final Words

Regression to the mean is a fundamental truth. When Sir Francis Galton, the British Victorian polymath, discovered this phenomenon and explained his findings to the general public, most people were not able to wrap their head around the core tenets of mean reversion or its implications. Sadly it is still the case today.

A very obvious implication with respect to the stock market is that one should not make predictions based on short term trends, like what Vishnu did, as it will invariably gravitate towards the mean. The more extreme the values, the more rapid will be the movement towards equilibrium. A period of 10 years is usually enough to allow the values to settle into a rhythm.

An investor should instead focus more on the process than on the outcome. As long as a strategy is sane, rational and has some merit, one should not get too much bothered by short term variations.

Regression to the mean is an inevitability which cannot be staved off or wished away. An investor who is able to separate true causation from spurious statistical causality and is able to stick to a well-defined rational long-term investment strategy will be more likely to have a peaceful and rewarding investment career.

You may also like


Subhodeep Mukhopadhyay

I am a Management Consultant in the Education Sector. In my previous corporate career, I have worked in Banking, Private Equity and Software industry. I am an MBA in Finance/ Computer Engineer and enjoy doing equity research and financial analysis in my free time.

2 thoughts on “Regression to the Mean and the Markets

  • April 2, 2016 at 8:20 am

    Your analysis is very much helpful and informative. Request you to let me know whether Regression to mean article is available in hindi or gujarati.


Leave a Reply

Your email address will not be published. Required fields are marked *