How an index is built, and why the rule matters

An index is a rule, not a list

An index is a rule, written down in advance, for choosing a set of shares and deciding how much of each one counts. The level on the screen is whatever that rule produced today. Change the rule and the same market, on the same day, produces a different number.

That is easy to lose sight of, because an index is presented as a measurement — the temperature of the market, taken by nobody in particular. It is nothing of the kind. Somebody decided which companies were eligible, somebody decided what sets their proportions, and somebody decided how often those two decisions get revisited.

Three choices do almost all the work. Which companies are eligible at all — the universe. What decides how much of each the index counts — the weighting. How often the first two get re-run — the reconstitution. Nearly everything an index does that surprises people is traceable to one of those three.

There is a consequence that follows from all of them and that most readers never reach. A large amount of money is invested to track indices rather than to pick shares, so the rule does not merely describe the market; it moves money around inside it. An index committee's decision is itself a market event, and the shares it names move for reasons unconnected to the businesses.

The first choice: who is eligible at all

Before anything is weighted, something decides which companies are in the running. A rulebook typically screens on where the share is listed, how long it has been listed, how much of it is actually available to trade, and how easily a reasonable order can be filled without pushing the price around.

The last of those is the one worth pausing on, because it looks like a formality and is a judgement. A liquidity screen is there because an index that trackers cannot replicate is of no use to anyone. If a constituent cannot absorb the buying that tracking it requires, the cost of holding it swamps whatever it adds to the sample.

Which produces a result nobody intends and everybody inherits. A headline index is a list of buyable companies rather than a list of a country's most important ones. A large, profitable, closely held business with a thin traded float can sit outside an index that contains companies a fraction of its size, and nothing has gone wrong — the rule did what it says.

Two more universe decisions matter and are usually invisible. A sector index requires somebody to have classified each company into one bucket, which is a straightforward call for a cement maker and a contestable one for a conglomerate. And some rulebooks cap how much weight a single constituent or a single sector may carry, precisely because the weighting rule left to itself can concentrate the index in one name. Whether a particular index does any of this is in its published methodology, and it is worth reading rather than assuming.

The second choice: what decides how much of each

Now the choice that changes the answer most, and the one almost nobody looks up. Here is what it does, in arithmetic.

Take a toy index of three companies, purely as an illustration: A worth ₹8,000 crore, B worth ₹1,500 crore and C worth ₹500 crore, so ₹10,000 crore in total. Weighted by market capitalisation, A is 80% of the index, B is 15% and C is 5%. On one day, A falls 5%, B rises 5% and C rises 40%.

The market-cap-weighted index closes at ₹9,875 crore against ₹10,000 crore, so it is down 1.25% on a day when two of its three companies rose. An equal-weighted version of the same index, counting each company one-third, returns the average of −5%, +5% and +40% — up 13.3%. Same three companies, same day, same prices, and the two indices disagree by nearly fifteen percentage points.

Neither number is wrong, and this is the point. The market-cap version answers what happened to the money invested in these companies. The equal-weighted version answers what happened to the average company. Those are different questions, and the weighting rule is where one of them gets chosen on your behalf.

Indian index providers add one more adjustment, as a matter of construction practice rather than any rule imposed on them. Weights are computed on the portion of each company actually available to trade — the free float — rather than on every share in existence, because a tracker can only buy shares that are for sale. Suppose A has 20% of its shares floating, B has 60% and C has all of them: the float values are ₹1,600 crore, ₹900 crore and ₹500 crore, so the weights become 53.3%, 30% and 16.7%. A is more than five times B by market cap and now carries under twice its weight. The same day's moves through those weights return +5.5%.

Three answers — −1.25%, +5.5% and +13.3% — from three companies whose prices did exactly one thing. What the free float is and why it differs from the share count is set out in market capitalisation explained; what matters here is that the weighting rule is the measurement, not a detail of how it is presented.

Only the first two maintain themselves. If a company doubles, a cap-weighted index's weight in it doubles without anybody trading, because the weight is the price. Every other rule drifts away from itself as prices move and has to be traded back — a cost that recurs, and one that smart beta explained works through, along with the harder question of who chose the rule and on what evidence.

The third choice: how often the rule is re-run

Companies get larger and smaller, list and delist, and grow or lose their traded float. An index that never re-ran its own rule would drift into being a list of the companies that qualified once. So the rule is applied afresh on a stated cycle, and the results are announced before they take effect.

Between two of those runs an index is a photograph of a ranking that has already moved on. That is not a defect, and the alternative is worse: re-run the rule constantly and every tracking fund trades constantly, which their holders pay for. The trade-off is explicit — a stale sample against a trading bill, and every rulebook picks a point on it.

One structural consequence deserves saying plainly, because it is routinely mistaken for incompetence. An index adds companies that have risen into its band and removes companies that have fallen out of it. Ranking on size and re-running the rule periodically buys after the rise and sells after the fall, by construction. No manager decided this and no better manager could avoid it while still tracking the rank.

The related point is that a company's band can change without its own price moving at all — if enough companies below it rally past it, its rank slips while its shares close flat. That is the ranking argument set out in market capitalisation explained, and it is what makes the next section's effects possible.

Why a committee's decision moves a share price

Because every fund tracking that index then has to buy the share, and not one of them is buying because the business improved. The buying is set by a rulebook, and the rulebook does not care what the share costs on the day it must be bought.

An index fund or ETF exists to match its index. Its objective is not to buy well; it is to hold the constituents in the proportions the rule specifies, so that its return equals the index's return. That mandate has a consequence people underrate: the fund's demand for a share is price-insensitive by design. If the rule says hold it, the fund holds it at whatever the price is on the day it must.

So when a name is added to an index, every fund tracking that index has to buy it, and none of them is buying because the business improved. Take the illustration through one multiplication. A share entering at a 1% weight, with ₹10,000 crore of money tracking that index, generates ₹100 crore of buying that has to happen — a number set by the weight and the tracked assets, not by anything about the company. Removal runs the same arithmetic in reverse, and the share being sold is by construction one that has shrunk out of the band the rule tests for.

Both dates are public. The change is announced, and it takes effect later, so the entire market knows in advance which shares a large block of price-insensitive money must own and roughly when. Anyone willing to hold the share in the meantime is on the other side of that trade. The mechanical demand is not in dispute; whether it leaves a lasting mark on the price or largely reverses afterwards is an empirical question that different studies and different periods answer differently, and it should not be asserted either way.

Which produces the specific mistake to watch for, and it is a common one. An index addition reads like an endorsement — the company has arrived, somebody selected it. Nobody selected it. A rank was recomputed and a threshold in a rulebook was crossed, possibly because other companies fell rather than because this one rose. Treating an inclusion as a verdict on the business mistakes the scoreboard for the judgement.

The cost is not free-floating either; somebody pays it. A tracking fund that buys at the elevated price and sells at the depressed one is transacting on behalf of its own unit holders, and the bill turns up in the gap between the fund's return and the index's. That gap has a name and a published figure, and index funds and ETFs sets out how to read it — which is the practical reason a fund tracking a high-turnover rule can lag one tracking a quiet rule at the same expense ratio.

The level you see is not the return you would have earned

One more thing the rule decides, and this one is a straight arithmetic gap that costs people money in comparisons they make every year.

A price index counts only the prices of its constituents. When a company pays a dividend, its share price adjusts down on the ex-date by roughly the amount paid — the mechanics are in corporate actions explained — so the index records the fall. It does not record the payment, because a payment to shareholders is not a price. The money went somewhere; the index simply does not look there.

A total return version of the same index fixes that by assuming each dividend is reinvested back into the index on the day it goes ex. Same constituents, same weights, same rule — one series counts the dividends and the other does not. The one quoted in a news report or on a front page is the price version, almost always, unless it says otherwise.

Over a year the difference is roughly the index's dividend yield, which sounds ignorable. It is not, because it compounds. Take a yield of 1% as an arithmetic input rather than as a figure for any particular index: twenty years of it is 1.01 to the twentieth power, or 1.22. The two series end 22% apart having tracked precisely the same companies throughout. Double the yield and the gap over the same span is about 49% rather than the 44% that doubling suggests, because the compounding is not linear either.

Now the mistake, which is made constantly and in one direction. A mutual fund's NAV already includes the dividends the scheme received on its holdings. Compare that NAV against a price index and the fund is being credited with dividends its benchmark was denied, so it looks better than it was by an entire dividend yield, compounded over however long the comparison runs. Which version a scheme is measured against is stated in its own documents, and it is one of the few lines in there worth looking up.

Four ways a rule gets misread

Each of these follows from taking the index as a measurement rather than as a rule, and each sounds perfectly reasonable while being wrong.

1. Reading the index as the market, when it is a sample chosen by an eligibility screen and weighted by one particular convention. It moved 1.25% down in the worked example above while two of its three companies rose. The market and the sample can differ for a whole session and neither one is lying.

2. Comparing two indices built differently. An equal-weighted index and a cap-weighted one over the same constituents will separate over any period, because they are asking different questions. Reading the gap as evidence about the companies, rather than about the two rules, is the error.

3. Treating inclusion as a quality signal, when a committee applying a size and liquidity rule has assessed a rank rather than a business. The buying that follows is real and is caused by the rule; that is a very different thing from the buying being informed.

4. Benchmarking against the price series. This is the quiet one, because nothing about it looks like an error. A comparison of anything that receives dividends against an index that ignores them is not a comparison, and the gap widens every year the comparison runs.

The single question that clears all four: what does the rule actually say? An index's methodology is published, and the three things worth reading in it are what makes a company eligible, what sets its weight, and how often both are redone. What the level did today is on every front page. Why the level is what it is sits in the rulebook.

Looking at the constituents rather than the level

Everything above reduces to one habit: treat an index level as the output of a rule and go looking for the inputs. The constituents, their sizes, their traded floats and the sectors they sit in are all published, and each of them changes what the level means.

FNOTrader's Stocks app runs on roughly 2,390 stocks and 17 NSE sector and size indices, which is also the universe its rank backtests run over. Ranking that population by size and re-running the ranking on a schedule is the operation an index performs, so it can be examined on real constituents rather than taken on trust.

None of that says what any index should contain. It says what the current one does contain and on what basis, which is the question that has to be settled before its level means anything.

Common questions

Continue reading

More in Investing Basics · App: Stocks · Definitions: glossary · Free tools: calculators · All: every article