Vanguard’s study, “Joined at the Hip: ETF and Index Development” found that these estimates are often unreliable. While 87% of the indices outperformed the broad U.S. stock market for the time in which back-tested data were used, only 51% did so after the index was launched.

Past performance data is not necessarily indicative of future results, but investors are making investment decisions based on this nonetheless. Joel Dickson, one of the study’s authors and a principal in Vanguard’s Investment Strategy Group, said that he thinks this decisionmaking strategy is a mistake.

“I think there’s too much rearview mirror investing,” Dickson told PLANADVISER. “Nobody would suggest driving your car with a rearview mirror because you’re going to run into the wall that’s right in front of you.”

In investing, that wall is underperformance, which many ETFs experience after an index goes live.

The performance of the 370 indices against the total U.S. stock market might look strong at an annualized excess return of 10.31%—but that percentage is based on back-filled data. After the index-live date, the performance of the indices was an annualized excess return of -0.93%—less than that of the broad U.S. stock market.

This raises the question: Why does the performance before the index-live date look so strong?

The reason for this is ETF makers are “cherry-picking” the indices that have recently performed well to track.

"There’s an inherent selection bias,” Dickson said. “Nothing sells like past performance.”

In addition, providers are increasingly releasing ETFs soon after indices are launched, leaving little time to collect live data. Vanguard found half of the1,000-plus indices studied  were less than six months old before an ETF was launched to track them. Dickson said this leads him to believe there might be a relationship between ETF and index creation.

“Indexes are being created with an ETF in mind,” Dickson said. “That is, an ETF sponsor actually may ultimately be the genesis of the index idea and then talks with the index provider who will brand it and license it.”

More than half the indices use back-tested performance data in lieu of or in addition to live-performance data. Naturally, ETF providers only choose market funds that have done well according to back-tested data.

“As any plan sponsor will attest, whether it’s past live performance or back-tested, you never see bad historical information,” Dickson said. “You only see that from your existing managers—never a prospective manager.”

But an index’s positive past-performance can lead investors to assume it will also perform well in the future. Dickson said this is not the case.

“The winners in the past are just as likely as any strategy to be winners or losers in the future,” he said.

Dickson said differentiating back-filled data and live date can be a challenge. In the study, researchers found that information through footnotes on index fact sheets on providers’ websites

“Even then it wasn’t always clear when the live date was versus any back-filled data,” Dickson said.

Different Back Test Rules 

While securities regulations prohibit the use of back-tested performance for most U.S. mutual fund and ETF providers, index providers are not held to the same rules.

Asked whether mutual fund and ETF providers should be given the same rules, Dickson said, “I’ll leave it up to the regulators to decide what or even if they’re able to do anything about it.”

Dickson suggested that before investing in an ETF, investors should observe the long-term endurance of the investment and see if it fits into their investment plans. Also, they should not view indices as objective and unbiased representations of market segments.

“Instead, recognize that indexes are being created because those areas have performed well and that facilitates the marketing and development of new investment products—largely ETFs—that track those areas,” Dickson said.

Investors have large investments in ETFs. Currently, more than $1.2 trillion is invested in about 1,400 U.S.-listed ETFs as of March 31, according to Strategic Insight’s Simfund, an Asset International company.

Dickson said ETFs are projected to grow at double the rate of traditional mutual funds.

Click here for the full report.