Here's Why T-Mobile's John Legere Thinks a Popular Study Is Wrong

Twice a year, wireless industry journalists breathlessly cover the results of the biannual RootMetrics Mobile Network Performance in the United States report.

As soon as the study is released, various news outlets jump on the numbers, and those stories tend to take a pretty surface view of the data. Sometimes it's as simple as "Verizon Wins Again" or maybe the headlines get nuanced enough to point out that AT&T has worked its way to a close second whileT-Mobile andSprint have improved.

Some media -- myself included -- have even taken a bit of a suspicious eye to the numbers, digging in a little deeper to offer context and some have even cited a competing report from OpenSignal that shows T-Mobile leading the way, trailed by Verizon, Sprint, and, lastly, AT&T. [Here's my take on the latest RootMetrics report.] But, in a broad sense, RootMetrics has been accepted as a viable standard by most, with Verizon, of course, embracing it enough to base its "A Better Network As Explained By Colorful Balls" ads on its data.

RootMetrics, however, does have one major critic. T-Mobile's John Legere has been a vocal opponent of the validity of the research and, in a recent email to the media, he said that the company "should be banned as an independent source for network benchmarking, period!"

That may sound like a bitter reaction from someone who leads a company that has consistently scored at the bottom of the report, but Legere backs up his hyperbole with some reasonable assertions.

What is Legere saying?Basically, Legere takes issue with the choices made by RootMetrics in how to test his company's network. Here, he charges that those decisions tilt the results in favor of its rivals:

Legere also questions exactly how independent the research firm is, asking "Do we have to pay RootMetrics millions like the other carriers do to get them to stop deliberately turning off significant portions of our network and skewing results during their drive tests? Coincidence? Doubt it." RootMetrics says anyone can purchase its highly detailed data, but that is not connected to its results.

Legere then invites people to email him(though no specific email is provided) for "more examples of their incomplete methodology."

T-Mobile CTO Neville Ray added that the current report is based on data that is seven months old while the Verizon ad is based on year-old information.

Verizon topped the report in overall performance again. Data source RootMetrics.

How did RootMetrics respond?In a broad sense, RootMetrics did not really disagree with what Legere had to say. It explained its logic in an email to The Motley Fool from Vice President Julie Dey, who answered a request to respond to Legere's remarks with statements including:

Basically, for better or for worse, RootMetrics said it conducts its test with devices benchmarked at the beginning of a six-month testing period then sticks with that until the next report. "Switching our methodology midstream would render a scientific apples-to-apples comparison impossible," Dey wrote.

She noted that voice-over-LTE testing will be part of the first half of 2016 testing that is currently under way."We are taking a wait-and-see approach to what the impact may (or may not be) for T-Mobile," Dey added.

The RootMetrics VP also responded in a second email exchange with theFool to Legere's charges that the other carriers pay the company. She made it clear that the surveys are done independently but wireless carriers are able to purchase access to the in-depth results.

That does not mean Verizon, Sprint, and AT&T in any combination are not RootMetrics customers. They may well be and they may spend millions as Legere claimed, but Dey is responding by laying out her case that the customer base does not dictate the results of the survey.

It's a matter of perceptionLegere has an issue with testing methodology and RootMetrics has responded in a way that can politely be described as "We see your point, but that's how we chose to do it," or if you take the impolite interpretation, "Stuff it." The CEO is not wrong in pointing out the age of the data and the research firm does not dispute. It's simply how it chooses to do the report.

The same could be argued on the issue of bias. If Legere considers the methodology unfair, which he clearly does, then he may draw the conclusion, even though he doesn't present evidence to back it up, that the research firm has allowed itself to be influenced by its paying customers -- which he presumes to include the other wireless carriers (or at least some of them).

This is a dispute where both sides appear right. Legere has valid points, but so does RootMetrics. The real issue is how consumers, the media, and even investors interpret the results. If you look at them through Legere's lens, then you see something very different than what the survey says and how these results get reported.

The article Here's Why T-Mobile's John Legere Thinks a Popular Study Is Wrong originally appeared on Fool.com.

Daniel Kline has no position in any stocks mentioned. He is a T-Mobile customer who kind of misses wireless phones with physical keyboards. The Motley Fool recommends Verizon Communications. Try any of our Foolish newsletter services free for 30 days. We Fools may not all hold the same opinions, but we all believe that considering a diverse range of insights makes us better investors. The Motley Fool has a disclosure policy.