Debunking a Debunking: Our Response to the NetsDaily

If the true purpose of the Emory Sports Marketing Analytics blog is to generate conversation, then our NBA brand equity study has been a success.  The reality is that we are a couple of sports fans that happen to be university professors.  I (Mike) spend most of my time developing dynamic models that predict customer behavior over time.  This blog is an effort to combine our jobs and hobbies.  An example of this is a paper on competitive balance in MLB.

Our blog is intended to present more quick hitting analyses of current issues in the world of sports.  With these studies we rely on publicly available data, and we tend to use relatively straight-forward statistical methods.  We really don’t begin with any agenda; we just let the numbers speak.

In the case of the NBA fan study, the numbers spoke and then the NetsDaily answered.  Given our love of debate, we can’t help but answer back.  Though all kidding aside, to the Nets/NetsDaily guys: we really do like what you guys have done.  You are obviously an exciting franchise with a lot of great marketing.

In what follows we reprint the NetsDaily notes in blue and then give our comments in bold:

Final Note: That Emory University study

We’re not going to devote a whole lot of effort to debunking the silly Emory College attempt at defining passionate fans by analytic means.  It found that the Nets were the worst home fans in the NBA and the Knicks the best.

Okay, we get the idea, the NetsDaily is not happy with the results.  A couple of things come to mind.  One, Emory is a University.  Second, debunking is probably the wrong way to start a discussion.  With almost any study like ours, there will need to be assumptions made.  As we noted in our original post,we are using a revenue premium brand equity model.  One could definitely argue that passion and revenue are different concepts.  It is probably more useful to understand what the study is saying than to claim it is wrong.

We would just like to point out some serious flaws in the study. The original study was so devoid of data that the authors were asked to provide some, which they did in a self-congratulatory addendum.

We are trying to strike a balance in the blog.  We could report all statistical models, but we are trying to keep things interesting so we emphasize intuition and report the interesting findings.  We are more than happy to share additional details.  And as academics, self-congratulation is probably our best hope for some positive feedback.

The authors note that key data they used to derive their conclusions is something called “home revenue.” They attempt to estimate “home revenue,” which is a finite, known but proprietary figure not available to them. Shouldn’t they note that, explain its relevance?

“The analysis begins with a model of box office revenue based on variables that correspond to market potential (capacity and market population), team quality (winning percentage) and entertainment value (number of all stars, payroll). The insight or theory that drives the analysis is that this model can be used to predict the revenue that is due to quality and market potential. Any difference between this predicted value and actual value is due to ‘fan loyalty’.”

So the reality is that they don’t have the finite, known but proprietary information that is the core of the theory so project it based on other data, including things as spurious as number of all-Stars, but ignore other data that might be important, like say MERCHANDISE SALES. Need we go there? The Nets now rank fourth in NBA merchandise sales. In the first several months after the merchandise was introduced, they ranked first.

The NetsDaily does not seem to properly understand the analysis.  We use a really simple estimate of home box office revenue using the popular Fan Cost Index and attendance reported by ESPN.  Is this the ideal way to determine home revenue?  Of course not!  Is it a reasonable way given that teams are private and do not report detailed financials?  Reasonable might even be too strong.  In fact, let’s say that it is a crude way to compute box office revenues. (Point for the Nets)

We then use this revenue measure as a dependent variable in a regression model that uses the previously mentioned factors.  We are not estimating revenue in terms of things like number of all-stars, we are developing a model that explains revenues by these factors that indicate team quality, market size and entertainment value (fans come out to see all-stars).  We then compare the difference between the predicted and the (crude) estimate of actual revenue.

The idea is to look at attendance AFTER controlling for how well the team did.  Does Miami selling out mean much given the quality of the team on the court?  Our goal is to really get at the true core support for a team.

On comparing the attendance between the Knicks and Nets, they use gross numbers of attendance.

“The teams share the largest population metropolitan areas but the Knicks achieve a 10.7% advantage in terms of attendance DESPITE charging much greater prices. It is this greater pricing power that pushes the two teams to opposite ends of the ranking.”

Suppose instead of gross numbers, they used capacity percentage. The original analysis appears to rely on ESPN attendance percentages, that is, the percentage of arena capacity sold out on average each game. We say “appears” because the original article notes, ” A quick look at attendance data from ESPN shows that the Trail Blazers regularly exceed capacity for entire seasons.” Two points: the Trail Blazers attendance last season was 95.4 percent, which did not exceed capacity (or it would have been in excess of 100 percent.)

This is an example of why it’s probably better to ask questions and have a discussion than to go on the attack.  We control for stadium capacity in the revenue prediction model to account for differences in stadium capacity.

But they do make a good point here.  When we talk about “best” fans there really is no obvious metric.  We choose revenue, the Nets suggest capacity utilization.  Also, the Trail Blazers attendance percentage was 102.6 percent of capacity in 2012, 102.7 percent in 2011, and 102.6 percent in 2010.  We believe that this is very consistent with our wording.

However, using ESPN data presents problems. It is inaccurate regarding the Nets. ESPN uses an NBA capacity of 18,000 for Barclays Center. That was the original number for NBA games. As the arena was completed, capacity was reduced to 17,732 (apparently to accommodate loge seating added late in construction.) The number can be found on the Nets website. So the actual percentage of seats sold this season is 96.9 percent, not 94.9. That would put the Nets at tenth (not 16th) in the NBA, just ahead of … drum roll … the Knicks at 96.3 percent and the Blazers.

Again, some good points are raised here by the NetsDaily.  As statisticians, we love to have very accurate data.  In reality, data almost always contains some noise or error.  The Nets would have a valid complaint if the publicly available data was somehow consistently biased against the Nets.  Does Team Marketing Report systematically underestimate the Nets’ prices, and does ESPN systematically underestimate the Nets’ attendance?  We don’t know.  If so, we apologize.

We leave it to the readers to decide whether our measure which includes quantity and prices is preferred to straight capacity utilization.

But let’s put aside the methodology and data and look at the final product, which suggests below average work.  Is there ANYONE in the NBA who believes that fan loyalty to the Dallas Mavericks, Phoenix Suns, and Orlando Magic is on the rise … or that their fans had greater loyalty than the teams that follow them in the Emory rankings: the Miami Heat and San Antonio Spurs??? You want an example of analytics gone wild???

The comment above seems to be a common misinterpretation.  The example of the Orlando Magic is really what is at the core of our study.  We are not saying that the Orlando Magic has more fan support than the Miami Heat this season.  We are saying that after you control for the difference in the quality of the teams it appears that the Orlando Magic have a more devoted fan base.  Over the past season, the Magic had an average home attendance of 17,595, while only winning 24% of their games.  The key question (and what we use the statistical models to get at) is what would Miami have drawn if they didn’t win 80% of their games and have 3 all-stars in the lineup?

If we were petty people we would also point out that the ESPN attendance figures for last season report that the Magic drew 721,414 fans while the Nets operating in the largest metropolitan area and winning about 60% of their games drew 704,702.  We usually aren’t petty, but they did call our study “silly”, and called us “C students” (Manish says thanks for the passing grade) on Twitter.

The study is also a rare, rare instance where Barry Baum, chief communications officer of the Nets, and Norman Oder, the leading critic and chronicler of Atlantic Yards find common ground.

Says Baum, after reading the original article, “With all due respect to Emory University, that is a seriously flawed study.”

Thank you Mr. Baum for getting the school name correct.

Says Oder, after reviewing the article and supporting data, “The Knicks’ attendance edge is magnified by an arena with greater capacity, and the willingness of Knicks fans to pay more. [It] has less to do with passion than a longstanding monopoly position in a large market.”

This is also a good point, and one that we acknowledged.  It seems likely that the Knicks may benefit from locational advantages that translate into greater pricing power.

If there is a continuing dispute on this, we suggest a review by Nate Silver, the New York Times stats guru … and Nets fan. We are sure he will get to the bottom of it.

We would of course very happy to extend the discussion.  We were in fact surprised when we ran the numbers, and found the Nets on the bottom.  And remember we did point out that the Nets were rapidly improving.

Mike Lewis and Manish Tripathi, Emory University 2013.