NFL Fan Equity: Method Limitations and Focus on the Falcons

Our analyses frequently generate criticism.  Our work has been described as “garbage,” “silly” and “annoying” (and this is just from Mike’s wife).  To us, one of the most interesting things about this project is that we are often surprised by whom we offend.  In the case of last week’s analysis, we were humored by the fact that Saints fans seemed equally interested in their 4th place ranking and the Falcons’ 31st place ranking.  Given that we are based in Atlanta, we thought it would be a good idea to discuss why the Falcons finished so low and, more importantly, how these results should be interpreted.

Our starting point in these analyses is that we are evaluating fandom from a marketing perspective.   This means that we are trying to identify which customer base is the most loyal in terms of their willingness to support their team through buying tickets.  This may seem like a crass measure to some, but it is at least an objective and observable metric.  Most critics seem to want us to somehow read the minds of the fans, and make ratings based on “passion.”  This is a fine notion but the implementation is somewhere between difficult and impossible.  Difficult, because a large scale survey would be needed to ask fans questions about how passionate they are, and nearly impossible because the survey would need to be repeated year after year to control for variation in team quality.

Our method, like all methods, has some limitations.  In our case, two limitations are most notable.  First, we rely on publicly available data (FCI pricing data, ESPN attendance estimates, Forbes’ team value estimates, US Census data, Title IX reporting data, etc.).  Publicly available data (and private data) will always contain inaccuracies.  The real question is whether the publicly available data is inherently biased against certain teams or types of teams.  We are happy to listen to debate about this issue.

The second limitation relates to a team’s marketing objectives.  One issue in sports marketing is that we do not get to observe true demand due to the constraints imposed by stadiums with finite capacities.  For this reason, we primarily rely on estimates of revenue.  This is an important distinction because it means that we implicitly make an assumption about how teams price.  The implicit assumption is that teams are attempting to maximize revenues.

You can definitely criticize this assumption.  This assumption comes into play when evaluating teams that regularly sellout (e.g. Green Bay).  How can these fans be any more loyal?  This leads to the question of why don’t teams like the Packers price higher.  I can think of a couple of potential answers.  One, perhaps they don’t have enough information or expertise to maximize revenues.  Demand forecasting for an NFL stadium is a non-trivial task.  Historical data is of limited use because demand for certain types of seats is censored.  The variation in the quality of tickets is also a problem as revenue maximizing teams would also need to understand the cross-elasticities across ticket types.

But the salient question is: if not a revenue maximizing assumption then what?  The best answer, we believe, is that some teams may systematically underprice in order to build or invest in their customer base.  The logic is that because the team lacks an extended tradition of success or that the team competes locally with other sports offerings, it makes sense to charge below market rates to get people into an exciting, sold-out stadium.  Of course, as more astute readers may have noticed, this explanation is also consistent with the story that the team lacks brand equity.  We could also make arguments that some team price too high and may therefore be “harvesting” brand equity.

This brings us back to the case of the Atlanta Falcons.  The explanation for why the Falcons finished low despite recent success on the field and in terms of sellout attendance is because they price lower than would be expected.  According to the Team Market Report’s fan cost index, over the last decade the Falcons have tended to price below the league average.  But it isn’t sufficient to just consider relative prices.  We also need to consider the “quality” of the market.  The Atlanta metro area has population and median income levels that are well above the league averages.

The other issue that was mentioned locally is: what does this mean for the Falcons’ quest for a new stadium?  A case can be made that our findings support the need for a new stadium. If we believe the assumption that professional sports are an important civic asset (because they draw attention, create economic value, enhance the culture, etc.) then it makes sense for the city to invest in the team.  The Falcons’ have a relatively short history, and play in a city full of transplants.  Just as the Falcons may be underpricing in order to develop their fan equity, it may make sense for the local community to also invest back into the team.

Click here for an alternative methodology for ranking fan bases that relies on social media data.

Mike Lewis & Manish Tripathi, Emory University 2013.

“Revenue Premium” Versus Survey-Based Attitudinal Measures

A criticism of our previous rankings of fan bases is that our approach is overly financial and doesn’t capture the “passion” of fans.  This critique has some validity but probably less than our critics realize.  When we talk about quantifying customer loyalty in sports or even in general marketing contexts we very quickly run into some challenges.

For example, when I speak to classes about what loyalty means, the first answer I get is that loyal customers engage in repeat buying of a brand.  I will then throw out the example of the local cable company.  The key to this example is that cable companies have very high repeat buying rates but they also frequently have fairly unhappy customers.  When asked if a company can have loyal but unhappy customers students quickly realize that it is difficult to cleanly measure loyalty.

Another distinction I make when teaching is the difference between observable and unobservable measures of loyalty.  As a marketer, I can often measure repeat buying and customer lifetime.  I can even convert this into some measure of customer lifetime value.  These are observable measures.  On the other hand other loyalty oriented factors such as customer satisfaction, preference or likelihood of repurchase are unobservable, unless I do an explicit survey.

I think what our critics are getting at is that they would prefer to see primary / survey data of customer preference or intensity (questions such as on a 1 to 7 scale rank how much you love the Florida Gators).  BUT, what our critics don’t seem to get is that this type of primary data collection would also suffer from some significant flaws.  First, whenever we do a consumer survey we worry about response bias.  The issue is how do we collect a representative sample of college or pro sports fans?  This is an unsolvable problem that we tend to live with in marketing since anyone who is willing to answer a survey (spend time with a marketing researcher) is by definition non-representative (a bit weird, I know).

A second and more profound issue is that it would be impossible to separate out the effects of current season performance from underlying loyalty using a survey.  I suspect that if you surveyed Michigan basketball fans this year you would find a great deal of loyalty to the team.  But I think we all know that fans of winning teams will be much happier and therefore respond much more positively during a winning season.

Related to the preceding two issues is that our critics seem to assume that they know what is in the heart of various fan bases.  Mike Decourcey took exception with our college basketball rankings that rated Louisville over Kentucky and Oklahoma State over Kansas.  A key mistake he makes is that he assumes that somehow he just knows that Kentucky fans are more passionate than Louisville’s, or that Kansas fans love their team more than Oklahoma State loves theirs.  He knows this based not on any systematic review of data, but based on a few anecdotes (this is especially convenient since the reliance on anecdotes means that there is no need to control for team quality) and his keen insight into the psyches of fans everywhere.

The other issue is whether our “Revenue Premium” captures fan passion or just disposable income.  This is another impossible question to fully answer, but in our defense the nice thing about this measure is that it is observable, and willingness to pay for a product is about the best measure of preference you can get short of climbing into someone’s head.  I think another way in which our critics are confused is that they associate noise with loyalty.  Is an active and loud student section a true measure of the fan base quality?  Perhaps so, but do we really believe that the 19 year old face painter is a better fan than the alumni who has been attending for 40 years but no longer stands up for the entire game?

Converting High School Talent into NBA Draft Picks: Ranking the ACC

The NBA Draft can be a time for college basketball fans to cheer about the “success” of their basketball program.  Kentucky, Duke, North Carolina, and Kansas fans can boast about the number of alums currently in the NBA.  This year, ESPN is taking that discussion one step farther by describing the quality of NBA players produced, and ranking the “NBA Pedigree” of colleges.

Our take is a bit different as we will examine the process of taking high school talent and converting it into NBA draft picks. In other words, we want to understand how efficient are colleges at transforming their available high school talent into NBA draft picks? Today, we launch our NBA draft series by ranking the schools in the ACC based on their ability to convert talent into draft picks.

The initial approach is fairly simple.  Each year, (almost) every basketball program has an incoming freshman class.  The players in the class have been evaluated by several national recruiting/ranking companies (e.g. Rivals, Scout, etc…).  In theory, these evaluations provide a measure of the player’s talent or quality*.  Each year, we also observe which players get drafted by the NBA.  Thus, we can measure conversion rates over time for each college.  Conversion rates may be indicative of the school’s ability to coach-up talent, to identify talent, or to invest in players.  These rates may also depend on the talent composition of all of the players on the team.  This last factor is particularly important from a recruiting standpoint.  Should players flock to places that other highly ranked players have selected?  Should they look for places where they have a higher probability of getting on the court quickly? Next week we will present a statistical analysis (logistic regression) that includes multiple factors (quality of other recruits, team winning rates, tournament success, investment in the basketball program, etc…). But for now we will just present simple statistics related to school’s ability to produce output (NBA draft picks) as a function of input (quality of recruits).

Our first set of rankings is for the ACC.  At the top of the list we have Boston College and Georgia Tech.  Boston College has done a good job of converting low-ranked talent into NBA picks (in this time period they had two three-star players and a non-rated player drafted).  Georgia Tech, on the other hand, has converted all of its five-star recruits, and several of its four-star recruits.  A result that may at first glance seem surprising is the placement of UNC and Duke.  However, upon reflection these results make a good deal of sense.  When players choose these “blue blood” programs they face stiff competition for playing time from both current and future teammates.

Here are some questions you probably have about our methodology:

What time period does this represent?

We examined recruiting classes from 2002 to 2011 (this represents the year of graduation from high school).  While the chart above ranks the ACC, we compiled data for over 300 Division 1 colleges (over 12,000 players).

How did you compute the conversion rate?

The conversion rate for each school is defined as (Sum of draft picks for the 2002-2011 recruiting classes)/(Weighted Recruiting Talent).  Weighted Recruiting Talent is determined by summing the recruiting “points” for each class.  These “points” are computed by weighting each recruit by the overall population average probability of being drafted for recruits at that corresponding talent level.  We are using ratings data from Rivals.com.  The weights for each “type” of recruit were 0.51 for each five star recruit, 0.13 for each four star, 0.03 for each three star, 0.008 for each two star, and 0.004 for each not ranked.  

Second-round picks often don’t even make the team.  What if you only considered first round picks?

We have also computed the rates using first round picks only, please see the table below.

NEXT: RANKING THE BIG 10

*We can already hear our friends at Duke explaining how players are rated more highly by services just because they are being recruited by Duke.  We acknowledge that it is very difficult to get a true measure of a high school player’s ability.  However, we also believe that over the last ten years, given all of the media exposure for high school athletes, this problem has attenuated.

Mike Lewis & Manish Tripathi, Emory University 2013.

Why Texas and Oklahoma State are Ahead of Kansas: The Best Fan Bases in the Big 12

One of the more entertaining aspects of producing the Emory Sports Marketing Analytics blog has been emotional nature of the criticisms that we have received.  Our series ranking fan bases has been particularly provocative.

What does the preceding have to do with the Big Twelve?  Some of our critics claimed that our rankings were “silly” because Kansas was not ranked in the Top Ten, while Oklahoma State and Texas were.   We thought that we would take a bit more time with this post to investigate how we could possibly come to this result.

As a starting point, if you had asked us to name the top fan bases in college basketball before we ran the numbers we would have said (in no particular order) Kentucky, Duke, North Carolina, Kansas and Indiana.  In other words, we would have bravely identified the conventional wisdom.  But our goal at Emory Sports Marketing Analytics is to go beyond the conventional wisdom, and to see what the numbers say.

Our emphasis on financial metrics also leads to some complaints.  This is somewhat odd given that we are covering sports that are clearly run like businesses.  It has been reported that Bill Self’s current deal with Kansas will pay him close to $50 million over ten years.  This suggests to us that Kansas very much views basketball through a financial lens.

Getting back to the conventional wisdom, we believe that Kentucky, Indiana, Kansas and Duke have exceptional fan bases.  However, we are not ready to concede that the passion felt by a Kansas fan exceeds that felt by an Oklahoma State or Texas fan.  Rather than rely on the noise created by fan bases, we examine how fans vote with their dollars.  And more to the point, we try to control for the role of on-court success.  While some may view this as crass, if you were the CEO of Apple or Coca Cola would you rather that your customers were highly loyal and willing to pay premium prices or would you rather that your brand was voted a fan favorite in an Internet poll?  The marketing concept that we are exploring is referred to as customer equity.  The basic idea is a brand’s ultimate source of revenues and profits is its customers.  Now a big caveat to this is that by measuring the value of the customer bases we are not controlling for how good of a job each institution does with managing its customer base.

The preceding list provides our breakdown of the Big Twelve.  Texas leads the way followed by Oklahoma State and then Kansas.  So what drives this result?  Over the last decade Texas has reported the largest basketball revenues in the conference followed by Kansas.  Texas’ advantage in revenue is slightly more than 4%.  More importantly, Texas generated this slightly higher revenue while winning around 5 games less per year than the Jayhawks.  Now one can argue that Texas has unique advantages or that Kansas could be generating more revenue, but our analysis is at least based on solid numbers and our dependent measure (revenue premium based brand equity) is an unambiguous term.

The other surprise was the ranking of Oklahoma State.  In this case, Kansas does produce about 25% more revenue than Oklahoma State.  But the Cowboys generated their revenue while winning about 35% less games per year and no national championships.  Both schools have proud histories and legendary past coaches.  What our analysis gets at is what would happen if both teams performed identically.  What would the environment be like at Gallagher-Iba arena if the Cowboys averaged 30 wins per year for a decade and had numerous trips to the Final Four?

(Note: The study examines 2001 to 2011, thus Nebraska, Colorado, Texas A&M, and Missouri are included in the Big 12)

Debunking a Debunking: Our Response to the NetsDaily

If the true purpose of the Emory Sports Marketing Analytics blog is to generate conversation, then our NBA brand equity study has been a success.  The reality is that we are a couple of sports fans that happen to be university professors.  I (Mike) spend most of my time developing dynamic models that predict customer behavior over time.  This blog is an effort to combine our jobs and hobbies.  An example of this is a paper on competitive balance in MLB.

Our blog is intended to present more quick hitting analyses of current issues in the world of sports.  With these studies we rely on publicly available data, and we tend to use relatively straight-forward statistical methods.  We really don’t begin with any agenda; we just let the numbers speak.

In the case of the NBA fan study, the numbers spoke and then the NetsDaily answered.  Given our love of debate, we can’t help but answer back.  Though all kidding aside, to the Nets/NetsDaily guys: we really do like what you guys have done.  You are obviously an exciting franchise with a lot of great marketing.

In what follows we reprint the NetsDaily notes in blue and then give our comments in bold:

Final Note: That Emory University study

We’re not going to devote a whole lot of effort to debunking the silly Emory College attempt at defining passionate fans by analytic means.  It found that the Nets were the worst home fans in the NBA and the Knicks the best.

Okay, we get the idea, the NetsDaily is not happy with the results.  A couple of things come to mind.  One, Emory is a University.  Second, debunking is probably the wrong way to start a discussion.  With almost any study like ours, there will need to be assumptions made.  As we noted in our original post,we are using a revenue premium brand equity model.  One could definitely argue that passion and revenue are different concepts.  It is probably more useful to understand what the study is saying than to claim it is wrong.

We would just like to point out some serious flaws in the study. The original study was so devoid of data that the authors were asked to provide some, which they did in a self-congratulatory addendum.

We are trying to strike a balance in the blog.  We could report all statistical models, but we are trying to keep things interesting so we emphasize intuition and report the interesting findings.  We are more than happy to share additional details.  And as academics, self-congratulation is probably our best hope for some positive feedback.

The authors note that key data they used to derive their conclusions is something called “home revenue.” They attempt to estimate “home revenue,” which is a finite, known but proprietary figure not available to them. Shouldn’t they note that, explain its relevance?

“The analysis begins with a model of box office revenue based on variables that correspond to market potential (capacity and market population), team quality (winning percentage) and entertainment value (number of all stars, payroll). The insight or theory that drives the analysis is that this model can be used to predict the revenue that is due to quality and market potential. Any difference between this predicted value and actual value is due to ‘fan loyalty’.”

So the reality is that they don’t have the finite, known but proprietary information that is the core of the theory so project it based on other data, including things as spurious as number of all-Stars, but ignore other data that might be important, like say MERCHANDISE SALES. Need we go there? The Nets now rank fourth in NBA merchandise sales. In the first several months after the merchandise was introduced, they ranked first.

The NetsDaily does not seem to properly understand the analysis.  We use a really simple estimate of home box office revenue using the popular Fan Cost Index and attendance reported by ESPN.  Is this the ideal way to determine home revenue?  Of course not!  Is it a reasonable way given that teams are private and do not report detailed financials?  Reasonable might even be too strong.  In fact, let’s say that it is a crude way to compute box office revenues. (Point for the Nets)

We then use this revenue measure as a dependent variable in a regression model that uses the previously mentioned factors.  We are not estimating revenue in terms of things like number of all-stars, we are developing a model that explains revenues by these factors that indicate team quality, market size and entertainment value (fans come out to see all-stars).  We then compare the difference between the predicted and the (crude) estimate of actual revenue.

The idea is to look at attendance AFTER controlling for how well the team did.  Does Miami selling out mean much given the quality of the team on the court?  Our goal is to really get at the true core support for a team.

On comparing the attendance between the Knicks and Nets, they use gross numbers of attendance.

“The teams share the largest population metropolitan areas but the Knicks achieve a 10.7% advantage in terms of attendance DESPITE charging much greater prices. It is this greater pricing power that pushes the two teams to opposite ends of the ranking.”

Suppose instead of gross numbers, they used capacity percentage. The original analysis appears to rely on ESPN attendance percentages, that is, the percentage of arena capacity sold out on average each game. We say “appears” because the original article notes, ” A quick look at attendance data from ESPN shows that the Trail Blazers regularly exceed capacity for entire seasons.” Two points: the Trail Blazers attendance last season was 95.4 percent, which did not exceed capacity (or it would have been in excess of 100 percent.)

This is an example of why it’s probably better to ask questions and have a discussion than to go on the attack.  We control for stadium capacity in the revenue prediction model to account for differences in stadium capacity.

But they do make a good point here.  When we talk about “best” fans there really is no obvious metric.  We choose revenue, the Nets suggest capacity utilization.  Also, the Trail Blazers attendance percentage was 102.6 percent of capacity in 2012, 102.7 percent in 2011, and 102.6 percent in 2010.  We believe that this is very consistent with our wording.

However, using ESPN data presents problems. It is inaccurate regarding the Nets. ESPN uses an NBA capacity of 18,000 for Barclays Center. That was the original number for NBA games. As the arena was completed, capacity was reduced to 17,732 (apparently to accommodate loge seating added late in construction.) The number can be found on the Nets website. So the actual percentage of seats sold this season is 96.9 percent, not 94.9. That would put the Nets at tenth (not 16th) in the NBA, just ahead of … drum roll … the Knicks at 96.3 percent and the Blazers.

Again, some good points are raised here by the NetsDaily.  As statisticians, we love to have very accurate data.  In reality, data almost always contains some noise or error.  The Nets would have a valid complaint if the publicly available data was somehow consistently biased against the Nets.  Does Team Marketing Report systematically underestimate the Nets’ prices, and does ESPN systematically underestimate the Nets’ attendance?  We don’t know.  If so, we apologize.

We leave it to the readers to decide whether our measure which includes quantity and prices is preferred to straight capacity utilization.

But let’s put aside the methodology and data and look at the final product, which suggests below average work.  Is there ANYONE in the NBA who believes that fan loyalty to the Dallas Mavericks, Phoenix Suns, and Orlando Magic is on the rise … or that their fans had greater loyalty than the teams that follow them in the Emory rankings: the Miami Heat and San Antonio Spurs??? You want an example of analytics gone wild???

The comment above seems to be a common misinterpretation.  The example of the Orlando Magic is really what is at the core of our study.  We are not saying that the Orlando Magic has more fan support than the Miami Heat this season.  We are saying that after you control for the difference in the quality of the teams it appears that the Orlando Magic have a more devoted fan base.  Over the past season, the Magic had an average home attendance of 17,595, while only winning 24% of their games.  The key question (and what we use the statistical models to get at) is what would Miami have drawn if they didn’t win 80% of their games and have 3 all-stars in the lineup?

If we were petty people we would also point out that the ESPN attendance figures for last season report that the Magic drew 721,414 fans while the Nets operating in the largest metropolitan area and winning about 60% of their games drew 704,702.  We usually aren’t petty, but they did call our study “silly”, and called us “C students” (Manish says thanks for the passing grade) on Twitter.

The study is also a rare, rare instance where Barry Baum, chief communications officer of the Nets, and Norman Oder, the leading critic and chronicler of Atlantic Yards find common ground.

Says Baum, after reading the original article, “With all due respect to Emory University, that is a seriously flawed study.”

Thank you Mr. Baum for getting the school name correct.

Says Oder, after reviewing the article and supporting data, “The Knicks’ attendance edge is magnified by an arena with greater capacity, and the willingness of Knicks fans to pay more. [It] has less to do with passion than a longstanding monopoly position in a large market.”

This is also a good point, and one that we acknowledged.  It seems likely that the Knicks may benefit from locational advantages that translate into greater pricing power.

If there is a continuing dispute on this, we suggest a review by Nate Silver, the New York Times stats guru … and Nets fan. We are sure he will get to the bottom of it.

We would of course very happy to extend the discussion.  We were in fact surprised when we ran the numbers, and found the Nets on the bottom.  And remember we did point out that the Nets were rapidly improving.

Mike Lewis and Manish Tripathi, Emory University 2013.

Methodology for Recruiting/NFL Draft Studies

The idea behind the Emory Sports Marketing Analytics initiative is to use statistical methods and marketing concepts to understand the decisions of players, teams and leagues with an eye on how these decisions effect fans.  Our feeling is that we can often generate some additional insights into the world of sports by digging into the data.  By and large we avoid too much discussion of statistics and focus mainly on the meaning of our analyses.  But readers can rest assured that the analyses behind the headlines are carefully executed.*

While we have just started the project, we have had a few requests for more details on the methods used to generate our posts.  In particular, our posts that examine the efficiency by which schools convert recruiting success to NFL draft have generated multiple questions.  The post that started the discussion was based on an analysis of six NFL drafts (2007-2012).  The analysis we reported used the number of draft picks divided by the number of elite (4 and 5 star) recruits who signed with the school.**  This ratio was then used in a linear regression that included data on each school’s investment in the football program, information of the schools recruiting success, winning rates, major bowl participation, conference memberships and other factors.

We do note that one issue in this model was in defining “recruiting success.”  Because there was no clear measure of “recruiting success” we tried multiple specifications.  These included the “recruiting points” as defined by rivals.com, recruiting class rank (averaged across multiple ratings groups) and the number of athletes at each star level.  Similarly, there may also be some debate as to what constitutes draft success.  While our reported analyses use number of picks as the key measure, one could also argue that first round picks or players selected in rounds one through three would also be appropriate.  Given the lack of obvious specification for the dependent measure of draft success and the independent variable of recruiting success, our approach was to estimate a wide variety of specifications and see what results are robust to the design of the specification.

In the case of the NFL draft analysis the finding that recruiting success tends to reduce the rate (NFL output / recruiting input) was amazingly robust.  Whether we predicted the number of day one picks or used recruiting rank the finding that top programs on average don’t produce as many NFL players as we might expect given their recruiting success was consistent.  We should, of course, emphasize that elite programs do produce more picks in absolute terms.  The key is that other programs also produce significant numbers of draft picks.

Following the 2013 NFL Draft, we have produced a series of studies that examine the “success” of colleges in converting recruiting talent into NFL draft picks.  As with any analysis based on essentially a single data point, it’s important to remember that these results are more anecdotal than conclusive.  For these studies, we produce a weighted-average of  “recruiting points” as defined by rivals.com for each school.  The weights are determined by the distribution of entering college class years for the players drafted in 2013.  The classes used are largely 2008, 2009, and 2010.  We divide the number of picks in the 2013 NFL draft by the weighted-average “recruiting points” measure for each school to determine its “success” score in the draft.  “Winners” are essentially the top quartile of scores in the conference, and “Losers” are the bottom quartile.

*Since both members of the team are business school professors we should probably make a distinction between academic publications and our blog posts.  In academic publications, methods tend to be fairly complex and are reported in great (painful?) detail.  In our blog posts we tend to use relatively simple methods such as linear and logistic regression.  In the blog posts we focus on robustness and consistency across multiple model specifications rather than on technical adjustments to the models.

** For example, the reason we used the sum of 4 and 5 star recruits was not because we were looking for a model that gave us the “right” answer but because the number of 4 and 5 star recruits tends to be in the range of about 250 per year.  This 250 number is relatively close to the approximately 220 players taken in the draft.  As such, we viewed these 250 recruits as approximating the set of projected NFL players in a given year.

By Mike Lewis & Manish Tripathi, Emory University 2013