Player Analytics Fundamentals: Part 2 – Performance Metrics

I want to start the series with the topic of “Metric Development.”  I’m going to use the term “metric” but I could have just as easily used words like stats, measures or KPIs.  Metrics are the key to sports and other analytics functions since we need to be sure that we have the right performance standards in place before we try and optimize.  Let me say that one more time – METRIC DEVELOPMENT IS THE KEY.

The history of sports statistics has focused on so called “box score” statistics such as hits, runs or RBIs in baseball.  These simple statistics have utility but also significant limitations.  For example, in baseball a key statistic is batting average.  Batting average is intuitively useful as it shows a player’s ability to get on base and to move other runners forward.  However, batting average is also limited as it neglects the difference between types of hits.  In a batting average calculation, a double or home run is of no greater value than a single.  It also neglects the value of walks.

These short-comings motivated the development of statistics like OBPS (on base plus slugging).  Measures like OBPS that are constructed from multiple statistics are appealing because they begin to capture the multiple contributions made by a player.  On the downside these types of constructed statistics often have an arbitrary nature in terms of how component statistics are weighted.

The complexity of player contributions and the “arbitrary nature” of how simple statistics are weighted is illustrated by the formula for the NFL quarterback ratings.

This equation combines completion percentage (COMP/ATT), yards per attempt (YARDS/ATT), touchdown rate (TD/ATT) and interception rate (INT/ATT) to arrive at a single statistic for a quarterback.  On the plus side the metric includes data related to “accuracy” (completion percentage) to “scale” (yards per), to “conversion” (TDs), and to “failures” (interceptions).  We can debate if this is a sufficiently complete look at QBs (should we include sacks?) but it does cover multiple aspects of passing performance.   However, a common reaction to the formula is a question about where the weights come from.  Why is completion rate multiplied by 5 and touchdown rates multiplied by 20?

Is it a great statistic?  One way to evaluate is via a quick check of the historical record.  Does the historical ranking jive with our intuition?  Here is a link to historical rankings.

Every sport has examples of these kinds of “multi-attribute” constructed statistics.  Basketball has player efficiency metrics that involve weighting a player’s good events (points, rebounds, steals) and negative outcomes (turnovers, fouls, etc…).  The OBPS metric involves an implicit assumption that “on base percentage” and “slugging” are of equal value.

One area I want to explore is how we should construct these types of performance metrics.  This is a discussion that involves some philosophy and some statistics.  We will take this piece by piece and also show a couple of applications along the way.

Decision Biases: Sports Analytics Series Part 4

One way to look at on-field analytics is that it is a search for decision biases.  Very often, sports analytics takes the perspective of challenging the conventional wisdom.  This can take the form of identifying key statistics for evaluating players.  For example, one (too) simple conclusion from “Moneyball” would be that people in baseball did not adequately value the value of being walked and on-base percentage.  The success of the A’s (again – way oversimplifying) was based on finding flaws in the conventional wisdom.

Examples of “challenges” to conventional wisdom are common in analyses of on-field decision making.  For example, in past decades the conventional wisdom was that it is a good idea to use a sacrifice bunt to move players into scoring position or that it is almost always a good idea to punt on fourth down.  I should note that even the term conventional wisdom is problematic as there have likely always been long-term disagreements about the right strategies to use at different points in a game.  Now, however, we are increasingly in a position to use data to determine the right or optimal strategies.

As we discussed last time, humans tend to be good at overall or holistic judgments while models are good at precise but narrow evaluations.  When the recommendations implied by the data or model are at odds with how decisions are made, there is often an opportunity for improvement.  Using data to find types of undervalued players or to find beneficial tactics represents an effort to correct human decision making biases.

This is an important point.  Analytics will almost never outperform human judgment when it comes to individuals.  What analytics are useful for is helping human decision makers self-correct.  When the model yields different insights than the person it’s time to drill down and determine why.  Maybe it’s a shortcoming of the model or maybe it’s a bias on the part of the general manager.

The term bias has a negative connotation.  But it shouldn’t for this discussion.  For this discussion a bias should just be viewed as a tendency to systematically make decisions based on less than perfect information.

The academic literature has investigated many types of biases.  Wikipedia provides a list of a large number of biases that might lead to decision errors.  This list even includes the sports inspired “hot-hand fallacy” which is described as a “belief that a person who has experienced success with a random event has a greater chance of further success in additional attempts.”  From a sports analytics perspective the question might be asked is whether the hot-hand is a real thing or just a belief. The analyst might be interested in developing a statistical test to assess whether a player on a hot streak is more likely to be successful on his next attempt.  This model would have implications for whether a coach should “feed” the hot hand.

Academic work has also looked at the impact of factors like sunk costs on player decisions.  The idea behind “sunk costs” is that if costs have already been incurred then those costs should not impact current or future decision making.  In the case of player decisions “sunk costs” might be factors like salary or when the player was drafted.  Ideally, a team would use the players with the highest expected performance.  A tendency towards playing individuals based on the past would represent a bias.

Other academic work has investigated the idea of “status” bias.  In this case the notion is that referees might call a game differently depending on the players involved.  It’s probably obvious that this is the case.  Going old school for a moment, even the most fervent Bulls fans of the 90’s would have to admit that Craig Ehlo wouldn’t get the same calls as Michael Jordan.

In these cases, it is possible (though tricky) to look for biases in human decision making.  In the case of sunk costs investigators have used statistical models to examine the link between when a player was drafted and the decision to play an athlete (controlling for player performance).  If such a bias exists, then the analysis might be used to inform general managers of this trait.

In the case of advantageous calls for high profile players, an analysis might lead to a different type of conclusion. If such a bias exists, then perhaps leagues should invest more heavily in using technology to monitor and correct referee’s decisions.

  • People suffer from a variety of decision biases. These biases are often the result of decision making heuristics or rules of thumbs.
  • One use of statistical models is to help identify decision making biases.
  • The identification of widespread biases is potentially of great value as these biases can help identify imperfections in the market for players or improved game strategies.

Medaling at the Olympics: Is Corruption the Golden Ticket?

A Guest post from my friend and colleague at Emory – Tom Smith!

by Thomas More Smith

Even before the Olympic flame in Rio was lit, there were significant concerns regarding doping and competitive balance. In June, 2016, the IAFF banned the Russian athletic team (those competing in track-and-field events) from the Rio Olympics after Russia failed to show it had made progress in light of the World Anti-Doping Agency’s report on state sponsored doping by Russia. After a considerable amount of concern and angst by Russian Olympians, the IOC decided not to ban the entire Olympic squad.

The issue of fair-play at the Rio Olympics has been front and center since the opening ceremonies. There is clearly some bad blood between competitors in the Olympic swimming events. At a press conference on Monday, August 9, Lilly King, the U.S. swimmer and Gold medalist of the 100-meter breast stroke, made pointed remarks about the Russian Silver medalist, Yuli Efimova, who was, until several weeks ago, banned from Olympic competition because of positive drug tests.  The Gold medalist of the men’s 200-meter freestyle even, Sun Yang, was the subject of testy comments from Camille Lacourt, who took fifth in the event. Lacourt suggested his Chinese competitor “pisses purple” in reference Sun’s failed drug test several years ago.

In both of these situations, athletes who had at one time been found to have taken PEDs were standing on the medal podium. Are these athletes clean now and will their medals stand? In 2012, Nadzeya Ostapchuk from Belarus won the Gold medal in women’s shot put. The IOC subsequently withdrew her medal and her standing after she tested positive for anabolic steroids.  Other athletes at the 2012 games and 2008 and 2004 games were stripped of their medals after they tested positive for various PEDs.

This leads to an interesting question – do dirty athletes win more medals? Or, perhaps, do athletes from “dirty” programs or countries win more medals?

How Much Advantage do PEDs Provide?

There is no data on athletes currently taking PEDs – we only know about the athletes that have taken PEDs and eventually tested positive for them. Also, we can suspect that some athletes did or didn’t take PEDs during the Olympics but we don’t really know unless they were tested and the results were positive. Still, some athletes have been able to avoid positive tests for years because of the drugs, testing facilities or advanced systems in place to mask the drugs (see, for example, Lance Armstrong.) As such, it is a little tricky to test the relationship between PED use and performance in sporting events. However, we can examine the relationship between Olympic performance and the perceived level of corruption of the country of the athlete – what I will call the “dirty” country hypothesis.

H0: Athletes from countries with more corruption are more likely to win Olympic medals.

Perceived Level of Corruption

The organization Transparency International compiles a corruption perception index tracking the level of perceived corruption by country and by year. The Corruption Perceptions Index scores countries on a scale from 0 (highly corrupt) to 100 (very clean). No county has a perfect score (100); the top four countries of Denmark, Finland, New Zealand and Sweden regularly score between 82 and 92. Nearly two-thirds of the 170 countries identified by Transparency International score below 50.

smith_fig1

Data: Transparency International and ESPN

Using data from the 2012 Olympics, I ran a correlation plot of the total Olympic medal count and the Corruption Perception Index (CPI) for each country with 10 or more total medals.  The plot of the total medal count for each country relative to the Country’s CPI is shown in the figure above. We can see that New Zealand, for example, is perceived as very un-corrupt (Index = 90) but also has a low medal count (13), while Russia has a much higher perceived level of corruption (index = 27) and a high medal count (79). The plot of the best-fit line shows a positive correlation. That is, although Russia and China have high medal counts and high levels of perceived corruption, the overall trend suggests that countries with less perceived corruption tend to also perform better at the Olympics.

Although it looks like some countries do poorly because of corruption, this may not be the case. Of course, correlation does not mean causation. In addition, this plot does not take into consideration the size of the Olympic team. Azerbaijan, for example, had 10 medals in the 2012 Olympics and had a PCI of 27. But, Azerbaijan only sent 53 athletes to the Olympics — a considerably smaller team than Ukraine, which had 19 medals, a PCI of 26 and 237 athletes. So, perhaps the countries with higher perceived corruption might have performed better at the Olympics if they had sent more athletes. When the medal count is adjusted for team size (Total Medals / Total Athletes) and plotted against the PCI, we get the figure below.

smith_fig2

Data: Transparency International and ESPN

In this figure, the correlation has reversed– countries with higher perceived corruption also have higher level of medals per athlete in general. When accounting for the size of the team, countries such as Kenya and Azerbaijan tend to do pretty well (as does China and Russia). The United States still performs well, but does not have as high a medal per athlete count as China or Kenya.

What does this mean?

It is unwise to use figures like this to suggest that the Kenya Olympic team are full of drug cheats or that the Chinese team is engaged in dubious behavior. It’s also unwise to suggest the United States has completely clean athletes (we know, for a fact, that this is not the case!) But, given that there are seemingly strong correlations between perceived corruption and Olympic performance, it is understandable that some athletes would be vocal about the behavior of the person in the next lane based on the country the athlete is playing for.

Amateur Sports and Brands

HBO Sports recently created a detailed report on the IOC.  The RIO Olympics do not come off well.  Pollution, doping, corruption and athlete exploitation are at the top of the list.  It is a fascinating story that seems to play out with each Olympic Games.

This issue of fair compensation for the athletes is high on the list. The number discussed in the report was $4 billion.  The question is whether and how this money from rights fees and sponsors should be allocated to the athletes.  Is (and should) there be an Olympic Ed O’Bannon?

In many respects this starts to sound like the debates about college sports in the US.  These debates are usually cast in terms of fairness.   to the athletes versus arguments about the purity of the sport or appropriateness of academic institutions running pro teams.

These debates are at best incomplete without considering the role of marketing and brands.  While college football players supply the product, the brands owned by the colleges or the Olympics is what drives fan interest.  Leonard Fournette is a Heisman favorite and a huge star.  But does he draw fans to LSU.  the truth is he probably doesn’t (in the short-term).  In the long-term its stars like Fournette that create the brand equity. 

635787203149253819-USATSI-8811280

Likewise, in the case of the Olympics – we could ask how much interest in driven by the current athletes?  and how much is driven by the attachment people have to the Olympics (the brand).

carllewis

I think (in the US) the Olympic brand is about Carl Lewis, Bruce Jenner, Mary Lou Retton, Jesse Owens, Cassius Clay or many others.  It remains to be seen who from the current crop breaks out.

The real problem, I believe is one of equity.  This is true in both college sports and the Olympics.  The fundamental issue is who gets to harvest the value of the brands.  The problem – to many folks – is that this seems to just end up being the people that control the institutions at any one moment.  The athletes that have built the brands (the stars of the past) and the athletes that create the product (this years athletes) tend to get left out in the cold.

 

The Best NFL Fans 2016: The Dynamic Fan Equity Methodology

The Winners (and Losers) of this years rankings!  First a quick graphic and then the details.

2016B_W

It’s become a tradition for me to rank NFL teams’ fan bases each summer.  The basic approach (more details here) is to use data to develop statistical models of fan interest.  These models are used to determine which cities fans are more willing to spend or follow their teams after controlling for factors like market size and short-term variations in performance.  In past years, two measures of engagement have been featured: Fan Equity and Social Media Equity.  Fan Equity focuses on home box office revenues (support via opening the wallet) and Social Media Equity focuses on fan willingness to engage as part of a team’s community (support exhibited by joining social media communities).

This year I have come up with a new method that combines these two measures: Dynamic Fan Equity (DFE).  The DFE measure leverages the best features of the two measures.  Fan Equity is based on the most important consumer trait – willingness to spend.  Social Equity captures fan support that occurs beyond the walls of the stadium and skews towards a younger demographic.  The key insight that allows for the two measures to be combined is that there is a significant relationship between the Social Media Equity trend and the Fan Equity measure.  Social media performance turns out to be a strong leading indicator for financial performance.

Dynamic Fan Equity is calculated using current fan equity and the trend in fan equity from the team’s social media performance.  I will spare the technical details on the blog but I’m happy to go into depth if there is interest.  On the data side we are working with 15 years of attendance data and 4 years of social data.

The Winners

We have a new number one on the list – the New England Patriots. Followed by the Cowboys, Broncos, 49ers and Eagles.  The Patriots victory is driven by fans willingness to pay premium prices, strong attendance and phenomenal social media following.  The final competition between the Cowboys and the Patriots was actually determined by the long-term value of the Patriots greater social following.  The Patriots have about 2.4 million Twitter followers compared to 1.7 for the Cowboys.  Of course this is all relative a team like the Jaguars has just 340 thousand followers.

The Eagles are the big surprise on the list.  The Eagles are also a good example of how the analysis works.  Most fan rankings are based on subjective judgments and lack controls for short-term winning rates.  This latter point is a critical shortcoming.  It’s easy to be supportive of a winning team. While Eagles fans might not be happy they are supportive in the face of mediocrity.  Last year the Eagles struggled on the field but fans still paid premium prices and filled the stadium.  We’ll come back to the Eagles in more detail in a moment.

The Strugglers

At the bottom we have the Bills, Rams, Chiefs, Raiders and Jaguars.  This is a similar list to last year.  The Jags, for example, only filled 91% of capacity (ranked 27th) despite an average ticket price of just $57.  The Chiefs struggle because the fan support doesn’t match the team’s performance.  The Chiefs capacity utilization rate ranks 17th in the league despite a winning record and low ticket prices.  The Raiders fans again finish low in our rankings.  And every year the response is a great deal of anger and often threats.

The Steelers

The one result that gives me the most doubt is for the Pittsburgh Steelers.  The Steelers have long been considered one of the league premier teams and brands.  The Steelers have a history of championships and have been known to turn opposing stadiums into seas of yellow and black.  So why are the Steelers ranked 18th?

steeler_atl

A comparison between the Steelers and the Eagles highlights the underlying issues.  Last year the Steelers had an average attendance of 64,356 and had an average ticket price of $84 (from ESPN and Team Market Report).  In comparison the Eagles averaged 69,483 fans with an average price of $98.69.  In terms of filling capacity the Steelers were at 98.3% compared to the Eagles at 102.8%.  The key is that the greater support enjoyed by the Eagles was despite a much worse record.

One issue to consider is that of pricing.  It may well be that the Steelers ownership makes a conscious effort to underprice relative to what the market would allow.  The high attendance rates across the NFL do suggest that many teams could profitably raise prices.  It’s entirely reasonable to argue that the Steelers relationship to the Pittsburgh community results in a policy of pricing below market.

In past years the Steelers have been our social media champions.  This past year did see a bit of a dip.  In terms of the Social Media Equity rankings the Steelers dropped to 5th.    As a point of comparison, the Steelers have about 1.3 million Twitter followers compared to 2.4 million for the Patriots and 1.7 million for the Cowboys.

 

The Complete List

And finally, the complete rankings.  Enjoy!


2016complete

End of an Era – Goodbye Manish

A fond farewell and a new era –

Things change.  Sometimes for the good and sometimes not.  We (Manish and myself) started this blog a few years ago as a means for turning our love into sports into an academic pursuit.  Its been a lot of fun and and a lot of work.  Its taken us into different ways of thinking and exposed us to a lot of interesting media.

Manish_0118

 

But its come to an inflection point.  Manish has decided to leave academia.  Nothing wrong with that, but it does mean he needs to step off the platform.  Its one thing for an academic to publish findings that insult Raiders or Duke Blue Devil fans.  Its another for someone in the corporate world.

He is already missed.  The best thing about this line of work was that it was fun and we had a shared purpose.   We also did a lot of other stuff related like teach several sports courses here at Emory.  We will have to see how all this evolves.  at a minimum there will likely be far more spelling errors and typos.  But fewer !!!!!

I won’t get too sentimental but its a huge loss.  And I’m genuinely sad.

 

 

 

2016 Pre-Season MLB Social Media Rankings: The Blue Jays Win!

Going into the baseball season, there are all sorts of expectations about how teams are going to perform.  This summer I thought it might be interesting to track social media across a season.  What this means is something of an open question.  I have a bunch of ideas but suggestions are welcome.

But the starting point is clear.  We open with social media equity rankings of MLB clubs.  The basic idea of the social media rankings is that we look at the number of social media followers of each team after statistically controlling for market differences (NY teams should have more followers than San Diego) and for short term changes in winning rates.  The idea is to get a measure of each teams’ fan base after controlling for short-term blips in winning and built in advantages due to market size.  A fuller description of the methodology may be found here.

Social Media Equity is really a measure of fan engagement or passion (no it’s not a perfect measure).  It captures the fact that some teams have larger and more passionate fan bases (again after controlling for market and winning rates) than others.  In this case the assumption is that engagement and passion are strongly correlated with social media community size.  Over the years we have looked at lots of social media metrics and my feeling, at least, is that this most basic of measures is probably the best one.

When we last reported our Social Media Equity ratings  the winners were the Red Sox, Yankees, Cubs Phillies and Cardinals.  The teams that struggled were the White Sox, Angels, A’s, Mets and Rays.  This was 2014.  Last summer was kind of a lost summer for the blog.

encarnacion-edwin-150826-620

But enough background…   The 2016 pre-season social equity rankings feature a top five of the Blue Jays, Phillies, Braves, Red Sox and Giants.  A lot of similarities from 2014, with the big change being the Blue Jays at the top of the rankings.  One quick observation (we have all summer for more) is that teams with “bigger” geographic regions like the Blue Jays (Canada?), Braves (the American South) and the Red Sox (New England) do well in this measure of brand equity since constraints like stadium capacity don’t play a role.

At the bottom of the rankings it’s the Marlins, Angels, Mariners, A’s and Nationals.  Again a good deal of overlap from earlier.  Maybe the key shared factor at the bottom is tough local competition.  The Angels struggle against the Dodgers, the A’s play second fiddle in the bay area and the Marlins lose out to the beach.

The table below provides the complete rankings and a measure of trend.  The trend shows the relative growth in followers from 2015 to the start of the 2016 season (again after controlling for factors such as winning rates).  The Cubbies are up and comers!  While the Mariners are fading.

Team Social Media Equity Rank Trend Rank
Blue Jays 1 4
Phillies 2 14
Braves 3 10
Red Sox 4 3
Giants 5 7
Yankees 6 21
Tigers 7 2
Reds 8 6
Rangers 9 17
Rays 10 13
Cubs 11 1
Pirates 12 9
Mets 13 5
Padres 14 23
Diamondbacks 15 8
Indians 16 11
Dodgers 17 15
Cardinals 18 25
White Sox 19 20
Brewers 20 22
Oriels 21 27
Astros 22 26
Twins 23 19
Royals 24 28
Rockies 25 16
Marlins 26 29
Angels 27 24
Mariners 28 30
A’s 29 12
Nationals 30 18

More to come….

Coaching Hot Seat Week 3 – Mack Brown and Lane Kiffen

Periodically, we like to do what we call “Instant Twitter Analyses.”  We do these in situations where consumer opinion is the key to understanding a sports business story.  In the case of “coaches on the hot seat” customer reactions are a critical factor.  While sports are a bit different than most marketing contexts, the basic principle that unhappy customers signal a problematic future remains true.

During this college football season we have been tracking fan base reactions to their coaches.  As we all know, there are two prominent programs (USC and Texas) with coaches in trouble.  The point of today’s post is to show how the Twitterverse has been reacting to these two coaches this season.

In the picture below we see the daily negative and positive posts for these two coaches.  The patterns and levels are remarkably similar.  But it does seem that Brown has a few more defenders at Texas (despite having two losses).   In fact over the first three weeks of the season Brown’s percentage of positive posts is 47.8% while Kiffin’s is 45.7%

This data indicates that in the court of public opinion these coaches are both in about the same shape.  We also suspect that an extended hot streak would save both coaches.  Perhaps the most interesting thing about this data is what it says about each job and fan base.  In the past we have ranked Texas as having the most loyal customer base and Forbes has ranked Texas as the most valuable athletic program.  To add to the Texas advantages, it seems that the fans are also a bit less critical.