I want to start the series with the topic of “Metric Development.” I’m going to use the term “metric” but I could have just as easily used words like stats, measures or KPIs. Metrics are the key to sports and other analytics functions since we need to be sure that we have the right performance standards in place before we try and optimize. Let me say that one more time – METRIC DEVELOPMENT IS THE KEY.
The history of sports statistics has focused on so called “box score” statistics such as hits, runs or RBIs in baseball. These simple statistics have utility but also significant limitations. For example, in baseball a key statistic is batting average. Batting average is intuitively useful as it shows a player’s ability to get on base and to move other runners forward. However, batting average is also limited as it neglects the difference between types of hits. In a batting average calculation, a double or home run is of no greater value than a single. It also neglects the value of walks.
These short-comings motivated the development of statistics like OBPS (on base plus slugging). Measures like OBPS that are constructed from multiple statistics are appealing because they begin to capture the multiple contributions made by a player. On the downside these types of constructed statistics often have an arbitrary nature in terms of how component statistics are weighted.
The complexity of player contributions and the “arbitrary nature” of how simple statistics are weighted is illustrated by the formula for the NFL quarterback ratings.
This equation combines completion percentage (COMP/ATT), yards per attempt (YARDS/ATT), touchdown rate (TD/ATT) and interception rate (INT/ATT) to arrive at a single statistic for a quarterback. On the plus side the metric includes data related to “accuracy” (completion percentage) to “scale” (yards per), to “conversion” (TDs), and to “failures” (interceptions). We can debate if this is a sufficiently complete look at QBs (should we include sacks?) but it does cover multiple aspects of passing performance. However, a common reaction to the formula is a question about where the weights come from. Why is completion rate multiplied by 5 and touchdown rates multiplied by 20?
Is it a great statistic? One way to evaluate is via a quick check of the historical record. Does the historical ranking jive with our intuition? Here is a link to historical rankings.
Every sport has examples of these kinds of “multi-attribute” constructed statistics. Basketball has player efficiency metrics that involve weighting a player’s good events (points, rebounds, steals) and negative outcomes (turnovers, fouls, etc…). The OBPS metric involves an implicit assumption that “on base percentage” and “slugging” are of equal value.
One area I want to explore is how we should construct these types of performance metrics. This is a discussion that involves some philosophy and some statistics. We will take this piece by piece and also show a couple of applications along the way.