Last time I introduced the topic of player “metrics.” (If you want to get caught up you can start with Part 1 and Part 2 of the series.) As I noted, determining the right metric is perhaps the most important task in player analytics. It’s almost too obvious of a point to make – but the starting point for any analytics project should be deciding what to measure or manage. It’s a non-trivial task because while the end goal (profit, wins) might be obvious, how this goal relates to an individual player (or strategy) may not be.
However, before I get too deep into metric development, I want to take a small detour and talk briefly about statistical models. We won’t get to modeling in this entry – the goal is to motivate the need for statistical models! If we are doing player analytics we need some type of tool kit to move us from mere opinion to fact based arguments.
To illustrate what I mean by “opinion” lets consider the example of rating quarterbacks. In the previous entry, I presented the Passer Rating Formula used to rate NFL quarterbacks. As a quick refresher let’s look at this beast one more time.The formula includes completion percentage (accuracy), yards per attempt (magnitude), touchdowns (ultimate success) and interceptions (failures). Let’s pretend for a second that the formula only contained touchdowns and interceptions (just to make it simple). The question then becomes how much should we weight touchdowns per attempt relative to interceptions per attempt? The actual formula is hopelessly complex in some ways – we have fractional weights and statistics in different units – so let’s take a step back from the actual formula.
Imagine we have two experts proposing Passer Rating statistics that are based on touchdowns and interceptions only. One expert might say that touchdowns per attempt are twice as important as interceptions. We will label this “expert” created formula as ePR1 for expert 1 Passer rating. The formula would be:
Maybe this judgment would be accompanied by some logic along the lines of “touchdowns are twice as important because the opposing team doesn’t always score as the result of an interception.”
However, the second expert suggests that the touchdowns and interceptions should be weighted equally. Maybe the logic of the second expert is that interceptions have both direct negative consequences (loss of possession) and also negative psychological effects (loss of momentum), and should therefore be weighted more heavily. The formula for expert 2 can be written as:
I suspect that many readers (or a high percentage of a few readers) are objecting to developing metrics using this approach. The approach probably seems arbitrary. It is. I’ve intentionally presented things in a manner that highlights the subjective nature of the process. I’ve reduced things down to just 2 stats and I’ve chosen very simple weights. But the reality is that this is the basic process through which novices tend to develop “new” or “advanced” statistics. In fact, it is still very much a standard practice. The decision maker or supporting analysts gather multiple pieces of information and then use a system of weights to determine a final “grade” or evaluation.
The question then becomes which formula do we use? Both formulas include multiple pieces of data and are based on a combination of logic and experience. I am ignoring (for the moment) a critical element of this topic – the issue of decision biases. In subsequent entries, I’m going to advocate for an approach that is based on data and statistical models. Next time, we will start to talk more about statistical tools.