In Part 3 we started to talk about the complementary role of human decision makers and models. Before we get to the next topic – Decision Biases – I wanted to take a moment to present an example that helps illustrate the points being made in the last entry.
I’m going to make the point using an intentionally nontraditional example. Part of the reason I’m using this example is that I think it’s worthwhile to think about what might be “questionable” in terms of the analysis. So rather than look at some well-studied relationships in contexts like NFL quarterbacks or NBA players, I’m going to develop a model of Fullback performance in Major League Soccer.
To keep this simple, I’m going to try and figure out the relationship between a player’s Plus-Minus statistic and a few key performance variables. I’m not going to provide a critique of Plus-Minus but I encourage everyone to think about the value of such a statistic in soccer in general and for the Fullback position in particular. This is an important exercise for thinking about combining statistical analysis and human insight. What is the right bottom line metric for a defensive player in a team sport?
The specific analysis is a simple regression model that quantifies the relationship between Plus-Minus and the following performance measures:
- % of Defensive Ground Duels Won
- % of Defensive Aerial Duels Won
- Tackling Success Rate (%)
- % of Successful Passes in the Opponents ½
This is obviously a very limited set of statistics. One thing to think about is that if I am creating this statistical model with even multiple years of data, I probably don’t have very many observations. This is a common problem. In any league there are usually about 30 teams and maybe 5 players at any position. We can potentially capture massive amounts of data but maybe we only have 150 observations a year. Note that in the case of MLS fullbacks we have less than that. This is important because it means that in sports contexts we need to have parsimonious models. We can’t throw all of our data into the models because we don’t have enough observations.
The table below lists the regression output. Basically, the output is saying that % Successful passes in the opponent’s half is the only statistic that is significantly and positively correlated with a Fullback’s Plus-Minus statistic.
||Pr > |t|
|% Defensive Ground Duels Won
|% Def Aerial Duels Won
| Tackling Success Percentage
|% Successful Passes in Opponents 1/2
The more statistically oriented reader might be asking the question of how well does this model actually fit the data. What is the R-Square? It is small. The preceding model explains about 5% of the variation in Fullback’s Plus-Minus statistics.
And that is the important point. The model does its job in that it tells us there is a significant relationship between passing skill and goal differential. But it is far from a complete picture. The decision maker needs to understand what the model shows. However, the decision maker also needs to understand what the model doesn’t reveal. This model (and the vast majority of other models) is inherently limited. Like I said last time – the model is a decision support tool / not something that makes the decision.
Admittedly I didn’t try to find a model that fits the data really well. But I can tell you that in my experience in sports and really any context that involves predicting or explaining individual human behavior, the models usually only explain a small fraction of variance in performance data.