Investigating Relationships Between PFF Grades
A refresh and expansion of my 2022 research
Regardless of personal leanings about Pro Football Focus (PFF) film grades, users and disparagers alike should be interested in unearthing objective measures about the quality of said grades.
One approach to scrutinizing PFF’s grades is investigating the ways the grading facets impact one another. PFF publishes team-level grades for each game. For this exercise (which I first shared in 2022) we’ll looking at the following grading facets:
On Offense:
PASS - Grade for the player dropping back to pass
RUN - Grade for the player carrying the football
RECV - Grade for players on pass routes
RBLK - Grade for non-ball carriers on running plays
PBLK - Grade for players in pass protection
On Defense:
PRSH - Grade for the players blitzing the passer
COV - Grade for players dropping back into pass coverage
RDEF - Grade for players defending the run
TACK - Grade for players attempting a tackle
There are three different types of relationships that are worth considering at the game level: 1) activities between players playing offense together, 2) activities between players playing defense together, and 3) the relationship between an offensive activity and an opposing defensive activity. Below, I created three grids to show the R-squared value between team-level grades for regular season NFL games played in the PFF era. The R-squared tells us the extent to which the grades are correlated with one another. An R-squared of 1.00 would mean that one grade tells us exactly what the other grade should be. An R-squared of 0.00 would mean the two grades are independent of one another.
The strongest correlation above (0.66 rsq) is between the offense’s pass blocking grade and the defense’s pass rushing grade. This intuitively makes sense. A rep between a pass blocker and blitzer will often have a winner and a loser, so it makes sense that knowing the offense’s pass blocking grade would give us a good sense of what the defense’s pass rushing grade might be. Similarly, we see a strong relationship (0.49 rsq) between the offense’s run blocking grade and the defense’s run defense grade.
On the flip side, there is virtually no relationship (0.001 rsq) in the way that PFF grades the players running the ball and the way PFF grades a team’s run blocking in a single game. This likely means one of two things. Either the quality of a team’s run blocking is unrelated success running the football, or (far more likely) PFF is able to evaluate these two facets without the result of one biasing the other. Put another way, when a team has a poor run blocking grade in a game, it doesn’t appear to be impacting the way PFF evaluates the runner.
The most intriguing item above to me is the three way relationship between the offense’s passing grade, receiving grade, and the defense’s coverage grade. Like the head-to-head nature of trench matchups mentioned above, it is reasonable to expect to see coverage grade and receiving grade tied together. The difference here though, is that passing grade has nearly as strong a tie to both of those facets individually. The fear (and I frankly don’t think it is this simple) is that the quality of the player passing the ball could be explaining a material amount of the variance in how route runners and coverage players are graded.
Even more significant, team passing grade and team receiving grade are the only grading facets with a high R-squared value that occur within the same team. The issue with positive correlation between grades is exacerbated over the course of a full season, where the receiving corps and primary passer remain the same week in and week out. Below, we can see that season-long R-squared values between passing grade and receiving grade have a strong correlation.
My takeaway here is that it seems possible that the quality of the passer influences the way that PFF grades the pass catching unit. In an ideal world, I’d like to see grading at every position that isolates players & matchups from the other activities on the field, even if the player’s production is dependent on a teammate. When it comes to evaluating players who line up in the trenches, PFF seems to be much better at separating success & failure from teammates engaged in related activities.
Given that the PFF era now spans 19 NFL seasons, we might reasonably wonder if any of the findings above have been consistent over time. After all, it wouldn’t be unreasonable to expect that process or quality have changed in some fashion over this period.
Above, I’ve singled out 8 relationships that have shown high correlations. A line that is trending downward tells us that the relationship is becoming less correlated over time, which is to say that PFF is getting better at distinguishing one facet from another. By and large, there isn’t a whole lot of change aside from a significant and sustained drop in offensive run blocking vs run defense (lower right) starting in 2013. There is also a slight downward slope in the two graphs featuring passing grade (top left), suggesting that PFF likely became aware of their struggles to isolate grades for passers from defenders & receivers years ago and worked to correct as well as they could. I’ve heard that PFF grades for receivers and coverage players tended to lean almost exclusively on targeted passes in the past and that there have been efforts in recent years to grade more frequently on untargeted reps. The trends above would seem to suggest that those efforts have had a small impact.
My second run at this exercise leaves me with conclusions that are essentially the same as those I had three years ago. I’m heartened that PFF grades for players in the trenches do not seem driven by the success or failure of passers and ball carriers. At the same time, I’m not sure what to make of the messy relationship between passer/receiver/coverage player.
This analysis still leaves plenty of room for additional exploration. I’m only looking at team-level grading here, so perhaps there are interesting nuggets to be uncovered with player-level grades. My analysis here also doesn’t explicitly tackle weekly or yearly variance of grades, which my past work has shown could be position/facet specific.