16 Sep 2009

Football stats: Through the eyes of a researcher

0 Comment

This Sunday marked the opening weekend for this season of professional football.  Prior to the opening kickoffs, several networks offer pregame shows filled with amusing anecdotes, insider knowledge, and expert analysis relevant to this weekend’s games.  A great deal of this flood of information is accompanied by a variety of statistics.  Generally the statistics presented are straightforward and any analysis that accompanies them is also pretty basic, and generally qualitative in nature.  I was intrigued however by some comments made by analyst and former professional football player Merril Hoge.  While discussing Peyton Manning and the Colts offense, the following statistics were displayed comparing Peyton Manning’s passing broken out by plays with 2 receivers on the field versus plays with 3 receivers on the field:

Peyton Manning 2008 Passing Statistics

2 Wide Receivers

3 Wide Receivers

Completions – Attempts

19 – 34

347  – 513

Completion Percentage



QB Rating



While the stats were displayed Merril asserted that Manning had much more success when three receivers were on the field, and then digressed into a discussion about how he thought the Colts would use Dallas Clark as part of their offensive scheme in the coming season.

As a research geek, I was immediately struck by the obvious differences in sample size.  Peyton Manning only threw 34 passes when the team was in a 2 receiver formation, while he threw 513 when there were 3 receivers on the field.  Some quick calculations let me know that with a sample size of 34 the error around a percentage would be approximately plus or minus seventeen percentage points.  This is more than enough to shed some doubt on Hoge’s claim if it was based on completion percentage.  Some additional calculations confirm that the differences in completion percentage are not significantly different even at the 90 percentile level.

A couple things to note here:  First, this doesn’t address if the difference between the QB rating is significant.  This is a more complicated question, as the QB rating is derived through a fairly complex formula, and is neither a mean, nor a proportion.  As a result, it is likely that it is necessary to derive an equation to calculate a z-stat to be able to make the comparison.  Second, both Merril Hoge in his qualitative discussion, and I in my rudimentary statistical analysis completely ignore other important differences in the data.  Clearly we have an endogeneity issue because the formation a team chooses to set up in will be dictated by the game situation and that could clearly influence the likelihood of success particularly in the two receiver formations.  Third, and more generally, football statistics are among the least clear in all of sports to interpret, because game situations so influence the statistical results (i.e.  a QB throwing for more yards might be doing so because his team is losing by a great deal, in this case do we interpret the yardage as a positive because of the success of those plays in gaining yards, or as a negative because of the poor team play that led to the need to throw for all of those yards).  As a result it more context might be required to determine if one set of stats are truly better than another.

Ultimately, I have every reason to believe the points that Merril Hoge made are valid.  I am sure he knows a great deal more about football than I.  Also in the end, because the statistics were only being used as a basis for a more qualitative discussion, there is no harm in them being presented and discussed in this manner.  As a researcher, however, it difficult to resist the urge to try and pick at the statistics presented a little bit, and attempt to determine if they hold up to additional scrutiny.

Kevin Knight
Sr. Research Associate