Are Some Offensive Positions Harder to Draft Than Others? Part 2: The Scouting Effect

This two-part series addresses a question that comes up repeatedly in fan discussions whenever I taken a look at drafting certain positions. Are some positions harder or easier to draft than others? I have wondered the same thing myself, but until recently have struggled to find a way to address it analytically.

That was until I stumbled on something when I was examining the balance of Opportunity to Opportunity Cost in drafting Quarterbacks. The metric I used as my measure of Opportunity was the Relative Availability of starting-quality QBs, calculated as the proportion of starting-quality QBs within the total pool of draftable QB prospects available at any given pick number in the draft. The Relative Availability metric turns out to be useful for addressing a range of questions related to the draft, including comparing the ease or difficulty of drafting players at different positions, as I will attempt to explain in this article.

The first article ended in a bit of a cliff hanger. I first showed that the past draft hit rates differed across offensive positions following a rank order that agreed with my expectations, and I would expect those of many readers. Offensive line seemed to be the easiest position group for teams to get right, followed in order of increasing difficulty by running backs, wide receivers/tight ends (hard to split), and quarterbacks.

I then demonstrated a possible alternative explanation by plotting the Relative Availabilities of good players at the same positions throughout the draft. Doing that revealed that the rank order of hit rates closely followed the rank order of Relative Availabilities, suggesting that past hit rates might have less to do with teams ability to identify the best players at a position than with the diminishing availability of good players at that position as the draft progresses.

I teased that Relative Availability has special significance in analysis of draft hit rates and provides the key to untangling how hard it really is for teams to identify the best players at different positions. Now I will explain what I meant. Fortunately for most readers, I explained most of the methods in the previous article, so I have just appended a section with a few details at the end for anyone who is interested.

Photo by Christian Petersen/Getty Images

---

The Scouting Effect

Some statistically-minded readers may have figured out that the Relative Availability metric is the best estimate of the chance hit rate in draft selection. That is, if the next team on the podium was planning on selecting, say, a tight end, and instead of going on his scouting department’s recommendation, the GM decided to make his selection by throwing a dart at a board of all the available tight ends while blindfolded, his chance of picking a good one is approximately equal to the proportion of good tight ends in the total available talent pool.

Why is that? It has to do with a basic premise of Statistical Sampling Theory. Don’t worry. It’s not very complicated. Imagine you are picking potatoes without looking from a giant bin containing spuds of all different sizes, and the average weight of the potatoes in the bin is five ounces. If you keep sampling at random for a while, you will eventually accumulate a pile of spuds that have an average weight of five ounces. That is because a random sample drawn from larger population tends to reflect the frequency of occurrence of items in the larger population.

Returning to our Cerrato-esque GM, who is now on the board at pick number 16. In the past 10 years there have been 19 TEs picked at pick number 16 or later in the draft who met my threshold definition of “good” (year 3 AV > 5), out of a total of 148 TEs drafted at pick number 16 or later, for a Relative Availability = 19/148 = 0.128. If we assume that this TE class is like the average of the last 10 years, his chance of picking a good TE at random is 0.128 or 12.8%, which is the same as the Relative Availability.

Being able to estimate the expected chance hit rate is very useful, because it allows us to determine how much of the draft hit rate is not due to chance. Let me show you how that works. The following graph shows the Hit Rates (burgundy and gold) and Relative Availabilities (blue) for running backs with AV > 5 selected in the drafts from 2009 to 2018.

The Hit Rate starts out much higher than the chance level in the first half of the first round. It tanks in the second half of the first round, where only one out of eight RBs selected met the criteria, before recovering in the second and third rounds where much larger numbers of RBs were selected. By about the fourth round, the Hit Rate is dropping to near the chance level. The Hit Rate continues to drop, reaching the chance level in the seventh round.

The difference between Hit Rate and the chance level (Relative Availability) at each point in the draft is a quantity I will call the Scouting Effect. The Scouting Effect provides a metric to quantify the net effect of everything that teams do to elevate their draft success rate above chance, through scouting reports, film review, combine measurements, interviews and rankings. Another way to think of the Scouting Effect is as the Hit Rate, corrected by removing hits expected due to chance. Or alternatively, it can be thought of as the Hit Rate minus the confounding influence of Relative Availability of players.

The Scouting Effect is just what we need to compare the draftability of different position groups because it allows us measure draft success in a way that is unaffected by the differences in Relative Availabilities of good players at different positions. In the case of running backs, we can see that the scouting process gives teams a really good ability to pick the best players early in the draft. But the effect seems to wear off by around the fourth to fifth rounds. What about the other positions?

---

Comparing the Difficulty of Drafting Different Positions

The next figure shows the Scouting Effects for all the offensive position groups.

On first glance, that’s kind of a mess. But there are a few key points to focus on. First, I think most readers would have expected QB to be the hardest position to draft because the Hit Rates are so low. Once Hit Rates are corrected for differences in Relative Availability, however it becomes apparent that that is a false impression. In fact, through the second round, QBs have among the highest Scouting Effects.

The next surprise for me is that, with a few exceptions at a few points in the draft, all of the position groups are kind of similar. After looking at this graph for a while, the position group that seems to be most different from the others is WR, because it has the most flat distribution of Scouting Effect throughout the draft, starting lower than the other groups in the first half of the first round and finishing higher than the others in rounds five and six.

The third point that stands out to me is that, by the fourth round, draft success at all positions has got pretty close to chance. After the fourth round, only WR and RB still seem to be hitting above chance. This seems to suggest that, as the draft progresses into day three, the proportion of draft success that is due to teams’ expertise at draft selection, as opposed to pure luck, seems to fall to about zero.

To nail that down, as a last step, I calculated the Scouting Effects as a proportion of total Hit Rates (ie. Scouting Effect/Hit Rate). Just for fun, I’ll call that ratio Scoutability. The results are shown in the next figure.

That makes it fairly clear. Through the first three rounds, the proportion of draft success which is not due to chance is fairly high and fairly constant, and there are not huge differences between position groups, aside from the glitch with running backs in the second half of the first round. Contrary to expectations, by this measure, quarterbacks are the most scoutable of all the position groups.

In the fourth round, the proportion of draft success which can be attributed to teams’ expertise in player evaluation starts to fall, and the positions start to go their separate ways. By the fifth through seventh rounds, we are really looking at small differences between small numbers (Hit Rates and Relative Availabilities are both very low), so I am not sure how informative this representation is in that range.

---

Conclusion

To bring this all back to where it started, I think it’s fair to say that we are all accustomed to believing that quarterback is a particularly difficult position to pick correctly in the draft because the hit rates fall precipitously from the first overall pick, and fall to 30% by the second round. This analysis suggests that the problem is not with the ability of teams to distinguish good QB prospects from bad ones, so much as it is that there just aren’t many NFL-quality QBs to go around. The fact that Relative Availability of QBs falls so sharply actually demonstrates that, overall, teams are adept at spotting the few good ones in each draft class and quickly remove them from the available talent pool as the draft proceeds.

Quarterback does not appear to be the only position where draft outcomes are mainly dictated by availability of good players. To the contrary, the differences in draft hit rates between positions, that we are used to thinking of as reflecting differences in the difficulty of scouting different skill sets, appear to simply reflect the differences in availability of good players at different positions.

Once differences in availability are corrected for, by calculating the Scouting Effect and Scoutability metrics, it appears that evaluating talent is equally challenging at all offensive positions.

The other big surprise for me to come out of this analysis is that the Scouting Effect all but evaporates and draft success falls to near chance levels by the fourth round. My expectation heading into this exercise was that, if that occurred it would be in the sixth or seventh round. I still find it difficult to get my head around the notion teams only do slightly better than guessing on prospects in the fourth round. But that seems to be what the numbers show.

Finally, in closing I would like to point out that the Relative Availability concept, which provided the key to this and the previous QB Sweet Spot analysis, has other uses as well. One in particular, which I might revisit next draft season is that, if one were looking for a more rational basis for a draft trade value chart, Relative Availability-derived metrics would seem to be the right way to go.

Photo by Tim Warner/Getty Images

---

Method Details

The analysis included all offensive players drafted from 2008 to 2019, with draft position as assigned by the Pro Football Reference database. The criterion for being a “good” player was having an Approximate Value (AV) of 6 or greater in the third season of the player’s career.

Hit Rates were calculated as the number of “good” players selected at each position divided by the total number of players at the same position in the following ranges of draft picks: 1-8, 9-16, 17-32, then by draft round.

Relative Availability was calculated the total number of “good” players in a position group selected at a given pick number, or later in the draft, divided by the total number of players in the same position group selected at that draft pick or later. Relative Availability was sampled with greater resolution than Hit Rate, being sampled in a doubling sequence of pick numbers through the first round (pick numbers 1, 2, 4, 8, 16, 32) then every 16 picks up to pick 240.

The Scouting Effect was calculated as the difference between Hit Rate and average Relative Availability within the same range of draft picks. To calculate average Relative Availability, the Relative Availability curves for different positions were fit with curves as shown in the second figure of the previous article. The Relative Availability values were read off the best fit curves at the beginning and end of each draft pick range and averaged. Because the fourth through seventh draft rounds start at different pick numbers each year, due to comp picks, the endpoints of these rounds were calculated as the average starting and ending pick numbers from 2009 to 2018.

The Scoutability index was calculated by dividing Scouting Effect by total Hit Rate for each position and pick range.

Acknowledgement

Thanks to James Dorsett for editing. All statistics from Pro Football Reference.

---