The statistical argument for trading down in the draft

I’m reaching into the distant past here — 2014 — to resurrect an article that was published by fivethirtyeight.com.

The article did a great job of putting together information from a number of statistical studies of NFL draft success. It looked at the information in many ways: the success of individual players versus draft position, the success rates of teams, the success of specific GMs or personnel executives, and so on.

I want to share a few ideas from that article, and discuss what they mean for NFL teams in general, and, perhaps, the Redskins in particular.

The article begins with an admission that the NFL draft is not exactly the same as the stock market (or other “efficient” markets):

The NFL’s draft market differs slightly from the financial markets Fama analyzed. There are legal opportunities for teams to gather inside knowledge through prospect workouts and interviews, which a buyer can’t do with stocks.³ But a large proportion of the information teams use to make their picks — tape of prospects’ college games, their college statistics, biometric data from the pre-draft combine — is available to every team. Teams, of course, differ in how they interpret this data, which is why not everybody wants the same players. That’s where teams’ scouting and, increasingly, quantitative analysis departments come in.

If certain teams had superior talent-evaluation abilities then we’d expect them to achieve a greater return on their draft picks than the average team, after adjusting for where the picks were made in the draft. But if the NFL Draft follows the same general guidelines financial markets do (at least, according to the efficient-market hypothesis), there wouldn’t be much of a relationship between a team or an executive’s drafting performance⁴ across multiple years’ worth of drafts.

The article relies heavily on the concept of Approximate Value (AV)

SportsReference.com has a clear explanation of the Approximate Value concept.

Created by PFR founder Doug Drinen, the Approximate Value (AV) method is an attempt to put a single number on the seasonal value of a player at any position from any year (since 1950).

Here’s the ‘conversational’ explanation of AV given by its creator:

“AV is not meant to be a be-all end-all metric. Football stat lines just do not come close to capturing all the contributions of a player the way they do in baseball and basketball. If one player is a 16 and another is a 14, we can’t be very confident that the 16AV player actually had a better season than the 14AV player. But I am pretty confident that the collection of all players with 16AV played better, as an entire group, than the collection of all players with 14AV.”

”Essentially, AV is a substitute for --- and a significant improvement upon, in my opinion --- metrics like ‘number of seasons as a starter’ or ‘number of times making the pro bowl’ or the like. You should think of it as being essentially like those two metrics, but with interpolation in between. That is, ‘number of seasons as a starter’ is a reasonable starting point if you’re trying to measure, say, how good a particular draft class is, or what kind of player you can expect to get with the #13 pick in the draft. But obviously some starters are better than others. Starters on good teams are, as a group, better than starters on bad teams. Starting WRs who had lots of receiving yards are, as a group, better than starting WRs who did not have many receiving yards. Starters who made the pro bowl are, as a group, better than starters who didn’t, and so on. And non-starters aren’t worthless, so they get some points too.”

That’s as technical as I want to get on the concept of Approximate Value. If you love the “Gruesome Details” then feel free to click here and read about them. Unless you love numbers in a special way, though, this site may be a no-go zone.

Let me just put my own words to what Doug Drinen has already said. Approximate Value is a statistical attempt to measure and compare the relative quality of NFL player performance on the field. In any given player-to-player situation, AV may not hold up to scrutiny, but when used to evaluate large groups of players, statisticians (and fans) should find a high level of reliability in the numbers.

The simple concept is that higher AV means better player performance, and, of course, lower AV means weaker player performance.

Okay... so what does Approximate Value have to do with the NFL Draft, again?

Somebody sat down one day with an abacus, a pencil and a ruler and did some work. For each drafted player from 1994 to 2013 (that’s 20 years of data on 256 players per year), the average Approximate Value of each player was calculated for his first 5 years in the league. If my abacus isn’t broken and there weren’t too many exceptions, then we’re talking about more than 5,000 players comprising over 25,000 seasons’ worth of data.

Remember how we were told that Approximate Value may not work for player-versus-player comps, but it was really good when you applied it to large groups? Well, 5,000 players over 20 years is the kind of group where the Approximate Value measurement should shine.

The guy with the abacus took his pencil and ruler and a piece of graph paper, and he put together a chart. For each player, he charted their draft position (1-256) on the x-axis, and the player’s Approximate Value on the y-axis.

The result looks like this:

Where I grew up, we used to call that strong correlation.

This chart shows us exactly what we should expect to find — that across this large sample of players, the earliest draft picks have the highest Approximate Value, and the Approximate Value of players falls as we move through the draft and reach the later picks.

You can still look at the dots on the chart and find players taken with the 250th pick who have higher AV than players taken 100 picks earlier, but the data is clustered pretty tightly around the red line, with minimal outliers.

No surprise here — early first round draft picks tend to be the NFL’s best performers, while the AV measure of on-field performance tends to tail off for later round picks.

Alright, that’s 1,000 words to tell me what I already knew by applying common sense. Is there a point here?

The fivethirtyeight.com article covers some interesting ground in which they show that teams and personnel people tend to be unable to outperform the market. In other words, over a period of three years or more, there’s little statistically identifiable difference between the best teams or GMs and the worst teams and GMs in terms of being able to draft more effectively.

What I’m saying in such a painfully roundabout way is that it doesn’t really matter who does the picking — NFL execs all get pretty much the same result. First round picks tend to score really high on the AV measure while later round picks don’t.

It might sound like I’m suggesting that if you put Howie Roseman, Doug Williams, John Lynch, Mike Mayock, and a monkey with a dartboard in charge of the draft, they’d all get the same results, but that’s not right.

The research suggests that the reason why no individual or organization consistently outperforms the others in the draft is that they are all skilled, all apply rigorous discipline to the process, and all have good access to information. Thus, the monkey with the dartboard wouldn’t get the same results, but NFL personnel execs, who all have roughly the same skill set, knowledge and professional practices, will tend to all get roughly the same results, which are displayed on the graph above.

But there is a “but”...

But...

There is a mismatch, according to the article, between the actual skill of the NFL personnel executives and how skilled they think they are at picking winners at the top of the draft.

NFL personnel executives think that, if they have a top-50 pick, then they can outperform the market and grab a more talented player. Further, the closer that pick is to #1 in the draft, the higher the confidence level of the personnel executives.

This leads to a mis-match between measurable results and perceived value of early draft picks.

NFL personnel executives, according to the 538 article, overvalue the top 50 picks in the draft because they feel that the closer they are to #1, the better chance they have of hitting a home run.

They are right about the relationship — top draft picks perform better — but they misjudge the degree of difference and, thus, the value of those picks.

How do we know this? Because of the famous “draft value chart” that was introduced by Jimmy Johnson so many years ago and updated over time.

As Massey and Thaler point out, the more that teams study players and gather information about them, the more assured they become in their ability to differentiate among prospects of roughly the same talent level. This leads to overconfidence, and the tendency to make what they call “non-regressive predictions” — forecasts that don’t appropriately account for the uncertainty in projecting college players’ performance in the NFL — about the future value of potential draftees.

This isn’t hard to show empirically, either. After examining 1,078 draft-pick swaps between 1983 and 2008, Massey and Thaler found that teams’ behavior when trading picks corresponds incredibly well to the famous draft-value chart popularized by former Dallas Cowboys and Miami Dolphins coach Jimmy Johnson.

The abacus guy got fancy and put an overlay of the NFL Draft Value Trade Chart on top of the red line we looked at above.

Here’s the result:

What you can see is that, at around pick #50, and again around pick #250, the two charts agree on the relative value of the picks.

But it shows that the NFL Trade Value Chart overestimates the value of the first 50 (or so) picks in the draft — and that the effect is more and more exaggerated the closer we are to the first pick in the draft.

This indicates that the NFL Trade Value Chart overestimates the value of the #1 pick by about 300%.

Conversely, the NFL Trade Value Chart, according to this analysis, overestimates the fall in value that occurs in the middle rounds. Draft picks that fall in the 3rd to 6th rounds are undervalued by the Chart, meaning that they are actually more valuable to teams (relatively speaking) than the chart would indicate, while top-50 picks are less valuable than the Chart says they are.

NFL decision-makers place an incredible premium on high draft picks. But the huge disparity between the observed performance of each pick and its apparent market value supports Massey and Thaler’s hypothesis that teams are not being realistic about their own ability to differentiate among prospects.

They should be. Research by TheBigLead’s Jason Lisk (then writing for Pro-Football-Reference) shows that teams with top-five picks in the draft correctly identify the player who goes on to have the best career only 10.3 percent of the time, a success rate that only gets worse as things progress deeper into the draft.¹¹ So a team that believes it could somehow beat the market if only it controlled its own fate can end up doing more harm than good if it trades away lower picks to move up in the draft. This is especially the case if a team uses Johnson’s unrealistically optimistic chart as justification for such behavior.

In the first two rounds, trade back

The general rule of thumb from this analysis is that teams with first round or early second-round picks should be happy to trade down in the draft if the terms of the trade reflect the NFL Trade Value Chart, and that teams holding picks in the middle rounds (3 - 6) should be hesitant about trading them away to move into the top-50 if the terms of the trade are based on the Chart.

So, what does that say about suggestions that the Redskins trade for the Jets 3rd overall pick in the 2019 draft?

This analysis screams “Don’t do it!!”

At least not if the trade follows the values in the Chart.

The #15 pick may be have an artificially inflated value on the Chart, but the #3 pick will be wildly inflated.

Even without knowing who the Redskins would take with that pick (the recent buzz has been Haskins, but that can change quickly) the analysis says don’t do it — not because of the specific skills of the player, but because of the relative value of the picks involved in the trade.

Redskins strategy - 1st round

If anything, the Redskins should seek to trade down from their 15th overall pick, hopefully adding some under-valued 3rd and 4th round selections.

Redskins strategy - 2nd round

The 46th overall pick, by contrast, is pretty much fairly valued by the Chart, so this analysis would suggest that the Redskins hold onto that one, or trade down no more than, say, a dozen spots in the draft. They don’t want to use the Chart values to trade out of the second round.

Redskins strategy - middle rounds

The 3rd, 5th and 6th round picks (#96, #153, #173, #206) can be traded, but NOT to move up into the top of the draft. Doing that would mean too much lost value. These picks should only be traded for other mid-round picks, which are all pretty much equally undervalued on the Chart.

Overall strategy

The Redskins need to fill roster holes, which means they are hungry for draft picks. If there is an opportunity to parlay the #15 pick into a later 1st rounder + some 3rd or 4th round supplemental picks, the Redskins should do that — if the terms of the trade follow the chart.

By way of example:

The chart says that #15 is worth 1050 points.

Trading with the Tennessee, say, would bring

• the Titans’ 1st round pick #19 (875 points)
• the Titans’ 3rd round pick #82 (180 points)

NFL executives overvalue first round draft picks. Bruce, Doug and Jay should know that and use it to help build the team.

---

You can follow this link to an interactive model of the 2019 version of the NFL Trade Value Chart

---