How predictive are “consensus” draft big boards?

Recently, MattinBrisVegas has written a couple of articles post-draft that have evaluated the Commanders’ draft haul against the “consensus big boards” of draft prognosticators and “experts.” And while I have little doubt that the combined estimates of 80+ evaluators are probably more accurate than those of any individual evaluator, particularly over time, I was curious to look at how “the wisdom of the crowd” holds up against the least forgiving evaluator there is - reality.

For this exercise, I looked back at one of the earlier “consensus big boards” I could find, the 2019 big board from the Athletic. This board was comprised of 60 individual boards, and the article linked actually dives a bit more into the variability between those boards, and is an interesting read in it’s own right.

The table below shows the top 32 “best players available” according to the consensus big board as well as several other data points, including “weighted approximate value (wAV)” (basically a metric evaluating actual performance in the league), wAV Rank (which is the relative wAV of each player in that draft against their fellow draftees), “actual reach/steal delta (ARSD)” (which is difference between that players’ consensus rank, minus their actual relative standing in the class), and “reach/steal” (which is a determination of whether they exceeded or fell short of their position relative to the expert opinion before the draft).

So, let’s take the first player as an example. Before the 2019 draft, Alabama’s DL Quinnen Williams was the “consensus” number one best player available. After 3 years in the league, Williams has the 38th highest weighted approximate value (wAV) in his class. That, essentially, means that if the experts had their way - and they basically did, as Williams was taken #3 overall - Williams was a “reach” by 37 draft slots. That would make him the 10th highest “reach” of the top 32 players according to the experts, and would place him at an early second round talent, in a perfect draft world.

As a, rare, example in the opposite direction, let’s look at AJ Brown. Brown was deemed the 27th overall player by the experts, was actually drafted in the middle of the second round, and has actually performed as the third best player in the 2019 draft. Not only is he an actual “steal” according to the expert rank, he was a super steal compared to where he was actually drafted by the Titans, at #51.

I recognize there’s a lot to look at there, but I would like to call out some highlights. Below is a list of top 20 actual players whom the “consensus experts” left out of their first round talent list entirely. Interestingly, the list includes a wide range of positions, from QB to RB and WR, and even C.

Notables Misses (wAv Rank):

Deebo Samuel - 5

Maxx Crosby - 6

Daniel Jones - 8

Terry McLaurin - 12

Elgton Jenkins - 13

Diontae Johnson - 13

Kaleb McCary - 16

David Montgomery - 16

David Edwards - 16

Miles Sanders - 20

Devin Singletary - 20

Some other key elements that stand out from the analysis:

• Of the top 32 “consensus expert picks,” 23 of 32 (72%) have ended up being “reaches.” Remember, this is not relative to where they were actually picked in the draft. This is relative to where the experts had each of them as “best player available.”
• The average (mean) delta between where the experts had these players ranked and their actual performance is -24, meaning that, of the 32 players on this list, they were generally taken about 24 spots ahead of where they should have been, if teams were truly looking for the BPA. Of course, it should also be pointed out that the variance is pretty wild, going from the aforementioned steal - Brown - to the biggest “expert” reach of Dwayne Haskins, who probably should have been selected in the 3rd-4th round range. Thanks, Danny.
• I’ll take the opportunity to also mention that Montez Sweat was good value where he was actually drafted (#26) but would have shown as a “slight reach” if he had been taken where the experts wanted him (#10).

I look forward to hearing your thoughts on this review, and whether you find it, or the “consensus big boards” of draft “experts” compelling. Given this, albeit limited, analysis, I’m - personally - less concerned about what appears to be a half to ²⁄₃ round reach for a first round pick, as some have suggested Jahan Dotson is, given the average magnitude of expert misses against actual performance. And, I can only imagine that “miss range” tends to grow as the universe of players expands into the perceived second and third round talents.

I’d love to hear your thoughts below.

BONUS CONTENT:

After publication, poster MFTA raised the following in the comments:

Of the top 32 “consensus expert picks,” 23 of 32 (72%) have ended up being “reaches.”

That’s all you need to know about self-styled draft experts. They have some ballpark ideas about talent, sure, but the scouts and GMs making the actual picks have far different opinions about what constitutes value.

So I decided to dig back into the data to see if, indeed, that was the case. First, I looked at how closely the team’s actual picks (made by their GMs and scouts, assuming their idiot owner isn’t doing the picking), adhered to the consensus expert projections. Turns out, it’s actually quite close. On average, for these first 32 players, the actual picks in the draft only varied by about 5 draft slots per pick from the draftniks’ rankings. In my opinion, that’s a ridiculously tight alignment, and can probably easily be explained simply by the fact that teams don’t really get to pick where they select in the draft (with the exception of some ability to trade around).

How did the scouts and GMs fare against reality? Not much better than the experts - which might be expected given their close alignment. On average, the scouts and GMs were off by about 19 drafts slots against actual player performance. Recall, the draftniks were off by an average of 24 draft slots. More information to chew on in the comments section.

BONUS CONTENT, PART 2:

The same exercise, run for the 2018 draft can be found below:

Key figures are these. The consensus big board was an average reach of 27 slots versus actual performance. Scouts and GMs were about 8 slots better, with an average reach of 19 slots, so, basically, while the draftniks improved slightly from 2018 to 2019, the scouts and GMs were almost in exactly the same spot both years.