clock menu more-arrow no yes

Filed under:

How predictive are “consensus” draft big boards?

It turns out, not very.

If you buy something from an SB Nation link, Vox Media may earn a commission. See our ethics statement.

Buffalo Bills v New England Patriots Photo by Maddie Malhotra/Getty Images

Recently, MattinBrisVegas has written a couple of articles post-draft that have evaluated the Commanders’ draft haul against the “consensus big boards” of draft prognosticators and “experts.” And while I have little doubt that the combined estimates of 80+ evaluators are probably more accurate than those of any individual evaluator, particularly over time, I was curious to look at how “the wisdom of the crowd” holds up against the least forgiving evaluator there is - reality.

For this exercise, I looked back at one of the earlier “consensus big boards” I could find, the 2019 big board from the Athletic. This board was comprised of 60 individual boards, and the article linked actually dives a bit more into the variability between those boards, and is an interesting read in it’s own right.

The table below shows the top 32 “best players available” according to the consensus big board as well as several other data points, including “weighted approximate value (wAV)” (basically a metric evaluating actual performance in the league), wAV Rank (which is the relative wAV of each player in that draft against their fellow draftees), “actual reach/steal delta (ARSD)” (which is difference between that players’ consensus rank, minus their actual relative standing in the class), and “reach/steal” (which is a determination of whether they exceeded or fell short of their position relative to the expert opinion before the draft).

So, let’s take the first player as an example. Before the 2019 draft, Alabama’s DL Quinnen Williams was the “consensus” number one best player available. After 3 years in the league, Williams has the 38th highest weighted approximate value (wAV) in his class. That, essentially, means that if the experts had their way - and they basically did, as Williams was taken #3 overall - Williams was a “reach” by 37 draft slots. That would make him the 10th highest “reach” of the top 32 players according to the experts, and would place him at an early second round talent, in a perfect draft world.

As a, rare, example in the opposite direction, let’s look at AJ Brown. Brown was deemed the 27th overall player by the experts, was actually drafted in the middle of the second round, and has actually performed as the third best player in the 2019 draft. Not only is he an actual “steal” according to the expert rank, he was a super steal compared to where he was actually drafted by the Titans, at #51.

2019 Consensus Big Board versus Actual Performance

Consensus Rank Player Position wAV wAV Rank ARSD Reach/Steal
Consensus Rank Player Position wAV wAV Rank ARSD Reach/Steal
1 Quinnen Williams DL 16 38 -37 Reach
2 Nick Bosa EDGE 24 7 -5 Reach
3 Ed Oliver DL 23 11 -8 Reach
4 Josh Allen EDGE 15 40 -36 Reach
5 Devin White LB 32 2 3 Steal
6 Brian Burns EDGE 19 22 -16 Reach
7 TJ Hockenson TE 13 51 -44 Reach
8 Jonah Williams OT 13 52 -44 Reach
9 Kyler Murray QB 44 1 8 Steal
10 Montez Sweat EDGE 19 22 -12 Reach
11 Jawaan Taylor OT 19 22 -11 Reach
12 Devin Bush LB 17 33 -21 Reach
13 Christian Wilkins DL 21 16 -3 Reach
14 Jeffrey Simmons DL 24 9 5 Steal
15 DK Metcalf WR 29 4 11 Steal
16 Dwayne Haskins QB 4 131 -115 Reach
17 Noah Fant TE 14 46 -29 Reach
18 Rashan Gary DL 12 58 -40 Reach
19 Andrew Dillard OT 6 105 -86 Reach
20 Clelin Ferrell EDGE 11 68 -48 Reach
21 Greedy Williams CB 7 98 -77 Reach
22 Byron Murphy CB 15 43 -21 Reach
23 Garrett Bradbury OC 19 22 1 Steal
24 Cody Ford G 13 52 -28 Reach
25 Josh Jacobs RB 22 13 12 Steal
26 Jerry Tillery DL 14 47 -21 Reach
27 AJ Brown WR 30 3 24 Steal
28 Marquise Brown WR 24 9 19 Steal
29 Dexter Lawrence DL 19 22 7 Steal
30 DeAndre Baker CB 6 106 -76 Reach
31 Dalton Risner G 17 35 -4 Reach
32 N'Keal Harry WR 6 107 -75 Reach

I recognize there’s a lot to look at there, but I would like to call out some highlights. Below is a list of top 20 actual players whom the “consensus experts” left out of their first round talent list entirely. Interestingly, the list includes a wide range of positions, from QB to RB and WR, and even C.

Notables Misses (wAv Rank):

Deebo Samuel - 5

Maxx Crosby - 6

Daniel Jones - 8

Terry McLaurin - 12

Elgton Jenkins - 13

Diontae Johnson - 13

Kaleb McCary - 16

David Montgomery - 16

David Edwards - 16

Miles Sanders - 20

Devin Singletary - 20

Some other key elements that stand out from the analysis:

  • Of the top 32 “consensus expert picks,” 23 of 32 (72%) have ended up being “reaches.” Remember, this is not relative to where they were actually picked in the draft. This is relative to where the experts had each of them as “best player available.”
  • The average (mean) delta between where the experts had these players ranked and their actual performance is -24, meaning that, of the 32 players on this list, they were generally taken about 24 spots ahead of where they should have been, if teams were truly looking for the BPA. Of course, it should also be pointed out that the variance is pretty wild, going from the aforementioned steal - Brown - to the biggest “expert” reach of Dwayne Haskins, who probably should have been selected in the 3rd-4th round range. Thanks, Danny.
  • I’ll take the opportunity to also mention that Montez Sweat was good value where he was actually drafted (#26) but would have shown as a “slight reach” if he had been taken where the experts wanted him (#10).

I look forward to hearing your thoughts on this review, and whether you find it, or the “consensus big boards” of draft “experts” compelling. Given this, albeit limited, analysis, I’m - personally - less concerned about what appears to be a half to 23 round reach for a first round pick, as some have suggested Jahan Dotson is, given the average magnitude of expert misses against actual performance. And, I can only imagine that “miss range” tends to grow as the universe of players expands into the perceived second and third round talents.

I’d love to hear your thoughts below.

BONUS CONTENT:

After publication, poster MFTA raised the following in the comments:

Of the top 32 “consensus expert picks,” 23 of 32 (72%) have ended up being “reaches.”

That’s all you need to know about self-styled draft experts. They have some ballpark ideas about talent, sure, but the scouts and GMs making the actual picks have far different opinions about what constitutes value.

So I decided to dig back into the data to see if, indeed, that was the case. First, I looked at how closely the team’s actual picks (made by their GMs and scouts, assuming their idiot owner isn’t doing the picking), adhered to the consensus expert projections. Turns out, it’s actually quite close. On average, for these first 32 players, the actual picks in the draft only varied by about 5 draft slots per pick from the draftniks’ rankings. In my opinion, that’s a ridiculously tight alignment, and can probably easily be explained simply by the fact that teams don’t really get to pick where they select in the draft (with the exception of some ability to trade around).

How did the scouts and GMs fare against reality? Not much better than the experts - which might be expected given their close alignment. On average, the scouts and GMs were off by about 19 drafts slots against actual player performance. Recall, the draftniks were off by an average of 24 draft slots. More information to chew on in the comments section.

BONUS CONTENT, PART 2:

The same exercise, run for the 2018 draft can be found below:

2018 Consensus Big Board versus Actual Performance

Consensus Rank Player Position wAV wAV Rank ARSD Reach/Steal
Consensus Rank Player Position wAV wAV Rank ARSD Reach/Steal
1 Saquon Barkley RB 28 14 -13 Reach
2 Quenton Nelson G 50 4 -2 Reach
3 Brad Chubb EDGE 23 28 -25 Reach
4 Josh Rosen QB 3 181 -177 Reach
5 Roquan Smith LB 39 8 -3 Reach
6 Minkah Fitzpatrick S 38 9 -3 Reach
7 Derwin James S 17 55 -48 Reach
8 Sam Darnold QB 22 36 -28 Reach
9 Denzel Ward CB 24 23 -14 Reach
10 Tremaine Edmunds LB 39 8 2 Steal
11 Baker Mayfield QB 42 5 6 Steal
12 Vita Vea DL 24 23 -11 Reach
13 Harold Landry EDGE 28 14 -1 Reach
14 Derrius Guice RB 2 194 -180 Reach
15 Isaiah Wynn G 16 62 -47 Reach
16 Jaire Alexander CB 22 36 -20 Reach
17 Mike McLinchey T 23 28 -11 Reach
18 Josh Jackson CB 7 132 -114 Reach
19 Maurice Hurst DL 9 118 -99 Reach
20 Calvin Ridley WR 26 19 1 Steal
21 Lamar Jackson QB 59 1 20 Steal
22 Rashaan Evans LB 25 20 2 Steal
23 Connor Williams T 22 36 -13 Reach
24 Daron Payne DL 27 15 9 Steal
25 Will Hernandez G 23 28 -3 Reach
26 Marcus Davenport EDGE 13 78 -52 Reach
27 Josh Allen QB 51 3 24 Steal
28 Taven Bryan DL 10 103 -75 Reach
29 James Daniels CB 18 50 -21 Reach
30 Leighton Vander Esch LB 27 15 15 Steal
31 Courtland Sutton WR 20 43 -12 Reach
32 DJ Moore WR 33 11 21 Steal

Key figures are these. The consensus big board was an average reach of 27 slots versus actual performance. Scouts and GMs were about 8 slots better, with an average reach of 19 slots, so, basically, while the draftniks improved slightly from 2018 to 2019, the scouts and GMs were almost in exactly the same spot both years.

Poll

What do you think about the value of "consensus big boards."

  • 5%
    They generally seem to have a strong relationship to actual performance.
    (16 votes)
  • 60%
    They’re a fun diversion, like astrology.
    (169 votes)
  • 33%
    They’re largely worthless at predicting actual outcomes.
    (93 votes)
278 votes total Vote Now