The Difference Between a Trade Value Chart and a Player Grade

There has been a lot of discussion on here about draft value and whether or not to trade down from #2 overall or take Chase Young. As often happens on Hogs Haven with a topic of contention, the debate has gone back and forth, and although no clear consensus has been reached, many good points have been made on both sides. I want to frame the discussion in a bit of a different light and make sense of my agreement with points being made on both sides. It really all hinges on a statistical valuation of draft picks (trade value chart) vs a specific valuation of draft picks (player grades).

A statistical approach uses lots of historical data to identify trends and assign probabilities based on what happened in the past over many cases. A specific approach involves taking a specific case and predicting what will happen based on a careful evaluation of that one case. To give a medical example, statistical studies might inform a doctor that their patient is in a high risk category for a disease based factors like age, diet, lifestyle, etc. But a doctor would never (I hope) tell their patient “I think you have cancer” simply based on statistical factors like that. Instead, if the statistical risks were high enough, they would ask questions and order tests that could better diagnose the presence of a disease in that specific patient. Both sets of information are useful, they just get used for different things.

Statistical value and the Trade Value Chart

Bringing it back to football, the statistical way to value draft picks is with a trade value chart (TVC). There are several such charts out there, but I’ll refer to the Jimmy Johnson Trade Value Chart because it’s also used in some of the examples I’ll mention and is commonly referenced as a standard for valuing draft picks. An example of the TVC is shown below.

via hambotalk.files.wordpress.com

MattInBrisVegas has done an excellent series of articles (article 1, article 2, article 3) investigating how well the TVC correlates with the actual ability of players taken at various positions in the draft. If you haven’t read the articles, you should, they’re illuminating. Matt investigated several measures of player ability and found that the chart actually correlated very strongly with the statistical probability that a player taken at that spot will be “elite” (defined as CarAV > 65, see the 3rd article for the details). The results of Matt’s analysis showing number of elite players taken at a given draft position superimposed over the scaled TVC value of the draft pick is shown below.

Plot by MattInBrisVegas, Hogs Haven

The correlation is so strong that I don’t think it can be coincidence. I suspect the TVC was crafted using a similar type of analysis and plotting the number of elite players in history taken at each draft position, then scaling it up to develop the “value” represented by each draft pick. They may have used a different definition of “elite”, but the results were the same. As such, I think this chart represents an excellent statistical valuation of draft picks.

The process and importance of scouting

So what do NFL scouts do exactly? There’s a series of articles on Bleacher Report by Greg Gabriel, who worked in several NFL front offices at positions as high as Director of College Scouting. In these articles he describes in a fair amount of detail how NFL teams scout and assess individual player talent and much of what he says sounds similar to what we know of the Redskins front office. I’ll try to summarize what I think are the most relevant parts for this article.

According to Gabriel, the college scouting process starts with area scouts. His team had 6 area scouts, each responsible for all the colleges in an “area” of the country (Northeast, Southeast, West Coast, etc). The areas were divided such that each area scout was responsible for 15 or so “major” schools and several more “mid-major” and “small” schools. Each scout visited the major schools at least 3 times per year, the mid-major schools at least twice a year (more if they had a top prospect), but would only visit the small schools if a coach told them about a legitimate prospect there. The bulk of the scout’s time was spent reviewing film on players to assess their talent and speaking with people who know them (coaches, teachers, teammates, etc) to assess their personal and football character. The character assessment was the most important duty of an area scout, because other people in the FO would cross-check the film, but only the scout knew the player well enough to assess their character.

Gabriel reiterated a theme that Kyle Smith has mentioned repeatedly that character is the most important trait in determining whether or not some of these players develop in the NFL or fail out of it. Gabriel felt the assessment of character was so important it deserved it’s own article, linked here. I won’t go into those details, but one takeaway from everything so far is how important an area scout’s relationships and connections are with the area he scouts. The scout relies on relationships with the small school coaches to tell him when they have potentially draftable players, as well as relationships with the coaches, teachers, and players in the larger schools to get accurate assessments of their character and understand things in context of the history of their teams. This is why area scouts usually stay scouting the same area for as long as possible and completely overhauling a front office (including firing the scouts) can be very disruptive.

Getty Images

Each week the scouts would tell the Director of College Scouting about any players they thought they should see, namely quality players that played a position of need and were fits to the offensive and defensive system. Starting in December, the cross-check phase began, where the scouting director and/or GM would meet with their scouts to go through all their prospects and cut a group of 1500 or so players down to 500. Most of the cuts were made based on character concerns or poor system fit. Each of the scouts would also be an expert in a specific position group (WR, TE, CB, etc), and the players would be re-assigned to the expert for their position group to re-evaluate their tape and re-assess their talent as a cross check.

Those position group experts would also have a close working relationship with their team’s position coaches in order to make sure they understood all the traits those coaches value. When the scouts then go to All Star games and the Combine, they focus on the players in their position groups in order to rank them. They also work with the team’s position coaches to get the coach’s assessment of the top 15-20 players in each position group. Gabriel stressed that the relationship between the position scout and the position coach was critical in getting players that would work well with the coach, as he had seen players that didn’t develop because they never got along with their position coach. This scout/coach familiarity is another reason why continuity of both coaching and scouting staff is important and turnover can be disruptive.

AP Images

After the Combine and Pro Days, there is a final elimination phase. Players are eliminated due to medical issues discovered at the Combine, poor athletic testing numbers, or newly discovered character concerns. After final cuts, the team’s big board will comprise around 120 players. The very last step in the process is assigning grades to each player on the board, which is done in the final weeks leading up to the draft once all the information is in.

I realize this is a lot to take in, but I wanted to go into this detail to show how painstaking the NFL scouting process is when done properly. The grades that get assigned to players are not arbitrary numbers that reflect the gut feel of a GM from watching him in a few games, but rather the culmination of a year of scouting around 1500 players, learning everything that is going on in their lives as well as studying film of them, cross-checking with other scouts and coaches, whittling that group down to around 120 players, then working to make sure the evaluations of those players are as accurate as possible. As such, they enable the best specific valuation of players that a team has.

As an aside, I also want to point out that not every NFL team operates this way, particularly the dysfunctional ones. One look at the Redskins draft history from 2005 to 2011 will show just how bad we have been at drafting players during long stretches of time where literally none of our picks outside of the 1st round turned into NFL starters. After reading through this section, hopefully it’s clear some of the reasons this could happen. Some teams are cheap, only hiring a small handful of scouts to cover the entire country, thus forcing those scouts to spend less time developing relationships and studying players at any one school and worsening their ability to evaluate talent (I’ve heard this about the Bengals). Some teams have perpetual turnover, firing coaches and front office personnel every few years and losing all the institutional knowledge and relationships they had built (the Browns have had 6 different GMs and 6 different HCs since 2012!). Some teams have poorly managed scouting departments, lacking the cross checks and close relationship with coaches that I described above (former Jets GM Mike Maccagnan refused to let coaches interact with scouts for fear of “biasing them”, so the scouts couldn’t assess system fit or coach/player compatibility). And some teams have arrogant owners or Team Presidents who choose to ignore the scouting recommendations and go with their gut valuations of players because they don’t respect the work that has been done for them by the scouts (you can guess who I’m referring to here).

Another point I want to make here is that this is a process that transcends any single front office official. Good player evaluations come from a good process, not a good talent scout. No matter how good a talent evaluator Kyle Smith is, he simply doesn’t have time in his day to investigate this many players and determine their value by himself. Instead, it is much more important that he set up a process like the one described that will empower the people around him to do that work effectively. I have long thought this was the failing of former GM Scot McCloughan, who was a much better talent scout than CEO of a scouting department. I suspect Kyle Smith’s success comes from not just being a good talent evaluator, but also a good communicator and organizer.

Specific value and player grades

The culmination of this intensive scouting process is the development of player grades for each of the 120 or so players on the team’s final big board. These grades represent the sum total knowledge about a player and their value within the team. It’s important to note here that the grade is not just about the raw talent of the player, but also their team fit. A smart, athletic offensive lineman like Ezra Cleveland will have a much higher value to a zone blocking team, whereas a bigger, stronger, but slower-footed offensive lineman like Isaiah Wilson will have a much higher value to a power blocking team. This is one of many reasons that different teams can have very different grades on players and is also why generic online player grades by draft “experts” will never reflect a player’s value to a specific team.

There are two different standard systems for grading players: a color-based system and a number-based system. The color-based system is older and the scale generally looks like the following:

A number-based grading system assigns a number grade to each player. It has the obvious advantage that it can have much finer-grained detail in assigning grades and it also lends itself readily to mathematical operations to determine relative value in trades (I don’t know how often this is actually done, it’s pure speculation on my part, but it makes sense). An example of a number-based grading scale that is used by Lance Zierlein of NFL.com is:

Notice that it is not a perfectly continuous number scale as there is no possibility to give a 6.9 or 7.2 grade, for example. Instead, it is designed to group players into tiers, which I’ve shown with boxes. This accentuates the difference between elite players, good players, starters, and developmental/backup-level players, while still allowing you to quantify differences within a tier (unlike the color scale). In a lot of ways, it is superior to the color scale and Kyle Smith mentioned during his Combine presser that the primary change he made to the Pro Personnel scouting department was switching them from a color-based grading system to a number-based grading system. A number grade applied to a player then represents the specific value of that player as seen by a given team.

How should these different tools be used?

Remember the analogy of the doctor trying to check the health of his patient. The statistical indicators (general risk factors based on age, diet, etc) give the doctor a good initial idea of what to be concerned about, but the doctor can’t make a diagnosis without specific indicators (positive blood tests and the like). Our equivalent is the TVC and the player grades, so how do we use them? I think there are two straightforward situations where one of these tools makes far more sense than the other.

• In determining which player to select when you’re on the clock, use the player grades. This seems obvious, but might as well spell it out. A statistical valuation doesn’t tell you which specific player to choose, that’s what the players grades are for.
• In determining fair trade value involving future draft picks or a trade up, use the TVC. If the draft pick in question is in a future round of the draft or even in a future draft, there’s no way to invoke player grades because you don’t know which players will be available at those picks. Therefore, a statistical valuation makes the most sense. Additionally, you should use the TVC as the basis for the initial offer that you make when proposing a trade up with another team, because you don’t know their player grades, which is what they will be using to determine the value of the trade. If they demand more compensation, your own player grades should set the maximum amount you’d be willing to offer (as described below).

Unfortunately, there is a third situation that arises, and that is where things get complicated:

• In determining fair trade value for a pick that is about to be made, use both the TVC and your player grades. In our case, the question isn’t “what’s a fair trade for a generic #2 pick?” The question is rather “how much is enough to pass on Chase Frickin’ Young?” You can substitute Tua, Okudah, or whomever is your highest graded player on the board. Either way, we shouldn’t just use a TVC value for that pick when we went through all the time and effort of determining the specific value of the player we’d lose the opportunity to draft. If our scouts think he’s the next Lawrence Taylor, we’d be fools to give him up so easily. This situation is complicated enough to deserve its own section.

How much is enough to pass on Chase Frickin’ Young?

In order to answer this, I think we should first answer what NFL teams are trying to do in the draft. Are they trying to fill holes on their roster with average players? No, they are trying to maximize the advantage their team has by getting elite players on cheap rookie contracts. If you remember Matt’s analysis on the TVC, he found it correlated most strongly with the probability of getting an elite player at that draft position. Not a good player, not a replacement-level player, but an elite player. The TVC curve would be much flatter if it were geared towards getting players who are just okay. Getting elite players was Jimmy Johnson’s goal in the draft and that should be the goal of every team that uses his value chart. Obviously teams know that not every player they draft will be elite, but the point is that they are trying to maximize talent in the draft in order to maximize the competitive advantage of having high-level players on cheap rookie contracts. They can fill roster holes in free agency with mid-tier veterans who are known, proven commodities with a much higher floor than draft picks. Players taken in the draft bust and wash out of the league all the time, so it’s very risky leaving holes on your roster to be filled by rookies in the draft. Better to fill those holes with dependable veterans and let the draft come to you.

Armed with the knowledge that the goal is to maximize talent and get elite players with our draft picks, the most obvious situation in which we’d trade down is if we think we’ll still be able to get an equally talented player at our new pick. As Ron Rivera said:

If you’re gonna make a trade, and you’re gonna go back, that guy you’re gonna take at that spot has to be able to make the kind of impact you need to validate missing an opportunity to take a player that’s a high impact guy. In other words, if you’re going to pass up player A, and you go back and you’re going to take player D, player D has to be equal to player A.

So for example, imagine the following player grades for two teams (note that I’ve grouped them by tier as above):

If the Redskins draft board matches that of Team A, then they don’t see any of these players as truly elite. Rather, they see Chase Young, Jeff Okudah, Isaiah Simmons, and Jedrick Wills as good, Pro Bowl level talents and some are better than others, but they are all in the same tier. In that case, if the Dolphins called and offered picks 5 and 18 (or even 5 and 26, if they are driving a hard bargain), it would make sense to accept that offer because at worst you will be able to draft a Pro Bowl level player in Wills (and most likely the Dolphins will use their pick on a QB, so either Okudah or Simmons will drop to 5). If the Lions call and offer picks 3 and 35 to trade up one spot, then you absolutely accept that offer because you will still be able to draft Okudah at 3 and he is graded very close to Young on your board, so you’re basically getting a free 2nd round pick.

However, the decision calculus is completely different if the Redskins draft board matches that of Team B. Here, they have Young graded as the only elite, perennial All Pro level player in the draft. Any trade down means losing an entire tier of talent at the very least. A team could still offer enough trade value to make it worth trading down, but it would have to be a LOT more than in the first situation. How much better is it to have Derrick Henry instead of Mark Ingram? Julio Jones instead of Amari Cooper? Aaron Donald instead of Akiem Hicks? Stephon Gillmore instead of Byron Jones? That’s the talent difference between those two tiers. If you’re having to pass on the chance to get one of those elite players on a rookie contract, you would have to be offered a LOT of value in return! Incidentally, the draft board for Team B happens to use the exact player grades Lance Zierlein gave these players in his prospect evaluations (the board for Team A is just made up). Zierlein, at least, thinks Young is the only elite player in this draft.

Trust your process and your board!

I can already hear an argument forming from some people that these player grades are fine, but even good teams are wrong on players all the time. Because the bust rate on players in the draft is so high, you should just trade back for “as many bites at the apple as possible.” To which I say what I’ve heard numerous front office personnel say when interviewed: you have to trust your board! If you don’t have some way of valuing these players individually to guide your draft decisions, then the draft is just a chaotic mess and you are piloting a ship without a rudder. Rather than getting more “bites at the apple,” a team that doesn’t trust its board to make these decisions and trades down blindly is like a dart thrower who chooses to throw three darts in the dark instead of one dart with the lights on. You get more chances, but you have nothing to guide any of them.

The first step for any front office is developing a scouting and player evaluation process they can trust, like the one outlined above. Dysfunctional teams may never get there, but they have to try. Once that is accomplished, they have to trust the results of that process when they go into the draft. They will undoubtedly miss on some of their picks, that’s just the way it is. Every miss is an opportunity to learn and improve the process next time, it shouldn’t leave you paralyzed with indecision.

Lastly, I want to offer up two brief cautionary examples of what happens when a team is good at one of these methods of valuation, but not the other.

What happens when you’re good at getting statistical value, but bad at evaluating player value?

Sashi Brown was effectively the general manager of the Cleveland Browns from 2016 to 2017, controlling the draft in both of those years. Despite having a legal rather than a football background, Brown was regarded as a very smart, analytics-oriented executive who intended to bring a “Moneyball” approach to the organization governed by maximizing the statistical value of their draft picks and player contracts. In his first year as an executive, he immediately began purging expensive veterans from the roster, trading them for draft capital when possible, cutting them when not. He also entered the 2016 draft with the number 2 overall pick in the draft and a strong intention to trade down to accumulate more draft picks. He ended up trading that #2 pick down with the Eagles for a return of their #8 and #77 picks in that draft, plus a 2017 1st round pick and 2018 3rd round pick. He then continued trading down in that draft, occasionally trading up, which makes the final tally difficult to estimate, but best estimates are that he turned that #2 pick into the following 11 players:

Corey Coleman (now on the Giants)

Shon Coleman (now on the 49ers)

Cody Kessler (currently a FA)

Derrick Kindred (now on the 49ers)

Spencer Drango (currently a FA)

Ricardo Louis (now on the Dolphins)

Jordan Payton (currently a FA)

Jabrill Peppers (currently on the Giants, though part of the trade that netted Odell Beckham)

Deshone Kizer (now on the Raiders)

Denzel Ward (still on the Browns, though picked by John Dorsey in 2018)

Chad Thomas (still on the Browns, also picked by John Dorsey in 2018)

Amazingly, Sashi Brown managed to turn the #2 pick into 11 players and none of the players he took are still on the team after only 4 years. The only draft picks he accumulated that are still on the Browns were picked by John Dorsey after Brown was fired. Meanwhile, the Eagles used that #2 pick to draft Carson Wentz, who looks to be their franchise QB. Sashi Brown made other questionable football decisions, but that single trade might be the best example of maximizing statistical value while returning very little real-world value due to poor player evaluations. In the end, Sashi left his team with a lot of draft capital and a ton of cap space, but almost completely bereft of talent.

What happens when you’re good at evaluating players, but bad at maximizing overall value?

Dave Gettleman is an old-school football GM who likes to build through the draft. Now don’t get me wrong, I’m not saying he’s great at evaluating players, but I think that is at least his focus and he isn’t terrible at it (unlike Sashi Brown). His focus is certainly not on accumulating draft capital, because he has never traded down in the 7 drafts he has presided over! Not once! Rather than going through all the ways in which Gettleman could have increased his draft capital, I’ll go through the one that I think is the most egregious.

In 2018, the Giants were well poised with the #2 pick in the draft. That draft class had several high-profile QBs in it, but only 2 were considered consensus franchise starters: Sam Darnold and Baker Mayfield. In addition, several teams including the Browns, Bills, Jets, and Cardinals were desperate to find their franchise signal-caller. The Giants should have also been considering a QB themselves. Gettleman was in a prime position to either take a QB himself (and solidify the franchise for years to come), or trade down and net lots of draft capital with which he could build out the roster. Instead, when his turn to pick approached, he chose RB Saquon Barkley, refusing to even listen to trade offers from other teams. Now don’t get me wrong, Barkley is a great player, but taking a RB this high when his team needed a QB and had lots of holes on the roster (especially the offensive line) is simply negligent. When questioned by reporters after the draft about his decision to not listen to offers, he said:

“People call you and they want the second pick of the draft for a bag of donuts, a hot pretzel and a hot dog,” Gettleman explained during his press conference. “Leave me alone. I don’t have time to screw around.”

What makes this decision particularly egregious is that he could have essentially had Barkley and those extra draft picks for the same price. The Jets had already traded up to #3 overall and had been telegraphing their intentions to take Darnold all offseason. As GM of a team that many thought should be considering taking a QB, Gettleman could have simply called the Jets and demanded a few extra draft picks to trade down one spot, similar to what the 49ers did to the Bears in the 2017 draft. Gettleman could have simply traded down one spot, still got his guy, and netted extra draft capital to boot. His refusal to even consider that option shows a complete ignorance of how to maximize the value of his picks that contributed to getting fired in Carolina and may yet result in his dismissal from the Giants.

Well, this article went longer than intended, but I hope it serves useful in re-framing the discussion about the choice between taking Chase Young or trading down. It’s not just a choice between maximizing our draft capital or getting the best player, it’s a careful balancing of how much better Young is than the next best options, an assessment the scouts have spent all year preparing and quantifying. So far, it seems from Rivera’s comments that they believe he is that much better, and if so then they need to follow their board. But whenever they find themselves with numerous players they have similarly ranked, they should absolutely entertain options to trade back.