Draft Logic PT II: SD 6’s and the TQ 10, How Do We Standardize Ratings?


Back in San Diego, a young lady (or man) might at best, be about a "6" on the dateability scale, but in the 5th month of deployment on Camp Taqqadum ("TQ"), Iraq, surrounded by a lack of other options, that same six gets graded on a highly subjective (and generous) curve. What we affectionately dubbed a "TQ 10".

My point here? Rating scales are subjective. Assigning numbers and value measurements to a person/performance can be difficult. Two peoples’ views of the same performance could be as far off as…


Well, that.

Last week in Draft Logic PT I, I suggested a matrix to calculate a prospect’s total value as a pickup. In each category, we would assign a number for how much a player helps us or how well a player did in a skill area. The next question was, where do the numbers come from? What prevents every rating from becoming 3 days of Mark and PiC arguing over whether Allen Robinson has 3 star speed or 4.5 star speed?

The answer? Description based value brackets.

The Marine Corps started using this method years ago, and civilian companies have been doing the same for a long time.

The Marine Example: A junior Marine gets two sets of ratings from 0-5. One for how he does his job (proficiency) and one for how he conducts himself in general (conduct). The problem was, when 80,000 people are collecting "Pro/Con" ratings from 10,000 different supervisors, the ratings can be pretty subjective. What rates 4.5 anyways? Next thing you know, the best ratings come from having the most generous raters. When all 80,000 people are competing against each other for promotions, this is a problem.


McKayla is not impressed with your evaluation system

Thus, the Corps released the "IRAM" (Individual Records Administration Manual)

This is a chart that if a Marine does certain tangible things he fits within a certain score range. The evaluator’s judgment can place him higher or lower within a box, but he should never go outside the box that describes him.


This simplified life completely. Brand new junior supervisors who weren’t sure how to assign grades, simply had to read the IRAM.

Junior Marines who thought their ratings were unfair didn’t have to blindly argue. They just had to "read the IRAM" and argue whether the block matched their performance or not.

When someone gave inaccurate ratings, the leader no longer had to debate them or blatantly override them. They just had to say "Why did you think this kid rated a 4.5/4.6? Go back and READ THE DAMN IRAM."

For football evaluators, the same type of block chart would apply. It would be better than simply rating a running back’s vision as a relative value ("well, Jashakeus Thighbones isn’t as good as LaSprintin Marshall, and we gave Marshall a 5, so I guess let’s give Thigbones a 4 and a half").

Instead, we’d watch all of the film available and match Thighbones up to the chart.


Using these description blocks does a few things.

1. It puts all the evaluators on the same page. When the coach or GM is looking at write ups on players from two different scouts, he knows that both reports are written in generally the same language.

2. It allows the team and evaluators to identify what matters ahead of time. These rating blocks work because the team meets and drafts them up at the beginning of the scouting period. This means that everyone comes to an agreement on what constitutes poor and what constitutes great. As the scouts review film, they also know exactly what they are watching out for.

3. It promotes accountability and discussion. Its much easier to argue a standard than an opinion. We Instead of weeks of opinions based on "this kid is awesome, didn’t you see the Auburn game?" "Awesome? He’s Awesome??? Did you see how BAD he was against LSU?" we can cite a hard standard. Its much easier to say "46??? Go read the 45-50 block and tell me it describes what he did this year."

4. It forces the scouts to do data analysis on game film.

New flash: Highlight reels are stupid.

Highlight reels are cherry picked collections of a player’s best plays. By their very nature that means they are statistical outliers. Why we you judge on that? That would be like buying every product based only on what you see in their own commercials. Even going "ten worst, ten best" is a stupid, stupid plan. Its still based off the statistical fringe, and not the normal performance of a player. When you get a stock broker to invest your money, do you want to hear about the best and worst days the stock has ever had, OR do you want to hear what the stock usually does the majority of the time?

We’ll call point #4, "The Shanahan Corollary".

5. I has applications to free agency. When the time comes to argue whether someone has lost a step, this tool comes in. Just like the military version, its best to keep it restricted to time blocks. A player's score for his 2013 performance isn't tainted by the name he built for himself in 2010. It also provides way to give clear, objective feedback to the players. They know where they stand, what they need to improve, and how the team views them from a salary perspective.

Let's be clear. This isn't an easy button.

To use the decision blocks accurately, you have to be willing to sit down with the tapes, turn the game on mute, and just WATCH all this guy’s plays to see what the trend is for him on each area. The end result is that you get a feel for what type of player he actually is, instead of "what kind of cool stuff he did in that one game". The end result may also be a very different opinion from what you started with.

To be upfront, doing something like this is NOT easy. It requires a lot of prep and planning in the beginning. It will make life easier and more effective in the end, but you are front loading a lot of the work.

Between rating scales and the TVI they support, a team has to draft up a requirements breakdown for every position and then a ratings scale breakdown for every requirement. Then, they have to be in agreement on it, and make sure every scout is educated on it.

Watching and re-watching film on all these players is time consuming. It will require a lot of hours from a decent sized staff.

Oh well.

Whatever it takes right? Scouts don’t count against the roster or the cap, so don’t be afraid to hire more. Then don’t be afraid to work them. Building a winning team is not always cheap and its not a 40 hour a week job.


I underestimated this...

As always, I'll take your thoughts in the comments. Thanks.

Log In Sign Up

Log In Sign Up

Please choose a new SB Nation username and password

As part of the new SB Nation launch, prior users will need to choose a permanent username, along with a new password.

Your username will be used to login to SB Nation going forward.

I already have a Vox Media account!

Verify Vox Media account

Please login to your Vox Media account. This account will be linked to your previously existing Eater account.

Please choose a new SB Nation username and password

As part of the new SB Nation launch, prior MT authors will need to choose a new username and password.

Your username will be used to login to SB Nation going forward.

Forgot password?

We'll email you a reset link.

If you signed up using a 3rd party account like Facebook or Twitter, please login with it instead.

Forgot password?

Try another email?

Almost done,

By becoming a registered user, you are also agreeing to our Terms and confirming that you have read our Privacy Policy.

Join Hogs Haven

You must be a member of Hogs Haven to participate.

We have our own Community Guidelines at Hogs Haven. You should read them.

Join Hogs Haven

You must be a member of Hogs Haven to participate.

We have our own Community Guidelines at Hogs Haven. You should read them.




Choose an available username to complete sign up.

In order to provide our users with a better overall experience, we ask for more information from Facebook when using it to login so that we can learn more about our audience and provide you with the best possible experience. We do not store specific user data and the sharing of it is not required to login with Facebook.