FanPost

Draft Logic PT II: SD 6’s and the TQ 10, How Do We Standardize Ratings?

2010-10-07-strip_71_deployment_goggles_web_medium

Back in San Diego, a young lady (or man) might at best, be about a "6" on the dateability scale, but in the 5th month of deployment on Camp Taqqadum ("TQ"), Iraq, surrounded by a lack of other options, that same six gets graded on a highly subjective (and generous) curve. What we affectionately dubbed a "TQ 10".

My point here? Rating scales are subjective. Assigning numbers and value measurements to a person/performance can be difficult. Two peoples’ views of the same performance could be as far off as…

Ashley-wagner-u-s_medium

Well, that.

Last week in Draft Logic PT I, I suggested a matrix to calculate a prospect’s total value as a pickup. In each category, we would assign a number for how much a player helps us or how well a player did in a skill area. The next question was, where do the numbers come from? What prevents every rating from becoming 3 days of Mark and PiC arguing over whether Allen Robinson has 3 star speed or 4.5 star speed?

The answer? Description based value brackets.

The Marine Corps started using this method years ago, and civilian companies have been doing the same for a long time.

The Marine Example: A junior Marine gets two sets of ratings from 0-5. One for how he does his job (proficiency) and one for how he conducts himself in general (conduct). The problem was, when 80,000 people are collecting "Pro/Con" ratings from 10,000 different supervisors, the ratings can be pretty subjective. What rates 4.5 anyways? Next thing you know, the best ratings come from having the most generous raters. When all 80,000 people are competing against each other for promotions, this is a problem.

Mckayla-maroney-pissed_medium

McKayla is not impressed with your evaluation system

Thus, the Corps released the "IRAM" (Individual Records Administration Manual)

This is a chart that if a Marine does certain tangible things he fits within a certain score range. The evaluator’s judgment can place him higher or lower within a box, but he should never go outside the box that describes him.

Iram-3-conduct-table_medium

This simplified life completely. Brand new junior supervisors who weren’t sure how to assign grades, simply had to read the IRAM.

Junior Marines who thought their ratings were unfair didn’t have to blindly argue. They just had to "read the IRAM" and argue whether the block matched their performance or not.

When someone gave inaccurate ratings, the leader no longer had to debate them or blatantly override them. They just had to say "Why did you think this kid rated a 4.5/4.6? Go back and READ THE DAMN IRAM."

For football evaluators, the same type of block chart would apply. It would be better than simply rating a running back’s vision as a relative value ("well, Jashakeus Thighbones isn’t as good as LaSprintin Marshall, and we gave Marshall a 5, so I guess let’s give Thigbones a 4 and a half").

Instead, we’d watch all of the film available and match Thighbones up to the chart.

Chart1_medium

Using these description blocks does a few things.

1. It puts all the evaluators on the same page. When the coach or GM is looking at write ups on players from two different scouts, he knows that both reports are written in generally the same language.

2. It allows the team and evaluators to identify what matters ahead of time. These rating blocks work because the team meets and drafts them up at the beginning of the scouting period. This means that everyone comes to an agreement on what constitutes poor and what constitutes great. As the scouts review film, they also know exactly what they are watching out for.

3. It promotes accountability and discussion. Its much easier to argue a standard than an opinion. We Instead of weeks of opinions based on "this kid is awesome, didn’t you see the Auburn game?" "Awesome? He’s Awesome??? Did you see how BAD he was against LSU?" we can cite a hard standard. Its much easier to say "46??? Go read the 45-50 block and tell me it describes what he did this year."

4. It forces the scouts to do data analysis on game film.

New flash: Highlight reels are stupid.

Highlight reels are cherry picked collections of a player’s best plays. By their very nature that means they are statistical outliers. Why we you judge on that? That would be like buying every product based only on what you see in their own commercials. Even going "ten worst, ten best" is a stupid, stupid plan. Its still based off the statistical fringe, and not the normal performance of a player. When you get a stock broker to invest your money, do you want to hear about the best and worst days the stock has ever had, OR do you want to hear what the stock usually does the majority of the time?

We’ll call point #4, "The Shanahan Corollary".

5. I has applications to free agency. When the time comes to argue whether someone has lost a step, this tool comes in. Just like the military version, its best to keep it restricted to time blocks. A player's score for his 2013 performance isn't tainted by the name he built for himself in 2010. It also provides way to give clear, objective feedback to the players. They know where they stand, what they need to improve, and how the team views them from a salary perspective.

Let's be clear. This isn't an easy button.

To use the decision blocks accurately, you have to be willing to sit down with the tapes, turn the game on mute, and just WATCH all this guy’s plays to see what the trend is for him on each area. The end result is that you get a feel for what type of player he actually is, instead of "what kind of cool stuff he did in that one game". The end result may also be a very different opinion from what you started with.

To be upfront, doing something like this is NOT easy. It requires a lot of prep and planning in the beginning. It will make life easier and more effective in the end, but you are front loading a lot of the work.

Between rating scales and the TVI they support, a team has to draft up a requirements breakdown for every position and then a ratings scale breakdown for every requirement. Then, they have to be in agreement on it, and make sure every scout is educated on it.

Watching and re-watching film on all these players is time consuming. It will require a lot of hours from a decent sized staff.

Oh well.

Whatever it takes right? Scouts don’t count against the roster or the cap, so don’t be afraid to hire more. Then don’t be afraid to work them. Building a winning team is not always cheap and its not a 40 hour a week job.

Haslam__pilot_j_investigation__a_distrac_1209890000_1732355_ver1

I underestimated this...

As always, I'll take your thoughts in the comments. Thanks.