Review quality lives in a handful of design choices that hold whatever tool, form or rating scale you use. A high-quality process has seven: a shared, explicit standard of what "good" looks like at each level, multiple credible inputs instead of one lead's memory, evidence over impression, calibration so a score reflects the work instead of the rater, a deliberate split between measuring and developing, an ending in a concrete forward plan, and a cadence that runs and reaches everyone.
Get those right and almost any format works. Get them wrong and no template saves you. When researchers decomposed the ratings of 4,492 managers, about 62% of the variance came from the individual rater's idiosyncrasies and only ~21% from the person's actual performance (Scullen, Mount & Goff, Journal of Applied Psychology, 2000). A review process is high-quality when it engineers that noise back down.
Why isn't a good form enough?
The form is the cheapest part of the system and the least predictive of the outcome. Teams spend their energy choosing a 5-point scale versus a no-ratings model versus a narrative template, then run the same thin process behind it: one manager, one memory, assembled the night before. The result is a number that mostly measures the rater. A single supervisor's rating of overall performance has an interrater reliability of just .52. Each rater is internally consistent, but two raters agree only moderately (Viswesvaran, Ones & Schmidt, Journal of Applied Psychology, 1996).
Inaccuracy is only part of the cost. Feedback can do harm: across 607 studies it raised performance on average, but over a third of feedback interventions made performance worse (Kluger & DeNisi, Psychological Bulletin, 1996). After a century of research there is little evidence that appraisal on its own improves performance at all (DeNisi & Murphy, Journal of Applied Psychology, 2017). So "high-quality" has to mean more than "a slicker form." It means the design choices that decide whether the process produces a fair, defensible number and whether anything changes afterwards.
What makes a review process high-quality?
There are seven design choices. They sit underneath whatever scale you pick, and they decide the quality you get.
- A shared, explicit standard. A written career framework that says, in concrete behaviours, what "good" looks like at each role and level. This is the practitioner standard for defining roles (CIPD, Competence and competency frameworks). Without a common bar, every reviewer invents their own and a "Grade 3" means something different to each lead.
- Multiple credible inputs. Evidence from the leads, peers and (where relevant) clients the person worked with, in place of one manager's recency-weighted recollection. More independent vantage points cancel noise a single rater can't, which answers that .52 reliability problem.
- Evidence over impression. Every judgment anchored to a concrete example tied to the framework instead of a global gut feel. Anchoring blunts recency, halo and leniency bias and turns "I think she's a 3" into "here's the behaviour at this level."
- Calibration across reviewers. A deliberate step where the leads who rated different people reconcile their scores against the shared bar. This attacks the 62% rater-idiosyncrasy problem head-on (Scullen, Mount & Goff, 2000) and keeps a score about the work instead of the rater's generosity.
- Separate "measure" from "develop." A defensible rating and an honest growth conversation pull in opposite directions. When both ride on one number, candour gets punished. High-quality processes design the two so a person can hear hard feedback without it threatening their grade.
- End in action. A review changes behaviour only with a concrete, forward-looking plan and follow-up. Improvement after feedback is generally small and conditional. It shows up when coaching and a plan follow the rating (Smither, London & Reilly, Personnel Psychology, 2005).
- It runs and it reaches everyone. On cadence, with high completion, even when the team is billable. If only half the firm completes your well-designed process, you are running a pilot.
What drives a rating?
Mostly the rater, unless you design against it. When Scullen, Mount and Goff decomposed the ratings of 4,492 managers, the largest single share of the variance traced to the quirks of whoever was holding the pen, ahead of how people performed (2000). That is the empirical case for choices 2, 3 and 4 above: multiple inputs, evidence and calibration exist to shrink the violet bar and grow the teal one.
Does the rating scale matter at all?
It matters, though less than the design choices. The well-known revolt against ratings came from cost and futility: 58% of executives said their performance-management approach drove neither engagement nor performance, and Deloitte alone was spending close to two million hours a year on forms, meetings and ratings (Buckingham & Goodall, Harvard Business Review, 2015). The dissatisfaction hasn't gone away. As of 2023, only 29% of HR leaders were confident their current process effectively helps people perform (Gartner, 2023).
Whether you abolish ratings or keep them, treat the scale as a downstream choice. Ratings, no-ratings, or both can each be high-quality if the seven design choices are in place, and each is low-quality without them. Pick the scale that fits how you make staffing and promotion decisions, then spend your real effort on the standard, the inputs, the calibration and the follow-up.
Why does this matter for agencies and consulting boutiques?
The generic answer assumes one manager who watched the person all year. In a project-based firm, that manager rarely exists. People are staffed across several engagements under different leads, rated per engagement by whoever ran it, and the score feeds grades, raises and the partner track. Here the 62% rater-idiosyncrasy finding (Scullen, Mount & Goff, 2000) decides whether a promotion case is fair or political.
In your firm, each design choice does a specific job. A shared bar lets different leads mean the same thing by a grade. Multiple credible inputs reconstruct a year no single lead saw. Calibration across leads makes a rating defensible when it decides the partner track. And because billable pressure eats the time, the process has to be light enough to run at high completion without burning the hours that pay for it. Get this right and people trust the review as fair. Get it wrong and the cost lands on the senior talent a boutique can least afford to lose, because an opaque verdict reads as an unfair one.
A quick self-check: is your review process high-quality?
Score one point per "yes". Six or seven means your process is high-quality. Four or fewer means you're relying on the form to do work it can't.
- There is a written, shared standard (a career framework) for what "good" looks like at each level.
- Ratings draw on multiple credible inputs beyond one lead's memory.
- Every judgment is anchored to concrete evidence tied to the framework.
- Leads calibrate scores against the shared bar before they're final.
- "Measure" and "develop" are designed so candour isn't punished.
- Every review ends in a concrete forward plan with a follow-up date.
- The cycle runs on cadence and reaches everyone, even when the team is billable.
FAQ
Is a performance review process about the rating scale?
No. The scale (5-point, no-ratings, narrative) matters far less than the design choices around it: a shared standard, multiple inputs, evidence, calibration, and a forward plan. Most of the variance in a rating traces to the rater rather than the performance (Scullen, Mount & Goff, 2000), so you win quality by engineering that noise down, and re-drawing the form does little.
What's the single biggest driver of a fair rating?
Calibration on top of multiple inputs. A single supervisor's rating of overall performance has an interrater reliability of only .52 (Viswesvaran, Ones & Schmidt, 1996). Adding independent vantage points and then reconciling them against a shared bar makes a score about the work rather than the rater.
Should we drop ratings entirely?
Not automatically. Cost and low usefulness drove the backlash: 58% of executives saw no engagement or performance benefit (Buckingham & Goodall, 2015), and only 29% of HR leaders were confident in their process in 2023 (Gartner, 2023). But ratings, no-ratings or a hybrid can each be high-quality with the seven design choices in place, and low-quality without them.
Why doesn't a good review automatically improve performance?
Appraisal on its own rarely does (DeNisi & Murphy, 2017), and feedback can backfire: over a third of interventions reduced performance (Kluger & DeNisi, 1996). Improvement shows up when a plan and follow-up come after the conversation (Smither, London & Reilly, 2005), which is why "end in action" counts as a design choice.
How do we keep quality high when everyone is billable?
Make the heavy parts one-time and the recurring parts light. You build the framework and templates once. Each cycle then mostly means assembling inputs you can collect asynchronously around client work, plus a short calibration. High completion comes from a process that fits the billable week.