When one person is staffed across several client projects, no single lead saw the whole cycle. Each watched a different slice, under a different scope, difficulty and client. You build a fair review on purpose from those slices. Averaging their scores won't get you there. You judge every engagement against one shared bar, weight each input by how much that lead saw, treat disagreement between projects as a signal to investigate, normalize for how hard each engagement was, and calibrate across the leads so one generous or harsh reviewer, or the single loudest project, doesn't decide the outcome. The result reflects the whole period, and the engagement that left the longest paper trail gets no extra say.
Why can't you average the scores? Most of any single score comes from the scorer. When researchers decomposed managers' ratings, about 62% of the variance came from the rater's idiosyncrasies and only ~21% from actual performance (Scullen, Mount & Goff, Journal of Applied Psychology, 2000), and two competent leads rating the same person agree only moderately, with interrater reliability around .52 (Viswesvaran, Ones & Schmidt, Journal of Applied Psychology, 1996). Averaging N partial, rater-flavoured views leaves that noise in the number. You remove it only by assembling the views on purpose.
Why is reviewing someone across several client projects so hard?
Nobody ever saw the thing you're rating whole. In a project-based firm a consultant or creative is spread across several engagements a cycle, and each lead saw only their slice: a different scope, a different duration, a different client, a different level of difficulty. Four problems follow from that:
- No one has the full picture. Each lead reviews a fraction and rates what they saw, which is reasonable. Stitch those fractions together without a method and you get a blur where the person should be.
- The loudest project dominates. The longest engagement, the most visible client win, or the project with the most documentation crowds out a quiet but harder piece of work. The project with the most paperwork gets more weight than it has earned.
- Each lead is a noisy instrument. A single reviewer's score is mostly their own perspective and leniency (Scullen, Mount & Goff, 2000). One vantage point is only moderately reliable (Viswesvaran, Ones & Schmidt, 1996).
- The bar drifts between leads. "Strong" on a demanding partner's engagement and "strong" on an easy retainer are not the same standard unless someone made them the same on purpose.
What does "fair" mean when every lead saw a different slice?
Fair means you judge every engagement against one shared bar instead of each lead's private standard. Before you reconcile anything, you need a single definition, in observable behaviours, of what "good" looks like at this person's level: what a consultant, a senior, a principal is expected to do, written down in advance (CIPD, Competence and competency frameworks). When the bar is explicit and common, a rating from a demanding partner and a rating from a gentle account lead mean the same thing, because both leads measured against the same yardstick rather than against their own mood.
It is also why you shouldn't average. If each score is partly the rater (Scullen, Mount & Goff, 2000) and single-rater agreement is only about .52 (Viswesvaran, Ones & Schmidt, 1996), treat the scores as evidence and weigh each one against a common standard. Kahneman, Sibony and Sunstein call the discipline that contains this kind of variability "decision hygiene": a shared scale, structure, and calibration (Kahneman, Sibony & Sunstein, Noise, 2021).
How do you map who saw what before you judge?
Build a coverage map first, because you can't weight inputs you haven't laid out. Before anyone proposes a rating, capture five things for each engagement the person worked on:
- The lead. Who is giving the input, and in what role did they see the person.
- Exposure. How much of the person's time and the cycle that engagement covered: a six-month staffing or a two-week overflow task.
- Scope. What the person was responsible for, and what this lead could observe (a lead who never saw them run a client meeting can't rate client presence).
- Difficulty and context. How hard and how ambiguous the engagement was: the client, the stakes, the constraints.
- Evidence. The concrete examples tied to the shared bar, gathered before any score is named.
The coverage map turns "here are five scores" into "here is who saw what, how much, and under what conditions". You can only combine the scores fairly on that basis.
How should you weight feedback across different leads and projects?
Weight each input by the lead's exposure and proximity to the work, and ignore how much they wrote. The lead who staffed the person for most of the cycle, on work central to their role, should count more than the lead who supervised a two-week overflow task, however detailed that lead's write-up. Proximity beats volume: a reviewer who watched the person handle the client is a better witness on client skill than a partner who only saw the final deck.
Averaging does the opposite. It treats a two-week task and a six-month engagement as equal votes. Exposure-weighting treats them as a glimpse and a long look. Collecting more inputs won't fix this on its own: multisource feedback improves things only when someone structures it and follows up, and gathering scores alone does little (Smither, London & Reilly, Personnel Psychology, 2005). You still have to do the weighting, the synthesis, and the follow-up.
What do you do when the projects disagree, and how do you handle difficulty and bench time?
Treat divergence as a signal to investigate instead of splitting it down the middle. When one lead rates the person a clear "exceeds" and another a flat "meets," the average ("slightly above") is usually the least true answer available. The useful question is why they diverge:
- Stretch versus steady state. One engagement pushed the person past their level, and the other kept them comfortable. Both are real. Rate what they did at the top of their range, and how consistently they did it.
- Difficulty of the engagement. A strong showing on a hard, ambiguous, high-stakes engagement is worth more than the same score on an easy retainer. Normalize for difficulty so the person who did well under pressure isn't rated below someone who coasted on something simple.
- Client fit. A bad client-personality fit can depress an otherwise strong performer's reviews on one project. Name it as context. Don't bake it into the number as if it were a skill gap.
Be honest about thin coverage and bench time, too. If no lead saw enough to judge a dimension, say so and don't manufacture a rating from a fragment. Time on the bench between staffings is no evidence of poor performance, so don't score it as such. Note it as a coverage gap and rate on the engagements you have. A century of appraisal research is blunt about this: the value lies in the reliable, useful judgment around the score (DeNisi & Murphy, Journal of Applied Psychology, 2017).
How do you calibrate across leads so one project (or one reviewer) doesn't skew it?
Calibrate the per-engagement inputs against the shared bar before anything rolls up into a decision. In calibration you compare the weighted, difficulty-adjusted inputs from different leads side by side so the outliers show up: the lead who rates everyone high to avoid a hard conversation, the partner who marks everyone down, the single dominant project trying to stand in for the whole cycle. You correct against the bar, with no quota to hit, so that one reviewer's leniency or severity, or one loud engagement, can't decide the outcome. This is the "decision hygiene" that reduces both bias and noise in professional judgment (Kahneman, Sibony & Sunstein, Noise, 2021), and it's the practitioner standard for fair, defensible appraisal (CIPD, Performance management).
Then end where it counts: one synthesized growth picture and a forward plan, kept separate from the calibrated rating. The rating answers "where did this person land against the bar this cycle". The growth plan answers "what happens next." Keep them apart, because feedback fused to a verdict can backfire: across 607 studies feedback raised performance on average, but over a third of interventions made it worse (Kluger & DeNisi, Psychological Bulletin, 1996). The synthesis has to look forward to be worth doing.
Why does this matter more for agencies and consulting boutiques?
In a client-services firm this is the default shape of every review. A consultant or creative is staffed on several engagements a year under different leads and partners, rated per engagement, and those per-engagement ratings roll up into calibration, promotion, and up-or-out on a partner track. Billable pressure guarantees the coverage is uneven: no one lead saw the whole person, and the engagements that happened to generate the most visible output aren't always the ones that show the most growth. Here the 62% rater-idiosyncrasy finding (Scullen, Mount & Goff, 2000) marks the gap between a defensible promotion case and a political one.
So the fair-review method belongs in how the firm runs. A coverage map stops the loudest project from standing in for the cycle. Exposure-weighted inputs keep a two-week overflow task from outvoting a six-month engagement. Difficulty normalization protects the person who did hard things well. Calibration across leads and partners catches the reviewer whose ratings say more about them than about the people they rated. That is what makes the result defensible when it decides who makes partner. And because a senior consultant reads an unfair verdict as exactly that and leaves over it, defensible reviews are also retention. Build the whole thing light enough to run between billable hours, or it won't run at all.
A quick self-check: is your multi-project review fair?
Score one point per "yes". Six or seven means your process assembles a fair picture. Four or fewer means the loudest project is still deciding the rating.
- You judge every engagement against one shared, written bar (observable behaviours at each level).
- You build a coverage map (lead, exposure, scope, difficulty, evidence) for each engagement before anyone names a score.
- You weight inputs by how much each lead saw, and the length of a write-up counts for nothing.
- You investigate divergence between projects (stretch, difficulty, fit) instead of averaging it away.
- You normalize for engagement difficulty so a hard project done well isn't undersold.
- You flag thin coverage and bench time as gaps and never score them as poor performance.
- You calibrate ratings across leads and partners before they feed promotion or partner-track decisions.
FAQ
How do you review someone who worked on several client projects with different leads?
Don't average the scores. Assemble a picture: set one shared bar for the person's level, build a coverage map of who saw what (lead, exposure, scope, difficulty, evidence), weight each input by how much that lead observed, investigate where projects disagree instead of splitting the difference, normalize for how hard each engagement was, and calibrate across leads before anything rolls up into a decision. Averaging fails because most of a single score comes from the rater (Scullen, Mount & Goff, 2000).
Why not just take the average of each engagement rating?
The scores aren't clean measurements. The individual rater flavours each one heavily (about 62% rater idiosyncrasy vs ~21% performance, per Scullen, Mount & Goff, 2000), and single-rater agreement is only about .52 (Viswesvaran, Ones & Schmidt, 1996). Averaging noisy, partial, unequal-exposure views produces a confident-looking number that hides all of that. Weighting by exposure and calibrating against a shared bar make the inputs comparable.
What do you do when two projects give very different ratings?
Treat the gap as information you need to explain. Ask why: was one engagement a stretch beyond the person's level and the other steady-state? Was one far harder or more ambiguous? Was there a client-fit problem that depressed otherwise strong work? Resolve the divergence against the shared bar and the difficulty of each engagement, and record the reasoning. A "slightly above" average is usually the least true answer.
How should bench time or thin coverage affect a review?
Neither should count as poor performance. Time on the bench between staffings is a coverage gap and tells you nothing about performance. Note it and rate on the engagements you have. If no lead saw enough to judge a dimension, say so rather than inventing a rating from a fragment, since an honest gap is worth more than manufactured precision.
Why does this problem hit agencies and consulting boutiques hardest?
Their people are staffed across multiple engagements under different leads every cycle, rated per engagement, and those ratings feed calibration, promotion, and partner-track decisions. Coverage is always uneven under billable pressure. Without a coverage map, exposure-weighting, difficulty normalization, and cross-lead calibration, the most visible project decides someone's career in place of the fullest picture.