How to get consistent, fair scores from your review committee.
Reviewer calibration is the work of getting every reviewer on a committee to use the scoring scale the same way. Without it, an abstract's score depends as much on who reviewed it as on how good it is.
Most reviewers aren't unfair on purpose. They just score on different scales. One gives a 4 out of 5 to anything competent, another saves a 4 for the best abstract they've read all year. Average those scores together and the author with the strict reviewer loses, even when the work is stronger. Calibration fixes that before, during and after review.
A few patterns show up in almost every committee. Lenient and strict reviewers use different parts of the scale. Middle scorers give nearly everything a 3, which hides real differences. Halo effects creep in when reviewers see a well-known name or institution. Fatigue changes scores between the first abstract of the evening and the fortieth. And vague criteria leave each reviewer to decide what "originality" or "relevance" means on their own.
None of these is solved by asking reviewers to try harder. They're solved by the setup.
Keep it short: three to five criteria that match what your program actually needs, such as scientific quality, relevance to your audience and clarity. Weight them if they don't matter equally.
The most important part is the anchors. For each criterion, describe what a low, middle and high score looks like in one sentence each. "5: presents new findings with a clear method and results" means far more to a reviewer than "5: excellent." Keep fit for the program, such as a track that's already full, out of the quality score. That's a committee decision, not a reviewer one.
Yes, if your committee has more than a handful of reviewers. Send everyone the same small set of abstracts, five to ten is usually enough, and ask them to score each one with the rubric. Then compare. Where reviewers are far apart, talk through why and tighten the anchor that caused it.
A calibration round takes reviewers an hour or two. It saves far more time later, when the committee would otherwise be arguing over scores that were never comparable.
Have at least two reviewers score each abstract, and three where you can. Mix assignments so reviewers overlap with several colleagues, not always the same one. That overlap is what lets you compare one reviewer's scale with another's.
Cap how many abstracts each reviewer gets, so nobody is scoring the fortieth abstract at midnight. Ask reviewers to declare conflicts of interest up front, and keep them away from abstracts from their own institution or co-authors.
Hiding author names and institutions from reviewers reduces the halo effect, so the abstract is judged on what it says. Many programs blind the first scoring round and give the committee full visibility later, when they need to check for duplicates, balance the program or confirm presenters. Decide this per round rather than for the whole process.
Compare each score with that reviewer's own average instead of the raw number. A 3 from a reviewer who averages 2.5 is a good score. A 3 from a reviewer who averages 4 is a weak one. The standard way to do this is a standardized score: subtract the reviewer's average from each of their scores and divide by how widely their scores spread. Every reviewer's scores then sit on the same scale.
This only works when each reviewer has scored enough abstracts and shares some with colleagues, which is why assignment matters. Also flag abstracts where reviewers disagree sharply. A 1 and a 5 that average to 3 doesn't mean "average." It means the committee should read it.
Watch completion rate against the deadline, and each reviewer's average score and spread as scores come in. A reviewer who has given every abstract a 3, or every abstract a 5, needs a short note now rather than an adjustment later. Our guide to event program KPIs covers the review metrics worth reporting to your committee.
Scores should rank, not decide. Use the adjusted scores to sort abstracts into clear accepts, clear declines and a borderline band, then spend committee time on the borderline band and the high-disagreement abstracts. Record the reason for each borderline decision. It makes the next call for abstracts easier and gives you an answer when an author asks.
In Sessionboard's Call for Papers & Grading, each review round gets its own scorecard and anonymization settings, with weighted rubrics that can vary by session type. You can assign reviewers by tag or track, cap reviews per reviewer and per submission, and watch review progress live. Reviewers score from a portal they open by magic link. AI Evaluators can add a first pass against your weighted rubric, with written reasoning for each score and outliers flagged. Your reviewers can override any AI score, and the decisions stay with the committee.
Want to see it with your own call for abstracts? Request a demo →
Reviewer calibration is the process of getting every reviewer on a committee to apply the scoring scale the same way. It combines clear rubric anchors, a practice round before review, balanced assignments and adjusting scores for each reviewer's tendencies.
Use a short rubric with clear anchors, blind author names and institutions in the first round, have at least two reviewers score each abstract, ask for conflicts of interest up front and adjust for lenient and strict reviewers before the committee meets.
At least two, and three where your reviewer pool allows it. More reviewers per abstract make individual quirks matter less and give you the overlap you need to compare reviewers' scales.
Score normalization adjusts each reviewer's scores relative to their own average and spread, so a strict reviewer's 3 and a lenient reviewer's 4 can be compared fairly. A common method is the standardized score: subtract the reviewer's average from each score and divide by the spread of their scores.
Three to five criteria that match what your program needs, such as quality, relevance and clarity, with weights if they aren't equally important. For each criterion, write one sentence describing a low, middle and high score.
AI can give a first pass against your rubric, explain its reasoning and flag outliers for reviewers to look at. It works best as support for human reviewers, who should be able to override any AI score, with final decisions staying with the committee.
Running review for your next call for abstracts? Sessionboard gives every round its own scorecard, blind review settings and live progress for your committee. See Call for Papers & Grading → or request a demo →
.png)
Stay up to date with our latest news
How to get consistent, fair scores from your review committee.
A step by step guide to running a poster session.
The KPIs that show whether your event program worked.
See how real teams simplify speaker management, scale content operations, and run smoother events with Sessionboard.
