Amar Cheema and Kieran O’Connor
How do evaluators decide which students enter the top universities and graduate schools, which athletes are considered the world’s best, and even which students receive the highest marks in class? New findings by McIntire Management Professor Kieran O’Connor and Marketing Professor Amar Cheema suggest that judges’ evaluations become more positive over time, pointing to a possibly widespread bias that could influence such ratings and other similar contexts when individuals are evaluated as part of a series, even one that is randomly ordered. Their research was recently published in Psychological Science and featured in The Economist.
In their study, “Do Evaluations Rise with Experience?” the two investigated how the decision-making process in sequential evaluations seems to become easier for raters over time, and can thereby produce inflated scores given later in a series. They found that judges’ ratings of professional dance competitors rose significantly across 20 seasons of the popular television series “Dancing With The Stars”; that university professors gave higher grades when the same course was offered multiple times; and in an experimental test of their hypotheses, that evaluations of randomly ordered short stories became more positive over a two-week sequence.
“As judges complete repeated evaluations, they experience more fluent decision making, producing more positive judgments,” O’Connor and Cheema say. “This seemingly simple bias has widespread and impactful consequences for evaluations of all kinds.”
