McIntire Information Technology Professor Sarah Lebovitz recently won two important awards recognizing her research in the field of artificial intelligence: the AIS Best Information Systems Publications Award and the MIS Quarterly Paper of the Year.
Her paper “Is AI Ground Truth Really True? The Dangers of Training and Evaluating AI Tools Based on Experts’ Know-What” earned the recognition on successive days, Dec. 11 and 12, 2022. Co-authored with Professor Natalia Levina of New York University and Professor Hila Lifshitz-Assaf of Warwick Business School, the paper details an in-depth field study investigating how managers evaluate the merit of artificial intelligence tools for making critical organizational decisions.
Lebovitz and her colleagues explain how AI performance claims must be evaluated to find the “ground truth” behind datasets that teach algorithms how to make predictions; for managers evaluating these tools, they must be sure that ground truth is high quality. Their research shows that the ground truth of many AI tools fails to replicate the full decision-making process that experts use, only capturing observable aspects of deep knowledge. The paper also offers advice for managers who discover that ground truth isn’t actually true—when less-than-ideal processes have been used to reach critical decisions in high-risk situations with ethical consequences.
According to AIS, its Best Information Systems Publications Award is given to five papers annually, as determined by the organization’s senior scholars. Established “to recognize the breadth of high quality work that is being published in the information systems discipline,” the accolade is designed “to bring outstanding papers across a range of journals to the attention of the IS community and to give due credit to the journals in which they are published.”
MIS Quarterly, a highly regarded scholarly journal focused on issues concerning the development of IT-based services; resources; and “the use, impact, and economics of IT with managerial, organizational, and societal implications,” honors one paper annually with its Paper of the Year award.
Read more about Lebovitz’s award-winning paper here.
