Schools are being encouraged to adopt AI tools partly on the strength of research reviews that report large gains in learning, and a new audit argues those reviews do not hold up. Patrick O’Neill, an associate professor at Ivy Tech Community College in Indiana, examined 14 meta-analyses, studies that combine the results of many smaller studies into one overall estimate. In a preprint discussed by Education Week on November 2, he concludes that none of them gives a valid basis for its claims. None clearly defined the kind of AI use it was measuring, and none properly checked for publication bias, the tendency for studies with good results to be published while disappointing ones are left out.

Some of the problems were basic. A study of an elementary school physical education class was grouped with one on simulations for medical students, a few included studies did not use AI at all, and results as different as test knowledge, motivation and mental health were added together. Some of the underlying studies contained impossible statistics or references that appear to be invented, including ChatGPT research dated before the tool was public. The reviews appeared in respected journals and have been cited thousands of times. O’Neill asks journals to require authors to share their data and to tighten peer review. A separate Stanford review of more than 800 studies found only 20 with a strong cause-and-effect design, and none of those involved schoolchildren.