ChatGPT in education: An effect in search of a cause

As researchers rush to investigate the potential of AI tools like ChatGPT to enhance learning, a number of well-known and well-documented pitfalls threaten the validity of this emerging research. Issues of media comparison research, where the confounding of instructional methods and technological affordances are unrecognized, render effects uninterpretable. Using a recent meta-analysis by Deng et al. (2025) as example, we revisit key insights from the media/methods debate to highlight recurring conceptual challenges in studying AI for education. We identify three considerations needed to interpret this research: the precise nature of the experimental treatment, the activities of the control group, and the validity of the outcome measures as indicators of learning. We contrast the nascent research on ChatGPT with the well-established literature on Intelligent Tutoring Systems (ITS). Our analysis underscores the importance of defining clear research questions, ensuring methodological rigor, and resisting the allure of “fast science.”

osf.io/preprints/psyarxiv/t6uzy_v2

About
Latest Posts

Ryan Watkins

Professor at George Washington University

I am a Professor with Human-Technology Collaboration and Educational Technology programs at George Washington University in Washington DC. I have written 12 books and more than 100 articles, and I co-host of the Parsing Science podcast where scientists tell the stories behind their research. I am also the developer of the WeShareScience.com online platform for sharing research videos, and SciencePods.com where researchers can create free podcasts about their science. My research interests include human interactions with intelligent machines, needs, needs assessments, and instructional design.