The problem with over-relying on quantitative evidence of validity

When presenting evidence of validity, social scientists tend to rely on popular methods such as reliability, regression, and confirmatory factor analysis. The quality of the model output is often determined by fairly arbitrary cutoff values, some of which arise from simulation studies that do not generalize to the user’s model subspace. Despite decades of warnings about improper use and the general inadequacy of relying solely on quantitative evidence of validity, quantitative methods continue to dominate the validation literature. In this paper, I argue that this over-reliance on quantitative evidence may have inadvertently caused psychologists to design survey instruments to fit psychometric models rather than psychometric theories. This may have unintentionally caused the structure of most psychological properties to be thought of as continuous, and made unidimensionality and conditional independence desirable properties of latent variables and corresponding survey items. Further, the quantitative imperative in publishing may have resulted in a dearth of qualitative evidence, such as testing items for interpretability and meaning in the population of interest. I encourage researchers to revisit the evidence that has been presented in support of instrument validation and use and to allow theory to guide psychological measurement.

Ryan Watkins

Professor at George Washington University

I am a Professor with Human-Technology Collaboration and Educational Technology programs at George Washington University in Washington DC. I have written 12 books and more than 100 articles, and I co-host of the Parsing Science podcast where scientists tell the stories behind their research. I am also the developer of the WeShareScience.com online platform for sharing research videos, and SciencePods.com where researchers can create free podcasts about their science. My research interests include human interactions with intelligent machines, needs, needs assessments, and instructional design.