Join us on Zoom for a Joint Quantitative Psychology Brownbag with Dr. Joshua Gilbert (Education Policy and Program Evaluation, Harvard Graduate School of Education)
This event is online only. Please join us using this meeting link.
Title: Estimating heterogeneous treatment effects with item-level data: Insights from item response theory
Abstract: Analyses of heterogeneous treatment effects (HTE) are common in applied causal inference research. However, when outcomes are latent variables assessed via psychometric instruments such as educational tests, standard methods ignore the potential HTE that may exist among the individual items of the outcome measure. Failing to account for “item-level” HTE (IL-HTE) can lead to both underestimated standard errors and identification challenges in the estimation of treatment-by-covariate interaction effects. We demonstrate how Item Response Theory (IRT) models that estimate a treatment effect for each assessment item can both address these challenges and provide new insights into HTE generally. This study articulates the theoretical rationale for the IL-HTE model and demonstrates its practical value using 75 datasets from 48 randomized controlled trials containing 5.8 million item responses in economics, education, and health research. Our results show that the IL-HTE model reveals item-level variation masked by single-number scores, provides more meaningful standard errors in many settings, allows for estimates of the generalizability of causal effects to untested items, resolves identification problems in the estimation of interaction effects, and provides estimates of standardized treatment effect sizes corrected for attenuation due to measurement error.
About Joshua Gilbert: Joshua Gilbert is a PhD candidate in Education Policy and Program Evaluation at the Harvard Graduate School of Education, where he works with James Kim and Luke Miratrix. His research interests include the intersection of causal inference and psychometric methods, and his work has been published in journals such as Developmental Psychology, Journal of Educational Psychology, Journal of Educational and Behavioral Statistics, Behavior Research Methods, Psychological Methods, and others.
Joint Quantitative Psychology Brownbags are organized by psychology programs at universities around the nation, including: The Ohio State University, University of Notre Dame, University of Maryland, College Park, University of North Carolina at Chapel Hill, University of Virginia, Vanderbilt University and University of South Carolina. Participating programs also include those from York University, McGill University, University of Missouri and University of British Columbia.