Some have argued that having students conduct rigorous replications of published studies would provide benefits to both psychological science and the students themselves. However, while it seems clear that replications are beneficial to psychological science, there is little empirical evidence that having students conduct replications provides benefits to students. In this study, I conducted a preliminary test (N = 37) of one purported benefit to students of conducting a classroom replication: the development of scientific critical thinking skills. Students completed a 1-term research methods course centered on a class replication project. I assessed students’ critical thinking development after completing the course. The results were largely inconclusive, showing no significant change in performance between a pretest (M = 11.00, SD = 3.44) and a posttest (M = 10.30, SD = 2.62), t(36) = −1.21, p = .23. This study highlights the need for additional research on the question of whether having students conduct replications provides educational benefits beyond those offered by other pedagogical methods.