Objective: The purpose of this study was to examine longitudinal associations between structural MRI and cognition in a diverse sample. Method: Older adults (n = 444; Mage = 74.5)—121 African Americans, 212 Whites, and 111 Hispanics—underwent an average of 5.3 annual study visits. Approximately half were cognitively normal at baseline (global Clinical Dementia Rating M = 0.5). Of the patients with dementia, most (79%) were diagnosed with Alzheimer’s disease (AD). MRI measures of gray matter volume (baseline and change), and hippocampal and white matter hyperintensity (WMH) volumes (baseline), were used to predict change in global cognition. Multilevel latent variable modeling was used to test the hypothesis that brain effects on cognitive change differed across ethnoracial groups. Results: In a multivariable model, global gray matter change was the strongest predictor of cognitive decline in Whites and African Americans and specific temporal lobe change added incremental explanatory power in Whites. Baseline WMH volume was the strongest predictor of cognitive decline in Hispanics and made an incremental contribution in Whites. Conclusions: We found ethnoracial group differences in associations of brain variables with cognitive decline. The unique patterns in Whites appeared to suggest a greater influence of AD in this group. In contrast, cognitive decline in African Americans and Hispanics was most uniquely attributable to global gray matter change and baseline WMH, respectively. Brain changes underlying cognitive decline in older adults are heterogeneous and depend on fixed and modifiable risk factors that differ based on ethnicity and race.